1. Introduction to Fabric Data Pipelines
In this video, we will cover the fundamentals of data pipelines, explore how to make them dynamic with parameters and variables, and finally learn to schedule them.
2. Data Pipelines in Microsoft Fabric
In Fabric's Data Factory, Data Pipelines handle data movement, ingestion, and transformation, while Dataflows offer 300+ UI-based transformations with real-time previews in Power Query.
In this lesson, we’ll focus on Data Pipelines. Pipelines automate the ETL process with minimal coding, seamlessly integrating various data sources. With a rich set of activities, they manage data ingestion and transformation efficiently. You can also run pipelines manually or schedule them with triggers for full control.
3. Activities in Data Pipelines
Let's dive deeper into activities within Data Pipelines
Activities in Data Pipelines are like steps in a car assembly line. Each activity performs a specific task, such as attaching wheels or painting the body. Similarly, pipeline activities are sequenced to process and transform data step-by-step, creating the final data outcome.
4. Types of Activities
Activities in Data Pipelines are categorized based on their roles.
First, Move & Transform activities handle data transfer and transformations, with the Copy Data activity moving data from one source to another.
Next, we have Metadata & Validation activities, which help with data quality and key information extraction. For instance, the Get Metadata activity retrieves file details like names, sizes, and timestamps, while Lookup dynamically selects items for later tasks.
Control Flow activities manage task execution order. For example, If Conditions execute tasks based on a conditions and ForEach iterate through a set of items.
5. Types of Activities
Next, Orchestrate activities synchronize multiple processes, like the Invoke Pipeline activity, which runs another pipeline within the current one.
Then we have, Notifications activities, like 365 Outlook or Teams, which send alerts or updates to notify stakeholders.
Finally, Transform activities perform data manipulations. The Notebook activity runs code to modify data, while the Stored Procedure activity executes predefined SQL scripts.
6. Pipeline Parameters and Variables
Next, let's look at paramaters and variables. These are essential for controlling and managing pipeline behavior dynamically
Parameters are set at runtime and allow you to adjust key inputs, like file paths, that impact the entire pipeline. They have a global scope, meaning they influence every part of the pipeline.
Variables, on the other hand, track and modify data within the pipeline as it moves through activities, with a local scope. They are commonly used to track execution status like 'In Progress' or 'Completed' and pass it to later activities for managing conditional logic or perform loggings.
7. Pipeline runs
In Microsoft Fabric, a pipeline run is when all activities in your pipeline are executed from start to finish.
You can run pipelines on-demand from the Fabric UI for immediate execution or set them to run on a regular schedule.
Each run is tracked by a unique Run ID, which you can monitor in the Monitor tab to stay updated on progress.
Before running, be sure to use the Validate option to catch any errors, ensuring a smooth pipeline experience.
8. Let's practice!
Now that you understand Data Pipelines, let’s start building them in Fabric!