Get startedGet started for free

Building focused pipelines

1. Building focused pipelines

Welcome back! Previously, you used a Copy job for a straightforward ingestion task. A pipeline is a more flexible tool. It helps when an ingestion task needs multiple steps, custom logic, or coordinated automation.

2. A pipeline is a planned route

Think of a pipeline as a planned railway route. The source is the departure station, the destination is the arrival station, and activities are the operations along the route. One pipeline run is one completed journey along that route. In Fabric, the pipeline editor lets you inspect each activity on the canvas and review its source and destination settings.

3. Starting a new pipeline

A new pipeline offers two ways to start: a blank canvas or guidance. Practice with sample data is the easiest. Its predefined template supplies a working source and copy activity, so you only need to choose the destination lakehouse.

4. Pipeline canvas

The pipeline canvas shows the pipeline's steps. Output reports whether each activity succeeded.

5. Copy data activity

A Copy data activity is a common pipeline activity. It works like a standalone Copy job, but its source and destination are defined inside the pipeline. It reads from a source and writes to a destination, such as a lakehouse table. After you select Run, Fabric shows run details and activity status.

6. Pipeline activity groups

The activity catalog groups activities by the work they perform. Move and transform activities move or process data. The Copy data activity belongs to this group. Control flow coordinates sequence, repetition, conditions, and dependencies. Metadata and validation activities read file properties or look up values before data moves. Notification activities send completion messages or failure alerts.

7. Parameters

A parameter is a value you supply before a run begins. It can specify a batch date, source file, or destination name, and different runs can receive different values. Each value stays fixed for the duration of that run, and activities and expressions use it as input.

8. Variables

A variable is a named value that activities can read and update during a run. Unlike a parameter, its value can change while the run is in progress. You might set a variable from a parameter value, store a temporary table name, or count the files processed.

9. Adding parameters and variables

To add either one, select an empty area of the canvas, open the Parameters or Variables tab, and select New. For example, a source_file parameter might default to orders.csv, and a files_processed variable to 0, which activities increase as each file loads.

10. Secure data connection

For each activity that reads data, you select a data source and set up a secure connection to it before any data moves.

11. Authenticating a secure connection

Next, you set the credentials for that connection. These credentials need to remain secret, so Fabric stores them in the connection rather than in parameters or variables.

12. Two ways to start a pipeline

You can run a pipeline on demand or on a schedule. An on-demand run starts when you select Run. It is useful for testing or urgent work. A scheduled run starts automatically at a configured frequency. It keeps recurring ingestion consistent. To add one, select Schedule on the Home tab; a pipeline can have several schedules.

13. Checking a pipeline run

A successful run status doesn't prove that the data arrived. First, check run details and activity status. Then open the destination lakehouse from the OneLake catalog and confirm that the expected table or files are present.

14. Let's practice!

Now let's practice running, classifying, parameterizing, and scheduling Fabric pipelines.

Create Your Free Account

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.