Adding and accessing data in Fabric

1. Adding and accessing data in Fabric

OneLake and the items inside it aren't useful until they hold data. In this video, we'll compare four ways to bring in data: uploading, pipelines, Dataflow Gen2, and Eventstreams.

2. Four ways to bring data in

Think of Fabric as a distribution center. Goods can arrive by hand, by a truck on a scheduled route, through a station that unpacks and relabels them, or on a conveyor belt that never stops. Fabric has an equivalent method for each, and we'll discuss them in that order.

3. Uploading data

Uploading data is like carrying a single package into the center by hand: we choose a file and place it where we want it. That works well for small files and one-off tasks, but we can't carry the same package in by hand every morning. When data has to arrive on a regular schedule, we need something automated.

4. Pipelines

A pipeline is that automated delivery route. It collects a shipment from its source, carries it to the center, and drops it at the right destination, and it can run on its own. In Fabric, a pipeline orchestrates a sequence of steps that move or process data. Pipelines are especially useful for batch ingestion, where data arrives in groups every hour, night, or week.

5. Choosing the pipeline data source

Let's build one. Because a pipeline transfers data, it needs both a source and a destination. The source is where the data comes from. There are several ways to configure it, but the quickest is the Practice with sample data option. It creates a sample source and sets up a Copy data activity, which copies data from the source to the destination.

6. Choosing the destination

The destination is where the data travels to. For a sample pipeline, we choose it from a lakehouse selection dropdown.

7. Creating a destination on the go

For some activities, like the Copy data activity, we can create the destination on the spot if it doesn't exist yet. Clicking Browse in the destination dropdown opens a dialog where we create a new lakehouse and assign it right away.

8. Running the pipeline

Once the pipeline is built, we run it with the Run button at the top of the pipeline UI to transfer data from source to destination.

9. Dataflow Gen2

A Copy data activity moves data exactly as it found it. But data often needs cleaning before anyone can use it, and that's the preparation and sorting station. Using low-code Power Query, Dataflow Gen2 lets us remove unneeded columns, rename fields, filter rows, and reshape data, then loads the result into our destination.

10. Eventstreams

Both of those run on a schedule. But some data can't wait for the next run. An Eventstream is the conveyor belt that never stops: it ingests data in real time, receiving events like website clicks, sensor readings, or transaction notifications as they happen. It can transform those events and route them to a destination for real-time analysis.

11. Importing data into a lakehouse

In a lakehouse, the Get data menu at the top of its UI brings most of these together: uploads, Dataflow Gen2, and Eventstreams. It also offers Copy job, a simpler way to copy data without building a pipeline.

12. Let's practice!

Now let's practice by building a batch ingestion pipeline.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.