Introduction to Dataflows Gen2

1. Introduction to Dataflows Gen2

In this video, we’ll explore how Dataflows Gen2 enhances data handling in Microsoft Fabric, its key features, supported destinations, and how to integrate it with pipelines for advanced data management.

2. Dataflows Gen2 in Microsoft Fabric

Dataflows Gen2 is a powerful tool for transforming and ingesting data efficiently, whether you're working with small or large datasets. Unlike Pipelines, which are more focused on orchestrating and moving data, Dataflows Gen2 specializes in transforming data, getting it ready for analysis. With Dataflows Gen2, you can perform complex transformations like merging tables, filtering fields, or pivoting columns. A key feature is its flexible schema options, allowing you to dynamically adjust your data structure, adapting to your changing needs. It also includes features like staging and fast copy for managing large datasets — we’ll cover these in detail later. Finally, with seamless integration into Data Pipelines, you’ll have everything you need for efficient data preparation.

3. Overview of the DataFlows Gen2 Interface

Let’s dive into the Dataflows Gen2 interface, where transforming data with Power Query Online is intuitive and powerful. Don’t worry, you’ll be getting hands-on with this in an upcoming exercise to see it all in action!

4. Dataflow Gen 2 Interface Components

First, we have the Power Query Ribbon — your control center for connecting data sources and applying transformations like merging and filtering, all without writing code. Next, in the Queries Pane, you’ll manage, rename, and duplicate data sources, and even enable staging for efficient data handling.

5. Dataflow Gen 2 Interface Components

Then, we have Diagram View which provides a visual map of how your data flows through each transformation, helping you track the process with ease. After that, the Data Preview Pane lets you see real-time snapshots of your data as you apply transformations, so you can immediately see the impact of each change.

6. Dataflow Gen 2 Interface Components

Finally, the Query Settings Pane records every step of your transformations, allowing you to review and adjust your process at any time.

7. Supported Data Destinations in DataFlows Gen2

In DataFlow Gen2, you can assign a specific destination for every data query, giving you the flexibility to control where your data lands. Each query can be sent to a different destination, like Azure SQL, Fabric Lakehouse, or Fabric Warehouse, depending on your needs. This approach allows you to choose the best storage option for each dataset, whether it’s relational data or large-scale analytics.

8. Integrating Dataflows Gen2 in Data Pipelines

Now, let’s talk about why we integrate Dataflows Gen2 into Data Pipelines. The answer is simple: it allows you to perform further operations on your transformed data and automate workflows. For example, using a For Each Loop in a pipeline, you can apply the same transformation across multiple datasets automatically. This is especially useful when you have many datasets that require identical processing, saving time and reducing manual work. To make this work, you can add Dataflows Gen2 as a pipeline activity. This integration enables you to streamline workflows and monitor performance in real-time, ensuring everything runs smoothly and efficiently.

9. Let's practice!

The concepts are clear, now let us take the next step and dive into the practical side!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.