The TaskFlow API

1. The TaskFlow API

Now that the architecture is fresh, let's look at how you write Dags with the TaskFlow API.

2. The classic approach

Here's the classic way to write a Dag. You define your functions, then wrap each one in a PythonOperator, passing the function as python_callable. Dependencies are wired manually. The code works, but there's a lot of repetition. The actual logic is only a few lines, while the rest is boilerplate.

3. The TaskFlow approach

With the TaskFlow API, the same pipeline looks like this. You import dag and task from the Task SDK. The @dag decorator replaces the context manager, and @task replaces PythonOperator. At the bottom, notice the two function calls: data equals extract_data() stores the return value, and print_summary(data) passes it into the next task. By passing the return value of one function call into another, Airflow creates the dependency for you automatically. To add a new step, you would define another @task function, call it, and pass data between them in the same way. We'll look at exactly how this data passing works in the next lesson.

4. Why TaskFlow?

So why use the TaskFlow API? It cuts the boilerplate: decorators replace operator instances and manual wiring. Dependencies happen implicitly when you pass data between function calls. And the result reads like a normal Python script. Classic operators still work and remain the right choice for providers that don't offer decorators, but for Python-based tasks, TaskFlow is the way to go. If your team has existing classic pipelines, converting them to TaskFlow is a good practice to reduce boilerplate and keep the codebase consistent.

5. Airflow UI: Grid view

Once your Dag runs, you can monitor it in the Airflow UI. Let's take a look at two key views you'll use throughout the course. The Grid view, accessible via the Dag run view, is a great monitoring tool. Each column represents a Dag run. Each row is a task. The colored squares tell you the status at a glance: green for success, red for failed. Click any square to open the task instance details, where you can read logs and see exactly what happened.

6. Airflow UI: Graph view

The graph view can display either the run-independent Dag or a specific run. Rather than comparing runs, it helps reveal task dependencies, parallel execution, and the types of operators used. You can still access task instance details by clicking a task box in the graph. Settings also let you adjust details such as the graph direction.

7. Airflow UI: Dag versioning

Airflow automatically tracks Dag versions. Each time you change a Dag's structure and run it, Airflow creates a new version. It links each run to the version that was active at the time. You can view the version in the Dag details panel. To understand how the Dag has changed over time, use the graph view settings to display the structure for a specific version. This also works in the code view and is a useful way to explore the history of a pipeline.

8. Let's practice!

Time to build some TaskFlow Dags and explore the UI.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.