Partition-aware scheduling with Asset Partitions

1. Partition-aware scheduling with Asset Partitions

Your upstream pipeline just finished loading today's sales data. The downstream report triggers. But which day's data should it process?

2. When ds is not enough

In the previous video, we used ds to filter SQL queries by date. That works for scheduled Dags, where ds reflects the logical date. But asset-triggered Dags have ds set to None, so it is simply not available. The downstream has no way to know which day's data was loaded. Asset Partitions solve this. The upstream attaches a partition_key to the asset event, and the downstream inherits it. The key can be a date, but it can also be any other data slice, like a region or a product.

3. CronPartitionTimetable

Let's build this, starting upstream. CronPartitionTimetable creates partitioned Dag runs automatically. Import it from airflow.sdk and pass it as the schedule. The cron expression sets the cadence, just like a regular schedule. The one shown here runs the Dag once a day at midnight UTC, the same cadence you'd get from a daily schedule. Every scheduled run gets a partition_key based on the run's logical date. Manual runs are not partitioned unless you provide a key.

4. Accessing partition keys

You can access the partition key in two ways. In a Python task, add dag_run as a function parameter, and Airflow injects it automatically. Then read partition_key off the dag_run object. In a SQL template, do the same thing through Jinja, inside double curly braces. Here, our SQL file uses the partition key to filter which rows to delete and insert, so each run processes exactly one day's data. Notice that the SQL example slices the first ten characters of the partition key. That's because the raw key is a full timestamp, like 2026-04-23T00:00:00, and we want just the date portion to match the format in our table. The partition key is available inside every task in a partitioned Dag run.

5. Partitioned asset events

Now connect partitions to assets so the key travels downstream. Define an Asset called daily_sales and set it as an outlet on the load task. The example here uses a Python task, but the same outlets argument works on any operator, including SQLExecuteQueryOperator, which is what you'll wire up in the exercise. When the task completes in a partitioned run, Airflow creates a partitioned asset event carrying the same partition key. This signals not just that data is ready, but which specific partition is ready.

6. PartitionedAssetTimetable

On the downstream side, schedule a Dag using PartitionedAssetTimetable. Pass the daily_sales asset to the assets parameter. This Dag triggers only on partitioned events for daily_sales. Regular asset events won't trigger it. The downstream run inherits the partition key, solving the earlier problem: the downstream now knows exactly which partition was updated.

7. StartOfDayMapper

The upstream produces keys as full timestamps. For daily reporting, we only need the date. StartOfDayMapper normalizes to a year-month-day format. Pass it in partition_mapper_config, mapping the asset to the mapper. The downstream now receives the formatted date instead of the full timestamp. Airflow ships temporal mappers for other grains, such as hour, week, and month. And partition keys don't have to be time at all. AllowedKeyMapper validates against a fixed list, like region names or departments.

8. The full flow

Let's put it together. The upstream uses CronPartitionTimetable, attaching a partition key to every run. The load task emits a partitioned asset event with that key. The downstream uses PartitionedAssetTimetable with a StartOfDayMapper. The mapper normalizes the timestamp to a date, and the downstream inherits the mapped key. There is no relying on ds and no custom logic needed. In the Airflow UI, you can verify that the partitioned asset events were created and see which partition key each event carries.

9. Let's practice!

Time to build partitioned pipelines.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.