From development to production

1. From development to production

Your pipeline works locally. How do you operate it in production?

2. The Airflow CLI

The Airflow CLI is your production toolkit alongside the UI. While the UI is great for monitoring and exploring, the CLI lets you automate operations and script deployments. Three commands you will use constantly: airflow dags trigger for manual runs, airflow dags list-runs for inspecting history, and airflow backfill create for reprocessing historical data.

3. Triggering a run

The trigger command creates a manual run. By adding the logical-date flag, we specify which date the run should process. This is useful for testing a specific date or reprocessing a single day after a data fix. After triggering, you see a confirmation with the run details. Notice the run_id starts with the manual prefix, because we triggered it by hand.

4. Listing runs

The list-runs command shows all runs for a Dag. The output includes the run_id, state, and logical date. The important detail is the run_id prefix: it tells you how each run was created. Scheduled runs start with the scheduled prefix, and our manual trigger created one with the manual prefix. These prefixes make it straightforward to tell run types apart in production.

5. Backfilling historical data

When historical data needs reprocessing, use airflow backfill create. Pass the dag-id, a from-date, and a to-date. Airflow creates one run per scheduled interval in that range, so a daily Dag backfilled over three days creates three runs. The max-active-runs flag controls how many of those runs go at the same time. That matters for databases like DuckDB that handle one write at a time. And backfill runs get their own prefix, simply called backfill.

6. Three types of runs

So there are three ways a Dag run gets created, each with its own prefix. Scheduled runs are created automatically by the scheduler based on the Dag's schedule. Manual runs come from the CLI trigger command or the UI trigger button. Backfill runs are created by the backfill command for reprocessing historical dates. The prefix in the run_id always tells you which one it is.

7. Production challenges

Moving from development to production introduces challenges in three areas. Build is about setting up a local environment, developing workflows efficiently, and deploying code reliably. Run is about scaling workers based on workload and handling failover when things go wrong. Observe is about investigating failures, tracking data freshness, and tracing dependencies across your pipelines. These are the three pillars of production Airflow.

8. Astro: Build

Astronomer's Astro platform addresses each pillar. For Build, the Astro CLI gives you a complete local Airflow environment with one command and deploys to production with another. Astro IDE is a browser-based editor for authoring Dags with AI assistance, with no local setup required. Both tools eliminate the friction of getting from code to a running production environment.

9. Astro: Run and Observe

For Run, Astro manages the entire infrastructure. Elastic auto-scaling adjusts worker count based on task queue depth and scales to zero when idle. High availability with multi-zone deployments and automatic failover keeps your pipelines running during outages. You focus on pipelines, not infrastructure. For Observe, Astro provides pipeline lineage across Dags and tables, proactive SLA alerts before deadlines are missed, and AI-powered log summaries that surface what broke, why, and how to fix it.

10. Let's practice!

Time to put it all together.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.