Deferrable operators

1. Deferrable operators

You already know that sensors can occupy worker slots while waiting for conditions. Now let's review the different sensor modes and learn how to configure deferrable operators for production use.

2. The sensor problem

Sensors are tasks that wait for a condition, such as a file appearing, before allowing downstream tasks to run. The problem is how they wait. In the default poke mode, a sensor occupies a worker slot the entire time. It checks the condition, sleeps for the poke interval, checks again, and keeps the slot locked between checks. With limited workers, a few long-running sensors can block the entire pipeline. Other tasks sit in the queue waiting for a slot that never frees up.

3. Reschedule mode

Reschedule mode improves this. Both examples use FileSensor from the standard sensors package. On the left is poke mode. The sensor checks every 30 seconds, but the worker slot stays occupied the entire time. On the right is reschedule mode. The same sensor has mode set to reschedule and a poke_interval of 300 seconds. After each check, the sensor releases the slot and asks the scheduler to wake it up after 5 minutes. But there's a trade-off: each time the sensor wakes up, it briefly grabs a worker slot again. With many sensors checking frequently, these brief slot grabs add up.

4. Deferrable mode

Deferrable mode takes this further. When a sensor defers, it hands off the wait entirely to the Triggerer, a separate Airflow process. The Triggerer uses async I/O, which lets it monitor hundreds of conditions at the same time without blocking on any one of them. Think of it as a single receptionist managing hundreds of phone lines instead of assigning one person per call. No worker slots are used during the entire wait. When the condition is met, the Triggerer signals the scheduler to resume the task on a worker.

5. Enabling deferrable

Enabling deferrable mode is straightforward. We create a FileSensor that waits for a CSV file to appear. Without any mode parameter, the sensor defaults to poke mode, holding a worker slot the entire time. The key parameter is deferrable=True. That's all it takes. Behind the scenes, the sensor creates a trigger, submits it to the Triggerer, and releases the worker. We still set poke_interval to control how often the Triggerer checks the condition, and timeout to set a maximum wait time. The only difference is who does the waiting: the Triggerer instead of a worker.

6. When to use each mode

So when do we use each mode? The table summarizes the tradeoffs. Poke is fine for sensors that resolve in under a minute. The overhead of deferring isn't worth it for near-instant checks. Reschedule works for moderate waits, especially if the environment doesn't have a Triggerer running. The slot is freed between checks, but briefly occupied on each wake-up. Deferrable is the clear winner for long waits and environments with many concurrent sensors. The worker slot is not needed for checks. If the Triggerer is running, deferrable is almost always the right default.

7. Global deferrable setting

To make all compatible operators use deferrable mode, we set the DEFAULT_DEFERRABLE configuration to True. Any operator or sensor that supports deferral will use it automatically, without changing a single line of Dag code. Most common sensors and several operators support deferrable mode, including file, time, and external task sensors. Check the Airflow documentation for the full list. If a specific sensor shouldn't defer, we can override it with deferrable=False.

8. Let's practice!

Let's configure some sensors.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.