Configuring a sensor for production
Your pipeline uses a FileSensor to wait for an incoming CSV report before processing it. In the default poke mode, the sensor holds a worker slot the entire time it waits, even when the file takes several minutes to arrive.
For production, a sensor should release the slot between checks and have a safety timeout so it doesn't wait indefinitely.
Deze oefening maakt deel uit van de cursus
Building Data Pipelines with Airflow
Interactieve oefening met praktijkervaring
Zet theorie om in actie met een van onze interactieve oefeningen
Begin oefening