Configuring retries with exponential backoff
Your team runs a pipeline that fetches weather data from an external API. The API occasionally times out during peak hours, but these are transient failures that usually resolve on their own.
A simple retry with a fixed delay can overwhelm the API during an outage, so the task needs a strategy that gives the endpoint progressively more time to recover between attempts.
This exercise is part of the course
Building Data Pipelines with Airflow
Hands-on interactive exercise
Turn theory into action with one of our interactive exercises
Start Exercise