1. SLAs and reporting in Airflow
Welcome back! Let's talk now about SLAs and reporting in Airflow.
2. SLAs
You may be wondering, what is an SLA?
SLA stands for Service Level Agreement. Within the business world, this is often an uptime or availability guarantee. Airflow treats it a bit differently - it's considered the amount of time a task or a DAG should require to run.
An SLA miss is any situation where a task or DAG does not meet the expected timing for the SLA. If an SLA is missed, an email alert is sent out per the system configuration and a note is made in the log.
Any SLA miss can be viewed in the Browse, SLA Misses menu item of the web UI.
3. SLA Misses
To view any given SLA miss, you can access it in the web UI, via the Browse: SLA Misses link.
It provides you general information about what task missed the SLA and when it failed. It also indicates if an email or notification has been sent when the SLA failed.
4. Defining SLAs
There are several ways to define an SLA but we'll only look at two for this course.
The first is via an sla argument on the task itself. This takes a timedelta object with the amount of time to pass.
The second way is using the default_args dictionary and defining an sla key. The dictionary is then passed into the default_args argument of the DAG and applies to any tasks internally.
5. timedelta object
We haven't covered the timedelta object yet so let's look at some of the details.
It's found in the datetime library, along with the datetime object.
Most easily accessed with an import statement of from datetime import timedelta.
Takes arguments of days, seconds, minutes, hours, and weeks. It also has milliseconds and microseconds available, but those wouldn't apply to Airflow.
To create the object, you simply call timedelta with the argument or arguments you wish to reference.
To create a 30 second time delta, call it with seconds equals 30.
Or weeks equals 2.
Or you can combine it into a longer mix of any of the arguments you wish (in this case, 4 days, 10 hours, 20 minutes, and 30 seconds).
6. General reporting
For reporting purposes you can use email alerting built into Airflow. There are a couple ways to do this.
Airflow has built-in options for sending messages on success, failure, or error / retry.
These are handled via keys in the default_args dictionary that gets passed on DAG creation.
The required component is the list of emails assigned to the email key. Then there are boolean options for email underscore on underscore failure, email underscore on underscore retry, and email underscore on underscore success.
In addition, we've already looked at the EmailOperator earlier but this is useful for sending emails outside of one of the defined Airflow options.
Note that sending an email does require configuration within Airflow that is outside the scope of this course. The Airflow documentation provides information on how to set up the global email configuration.
7. Let's practice!
Let's finish up this chapter by practicing what you've learned!