The HITL Operator

1. The HITL Operator

Welcome back! Let's take a look at our last new topic for this course: human-in-the-loop operators.

2. Human-In-The-Loop?

Before we discuss specifics, let's note that our all our current workflows run automatically from start to finish. As we've seen, we can handle some common issues - task failures, adding notifications, and so forth. Even with that functionality, we can't currently double-check any concerning or particularly important details within a workflow. Human-In-The-Loop operators let us add human level interaction to our workflows.

3. Different operators

There are actually different operators in the HITL class. The different operators give different interaction capabilities for each type. There are four types of HITL operators in the current version of Airflow.

4. HITL Operators

The first is the HITLOperator. This is the base class for all the HITL options. It puts any HITL task into a deferred state which lets Airflow run other tasks while waiting for other Dags. The HITLBranchOperator is used for manual decision making, which then branches according to the Dag's downstream settings. The HITLEntryOperator allows users to enter structured input on the Airflow UI which can then be used for further branching. The last one is the ApprovalOperator, which allows for simple "Approve" or "Reject" options along with a detailed message. These messages and options appear within the Airflow UI. This is the portion we'll focus on for this course.

5. ApprovalOperator

The ApprovalOperator is part of the airflow.providers.standard.operators.hitl library. It includes several attributes you'll use often. These include the subject, which is the title or subject of the approval. The body is the content of the message posted to the Airflow user. The execution_timeout is a timedelta object that determines how long to wait for a response before timing out. The defaults attribute lets you specify whether to approve or reject the request if an execution_timeout is reached. Finally, the assigned_users attribute takes a list of the users who can approve or reject the request.

6. ApprovalOperator example

Here's an example calling the ApprovalOperator with the subject for Data import - Approval Required and the body of the message shown to the user.

7. ApprovalOperator in Airflow UI

The Airflow UI will alert on the Dag run view of any approvals that are needed. You can tell by the Approval required status along with the name of the task.

8. ApprovalOperator Home page

It's also accessible from the main Airflow home page under the Required Actions section.

9. ApprovalOperator message in Airflow UI

This illustrates the actual message provided to a user for Approval or Rejection. The message can contain specific info about the Dag run, or simply be a reminder / pause for review prior to import task runs.

10. Let's practice!

Human-In-The-Loop functions add a whole new level of capability in Airflow workflows. Let's practice what we've learned.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.