Airflow Variables

1. Airflow Variables

Let's take a look at another aspect of interacting with Airflow: Variables.

2. What are variables?

When building Dags, you'll often need configuration values that shouldn't be hardcoded. These include items like file paths, API endpoints, or environment names. This is where Airflow Variables come in. Variables are a general key/value store that can be queried from your tasks, and easily set through Airflow's user interface or defined in a JSON file. They act as settings that exist outside of your Dag code, allowing you to make changes based on scenarios, such as staging or production environments, without modifying the Dag code.

3. Accessing variables in Airflow UI

To access variables in the Airflow UI, select the Admin: Variables menu option.

4. Viewing variables in Airflow UI

Here we see the Variables list, accessible by going to Admin > Variables. Currently, no variables are defined. This is where all variables will appear once created, and where you can add, edit, or delete them.

5. Adding a variable in Airflow

To add a variable, we click the Add Variable button.

6. Variable editor

To create or edit a variable, we add the key and value. This is just like a Python dictionary, where the key is how we access the content stored in the value field. We can also add a description if desired, then click Save.

7. Variables present in the UI

Once we've added our variable, it will appear in this list. We can add more, edit this one, or delete it as well.

8. Reading a variable in Python

To actually use variables, we need to access them within our Python code for the Dags. This is accomplished with the airflow.sdk.Variable library. To read a variable, we use the Variable.get(Variable_name) method. If we try Variable.get("region"), my_region would return us-eastern as defined in the previous slides. We can also provide a fallback if a variable doesn't exist currently. If we try to access the current_environment variable, it doesn't exist, so env is None.

9. Accessing variables in Jinja templates

We can also access variables with Jinja templates, using the var.value.variable_name syntax. In this case, we can define our FileSensor to watch for a path based on the region variable. The actual filepath would be /data/us-eastern/input.csv.

10. Accessing variables through CLI

Variables can be read, updated, deleted or listed using the Airflow variables command. The subcommands include get to read, set to write, delete to remove, and list to display all variables.

11. JSON variables

In more complex scenarios, variable values can be provided as full JSON strings. To read this directly in Python, we use the syntax Variable.get variable_name deserialize_json=True. Assuming the JSON string is valid, this will automatically return a Python dictionary that mirrors the structure of the JSON file. In Jinja scenarios, we can access this data using the syntax var.json.variable_name.

12. Setting variables

It is also possible to set variables from within your Dag. This can be done using the syntax Variable.set variable_name, variable_value. If you are writing a Python dict object as the value, you can include the option serialize_json=True. Note that this should be used sparingly as it may affect other Dags that access the same variables in unforeseen ways. For example, since multiple Dags often run simultaneously in production, consider the impact of changing a region variable that is used by other Dags mid-run.

13. Let's practice!

Now that we've learned about variables, let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.