More templates

1. More templates

Welcome back! Hopefully you've cemented the basics of templates in your mind. Now let's look at some of the more powerful options they provide.

2. Quick task reminder

In our last lesson we were given the task of taking a list of filenames and printing "Reading filename" to the log or output. In our templated version, we created a templated command that substituted the filename value in place of params dot filename. We also used two tasks with different filename values to demonstrate one way to use templates without changing the actual bash command. Now, let's consider another way to perform the substitution.

3. More advanced template

Jinja templates can be considerably more powerful than we've used so far. It is possible to use a for construct to allow us to iterate over a list and output content accordingly. Let's change our templated command to the following. We start with an open curly brace and the percent symbol, then use a normal python command of for filename in params dot filenames then percent close brace. Then we modify our output line to be echo Reading open curly braces filename close curly braces. We then use a Jinja entry to represent the end of the for loop, open curly brace percent endfor percent close curly brace. Note that this is required to define the end of the loop, as opposed to Python's typical whitespace indention. Now let's look at our BashOperator task. It looks similar except we've modified the params key to be filenames, and the value is now a list with two strings, file1 dot txt and file2 dot txt. When Airflow executes the BashOperator, it will iterate over the entries in the filenames list and substitute them in accordingly. Our output is the same as before, with a single task instead of two. Consider too the difference in code if you had 100 files in the list?

4. Variables

As part of the templating system, Airflow provides a set of built-in runtime variables. These provide assorted information about DAG runs, individual tasks, and even the system configuration. Template examples include the execution date, which is ds in the double curly brace pairs. It returns the date in a 4 digit year dash 2 digit month dash 2 digit day format. There's also a ds underscore nodash variety to get the same info without dashes. Note that this is a string, not a python datetime object. Another variable available is the prev underscore ds, which gives the date of the previous DAG run in the same format as ds. The nodash variety is here as well. You can also access the full DAG object using the dag variable. Or you can use the conf object to access the current Airflow configuration within code. There are many more variables available - you can refer to the documentation for more information.

5. Macros

In addition to the other Airflow variables, there is also a macros variable. The macros package provides a reference to various useful objects or methods for Airflow templates. Some examples of these include the macros dot datetime, which is the Python datetime dot datetime object. The macros dot timedelta template references the timedelta object. A macros dot uuid is the same as Python's uuid object. Finally, there are also some added functions available, such as macros dot ds underscore add. It provides an easy way to perform date math within a template. It takes two arguments, a YYYYMMDD datestring and an integer representing the number of days to add (or subtract if the number is negative). Our example here would return April 20, 2020. These are not all the available macros objects - refer to the Airflow documentation for more info.

6. Let's practice!

We've seen several interesting aspects of Airflow templates - let's practice using them now.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.