1. Working with templates
Great work getting to the last chapter of this course! You've learned a lot, but let's take a look at some of the more advanced features of Airflow, starting with templates.
2. What are templates?
You may be wondering what templates are and what they do in the case of Airflow.
Templates allow substitution of information during a DAG run. In other words, every time a DAG with templated information is executed, information is interpreted and included with the DAG run.
Templates provide added flexibility when defining tasks. We'll see examples of this shortly.
Templates are created using the Jinja templating language. A full explanation of Jinja is out of scope for this course, but we'll cover some basics in the coming slides.
3. Non-Templated BashOperator example
Before we get specifically into a templated example, let's consider what we would do for the following requirement. Your manager has asked you to simply echo the word "Reading" and a list of files to a log / output / etc.
If we were to do this with what we currently know about Airflow, we would likely create multiple tasks using the BashOperator.
Our first task would setup the task with the intended bash command - in this case echo Reading file1 dot txt, as an argument to the BashOperator.
If we had a second file, we would create a second task using the bash command echo Reading file2 dot txt.
This type of code would continue for the entire list of files. Consider what this would look like if we had 5, 10, or even 100+ files we needed to process. There would be a lot of repetitive code. Not to mention what if you needed to change the command being used / etc.
4. Templated BashOperator example
Let's take a look at how we would accomplish the same behavior with templates.
First, we need to create a variable containing our template - which is really just a string with some specialized formatting.
Our string is the actual bash command echo and instead of the file name, we're using two open curly braces, the term params dot filename, and then two closing curly braces. The curly braces when used in this manner represent information to be substituted. This will make more sense in a moment.
If you've done any web development or worked with any reporting tools, you've likely worked with something similar.
Next, we create our Airflow task as we have previously.
We assign a task_id and dag argument as usual, but our bashcommand looks a little different. We set the bashcommand to use the templated command string we defined earlier. We also have an additional argument called params. In this case, params is a dictionary containing a single key filename with the value file1 dot txt.
Now, if you look back at the templated command, you'll notice that the term in the curly braces is params.filename. At runtime, Airflow will execute the BashOperator by reading the templated command and replacing params dot filename with the value stored in the params dictionary for the filename key. In other words, it would pass the BashOperator echo Reading file1 dot txt. The actual log output would be Reading file1 dot txt (after the BashOperator executed the command).
5. Templated BashOperator example (continued)
Now, let's consider one way to use templates for our earlier task of outputting Reading file1 dot txt and Reading file2 dot txt.
First, we create our templated command as before.
Next, create the first task and pass the params dict with a filename key and the value file1 dot txt.
To pass another entry, we can create a second task and modify the params dict accordingly. This time the filename would contain file2 dot txt and Airflow would substitute that value instead. The resulting output would be as expected.
Note, you may be wondering what templates do for you here. You'll see more in the coming exercises and lessons.
6. Let's practice!
We'll have more to discuss about templates in the next lesson but let's practice what we've learned and I'll see you back shortly.