Congratulations!

1. Congratulations!

Congratulations on successfully completing this introduction to Airflow. We've covered a lot of ground since the first chapter and you should be pleased to have come this far. Let's take a moment to review what we've learned and cover some next steps.

2. What we've learned

Let's review everything we've worked with during this course. We started with learning about workflows and DAGs in Airflow. We've learned what an operator is and how to use several of the available ones. We learned about tasks and how they are defined by various types of operators. In addition, we learned about dependencies between tasks and how to set them with bitshift operators. We've used sensors to react to workflow conditions and state. We've scheduled DAGs in various ways. We used SLAs and alerting to maintain visibility on our workflows. We learned about the power of templating in our workflows for maximum flexibility when defining tasks. We've learned how to use branching to add conditional logic to our DAGs. Finally, we've learned about the Airflow interfaces (command line and UI), about Airflow executors, and a bit about how to debug and troubleshoot various issues with Airflow and our own workflows.

3. Next steps

A few suggestions for next steps include setting up your own environment for practice. You can follow the installation instructions in the Airflow documentation or use a cloud-based Airflow service. Look into other operators or sensors - there are operators available for Amazon's S3, Postgresql operators, HDFS sensors, and so forth. Experiment with dependencies with a large number of tasks. Consider how you expect the workflow to progress and always try to leave as much up to the scheduler as possible to achieve the best performance. Given the length of the course, there is only so much we could cover and we left out some important parts of Airflow such as XCom, connections, and managing the UI further. Refer to the documentation for more ideas. Finally and most importantly, keep building workflows. When you're uncertain how something works, try to build an example that covers what you'd like to accomplish. Look at the views within the Airflow UI to better understand how the system interprets your DAG code. The more you experiment, the better your understanding will grow.

4. References

There are many sources of information for learning more about Airflow, including the documentation found at airflow.apache.org/docs. The Airflow Community slack channel is particularly active and covers many topics on Airflow including development, troubleshooting, and various components. It can be a little overwhelming, but people are quite happy to assist. There are of course countless blogs and newsletters covering Airflow (including my own at news.thedatarodeo.com) in various forms. There will also likely be future DataCamp courses covering more advanced usage of Airflow.

5. Thank you!

Finally, thank you for taking this course and giving me the opportunity to introduce you to Airflow. Good luck on your future learning opportunities!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.