1. Congratulations
Awesome work. You've finished the course. You're well on your way of becoming a professional data engineer! Let's look at a short recap of what we've learned.
2. Introduction to data engineering
In the first chapter, you built a solid understanding of the tasks of a data engineer. You also got acquainted with the different kinds of tools. We finished with some introduction to cloud service providers.
3. Data engineering toolbox
In the second chapter, you took a deep dive into the data engineering toolbox. Starting with a lesson about databases, we talked about parallel computing using frameworks like Spark and finished it off with some workflow scheduling in Airflow.
4. Extract, Load and Transform (ETL)
The third chapter introduced the concept of ETL, or Extract, Load, and Transform. This well-established procedure consists of the extract phase, where we extract data from several sources. Afterward, we transform it using parallel computing frameworks. Finally, we load the result into a target database.
5. Case study: DataCamp
We ended with a hands-on example on recommendations at DataCamp, represented as an ETL task. We first fetched data from multiple sources, transformed it to form recommendations, and loaded it into a database that is ready to use by data products.
6. Good job!
I hope you enjoyed this course as much as I did and hope it motivates you to learn more about data engineering. See you soon!