1. Congratulations!
Congratulations! Throughout this course, we’ve explored the ins and outs of designing, building, and maintaining data pipelines. Let’s take a look back at all that we covered.
2. Designing and building data pipelines
First, we were introduced to both ETL and ELT pipelines. We explored characteristics of the best data pipelines to emulate, and how to properly design a data pipeline.
After outlining the foundations of ETL pipelines, we jumped right into developing extract, transform, and load logic to pipeline data from a source system to a storage medium. We incorporated techniques including exception handling and logging to provide visibility into a data pipeline.
3. Advanced ETL techniques
In Chapter 3, we built on foundational ETL logic to handle different data types and formats, such as flattened and nested JSON data. We also practiced advanced transformation logic with pandas, and techniques to persist data to dynamic storage media.
4. Deploying and maintaining data pipelines
Finally, we explored tools to validate and test a data pipeline to ensure data has been extracted, transformed, and loaded before being run in a production setting. We also looked at orchestration tools, such as Apache Airflow, to help assist with running data pipelines in a production setting.
5. Next steps
Now that you've mastered the basics of designing, building, and maintaining data pipelines, check out a few of these DataCamp courses and other resources to continue building your data engineering skill-set.
6. Thank you!
Thank you so much for joining me for this course. Best of luck as you continue your journey designing, building, and deploying data pipelines!