Airflow, Luigi, dan cron

Dalam video, Anda melihat beberapa alat yang dapat membantu Anda menjadwalkan pekerjaan Spark. Anda melihat keterbatasan cron dan bagaimana hal itu mendorong pengembangan kerangka kerja seperti Airflow dan Luigi.

Ada banyak fitur berguna di Airflow, tetapi bisakah Anda memilih fitur dari daftar di bawah ini yang juga disediakan oleh cron?

Latihan ini adalah bagian dari kursus

Pengantar Data Engineering

Latihan interaktif praktis

Ubah teori menjadi tindakan dengan salah satu latihan interaktif kami.

Mulai berolahraga

Latihan ini adalah bagian dari kursus

Pengantar Data Engineering

SkillTag.level.intermediateSkillTag.label

4.8+

Mulai Kursus Gratis

In this first chapter, you will be exposed to the world of data engineering! Explore the differences between a data engineer and a data scientist, get an overview of the various tools data engineers use and expand your understanding of how cloud technology plays a role in data engineering.

Exercise 1: What is data engineering?Exercise 2: Tasks of the data engineer Exercise 3: Data engineer or data scientist?Exercise 4: Data engineering problems Exercise 5: Tools of the data engineer Exercise 6: Kinds of databases Exercise 7: Processing tasks Exercise 8: Scheduling tools Exercise 9: Cloud providers Exercise 10: Why cloud computing?Exercise 11: Big players in cloud computing Exercise 12: Cloud services

Now that you know the primary differences between a data engineer and a data scientist, get ready to explore the data engineer's toolbox! Learn in detail about different types of databases data engineers use, how parallel computing is a cornerstone of the data engineer's toolkit, and how to schedule data processing jobs using scheduling frameworks.

Exercise 1: Database Exercise 2: SQL vs NoSQL Exercise 3: Skema basis data Exercise 4: Menggabungkan berdasarkan relasi Exercise 5: Diagram star schema Exercise 6: Apa itu komputasi paralel Exercise 7: Mengapa komputasi paralel?Exercise 8: Dari tugas ke sub-tugas Exercise 9: Menggunakan DataFrame Exercise 10: Kerangka komputasi paralel Exercise 11: Spark, Hadoop, dan Hive Exercise 12: GroupBy di PySpark Exercise 13: Menjalankan berkas PySpark Exercise 14: Kerangka kerja penjadwalan alur kerja Exercise 15: Airflow, Luigi, dan cron

Latihan Saat Ini

Exercise 16: DAG Airflow

Having been exposed to the toolbox of data engineers, it's now time to jump into the bread and butter of a data engineer's workflow! With ETL, you will learn how to extract raw data from various sources, transform this raw data into actionable insights, and load it into relevant databases ready for consumption!

Exercise 1: Extract Exercise 2: Data sources Exercise 3: Fetch from an API Exercise 4: Read from a database Exercise 5: Transform Exercise 6: Splitting the rental rate Exercise 7: Prepare for transformations Exercise 8: Joining with ratings Exercise 9: Loading Exercise 10: OLAP or OLTP Exercise 11: Writing to a file Exercise 12: Load into Postgres Exercise 13: Putting it all together Exercise 14: Defining a DAG Exercise 15: Setting up Airflow Exercise 16: Interpreting the DAG

Cap off all that you've learned in the previous three chapters by completing a real-world data engineering use case from DataCamp! You will perform and schedule an ETL process that transforms raw course rating data, into actionable course recommendations for DataCamp students!

Exercise 1: Course ratings Exercise 2: Exploring the schema Exercise 3: Querying the table Exercise 4: Average rating per course Exercise 5: From ratings to recommendations Exercise 6: Filter out corrupt data Exercise 7: Using the recommender transformation Exercise 8: Scheduling daily jobs Exercise 9: The target table Exercise 10: Defining the DAG Exercise 11: Enable the DAG Exercise 12: Querying the recommendations Exercise 13: Congratulations