Defining the DAG
In the previous exercises, you've completed the extract, transform and load phases separately. Now all of this is put together in one neat etl() function that you can discover in the console.
The etl() function extracts raw course and ratings data from relevant databases, cleans corrupt data and fills in missing value, computes average rating per course and creates recommendations based on the decision rules for producing recommendations, and finally loads the recommendations into a database.
As you might remember from the video, etl() accepts a single argument: db_engines. Both etl and db_engines are available in your workspace, so the task you define only has to call one with the other.
This exercise is part of the course
Introduction to Data Engineering
Exercise instructions
- Complete the DAG definition, so it runs daily. Make sure to use the cron notation.
- Complete the task so it calls the
etl()function with the database engines.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Define the DAG so it runs on a daily basis
@dag(dag_id="recommendations",
start_date=datetime(2024, 1, 1),
schedule="____")
def recommendations():
# Make sure the task calls etl() with the database engines
@task(task_id="recommendations_task")
def recommendations_task():
____(____)
recommendations_task()
# Run the DAG
recommendations()