Pairing inputs with zip
Your team has a pipeline that uploads files to specific cloud storage locations. A colleague uses zip() with expand_kwargs() to pair each file with its cloud storage destination:
@task
def get_files():
return ["users.csv", "orders.csv", "products.csv"]
@task
def get_destinations():
return ["s3://bronze/users/", "s3://bronze/orders/", "s3://bronze/products/"]
@task
def upload(file_to_upload):
# Simulate uploading a file to cloud storage
print(f"Uploading {file_to_upload[0]} to {file_to_upload[1]}")
files = get_files()
destinations = get_destinations()
upload.expand(file_to_upload=files.zip(destinations))
How many instances of upload will Airflow create?
Deze oefening maakt deel uit van de cursus
Building Data Pipelines with Airflow
Interactieve oefening met praktijkervaring
Zet theorie om in actie met een van onze interactieve oefeningen
Begin oefening