Aan de slagBegin gratis

Pairing inputs with zip

Your team has a pipeline that uploads files to specific cloud storage locations. A colleague uses zip() with expand_kwargs() to pair each file with its cloud storage destination:

@task
def get_files():
    return ["users.csv", "orders.csv", "products.csv"]

@task
def get_destinations():
    return ["s3://bronze/users/", "s3://bronze/orders/", "s3://bronze/products/"]

@task
def upload(file_to_upload):
    # Simulate uploading a file to cloud storage
    print(f"Uploading {file_to_upload[0]} to {file_to_upload[1]}")

files = get_files()
destinations = get_destinations()
upload.expand(file_to_upload=files.zip(destinations))

How many instances of upload will Airflow create?

Deze oefening maakt deel uit van de cursus

Building Data Pipelines with Airflow

Bekijk cursus

Interactieve oefening met praktijkervaring

Zet theorie om in actie met een van onze interactieve oefeningen

Begin oefening