Serverless Data Processing with Dataflow - Writing an ETL pipeline using Apache Beam and Dataflow (Python)

This exercise is part of the course

Serverless Data Processing with Dataflow: Develop Pipelines

Exercise instructions

In this lab, you a) build a batch ETL pipeline in Apache Beam, which takes raw data from Google Cloud Storage and writes it to BigQuery b) run the Apache Beam pipeline on Dataflow and c) parameterize the execution of the pipeline.

This exercise is part of the course

Serverless Data Processing with Dataflow: Develop Pipelines

AdvancedSkill Level

4.5+

Start Course for Free

This module introduces the course and course outline

Exercise 1: Course Introduction

Review main concepts of Apache Beam, and how to apply them to write your own data processing pipelines.

Exercise 1: Beam Basics Exercise 2: Utility Transforms Exercise 3: DoFn Lifecycle Exercise 4: Serverless Data Processing with Dataflow - Writing an ETL pipeline using Apache Beam and Dataflow (Python)

Current Exercise

Exercise 5: Quiz Question 1 Exercise 6: Quiz Question 2 Exercise 7: Quiz Question 3 Exercise 8: Module Resources

In this module, you will learn about how to process data in streaming with Dataflow. For that, there are three main concepts that you need to learn: how to group data in windows, the importance of watermark to know when the window is ready to produce results, and how you can control when and how many times the window will emit output.

Exercise 1: Windows Exercise 2: Watermarks Exercise 3: Triggers Exercise 4: Serverless Data Processing with Dataflow - Batch Analytics Pipelines with Dataflow (Python)Exercise 5: Serverless Data Processing with Dataflow - Using Dataflow for Streaming Analytics (Python)Exercise 6: Quiz Question 1 Exercise 7: Quiz Question 2 Exercise 8: Quiz Question 3 Exercise 9: Quiz Question 4 Exercise 10: Module Resources

In this module, you will learn about what makes sources and sinks in Dataflow. The module will go over some examples of TextIO, FileIO, BigQueryIO, PubsubIO, KafKaIO, BigtableIO, Avro IO, and Splittable DoFn. The module will also point out some useful features associated with each I/O.

Exercise 1: Sources & Sinks Exercise 2: TextIO & FileIO Exercise 3: BigQueryIO Exercise 4: PubsubIO Exercise 5: KafkaIO Exercise 6: BigtableIO Exercise 7: AvroIO Exercise 8: Splittable DoFn Exercise 9: Quiz Question 1 Exercise 10: Quiz Question 2 Exercise 11: Module Resources

This module will introduce schemas, which give developers a way to express structured data in their Beam pipelines.

Exercise 1: Beam schemas Exercise 2: Code examples Exercise 3: Serverless Data Processing with Dataflow - Branching Pipelines (Python)Exercise 4: Quiz Question 1 Exercise 5: Quiz Question 2 Exercise 6: Module Resources

This module covers State and Timers, two powerful features that you can use in your DoFn to implement stateful transformations.

Exercise 1: State API Exercise 2: Timer API Exercise 3: Summary Exercise 4: Quiz Question 1 Exercise 5: Quiz Question 2 Exercise 6: Module Resources

This module will discuss best practices and review common patterns that maximize performance for your Dataflow pipelines.

Exercise 1: Schemas Exercise 2: Handling un-processable data Exercise 3: Error handling Exercise 4: AutoValue code generator Exercise 5: JSON data handling Exercise 6: Utilize DoFn lifecycle Exercise 7: Pipeline Optimizations Exercise 8: Serverless Data Processing with Dataflow - Advanced Streaming Analytics Pipeline with Dataflow (Python)Exercise 9: Quiz Question 1 Exercise 10: Quiz Question 2 Exercise 11: Quiz Question 3 Exercise 12: Module Resources

This modules introduces two new APIs to represent your business logic in Beam: SQL and Dataframes.

Exercise 1: Dataflow and Beam SQL Exercise 2: Windowing in SQL Exercise 3: Beam DataFrames Exercise 4: Quiz Question 1 Exercise 5: Quiz Question 2 Exercise 6: Module Resources

This module will cover Beam notebooks, an interface for Python developers to onboard onto the Beam SDK and develop their pipelines iteratively in a Jupyter notebook environment.

Exercise 1: Beam Notebooks Exercise 2: Quiz Question 1 Exercise 3: Quiz Question 2 Exercise 4: Module Resources

This module provides a recap of the course

Exercise 1: Course Summary