Serverless Data Processing with Dataflow - Custom Dataflow Flex Templates (Python)

This exercise is part of the course

Serverless Data Processing with Dataflow: Operations

Exercise instructions

In this lab, you (i)convert a custom pipeline into a Dataflow Flex Template, (ii)run a Dataflow Flex Template.

This exercise is part of the course

Serverless Data Processing with Dataflow: Operations

AdvancedSkill Level

5.0+

Start Course for Free

This module covers the course outline

Exercise 1: Course Introduction

In this module, we learn how to use the Jobs List page to filter for jobs that we want to monitor or investigate. We look at how the Job Graph, Job Info, and Job Metrics tabs collectively provide a comprehensive summary of your Dataflow job. Lastly, we learn how we can use Dataflow’s integration with Metrics Explorer to create alerting policies for Dataflow metrics.

Exercise 1: Job List Exercise 2: Job Info Exercise 3: Job Graph Exercise 4: Job Metrics Exercise 5: Metrics Explorer Exercise 6: Quiz Question 1 Exercise 7: Quiz Question 2 Exercise 8: Additional Resources

In this module, we learn how to use the Log panel at the bottom of both the Job Graph and Job Metrics pages, and learn about the centralized Error Reporting page.

Exercise 1: Logging Exercise 2: Error Reporting Exercise 3: Quiz Question 1 Exercise 4: Quiz Question 2 Exercise 5: Additional Resources

In this module, we learn how to troubleshoot and debug Dataflow pipelines. We will also review the four common modes of failure for Dataflow: failure to build the pipeline, failure to start the pipeline on Dataflow, failure during pipeline execution, and performance issues.

Exercise 1: Troubleshooting workflow Exercise 2: Types of troubles Exercise 3: Quiz Question 1 Exercise 4: Quiz Question 2 Exercise 5: Serverless Data Processing with Dataflow - Monitoring, Logging and Error Reporting for Dataflow Jobs Exercise 6: Additional Resources

In this module, we will discuss performance considerations we should be aware of while developing batch and streaming pipelines in Dataflow.

Exercise 1: Pipeline Design Exercise 2: Data Shape Exercise 3: Source, Sinks & external systems Exercise 4: Shuffle and streaming engine Exercise 5: Quiz Question 1 Exercise 6: Quiz Question 2 Exercise 7: Quiz Question 3 Exercise 8: Additional Resources

This module will discuss unit testing your Dataflow pipelines. We also introduce frameworks and features available to streamline your CI/CD workflow for Dataflow pipelines.

Exercise 1: Testing and CI/CD Overview Exercise 2: Unit Testing Exercise 3: Integration Testing Exercise 4: Artifact Building Exercise 5: Deployment Exercise 6: Quiz Question 1 Exercise 7: Quiz Question 2 Exercise 8: Serverless Data Processing with Dataflow - Testing with Apache Beam (Java)Exercise 9: Serverless Data Processing with Dataflow - Testing with Apache Beam (Python)Exercise 10: Additional Resources

In this module we will discuss methods for building systems that are resilient to corrupted data and data center outages.

Exercise 1: Introduction to Reliability Exercise 2: Monitoring Exercise 3: Geolocation Exercise 4: Disaster Recovery Exercise 5: High Availability Exercise 6: Quiz Question 1 Exercise 7: Quiz Question 2 Exercise 8: Additional Resources

This module covers Flex Templates, a feature that helps data engineering teams standardize and reuse Dataflow pipeline code. Many operational challenges can be solved with Flex Templates.

Exercise 1: Classic templates Exercise 2: Flex templates Exercise 3: Using flex templates Exercise 4: Google-provided templates Exercise 5: Quiz Question 1 Exercise 6: Quiz Question 2 Exercise 7: Quiz Question 3 Exercise 8: Serverless Data Processing with Dataflow - Custom Dataflow Flex Templates (Java)Exercise 9: Serverless Data Processing with Dataflow - Custom Dataflow Flex Templates (Python)

Current Exercise

Exercise 10: Additional Resources

This module reviews the topics covered in the course

Exercise 1: Course Summary