1. Aprende
    2. /
  2. Cursos
    3. /
  3. Building Data Engineering Pipelines in Python
  • 1

    Ingesting Data

    Gratuito

    After seeing this chapter, you will be able to explain what a data platform is, how data ends up in it, and how data engineers structure its foundations. You will be able to ingest data from a RESTful API into the data platform’s data lake using a self-written ingestion pipeline, made using Singer’s taps and targets.

  • 2

    Creating a data transformation pipeline with PySpark

    You will learn how to process data in the data lake in a structured way using PySpark. Of course, you must first understand when PySpark is the right choice for the job.

  • 3

    Testing your data pipeline

    Stating “it works on my machine” is not a guarantee it will work reliably elsewhere and in the future. Requirements for your project will change. In this chapter, we explore different forms of testing and learn how to write unit tests for our PySpark data transformation pipeline, so that we make robust and reusable parts.

  • 4

    Managing and orchestrating a workflow

    We will explore the basics of Apache Airflow, a popular piece of software that allows you to trigger the various components of an ETL pipeline on a certain time schedule and execute tasks in a specific order. Here too, we illustrate how a deployment of Apache Airflow can be tested automatically.

Initializing

Ejercicio

Grouping and aggregating data

Often you will want to compute a metric over a set of values that share a common characteristic, like the average price of a house in a certain region. To achieve this, you would need to group the data by region and compute an aggregate metric on that subgroup of data.

We’ve already seen in the video a couple of these aggregation metrics, on landingprices.csv_. We’ll inspect a few more now and apply them to _~/workspace/mnt/data_lake/landing/purchased.csv_. In particular, you’ll use the spark.sql aggregation functions avg() to compute the average value of some column in a group, stddev_samp() to compute the standard (sample) deviation and max() (which we alias as sfmax so as not to shadow Python’s built-in max()) to retrieve the largest value of some column in a group.

Instrucciones

100 XP
  • Use the .groupBy() method to group the data by the “Country” column.
  • In these groups, compute the average of the “Salary” column and name the resulting column “average_salary”.
  • Compute the standard deviation of the “Salary” column in each group in the same aggregation.
  • Retrieve the largest “Salary” in each group, in the same aggregation, and name the resulting column “highest_salary”.