CommencerCommencer gratuitement

Integers in PySpark UDFs

This exercise covers UDFs, allowing you to understand function creation in PySpark! As you work through this exercise, think about what this would replace in a data cleaning workflow.

Remember, there's already a SparkSession called spark in your workspace!

Cet exercice fait partie du cours

Introduction to PySpark

Afficher le cours

Instructions

  • Register the function age_category as a UDF called age_category_udf.
  • Add a new column to the DataFrame df called "category" that applies the UDF to categorize people based on their age. The argument for age_category_udf() is provided for you.
  • Show the resulting DataFrame.

Exercice interactif pratique

Essayez cet exercice en complétant cet exemple de code.

# Register the function age_category as a UDF
age_category_udf = ____(age_category, StringType())

# Apply your udf to the DataFrame
age_category_df_2 = age_category_df.withColumn("category", ____(age_category_df["age"]))

# Show df
age_category_df_2.____
Modifier et exécuter le code