Splitting & Exploding
Being able to take a compound field like GARAGEDESCRIPTION
and massaging it into something useful is an involved process. It's helpful to understand early what value you might gain out of expanding it. In this example, we will convert our string to a list-like array, explode it and then inspect the unique values.
This exercise is part of the course
Feature Engineering with PySpark
Exercise instructions
- Import the needed functions
split()
andexplode()
frompyspark.sql.functions
- Use
split()
to create a new columngarage_list
by splittingdf['GARAGEDESCRIPTION']
on ', ' which is both a comma and a space. - Create a new record for each value in the
df['garage_list']
usingexplode()
and assign it a new columnex_garage_list
- Use
distinct()
to get unique values ofex_garage_list
andshow
the 100 first rows, truncating them at 50 characters to display the values.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Import needed functions
____ ____ ____ ____, ____
# Convert string to list-like array
df = df.withColumn(____, ____(____, ____))
# Explode the values into new records
ex_df = df.withColumn(____, ____(____))
# Inspect the values
ex_df[['ex_garage_list']].____().____(100, ____=____)