Time Components

Being able to work with time components for building features is important but you can also use them to explore and understand your data further. In this exercise, you'll be looking to see if there is a pattern to which day of the week a house lists on. Please keep in mind that PySpark's week starts on Sunday, with a value of 1 and ends on Saturday, a value of 7.

This exercise is part of the course

Feature Engineering with PySpark

View Course

Exercise instructions

Import to_date() and dayofweek() functions from pyspark.sql.functions
Use the to_date() function to convert LISTDATE to a Spark date type, save the converted column in place using withColumn()
Create a new column using LISTDATE and dayofweek() then save it as List_Day_of_Week using withColumn()
Sample half the dataframe and convert it to a pandas dataframe with toPandas() and plot the count of the pandas dataframe's List_Day_of_Week column by using seaborn countplot() where x = List_Day_of_Week.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Import needed functions
from ____ import ____, ____

# Convert to date type
df = df.____(____, ____(____))

# Get the day of the week
df = df.____(____, ____(____))

# Sample and convert to pandas dataframe
sample_df = df.sample(False, ____, 42).____()

# Plot count plot of of day of week
sns.____(x="List_Day_of_Week", data=____)
plt.show()

Edit and Run Code