Imputing Missing Data
Missing data happens. If we make the assumption that our data is missing completely at random, we are making the assumption that what data we do have, is a good representation of the population. If we have a few values we could remove them or we could use the mean or median as a replacement. In this exercise, we will look at 'PDOM'
: Days on Market at Current Price.
This exercise is part of the course
Feature Engineering with PySpark
Exercise instructions
- Get a count of the missing values in the column
'PDOM'
usingwhere()
,isNull()
andcount()
. - Calculate the mean value of
'PDOM'
using the aggregate functionmean()
. - Use
fillna()
with the value set to the'PDOM'
mean value and only apply it to the column'PDOM'
using thesubset
parameter.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Count missing rows
missing = df.____(df[____].____()).____()
# Calculate the mean value
col_mean = df.____({____: ____}).____()[0][0]
# Replacing with the mean value for that column
df.____(____, ____=[____])