Dropping a list of columns
Our data set is rich with a lot of features, but not all are valuable. We have many that are going to be hard to wrangle into anything useful. For now, let's remove any columns that aren't immediately useful by dropping them.
'STREETNUMBERNUMERIC': The postal address number on the home'FIREPLACES': Number of Fireplaces in the home'LOTSIZEDIMENSIONS': Free text describing the lot shape'LISTTYPE': Set list of values of sale type'ACRES': Numeric area of lot size
This exercise is part of the course
Feature Engineering with PySpark
Exercise instructions
- Read the list of column descriptions above and explore their top 30 values with
show(), the dataframe is already filtered to the listed columns asdf - Create a list of two columns to drop based on their lack of relevance to predicting house prices called
cols_to_drop. Recall that computers only interpret numbers explicitly and don't understand context. - Use the
drop()function to remove the columns in the listcols_to_dropfrom the dataframedf.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Show top 30 records
df.____(____)
# List of columns to remove from dataset
cols_to_drop = [____, ____]
# Drop columns in list
df = df.____(____)