Percentage based outlier removal
One way to ensure a small portion of data is not having an overly adverse effect is by removing a certain percentage of the largest and/or smallest values in the column. This can be achieved by finding the relevant quantile and trimming the data using it with a mask. This approach is particularly useful if you are concerned that the highest values in your dataset should be avoided. When using this approach, you must remember that even if there are no outliers, this will still remove the same top N percentage from the dataset.
This exercise is part of the course
Feature Engineering for Machine Learning in Python
Exercise instructions
- Find the 95th quantile of the
ConvertedSalarycolumn. - Trim the
so_numeric_dfDataFrame to retain all rows whereConvertedSalaryis less than it's 95th quantile. - Plot the histogram of
so_numeric_df[['ConvertedSalary']]. - Plot the histogram of
trimmed_df[['ConvertedSalary']].
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Find the 95th quantile
quantile = so_numeric_df['ConvertedSalary'].____(____)
# Trim the outliers
trimmed_df = so_numeric_df[so_numeric_df['ConvertedSalary'] < ____]
# The original histogram
so_numeric_df[['ConvertedSalary']].____()
plt.show()
plt.clf()
# The trimmed histogram
trimmed_df[['ConvertedSalary']].____()
plt.show()