Filtering out highly correlated features
You're going to automate the removal of highly correlated features in the numeric ANSUR dataset. You'll calculate the correlation matrix and filter out columns that have a correlation coefficient of more than 0.95 or less than -0.95.
Since each correlation coefficient occurs twice in the matrix (correlation of A to B equals correlation of B to A) you'll want to ignore half of the correlation matrix so that only one of the two correlated features is removed. Use a mask trick for this purpose.
This is a part of the course
“Dimensionality Reduction in Python”
Exercise instructions
- Calculate the correlation matrix of
ansur_dfand take the absolute value of this matrix. - Create a boolean mask with
Truevalues in the upper right triangle and apply it to the correlation matrix. - Set the correlation coefficient threshold to
0.95. - Drop all the columns listed in
to_dropfrom the DataFrame.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Calculate the correlation matrix and take the absolute value
corr_df = ansur_df.____().____()
# Create a True/False mask and apply it
mask = np.____(np.____(corr_df, dtype=____))
tri_df = corr_df.____(mask)
# List column names of highly correlated features (r > 0.95)
to_drop = [c for c in tri_df.columns if any(tri_df[c] > ____)]
# Drop the features in the to_drop list
reduced_df = ansur_df.____(____, axis=1)
print(f"The reduced_df DataFrame has {reduced_df.shape[1]} columns.")