Filtering out highly correlated features

You're going to automate the removal of highly correlated features in the numeric ANSUR dataset. You'll calculate the correlation matrix and filter out columns that have a correlation coefficient of more than 0.95 or less than -0.95.

Since each correlation coefficient occurs twice in the matrix (correlation of A to B equals correlation of B to A) you'll want to ignore half of the correlation matrix so that only one of the two correlated features is removed. Use a mask trick for this purpose.

Calculate the correlation matrix of ansur_df and take the absolute value of this matrix.
Create a boolean mask with True values in the upper right triangle and apply it to the correlation matrix.
Set the correlation coefficient threshold to 0.95.
Drop all the columns listed in to_drop from the dataframe.

Exercise

Filtering out highly correlated features

Instructions