Cross-validation statistics

You used grid search CV to tune your random forest classifier, and now want to inspect the cross-validation results to ensure you did not overfit. In particular you would like to take the difference of the mean test score for each fold from the mean training score. The dataset is available as X_train and y_train, the pipeline as pipe, and a number of modules are pre-loaded including pandas as pd and GridSearchCV().

This exercise is part of the course

Designing Machine Learning Workflows in Python

View Course

Exercise instructions

Create a grid search object with three cross-validation folds and ensure it returns training as well as test statistics.
Fit the grid search object to the training data.
Store the results of the cross-validation, available in the cv_results_ attribute of the fitted CV object, into a dataframe.
Print the difference between the column containing the average test score and that containing the average training score.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Fit your pipeline using GridSearchCV with three folds
grid_search = GridSearchCV(
  pipe, params, ____=3, return_train_score=____)

# Fit the grid search
gs = grid_search.____(____, ____)

# Store the results of CV into a pandas dataframe
results = pd.____(gs.____)

# Print the difference between mean test and training scores
print(
  results[____]-results['mean_train_score'])

Edit and Run Code

Designing Machine Learning Workflows in Python

AdvancedSkill Level

4.8+

10 reviews

In the previous chapters you established a solid foundation in supervised learning, complete with knowledge of deploying models in production but always assumed you a labeled dataset would be available for your analysis. In this chapter, you take on the challenge of modeling data without any, or with very few, labels. This takes you into a journey into anomaly detection, a kind of unsupervised modeling, as well as distance-based learning, where beliefs about what constitutes similarity between two examples can be used in place of labels to help you achieve levels of accuracy comparable to a supervised workflow. Upon completing this chapter, you will clearly stand out from the crowd of data scientists in confidently knowing what tools to use to modify your workflow in order to overcome common real-world challenges.

Exercise 1: Anomaly detection Exercise 2: A simple outlier Exercise 3: LoF contamination Exercise 4: Novelty detection Exercise 5: A simple novelty Exercise 6: Three novelty detectors Exercise 7: Contamination revisited Exercise 8: Distance-based learning Exercise 9: Find the neighbor Exercise 10: Not all metrics agree Exercise 11: Unstructured data Exercise 12: Restricted Levenshtein Exercise 13: Bringing it all together Exercise 14: Concluding remarks