Challenge the champion

Having pushed your random forest to production, you suddenly worry that a naive Bayes classifier might be better. You want to run a champion-challenger test, by comparing a naive Bayes, acting as the challenger, to exactly the model which is currently in production, which you will load from file to make sure there is no confusion. You will use the F1 score for assessment. You have the data X_train, X_test, y_train and y_test available as before and GaussianNB(), f1_score() and pickle().

This exercise is part of the course

Designing Machine Learning Workflows in Python

View Course

Exercise instructions

Load the existing model from memory using pickle.
Fit a Gaussian Naive Bayes classifier to the training data.
Print the F1 score of the champion and then the challenger on the test data.
Overwrite the current model to disk with the one that performed best.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Load the current model from disk
champion = pickle.____(open('model.pkl', ____))

# Fit a Gaussian Naive Bayes to the training data
challenger = ____.____(X_train, y_train)

# Print the F1 test scores of both champion and challenger
print(____(y_test, champion.____(X_test)))
print(____)

# Write back to disk the best-performing model
with open('model.pkl', 'wb') as file:
    pickle.____(____, file=file)

Edit and Run Code

Designing Machine Learning Workflows in Python

AdvancedSkill Level

4.8+

10 reviews

In the previous chapters you established a solid foundation in supervised learning, complete with knowledge of deploying models in production but always assumed you a labeled dataset would be available for your analysis. In this chapter, you take on the challenge of modeling data without any, or with very few, labels. This takes you into a journey into anomaly detection, a kind of unsupervised modeling, as well as distance-based learning, where beliefs about what constitutes similarity between two examples can be used in place of labels to help you achieve levels of accuracy comparable to a supervised workflow. Upon completing this chapter, you will clearly stand out from the crowd of data scientists in confidently knowing what tools to use to modify your workflow in order to overcome common real-world challenges.

Exercise 1: Anomaly detection Exercise 2: A simple outlier Exercise 3: LoF contamination Exercise 4: Novelty detection Exercise 5: A simple novelty Exercise 6: Three novelty detectors Exercise 7: Contamination revisited Exercise 8: Distance-based learning Exercise 9: Find the neighbor Exercise 10: Not all metrics agree Exercise 11: Unstructured data Exercise 12: Restricted Levenshtein Exercise 13: Bringing it all together Exercise 14: Concluding remarks