Selecting the best precision model
Your boss has offered to pay for you to see three sports games this year. Of the 41 home games your favorite team plays, you want to ensure you go to three home games that they will definitely win. You build a model to decide which games your team will win.
To do this, you will build a random search algorithm and focus on model precision (to ensure your team wins). You also want to keep track of your best model and best parameters, so that you can use them again next year (if the model does well, of course). You have already decided on using the random forest classification model rfc
and generated a parameter distribution param_dist
.
This exercise is part of the course
Model Validation in Python
Exercise instructions
- Create a precision scorer,
precision
usingmake_scorer(<scoring_function>)
. - Complete the random search method by using
rfc
andparam_dist
. - Use
rs.cv_results_
to print the mean test scores. - Print the best overall score.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
from sklearn.metrics import precision_score, make_scorer
# Create a precision scorer
precision = ____(____)
# Finalize the random search
rs = RandomizedSearchCV(
estimator=____, param_distributions=____,
scoring = precision,
cv=5, n_iter=10, random_state=1111)
rs.fit(X, y)
# print the mean test scores:
print('The accuracy for each run was: {}.'.format(rs.cv_results_['____']))
# print the best model score:
print('The best accuracy for a single model was: {}'.format(rs.____))