Calculate specificity
Using different measures for model performance allows you to more accurately assess it. There are several metrics for different use cases. Specificity measures the proportion of true negative outcomes correctly identified:
$$\text{specificity} = \frac{TN}{TN + FP}$$
This formula implies that with specificity approaching 100%, the number of false positives (FP) approaches 0.
In this exercise, you are going to investigate the out-of-sample specificity of your model with cross-validation.
Pre-loaded is the training data of the credit card customers dataset, customers_train
, and a decision tree specification, tree_spec
, which was generated using the following code:
tree_spec <- decision_tree() %>%
set_engine("rpart") %>%
set_mode("classification")
This exercise is part of the course
Machine Learning with Tree-Based Models in R
Exercise instructions
- Create three CV folds of
customers_train
and save them asfolds
. - Calculate cross-validated
specificity
using thefit_resamples()
function that takes your specificationtree_spec
, a model formula, the CV folds, and an appropriate metric set. Use all predictors to predictstill_customer
, saving the results tospecificities
. - Aggregate the results using a single function.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Create CV folds of the training data
folds <- ___(customers_train, v = ___)
# Calculate CV specificity
specificities <- ___(___,
___,
resamples = ___,
metrics = ___)
# Collect the metrics
___(specificities)