1. Learn
    2. /
  2. Courses
    3. /
  3. Introduction to PySpark

Exercise

Create the evaluator

The first thing you need when doing cross validation for model selection is a way to compare different models. Luckily, the pyspark.ml.evaluation submodule has classes for evaluating different kinds of models. Your model is a binary classification model, so you'll be using the BinaryClassificationEvaluator from the pyspark.ml.evaluation module.

This evaluator calculates the area under the ROC. This is a metric that combines the two kinds of errors a binary classifier can make (false positives and false negatives) into a simple number. You'll learn more about this towards the end of the chapter!

Instructions

100 XP
  • Import the submodule pyspark.ml.evaluation as evals.
  • Create evaluator by calling evals.BinaryClassificationEvaluator() with the argument metricName="areaUnderROC".