Training a spam classifier
The SMS data have now been prepared for building a classifier. Specifically, this is what you have done:
- removed numbers and punctuation
- split the messages into words (or "tokens")
- removed stop words
- applied the hashing trick and
- converted to a TF-IDF representation.
Next you'll need to split the TF-IDF data into training and testing sets. Then you'll use the training data to fit a Logistic Regression model and finally evaluate the performance of that model on the testing data.
The data are stored in sms and LogisticRegression has been imported for you.
Questo esercizio fa parte del corso
Machine Learning with PySpark
Istruzioni dell'esercizio
- Split the data into training and testing sets in a 4:1 ratio. Set the random number seed to 13 to ensure repeatability.
- Create a
LogisticRegressionobject and fit it to the training data. - Generate predictions on the testing data.
- Use the predictions to form a confusion matrix.
Esercizio pratico interattivo
Prova a risolvere questo esercizio completando il codice di esempio.
# Split the data into training and testing sets
sms_train, sms_test = sms.____(____, ____)
# Fit a Logistic Regression model to the training data
logistic = ____(regParam=0.2).____(____)
# Make predictions on the testing data
prediction = logistic.____(____)
# Create a confusion matrix, comparing predictions to known labels
prediction.groupBy(____, ____).____().____()