1. Learn
    2. /
  2. Courses
    3. /
  3. Building Recommendation Engines with PySpark

Exercise

Generate predictions and calculate RMSE

Now that we have a model that is trained on our data and tuned through cross validation, we can see how it performs on the test dataframe. To do this, we'll calculate the RMSE.

As a side note, the generation of test predictions takes more than a few minutes with this dataset. For this reason, the test predictions have been generated already and are provided here as a dataframe called test_predictions. For your reference, they are generated using this code: test_predictions = best_model.transform(test).

Instructions

100 XP
  • The dataframe test_predictions contains predictions that our cross-validated ALS model generated using the test set that we created previously. Use the .show() method to take a look at it and see if the predictions seem close.
  • Use the evaluator that you built previously to calculate the RMSE by calling the .evaluate() method on the test_predictions generated. Call this RMSE.
  • Print the RMSE.