Flight duration model: Pipeline model
You're now ready to put those stages together in a pipeline.
You'll construct the pipeline and then train the pipeline on the training data. This will apply each of the individual stages in the pipeline to the training data in turn. None of the stages will be exposed to the testing data at all: there will be no leakage!
Once the entire pipeline has been trained it will then be used to make predictions on the testing data.
The data are available as flights, which has been randomly split into flights_train and flights_test.
This exercise is part of the course
Machine Learning with PySpark
Exercise instructions
- Import the class for creating a pipeline.
- Create a pipeline object and specify the
indexer,onehot,assemblerandregressionstages, in this order. - Train the pipeline on the training data.
- Make predictions on the testing data.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Import class for creating a pipeline
from pyspark.____ import ____
# Construct a pipeline
pipeline = ____(____=[____])
# Train the pipeline on the training data
pipeline = pipeline.____(____)
# Make predictions on the testing data
predictions = ____.____(____)