Model Scoring

1. Model Scoring

Now let's talk about model scoring,

2. Workflow

which is the step after training the models.

3. Reformat the backtesting output

The `cross_validation()` function returns the model's forecast for each partition using a wide format. The `unique_id`, `ds`, `y`, and `cutoff` fields represent the series id, timestamp, actual, and partition label, respectively. Where, partition label is set by the partition first timestamp. The rest of the fields represent the forecast values and their corresponding prediction intervals.

4. Reformat the backtesting output

For example, the `lightGBM` model will have the following three fields, representing the model forecast and lower and upper 95% conformal prediction intervals.

5. Reformat the backtesting output

Let's start by creating a mapping for the partitions label and their numeric values

6. Reformat the backtesting output

and for the models names and their labels.

7. Reformat the backtesting output

For convenience, let's reformat the data frame from a wide to a long format. We will extract the corresponding model's field names and the lower and upper conformal prediction intervals using the pandas `melt()` function.

8. Reformat the backtesting output

This will transform the backtesting table to long format where the model label represents the corresponding forecast and the conformal prediction intervals labels.

9. Reformat the backtesting output

We will set a custom function to parse and reformat the conformal prediction intervals and forecast labels into lower, upper, and forecast, respectively. and apply it using the `zip()` and `map()` functions to assign the reformatted labels into a new field called type.

10. Reformat the backtesting output

Lastly, we merge the partition mapping and pivot the type field into three new columns - forecast, lower, and upper, and merge the output with the models mapping table.

11. Reformat the backtesting output

Here is a preview of the long-format DataFrame. Now we can score the models performance.

12. Performance metrics

As before, we will use the following functions to calculate the MAPE, RMSE, and coverage for each model.

13. Performance metrics

Let's also define a helper function to create the score fields for each performance metric,

14. Performance metrics

and then group the backtesting table by `unique_id`, `model_label` and `name`, and `partition`, using the `score` function to calculate the performance metrics.

15. Performance metrics

This returns the models score on each testing partition.

16. Workflow

We can now move to the next step and log the model's performance.

17. MLflow

To demonstrate the models' logging and tracking we will use the MLflow library, which is an open-source platform and industry standard for experimentation tracking.

18. MLflow workflow

To log the model's score we will: Refer to a new or existing experiment, use the run command to log the model metadata, including model parameters and score, then launch the MLflow UI to analyze the experiment results.

19. Log the backtesting

Let's import the mlflow and datetime libraries, and define the experiment name and the path to save the experiment metadata. Next, we will define the experiment tag, in this case, logging the backtesting parameters.

20. Log the backtesting

Let's create a new experiment with the `create_experiment` function, and pull the experiment metadata with the `get_experiment_by_name` function. We can simply pull the experiment metadata by its name if the experiment already exists. Next, we will set the experiment label using the run time.

21. Log the backtesting

We will loop over the score DataFrame rows and log each models' performance by partition. We will label each model using the model label and experiment run time. Next, we will use the `start_run()` function to tag the model's metadata and log the model's parameters and the model's score. In the next section, we will analyze the experiment results using the MLflow UI.

22. Let's practice!

Now, let's practice scoring forecasting models!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.