1. Time series cross-validation
Up to now, we have used this traditional evaluation set up.
2. Time series cross-validation
The problem is that it is wasteful. We have a relatively small test set, and it is possible that we could draw conclusions that work for that test set but which are not reliable for future times.
Time series cross-validation is a solution to this problem.
3. Time series cross-validation
In time series cross-validation, we have a series of training and test sets. Assuming we are interested in one-step forecasts, the set up would look like this. For each row, the blue dots show the training set, and the red dot the test set. The white dots are not used. Each training set consists of just one more observation than the previous training set. In this way, many more observations can be used in the test set, and we can evaluate how good the method is for one-step forecasting by averaging the error over all those tiny test sets.
This is known in econometrics as "forecast evaluation on a rolling origin". (For some reason, econometricians never seem to come up with memorable names!) The forecast origin is the time at the end of the training data. And it rolls forward in time. Hence the name.
But I like to call it "time series cross-validation", because it is analogous to cross-validation for non-time-series problems.
We can do something similar for multi-step forecasting,
4. Time series cross-validation
such as 2-steps ahead,
5. Time series cross-validation
and 3-steps ahead.
It looks like a lot of work to do, but there is a neat R function that does everything for you.
6. tsCV function
tsCV applies a forecasting method on a sequence of training sets computed from a time series. It works like this.
The resulting forecast errors are saved in the object e. There may be some missing values, especially at the start of the series, because it is simply not possible to compute the forecast when the training error is too small.
You need to compute your own error measures when you use this function. I've computed the mean squared error for you here as an example.
This is actually a strange example, because the naive method has no parameters to estimate. Therefore, tsCV will produce the same values as the residuals function in this case.
7. tsCV function
Here is a more sophisticated use of the function. I am computing the mean squared error for different forecast horizons based on time series cross-validation. The pipe operator comes in handy here to simplify my code. However, I did need to write my own square function because the usual approach won't work inside a sequence of pipes.
Notice how the MSE increases with the forecast horizon. The further ahead we forecast, the less accurate our forecasts are.
8. tsCV function
In summary, time series cross-validation is very useful for selecting a good forecasting model. In general, a good rule is to choose the model with the smallest MSE.
If a particular forecast horizon is of interest, then compute the cross-validated MSE at that horizon. That way, you are choosing the best forecast model for your purpose.
9. Let's practice!
Now, let's use the tsCV function on a different data set in the next exercise.