Training and test sets

1. Training and test sets

To really test how good your forecasting method is, there is no substitute for actually using it to forecast future observations.

2. Training and test sets

You probably don't want to wait around for a few years to see how it goes.

3. Training and test sets

Instead, you can hide some observations at the end of the series, then try to forecast them. No peeking!

4. Training and test sets

If you look at the hidden observations, and that influences your forecasts, then it is not a fair test. The observations used to build your forecasts are called the "training set". The remaining hidden observations form the test set. It is easy to build a very complicated model on the training data that has tiny residuals and looks like a great fit. But it produces terrible forecasts. This is called over-fitting. Checking the forecast performance on the test set helps overcome the problem of over-fitting the training set.

5. Example: Saudi Arabian oil production

In this example, we split the data into training and test sets using the window function. We keep the last 10 years of data for testing, and compute naive forecasts. Because there are no parameters associated with a naive forecast, there actually isn't much point using a test set in this example. But the principle is important, and we will use this approach when we have more complicated forecasting models.

6. Forecast errors

Forecast errors are the differences between the test set observations and the point forecasts. This is different from residuals in two ways. First, residuals are on the training set, while forecast errors are on the test set. Second, residuals are based on one-step forecasts, while forecast errors can be from any forecast horizon. We compute the accuracy of our method using the forecast errors calculated on the test data.

7. Measures of forecast accuracy

There are a number of ways to measure forecast accuracy. We can take the average absolute error, the average squared error or the average percentage error. All of these are widely used and their equations are shown here. But there are problems with these simple measures. If we want to compare forecast accuracy between two series on very different scales, we can't compare the MAE or MSE because their size depends on the scale of the data. MAPE is better for comparisons, but only if our data are all positive and have no zeros or small values. MAPE also assumes there is a natural zero, so it can't be used with temperature forecasts, for example, as the Fahrenheit and Celsius scales have arbitrary zero points. A solution is the mean absolute scaled error or MASE, which is like the MAE but is scaled so that it can be compared across series. In all cases, a small value indicates a better forecast.

8. The accuracy() command

Once again, R makes our life easy by providing a function that does most of the work for us. The accuracy command computes all of these measures, plus a few others that we won't discuss here. The training set measures are based on the residuals, while the test set measures are based on the forecast errors. In most cases, we are interested in the test set error measures. On their own, these don't tell us much. But when we compare different forecast methods on the same data, these will be very useful in telling us what works and what doesn't.

9. Let's practice!

Now it's time for you to practice using the accuracy function.

This exercise is part of the course

Forecasting in R

IntermediateSkill Level

4.8+

Start Course for Free

The first thing to do in any data analysis task is to plot the data. Graphs enable many features of the data to be visualized, including patterns, unusual observations, and changes over time. The features that are seen in plots of the data must then be incorporated, as far as possible, into the forecasting methods to be used.

Exercise 1: Welcome to Forecasting Using R Exercise 2: Creating time series objects in R Exercise 3: Time series plots Exercise 4: Seasonal plots Exercise 5: Trends, seasonality, and cyclicity Exercise 6: Autocorrelation of non-seasonal time series Exercise 7: Autocorrelation of seasonal and cyclic time series Exercise 8: Match the ACF to the time series Exercise 9: White noise Exercise 10: Stock prices and white noise

In this chapter, you will learn general tools that are useful for many different forecasting situations. It will describe some methods for benchmark forecasting, methods for checking whether a forecasting method has adequately utilized the available information, and methods for measuring forecast accuracy. Each of the tools discussed in this chapter will be used repeatedly in subsequent chapters as you develop and explore a range of forecasting methods.

Exercise 1: Forecasts and potential futures Exercise 2: Naive forecasting methods Exercise 3: Fitted values and residuals Exercise 4: Checking time series residuals Exercise 5: Training and test sets

Current Exercise

Exercise 6: Evaluating forecast accuracy of non-seasonal methods Exercise 7: Evaluating forecast accuracy of seasonal methods Exercise 8: Do I have a good forecasting model?Exercise 9: Time series cross-validation Exercise 10: Using tsCV() for time series cross-validation

Forecasts produced using exponential smoothing methods are weighted averages of past observations, with the weights decaying exponentially as the observations get older. In other words, the more recent the observation, the higher the associated weight. This framework generates reliable forecasts quickly and for a wide range of time series, which is a great advantage and of major importance to applications in business.

Exercise 1: Exponentially weighted forecasts Exercise 2: Simple exponential smoothing Exercise 3: SES vs naive Exercise 4: Exponential smoothing methods with trend Exercise 5: Holt's trend methods Exercise 6: Exponential smoothing methods with trend and seasonality Exercise 7: Holt-Winters with monthly data Exercise 8: Holt-Winters method with daily data Exercise 9: State space models for exponential smoothing Exercise 10: Automatic forecasting with exponential smoothing Exercise 11: ETS vs seasonal naive Exercise 12: Match the models to the time series Exercise 13: When does ETS fail?

ARIMA models provide another approach to time series forecasting. Exponential smoothing and ARIMA models are the two most widely-used approaches to time series forecasting, and provide complementary approaches to the problem. While exponential smoothing models are based on a description of the trend and seasonality in the data, ARIMA models aim to describe the autocorrelations in the data.

Exercise 1: Transformations for variance stabilization Exercise 2: Box-Cox transformations for time series Exercise 3: Non-seasonal differencing for stationarity Exercise 4: Seasonal differencing for stationarity Exercise 5: ARIMA models Exercise 6: Automatic ARIMA models for non-seasonal time series Exercise 7: Forecasting with ARIMA models Exercise 8: Comparing auto.arima() and ets() on non-seasonal data Exercise 9: Seasonal ARIMA models Exercise 10: Automatic ARIMA models for seasonal time series Exercise 11: Exploring auto.arima() options Exercise 12: Comparing auto.arima() and ets() on seasonal data

The time series models in the previous chapters work well for many time series, but they are often not good for weekly or hourly data, and they do not allow for the inclusion of other information such as the effects of holidays, competitor activity, changes in the law, etc. In this chapter, you will look at some methods that handle more complicated seasonality, and you consider how to extend ARIMA models in order to allow other information to be included in the them.

Exercise 1: Dynamic regression Exercise 2: Forecasting sales allowing for advertising expenditure Exercise 3: Forecasting electricity demand Exercise 4: Dynamic harmonic regression Exercise 5: Forecasting weekly data Exercise 6: Harmonic regression for multiple seasonality Exercise 7: Forecasting call bookings Exercise 8: TBATS models Exercise 9: TBATS models for electricity demand Exercise 10: Your future in forecasting!