1. The intuition behind stacking
Welcome to the final chapter! Here, you'll learn how to build and apply an ensemble method known as Stacking. Let's begin by understanding the idea behind stacking predictive models.
2. Relay races
Consider a relay race, in which sprinters run until they pass the baton over to the next on track. This is a good example of teamwork. While all team members must be strong competitors, each individual has a special role to play based on their abilities. For example, the anchor is the last team member to receive the baton and the one who completes the race. In some sense, the anchor is like the leader of the team.
In order to be effective in relay races, team anchors must display certain characteristics. First, they must have an awareness of the individual strengths and weaknesses of each team member. Second, they should clearly define tasks. Each team member must know their responsibilities and focus on them. Finally, the anchor must participate in the race. Participation from the leader is important.
3. Relay race for models
Stacking ensembles leverage this idea of relay races. Instead of passing a baton, individual models pass their predictions together with the input features to the next model.
4. Stacking architecture
Here's a diagram depicting the architecture of stacking ensembles. Each individual model uses the same dataset and input features. These are the first-layer estimators. Then, estimators pass their predictions as additional input features to the second-layer estimator.
So far, we have seen ensemble methods that use simple arithmetic operations like the mean or the mode as combiners. However, in Stacking, the combiner is itself a trainable model.
In addition, this combiner model has not only the predictions as input features, but also the original dataset. This allows it to determine which estimator is more accurate depending on the input features.
5. Combiner model as anchor
The combiner model plays a similar role to the anchor in the relay race. It is also the last team member and the one which provides the final predictions.
In order to be effective, the combiner model must display the same characteristics as the team anchor.
It should learn to identify the strengths and weaknesses of the individual estimators.
It also defines tasks. Depending on the input features, it chooses which model provides the best prediction.
In addition, the combiner is itself a model, and therefore takes part in the team job of learning useful patterns to predict the target.
6. Time to practice!
Let's now get some practice using stacking ensembles!