1. Automatic feature selection
Unlike some machine learning methods, regression typically asks the human to specify the model's predictors ahead of time.
Thus, each of the donation models you've built so far required a little bit of fund-raising subject-matter expertise to identify the variables that may be predictive of donations.
Sometimes you may not have this type of insight ahead of time. You may not know what all of the predictors mean, or you may have so many predictors there's no easy way to sort through them all.
A process called automatic feature selection can be used here, but as you'll soon see, with this great power comes great responsibility to apply it carefully.
2. Stepwise regression
Stepwise regression involves building a regression model step by step, evaluating each predictor to see which ones add value to the final model.
A procedure called backward deletion begins with a model containing all of the predictors. It then checks to see what happens when each one of the predictors is removed from the model. If removing a predictor does not substantially impact the model's ability to predict the outcome, then it can be safely deleted. At each step, the predictor that impacts the model the least is removed-- assuming, of course, it has minimal impact. This continues step-by-step until only important predictors remain.
The same idea applied in the other direction is called forward selection. Beginning with a model containing no predictors, it examines each potential predictor to see which one, if any, offers the greatest improvement to the model's predictive power. Predictors are added step-by-step until no new predictors add substantial value to the model.
Keep in mind that although the figures here show the same final model for backward and forward stepwise, this is not always the case. It is possible that the two could come to completely different conclusions about the most important predictors.
3. Stepwise regression caveats
This is just one of the potential caveats of stepwise regression. Not only can backward and forward selection create completely different models, but neither is guaranteed to find the best possible model.
Statisticians also raise concerns about the fact that a stepwise model violates some of the principles that allow a regression model to explain data as well as predict. Of course, if you only care about the PREDICTIVE power, this may not be a very big concern-- the use of stepwise doesn't mean the model's predictions are worthless; it simply means that the model may over or understate the importance of some of the predictors.
Perhaps most importantly, feature selection methods like stepwise regression allow the model to be built in the absence of theory or even common sense. This can result in a model that seems counterintuitive in the real world.
It may be best to consider stepwise regression as just one tool for exploring potential models in the absence of another good starting point.
4. Let's practice!
You'll have a chance to see how to build a stepwise regression model during the next coding exercise.