Assessing model performance
1. Assessing model performance
The big benefit of using more than one explanatory variable in a model is that you can sometimes get a better fit than when you use a single explanatory variable.2. Model performance metrics
In the last course, you saw two metrics for measuring model performance: the coefficient of determination, and the residual standard error. The coefficient of determination, sometimes known as the R-squared value, measures how good the regression's prediction line fits the observed values, and a larger number is better. The residual standard error, sometimes abbreviated as RSE, is - loosely speaking - the typical size of the residuals. Let's see if these metrics improve when both explanatory variables are included in the fish model.3. Getting the coefficient of determination
To easily get the coefficient of determination, you can use the rsquared attribute of the fitted model. For the mass versus length model, the coefficient of determination is zero-point-eight-two, where zero is the worst possible fit and one is a perfect fit. For the mass versus species model, the coefficient of determination is worse, at zero-point-two-five. For the mass versus both model, the coefficient of determination is the highest, at zero-point-nine-two. Using this metric, the model with both explanatory variables is the best one, since it has the highest coefficient of determination.4. Adjusted coefficient of determination
Adding more explanatory variables often increases the coefficient of determination for a model, but there is a problem. Including too many explanatory variables in your model can lead to a phenomenon called overfitting. That's when your model is optimized to provide the best fit for that particular dataset, but no longer reflects the general population. In this case, the model would be overfit if it performed well on this fish dataset, but badly on a different fish dataset. A variant metric called adjusted coefficient of determination includes a small penalty term for each additional explanatory variable to compensate for this effect. Its a better metric than the plain coefficient of determination. Its equation is based on the plain coefficient of determination, the number of observations, and the number of explanatory variables, including interactions. The penalty is big enough to worry about if the plain coefficient of determination is small, or if the number of explanatory variables is a sizable fraction of the number of observations. To get this metric, we use the rsquared_adj attribute.5. Getting the adjusted coefficient of determination
To see the effect of penalization, let's look at the unadjusted and adjusted coefficients side-by-side. For the mass versus length model, the R-squared metric is point-822, while the adjusted R-squared of that same model is point-821 - a tiny difference. Since each model only contains one or two explanatory variables, the effect is minimal in all models.6. Getting the residual standard error
The code to get the residual standard error is the same as before. RSE isn't directly available as an attribute, but the MSE is. Recall that the MSE is the mean squared error. Consequently, you take the square root of the mse_resid attribute to get the RSE. The mass versus length model has an RSE of just over one hundred fifty. The mass versus species model has an RSE of about three hundred and thirteen. Finally, the mass versus both model has an RSE of just over one hundred, meaning that it typically gets the mass wrong by about one hundred grams. Since that number is the lowest of the three, by this metric, the mass versus both model is best. That means that all metrics indicate that the model with two explanatory variables is better than the models with just one explanatory variable.7. Let's practice!
Your turn to look at some model metrics.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.