Building models for prediction · 117 monthly utility bills
Train, test, repeat
Each dot is one month of one household's gas bill: the average temperature that month and the gas spend per day. We want a model that predicts next month's bill, not one that just describes the months we already have.
The trick is to hide some of the data. Fit the model on the training set, then score it on the test set, months the model never saw. The error on the test set, the RMSPE, is our estimate of how it will do on new data.
Step 1
Fit on training, score on test
Pick a polynomial degree. The blue curve is fit to the gray training points only. The orange lines show how far each test point lands from that curve. Those misses are what RMSPE measures.
Step 2
Training error always falls. Test error doesn't.
Here is the same split scored at every degree from 1 to 10. Each extra term gives the curve more freedom to chase the training points, so training error can only go down. Test error drops while the extra terms capture real curvature, then rises once they start fitting noise. That rise is overfitting.
To choose a model, fit each candidate on the training set, compute RMSPE on the test set, and keep the one with the lowest RMSPE. Never pick based on training error. It always prefers the most complicated model.
Step 3
One split is one draw. Luck matters.
The test months are chosen at random, so a different shuffle gives a different RMSPE. Draw 200 random splits at the current test size and watch how much the answer moves for the three models from class.
| Model | Training RMSE (all 117) | Your split's RMSPE | Average over splits | Middle 80% of splits |
|---|
Try changing how many of the 117 utility bills go into the test set, using the Test size slider in the bar at the top, then press Draw 200 splits again.
- Small test set (12 bills): the dots spread out. Each RMSPE is an average of only 12 misses, so it depends heavily on which 12 bills happened to land in the test set.
- Large test set (58 bills): the dots bunch together, because each RMSPE averages more misses. But now the models are trained on only 59 bills instead of 93, so they're fit with less information.
More test data gives a steadier estimate of prediction error. More training data gives a better model. The common 80/20 split is a compromise between the two.
Averaging over many splits instead of trusting one is the idea behind cross-validation. With k-fold CV, every month gets to be in the test set exactly once.
Step 4
Why the split has to be random
Switch the split to Hottest months in the bar at the top. Now the model trains only on cooler months and is tested on the hottest ones, so it has to extrapolate. The 9th-degree curve has nothing holding it down past the last training point, and its test error explodes. A random split makes the test set look like the data the model will actually face.
In R
The same steps in code
Data: 117 monthly bills from the mosaicData Utilities set; dailyspend = gas bill ÷ billing days, temperature in °F. Fit to all 117 months, the degree 1, 2 and 9 models have RSE 0.866, 0.755 and 0.750, matching the course output. Splits here use their own random number generator, so they won't match set.seed(9529) exactly. The page opens on a split that shows the typical pattern; press New random split to see others. Companion lab: Fold by Fold.