Building models for prediction · 117 monthly utility bills

Train, test, repeat

Each dot is one month of one household's gas bill: the average temperature that month and the gas spend per day. We want a model that predicts next month's bill, not one that just describes the months we already have.

The trick is to hide some of the data. Fit the model on the training set, then score it on the test set, months the model never saw. The error on the test set, the RMSPE, is our estimate of how it will do on new data.

Split
24 of 117 (21%)

Step 1

Fit on training, score on test

Pick a polynomial degree. The blue curve is fit to the gray training points only. The orange lines show how far each test point lands from that curve. Those misses are what RMSPE measures.

Polynomial degree
Training monthTest monthModel fit to training only
Training RMSE0.73How well it fits months it has seen
Test RMSPE0.85How well it predicts months it hasn't
Gap+16%Test error is usually worse. That gap is optimism you'd have reported if you'd only checked the training data.

Step 2

Training error always falls. Test error doesn't.

Here is the same split scored at every degree from 1 to 10. Each extra term gives the curve more freedom to chase the training points, so training error can only go down. Test error drops while the extra terms capture real curvature, then rises once they start fitting noise. That rise is overfitting.

Training RMSETest RMSPELowest test error

To choose a model, fit each candidate on the training set, compute RMSPE on the test set, and keep the one with the lowest RMSPE. Never pick based on training error. It always prefers the most complicated model.

Step 3

One split is one draw. Luck matters.

The test months are chosen at random, so a different shuffle gives a different RMSPE. Draw 200 random splits at the current test size and watch how much the answer moves for the three models from class.

Each dot will be one random split. The ring is your current split.
ModelTraining RMSE (all 117)Your split's RMSPEAverage over splitsMiddle 80% of splits

Try changing how many of the 117 utility bills go into the test set, using the Test size slider in the bar at the top, then press Draw 200 splits again.

More test data gives a steadier estimate of prediction error. More training data gives a better model. The common 80/20 split is a compromise between the two.

Averaging over many splits instead of trusting one is the idea behind cross-validation. With k-fold CV, every month gets to be in the test set exactly once.

Step 4

Why the split has to be random

Switch the split to Hottest months in the bar at the top. Now the model trains only on cooler months and is tested on the hottest ones, so it has to extrapolate. The 9th-degree curve has nothing holding it down past the last training point, and its test error explodes. A random split makes the test set look like the data the model will actually face.

In R

The same steps in code

set.seed(9529) randomized <- slice_sample(utilities, n = 117) # shuffle utilities_test <- randomized[1:24, ] # about 20% for testing utilities_train <- randomized[25:117, ] # the rest for training model2 <- lm(dailyspend ~ temp + I(temp^2), data = utilities_train) predictions2 <- predict(model2, utilities_test) sqrt(mean((utilities_test$dailyspend - predictions2)^2)) # RMSPE

Data: 117 monthly bills from the mosaicData Utilities set; dailyspend = gas bill ÷ billing days, temperature in °F. Fit to all 117 months, the degree 1, 2 and 9 models have RSE 0.866, 0.755 and 0.750, matching the course output. Splits here use their own random number generator, so they won't match set.seed(9529) exactly. The page opens on a split that shows the typical pattern; press New random split to see others. Companion lab: Fold by Fold.