Building models for prediction · 117 monthly utility bills
Fold by fold
One train/test split can be lucky or unlucky. Cross-validation fixes that by giving every month a turn in the test set. Shuffle the data, cut it into k folds, test on each fold once, and average the errors.
Step 1
Shuffle, then cut into folds
Each square is one month, in shuffled order. The shuffle mixes hot and cold months, so every fold looks like the full data set.
Step 2
One round: train on the rest, test on one fold
The blue curve is fit to the training folds only. The orange rings are the test fold, months this model never saw. The orange lines are its misses, and RMSPE boils them down to one number.
Step 3
Every fold takes a turn, then average
After k rounds, every month has been tested exactly once. The fold errors bounce around because some folds are harder to predict than others. Their average is the cross-validation estimate, and it is steadier than any single split.
| Round | Train on | Test on | Test months | RMSPE |
|---|---|---|---|---|
| Average |
Step 4
Use it to pick a model
Run the same folds for each model and compare their averages. Each dot is one fold's RMSPE; the black bar is the average.
| Model | RSE (all data) | CV RMSPE | Best and worst fold |
|---|
Step 5
How many folds?
Switch Folds (k) in the bar at the top and watch the table above.
- More folds: each model trains on more of the data, and you average more rounds. The cost is more models to fit.
- Fewer folds: less work, which is why we do k = 3 by hand in class.
Five or 10 folds are common in practice. Whatever k you choose, the quadratic model comes out on top here, and the 9th-degree model never beats it.
In R
3-fold cross-validation by hand
Data: 117 monthly bills from the mosaicData Utilities set; dailyspend = gas bill ÷ billing days, temperature in °F. The shuffle is the one R makes with set.seed(9529) and slice_sample(), so the 3-fold numbers match lecture exactly. For k = 5 and 10, the same shuffled order is cut into k nearly equal pieces. Companion lab: Train, Test, Repeat.