Model building for prediction

STA 235 Week 10

Emre Yucel

University of Texas at Austin

2026-10-28

Two reasons for building regression models

  • Prediction: Predict the value of Y based on the values of the X variables (today)
  • Understanding: Understand the relationships between Y and the X variables (later in the semester)

How we evaluate a model depends on what our use for it is.

Remember overfitting?

In week 6 we kept adding polynomial terms, and R^2 kept going up.

But a very wiggly curve overfits: it follows the random noise in this data set, so it does badly on new data.

It’s like memorizing the answers to a practice exam:

  • You’d ace the practice exam (the data the model already saw)
  • You’d struggle on the real exam (new data)

Today: how to catch overfitting before it costs us.

Evaluating forecasts

How should we evaluate a forecast?

A model for Apple’s revenue

Like Moore’s law in week 6, Apple’s revenue grows by a roughly steady percentage each quarter. So we model log revenue with a straight-line trend:

lm(log(Revenue) ~ Period, data=apple)

(Period counts quarters: 1, 2, 3, …)

Its R^2 is 0.75. Does that tell us how well it will forecast next year?

R^2 and the residual standard error (RSE) measure how well the model fits past data — the same data we used to build it.

That’s the practice exam: the model has already seen the answers.

If you were an analyst, how impressed would your boss be if you claimed to be able to predict the past? (Not very — they want you to be able to predict the future!)

How should we evaluate a forecast?

  • R^2 and the residual standard error are optimistic measures of predictive (future) performance.
  • They measure the quality of predictions using the same data used to fit the model.
  • But we’re interested in how the model predicts into the future, not how well it fits the past!

We can mimic predicting on future data by holding back the most recent observations.

Fit the model to the “training” data only (blue) and test the model by forecasting the most recent year of data (red).

Try it: how much to hold out?

Fitting the model to the training set

We want to split our data into a test set (last 4 quarters) and a training set (everything else).

training <- apple[1:54,]
test <- apple[55:58,]

Then we fit the model to the training data only. (This is the same model as before.)

apple_model <- lm(log(Revenue) ~ Period, data=training)

Testing the model on the test set

Now we can see how well the model predicts the test set.

predict(apple_model, test)
   55    56    57    58 
4.710 4.730 4.751 4.771 

These are predictions for log revenue. exp() undoes log(), so we use it to get back to revenue in $B:

exp(predict(apple_model, test))
   55    56    57    58 
111.0 113.3 115.6 118.0 

How do we compare this to the actual revenues of the last 4 quarters?

test$Revenue
[1]  95.4  94.0 102.5 143.8

Measuring prediction error

We want one number that says how far off our 4 predictions were.

Root Mean Squared Prediction Error (RMSPE) — read the name backwards:

  1. Prediction Error: for each quarter, find the miss: actual - predicted
  2. Squared: square each miss (so negative misses don’t cancel positive ones)
  3. Mean: average the squared misses
  4. Root: take the square root (to get back to the units of Y, here $B)

Interpret RMSPE just like the residual standard error: it’s the size of a typical miss. Smaller is better!

The same recipe as a formula

For reference, here are the four steps written in math:

\text{RMSPE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (Y_i - \hat Y_i)^2}

  • Y_i - \hat Y_i: the miss for observation i (actual - predicted)
  • (\ldots)^2: square it
  • \frac{1}{n}\sum: add them up and divide by n (the size of the test set; n=4 in this case)
  • \sqrt{\ldots}: take the square root

Calculating RMSPE

Quarter Actual Y Predicted \hat Y Miss Y - \hat Y Miss squared
2025 Q1 95.40 111.05 -15.65 244.86
2025 Q2 94.00 113.32 -19.32 373.35
2025 Q3 102.50 115.64 -13.14 172.74
2025 Q4 143.80 118.01 25.79 665.04
Total 1456.00

Mean: 1456.00 \div 4 = 364.00

Root: \sqrt{364.00} = 19.08

What does 19.08 mean?

When forecasting the future, a typical miss for this model is about $19.08 billion.

Is that big? Apple’s revenue in these quarters was about $94–144 billion, so a typical miss is roughly 18% of revenue.

Notice the pattern: the model guesses too high in Q1–Q3 and too low in Q4, the holiday quarter. A straight-line trend can’t see seasons.

A shortcut for calculating RMSPE

# exp here only because we're predicting log(Revenue) instead of Revenue
predictions <- exp(predict(apple_model, test))
actuals <- test$Revenue
sqrt(mean((actuals - predictions)^2))
[1] 19.08

Read it from the inside out: miss \to square \to mean \to square root.

RMSPE vs residual standard error

  • Residual standard error: typical miss on the data the model was fit to (in-sample; the practice exam)
  • RMSPE: typical miss on data the model has never seen (out-of-sample; the real exam)

When we care about prediction, RMSPE is the one that matters.

Training/test splits in general

Train/test splits to measure performance

  • Shuffle the rows, then split them, so both sets look like the full data set
  • 80% training / 20% testing is common
  • The tradeoff:
    • Bigger test set \to more trustworthy estimate of prediction error
    • Bigger training set \to a better model

Which rows go in the test set?

  • Data in time order (like Apple’s revenue): hold out the most recent rows. We want to predict the future, so test on the “future.”
  • Everything else (like today’s next example): shuffle first, then hold out a random 20%.

Why shuffle? If the rows were sorted (say, by month), the test set might be all summer days. A model trained on fall–spring would be tested on weather it never saw.

Predicting utility expenditures

Creating a train/test split

Let’s randomly split the utilities data into a training and test set.

library(tidyverse)
set.seed(9529)

# Ask R to shuffle all 117 observations
randomized <- slice_sample(utilities, n = 117)

# 24 is about 20% of 117, so we'll use the first 24 rows as our test set
utilities_test <- randomized[1:24,]

# The rest are our training set
utilities_train <- randomized[25:117,]

set.seed() makes the “random” shuffle the same every time you run the code. Any number works.

Creating a train/test split

Black = training data, Red = testing data

The red points are scattered across all temperatures — that’s what shuffling buys us.

Using training/test splits to pick between competing models

When comparing models:

  1. Fit each model using the training set
  2. Test each model on the test set (calculate RMSPE)
  3. Pick the model with the smallest RMSPE

If we fit the model and evaluate it using the same data, we might overfit and get an overly optimistic estimate of how well we’ll predict new data.

Three candidate models

Linear (fit to all the data)

Quadratic

9th degree

Linear Quadratic 9th degree
R^2 0.841 0.880 0.889
RSE 0.866 0.755 0.750

Judged by R^2 and RSE, the 9th-degree model “wins.” Do you trust it?

Building the models

Now let’s do it properly. Fit the models using the training set:

model1 <- lm(dailyspend ~ temp, data=utilities_train)
model2 <- lm(dailyspend ~ temp + I(temp^2), data=utilities_train)
model9 <- lm(dailyspend ~ temp + I(temp^2) + I(temp^3) + I(temp^4)
              + I(temp^5) + I(temp^6) + I(temp^7) + I(temp^8) + I(temp^9),
              data=utilities_train)

Get their predictions on the test set:

actuals <- utilities_test$dailyspend
predictions1 <- predict(model1, utilities_test)
predictions2 <- predict(model2, utilities_test)
predictions9 <- predict(model9, utilities_test)

Measuring performance

Now let’s calculate RMSPE on the testing data:

sqrt(mean((actuals - predictions1)^2))
[1] 1.014
sqrt(mean((actuals - predictions2)^2))
[1] 0.8535
sqrt(mean((actuals - predictions9)^2))
[1] 0.9483
Model RSE (training data) RMSPE (test data)
Linear 0.83 1.01
Quadratic 0.73 0.85
9th Degree 0.71 0.95
  • Every model looks worse on new data than on the data it was fit to.
  • The 9th-degree model has the smallest RSE, but the quadratic has the smallest RMSPE. The quadratic is the best of the three for predicting new data.

Training error falls, test error doesn’t

  • Every extra term lets the curve chase the training points, so training error keeps going down.
  • Test error drops at first (real curvature), then climbs once the extra terms start fitting noise. That climb is overfitting.
  • Never pick a model by training error: it always prefers the most complicated one.

The problem with this approach

Our result depends on which rows landed in the test set. Here is the same comparison with different random splits:

Split Linear Quadratic 9th degree Winner
Ours (seed 9529) 1.01 0.85 0.95 Quadratic
Seed 1 0.73 0.54 0.64 Quadratic
Seed 2 1.07 0.96 0.91 9th degree

Out of 200 different random splits, the quadratic won 132 times, the 9th degree won 63 times, and the linear model won 5 times.

One split can be “unlucky.” Fix: give several different test sets a turn, and average.

Try it: train, test, repeat

Cross-validation

k-fold Cross-Validation

  • Split the data into k groups (“folds”) of about the same size
  • Each fold takes one turn as the test set; the model is trained on the other folds
  • Every row gets tested exactly once
  • Average the k RMSPEs to get the final estimate

Let’s work through an example with k=3

First we need to split the data into 3 random groups of roughly the same size (39\times 3=117):

fold1 <- randomized[1:39,]
fold2 <- randomized[40:78,]
fold3 <- randomized[79:117,]

Now we need to build the model 3 times:

Train on Test on
Model A fold2 and fold3 fold1
Model B fold1 and fold3 fold2
Model C fold1 and fold2 fold3

Building the models

modelA <- lm(dailyspend ~ temp, data=rbind(fold2, fold3))
modelB <- lm(dailyspend ~ temp, data=rbind(fold1, fold3))
modelC <- lm(dailyspend ~ temp, data=rbind(fold1, fold2))

rbind() stacks two data sets on top of each other.

Getting the predictions

Each model predicts the fold it was not trained on:

predictionsA <- predict(modelA, fold1)
predictionsB <- predict(modelB, fold2)
predictionsC <- predict(modelC, fold3)

Calculating RMSPE

We now get 3 different RMSPE estimates:

actualsA <- fold1$dailyspend
sqrt(mean((actualsA - predictionsA)^2))
[1] 0.9898
actualsB <- fold2$dailyspend
sqrt(mean((actualsB - predictionsB)^2))
[1] 0.745
actualsC <- fold3$dailyspend
sqrt(mean((actualsC - predictionsC)^2))
[1] 0.8463

Our final estimate of RMSPE is the average of these 3 estimates:

\text{RMSPE} = \frac{ 0.99 + 0.75 + 0.85 }{3} = 0.86

Re-doing our model comparisons

Model RSE (all data) RMSPE (3-fold CV) RMSPE (single train/test split)
Linear 0.866 0.860 1.014
Quadratic 0.755 0.765 0.853
9th Degree 0.750 0.907 0.948
  • By RSE, the 9th-degree model looks best (0.750).
  • By cross-validation, it’s the worst (0.907). It memorized noise — that’s overfitting, caught.
  • The quadratic has the smallest RMSPE in both test-based columns: it’s the best model for prediction.
  • Our single split was unlucky: it made every model look worse than cross-validation does.

Big ideas from today

  1. R^2 and RSE grade a model on data it has already seen (the practice exam). They are too optimistic about the future.
  2. To judge predictions, hold out data the model never sees: the most recent rows for time data, a random 20% otherwise.
  3. RMSPE = the typical miss on that held-out data. Miss \to square \to mean \to root. Smaller is better.
  4. More complicated isn’t better: the 9th-degree model looked best on old data and did worst on new data.
  5. Cross-validation gives every row a turn in the test set and averages the results, so one unlucky split can’t fool us.