Model building for Explanation

STA 235 Lecture 9

Emre Yucel

University of Texas at Austin

2026-11-04

Building models for understanding and explanation

Understanding vs predicting

  • In a purely predictive model, we should include terms that reduce out-of-sample prediction error (i.e., predictions errors of the model on new data).
  • With lots of variables, many models will have essentially identical out-of-sample predictive error, and we can use any of them.
  • If we are trying to understand the relationships between a subset of the X variables and Y, the model form is much more important.

The data

Model 1: Predicting term GPA from total sleep time

model1 <- lm(term_gpa~ total_sleep_time, data=sleep)
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        2.7913     0.1745    16.0  < 2e-16 ***
total_sleep_time   0.0989     0.0260     3.8  0.00016 ***
confint(model1)
                   2.5 % 97.5 %
(Intercept)      2.44860  3.134
total_sleep_time 0.04774  0.150

Each additional hour of sleep increases predicted GPA by about 0.10 points with a 95% CI of (0.05, 0.15).

Are we done?

Variables in the sleep study

  • total_sleep_time (TST): Average daily sleep time, recorded early in the term.
  • cumulative_gpa: GPA at the beginning of the term.
  • midterm: Midterm grade, recorded midway through the term.
  • term_gpa: GPA at the end of the term.
  • study: Four studies using different cohorts at different universities:
    • Study 2, 3: “large public university”
    • Study 4: “private Catholic university”
    • Study 5: “private, STEM-focused university”
  • race: 1 if either parent from an under-represented group (non-White/Asian).
  • first_gen: 1 if neither parent attended any college.
  • gender: 1 if reported male, 0 otherwise.
  • height: height in inches.

Building a model for understanding

  • We would like to be able to conclude that more sleep causes higher GPAs, but this model will not let us do that.
  • We can’t conduct a randomized controlled trial, because that would be unethical.
  • We can rule out potential confounders by adding them to the model (along with other predictive variables), and avoid causal language.
  • How do we decide which variables to include?

How do we decide which variables to include?

  • Theory-driven, often “pre-registered” to avoid p-hacking (https://osf.io/5xngv for this study).
  • Data-driven, but be careful—we don’t want to overfit!

Variables to always include

  • Confounders are variables that causally influence both total sleep time and term GPA
    • Because of their collinearity with sleep time, the sleep time coefficient will be different if omitted from the model
    • To ensure we are measuring the effect of sleep and not the confounder, they must be in the model!
  • Variables that are strongly predictive of term GPA and are (nearly) uncorrelated with total sleep time.
    • By reducing the residual standard error, these variables reduce the size of confidence intervals for the regression coefficient of interest.

Variables you may want to include

  • Variables that are weakly predictive of term GPA and (nearly) uncorrelated with total sleep time.
    • Is the more complex model worth the extra predictive power?
  • Variables that someone might think are confounders, even if you don’t agree (or they don’t appear to be after seeing the data).
    • This lends your analysis some credibility.

Variables you should NEVER include

  • Variables that are causally influenced by the outcome Y (term GPA).
    • These variables can introduce bias, especially if they’re also causally influenced by total sleep time (a collider).
    • This can happen by accident when constructing the final dataset (e.g., if we excluded students who dropped out at the end of the term).
  • Variables that are causally influenced by the X variable of interest (total sleep time).
    • Adjusting for these variables removes a part of the effect we’re interested in!

Why these rules? Simulate it

  • Invent a world where we know the effect of sleep on GPA (0.10 points per hour).
  • Run the same study 300 times, with and without a third variable Z.
  • Before each click: where will the estimates land?

Try it: what does Z do?

Cumulative GPA after fall term

Should we include or exclude this predictor?

Cumulative GPA after fall term

Cumulative GPA

  • Should we include this predictor? Yes!
  • Almost certainly a strong predictor, potentially a confounder, can’t be causally influenced by winter TST

Study number/Cohort

  • Recall: Four separate studies in different cohorts (large public uni, small private Catholic uni, private STEM-focused uni)
  • Would each cohort have different average GPA and/or sleep patterns on average?

Study number/cohort

Study number/cohort

  • Should we include this predictor? Yes!
  • Almost certainly a strong predictor, potentially a confounder, can’t be causally influenced by winter TST

Demographics

  • Demographic predictors include gender, race, and whether the student is a first-generation college student
  • Should we include these predictors?

Demographics: Gender

Demographics: Race

Demographics: First-generation

Demographics

  • Should we include these predictors?
    • Race, first-generation seem to predict term GPA, at least weakly, but maybe not sleep time
    • Not much going on with gender
  • Previous literature indicates all three may be associated with sleep patterns
    • This is an argument for including them all in the model, at least as a robustness check

Recall: Variables you may want to include

  • Variables that are weakly predictive of total sleep time and (nearly) uncorrelated with total sleep time
  • Variables that someone might think are confounders, even if you don’t agree (or they don’t appear to be after seeing the data)

Another hypothetical variable: Midterm grades

  • Sleep patterns were all measured before administration of midterm exams
  • Should our model include midterm exams? Why or why not?

Another hypothetical variable: Midterm grades

  • Should our model include midterm exams? No!
  • If sleep causally influences term GPA, presumably it also influences midterm scores
  • Part of the effect of early-term sleep on term GPA “flows through” midterm exams; if we include midterm grades, we remove that portion of the effect
  • Just like course evaluation participation rate in the pre-class reading

Try it: add the midterm and watch the sleep estimate

Overall relationship

fit0 <- lm(term_gpa~ total_sleep_time, data=sleep)
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        2.7913     0.1745    16.0  < 2e-16 ***
total_sleep_time   0.0989     0.0260     3.8  0.00016 ***
confint(fit0)
                   2.5 % 97.5 %
(Intercept)      2.44860  3.134
total_sleep_time 0.04774  0.150

Each additional hour of sleep increases predicted GPA by about 0.10 points, with a 95% CI of (0.05, 0.15)

Adjusting for cumulative GPA

fit0_gpa <- lm(term_gpa~ total_sleep_time + cumulative_gpa, data=sleep)
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        0.5638     0.1754    3.21   0.0014 ** 
total_sleep_time   0.0554     0.0200    2.76   0.0059 ** 
cumulative_gpa     0.7275     0.0371   19.60   <2e-16 ***
confint(fit0_gpa)
                   2.5 %  97.5 %
(Intercept)      0.21926 0.90841
total_sleep_time 0.01602 0.09479
cumulative_gpa   0.65458 0.80036

Comparing two students with identical cumulative GPA at the start of the term, if one has an additional hour of sleep time their predicted end of term GPA will be 0.05 points higher (95% CI [0.016, 0.095])

Adjusting for cumulative GPA and study/cohort

fit1 <- lm(term_gpa~ total_sleep_time + cumulative_gpa +
            factor(study), data=sleep)
Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        0.6559     0.1815    3.61  0.00033 ***
total_sleep_time   0.0690     0.0205    3.37  0.00081 ***
cumulative_gpa     0.6768     0.0383   17.68  < 2e-16 ***
factor(study)3    -0.0651     0.0447   -1.46  0.14607    
factor(study)4     0.1050     0.0453    2.32  0.02072 *  
factor(study)5    -0.0791     0.0459   -1.72  0.08566 .  

Comparing two students from the same cohort with identical cumulative GPAs, if one has an additional hour of sleep time their predicted end of term GPA will be 0.07 points higher (95% CI [0.029, 0.011] – see R script)

Including the kitchen sink

fit2 <- lm(term_gpa~ total_sleep_time + cumulative_gpa +
            factor(study) + gender + race + firstgen, data=sleep)
Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        0.6722     0.1868    3.60  0.00035 ***
total_sleep_time   0.0678     0.0205    3.31  0.00099 ***
cumulative_gpa     0.6624     0.0392   16.90  < 2e-16 ***
factor(study)3    -0.0669     0.0448   -1.49  0.13580    
factor(study)4     0.0991     0.0467    2.12  0.03410 *  
factor(study)5    -0.0890     0.0470   -1.90  0.05861 .  
gender            -0.0179     0.0322   -0.56  0.57910    
race               0.0743     0.0402    1.85  0.06534 .  
firstgen          -0.0207     0.0439   -0.47  0.63671    

Comparing two students from the same cohort with identical cumulative GPAs and the same demographics, if one has an additional hour of sleep time their predicted end of term GPA will be 0.068 points higher (95% CI [0.028, 0.011] – see R script)

Can we simplify the model?

confint(fit2)
                     2.5 %   97.5 %
(Intercept)       0.305178 1.039228
total_sleep_time  0.027587 0.108011
cumulative_gpa    0.585449 0.739448
factor(study)3   -0.154829 0.021065
factor(study)4    0.007466 0.190820
factor(study)5   -0.181315 0.003254
gender           -0.081227 0.045438
race             -0.004735 0.153357
firstgen         -0.106904 0.065442

What if we omit gender and firstgen?

Simplifying…

fit3 <- lm(term_gpa~ total_sleep_time + cumulative_gpa +
            factor(study)  + race, data=sleep)
Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)        0.6435     0.1812    3.55  0.00042 ***
total_sleep_time   0.0680     0.0204    3.33  0.00094 ***
cumulative_gpa     0.6651     0.0387   17.18  < 2e-16 ***
factor(study)3    -0.0647     0.0446   -1.45  0.14747    
factor(study)4     0.1071     0.0452    2.37  0.01815 *  
factor(study)5    -0.0834     0.0459   -1.82  0.06969 .  
race               0.0749     0.0398    1.88  0.06050 .  

Nearly identical results (CIs too) – and very, very close to the model that omits demographics

Reporting results

Report findings from a range of models:

  • Models with few or no other variables.
  • Models that adjust for theory-driven confounders and predictors.
  • Models that add data-driven confounders and predictors – specifically note these!
  • Models that add or remove subsets of potential confounders, to assess how robust the findings are.

Try it: every possible model, at once

So, what did we find?

  • After adjusting for cohort, cumulative GPA, and demographic variables, each additional hour of sleep is associated with an increase in predicted GPA of about 0.05 points, with a 95% CI of about [0.03, 0.11].
  • The effect estimate is robust to the specific adjustment for demographic variables hypothesized to correlate with sleep patterns and academic performance—this strengthens our argument!
  • In each case our findings are statistically significant, and the estimated effects (and their plausible values based on the CIs) are all positive.

In other words, you should probably get more sleep!

Appendix: A case where we know the answer

Does sales training work?

  • A software company sent some of its sales reps through a training program. Managers chose who went, so this is not a randomized experiment.
  • The data are simulated, so we know the truth: training raises quarterly sales by $12K. About $8K of that comes from reps booking more client meetings; $4K is direct.
  • sales_k: Quarterly sales, in thousands of dollars.
  • trained: 1 if the rep completed the training program at the start of the quarter.
  • experience_yrs: Years of sales experience.
  • territory_potential: Market size of the rep’s territory (assigned by geography).
  • client_meetings: Client meetings per week during the quarter.
  • review_score: Performance review written after the quarter closed.
  • office_floor: Which floor of headquarters the rep sits on.

Who got trained?

Trained reps are more experienced, but their territories look the same.

Five models, five answers

Model Estimate 95% CI Residual SE
No controls $23.2K [20.5, 25.9] 19.2
+ experience $12.3K [9.7, 14.9] 16.3
+ experience + territory $11.9K [10.0, 13.8] 11.7
+ experience + territory + meetings $4.1K [2.3, 6.0] 10.1
+ experience + territory + review -$7.2K [-9.4, -5.0] 9.1

The true effect is $12K. Which models get it right?

What happened in each model?

  • No controls ($23K): experience_yrs is a confounder. Managers send veterans to training, and veterans sell more anyway.
  • + experience ($12K): Correct!
  • + territory ($12K): territory_potential is a strong predictor of sales that’s unrelated to training. Same estimate, narrower CI.
  • + meetings ($4K): client_meetings is a mediator. Training causes more meetings, so holding meetings fixed removes $8K of the effect.
  • + review (−$7K): review_score is a collider. It is caused by both training and sales, so comparing reps with the same review makes training look harmful.

The lesson

  • The two models with bad controls have the lowest residual standard error. They predict sales better and answer our question worse.
  • When building a model for explanation, choose controls by asking what causes what and when it was measured, not by which model fits best.

Try it: what does Z do?

Try it: choose your controls