Building models for explanation · sleep and GPA

What does Z do?

We want the effect of sleep (X) on term GPA (Y). Someone hands you a third variable, Z. Should it go in the model?

The answer never comes from the regression output alone. It depends on where Z sits in the causal story: what causes Z, and what Z causes. Here we invent a world where we know the truth, run the same study 300 times, and see what adding Z does to the slope on sleep.

In every world below, one more hour of sleep raises GPA by 0.10 points. That is the number a good model should recover.

Z is a
strong
300

Step 1

Draw the arrows first

An arrow means "causes." Thicker arrows are stronger effects. The arrow from sleep to GPA is the one we want to measure; everything else is a possible way to get it wrong.

Step 2

Run the study 300 times

Each dot will be one simulated study: draw a new sample of students, fit the regression, record the slope on sleep. The orange row leaves Z out. The blue row puts Z in. The green line is the true effect. Before each click, predict where the dots will land.

lm(gpa ~ sleep)lm(gpa ~ sleep + z)True effect, 0.10Bar = middle 95% of estimates
Two things can go wrong: the cloud can be in the wrong place (bias), or it can be wide (imprecise).

Leave Z out

Put Z in

Step 3

Mystery Z

Now go the other way. A role is picked at random and hidden. Look only at the two clouds and the numbers above, then guess what kind of variable Z is. This is the reasoning you'll do on real data, except there you never get to check.

Hint: a mediator and a confounder can produce clouds that look alike. The data can't tell them apart; only the time order and the story can. That is why we decide on controls by reasoning, not by p-values.

Summary

The rule, in one table

Z is a…ArrowsLecture exampleInclude?Why

The two "never include" rows look harmless in the output. The coefficient on Z is significant, R² goes up, and the residual error goes down. The model predicts better and explains worse. That is the main difference between building a model for prediction (last week) and building one for explanation (this week).

In R

One simulated study, by hand

The simulation draws sleep and Z with standard deviation 1 and adds noise with SD 0.35 to GPA, so the numbers are in "GPA points per hour of sleep." For the mediator, the effects are split so the total effect of sleep stays at 0.10: part goes directly to GPA and part flows through Z. Companion lab: Choose Your Controls, the same ideas on the sleep study and a sales training program.