Recall from last week

We need four things to be true for regression to work properly:

• Linearity: \(Y\) is a linear function of the \(X\)s
• Independence: The prediction errors (residuals) are independent
• Normality: The prediction errors (residuals) are normally distributed
• Equal variance: The variance of \(Y\) is the same for any value of the \(X\)s (also called homoskedasticity)

Visualizing the linear model

What if this were the true relationship?

Linearity assumption

A linear function should represent the overall pattern of the data (otherwise, it wouldn’t make sense to fit a linear function to predict \(Y\)!)

Linearity Example 1

Linearity is clearly satisfied

Linearity Example 2

Linearity is not satisfied—a line is not a good fit for the data:

Linearity Example 3

This one is a lot harder to tell!

Regression diagnostic plots

We’ll use regression diagnostic plots to help us evaluate the L, N, and E assumptions.

The residuals vs fitted graph plots:

• Residuals on the \(Y\)-axis
• Fitted values (predicted \(Y\) values) on the \(X\)-axis

This graph effectively subtracts out the linear trend between \(Y\) and the \(X\)’s, so we want to see no trend left in this graph.

Linearity Example 1

Linearity is clearly satisfied:

Linearity Example 2

Linearity is not satisfied—a line is not a good fit for the data:

Linearity Example 3

This one is a lot harder to tell!

Now it’s clear that linearity is not satisfied:

Normality assumption

To build our confidence intervals for slopes and predictions, we’re assuming that the residuals are normally distributed.

Normality example 1

Hard to tell what the distribution of the residuals is from the scatterplot:

Normality example 1

We could make a histogram of the residuals, but it’s hard to tell if the distribution is really Normal or not:

Normality example 1

The key is to look at the Normal Q-Q plot which compares the distribution of our residuals to a perfect Normal distribution.

If the dots line up along an (approximately) straight line then the Normality assumption is satisfied.

Normality example 2

Hard to tell what the distribution of the residuals is from the scatterplot:

Now it’s clear that normality is not satisfied:

Equal variance (“homoscedasticity”)

The variance of \(Y\) should be about the same at any \(X\) value (or combination of values for the \(X\)’s).

In other words, the vertical spread of the points should be the same anywhere along the \(X\)-axis.

Equal variance example 1

Equal variance looks good:

Equal variance example 2

Equal variance is clearly violated—the vertical spread of the points is larger along the right side of the graph:

Equal variance example 3

This one is a lot harder to tell!

Regression diagnostic plots

Check the equal variance assumption by looking at the vertical spread of the points in the scale-location plot or the residuals vs fitted plot.

There should be no trend or pattern: if the vertical spread is about the same all the way across the graph, equal variance is satisfied.

Equal variance example 1

Equal variance looks good:

Equal variance example 2

Equal variance is clearly violated—the vertical spread of the points is larger along the right side of the graph:

Equal variance example 3

This one is a lot harder to tell!

Now it’s clear that equal variance is not satisfied:

Use the regression diagnostic plots!

• Sometimes, violations of assumptions are hard to see unless you look at the regression diagnostic plots, but then they are clear in the regression diagnostic plots.
• Look only for clear violations in the regression diagnostic plots, as real data is never perfect: if it’s not an obvious violation then it’s OK to take the assumption as satisfied.
• Don’t focus on a small number of weird observations when the data set is large.

Independence

Independence means that knowing the residual for one observation doesn’t tell you anything about the residual for another observation.

• Independence is tricky to identify, because we don’t have a convenient plot like we do for the other 3 assumptions.
• If the data were collected via a simple random sample then there’s a good chance that we can assume independence.
• But if there is a “hidden” structure, such as geography, grouping, or missing variables, we could still have a violation of independence.

Residuals vs Last Sold Date

We tend to under-predict houses sold recently, while over-predicting earlier sales.

Residuals vs Last Sold Date

If we add latest_saledate to our model, then this dependence disappears.

Residuals vs Zipcode

Using our simple model of price regressed on area, we seem to underpredict in 78731 and overpredict in 78721 and 78723.

# A tibble: 3 × 2
  zipcode mean_residual
    <dbl>         <dbl>
1   78731        61539.
2   78721       -33435.
3   78723       -37908.

Adding zipcode to our model

If we add zipcode as a predictor in our model this correlation goes away!

# A tibble: 3 × 2
  zipcode mean_residual
    <dbl>         <dbl>
1   78723      5.66e-12
2   78731      4.46e-12
3   78721     -2.38e-11

Independence example 1

• We want to know how high school SAT scores predict college GPAs.
• A data set consists of the SAT scores (\(X\)) and college GPAs (\(Y\)) of a simple random sample of UT students.
• Because this is a SRS, then knowing that Alice’s GPA is higher than we would expect based on her SAT score, doesn’t tell me anything about whether Betsy’s GPA is higher or lower than we would expect based on her SAT score.
• Independence is satisfied.

Independence example 2: Time series data

• We want to predict how Facebook’s stock price performs over time.
• A data set consists of the day of the year (\(X\)) and Meta’s closing daily stock price (\(Y\)) for ~100 days.
• If I know that Meta’s stock was higher than expected on June 1, I have pretty good guess that it was higher than expected on June 2 too.
• If this were true, then independence would not be satisfied.

Independence example 2: Time series data

We can see that independence is not satisfied because the residuals are correlated with the \(X\) variable (residuals shouldn’t correlate with any \(X\) or potential \(X\) variable!):

Independence example 3: Spatial correlation

• We want to predict house prices in a city.
• Houses close to each other often share neighborhood characteristics that aren’t fully captured by the model (crime, schools, noise).
• Nearby residuals tend to look alike, creating spatial correlation.
• Independence is not satisfied.

Independence example 4: Repeated measurements

• We want to predict weight loss of the same individuals across weeks.
• Multiple measurements from the same person are not independent.
• The residuals from week to week for each person will be related.
• Independence is not satisfied.

Potential independence violations

• Time series data: residuals are often correlated over time.
• Spatial correlation: residuals are often correlated in space.
• Repeated measurements: residuals are often correlated within subjects (i.e., repeated measurements on the same individual).
• If you see a correlation between residuals and any \(X\) variable, then you know that independence is not satisfied, but usually you just have to think conceptually about the situation to make a judgement about whether independence is satisfied.

What do I do if assumptions are violated?

Violations of linearity are the most serious: Everything is wrong!

• If linearity holds, we can make good predictions…
• …but not prediction intervals, coefficient estimates, or confidence intervals
• Can try transformations of variables (later in the course)

What do I do if assumptions are violated?

For the others:

• Violations of normality are the least serious: Prediction intervals are wrong, but everything else is OK if sample size is large enough
• Violations of normality/equal variance can be addressed with bootstrap techniques (beyond the scope of this class)
• Violations of independence are best addressed by modeling the dependence
  • If we don’t, our intervals will tend to be too small
• Often fixing a linearity problem will fix other issues as well, so we usually check this first.

What a single observation can do

Let’s take some hypothetical sample data:

What a single observation can do

Adding a new observation with \(X\) near the mean of \(X\) doesn’t matter much even if it’s out of line with the rest of the data:

What a single observation can do

What a single observation can do

Adding a new observation with \(X=5\) doesn’t matter much when it’s in line with the rest of the data.

What a single observation can do

Adding a new observation with \(X=5\) starts making the line a bad fit for the rest of the data when it isn’t in line with the rest of the data.

What a single observation can do

It gets worse when the new observation is far off the overall trend (i.e., large residual):

What a single observation can do

It gets worse when the new observation is far off the overall trend (i.e., very large residual):

Before trusting a model, we want to determine if any influential observations are making the model not represent the bulk of the data.

We need both a large residual and high leverage for an observation to be influential:

\[\text{large residual} + \text{high leverage} = \text{high influence}\]

How can I find influential observations?

Look at the residuals vs leverage diagnostic plot, for points with high leverage and a large (positive or negative) residual (upper right or lower right):