Logistic regression 1

STA 235 Week 9

Emre Yucel

University of Texas at Austin

2026-10-21

The basic logistic regression model

The LendingClub data set

  • The LendingClub data set contains information about 96736 funded loans for home improvement, debt consolidation, or small business purposes.
  • We have data on each borrower’s credit sub-grade, loan amount, interest rate, income, and more. We also know whether each loan was fully paid or charged off (defaulted).
  • Let’s see if we can predict whether a loan will be fully paid based on the borrower’s credit score (a 0–100 scale derived from LendingClub’s sub-grade system).

What’s wrong with this regression?

\begin{align*} \text{fully.paid} = \hat{\beta_0} + \hat\beta_1\cdot\text{credit\_score} \end{align*}

The Y variable here is categorical (two levels—a loan is either fully paid or it isn’t), so regular linear regression won’t work here.

The idea behind logistic regression

  • Instead of predicting whether a loan is fully paid, let’s predict the probability that it is fully paid.
  • In logistic regression, one level of Y is always called “success” and the other “failure.” Since Y=1 for fully paid loans, we designate fully paid as “success.”
  • Logistic regression predicts the probability of success p as a function of X indirectly, by modeling the log-odds of success as a linear function of X.

Odds

  • When something has “even (1/1) odds,” the probability of success is 1/2
  • When something has “2/1 odds,” the probability of success is 2/3
  • When something has “3/2 odds,” the probability of success is 3/5
  • In general, the odds of something happening are p/(1-p)
  • …and so the probability of something happening is \text{odds}/(1+\text{odds})

Odds

Before Texas played Gonzaga in the 2026 NCAA Tournament Second Round, DraftKings listed Texas at +200—meaning a $100 bet on Texas winning would pay out $200.

That is, they were giving Texas odds to win of \$100/\$200 = 1/2 (ignoring house edge).

What’s their implied probability that Texas would win?

\frac{p}{1-p} = \frac{1}{2} \Rightarrow p = \frac{1/2}{1+1/2} \approx 0.333

Odds

Before Texas played Purdue in the 2026 NCAA Tournament Sweet 16, DraftKings listed Texas at +290—meaning a $100 bet on Texas winning would pay out $290.

That is, they were giving Texas odds to win of \$100/\$290 \approx 0.345. What’s their implied probability that Texas would win?

\frac{p}{1-p} = \frac{1}{2} \Rightarrow p = \frac{0.345}{1+0.345} \approx 0.256

Probability vs odds vs log odds

Odds and log odds are just different ways of expressing probabilities:

Probability Odds Log odds
p p/(1-p) \log(p/(1-p))
Between 0 and 1 Between 0 and \infty Between -\infty and \infty
0 0 -\infty
0.25 0.33 -1.10
0.5 1 0
0.75 3 1.10
0.8 4 1.39
0.9 9 2.20
0.95 19 2.94
1 \infty \infty

The logistic regression model

Logistic regression predicts the log odds of success p as a linear function of X: \log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 X This fits an S-shaped curve to the data (we’ll see what it looks like shortly).

Let’s try it

# Fit logistic regression: predict P(fully paid) from credit score
model <- glm(fully.paid ~ credit_score, data = loans,
             family = binomial)
summary(model)

Call:
glm(formula = fully.paid ~ credit_score, family = binomial, data = loans)

Coefficients:
             Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -5.86756    0.08163   -71.9   <2e-16 ***
credit_score  0.10464    0.00123    85.2   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 107169  on 96735  degrees of freedom
Residual deviance:  99212  on 96734  degrees of freedom
AIC: 99216

Number of Fisher Scoring iterations: 4

How to interpret the curve?

The regression output tells us that our prediction is \begin{aligned} \log(\text{odds}) &= \log\left(\frac{P(\text{fully paid})}{1-P(\text{fully paid})}\right) \\ &= -5.868 + 0.105 \cdot\text{credit\_score}. \end{aligned}

Solving for P(\text{fully paid}):

\widehat{P(\text{fully paid})} = \frac{e^{-5.868 + 0.105 \cdot\text{credit\_score}}}{1 + e^{-5.868 + 0.105 \cdot\text{credit\_score}}}

Making predictions

We can use predict to automate plugging into the equation:

# Predict probability of full repayment for a borrower with credit_score = 65
predict(model, list(credit_score = 65), type = "response")
    1 
0.718 

\frac{e^{-5.868 + 0.105 \cdot 65}}{1 + e^{-5.868 + 0.105 \cdot 65}} = 0.72

We predict that a borrower with credit score 65 has about 72% chance of fully repaying their loan.

Interpreting the model results

Visualizing the model

How to interpret the curve?

\widehat{P(\text{fully paid})} = \frac{e^{-5.868 + 0.105 \cdot\text{credit\_score}}}{1 + e^{-5.868 + 0.105 \cdot\text{credit\_score}}}

Interpreting the coefficients

Our prediction equation is: \log\left(\frac{P(\text{fully paid})}{1-P(\text{fully paid})}\right) = -5.868 + 0.105 \cdot\text{credit\_score}.

  • We give our interepretations in odds, since log-odds are less intuitive.
  • This works just like a linear regression for \log(Y)!

Interpreting the coefficients

Rewriting the prediction equation as: \text{Predicted odds of full repayment} = e^{-5.868 + 0.105 \cdot\text{credit\_score}}

  • Predicted odds when credit score = 0: e^{-5.868} = 0.003.
  • Minimum credit score in the data is 45, so not meaningful on its own here.

Interpreting the coefficients

Rewriting the prediction equation as: \text{Predicted odds of full repayment} = e^{-5.868 + 0.105 \cdot\text{credit\_score}}

Increasing credit score by 1 point multiplies the odds by e^{0.105}=1.11; i.e., increases the odds by 100(e^{0.105} - 1) = 11%.

Increasing credit score by 2 points multiplies the odds by e^{2 \cdot 0.105}=1.23; i.e., increases the odds by 100(e^{2\cdot 0.105} - 1) = 23%.

Logistic regression with 2+ predictors

Adding another predictor: can we do better?

  • Just like with a linear regression model, we can add additional predictors to the model.
  • Our interpretation of the coefficients in multiple logistic regression is similar to multiple linear regression: each coefficient represents the predicted effect of one X on Y, holding the other X variables constant.

Does loan purpose predict repayment?

# Proportion fully paid within each loan purpose
xtabs(~ purpose + fully.paid, data = loans)  %>% 
  prop.table(margin = 1)  %>% 
  round(2)
                    fully.paid
purpose                 0    1
  debt_consolidation 0.25 0.75
  home_improvement   0.20 0.80
  small_business     0.32 0.68

Does loan purpose predict repayment?

Adding another predictor

Let’s add loan purpose as a second predictor of repayment, in addition to credit score:

# Add loan purpose (reference level: debt_consolidation)
model2 <- glm(fully.paid ~ credit_score + purpose,
              data = loans, family = binomial)

summary(model2)

Call:
glm(formula = fully.paid ~ credit_score + purpose, family = binomial, 
    data = loans)

Coefficients:
                        Estimate Std. Error z value Pr(>|z|)    
(Intercept)             -5.85019    0.08188  -71.45  < 2e-16 ***
credit_score             0.10419    0.00123   84.54  < 2e-16 ***
purposehome_improvement  0.13828    0.02679    5.16  2.5e-07 ***
purposesmall_business   -0.06753    0.05737   -1.18     0.24    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 107169  on 96735  degrees of freedom
Residual deviance:  99183  on 96732  degrees of freedom
AIC: 99191

Number of Fisher Scoring iterations: 4

Interpreting coefficients

Our prediction equation is: \begin{aligned} \log\!\left(\frac{p}{1-p}\right) &= -5.850 + 0.104 \cdot\text{credit\_score} \\ &\quad + 0.138 \cdot\text{home\_impr.} \\ &\quad - 0.068 \cdot\text{small\_bus.} \end{aligned}

  • For a debt consolidation loan with credit score 0, the predicted log odds are -5.850 (the intercept).
  • Among loans with the same purpose, each additional credit score point corresponds to a 11% increase in predicted odds of full repayment (multiplied by e^{0.104} = 1.11).

Interpreting coefficients

\begin{aligned} \log\!\left(\frac{p}{1-p}\right) &= -5.850 + 0.104 \cdot\text{credit\_score} \\ &\quad + 0.138 \cdot\text{home\_impr.} \\ &\quad - 0.068 \cdot\text{small\_bus.} \end{aligned}

  • Among loans with the same credit score, home improvement loans have 15% higher predicted odds of full repayment vs. debt consolidation (multiplied by e^{0.138} = 1.15).
  • Among loans with the same credit score, small business loans have 7% lower predicted odds of full repayment vs. debt consolidation (multiplied by e^{-0.068} = 0.93).

Visualizing the model

Which curve is which?

Visualizing the model

Which curve is which?

Visualizing the model

Evaluating the model

How good is our model?

  • R^2 and residual standard error metrics aren’t available for logistic regression.
  • There are many “pseudo-R^2” metrics that indicate model fit.
  • But most of these are difficult to interpret, so we’ll focus on other metrics

How many cases did we accurately predict?

We could classify each loan using a 0.5 probability threshold:

\text{Prediction} = \begin{cases} \text{fully paid}, & \text{if } \widehat{P(\text{fully paid})} \geq 0.5, \\ \text{charged off}, & \text{if } \widehat{P(\text{fully paid})} < 0.5. \end{cases}

predicted.paid <- ifelse(
  predict(model2, type = "response") >= 0.5, 1, 0)
round(mean(predicted.paid == loans$fully.paid), 4)
[1] 0.76

We correctly classify 76% of the loans. Is that good?

What should we compare that against?

What accuracy would we get if we just predicted “fully paid” for every loan, without using any information?

# Fraction of loans that are fully paid (the "predict-everyone-pays" accuracy)
mean(loans$fully.paid)
[1] 0.758

A model that predicts “fully paid” for every single loan would be right 75.75% of the time!

Our model’s accuracy: 76.02% — a lift of 0.27 percentage points over naive. Is that good?

Defaults are expensive!

  • Overall model accuracy is similar to the baseline, but where do they make their mistakes?
    • Baseline predicts no defaults, so it gets all the fully paid loans right but all the defaults wrong.
    • Our model predicts 5549 defaults (out of 96736 total loans), of which 2904 are actually defaults
  • If we had avoided writing any loans predicted to default we might have saved a lot of money!

Reviewing a confusion matrix

Actual negative Actual positive
Predict negative True negative False negative
Predict positive False positive True positive

For example, there were 70,634 true positives: loans we predicted would be fully paid that actually repaid in full.

         Actual
Predicted     0     1
        0  2904  2645
        1 20553 70634

Here, positive means fully paid (Y=1), and negative means charged off (Y=0).

False positives and false negatives

In this model, positive means fully paid (Y=1), and negative means charged off (Y=0).

Actual charged off (Y=0) Actual fully paid (Y=1)
Predict charged off True negative False negative
Predict fully paid False positive True positive
  • False positive: predict full repayment, but the borrower defaults.
  • False negative: predict default, but the borrower repays in full.

The labels depend on which outcome we call positive.

The confusion matrix

The confusion matrix compares predictions with actual outcomes.

# Rows: predictions. Columns: actual outcomes.
# 0 = charged off, 1 = fully paid.
addmargins(confusion)
         Actual
Predicted     0     1   Sum
      0    2904  2645  5549
      1   20553 70634 91187
      Sum 23457 73279 96736

Accuracy counts the two correct cells:

\text{Accuracy} = \frac{\text{TN} + \text{TP}}{n} = \frac{2904 + 70634}{96736} = 76.02\%.

False positive rate

Among loans that actually defaulted, what fraction did we predict would be fully paid?

\text{False positive rate} = \frac{\text{FP}}{\text{TN} + \text{FP}} = \frac{20553}{23457} = 87.62\%.

  • These are defaults the model missed by predicting full repayment.
  • The denominator includes all actual defaults, regardless of the prediction.

False negative rate

Among loans that actually repaid in full, what fraction did we predict would default?

\text{False negative rate} = \frac{\text{FN}}{\text{FN} + \text{TP}} = \frac{2645}{73279} = 3.61\%.

  • If we declined these loans, we would miss borrowers who repay in full.
  • The denominator includes all actual fully paid loans, regardless of the prediction.

Changing the probability cutoff

Suppose we predict full repayment only when \widehat{P(\text{fully paid})} \geq 0.7, instead of 0.5.

predicted.paid.70 <- ifelse(
  predict(model2, type = "response") >= 0.7, 1, 0)
table(Predicted = predicted.paid.70,
      Actual = loans$fully.paid) %>% addmargins()
         Actual
Predicted     0     1   Sum
      0   10011 14215 24226
      1   13446 59064 72510
      Sum 23457 73279 96736

We now require a higher predicted probability before classifying a loan as fully paid. What happens to each kind of error?

Comparing the two cutoffs

Cutoff 0.5 Cutoff 0.7
False positives 20553 13446
False positive rate 87.62% 57.32%
False negatives 2645 14215
False negative rate 3.61% 19.40%
  • A higher cutoff lowers the false positive rate but raises the false negative rate.
  • A lower cutoff lowers the false negative rate but raises the false positive rate.

The cutoff tradeoff

Changing the cutoff changes which mistakes the model makes.

Defaults are expensive!

  • A false positive can mean funding a loan that defaults.
  • A false negative can mean declining a loan that would have repaid in full.
  • A lender can choose a cutoff based on the relative costs of these errors, instead of choosing 0.5 automatically.
  • Overall accuracy alone does not tell us which cutoff makes the best lending decision.