Topics

• Calculating predicted probabilities from the logistic regression model

• Using the predicted probabilities to make a "yes/no" decision for a given observation

• Assessing model performance using

  • Confusion matrix
  • ROC curve

Risk of coronary heart disease

This dataset is from an ongoing cardiovascular study on residents of the town of Framingham, Massachusetts. We want to examine the relationship between various health characteristics and the risk of having heart disease in the next 10 years.

high_risk: 1 = High risk, 0 = Not high risk

age: Age at exam time (in years)

totChol: Total cholesterol (in mg/dL)

currentSmoker: 0 = nonsmoker; 1 = smoker

Modeling risk of coronary heart disease

Using the model for prediction

We are often interested in predicting whether a given observation will have a "yes" response

To do so

• Use the logistic regression model to calculate the predicted log-odds that an observation has a "yes" response.

• Then, use the log-odds to calculate the predicted probability of a "yes" response.

• Then, use the predicted probabilities to classify the observation as having a "yes" or "no" response.

Calculating the predicted probability

log(ˆπi1−ˆπi)=ˆβ0+ˆβ1xi

⇒exp{log(ˆπi1−ˆπi)}=exp{ˆβ0+ˆβ1xi}

ˆπ vs. ^log-odds

ˆπi=exp(ˆβ0+ˆβ1xi)1+exp(ˆβ0+ˆβ1xi)=exp(ˆlog-odds)1+exp(^log-odds)

Predicted response for a patient

Suppose a patient comes in who is 60 years old, does not currently smoke, and has a total cholesterol of 263 mg/dL.

Predicted log-odds that this person is high risk for coronary heart disease in the next 10 years:

^log-odds=−6.638+0.082×60+0.002×263+0.457×0=−1.192

^probability=exp{−1.192}1+exp{−1.192}=0.233

Predictions in R

x0 <- data_frame(age = 60, totChol = 263,
                 currentSmoker = factor(0))

predict(risk_m, x0)

##         1 
## -1.214193

predict(risk_m, x0, 
      type = "response")

##       1 
## 0.22896

Is this patient high risk?

The probability the patient is at risk for coronary heart disease is 0.229.

Confusion Matrix

• We can use the predicted probability to predict the outcome for a given observation

  • In other words, we can classify the observations into two groups: "yes" and "no"

• How: Establish a threshold such that y=1 if predicted probability is greater than the threshold (y=0 otherwise)

• To assess the accuracy of our predictions, we can make a table of the observed (actual) response versus the predicted response.

  • This table is the confusion matrix

Confusion Matrix

Suppose we use 0.3 as the threshold to classify observations.

If ˆπi>0.3, then risk_predict = "Yes". Otherwise, risk_predict = "No".

Confusion matrix

Confusion matrix: 2 X 2 table

In practice, you often see the confusion matrix presented as a 2 × 2 table as shown below:

Receiver Operating Characteristic (ROC) curve

Sensitivity & Specificity

• Sensitivity: Proportion of observations with y=1 that have predicted probability above a specified threshold

  • Called true positive rate (y-axis)

• Specificity: Proportion of observations with y=0 that have predicted probability below a specified threshold

  • (1 - specificity) called false positive rate (x-axis)

• What we want:

⬆️ High sensitivity

⬇️ Low values of 1-specificity

ROC curve in R

library(yardstick)
# Need to put 1 as the first level
risk_m_aug <- risk_m_aug %>%
  mutate(high_risk = fct_relevel(high_risk, c("1", "0")))
# calculate sensitivity and specificity at each threshold
roc_curve_data <- risk_m_aug %>%
  roc_curve(high_risk, .fitted) 
# plot roc curve
autoplot(roc_curve_data)

ROC curve

Area under curve (AUC)

We can use the area under the curve (AUC) as one way to assess how well the logistic model fits the data

• AUC=0.5 very bad fit (no better than a coin flip)
• AUC close to 1: good fit

risk_m_aug %>%
  roc_auc(high_risk, .fitted) %>%
  pull(.estimate)

## [1] 0.6955

Which threshold would you choose?

Recap

• Calculating predicted probabilities from the logistic regression model

• Using the predicted probabilities to make a "yes/no" decision for a given observation

• Assessing model performance using

  • Confusion matrix
  • ROC curve