ROC Curves

Receiver Operating Characteristic - Evaluating Binary Classifiers

What is a ROC Curve?

A ROC curve plots the performance of a binary classification model at all classification thresholds. It shows the trade-off between sensitivity and specificity.

X-axis: False Positive Rate (FPR)

Y-axis: True Positive Rate (TPR)

TPR = TP / (TP + FN) = Sensitivity

FPR = FP / (FP + TN) = 1 - Specificity

AUC (Area Under Curve) summarizes the entire ROC curve into a single number between 0 and 1.

One model, many thresholds

Move the threshold to see which examples are called positive. The model scores stay fixed; only the decision rule changes.

Model (AUC 0.84)Random ranking (0.5)

Decision threshold: 0.50

Call an example positive when its score ≥ threshold. Use arrow keys for steps of 0.05; Home / End select the extremes.

True positives (TP): 4 / 5
False positives (FP): 2 / 5
False negatives (FN): 1 / 5
True negatives (TN): 3 / 5

Try 1.00, then lower the threshold: more positives are found, but more negatives are flagged too. The dot moves along the curve; AUC stays fixed.

Inspect the synthetic examples

Five actual positives: 0.95, 0.85, 0.75, 0.55, 0.35.
Five actual negatives: 0.80, 0.50, 0.30, 0.20, 0.10.

Scores are illustrative, not calibrated probabilities. Each vertical step adds one true positive; each horizontal step adds one false positive. A positive outranks a negative in 21 of 25 pairs, so AUC = 0.84.

AUC Interpretation Guide

AUC represents the probability that the model ranks a random positive example higher than a random negative example.

1.0

Perfect
Classifier

0.8-0.9

Excellent
Performance

0.7-0.8

Good
Performance

0.5

Random
Guess

Advantages of ROC/AUC

Scale-invariant: measures quality of predictions
Classification-threshold-invariant: measures across all thresholds
Easy to compare different models
Single number summary of performance
Intuitive visual representation

Limitations to Consider

Overly optimistic for imbalanced datasets
May not reflect real-world performance costs
Different curves can have same AUC
Doesn't show threshold-specific performance
Consider Precision-Recall for skewed data

Practical Tips for Students

When to Use ROC

Best for balanced datasets or when false positives and false negatives have similar costs. Use for screening tests, spam detection, or general classification tasks.

When to Avoid ROC

Avoid for highly imbalanced datasets (e.g., 1% positive class). Use Precision-Recall curves instead for rare disease detection or fraud detection.

Reading the Curve

Look at the curve shape, not just AUC. A curve closer to the top-left corner is better. The optimal threshold depends on your specific use case.

Model Selection

Compare multiple models by plotting their ROC curves together. The model with the highest AUC isn't always the best for your specific problem.