Generalization, Leakage, and the Limits of a Good Fit

The purpose of supervised learning is not to memorize a table. It is to learn a relationship that remains useful on new cases from the problem we care about.

That ability is called generalization.

A model can fail to generalize for several reasons. Two of the most common are overfitting and leakage.

Overfitting learns the training set too specifically

Imagine a classifier that creates a special rule for every training row. It could reach 100% training accuracy while having no useful rule for a new distance value.

That is overfitting: the model captures details of the training examples that do not hold up outside them.

A gap between training performance and held-out performance is one warning sign. It does not tell us exactly why the model failed, but it tells us the training score alone was insufficient evidence.

Leakage lets the answer sneak into the experiment

Leakage is different. It occurs when information that should not be available to the model influences training or evaluation.

Examples include:

• including the target label as an input feature;
• normalizing the entire dataset before creating the train/test split when that calculation reveals information from the test set;
• tuning repeatedly against the final test set;
• using a timestamp or identifier that indirectly reveals the answer because of how the data was collected.

Suppose all stop examples were collected on Monday and all continue examples on Tuesday. A model using day_of_week might appear excellent. If deployment happens on Wednesday, the shortcut collapses.

The model did learn a real pattern in the table. The problem is that the pattern did not represent the mechanism we intended to test.

A good result still needs a bounded claim

Even a clean held-out result does not prove that the model is safe for every environment. It tells us how the fixed model behaved on a defined set of examples.

A stronger evaluation report therefore records:

• how the data was collected;
• which features were allowed;
• how the split was created;
• what metric was calculated;
• which errors occurred;
• what conditions were not tested.

This is the beginning of an evidence discipline you will use for the rest of the course.

Before the lab

Write one sentence that distinguishes these two failures:

1. a model memorizes details of the training rows;
2. the experiment accidentally exposes information from the test answers.

Then open the lab and see how quickly a clean-looking classifier changes when labels are damaged.