Review & Self Study — What Did the Model Actually Learn?

Before leaving supervised learning, separate the mechanics from the claims.

You should now be able to trace this chain:

examples → features + labels → training → fitted parameters → held-out test → bounded claim

If any link is missing, the final score becomes harder to interpret.

Review questions

1. Why is a label different from a feature?
2. What does training change in the threshold classifier?
3. Why should a final test set remain untouched while the model is being tuned?
4. How can a model reach excellent training performance and poor held-out performance?
5. What is one example of leakage?
6. Why is “the model is 92% accurate” weaker than a sentence that names the test set and conditions?

Compare these claims:

The classifier is reliable.

On 40 held-out examples collected under the stated test conditions, the fixed classifier produced 37 correct labels and 3 errors.

The second claim may sound less impressive, but it is more useful because another person can inspect its boundary.

Self study

Try one or more of these without changing the course requirements:

• Draw a train/test split for a dataset of 20 examples and explain which rows may influence fitting.
• Find a public explanation of overfitting and compare its definition with the evidence pattern used here.
• Invent a feature that looks predictive but would probably fail after deployment because it captures a shortcut.
• Return to the Classification Playground and find two thresholds with similar accuracy but different precision or recall.

One sentence to keep

A model learns from the evidence we give it, and our claims should stay inside the evidence used to test it.