Week 14 · lesson
Four Outcomes Behind a Model Score
For a binary classifier, every evaluated prediction lands in one of four places.
| Actual | Predicted | Name |
|---|---|---|
| positive | positive | true positive |
| negative | negative | true negative |
| negative | positive | false positive |
| positive | negative | false negative |
Those four counts form a confusion matrix. The word “confusion” does not mean the model is thinking. It means the table records where predictions matched or disagreed with known outcomes.
Suppose an inspection model flags defective parts. On 100 test parts it produces:
- 18 true positives;
- 70 true negatives;
- 7 false positives;
- 5 false negatives.
The total is 100, so we can already see that 88 predictions were correct and 12 were errors. But that single correct/incorrect split hides something important: the two error types may have very different consequences.
A false positive sends a good part for extra inspection. A false negative allows a defective part to pass. If the consequences differ, a metric that merges both errors into one percentage can hide the decision we actually care about.
Read the matrix before calculating metrics
Before reaching for formulas, ask what each cell means in the real problem.
For an obstacle detector:
- false positive: the system warns when no obstacle is present;
- false negative: the system misses an obstacle.
For spam filtering, the meanings change. For medical screening, they change again.
The same mathematical matrix can describe very different operational risks.
Before moving on
Choose a binary decision you understand. Describe its false positive and false negative in plain language. Which one would you investigate first, and why?
Experiment pipeline
From scored cases to a metric decision
Follow one fixed result through expected and failure traces to a bounded system decision.
Input preparation
Scored Evaluation Set
Hold twenty supplied labels and scores fixed: five inspection-required cases and fifteen no-inspection cases.
Model configuration
Threshold Configuration
Choose the score boundary that converts each numeric score into inspect or no-alert.
Evaluation
Confusion Counts
Count true positives, false positives, true negatives, and false negatives for one threshold.
Metric Calculations
Calculate accuracy, precision, and recall from the same four outcome counts.
Threshold Tradeoff
Compare all three settings and identify which error decreases while the other increases.
Decision
Metric Decision
Retain a threshold only after naming the fictional consequence priority and evidence limit.
Read this concept flow as plain text
- Scored Evaluation Set. Hold twenty supplied labels and scores fixed: five inspection-required cases and fifteen no-inspection cases. CLASS BALANCE: 5 POSITIVE / 15 NEGATIVE.
- Threshold Configuration. Choose the score boundary that converts each numeric score into inspect or no-alert. THRESHOLDS: 0.35 · 0.55 · 0.75.
- Confusion Counts. Count true positives, false positives, true negatives, and false negatives for one threshold. AT 0.55: TP 3 · FP 3 · TN 12 · FN 2.
- Metric Calculations. Calculate accuracy, precision, and recall from the same four outcome counts. AT 0.55: 75% · 50% · 60%.
- Threshold Tradeoff. Compare all three settings and identify which error decreases while the other increases. ACCURACY VS RECALL: 80% / 40% AT 0.75.
- Metric Decision. Retain a threshold only after naming the fictional consequence priority and evidence limit. DECISION RULE: CONSEQUENCE BEFORE SCORE.