Thresholds Trade Sensitivity for Noise

Many detections use thresholds.

A lower threshold may catch more true cases but create more false positives. A higher threshold may reduce noise but miss meaningful cases.

There is no universal “best” threshold without context.

Use a labeled synthetic dataset

For this lesson, the fictional dataset already tells us whether each 10-minute account window should be considered review-worthy according to the teacher-provided scenario.

That label lets us evaluate a rule.

Rule candidate:

alert if failures >= threshold

Threshold = 3

Predictions:

• A no alert → true negative
• B no alert → true negative
• C alert → true positive
• D alert → true positive
• E no alert → false negative
• F alert → true positive
• G alert → false positive
• H no alert → true negative

Counts:

TP=3
TN=3
FP=1
FN=1

Accuracy:

(TP + TN) / total = 6 / 8 = 75%

Precision:

TP / (TP + FP) = 3 / 4 = 75%

Recall:

TP / (TP + FN) = 3 / 4 = 75%

The symmetry is accidental to this small dataset.

Threshold = 2

Now more windows alert.

You should calculate the new confusion counts.

Expect higher recall but more false positives in this particular dataset.

Threshold = 4

Now fewer windows alert.

You should expect fewer false positives but more false negatives.

Again: calculate it. Do not rely on intuition.

Which error matters more?

That depends on the detection purpose.

For a low-cost review queue, you may tolerate some false positives.

For an automated control that disables accounts, false positives may cause significant harm — which is one reason high-impact actions often need additional evidence or human approval.

Detection policy should match response consequence.

Accuracy can hide weak behavior

Imagine 1,000 windows where only 5 are truly review-worthy.

A detector that never alerts gets:

995 / 1000 = 99.5% accuracy

and catches zero meaningful cases.

So evaluation must consider class balance and the metric connected to the operational goal.

Activity: compare three thresholds

For thresholds 2, 3, and 4, calculate:

• TP;
• TN;
• FP;
• FN;
• accuracy;
• precision;
• recall.

Then choose a threshold for one stated response policy.

Your justification must mention:

• cost of false positive;
• cost of false negative;
• volume/noise;
• whether an alert triggers human review or automatic action; and
• dataset limitations.

Tune with evidence, not to make the dashboard green

A dangerous tuning process is:

Too many alerts. Raise threshold until alerts disappear.

A stronger process:

label representative cases
   ↓
measure rule behavior
   ↓
inspect FP/FN mechanisms
   ↓
adjust logic/threshold
   ↓
retest old + new cases
   ↓
document tradeoff

Sometimes the right fix is not threshold adjustment. It may be better context or a different feature.

Example:

failed-login burst + privileged account

may be more useful for a particular response than lowering the threshold for everyone.

Extend your Detection Engineering Record

Add:

• labeled synthetic dataset;
• three threshold evaluations;
• confusion counts;
• chosen threshold;
• operational reason;
• false-positive example;
• false-negative example;
• one additional context field that could improve the rule.

Lesson 3 turns the detection into a maintained testable component.