Detections Need Tests and Maintenance

A detection rule is code or logic operating on changing data.

That means it can regress.

Fields can be renamed. Log sources can disappear. Time formats can change. A service can move. A new legitimate workflow can create noise.

A detector that worked last month can silently stop working today.

Give every detection a test set

For a failed-login rule, include:

Positive test

Known synthetic event set that should alert.

Negative test

Known benign event set that should not alert.

Boundary test

Exactly one below and exactly at threshold.

Regression test

A previously fixed false positive remains fixed.

Pipeline-health test

Required telemetry is still arriving with expected fields.

Treat detection content like a governed system

A detection record should include:

rule ID
owner
purpose
version
required data sources
logic
severity/priority
expected response
known benign cases
blind spots
test cases
last review date
change history

This prevents mystery alerts that nobody understands.

Supplied regression failure

Original event field:

result=failed

New application version emits:

outcome=failure

Detection still queries:

result == "failed"

Result:

zero alerts

Does that prove failed logins stopped?

No.

Pipeline/schema evidence shows the rule no longer matches the new event format.

This is a detection regression.

A canary event can test the full path

A controlled synthetic test event can verify:

source emits
   ↓
forwarder receives
   ↓
collector parses
   ↓
rule matches
   ↓
alert appears

The event must be clearly labeled as synthetic so it is not confused with a real incident.

Example:

test_event=true scenario=DETECTION-CANARY-14

Do not generate harmful behavior merely to make an alert. Generate safe synthetic telemetry that exercises the rule.

Response automation raises the evidence bar

If an alert only creates a low-priority review task, some false positives may be tolerable.

If an alert automatically disables an account or isolates a device, mistakes can cause larger harm.

The stronger the automated response, the stronger the requirements for:

• signal quality;
• multiple evidence sources;
• exception handling;
• rollback/recovery;
• human authority;
• audit logs; and
• safe testing.

This course does not assume “automate response” is always better.

Lab: repair the broken detector

Fictional baseline:

rule expects result=failed
positive test alerts
negative test quiet

Changed source schema:

outcome=failure

Observed:

source events present
collector receives events
rule alert count = 0

Correction:

Update the normalized field mapping so both source versions produce canonical:

auth_result=failed

Retest:

• old-format positive case;
• new-format positive case;
• negative case;
• threshold boundary;
• known false positive.

Now the detection survives the represented schema change.

Blind spots should be written down

Possible blind spots:

• source does not log event;
• identity split across multiple account names;
• distributed activity below per-account threshold;
• clock/data delay;
• log collector outage;
• encrypted application behavior not represented in logs;
• rule only covers one environment.

A detector with documented blind spots is stronger than a detector marketed as “complete.”

Finish the Detection Engineering Record

Submit:

1. rule purpose/owner;
2. required telemetry;
3. deterministic logic;
4. labeled dataset;
5. threshold comparison;
6. chosen tradeoff;
7. positive/negative/boundary/regression tests;
8. source-schema regression and repair;
9. canary/pipeline-health plan;
10. blind spots; and
11. limitation.

A strong final claim:

The Week 14 rule detects the tested synthetic conditions under the documented normalized schema and chosen threshold. The regression test demonstrates recovery from one represented field change; the rule remains blind to events or identities not present in its required telemetry.

That is detection engineering, not alert worship.