RobotnixAcademy

Week 13 · lesson

Missing Logs Are Also a System Condition

A monitoring system can fail while the application keeps working.

That creates a dangerous illusion: no alerts may mean “nothing happened,” or it may mean “we stopped seeing.”

Observability has to be monitored too.

Model the telemetry pipeline

APPLICATION
   ↓
LOCAL LOG OUTPUT
   ↓
FORWARDER / AGENT
   ↓
NETWORK / QUEUE
   ↓
COLLECTOR
   ↓
PARSER / NORMALIZER
   ↓
STORAGE
   ↓
SEARCH / ALERT

A failure at any stage can create a gap.

Absence of evidence needs context

Suppose the query returns zero failed logins.

Can you claim there were no failed logins?

Only if you also have reasonable evidence that:

  • the source was logging failures;
  • forwarding was healthy;
  • collection covered the time window;
  • parsing did not discard them;
  • storage retained them;
  • your query/filter was correct.

Otherwise the safest statement may be:

No failed-login records were returned by this query from the available dataset.

That sentence is less dramatic and more defensible.

Supplied forwarding failure

Application events:

15:00:00 event=service_health status=ok
15:01:00 event=service_health status=ok
15:02:00 event=service_health status=ok
15:03:00 event=service_health status=ok

Collector receives:

15:00:00 received
15:01:00 received
[gap]
[gap]

Forwarder health:

15:01:20 state=retrying destination=collector
15:01:50 state=queue_full dropped=12

Now the gap has supporting pipeline evidence.

You can say the collector dataset is incomplete for the affected period in the fictional model.

Heartbeats make silence observable

A periodic heartbeat can say:

source=log-forwarder event=heartbeat status=healthy queue_depth=0

If heartbeats stop, the monitoring system can alert on the absence of expected telemetry.

This is different from alerting on a bad application event.

Retention is part of evidence quality

Suppose policy keeps:

  • application operational logs: 14 days;
  • privileged-role change logs: 180 days;
  • backup job records: 90 days.

Why different periods?

Because evidence value, privacy, storage, regulation/policy, and operational requirements differ.

“Keep everything forever” creates cost and privacy risk.

“Delete everything tomorrow” may destroy needed evidence.

Retention should be intentional.

Integrity and access controls matter

If anyone can rewrite logs without evidence, accountability weakens.

Useful design questions:

  • who can read logs?
  • who can delete them?
  • who can change logging configuration?
  • are privileged changes themselves logged elsewhere?
  • can a local service erase centralized records?
  • is time synchronized?
  • how is retention enforced?

You do not need an immutable blockchain to ask good governance questions.

Lab: diagnose an evidence gap

Fictional incident:

application healthy
collector missing 12 minutes of events
forwarder queue reported full
heartbeat absent for same interval
network later recovered
forwarder resumed but reported dropped_count=48

Build the strongest bounded claim.

A good answer:

The supplied forwarder and collector evidence indicates the centralized dataset is incomplete for the 12-minute interval and that the forwarder reported 48 dropped events after its queue filled. The evidence does not reveal the contents of the dropped events.

Exactly.

Recovery test

After the simulated pipeline recovery:

  • heartbeat returns;
  • new application events reach collector;
  • queue depth returns to zero;
  • alert clears only after several healthy intervals;
  • gap remains documented rather than silently backfilled with invented data.

Do not pretend missing evidence can be reconstructed when it cannot.

Finish the Telemetry and Timeline Record

Submit:

  1. telemetry pipeline diagram;
  2. event dictionary;
  3. eight-event multi-source timeline;
  4. clock-normalization example;
  5. missing-event/gap analysis;
  6. heartbeat/forwarder health evidence;
  7. retention requirement for three event classes;
  8. recovery verification; and
  9. confidence + limitation statement.

A strong final claim:

The Week 13 timeline correlates the supplied identity, application, and database records while preserving source timestamps and known clock adjustments. A separate forwarding failure makes one interval explicitly incomplete; the artifact does not infer the content of records that were never collected.

That is how logs become evidence instead of decoration.