Troubleshooting Is Hypothesis Testing

A service fails at 11:20.

A configuration file's hash changed at 11:18.

Did the configuration change cause the failure?

Maybe.

That is not the same as “yes.”

Troubleshooting becomes engineering when you turn observations into hypotheses and then try to disprove them with evidence.

Continue the fictional service

We return to RNX-LAB-SVC-01 from Week 2.

Its status page should return a normal response using this intended configuration:

listen: 127.0.0.1:8080
template: /opt/rnx/templates/status.html
status_file: /var/lib/rnx/status.json

At 09:00, the instructor records a SHA-256 baseline for the configuration:

artifact: /etc/rnx/status-service.yaml
sha256: 6d31b8e7...baseline-example...f09a
source: approved lab image after provisioning

The digest is illustrative course evidence, not a real package checksum.

Failure evidence

At 11:20 the supplied application log shows:

11:20:04Z level=ERROR event=startup reason=status-file-not-found path=/var/lib/rnx/stats.json
11:20:04Z level=ERROR event=service-ready result=false

The service-manager snapshot shows:

rnx-status.service
state: failed
exit_reason: configuration-error

The current configuration is:

listen: 127.0.0.1:8080
template: /opt/rnx/templates/status.html
status_file: /var/lib/rnx/stats.json

And its current digest is:

sha256: e9507c11...current-example...3b27

The hashes differ.

Observation first

List only what the supplied evidence directly supports.

1. The current configuration digest differs from the 09:00 baseline digest.
2. The current configuration names /var/lib/rnx/stats.json.
3. The intended baseline content names /var/lib/rnx/status.json.
4. The application log reports status-file-not-found for /var/lib/rnx/stats.json.
5. The service manager reports a configuration-related failure.

Those observations align around one mechanism.

Build the hypothesis

A useful hypothesis is specific and testable:

The service fails because the current configuration points to /var/lib/rnx/stats.json instead of the intended /var/lib/rnx/status.json.

This hypothesis explains:

• the changed file digest;
• the exact content difference;
• the path named by the application error; and
• the service-manager failure classification.

That is stronger than “the hash changed, therefore the service broke.”

The content difference is the proposed mechanism.

What evidence would weaken the hypothesis?

Good troubleshooters ask how they could be wrong.

The hypothesis would weaken if:

• /var/lib/rnx/stats.json actually existed and was the approved new path;
• the service continued to fail with the baseline path restored;
• logs showed a different failure before the configuration was read;
• the baseline was stale and the new configuration was expected; or
• another required resource was independently broken.

A hypothesis that cannot be challenged is not useful.

The evidence stack

Notice how several sources contribute different pieces.

None of those artifacts needs to be dramatic.

Together, they form a coherent explanation worth testing.

File metadata can help—but it has limits

Suppose the supplied metadata says:

path: /etc/rnx/status-service.yaml
modified: 11:18:51Z
owner: root
size: 116 bytes

This gives us time and filesystem metadata.

It does not prove that “root changed the file.” File ownership is not an audit log of who performed the last edit.

Again: distinguish system state from event history.

If attribution mattered, you would need a trustworthy source that records the change event or deployment action.

Troubleshooting without random resets

A weak response to a failed service is:

restart
reinstall
change permissions
reboot
change another setting
restart again

Even if the service eventually works, you may not know which change mattered.

A stronger response is:

1. Define expected function.
2. Identify the observed failure.
3. Compare current state with baseline.
4. Form one hypothesis.
5. Change one relevant variable in the isolated lab.
6. Retest the expected function.
7. Retest the preserved boundaries.
8. Record the result.

This is why troubleshooting and security engineering share a mindset.

Both depend on controlled evidence.

Optional local exercise

On an approved local file, you can observe metadata and digest with commands such as:

ls -l approved-lab-file.txt
sha256sum approved-lab-file.txt

If your instructor has not provisioned a local environment, use the supplied synthetic evidence. Do not substitute a random system or downloaded executable.

Activity: evidence or explanation?

Classify each statement.

1. The two SHA-256 values differ.
2. The configuration text contains stats.json instead of status.json.
3. A malicious user edited the configuration.
4. The application log reports status-file-not-found for stats.json.
5. Restoring the intended path should be tested as a possible correction.
6. The configuration change definitely caused every service problem.

Statements 1, 2, and 4 are observations from supplied evidence.
Statement 3 invents attribution.
Statement 5 is a defensible hypothesis-driven next action.
Statement 6 is stronger than the evidence supports.

Add to your Integrity and Troubleshooting Record

Record:

Observed mismatch:
Relevant content difference:
Failure evidence:
Hypothesis:
Evidence supporting it:
Evidence that would weaken it:
Controlled change to test:
Preserved boundary to retest:

Do not write the conclusion yet.

Lesson 3 provides the controlled retest.