Lesson 3: Diagnose the Seeded Faults

Harborview is configured.

Now it starts behaving like a real environment.

Something changes. A user notices a symptom. One part of the system stops meeting the requirement while other parts keep working.

That is the point of the seeded faults.

You are not being tested on whether you recognize the title of the lesson where the answer first appeared.

You are being tested on whether you can use surviving function to locate the failure.

Use the capstone incident model

For every incident:

user report
   ↓
scope
   ↓
last known change
   ↓
known-good boundaries
   ↓
first failed / uncertain boundary
   ↓
question-driven evidence
   ↓
strongest theory + competing theory
   ↓
controlled action
   ↓
rollback / recovery
   ↓
verification

Do not start with the fix.

Start with the system.

Incident 1: workstation stops completing POST after a memory change

User report:

One workstation turns on, but there is no normal startup screen.

Supplied evidence:

before change: workstation booted normally
recent change: additional memory module installed
power LED: on
fans: spinning
monitor/cable: verified elsewhere
POST completion: not observed
motherboard diagnostic indicator: DRAM
new module: correct DDR generation, full compatibility not yet proven

What the evidence gives you

The system receives some power.

The display equipment is known-good elsewhere.

The machine has not completed POST.

A memory change happened immediately before the failure.

The board is reporting a memory-related initialization problem.

Windows has not earned a place in the theory yet.

Strong first path

• power off safely;
• verify module and board compatibility;
• inspect recommended DIMM population;
• verify seating;
• restore the last known-good memory configuration when appropriate;
• retest POST.

Competing theory

A motherboard or memory-slot fault remains possible.

What would make that theory stronger?

If the known-good original memory configuration still fails in a previously working slot, the changed-module-only theory becomes weaker.

Do not call the new DIMM defective merely because the failure appeared after installation. It could be incompatible, mis-seated, or placed incorrectly.

Robotnix service-port simulation
This is deterministic lesson state, not a real host or service.

$ ping -c 2 192.0.2.20
64 bytes from 192.0.2.20: icmp_seq=1 ttl=64 time=0.9 ms
64 bytes from 192.0.2.20: icmp_seq=2 ttl=64 time=0.8 ms

$ ss -lnt
State  Recv-Q Send-Q Local Address:Port
LISTEN 0      128    127.0.0.1:22

$ curl -I http://192.0.2.20:8080
curl: (7) Failed to connect to 192.0.2.20 port 8080: Connection refused

$ systemctl --user start helpdesk-demo
Started helpdesk-demo.service in simulated state.

Incident 2: valid network path, failed DNS

User report:

The class portal is down.

Evidence:

wired link: up
IP configuration: valid
gateway: reachable
remote documentation IP: reachable
portal IP: reachable
portal hostname lookup: fails

The phrase portal is down is already too broad.

The endpoint can reach the remote IP.

The first failed service is name resolution.

Useful evidence

• configured DNS server;
• supplied query result;
• expected portal record;
• local static name override where applicable.

Irrelevant first move

Replacing the Ethernet cable.

The same cable is already carrying successful routed IP traffic.

Verification

• hostname resolves to the expected address;
• portal opens by name;
• required login/application function works.

Incident 3: printer is healthy, one workstation cannot print correctly

User report:

The front-office printer is broken from this computer.

Evidence:

printer internal test page: correct
other workstations: print normally
printer network: reachable
problem workstation: jobs go to old/stale queue
new approved shared queue: available

The printer is not the strongest failure boundary.

It creates correct pages and other clients use it normally.

The affected workstation's queue/configuration is the narrower boundary.

Controlled correction

Remove or replace the stale mapping according to the approved support procedure and connect the workstation to the intended shared queue.

Verification

Print the same supplied test document from the affected workstation.

Do not stop because the queue icon appears.

The user needs paper output.

Incident 4: display/application regression after an approved update

User report:

The graphics application stopped working after yesterday's update.

Evidence:

Windows boot: normal
other apps: normal
problem app: crashes during plugin/graphics initialization
recent change: approved driver or plugin update
application log: identifies updated component during failure
previous approved version/configuration: available

The scope is narrow.

The timing is strong.

The change record gives you a rollback path.

Strong theory

The updated component introduced a compatibility regression in the affected application path.

Controlled action

Use the approved rollback procedure for the implicated driver/plugin.

What would weaken the theory?

If the old version is restored and the same failure remains unchanged, the update-only theory loses strength.

Record that result instead of escalating the destructiveness of the fix just to produce movement.

Incident 5: managed mobile synchronization disabled by policy/configuration

User report:

My phone stopped syncing. Remove the management profile so it works again.

Evidence:

Wi-Fi: connected
IP/gateway: valid
cloud sign-in: successful
other approved cloud services: available
recent MDM/profile update: yes
required sync feature: disabled by supplied policy state

The local network and account are already proven well enough to reach the cloud service.

The failed requirement is controlled by the management configuration.

Bad fix

Remove MDM.

That solves one symptom by destroying the management boundary.

Better response

• verify what the intended policy should be;
• document the mismatch;
• correct or escalate the specific profile setting through the approved management path;
• verify sync returns;
• verify the device remains enrolled and compliant.

Incident 6: unexpected MFA prompts

User report:

My phone keeps asking me to approve sign-ins. Turn MFA off.

Evidence:

user is not initiating sign-ins
multiple MFA prompts: observed
prompts denied/not approved
normal account access: still available
endpoint malware evidence: none supplied
account-security escalation path: available

MFA is not the problem.

MFA is exposing the problem.

What is proven

Unexpected authentication attempts reached the approval stage.

What is not proven

• endpoint malware;
• password theft;
• identity-provider compromise;
• who initiated the attempts.

Defensive path

Follow the approved account-security process: preserve the timing/alert evidence, review account/session state, protect or reset credentials when authorized, revoke suspicious sessions where required, verify expected sign-in, and leave MFA enabled.

Incident 7: VM cannot reach the internet

User report:

The practice VM has no network.

Evidence:

host network: healthy
host internet: healthy
VM boots normally
VM virtual NIC: connected
VM network mode: host-only/internal
other host-only peer communication: works
client requirement: external access is now required for approved exercise

The virtual network is not necessarily broken.

It is doing what host-only networking is supposed to do.

The configuration no longer matches the requirement.

Controlled change

Move the VM to the approved external/NAT/bridged model defined by the lab, with authorization and awareness of the exposure change.

Verification

Test both:

• required external destination works;
• any required isolation boundary still behaves as designed.

“Internet works” is not enough if the VM was also supposed to remain separated from another network.

Incident 8: low/degraded storage affects application performance

User report:

The file service and application are slow today.

Evidence:

network latency: normal
CPU: normal
RAM: normal
storage free space: critically low
application log: write failures/timeouts
optional scenario variation: RAID/controller degraded state

The user experiences performance.

The system evidence points toward storage.

Strong questions

• Is the problem capacity, health, or both?
• Which data is consuming space?
• Is backup current?
• Is the degraded member identified?
• What action is safe without destroying recoverable data?

Do not delete files because a directory looks large.

Find the owner and retention/recovery requirement first.

Incident 9: approved change causes a compatibility regression

User report:

The update fixed the original problem, but the reporting application no longer works.

Evidence:

change ticket: approved
baseline: documented
backup: verified before change
new version: installed successfully
original defect: resolved
business-critical reporting app: incompatible
rollback package/configuration: available
maintenance window: still open

The deployment tool may say success.

The change is still unacceptable because the business requirement failed.

Correct operational response

Use the documented rollback.

Then verify:

• the previous supported state is restored;
• the reporting application works;
• the original defect is reopened/documented rather than falsely marked resolved.

Rollback is not failure.

Rollback is a control working as designed.

Incident 10: backup exists, restore has never been tested

User report:

We have backups, so recovery is covered.

Evidence:

backup jobs: report success
backup destination: available
restore test history: none
client requirement: recover important staff files

Do you have proven recovery?

No.

You have backup-job evidence.

The missing boundary is restore verification.

Controlled recovery test

Use an approved non-destructive test item and restore it to an alternate location.

Verify:

• restored file exists;
• file opens or validates correctly;
• expected version/date is correct;
• recovery steps are documented.

Do not overwrite the production copy just to prove restoration.

The faults cross domains on purpose

Notice how several incidents refuse to stay inside one category.

The no-POST case involves:

• hardware compatibility;
• firmware/POST;
• safe service;
• documentation.

The mobile sync case involves:

• networking;
• cloud identity;
• mobile management;
• policy;
• user communication.

The compatibility-regression case involves:

• applications;
• change management;
• backup;
• rollback;
• business requirements.

That is what systems actually look like.

Your assigned incidents

Each learner or team receives at least three incidents.

The set must include:

• at least one Core 1-heavy incident;
• at least one Core 2-heavy incident;
• at least one case where the user report points toward the wrong layer.

For each incident, produce:

incident ID:
user complaint:
scope:
last known change:
known-good boundaries:
first failed / uncertain boundary:
strongest evidence:
strongest theory:
competing theory:
what evidence would weaken the strongest theory:
controlled action:
safety/privacy/authorization boundary:
rollback/recovery:
verification of real requirement:
user explanation:
documentation/escalation:
remaining uncertainty:

Do not optimize for speed

The fastest student is not necessarily the strongest technician.

A five-minute diagnosis that destroys evidence, bypasses policy, or creates a second problem is worse than a ten-minute diagnosis that can be defended.

The goal is not hesitation either.

It is controlled certainty.

Before you move on

By the end of this lesson, Harborview should be back in a working state or have a clearly documented unresolved/escalated state for anything outside local authority.

Next you challenge the repaired environment itself:

• Is the access model still secure?
• Can required data actually be restored?
• Are updates and endpoint controls healthy?
• Does every required client function still work?

A repair is only complete when the whole requirement survives it.