Process and Service State Are Evidence

One system can tell several different stories at the same time.

The process list can say a program is running. The service manager can say a service is active. A log can show repeated errors. A listening-socket view can show a port bound only to the local machine.

None of those views automatically cancels the others.

They answer different questions.

Four evidence views

For the fictional RNX-LAB-SVC-01 status service, we care about four evidence families.

The last row is a common reasoning failure.

Listening and reachable are not synonyms.

Synthetic evidence set

Process evidence

PID   USER          COMMAND
733   rnx-status    status-web
748   rnx-status    status-worker

Service-manager evidence

rnx-status.service
state: active
main_pid: 733
started: 11:00:04Z
restart_count: 0

Listening-socket evidence

ADDRESS       PORT   PROCESS
127.0.0.1     8080   status-web(pid=733)

Application log

11:02:11Z level=INFO  event=startup bind=127.0.0.1:8080
11:02:14Z level=INFO  event=status-read records=6
11:04:02Z level=ERROR event=template-load path=/opt/rnx/templates/status.html result=permission-denied
11:04:02Z level=WARN  event=response route=/status result=500
11:04:40Z level=INFO  event=status-read records=6

What can we prove?

Claim: “The web process is running.”

Supported by the process inventory and service-manager state.

Claim: “The status service is healthy.”

Not supported.

The log records a permission error and a 500 response.

Claim: “Port 8080 is open to the internet.”

Not supported.

The listener is shown on 127.0.0.1, the loopback address. The supplied evidence does not show an external network path.

Claim: “The template permission caused the 500 response.”

Plausible and fairly strong because the events occur together and the error names the template path, but a careful investigator may still want a controlled retest after the permission issue is corrected.

Correlation strengthens when independent sources agree

Imagine you had only this log line:

event=template-load ... result=permission-denied

Useful, but narrow.

Now add:

• the process owner rnx-status;
• file permission evidence showing the service identity cannot read the template;
• a 500 response at the same time; and
• a successful response after a controlled permission correction.

The explanation becomes stronger because different evidence sources converge on the same mechanism.

This is corroboration.

It is one of the most important habits in technical troubleshooting and security analysis.

Local observation path

On an instructor-provided isolated Linux environment, these commands can show different views:

ps -eo pid,user,comm,args
systemctl status <approved-lab-service>
ss -lntp

Use only the service named by the lab. Do not turn a local observation lesson into scanning other hosts.

ss -lntp shows local listening TCP sockets and associated process information when permissions allow. It does not tell you that a remote system can connect through routing, firewalls, NAT, cloud security controls, or another boundary.

No-Linux alternative

Use the supplied evidence set above. The learning target is correlation, not command memorization.

Activity: evidence matrix

Create a matrix for these questions:

1. Is the process running?
2. Which identity owns it?
3. Is the service manager reporting it active?
4. Is there a listener?
5. What address is it bound to?
6. Did the application report a failure?
7. Which resource is named in that failure?
8. What still needs to be inspected?

For each answer, name the evidence source.

Do not write “the system shows.” Write “the process inventory shows,” “the service record shows,” or “the application log records.”

Source precision makes your reasoning auditable.

A useful troubleshooting order

When a service fails, random resets destroy evidence and make learning harder.

A better sequence is:

EXPECTED FUNCTION
      ↓
PROCESS STATE
      ↓
SERVICE STATE
      ↓
RESOURCE / PERMISSION
      ↓
SOCKET / BINDING
      ↓
APPLICATION LOG
      ↓
CONTROLLED RETEST

The exact order can change, but the principle remains: test a hypothesis with evidence instead of changing five things at once.

Add to your Service Responsibility Map

Extend your Week 2 artifact:

Your final row should say what evidence you need next.

For this scenario, the next useful source is the file permission state for /opt/rnx/templates/status.html.

Lesson 3 gives you that evidence and asks whether the service has more privilege than its job actually requires.