The Agent Said It Was Done. Prove It.

Core path: 30 minutes

A coding agent saying:

Done. Tests pass and the feature is complete.

is a status message from the agent.

It is not your evidence.

This lesson is where you verify the claim yourself.

Start with the repository, not the chat

Your first questions are:

What files changed?
What changed inside them?
Do those files match the approved scope?
Do the tests still encode the requirement?
Does the runtime behavior match the claim?

Use the terminal simulation below to practice the verification sequence.

Robotnix agent-verification simulation
A coding agent changed app.py and tests/test_app.py.
The prompt reflects the current Git state while you gather evidence.
Type 'help' for supported commands.

$ git status
Changes not staged for commit:
        modified:   app.py
        modified:   tests/test_app.py

$ git diff --stat
 app.py            | 8 +++++---
 tests/test_app.py | 6 ++++++
 2 files changed, 11 insertions(+), 3 deletions(-)

$ git diff
diff --git a/app.py b/app.py
+def export_players(players):
+    ...
diff --git a/tests/test_app.py b/tests/test_app.py
+def test_export_players_empty_list():
+    ...

$ python -m pytest
============================= test session starts =============================
collected 6 items

tests/test_app.py ......                                              [100%]

============================== 6 passed in 0.18s ==============================

$ git add .

$ git commit -m "Verify bounded agent change"
[main c93d1a8] Verify bounded agent change
 2 files changed, 11 insertions(+), 3 deletions(-)

$ git push origin main
To github.com:student/python-project.git
   44aa902..c93d1a8  main -> main

Try to stage and commit immediately.

The simulation should block the commit until you inspect the diff and run tests.

That is deliberate.

git status tells you the blast radius

Run:

git status

Compare the changed paths against the Feature Brief.

If the task allowed:

reports.py
tests/test_reports.py

and status shows:

storage.py
main.py
README.md

as additional changes, stop.

You have a scope question before you have a code-quality question.

git diff --stat gives you the shape

Run:

git diff --stat

This is a fast first look at the size and file distribution of the change.

A task described as "change one comparison and add one test" should not casually become 400 changed lines across six modules.

The stat does not tell you whether the code is correct. It tells you where to look.

git diff gives you the actual candidate

Now inspect:

git diff

Read the changed lines.

Check:

• operators at boundaries;
• removed validation;
• altered tests;
• unrelated refactors;
• new dependencies;
• changed function signatures;
• deleted behavior;
• comments or TODOs hiding unfinished work.

An agent summary can omit any of those accidentally or confidently.

The diff cannot hide the fact that the file changed.

Run the tests yourself

python -m pytest

Do not accept:

The agent already ran them.

Run them in the environment you are using for verification.

If tests fail, preserve the failure output before asking for another change.

If tests pass, ask which requirements they actually cover.

Add the test the agent forgot

For the leaderboard example:

requirement: include score >= minimum

The high-value adversarial case is:

player score == minimum

That exact boundary catches > versus >=.

Add or run a test for it.

Generated tests tend to prove the cases the generator remembered. Your job is to look for the case that would embarrass the implementation.

Separate capability, permission, and authority

An agent may be technically capable of editing storage.py.

The tool permission may even allow it.

But the Feature Brief may authorize only:

reports.py
tests/test_reports.py

Those are three different ideas:

CAPABILITY
what the agent can technically do

PERMISSION
what the tool/environment allows

AUTHORITY
what this task actually authorizes

Do not collapse them into one green button.

Recovery is part of verification

If the diff contains an unauthorized change, recover deliberately.

Before recovery, record:

unapproved file:
change detected:
why it violates scope:

Then restore/revert according to your Git state and classroom workflow.

Afterward:

git status
git diff
python -m pytest

Prove the repository is back inside the authorized boundary.

Build the verification record

For the final agent-assisted change, preserve:

agent claim:
approved files:
actual changed files:
diff inspected: yes/no
one suspicious line or assumption reviewed:
test command:
test result:
adversarial test:
manual check:
recovery action, if any:
final git status:

Then answer one question without referring back to the agent conversation:

Why should another developer believe this change is acceptable?

If your answer is "because the agent said it finished," the review failed.

The agent can move fast.

You still own reality.