Review AI Output Like a Pull Request

Core path: 30 minutes

AI-generated code should enter your project the same way any other untrusted change should: as a proposal that has to survive review.

Not because AI is uniquely bad.

Because any code you did not fully reason through can carry assumptions you did not notice.

The useful mental shift is:

AI RESPONSE ≠ PROJECT CHANGE

There is a review boundary in between.

Start with the diff, not the explanation

An AI response may include a very confident paragraph saying:

I added CSV export, preserved your existing behavior, and updated the tests.

Fine.

Now inspect what actually changed.

git diff --stat
git diff

The prose is a claim.

The diff is evidence about the files.

If the response says one file changed and the diff shows six, believe the diff.

Review scope first

Suppose your request allowed:

main.py
export.py

But the candidate also changed:

storage.py
tests/test_storage.py
README.md

Those changes may be reasonable.

They are still outside the original scope.

Your choices are:

reject them
revise the task and approve them deliberately
or ask for a narrower change

What you should not do is let scope expand invisibly because the generated patch "looked clean."

Check assumptions against the current code

Generated code often reveals what the model assumed your project looked like.

Examples:

player["name"]

when your project now uses:

player.name

or:

open("players.json")

when your project centralizes paths in storage.py.

The code may be valid Python and still be invalid architecture for your project.

Review means comparing the candidate to the system that exists now.

Tests generated by AI are candidates too

An AI can write a test that passes because it accidentally weakens the requirement.

Example:

Requirement:

CSV export must include name and score for every player.

Weak generated test:

assert output_file.exists()

The file existing does not prove the rows are correct.

A better test would inspect the actual contents.

So when AI changes tests, ask:

• Did the test become stricter or weaker?
• Does it still encode the original requirement?
• Did the candidate delete a failing assertion instead of fixing the code?
• Does the test actually exercise the feature?

Never treat green generated tests as automatic proof.

Hallucination can look like architecture

A coding assistant may confidently reference:

functions that do not exist
packages not installed
files not in the repository
API methods from another version
configuration keys your project never defined

That is why the right response is not:

The AI hallucinated. AI is useless.

It is:

Which claim can I verify against the repository, documentation, runtime, or tests?

Then verify it.

Compare three layers

For one AI-assisted change from Lesson 4, create this review:

REQUEST
what I asked for

CANDIDATE
what the AI proposed

PROJECT EVIDENCE
what the diff/tests/runtime actually show

Mark every mismatch.

Examples:

requested 2 files → candidate changed 4
requested standard library only → candidate imported pandas
candidate says empty roster works → test shows crash
candidate says no existing behavior changed → diff removed an old branch

Now you are reviewing the change instead of reviewing the tone of the response.

Human revision is not failure

If you change the generated code before accepting it, that is normal.

You may:

• rename a function;
• restore an architecture boundary;
• improve error handling;
• remove an unnecessary package;
• add a missing test;
• reject half of the candidate.

The goal was never "use the AI output unchanged."

The goal is to use assistance without surrendering judgment.

Build a one-page review record

For the CSV feature, record:

Original request:
Allowed files:
Candidate files changed:
Assumption I found:
Generated test I inspected:
One revision I made:
One rejection I made:
Verification command:
Verification result:
Final diff summary:

Then answer:

What evidence would convince another developer that the final feature is correct even if they never saw the AI conversation?

That is the standard.

The model can propose. The repository has to prove.

Run the code to see output.