How to Review AI-Generated Pull Requests

To review an AI-generated pull request, check it against what was asked for rather than reading the diff from top to bottom. Start from the linked issue and read the dependency and test changes before the implementation. Save line-by-line reading for the files that can do real damage, and ask for evidence wherever code only looks right. The few questions you can't settle go to the one person who can answer them.

Why AI-generated pull requests need a different review

A tired human leaves visible traces: inconsistent naming, a forgotten case, a sloppy commit. Agent output looks finished whether or not it is. Aviator's guide describes it as code that looks "written by a meticulous engineer who also happened to be confidently wrong." Your job shifts from spotting sloppiness to testing whether plausible code is actually correct.

GitHub's, CodeRabbit's and Aviator's guides describe three recurring failure modes:

• Wrong target. The code works but solves a slightly different problem. CodeRabbit points out that the pull request description reports what the agent produced, not what was requested.
• Invented or unneeded dependencies. GitHub's guide warns about hallucinated packages that don't exist, and about slopsquatting, which it describes as a theoretical attack on LLMs using fake or malicious packages.
• Tests that prove nothing. GitHub lists tests that were deleted or skipped instead of fixed. Aviator adds tests that assert too little or mock away the logic they claim to cover.

Size makes all three worse. CodeRabbit's guide cites LinearB's 2026 benchmark report, which found AI-assisted pull requests 2.6 times larger than unassisted ones. Reading every line doesn't scale to that, so the process below decides where line-by-line reading is worth the time.

Step 1: Start from the linked intent

Open the issue, ticket or spec before you open the diff. Write down in two to four lines what must be true when the change merges: the behavior, the constraints, and anything explicitly out of scope. That list is what you review against.

Then treat the description as a claim to verify:

• Coverage. Each acceptance criterion maps to code and to a test. Aviator calls the gap the "looks done" trap: stubbed functions, leftover TODOs, or part of the request silently dropped.
• Scope. Each changed file is justified by the request. Unrelated edits get split out or explained.
• Omissions. Ask the author, or the agent, to state what it assumed and what it didn't do.

If no intent is linked, send the pull request back. Without a stated goal you have nothing to review against, and passing tests only confirm that the code does what the code does.

Step 2: Read dependency and test changes first

These areas are small, informative, and the cheapest place to catch agent mistakes. Check out the branch and read the file list before any logic:

gh pr checkout 123

# Every changed file, including ones GitHub collapses
git diff --stat origin/main...HEAD

# Deleted files, often removed tests
git diff --name-only --diff-filter=D origin/main...HEAD

# Manifest and lockfile changes
git diff origin/main...HEAD -- '*package.json' '*package-lock.json' \
  '*pnpm-lock.yaml' '*yarn.lock' '*go.mod' '*go.sum' \
  '*requirements*.txt' '*pyproject.toml' '*poetry.lock' '*Cargo.toml' '*Cargo.lock'

Dependencies. For each added or upgraded package, check four things:

• It exists under exactly that name.
• It is maintained.
• Its license fits your project. GitHub's guide gives the example of an AGPL-3.0 package in an MIT project.
• The change actually needs it. Aviator flags a new dependency for a trivial task, and a heavy library pulled in for one function.

Read the lockfile as well as the manifest, because one direct upgrade can move many indirect dependencies. GitHub's dependency review shows these changes as a rich diff in the Files changed tab: added, removed and updated packages, with release dates and vulnerability data. The dependency-review-action can fail a check when a change introduces a vulnerable version. As of October 2026, GitHub's docs list the action for all public repositories, and for private repositories with GitHub Code Security or GitHub Advanced Security.

Tests. Search the added lines for anything that switches a check off:

git diff origin/main...HEAD | grep -nE '^\+.*(\.skip\(|xdescribe\(|@pytest\.mark\.(skip|xfail)|t\.Skip\(|#\[ignore\]|@ts-ignore|@ts-expect-error|eslint-disable|type: *ignore|nolint)'

When a test changes in the same pull request as the code it covers, ask why the expectation changed: because the requirement changed, or because the code did? Also read any edits to .github/workflows/, coverage thresholds and lint configuration. Lowering a bar turns CI green without fixing anything.

GitHub hides paths marked linguist-generated in .gitattributes by default in diffs. If the pull request touches .gitattributes, or edits files that GitHub collapses, rely on the --stat output for the complete file list.

Step 3: Triage the rest by risk

Once intent, dependencies and tests are settled, sort the remaining files into tiers. Aviator's version: auth, payments, data handling, concurrency, and anything user-facing or hard to undo get priority, while copy tweaks, log lines and icons can be skimmed.

Within a tier, read foundations before the code that depends on them. CodeRabbit recommends starting with schemas, shared types, interfaces, configuration, migrations and utilities, and reviewing mechanical edits in a separate pass from behavioral ones. It also treats 10 files or 400 changed lines as the point to plan the review deliberately or ask for a split. If scanners already run in CI, read their findings before the raw diff. The DEV Community walkthrough on triaging a 500-line AI pull request builds its whole workflow on reviewing ranked findings instead of lines.

Step 4: Ask for evidence over plausibility

"Looks right" is the weakest thing you can say about agent code. For every claim the pull request makes in a high- or medium-tier file, ask for something you can check:

You can check the first row yourself by running the new test against the old code:

base=$(git merge-base origin/main HEAD)
git restore --source="$base" -- src/billing/retry.ts   # old code, new test
npm test -- src/billing/retry.test.ts                  # should fail
git restore -- src/billing/retry.ts                    # back to the PR version

If the test passes against the old code, it doesn't test the fix.

Evidence matters whoever wrote the code. On 2026-10-07, Warpway replayed a merged Hono pull request as it stood before merge. The change added a fast path for c.json() that used Response.json(), which throws for undefined on Node. The replay flagged it, and the real change had been reverted after four days. The question that exposes this kind of bug is mechanical: does the new path return the same result for every input the old path accepted?

Step 5: Escalate only the real open questions

Some questions code can't answer. Must revoking a role end active sessions immediately? Do unused credits survive cancellation? These need a person, and for agent-written code the author often isn't that person: they wrote the prompt, not the policy.

Before escalating, try to settle the question yourself from the callers, the tests, the linked issue and earlier pull requests. Send what remains to whoever owns the decision: the product owner for intended behavior, the code or database owner for data, the security team for trust boundaries. CODEOWNERS and history help you find them:

git log --since=6.months --format='%an' -- src/billing/ | sort | uniq -c | sort -rn | head

Write the question so one reply can answer it:

• Weak: "Can someone from security take a look?"
• Answerable: "This caches permission checks for 15 minutes, and role changes don't clear the cache (permissions.ts:41-72). I found no spec for revocation timing. Must revoking an admin role end sessions immediately? Immediate / 15 minutes is acceptable / needs discussion."

How Warpway automates this triage

Warpway is a GitHub App. It reviews every eligible pull request: when one is opened, reopened, receives new commits, or is marked ready for review. It analyzes the change, tags it with semantic categories such as auth, migration or dependencies, and plans which lenses apply:

• Correctness, Security and Testing / Reliability run on every pull request.
• Backward Compatibility, Product Semantics and Database Safety become required when the change touches what they cover.
• Architecture and Performance warn but don't block.

Each lens ends in one of four states: Cleared, Issue found, Human decision required, or Incomplete. Warpway fails closed, so a lens that couldn't finish is never reported as Cleared. A human task must include seven parts: the question, why it matters, why AI couldn't resolve it, the evidence, the relevant code, the expected kind of answer, and what each answer leads to. Answers are checked before they count: a reply like "Sounds fine" gets one follow-up instead of resolving the task.

Routing picks the recipient from rule and lens owners, CODEOWNERS, review and commit history, and linked issues. Questions arrive in GitHub, or in Slack on the Team plan.

Two details matter for agent-written changes:

• Warpway reads its configuration from the base branch, so a pull request can't weaken its own review.
• Text in the code or the pull request is treated as data. A comment telling the reviewer to approve has no effect, and the Security lens reports the attempt.

The policy below, in .warpway.yml (Team plan), encodes steps 2 and 5:

version: 1
rules:
  - id: tests-not-weakened
    lens: testing
    instructions: >
      Flag tests this pull request deletes, skips or loosens. A test edited
      alongside the code it covers must still assert the behavior the
      linked issue asks for.
    severity: high
  - id: new-dependencies
    lens: security
    applies_to:
      categories:
        - dependencies
    instructions: >
      For every added package, confirm the code uses it and that an
      existing dependency does not already cover the need.
    owner: "@acme/platform"
routing:
  product_semantics:
    slack_user_group: "@product-leads"
    prefer_slack: true

Tools draw the triage line in different places. CodeRabbit's Triage, for example, orders a cross-repository queue of open pull requests by what each one needs next. Warpway works inside each pull request and decides which of its questions need a person.

Know the limits before you adopt it:

• Warpway works with GitHub pull requests only.
• New repositories start at the Assist trust level, where questions are routed but the check never blocks a merge.
• Blocking requires the Team plan, trust level Gate or higher, and merge gating set up in GitHub. The Free plan caps at Assist.
• Warpway never merges. It approves only at the Auto-approve level, and only when every deterministic condition passes.