How to Roll Out AI Code Review to Your Team in Stages

Roll out AI code review in five stages, one repository at a time:

• Observe: it reviews every pull request but asks nobody to act and blocks nothing.
• Assist: its comments and questions reach people but never block.
• Gate: a required check blocks merges only on the finding classes you have measured as reliable.
• Clear: it resolves routine concerns it can back with evidence.
• Auto-approve: it approves narrow classes of change when every deterministic condition passes.

Record your current review metrics before stage one, and write down each stage's exit criterion before you enter it.

The five stages at a glance

Published playbooks agree on how to start:

• CodeAnt's rollout guide opens with a two-week pilot in observation mode, with no blocking.
• cubic has a Shadow mode for auto-approval.
• Rewind ran about a week of shadow reviews before letting its internal reviewer approve anything.

They differ on the later stages, and that is where the hard decisions are.

Before you start: record a baseline

Without a baseline you can't tell whether a stage helped or just moved the wait somewhere else. Take four to eight weeks of merged pull requests from each pilot repository and record:

The first five come out of the GitHub CLI and jq:

gh pr list --state merged --limit 500 \
  --search "merged:>=2026-09-15" \
  --json number,author,createdAt,mergedAt,additions,deletions,reviews \
  > baseline.json

jq '
  def hours(a; b): ((b | fromdateiso8601) - (a | fromdateiso8601)) / 3600;
  def median: sort | if length == 0 then null else .[length / 2 | floor] end;
  map(.author.login as $me
      | . + {firstReview: ([.reviews[] | select(.author.login != $me) | .submittedAt] | min)})
  | {
      prs: length,
      median_hours_to_first_review:
        (map(select(.firstReview != null) | hours(.createdAt; .firstReview)) | median),
      median_hours_to_merge: (map(hours(.createdAt; .mergedAt)) | median),
      median_changed_lines: (map(.additions + .deletions) | median),
      merged_without_other_review: (map(select(.firstReview == null)) | length),
      changes_requested_per_pr:
        ((map([.reviews[] | select(.state == "CHANGES_REQUESTED")] | length) | add) / length)
    }' baseline.json

Two adjustments to the numbers:

• If a bot already reviews your pull requests, exclude its login. Otherwise an existing AI reviewer counts as the "first review."
• Report Dependabot pull requests separately. They are your most likely first auto-approve class.

Then agree in writing on how you will label findings, as Qodo's guide recommends for acceptance rate. A simple scheme has three labels:

• real and fixed
• real but not worth fixing
• wrong

Keep a separate log of missed problems that people or incidents found later.

Stage 1: Observe without blocking

The reviewer runs on every eligible pull request, nobody is asked to do anything, and its check never fails. Some tools keep results in a dashboard; others post them on the pull request. Either way, the point of this stage is to produce labels.

• Label per class, not overall. Track precision by category and severity. A single false-positive rate hides the case that matters: a reviewer can be reliable on injection and unreliable on concurrency, and only the first should ever block.
• Replay known failures. Pick a few merged pull requests that later needed a revert or hotfix. Recreate each one from its pre-merge head on a scratch branch, and see whether the reviewer flags the problem. Warpway's case studies are built this way. For example, a replay run on 2026-10-07 of the requests 2.32 pull request, as it stood before merge, flags the client-certificate mix-up that was later reported as #6726.
• Pilot where pull requests flow. Qodo's guide suggests starting where PRs actually flow, such as a dependency cluster in a multi-repo estate.

Set the exit threshold in numbers before you start. CodeAnt's playbook, for example, ends week two with a go/no-go that includes a false-positive rate under 12%. Pick your own, and count findings rather than days: a quiet repository can need several weeks to produce enough of them.

Stage 2: Assist

Findings become comments, and questions the reviewer can't settle go to people. Nothing blocks. This stage shows what Observe could not: how people respond.

• Acceptance: the share of comments that led to a change.
• Noise: comments per pull request. Exclude generated code, lockfiles and snapshots before they produce comments.
• Routing: whether questions reached someone who could answer them.
• Wait: time to first human review, compared with the baseline.

Before you promote, downgrade or suppress the classes people consistently dismiss. How to reduce AI code review noise covers the tuning.

Stage 3: Gate, and how to decide what may block a merge

A gate is where an AI reviewer can cost you the most time, so start narrow. A finding class earns blocking status only when all four of these hold:

1. Measured precision. Your labels show it is right often enough to meet the threshold you set.
2. A costly miss. For example security flaws, data loss, destructive migrations and incompatible API changes.
3. Actionable by the author. The finding names the code and the fix. A question that needs a domain owner can block too, but only once routing has shown it reaches someone who answers.
4. A recorded way out. Someone can override the block, and the override is logged with a reason.

A starting policy:

The last row matters. A gate that reports success when the review never ran is worse than no gate, so find out whether your tool fails open or closed.

In GitHub, make the reviewer's check a required status check in a ruleset or branch protection. Two details from GitHub's protected-branch docs matter here:

• A required check passes with a successful, skipped or neutral status. A reviewer that reports neutral therefore blocks nothing, even when the check is required.
• By default, branch protection restrictions don't apply to repository admins, so decide who may bypass the gate.

How to block merges on AI code review findings walks through the setup.

After that, watch overrides and compare time to merge with the baseline. CodeAnt's playbook downgrades or disables any rule that is overridden more than 15% of the time.

Stage 4: Clear routine pull requests

At Clear, the reviewer may resolve a concern without a person when it has evidence: the change crosses no trust boundary, or tests cover the changed behavior. Human reviewers then spend their time on what is left. The final approval still comes from a person.

Decide which concerns AI may never clear alone, such as authentication, payments or regulated data, and require a human sign-off on every pull request they apply to. Before you promote further, spot-check a sample of AI-cleared pull requests each week and compare escaped defects in those areas with the baseline.

Stage 5: Auto-approve under strict conditions

Approval without a person is only for narrow, low-risk classes of change, such as documentation and test-only changes or small dependency updates. Every condition should be deterministic and checked against the exact commit:

• the head commit is the commit that was reviewed;
• the named CI checks succeeded, and nothing else is failing or pending;
• no findings at a blocking severity and no questions are open;
• changed lines and changed files are under a cap;
• the change touches no excluded paths or categories, such as auth, migrations, infrastructure or billing;
• the author is on an allowlist, for example Dependabot.

Never approve because a model reported high confidence. Rewind's post describes a deterministic, unit-tested verdict engine that makes the decision. Changes that touch authentication, authorization, secrets, encryption or governance files always go to a human.

Run auto-approval in shadow before it goes live:

• As of November 2026, cubic's auto-approval docs describe a Shadow behavior that keeps cubic in comment-only mode and adds a summary of whether it would have approved. The docs recommend watching real pull requests, adding never-auto-approve rules for sensitive paths, and then switching to Live.
• Rewind required at least 90% agreement between its bot's approvals and human reviews of the same pull requests.

Finally, turn on GitHub's option to dismiss stale approvals when new commits are pushed, so an approval never outlives its commit. See how to auto-approve pull requests in GitHub safely.

Lock the policy before you gate

Once a reviewer can block or approve, its configuration becomes a target: a pull request could edit the config to remove the rule that would block it. Before stage 3, confirm three things about your tool:

• Configuration is read from the base branch, so a pull request is reviewed under the policy it is trying to change.
• The organization can set a ceiling that repositories can lower but not raise.
• Changes to stages and policy are audit-logged, with who made them and when.

Running the stages with Warpway

Warpway's trust levels correspond to these five stages and are set per repository:

Things to know before you start:

• New repositories start at the organization default, which is Assist unless an Owner changes it. Set Observe explicitly if you want a stage 1.
• The effective level is the lowest of several settings: the dashboard level, any trust_level in organization settings, repository settings or the base branch's .warpway.yml, and the plan maximum. The file can lower the level but never raise it.
• An Owner can lock keys for every repository, including trust_level, merge_gate and auto_approve. See configuration as code.
• Warpway fails closed. A required lens that cannot finish is Incomplete and blocks, and the check never reports success for a review it did not do.
• Feedback. People can mark a finding as a false positive, say a question went to the wrong person, or report an issue Warpway missed. Warpway uses this feedback to measure and improve review quality. Keep your own labels for the per-class precision numbers.
• Limits: Warpway works with GitHub only, never merges, and does not report on merge-queue groups.

Here is a .warpway.yml for a repository moving from Gate toward Auto-approve:

version: 1

merge_gate:
  # The default also blocks on medium; add it back once medium findings prove precise.
  block_on_severities:
    - critical
    - high
  fail_on_incomplete: true

lenses:
  auth_security:
    name: Auth Security
    extends: security
    required: true
    applies_when:
      paths:
        - src/auth/**
    ai_clear: false   # at Clear and above, a person signs off

auto_approve:
  enabled: true
  docs_tests_only: true
  max_changed_lines: 300
  excluded_paths:
    - src/auth/**
    - db/migrations/**
  required_ci_checks:
    - unit-tests

auto_approve.enabled does nothing on its own. Auto-approval also needs all of these:

• the Team plan;
• trust level 4;
• an Owner turning on the organization switch;
• an Owner enabling it for the repository.

The full list of conditions is in auto-approval.