How to Reduce AI Code Review Noise on Pull Requests

AI code review gets noisy when the reviewer comments on everything it sees and gives a renamed variable the same weight as a broken authorization check. To reduce the noise, first measure how many of the bot's comments actually lead to a code change. Then change what the reviewer is allowed to post: nothing when it found nothing, a finding only when it can show evidence, and one specific question when it can't decide on its own.

Why AI reviewers over-comment

Most of the noise comes from four mechanisms, and each needs a different fix.

The reviewer sees the diff, not the system. A model that reads only the changed hunks can't tell whether a function it calls "unused" is imported by other files. Octopus uses this as its example of a context-blind false positive: a function flagged as unused "because it can't see the three files that import it." CodeAnt's write-up lists context blindness alongside three other failure modes: volume without severity calibration, findings without reproduction steps, and comments that repeat rules your linter already enforces. Some tools now read beyond the diff. GitHub's docs say Copilot code review's full project context gathering "analyzes your entire repository" through GitHub Actions. But more context fixes only this first problem.

Every change gets the same weight. A diff carries no sense of risk. Left alone, a reviewer spreads its attention evenly, so a 40-line cleanup of log formatting draws as many comments as a 4-line change to session handling. When a style nit looks as urgent as a security flaw, engineers start treating everything as optional. Octopus calls this the "cry wolf" effect.

Saying nothing looks like failure. A reviewer that posts "looks good, but consider…" on a clean pull request seems to have done its job. A reviewer that posts nothing seems broken, unless it can show what it checked. If a tool has no way to report a clean result, it fills the space with suggestions.

Automatic triggers multiply everything. Reviewing on every push raises the same points again. In a GitHub community discussion from August 2026, a user describes each new commit adding about five automated Copilot comments that bury the human conversation. They wrote that the reviews were good and they didn't want to turn them off. The problem was the volume.

Measure signal before you change anything

You can't tell whether a tuning change helped unless you measured before it. Three measures are worth tracking, listed from most to least reliable.

Resolution rate

Resolution rate is the share of AI comments that are followed by a code change addressing them before merge. It's the closest proxy for "was this worth reading?" Two published reference points:

• An arXiv study of 16 AI code review GitHub Actions covered more than 22,000 review comments in 178 repositories. It found that 60% of valid human review comments led to code changes, compared with 0.9% to 19.2% of valid AI comments, depending on the tool. Higher addressing rates were associated with concise comments, comments that include code, comments attached to hunks rather than whole files, and manually triggered reviews. The authors note that these are associations, not causal effects.
• Cursor measures Bugbot's resolution rate by using AI to determine, at merge time, which flagged bugs the author actually resolved in the final code. Its January 2026 post reports the rate rising from 52% to over 70%, and calls the metric "a much clearer signal than anecdotal feedback or reactions on comments." Cursor measured this for its own tool.

Reactions and resolved threads

Thumbs up, thumbs down and "Resolve conversation" clicks are cheap to collect, but they record mood and housekeeping, not fixes. A resolved thread can mean "fixed" or "go away." Use them to spot comment categories people dislike, not as your main number.

Signal ratio from a hand-labeled sample

Jet Xu's framework sorts comments into three tiers:

1. Critical signal, such as runtime errors, breaking changes and exploitable vulnerabilities.
2. Important signal, such as architectural inconsistencies and measurable performance regressions.
3. Noise, such as style and subjective opinions.

Signal ratio is tiers 1 and 2 divided by total comments. His thresholds (above 60% is good, above 80% is great) are his own, and he builds one of the tools he compares. The labeling exercise is still worth doing on its own. CodeAnt suggests a similar exercise: take 20 to 30 dismissed findings and sort them by failure mode.

Pulling the data from GitHub

In GitHub's GraphQL reference, each pull request review thread reports whether it has been resolved (isResolved) and whether it "was outdated by newer changes" (isOutdated). "Outdated" is only a rough proxy for "the author fixed it": the field doesn't say whether the newer changes addressed the comment. Read a sample before you rely on it.

# Review threads started by your AI reviewer on one merged PR.
# Copy the bot's exact login from one of its comments first.
gh api graphql -F owner=acme -F repo=api -F pr=1234 -f query='
query($owner: String!, $repo: String!, $pr: Int!) {
  repository(owner: $owner, name: $repo) {
    pullRequest(number: $pr) {
      reviewThreads(first: 100) {
        nodes {
          isResolved
          isOutdated
          path
          comments(first: 1) { nodes { author { login } } }
        }
      }
    }
  }
}' --jq '.data.repository.pullRequest.reviewThreads.nodes[]
  | select(.comments.nodes[0].author.login == "your-review-bot")
  | [.path, .isOutdated, .isResolved] | @tsv'

Run it across 30 to 50 merged pull requests, then label each comment:

Teams tend to overlook the last row, and it points to the third fix below.

Three alternatives to commenting on everything

1. Stay silent when nothing is found

A clean investigation should produce one status, not a compliment or a list of optional ideas. Silence can only be trusted if the tool also says what it checked and reports clearly when it couldn't finish. A reviewer that crashed and a reviewer that found nothing must not look the same.

2. Clear findings with evidence

A finding earns a line comment when it says what goes wrong, where, and why the reviewer believes it. The evidence should be concrete: the caller that passes the bad value, the test that fails, or the code path that skips a check. Anything that can't meet that bar should either not be posted or appear as collapsed, non-blocking advice. Severity should decide what blocks, and findings below your threshold shouldn't interrupt anyone.

3. Ask one bounded question

Much of the "Vague" row is real uncertainty: the diff changes behavior, and nothing the tool can read says whether that behavior is intended. Posting it as a soft suggestion to everyone on the pull request is noise. Asking it once, as a specific question with defined answers, to the person who knows, turns it into a decision.

Worked example: Warpway's lens end states and human tasks

Warpway is built around these three patterns, so it works as a concrete reference. It investigates every eligible GitHub pull request through lenses such as Correctness, Security and Testing. Each lens uses read-only tools on a snapshot of the head commit: reading files, finding references, history and blame, related tests and CI results. Every lens ends in exactly one of four states:

Each result also states its coverage: what was investigated, what wasn't, and a level from strong to incomplete. Warpway doesn't present a lens's internal confidence as a probability that the change is safe.

On GitHub you get one Warpway / Review check and one summary comment that's edited in place as the review progresses. When a question goes to someone on GitHub, Warpway also posts a comment that mentions them with the question, and later questions get a new notifying comment. Concrete findings appear as line annotations, and uncertainty never does. By default, the Architecture and Performance lenses warn but don't block.

Uncertainty becomes a human task, which must have seven parts before it reaches anyone:

1. The question.
2. Why it matters.
3. Why AI couldn't resolve it.
4. The evidence gathered.
5. The relevant code.
6. The expected kind of answer.
7. The effect of each answer.

A task missing any part is rejected, so a request like "Security confidence is 61%. Please inspect." doesn't go out. Routing sends each task to the person most likely to know, with an explanation of why they were picked. Answers are checked too: a reply of "Sounds fine" gets one concise follow-up and isn't recorded as a yes.

Two behaviors deal with repetition across pushes. When new commits arrive, a lens whose files didn't change keeps its previous result, unless that result was Incomplete or the lens or policy changed. An earlier answer is reused as long as the code its question depended on is unchanged, so nobody is asked the same unchanged question twice.

You can also tell Warpway what to skip and what should never block. Configuration as code requires the Team plan and is read from the base branch:

version: 1
review:
  ignore_paths:
    - "**/generated/**"
    - pnpm-lock.yaml
  ignore_authors:
    - "renovate[bot]"
lenses:
  architecture:
    blocking: none
merge_gate:
  block_on_severities:
    - critical
    - high

Feedback such as marking a finding as a false positive, or flagging that "AI should have resolved this," is used to measure and improve review quality. It never changes your policy by itself.

For an example of a finding with evidence, see the requests 2.32 SSLContext case study. It's a replay: on 2026-10-07, the merged pull request was run through Warpway as it stood before merge, and Warpway flagged the client-certificate mix-up later reported as #6726.

Some limits to know:

• Warpway works with GitHub only.
• Repositories start at the Assist trust level by default, where the check never blocks a merge.
• Blocking merges requires the Gate level and the Team plan. The Free plan caps the trust level at Assist. See merge gating.

Tune the reviewer you already have

If you're not switching tools, these steps target the mechanisms above:

1. Measure for two weeks first with the labels above, so you can tell whether each change helps.
2. Stop reviewing every push if your tool allows it. GitHub's docs, for example, describe a "Review new pushes" setting for Copilot code review. Without it, Copilot reviews a pull request once, and you can re-request a review later. In the arXiv study, manually triggered comments were addressed more often.
3. Exclude what nobody reads, such as generated code, lockfiles and snapshots.
4. Remove linter overlap. If ESLint or your SAST tool already enforces a rule, tell the AI reviewer not to comment on it.
5. Write down what not to comment on in the tool's instructions: naming, formatting, and "consider adding tests" without a named test case.
6. Gate by severity. Let only high-severity findings interrupt people, and collapse or suppress the rest.
7. Measure again. If the resolution rate doesn't move, the configuration probably isn't the problem; what the tool is allowed to post is. That's the point to evaluate a different approach. The Greptile alternative guide and how to review AI-generated pull requests cover that next step.