How reviews work
The life of a Warpway review: from webhook to plan, lenses, evidence, human tasks, policy and the GitHub check.
A Warpway review answers seven questions for every eligible pull request: what changed, what risks were investigated, what evidence was gathered, what problems were found, what could not be resolved, who should resolve each open question, and whether your review policy is satisfied.
What starts a review
| Trigger | Notes |
|---|---|
| Pull request opened or reopened | |
| New commits pushed (synchronize) | Starts a new review of the new head commit. |
| Draft marked ready for review | Drafts are skipped unless review.include_drafts is true. |
Re-run on the Warpway / Review check, Rerun in the dashboard, or a /warpway rerun comment on the pull request | Reviews the same commit again. |
| An answer, CI result or runtime result arrives | Re-evaluates only the affected lenses of the current review. |
Pull requests can also be left out by configuration: authors listed in review.ignore_authors, base branches outside review.base_branches when that list is set, and pull requests whose changed files all match review.ignore_paths.
One review per commit
A review run is bound to one repository, one pull request, one exact head commit and one policy version (the resolved configuration, identified by its hash), and it records the model, prompt, lens and tool versions it used. A run's commit and policy version never change. A new commit creates a new run, and the previous run is marked superseded.
The steps of a review
- Receive the event. The webhook signature is verified and the delivery ID recorded, so a repeated delivery does nothing.
- Fetch context. Pull request metadata, changed files and diff, commits, existing reviews and comments, linked issues, the CI checks on the head commit, and
CODEOWNERSand.warpway.ymlfrom the base commit. - Resolve the policy. Defaults, organization settings, repository settings and the base-branch configuration file are combined, organization locks and plan limits applied, and the result stored as a policy version. See Configuration as code.
- Start the check.
Warpway / Reviewis created on the head commit with statusin_progress. - Analyze the change. A summary of what changed, affected subsystems, risk areas, semantic categories (such as
auth,billingormigration) and files worth reading. - Plan the review. Which lenses apply and whether each is required, which rules and organization knowledge are relevant, which tools each lens may use, which runtime profiles to run, and who is likely to answer questions. The planner can add lenses but cannot drop a required one.
- Run the lenses. Each lens investigates independently with read-only tools on a snapshot of the head commit: reading files and line ranges, searching, finding symbols and references, history and blame, related tests, CI status, linked issues, previous reviews and organization knowledge. Every tool result becomes evidence with its provenance.
- Arbitrate. Completed required lenses and their declared dependencies make their results and questions available while unrelated advisory lenses continue. Findings and compatible questions are deduplicated, claims are checked against cited evidence, and contradictions are exposed. An Incomplete required investigation never becomes Cleared because another lens finished. Later advisory results update the current head; a newly demonstrated material defect follows the configured policy, receives an audit event and a GitHub notification, and invalidates auto-approval eligibility where applicable.
- Route human tasks. New questions record the applicable required decision or their explicit advisory classification, the missing fact, why it matters to that decision, the recipient’s role and each offered answer’s effect. References are checked against trusted planning and applicable rules. At trust level 1 and above, each open question is assigned to the person most likely to know, in Slack or GitHub. See Routing.
- Evaluate the policy. Deterministic code decides whether the policy is satisfied and lists every blocker.
- Publish. The check run and the summary comment are updated, but only if this run is still the current one for the pull request's head commit.
The four lens states
Every lens ends in exactly one state:
| State | In the check | Meaning |
|---|---|---|
| Cleared | ✓ | The required investigation was completed and no material concern remains. |
| Issue found | ✗ | A concrete, actionable defect or risk was identified. |
| Human decision required | ⚠ | A material question remains that needs human judgment or organizational knowledge. |
| Incomplete | ◌ | Required work could not be completed: missing context, a tool or model failure, missing permissions or a failed verification run. |
Waiting for people
Nothing waits in memory. When a lens needs a human answer, the review records its state and the check reports what it is waiting for. When the answer arrives, Warpway checks that it actually settles the question, records it as evidence, re-runs only the affected lenses plus a global sanity pass over the whole change, evaluates the policy again and updates GitHub. If a worker restarts in the middle, the review resumes from its stored state.
New commits
When new commits arrive:
- a new review run starts for the new head commit and the old run is superseded. The old run may finish its work, but it can no longer update the check, the summary or anything else on the pull request;
- Warpway compares the new head with the last reviewed head to find what changed;
- a lens whose files did not change keeps its previous result instead of investigating again, unless that result was Incomplete or the lens or policy changed;
- earlier human answers are reused as they are if the code the question depended on has not changed; otherwise each answer is checked against the new code, and its task reopens if the answer no longer holds;
- findings that the new commit fixed are recognized and closed;
- affected lenses run again, followed by a global sanity pass.
Nobody is asked the same unchanged question twice.
When something fails
Warpway fails closed. If a model call fails, required context cannot be loaded, the repository cannot be fetched, a runtime verification run fails or times out, an answer is ambiguous, or the configuration is invalid, the affected work is not marked Cleared: a lens that could not finish reports Incomplete with the reason, an ambiguous answer leaves its task open, and an invalid configuration is reported as a configuration error. The review policy is then not satisfied, so the check never reports success. Transient failures are retried with backoff first; work that still cannot complete stays visible as a blocker.
Usage limits
If your plan's review limit is reached, Warpway does not review the pull request and the check says Usage limit reached. It never reports success for a review it did not do. See Billing.
What you see on the pull request
The check run. One check, Warpway / Review, bound to the head commit. Its summary says in one line what remains, then shows each lens's result:
1 open question waiting on Priya Shah · nothing else is blocking
| Lens | Result |
| Correctness | ✅ Passed |
| Security | ✅ Passed |
| Testing / Reliability | ✅ Passed |
| Product Semantics | 🙋 Waiting on Priya Shah |
1 question answered · Trust level 2 (Gate) · Open in WarpwayOn GitHub the lens states read in plain words: Passed (Cleared, or AI-cleared at Clear and above), Issues found, Waiting on the person a question went to (Human decision required) and Couldn't finish (Incomplete).
A continuing advisory lens is shown separately from an Incomplete investigation. An optional lens configured to block on incomplete work or always require human signoff is a declared dependency of required work. The check's title sums up the result, for example Waiting on 2 open questions; Merge gating lists every conclusion. The check's details list the findings to fix first, then the open questions with why each person was asked, every policy blocker, and the model, prompt and policy versions the review used, each collapsed until opened.
The dashboard records when required results became usable separately from advisory completion. Stage timings distinguish queue wait, analysis, planning, required/advisory execution spans, consolidation, answer validation and observed task delivery/human waiting. Concurrent spans overlap, and summed task clocks are not reviewer effort. Legacy or reused stages without an observation remain unknown. These measurements do not establish a response promise or customer time savings.
Concrete findings appear as line annotations. Uncertainty never does.
The summary comment. One comment, edited in place as the review progresses. It leads with where the review stands and what remains; everything else is one click away:
⛔ Changes needed 3f9c2e1
1 blocking finding · 1 open question waiting on @marcus
Fix before merging
▸ 🔴 High · Retry loop can enqueue the same invoice twice. · src/billing/retry.ts:42-58
Open questions
▸ 🙋 Revoking an administrator role · @marcus · T4F2A9C
▸ Lens results · 3 passed · 1 with issues · 1 needs a decision
▸ Resolved · 1 answered
Trust level 2 (Gate) · Open in WarpwayEach finding and question is a collapsed row (▸): a finding opens to its explanation, the suggested fix and the lens that found it; a question opens to the question, its choices with what each one records, how to answer and why that person was asked. Lens results, non-blocking findings, resolved questions and lenses that disagree are collapsed below. When they apply, the comment also lists configuration errors, failing required checks, and lenses that couldn't finish with their reasons, and it notes when the pull request changes Warpway's configuration, which only applies after merge. When the review recorded them, it ends with the human review surface of the commit: changed lines analyzed and lines flagged for human review. These are counts, not an estimate of time saved.
Answering on GitHub. Each open question has a short code such as T4F2A9C. In a pull request comment, /warpway answer <code> <answer> answers it (use the choice id when choices are listed), /warpway wrong-person <code> says you are not the right person, and /warpway rerun reviews the current commit again. A person asked on GitHub also gets one comment that mentions them, with each question, its choices and how to answer; the pull request's author cannot answer its own review questions.
Reviewer requests. When a question is best answered by an engineer, Warpway can request them as a reviewer, at most two per pull request by default. People reached only in Slack are not. See Routing.
The dashboard. The pull request page shows the change summary, head commit, review plan, every lens with its evidence, findings, human tasks and their conversations, the policy state, an audit timeline and a rerun button.
Something unclear or missing? Email marcus@cmglabs.ai.