workflowpull-requestscode-reviewagentsevalsautomationhuman-in-the-loopUpdated 2026-08-11

Self-Improving PR Review Workflow

Separate the per-PR review loop from the slower learning loop: reviews produce structured evidence; human corrections become measured, approval-gated proposals to improve the review skill.

Inner Loop — Review A Pull Request

  1. Load the PR description, diff, repository rules, and optional specification.
  2. Run the existing review capability with permission to read, build, test, and validate suggestions.
  3. Emit deterministic review.json containing findings, severity, file/line targets, evidence, checks run, uncertainty, and proposed comment text.
  4. Keep the reviewing agent read-only with respect to the PR and GitHub discussion.
  5. Let a narrow trusted publisher validate the JSON schema and post approved comments. This reduces the prompt-injection and over-broad-token surface.
  6. Record which comments humans accepted, corrected, dismissed, or ignored.

Outer Loop — Improve The Reviewer

Run on a batch or daily schedule, not after every comment:

  1. Aggregate recent human feedback and failed/missed review examples.
  2. Group recurring errors by rule, false-positive pattern, repository context, or missing verification.
  3. Propose the smallest change to the review skill, deterministic helper, rubric, or eval set.
  4. Run regression evals against both the motivating cases and a stable holdout set.
  5. Open a reviewable PR with before/after metrics, changed rules, cost impact, and rollback path.
  6. Require human approval before merge. Never let the reviewing system silently rewrite its own production instructions.

The proposer and acceptor must be operationally distinct. A different role label is insufficient when the same model, prompt, retrieval, tools, or hidden feedback produces correlated errors. Keep a fixed incumbent arm, measure accepted and dismissed findings by severity and rule, preserve false positives and misses, and replay the candidate on untouched pull requests. If the skill, rubric, model, publisher, or repository policy changes, record the new digest and invalidate comparisons that depended on the displaced condition.

The current Warp demo source at bd3a79ae contains seven skill files, five workflow files, and one test file under MIT. It remains a method fixture, not a production dependency or quality proof. Its strongest transferable boundary is still deterministic review.json plus a narrow publisher: the read-only agent may inspect/build/test, but trusted code turns validated findings into GitHub comments. Source: recovered Warp X Articles 2071663234772574209 and 2077425822084759552; warpdotdev-demos/cloud-factory-demo@bd3a79ae, reviewed 2026-08-12

Metrics

  • human correction and dismissal rate;
  • accepted high-severity finding rate;
  • missed-regression rate from later incidents or reviewers;
  • review cost, wall time, and validation cycles;
  • recurrence of a corrected failure after a skill update;
  • performance on a fixed holdout eval set.

Local Adoption

Use the existing local pr-review and agent-eval-library capabilities. Add portable structured output, a trusted publisher boundary, feedback capture, and an approval-gated improvement PR. Warp itself is reference evidence; it is not a required runtime dependency. Source: Warp article and demo repository, reviewed 2026-07-15

Failure Modes

  • treating comment volume as review quality;
  • letting the agent publish arbitrary prose with a write-capable token;
  • training on raw reviewer preference without distinguishing correctness from style;
  • updating the skill from one anecdote without holdout regression checks;
  • auto-merging self-modifications;
  • optimizing acceptance rate until the reviewer stops surfacing uncomfortable risks.

Timeline

Related Pages

Security and Review Skillsagent-eval-libraryAutomation OrchestrationHuman-in-the-Loop ControlProof-Carrying Work

4 pages link here

Full Corpus Workflow and Capability ProgramArchitectureHarness and Brain Workflow Refactor PlanArchitectureOpenCodeReviewToolsStraight Fats Harness Notes — July 15, 2026Research