Coding agent review debt: a measurement guide
Measure the hidden human review, correction, CI rerun, defect, and rollback work that can offset coding-agent output.
Compare the tools behind this article on ToolVerse.
Open ToolVerse for evidence, pricing context, alternatives, and current review status. Every link below navigates to the external ToolVerse directory.
Compare GitHub Copilot and SWE-agent and Promptfoo Open on ToolVerse · externalQuick answer
A coding agent can make a pull request appear cheap because its first patch arrives quickly. That is only the beginning of the work. The engineering system still has to read the diff, correct mistaken assumptions, rerun CI, answer review questions, operate the changed service, and repair failures found after merge. Call that displaced work review debt. It is a measurement category, not an accusation: the point is to reveal whether the agent increases accepted capacity or merely moves labor into less visible queues.
For an engineering leader, the decisive unit is an accepted, supportable change in a defined task class. Count a patch only after it clears the repository’s normal controls and its owner can operate it. A generated diff, a green agent self-test, an AI review comment, and an opened pull request are intermediate signals. They are not completed work. This guide gives a compact scorecard and a four-week protocol for deciding where coding agents actually create net capacity.
Definition: what review debt includes
One submitted attempt is one immutable row keyed by (task_id, change_id): the task identifier plus its pull-request number or equivalent change set. It is not an agent run, a correction push, or a CI rerun. Every submitted attempt has exactly one terminal disposition: merged-retained, rejected, abandoned, or reverted-after-merge. Correction pushes, workflow runs, and incidents remain linked events on that row, so a retry cannot create a second numerator or disappear from the denominator.
Review debt is the incremental active human effort caused by an agent-assisted change after handoff. Every active human person-minute is assigned once to one and only one category: implementation, review, clarification, correction, security-exception handling, or incident work. Implementation is active work on the initial proposed change; correction is active revision after feedback or failed evidence. Review covers diff and evidence inspection plus the approval decision; clarification covers requirement or context answers without changing code. Security-exception handling covers investigating, blocking, or approving a policy or permission exception, and incident work covers diagnosis, mitigation, and recovery after release.
Use the category that performs the action in that minute when an activity spans categories. For example, a reviewer who writes a test is recording correction, a security specialist who reads an ordinary diff is recording review, and a security specialist deciding an exception is recording security-exception handling. Split a minute only at a real change of activity; never record the same person-minute in two categories. This allocation rule keeps review debt person-minutes disjoint and auditable.
Agent cycle minutes means agent active/runtime minutes only: model or tool execution while the agent is working. Exclude human review, human clarification, human correction, security-exception handling, incident work, CI queue elapsed, CI runner minutes, and waiting. Record CI runner minutes and queue/wall-clock delay separately for both cohorts. They are automation consumption and elapsed-flow measures, not review-debt person-minutes; report their monetary cost separately if a financial decision needs it.
The basic formulas should be visible in the same dashboard as delivery metrics:
accepted change rate = merged-retained attempts / submitted attempts
review debt person-minutes = review + clarification + correction + security-exception handling + incident work
baseline active human person-minutes = implementation + review + clarification + correction + security-exception handling + incident work
net human person-minutes saved = baseline active human person-minutes - (treatment implementation person-minutes + treatment review debt person-minutes)The baseline and treatment use the same active-person-time boundary and the same separate automation-cost ledger. For a manual baseline, record implementation, review, clarification, correction, security-exception handling, incident work, CI runner minutes, queue/wall-clock delay, and any automation spend under those same definitions. For treatment, record the same human categories plus agent active/runtime minutes. Never assert that runtime minutes equal person-minutes: compare human capacity in the net-human-person-minutes formula, and compare model, CI, and other automation consumption in their own runtime or currency measures.
Use the formula per task class, not as a single team-wide average. A small test fixture, a dependency update, a production migration, and a permissions change have different acceptance bars and failure costs. A coding agent may save time on a narrow maintenance queue while creating negative net value on changes that require architecture or security context. That is a useful result because it defines an operational boundary.
Review debt is also not a claim that every agent slows every team. DORA’s 2025 research describes AI as an amplifier of the surrounding organizational system, while its aggregate findings are not a causal measurement for one repository. METR’s randomized study of experienced open-source developers using early-2025 tools found a slowdown in that bounded setting; its authors also describe the result as a snapshot rather than a universal forecast. Those findings justify measurement, not a predetermined conclusion.
Why generated code volume is a misleading metric
Generated lines, tasks started, time to first patch, and comments created are easy to collect. They are weak measures of useful delivery because they ignore whether a change was accepted, whether it created future work, and whose time was consumed. A team can double the number of proposed patches while reducing actual capacity if reviewers spend the recovered author time resolving ambiguity or if CI queues become the new bottleneck.
Google’s reviewer guidance makes this distinction concrete: review has to cover design, functionality, tests, complexity, documentation, and the broader system context. A reviewer who cannot understand a human-written block should request clarification; an agent label does not lower that threshold. Treat this as the cost model for every agent-generated pull request, especially when the patch crosses concurrent code, privacy boundaries, infrastructure, or an unfamiliar subsystem.
Benchmarks are necessary but insufficient. SWE-bench evaluates whether a model can produce a patch for a real-world GitHub issue in a reproducible harness. Its containerized evaluation process is valuable for comparing candidates, but it does not measure your code-owner routing, reviewer calibration, CI policy, on-call consequences, or customer impact. Keep benchmark results beside, not in place of, internal review-debt measurements.
Instrumentation: capture the handoffs, not just the merge
Start with the pull request and issue tracker; do not add a parallel surveillance system. Label an attempt as agent-assisted only when the agent materially authored or changed code. Record a task ID, task class, repository, risk tier, starting commit, owner, agent and model configuration, and the exact permission boundary. The label needs a human override because a developer may use an assistant for brainstorming without delegating implementation.
For each attempt, capture timestamps for assignment, first patch, review requested, first reviewer response, every correction push, CI start and completion, approval, merge, rollback, and incident close. Store active human person-minutes in the six exclusive categories, using a short self-reported range or sampled time log; do not infer person-time from elapsed calendar time. Record agent active/runtime minutes, CI runner minutes, and queue/wall-clock delay as separate fields. Also record reviewer count, code-owner involvement, required-check state, failed and rerun workflows, rejected attempts, revert links, and post-merge defects tied to the change. Redact prompts, source code, credentials, and customer data according to your existing retention policy.
GitHub can request code-owner review and can require approvals, status checks, and other conditions through protected branches or rulesets. Those controls are an enforcement boundary, not evidence that an agent’s analysis was correct. Capture the actual merge-box state for the head commit and distinguish a human approval, an advisory AI comment, a deterministic status check, a skipped workflow, and an administrator bypass. Otherwise a missing review can masquerade as a cheap review.
Instrument only the measures needed for a decision. A useful minimum schema has the immutable attempt row, a linked event row for review, clarification, correction, security exception, CI run, release, and incident events, and a separate cost ledger for agent runtime, CI runner consumption, and queue/wall-clock delay. Define attribution before the pilot: for example, count a defect when the post-merge investigation identifies the agent-authored change as a material contributor, and count a revert even if the code is later reintroduced in another form. A disputed attribution should remain visible as disputed rather than silently removed.
Scorecard and decision matrix
Review the scorecard weekly by task class, risk tier, repository, and agent configuration. Show raw counts and medians with ranges; small cohorts make precise-looking percentages fragile. Do not offset a security exception with fast completion. Security, ownership, and irreversible-data gates are non-compensating: a cohort with a critical breach fails its rollout criterion even if it improves throughput.
| Decision dimension | What to record | Healthy direction | Escalate or stop when |
|---|---|---|---|
| Task class | Repair, test, maintenance, feature, migration, or security work | Comparable work is segmented | Mixed work hides risk and effort |
| Acceptance | Merged-quality changes and rejected submissions | Higher accepted change rate | Acceptance falls or reviewers approve without evidence |
| Reviewer time | Minutes for reading, clarification, and final decision | Lower without weaker review | Review queue or specialist load rises |
| Defect escape | Confirmed post-merge defects and severity | No increase by risk class | A material defect lacks a safe response |
| Retry count | Agent correction pushes; CI runner and queue delay reported separately | Fewer retries per accepted change | Reruns crowd out normal CI work |
| Security exceptions | Policy, secret, permission, or boundary violations | Zero critical exceptions | Any critical exception or unowned exception occurs |
| Reversions | Reverts, hotfixes, and rollback minutes | No increase | Reversion or rollback rate rises materially |
| Ownership | Named maintainer and operational handoff | Every accepted change has an owner | Nobody can explain, test, or operate it |
Use both the formulas and the table. A team might see a high accepted change rate with low visible reviewer minutes, then discover that the agent’s correction loops are consuming CI capacity or that reviewers are deferring work until after merge. The scorecard makes those transfer costs discussable. It should never be used as a quota for reviewers to approve faster.
Cohort design: make the comparison fair enough to learn
Build paired cohorts from work that would otherwise enter the same queue. A practical starting split is low-risk tests and small maintenance tasks, with a matched manual baseline drawn from recent work in the same repository. Exclude emergency incidents, large rewrites, and tasks whose requirements changed materially after assignment; record exclusions and why. Random assignment is strongest when feasible, but a transparent matched comparison is better than an uncontrolled before-and-after chart.
Freeze the task definition, repository revision, allowed tools, budget, network policy, required checks, and reviewer standard before the agent starts. The coding-agent benchmark selection guide helps separate a public capability benchmark from an internal adoption cohort. For tasks that use a private evaluation set, keep the holdout out of prompts, vendor support requests, and agent memory so configuration is not tuned to the answer key.
Stratify rather than average away important differences. Compare review debt for test generation separately from defect repair, dependencies, migrations, and security-sensitive work. Ensure both cohorts include similar file counts, language areas, and risk tiers. If a single staff engineer reviews every agent patch, report that concentration; the pilot may be measuring a temporary expert shield rather than a scalable workflow.
The AI coding-agent evaluation guide provides the acceptance rubric and controlled task-set method. This guide adds the downstream labor measure. Join the two records by task ID so a failed benchmark attempt, a rejected PR, and a later rollback remain parts of one cycle rather than disappearing from the success metric.
Failure modes
Output-count optimism. A dashboard celebrates patches proposed or lines generated while rejected changes and reviewer correction remain outside the denominator. Use submitted changes and accepted changes together, then count every rejection and retry.
Approval theater. An AI review comment or a passing optional workflow is treated as approval. GitHub separates comments, approvals, and requested changes; repository rules determine what can merge. Preserve a named human owner and test the enforced controls on the actual target branch.
Review compression. Review minutes fall because reviewers skim large diffs or defer validation. Counter this with sampled independent review, post-merge defect tracking, and explicit evidence requirements from the AI PR review checklist. Less time is not automatically better.
Benchmark transfer. A strong SWE-bench or vendor result is announced as proof of business productivity. The benchmark’s task format and harness are bounded. Validate with representative repository work and report its limits.
Retry invisibility. The agent keeps revising until CI is green, but only the final attempt is counted. Keep all correction pushes, failed workflows, and tool costs attached to the original submission.
Ownership erosion. A maintainer merges a plausible patch but cannot explain its runtime behavior or rollback. Require a human owner for every accepted change; route sensitive code to qualified reviewers rather than distributing the risk across a general queue.
Confounded comparison. The agent cohort receives simpler tasks, better prompts, or faster reviewers. Preserve assignment rules and cohort exclusions, and report uncertainty instead of forcing a winner.
Validation protocol: a four-week pilot
Week 0 — prepare. Choose one repository and two low-to-medium-risk task classes. Write the acceptance rubric, define the baseline window, configure required reviews and deterministic checks, name the incident owner, and test the measurement pipeline with two manual historical changes. Announce that the pilot measures the workflow, not individual developer performance.
Week 1 — establish the baseline. Sample recent manual work matching the chosen classes. Capture implementation, review, clarification, correction, security-exception, and incident person-minutes under the same six-category rule; record CI runner minutes and queue/wall-clock delay separately. Capture accepted changes, defects, reverts, and known external waits. Review the coding-agent pricing brief alongside the separate automation ledger so model fees, CI runner costs, and platform costs are not disguised as human labor.
Weeks 2–3 — run constrained agent cohorts. Give the agent identical repository access, branch restrictions, test commands, time budget, and reviewer standard. Keep changes small and stop the attempt on an unapproved network action, secret exposure, unsafe permission expansion, destructive operation, or failed required control. Reviewers should use the same checklist for manual and agent-assisted work and record correction time without seeing a dashboard target.
Week 4 — adjudicate and decide. Compare the cohorts by task class. Report raw attempt counts, median and tail agent active/runtime minutes, accepted change rate, review-debt person-minutes, implementation person-minutes, defects, reversions, and exceptions. Report CI runner minutes and queue/wall-clock delay beside—not inside—those person-minute measures, then report monetary automation cost per accepted change separately. Read a sample of accepted patches after merge. Ask on-call and dependent teams whether waiting, support, or recovery labor moved outside the PR record.
Expand only if the selected task class shows a credible improvement in net human person-minutes saved, stable or better acceptance, no critical exception, no material defect or rollback increase, and durable human ownership. Keep the pilot constrained or stop it when the evidence is mixed. The decision can be “use the agent only for tests in this repository”; a narrow approved boundary is more valuable than a broad, unsupported rollout.
Recommendation
Adopt coding agents by measured task class, not by a blanket developer-productivity target. Start with a constrained cohort, retain existing code-owner and required-check controls, and publish the scorecard to the people who bear the review and on-call cost. Treat faster first patches as a hypothesis. The release decision should follow accepted work, review debt, safety exceptions, and ownership evidence.
For a practical comparison, GitHub Copilot, SWE-agent, and Promptfoo represent complementary surfaces: repository-native assistance, autonomous repository-task work, and repeatable model-behavior evaluation. Compare the approved set in the ToolVerse decision workspace. Do not use that comparison as a substitute for the internal four-week protocol; product capability does not establish your team’s net capacity.