How to evaluate AI coding agents with repository tasks

Quick answer

Build a small, versioned repository benchmark that represents the work you might actually delegate. Run candidates with identical permissions and budgets, then score accepted outcomes rather than self-reported completion. This guide is for engineering leaders choosing whether a coding agent belongs in issue triage, maintenance, testing, refactoring, or feature work.

Build the task set

Use 20–40 tasks across four classes:

• Repair: reproduce and fix a known defect with a regression test.
• Maintenance: update a dependency or API without unrelated churn.
• Test: add meaningful coverage for an existing behavior.
• Feature: implement a bounded change with acceptance criteria.

Include ambiguous requirements, stale documentation, permission boundaries, and at least five tasks the agent should decline or escalate. Freeze the starting commit and expected environment so results remain comparable.

Score outcomes

Metric	Calculation	Why it matters
Accepted completion	Merged-quality solutions / attempted tasks	Filters out confident but unusable outputs
Escaped defects	Defects found after evaluator acceptance	Exposes weak review rubrics
Reviewer minutes	Human review and repair time per accepted task	Captures shifted labor
Boundary violations	Unauthorized files, commands, network calls, or secrets	Tests safe operation
Total task cost	Model, compute, CI, sandbox, and reviewer cost	Supports a buying decision

A useful decision rule is task-class specific. An agent may be excellent at test scaffolding and poor at migrations; a single blended score hides that difference.

Run a fair evaluation

1. Give every candidate the same repository snapshot, task text, tools, time limit, and network policy.
2. Isolate execution and provide only the minimum credentials needed.
3. Capture commands, changed files, test output, time, and usage cost.
4. Have a reviewer who did not operate the agent apply the AI PR checklist.
5. Re-run accepted tasks after model, prompt, tool, or repository-policy changes.

Public benchmarks such as SWE-bench help with external comparison, but your repository set answers the adoption question. A leaderboard cannot measure your architecture, review conventions, or cost of failure.

Example: two agents, different value

Agent A solves 16 of 20 tasks but needs 22 reviewer minutes per accepted patch. Agent B solves 13 but needs 7 minutes and never crosses a forbidden path. For a constrained maintenance queue, Agent B may create more capacity despite the lower raw completion rate. Report the tradeoff rather than forcing a universal winner.

Selection path

Use Promptfoo when prompts and model behavior need repeatable evaluation, and review the agents tag or automation category when comparing orchestration layers. The commercial comparison in best AI coding agent tools maps tool types to evaluation, review, and rollout jobs. When automated review is part of the candidate workflow, apply the AI PR review tools comparison to the same historical task set rather than trusting vendor demos.

Risks and limits

Repository benchmarks can overfit if tasks leak into prompts or remain unchanged for too long. Rotate a private holdout set, record failed attempts, and review the evaluator itself. Never let high completion compensate for secret exposure, destructive commands, or bypassed approvals.

Rubrics, environment, and failure analysis

Write acceptance rubrics before seeing candidate output. A repair task may require reproduction of the original failure, a minimal fix, a regression test that fails on the starting commit, and no unrelated changes. A maintenance task may require compatibility with supported versions, lockfile integrity, release-note evidence, and unchanged public behavior. Map every feature criterion to a test or review observation.

Critical security gates are non-compensating. Accessing a forbidden credential, disabling a protection, or performing an unapproved external write fails the task even if the feature works. Use weighted quality scores only after those gates pass. Keep reviewer notes with the score so future teams understand why an output was accepted.

Create a disposable workspace for every attempt and start from the same commit. Pin tool versions, record model identifiers and settings, and state the network policy. Use synthetic credentials or a restricted test account. Capture commands, changed files, tool approvals, and network destinations, but redact secrets. Observing only the final diff cannot distinguish a safe process from one that happened to end safely.

Classify failures by misunderstood requirement, missing context, invalid tool use, environment problem, weak test, unsafe action, or reviewer disagreement. Count every attempt; rerunning until success without earlier failures inflates completion. Compare failure distributions by task class and use the diagnosis to improve repository documentation, permissions, deterministic tooling, or task scope.

From evaluation to rollout policy

Document which task classes the agent may attempt, allowed repositories and branches, commands and destinations, required approvals, and stop conditions. Permission can allow test generation while prohibiting migrations, or allow issue triage while requiring a human to initiate every patch. This is more useful than a universal enabled switch.

Publish median and tail results: accepted completion by task class, reviewer time, critical failures, p95 runtime, and cost per accepted patch. Include the manual baseline and acknowledge uncertainty when the sample is small. Add the coding-agent review-debt scorecard so correction loops, CI reruns, defect escapes, and rollback work remain attached to the task rather than becoming invisible reviewer labor. A finding that the agent creates value only on two narrow queues is a successful evaluation because it produces a defensible boundary.

Rotate a private holdout set and trigger re-evaluation after model updates, new tools, expanded permissions, material repository changes, or a serious incident. Early rollouts should be reviewed monthly until outcomes and incident rates stabilize.

Sample-size and reviewer controls

Do not claim a winner from a handful of tasks. Report raw counts and segment results by difficulty so easy documentation changes do not dominate complex repairs. Preserve failed patches for audit. Use two calibrated maintainers on a sample, hide candidate identity where practical, and measure agreement. Resolve disagreement by improving rubric anchors instead of averaging incompatible judgments.

Reserve tasks that operators have not seen. A repeatedly tuned set measures familiarity with the harness as much as generalization. Refresh the holdout from real adjudicated work and prevent it from entering prompts, vendor support, or training examples. Assisted and autonomous performance should be reported separately because extra human steering changes both cost and safety.

Decision

Approve a limited rollout only for task classes with higher accepted throughput, bounded reviewer cost, and zero critical boundary violations. Keep other work manual until evidence changes.