Prompt evaluation playbook for production AI workflows

Quick answer

Version prompts, datasets, graders, model settings, and release thresholds together. A useful evaluation asks whether a proposed change improves the real workflow, not whether several hand-picked answers look better. This playbook is for product and engineering teams that need a repeatable approval gate before changing prompts, models, retrieval, or tools.

Design the dataset

Start with 50–200 cases drawn from production-like work:

• ordinary inputs weighted by actual frequency;
• boundary conditions, missing context, and conflicting instructions;
• sensitive or disallowed requests that require refusal or escalation;
• tool failures and stale retrieval results;
• a private holdout set to detect evaluator overfitting.

Each case needs an expected behavior, not always an exact answer. Define acceptable evidence, prohibited claims, required fields, allowed tools, and escalation rules.

Use several kinds of checks

Check	Best use	Limitation
Exact or schema assertion	IDs, JSON, citations, required fields	Misses semantic quality
Reference rubric	Known-answer tasks	Can punish valid alternatives
Model grader	High-volume semantic comparison	Needs calibration and bias checks
Blinded human review	Tone, usefulness, consequential judgment	Slower and more expensive
Runtime trace check	Tool choice, permission, and action sequence	Requires instrumentation

Calibrate automated graders against a human-reviewed sample. A grader that agrees only on easy cases is not a release authority.

Release scorecard

Track task success, critical failure count, unsupported claims, refusal precision, p95 latency, token cost, and reviewer preference. Define non-compensating gates: no average quality gain should offset a new secret leak, unsafe action, or broken high-value workflow.

Example: a shorter prompt that costs more

A support team shortens its system prompt and sees faster first-token latency. The new version retrieves twice as many documents and increases total cost while citing outdated policy in 3% of cases. A polished answer sample misses both regressions. The evaluation catches them because retrieval freshness and cost are first-class metrics.

Implementation path

1. Store the current prompt and model configuration as the baseline.
2. Add observed failures before tuning the prompt.
3. Run candidates with fixed seeds where supported and repeated trials where variability matters.
4. Review disagreements between deterministic checks, graders, and humans.
5. Publish a compact diff: gains, regressions, unresolved risks, and rollback version.
6. Monitor post-release failures and feed adjudicated examples back into the dataset.

Tool selection

Promptfoo is a practical open-source candidate for matrix tests and red-team cases. Ragas is relevant when retrieval quality is central, while Guardrails AI can support structured validation. Compare broader options through the workflows hub.

Risks and limits

Evaluation data can contain personal or confidential information; minimize, mask, and govern it like production data. Model graders can reward their own style, and public test sets can become training data. Keep human adjudication, a private holdout, and a rollback path.

Rubrics and system-level tests

Separate correctness, completeness, grounding, style, safety, and action behavior. Define observable anchors: a grounded answer cites a source that supports the claim; a complete answer covers required decision factors; a safe tool-using answer stays inside the allowlist and stops when approval is missing. A blended quality score can let fluent prose conceal unsupported claims, so critical safety and schema checks should stand alone.

Prompt behavior depends on retrieval, tools, and output parsing. Store the retrieval query, source identifiers, tool definitions, tool results, and final answer for each case. Add fixtures for unavailable tools, permission denials, contradictory sources, expired policies, malformed responses, and hostile instructions inside retrieved content. The expected result may be a bounded retry, clarification, or escalation rather than an answer.

Use deterministic assertions for schema validity, required citations, prohibited tokens, tool allowlists, latency, and cost. Calibrate semantic graders against independently reviewed examples, especially unsafe and ambiguous cases. Track a grader’s false passes on critical failures separately from overall agreement. Explanations help debugging but do not prove the grader’s judgment is faithful.

Release operation

Run baseline and candidate in the same infrastructure window and report paired differences for each case. Repeat variable cases and compare distributions. Freeze the winning prompt, model, tool schema, retrieval configuration, parser, dataset, and thresholds as one release record. Rolling back only prompt text may not restore known behavior.

After release, sample production outcomes with the same rubric and compare them with evaluation predictions. Segment results by language, customer tier, document type, and action class, because a stable aggregate can hide a serious regression in a smaller group. Add adjudicated incidents and high-cost reviewer corrections to the dataset.

Evaluation cases may contain personal or confidential information. Minimize and mask inputs, control access, set retention, and keep them out of ordinary logs. Name owners for datasets, thresholds, graders, monitoring, and rollback. A model or vendor change should not reach production merely because the API remains compatible.

Diagnose disagreements

When baseline and candidate trade wins, inspect cases rather than trusting the aggregate. A shorter answer may lose with a verbosity-biased grader while users prefer it. A cautious version may refuse too often on low-risk work but correctly stop a dangerous action. Label each disagreement as requirement ambiguity, grader error, expected tradeoff, or real regression.

Maintain an adjudication log with case, decision, rationale, and rubric change. Re-run both versions after changing a grader so the measurement itself does not silently move. Use cheap deterministic checks first, semantic grading second, and targeted expert review for high-value or uncertain cases. This keeps the gate affordable enough for CI without delegating consequential judgment to one model.

Publish the unresolved cases with the release decision. Product owners can then see which behavior remains uncertain, which segment is affected, and what monitoring or human review compensates for it. Hiding disagreement behind a pass percentage makes later incidents harder to diagnose.

Decision

Ship only when the candidate clears critical gates and improves the metrics that matter for the workflow. Record the evidence so the next model or prompt change starts from a known baseline.