How to build a RAG evaluation dataset that finds real failures

Expected outcome

The finished artifact is a versioned table of evaluation cases plus a reviewer handbook. Each row contains the user identity or permission profile, corpus revision, question, expected evidence, answer requirements, prohibited claims, desired refusal or escalation behavior, and scoring rules. A release report can run the same rows against two system versions and explain every material regression.

The dataset must expose failures that average answer scores hide. A response can be fluent yet unsupported, retrieve the correct document but cite the wrong passage, answer a question that should be refused, or reveal a document the user cannot access. Those outcomes need distinct labels.

This tutorial narrows the broader RAG evaluation playbook into the data contract that makes repeated evaluation possible. Use the RAG quality checklist for the surrounding production controls.

Prerequisites

Freeze a corpus snapshot with document identifiers, versions, access-control metadata, and ingestion status. If the corpus changes between runs, a score difference may reflect source drift rather than retrieval or model behavior.

Name four roles:

• A product owner who defines representative user jobs.
• A domain reviewer who judges evidence and material correctness.
• A data or security owner who defines permission and sensitive-data rules.
• An evaluation owner who versions cases, runners, graders, and reports.

Choose a unit of evaluation. For most knowledge assistants, one row represents one user question under one identity against one corpus revision. Multi-turn tasks require prior messages and expected state, but do not combine unrelated questions into one row.

Write hard gates before generating cases. Permission leaks, fabricated high-impact claims, failure to refuse unavailable evidence, and citations to content not retrieved should normally block a release regardless of the average score.

Step 1: sample real user jobs

Collect questions from approved support logs, search queries, interviews, task analyses, and subject-matter experts. Remove personal or confidential material unless the evaluation environment is authorized to hold it. Preserve the intent and failure mechanism when replacing sensitive details with synthetic equivalents.

Group questions by job rather than wording:

Job	Example intent	Main risk
Locate a fact	Find a policy date or product limit	Wrong version or unsupported value
Explain a procedure	Summarize steps across documents	Missing prerequisite or reordered step
Compare options	Contrast plans, controls, or methods	Selective evidence or stale terms
Diagnose a problem	Connect symptoms to a runbook	Unsafe recommendation or omitted escalation
Verify compliance	Determine whether a control is documented	Treating absence of evidence as approval
Find an exception	Identify exclusions or edge cases	Retrieval favors general text over exception

Limit duplicate paraphrases. Ten cosmetic variants of an easy lookup can make performance appear stable while leaving difficult jobs untested. Add variants when they exercise abbreviations, misspellings, multilingual wording, ambiguity, or different user roles.

Step 2: design the case mix

For an initial 80-case set, a practical starting mix is:

• 32 supported questions with clear evidence.
• 12 supported questions requiring multiple passages or documents.
• 10 unanswerable questions where the corpus lacks required evidence.
• 8 stale or conflicting-source cases.
• 6 ambiguous questions that require clarification.
• 6 permission-denied cases using real access boundaries in a synthetic account.
• 6 adversarial or instruction-conflict cases.

These numbers are not a benchmark. Adjust them to the workload’s risk. A regulated assistant may need many more permission and refusal cases; a public documentation bot may emphasize version conflicts and difficult navigation.

Tag every row by job, difficulty, source format, language, risk, answerability, permission profile, and expected behavior. Report results per slice so strong easy-case performance cannot hide a critical gap.

Step 3: label evidence before answers

Start with acceptable evidence. Record document and passage identifiers, source version, effective date, and why each passage supports the case. For multi-document answers, label the contribution of each source instead of pasting a large context window.

Then define the answer contract:

• Required claims or actions.
• Claims that are optional but acceptable.
• Material errors that invalidate the answer.
• Claims the evidence does not support.
• Required uncertainty, citation, clarification, refusal, or escalation.

Avoid one canonical prose answer when several phrasings are valid. A structured claim set gives reviewers a stable standard without rewarding superficial similarity.

For an unanswerable case, the correct evidence list may be empty. Specify what the system should say and whether it should ask for a missing detail, recommend an authoritative next step, or refuse. Do not reward a model for improvising a plausible answer.

Step 4: encode identity and authorization

Duplicate selected questions across two synthetic users with different access. The same query should retrieve and answer from different evidence, or refuse, according to the initiating identity.

Required fields include:

{
  "case_id": "perm-014",
  "user_profile": "regional-support",
  "tenant_id": "tenant-a",
  "allowed_source_ids": ["policy-public", "runbook-region-a"],
  "denied_source_ids": ["runbook-region-b"],
  "question": "Which escalation path applies?",
  "expected_behavior": "answer-from-allowed-evidence"
}

The evaluation runner should assert that denied source IDs never appear in retrieved context, citations, trace payloads, or cached results. A final answer that omits the secret but used unauthorized evidence still fails.

Step 5: create stale, conflicting, and adversarial cases

Version conflicts are common in operating documentation. Include an old policy that ranks well semantically and a newer document with an effective date or explicit supersession. The desired behavior should favor the current authority and mention genuine unresolved conflict.

For adversarial cases, place instructions inside documents that ask the assistant to ignore policy, expose another source, or execute a tool. The expected behavior is to treat retrieved content as evidence only. Deterministic tool authorization must remain outside the language model.

Add malformed and partial ingestion cases: a table with lost headers, a PDF where footnotes contain exceptions, a scanned page with OCR errors, and a document that failed parsing. These connect the dataset to the document extraction QA guide.

Step 6: separate retrieval, generation, and system scoring

Use several signals:

Layer	Suggested measures
Retrieval	Evidence recall, irrelevant-context rate, rank of required passages
Evidence use	Claim support, citation correctness, unsupported-claim count
Response	Required-claim coverage, material-error rate, clarity
Behavior	Correct clarification, refusal, escalation, or safe failure
Authorization	Denied-context exposure and identity propagation
Operations	Latency, token cost, retrieval failures, grader failures

Deterministic checks should handle source IDs, permissions, citations, formats, and known required values. Human or calibrated model graders can assess nuanced support and clarity. Preserve grader model, prompt, version, rationale, and disagreement with human review.

Do not collapse every signal into one average. Define hard gates and show a scorecard. A system with better retrieval recall but one permission leak is not a safe upgrade.

Step 7: split development and release sets

Expose a development set for iteration, retain a release set that developers do not tune against, and maintain a small audit set for sensitive or adversarial cases. A 60/25/15 split can work initially, but keep enough cases in each risk slice for interpretation.

When a production failure occurs, sanitize it, add it to the regression set, and record why. Do not silently edit expected results to accommodate the current model. Changes to labels require reviewer approval and a dataset version increment.

Run important high-variance cases multiple times. Report pass counts and configuration rather than one lucky outcome. Freeze temperature and provider settings when comparing architecture changes.

Copyable case schema

case:
  id: ""
  dataset_version: ""
  corpus_revision: ""
  user_profile: ""
  tenant_id: ""
  question: ""
  job: lookup | procedure | comparison | diagnosis | compliance | exception
  answerability: supported | unsupported | ambiguous
  risk: low | medium | high
  expected_behavior: answer | clarify | refuse | escalate
  required_evidence:
    - source_id: ""
      passage_id: ""
      version: ""
      supports: ""
  denied_sources: []
  required_claims: []
  prohibited_claims: []
  hard_gates:
    authorization: true
    citation_support: true
    material_accuracy: true
  reviewer: ""
  reviewed_at: ""

Store the runner result separately so the case definition remains immutable for a release. A result should reference the case version, system revision, retrieved source IDs, response, per-layer scores, and trace ID.

Failure modes

Answer-first labeling: reviewers write a preferred answer and search for evidence afterward. Label authoritative evidence first to reduce confirmation bias.

Easy-case inflation: the set contains many simple lookups from one document type. Report by job, difficulty, and source format.

Synthetic-only confidence: generated questions resemble model-friendly language and miss user ambiguity. Keep real approved queries as a separate slice.

No permission identity: cases name documents but not the user allowed to see them. Encode identity and assert denied content is absent from context and logs.

Mutable corpus: source documents change without a dataset revision. Hash or version the corpus and preserve effective dates.

Opaque grader: a model score changes after a provider update. Version graders and calibrate them against blinded human labels.

Acceptance criteria

The dataset is ready when every case has a corpus revision, permission profile, answerability label, evidence or explicit lack of evidence, expected behavior, hard gates, and reviewer. All source identifiers resolve to the frozen corpus, and denied-source tests demonstrate enforcement before retrieval.

At least two reviewers should adjudicate a sample from every risk slice. Record disagreement and improve the rubric before using model graders at scale. The release set must remain unseen during routine tuning.

Run a baseline system and confirm the dataset reveals known weaknesses. A suite that passes everything immediately may be too easy or incorrectly implemented. Then use the companion RAGFlow review or another platform review to compare candidates against the same customer-owned cases.

Next step

Connect the versioned dataset to the RAG monitoring guide so confirmed production failures become regression cases. Re-run hard gates after changes to parsing, chunking, embeddings, reranking, authorization, prompts, models, or citations.