A fluent RAG answer can still be wrong, incomplete, or unsupported by the retrieved documents. Quality review starts by checking what the answer claims and what evidence actually supports it.
Why this Skill is needed
The RAG Quality Testing Skill evaluates the answer layer of a retrieval-augmented system. It separates grounding, relevance, completeness, citation support, and abstention so a polished response does not hide a missing or contradictory source.
What the Skill does
The key question is whether the answer says only what the available evidence can support.
Use this Skill to review an Agent, RAG, or LLM result against supplied context and expected behavior. It is evidence-bounded analysis, not a claim that a small set of examples proves model safety.
A good review links each answer judgment to the question, retrieved context, cited source, expected answer or abstention rule, evidence state, and next validation action.
Use it when an internal knowledge assistant answers policy, support, or operational questions and the team needs to know whether the answer is supported, not merely helpful-sounding.
What it does not do
The RAG Quality Testing Skill can organize material, evidence, and next actions for Grounding. It does not replace:
- Confirmation of rules, scope, and risk by the accountable domain owner.
- A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
- Authorization for release, compliance, production actions, or residual-risk acceptance.
- A Human decision when supplied sources conflict.
What it checks
The value of this Skill is not another keyword list. It connects each focus area to an observable input, a judgment, and a way to close the loop. Start with a small matrix based on the source Skill’s output contract:
| Focus | Question before analysis | Handoff output |
|---|---|---|
| Grounding | Whether claims are entailed by the supplied context | Claim-to-source evidence |
| Relevance and completeness | Whether the answer addresses the question without material omissions | Question and expected behavior |
| Citation support | Whether citations support the exact claim and version | Span, source, freshness |
| Abstention | Whether missing support triggers refusal or clarification | Policy and observed output |
If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.
Audit inputs before you start
Before analyzing Grounding, classify the input into six evidence states. A gap is not automatically a failure, but it must not disappear inside the conclusion.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.
Keep the decision layers separate
For Grounding, do not compress four different kinds of language into “recommended to pass”:
| Layer | How to write it | Application here |
|---|---|---|
| Fact | What the supplied material directly shows | Cite the source, version, input, or run record for the focus |
| Evidence-backed Inference | What several facts support together | Show the inference chain and retain uncertainty |
| Recommendation | The smallest next action | Name the evidence, review, execution, or regression path |
| Human Decision | What an accountable person must decide | Leave scope, risk acceptance, resources, and release meaning to the owner |
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Question set | Policy questions covering direct answers, missing evidence, and conflicting documents | Mark coverage gaps instead of inventing expected answers |
| Retrieved context | Document IDs, versions, chunks, metadata, and citation spans | Keep grounding unassessed without actual context |
| Expected behavior | Answer key, citation rule, completeness expectation, and abstention policy | Ask for the missing decision rule |
| Runtime evidence | Prompt, model version, output, trace, and evaluator notes | Stay in static review without a run artifact |
Use a request like this:
Use the rag-quality-testing Skill.
Task: Review an HR policy assistant's answers about parental leave, travel reimbursement, and an unsupported benefits question.
Inputs: [questions, retrieved chunks, document versions, model outputs, citations, expected behavior]
Scope: [policy corpus, model version, evaluation window]
Constraints: [masked employee data, no production writes]
Assess grounding, relevance, completeness, citation support, and abstention separately. Link findings to answer-level evidence and label unsupported conclusions pending or unassessed.Analysis
Use the input, matrix, and evidence state to form the judgment before writing the Finding; keep missing material as a gap.
Finding
Focused example: Review a policy answer that sounds confident
The assistant says “employees may carry over 10 days” and cites a handbook chunk that only covers annual-leave accrual. The Skill should flag the carry-over claim as unsupported, distinguish citation presence from citation support, and ask for the governing policy version. A benefit absent from the corpus should produce a pending or abstention finding, not an invented answer.
This review classifies supplied outputs against supplied evidence. It cannot prove general model behavior, production safety, or a pass rate beyond the evaluated question set and version.
Example finding: turn one problem into a handoff
The field example below shows the recording pattern; it is not an execution result.
If the supplied material cannot prove that Grounding meets its contract, write the finding like this. It does not invent the missing rule or turn missing evidence into a failure.
| Field | Example wording |
|---|---|
| Source and scope | Record the requirement, version, environment, and the concrete object for Grounding |
| Finding | The condition or result for Grounding is not yet traceable to evidence |
| Evidence state | missing / assumptions; use conflicting when sources disagree |
| Impact and priority | Name the affected user, journey, or delivery decision without inflating severity |
| Owner and Human decision | Ask the product, engineering, security, or test owner to confirm the rule and trade-off |
| Action and close condition | Add the smallest missing evidence; close only when source, judgment, and owner can be reviewed |
| Validation | Name one repeatable check, query, or run and retain the raw artifact |
The point is to let the next person walk from the finding back to the source and run an action that can change the decision.
Decision
The accountable owner confirms the decision question and risk trade-off; the Skill does not make that choice.
Validation
Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.
How a Finding enters the next stage
Handoff output
| Output field | Why it exists | Example status |
|---|---|---|
| Grounding | Whether claims are entailed by the supplied context | Claim-to-source evidence |
| Relevance and completeness | Whether the answer addresses the question without material omissions | Question and expected behavior |
| Citation support | Whether citations support the exact claim and version | Span, source, freshness |
| Abstention | Whether missing support triggers refusal or clarification | Policy and observed output |
Every conclusion should point to a source, evidence state, and next action. If evidence is missing, use pending, blocked, unassessed, or NOT_SCORED instead of filling the gap with confidence.
Next-stage route
At minimum, hand off the source, evidence state, owner, close condition, and validation action; the next-stage conclusion remains bounded by the evidence state.
How to prepare better input
Provide paired question, retrieved context, answer, citation, and expected-behavior records. Include negative questions where the corpus is silent and conflict cases between document versions; otherwise abstention and freshness cannot be assessed.
A useful handoff includes input versions, scope, time window, evidence index, assumptions, Human decision boundary, and the smallest validation action.
Working with other Skills
- RAG Retrieval Testing:checks whether the right evidence reached the answer layer.
- LLM Hallucination Testing:explores unsupported claims and invented details.
- AI Safety Testing:adds harm and escalation boundaries for high-impact outputs.
Common traps
- Scoring citation presence as grounding without checking claim-level support.
- Treating a relevant paragraph as a complete answer when an exception or effective date is missing.
- Using a small set of successful answers to claim broad RAG reliability.
Install and invoke
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/rag-quality-testing -gUse the rag-quality-testing Skill.
Include objective, scope, versions, evidence paths, constraints, and decision boundary.
Audit inputs first, preserve evidence states, and finish with owner, close condition, and validation method.FAQ
Can I review answers without an expected answer key?
Yes, but mark completeness and correctness unassessed where no decision rule or authoritative source exists. Preserve a question for the domain owner.
Does a citation automatically make an answer grounded?
No. Check claim entailment, source applicability, version, and whether the citation covers the whole claim.
The Skill is most useful when attached to one real project artifact and kept with its source evidence. Start narrow, validate the uncertain part, and expand only when evidence supports it.
References
Source Skill and execution contract
The complete execution contract lives in RAG Quality Testing prompt. The source directory may also contain evaluation cases and supporting material.
Static plans, file presence, and dry runs keep their evidence state. They do not become runtime proof, an all-passed claim, or release approval.
Reference links
- RAG Quality Testing prompt:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/rag-quality-testing/prompts/rag-quality-testing.md
- RAG Quality Testing Skill source:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/rag-quality-testing
- RAG Quality Testing details:https://inaodeng.com/en/qaskills/rag-quality-testing/
- Awesome QA Skills on GitHub:https://github.com/naodeng/awesome-qa-skills