A fluent answer can contain a claim that no supplied source supports. LLM Hallucination Testing checks each actionable claim against its evidence and distinguishes contradiction, missing support, and evidence that cannot be assessed.
Build a claim-to-source ledger
Split an answer into independently checkable claims, including numbers and units, entities, conditions, dates, and citations. Preserve the exact answer text, evidence version, and source location so another reviewer can reproduce the judgment.
| Status | Use it when |
|---|---|
| SUPPORTED | Evidence supports the claim with the same scope and conditions. |
| CONTRADICTED | Supplied evidence explicitly says something different. |
| MISSING_SOURCE | No supplied evidence supports the claim, and no direct contradiction is available. |
| UNASSESSABLE | Inaccessible, unclear-scope, or outdated evidence prevents a valid comparison. |
Missing support alone does not prove that a claim was fabricated. Keep mixed answers at claim level instead of assigning one status to an entire paragraph.
Check citations and abstention separately
For every citation, check that it exists, can be accessed, is relevant, and supports the specific claim. A genuine handbook can still contradict the answer’s date or eligibility statement. Record citation failure separately from claim support.
When evidence is missing or ambiguous, review whether the answer states its uncertainty, narrows its scope, or abstains and asks for an authorized source. Do not treat a confident tone, a nearby citation, or a plausible answer as evidence.
Worked example: a policy answer
Suppose a policy assistant answers whether contractors qualify for a benefit, while the supplied handbook only describes full-time employees. Split the answer into its eligibility claims, compare each with the handbook’s actual scope, and preserve unsupported cells as MISSING_SOURCE or UNASSESSABLE according to the available evidence. The review should recommend a source request or safe abstention; it should not infer the missing policy.
Report supported claims and unknown claims with their own denominators. Without a suitable reference set and representative sample, provide a validation plan rather than a factual-accuracy rate.
Record an LHT-## finding
Identify the claim, exact answer span, evidence version and location, citation status, claim status, impact, priority, owner role, close condition, and validation method. Separate facts, evidence-backed inferences, recommendations, and Human decisions. Static review is not proof of model behavior across a live population.
Related Skills
- RAG Retrieval Testing checks retrieval coverage and source selection.
- RAG Quality Testing reviews the wider retrieval-augmented answer path.
- LLM Evaluation Design defines evaluation cases and judgment contracts.
Install and invoke
Install the individual Skill:
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/llm-hallucination-testing -g
Invoke llm-hallucination-testing with the answer set, source corpus and versions, claim scope, citation rules, and abstention requirements.
Source
Read the LLM Hallucination Testing prompt before use. It defines the LHT-## finding contract and the four evidence labels used for claim comparison.