LLM Consistency Testing asks which parts of an output must remain invariant across controlled repetitions and which variations are acceptable. It is not a demand for identical wording or a claim that one repeated answer proves reliability.
Fix the comparison conditions
Record a stable sample ID and the model, prompt, tools, data, temperature, seed where supported, and other sampling parameters. Use the repetition count specified by the project. Separate model, prompt, retrieval, and parameter versions instead of pooling them into one stability result.
| Comparison layer | What to define |
|---|---|
| Structural invariants | Schema, required fields, allowed enums, and policy constraints |
| Semantic invariants | Decisions, exclusions, dates, amounts, and source references that must stay stable |
| Permitted variation | Paraphrases, ordering, and other changes allowed by the rubric |
| Variance measure | Named numeric variable, unit, denominator, and calculation rule |
Check structural requirements first, then compare meaning with the supplied rubric. A wording difference rate is not a quality-variance measure. If no rubric defines material semantic change, keep that judgment UNKNOWN.
Analyze repetitions without inventing a baseline
Compare runs for the same sample and condition. Report the repetition denominator, structural failures, semantic failures, and intermittent failures per sample. Keep unknown or unassessable outputs visible instead of excluding them from the denominator.
A single observation establishes no repeated consistency. Matching seeds do not prove reproducibility across model versions. If the project has not specified a run count, variance formula, or acceptable threshold, record that gap and propose the smallest decision needed; do not supply arbitrary values.
Worked example: policy summaries
For a policy summarizer, the team may require deadlines, exclusions, and source references to remain unchanged while allowing sentence order or phrasing to vary. Compare repeated outputs against that contract, record every model and prompt version, and separate formatting failures from changes to the actual policy decision.
If an output changes an eligibility decision, record the affected invariant and source run. Do not call two summaries inconsistent merely because their wording differs, and do not infer stability from one matching pair.
Record an LCT-## finding
Identify the sample and condition group, invariant or comparison rule, source versions, repetition denominator, structural and semantic results, variance variable and formula if defined, evidence state, impact, owner role, close condition, and validation method. Keep each parameter/version group separate. A missing rubric leaves semantic consistency UNKNOWN; a missing run set leaves repeated behavior unassessed.
Related Skills
- LLM Evaluation Design defines cases and judgment contracts.
- LLM Testing covers broader model behavior checks.
- LLM Hallucination Testing checks claim support against evidence.
Install and invoke
Install the individual Skill:
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/llm-consistency-testing -g
Invoke llm-consistency-testing with the sample set, project run count, model and prompt versions, permitted variation, semantic rubric, and comparison conditions.
Source
Read the LLM Consistency Testing prompt before use. It defines the LCT-## contract, repeat denominators, parameter groups, and variance-evidence boundary.