LLM Testing - QA Skill Library

Why this Skill

It turns this Skill's method into a quality input that can be executed, reviewed, and reused.

Use this skill when you need to test LLM applications for task quality, robustness, safety, factuality, and version regressions.
avoid exact-string assertions alone
pin and record model settings
evaluate stochastic outputs with repetitions and distributions
Listing checks without preconditions, expected outcomes, or evidence.

When to use

Use this Skill

Use this skill when you need to test LLM applications for task quality, robustness, safety, factuality, and version regressions.
Use it to review an existing plan, result, or evidence set and produce actionable improvements.
Use it when context is incomplete but a bounded first pass is still valuable.

Common pitfalls

Listing checks without preconditions, expected outcomes, or evidence.
Marking everything high priority and avoiding tradeoffs.
Substituting tool names or generic theory for domain reasoning.
Refusing incomplete input, or pretending incomplete evidence supports certainty.

Input

Minimum Input

The current task scope, objective, and subject under review.

Recommended Input

Project goal
Test scope
Constraints

Optional Context

Relevant code or configuration
Historical results
Logs and metrics

Output

The output follows this Skill's method and makes facts, assumptions, risks, and next steps explicit.

Judge the output value before installing

01Read and follow prompts/llm-testing.md, including its input contract, execution rules, minimum coverage, and output order.
02Add only context that changes the decision: scope, environment, version, constraints, evidence, and success criteria.
03Audit the input, then separate confirmed facts, working assumptions, and open questions.
04Rank by risk and evidence strength, and produce an artifact that can be executed or reviewed directly.

View full output structure

05If information is missing, deliver a bounded first pass and state which conclusions remain unsupported.

How It Works

01Read and follow prompts/llm-testing.md, including its input contract, execution rules, minimum coverage, and output order.
02Add only context that changes the decision: scope, environment, version, constraints, evidence, and success criteria.
03Audit the input, then separate confirmed facts, working assumptions, and open questions.
04Rank by risk and evidence strength, and produce an artifact that can be executed or reviewed directly.
05If information is missing, deliver a bounded first pass and state which conclusions remain unsupported.

Install & Quick Start

Install command / SHELL

npx skills add \
  https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/llm-testing
  -g

llm-testing.prompt

@skill llm-testing

Using the current project context, produce an actionable result following this Skill.

Additional context:
[Paste project context or requirement]

---
name: llm-testing
description: Use this skill when you need to test LLM behavior, failure modes, and evidence-based quality boundaries; triggers include llm testing.
---

# LLM Testing

## When to Use

- Use this skill when you need to test LLM applications for task quality, robustness, safety, factuality, and version regressions.
- Use it to review an existing plan, result, or evidence set and produce actionable improvements.
- Use it when context is incomplete but a bounded first pass is still valuable.

## Output Format Options

- Default to Markdown for review, execution, and incremental refinement.
- When the user requests tables, CSV, JSON, or ticket fields, preserve risk, evidence, priority, and boundary information.
- For machine-consumed output, confirm the schema, enums, and required fields first.

## How to Use

1. Read and follow `prompts/llm-testing.md`, including its input contract, execution rules, minimum coverage, and output order.
2. Add only context that changes the decision: scope, environment, version, constraints, evidence, and success criteria.
3. Audit the input, then separate confirmed facts, working assumptions, and open questions.
4. Rank by risk and evidence strength, and produce an artifact that can be executed or reviewed directly.
5. If information is missing, deliver a bounded first pass and state which conclusions remain unsupported.

## Reference Files

- Always read `prompts/llm-testing.md`; it is the complete execution specification for this skill.
- For evaluation or regression, read `evals/eval.yaml` and the relevant cases under `evals/cases/`.
- Load `references/`, `examples/`, `scripts/`, or `output-formats.md` only when those directories exist and the task needs them.

## Core Constraints

- avoid exact-string assertions alone
- pin and record model settings
- evaluate stochastic outputs with repetitions and distributions
- Never invent system behavior, fields, data, metrics, or root causes absent from the evidence.
- Link important conclusions to evidence; mark unsupported conclusions as hypotheses with a verification method.
- Explain priority using business impact, likelihood, or detectability.

## Delivery Checklist

- [ ] Covered: instruction following, factuality, robustness, safety, bias, context length, multilingual behavior, stochasticity, cost and latency.
- [ ] Separated facts, assumptions, gaps, and recommendations.
- [ ] Gave high-risk items a priority, evidence basis, owner or next action.
- [ ] Defined verifiable decision criteria instead of generic advice.
- [ ] Performed no unauthorized production writes or destructive actions.

## Common Pitfalls

- Listing checks without preconditions, expected outcomes, or evidence.
- Marking everything high priority and avoiding tradeoffs.
- Substituting tool names or generic theory for domain reasoning.
- Refusing incomplete input, or pretending incomplete evidence supports certainty.

## Best Practices

- Start with paths most likely to cause business loss, safety issues, or release blockage.
- Reduce uncertainty through the smallest verifiable experiment and record reproduction conditions.
- Make the artifact executable and independently reviewable by another engineer.