Reliability is not a claim that a service “usually works.” It is a relationship between an availability objective, realistic failure modes, exposure windows, and evidence that can be checked.
Why this Skill is needed
The Reliability Testing Skill prepares an evidence-bounded view of service reliability. It models failure modes and exposure without treating an architecture diagram or an incident-free week as proof.
What the Skill does
The review should identify which failure to isolate and what evidence could change the reliability judgment. Map each important failure mode to expected behavior, affected exposure, evidence source, validation method, owner, and stop condition.
Use the Skill to assess availability objectives, dependency failures, degraded modes, exposure windows, and evidence readiness. It is useful when planning a resilience exercise, after a dependency or traffic change, or when an SLO lacks a clear validation path. It does not inject faults or access production systems by itself.
What it does not do
The Reliability Testing Skill helps organize evidence and next steps for a reliability review. It does not replace:
- Confirmation of rules, scope, and risk by the accountable domain owner.
- A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
- Authorization for release, compliance, production actions, or residual-risk acceptance.
- A Human decision when supplied sources conflict.
What it checks
The value of this Skill is not another keyword list. It connects each focus area to an observable input, a judgment, and a way to close the loop. Start with a small matrix based on the source Skill’s output contract:
| Focus | Question before analysis | Handoff output |
|---|---|---|
| reliability objective | Expected availability or degraded behavior | Source and measurement window |
| Failure-mode judgment | Expected behavior under one failure | Scenario and evidence state |
| Exposure analysis | Who or what is affected and for how long | Blast radius and unknowns |
| Evidence plan | Metric, trace, or exercise that reduces uncertainty | Owner and stop condition |
If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.
Define the SLI and observation window first
Before comparing an availability objective, record the project’s SLI event definition, success/failure conditions, denominator, exclusions, measurement window, and data source. Request success rate and time-based availability are different measures; do not substitute or combine them.
| Metric type | Evidence to retain | Review focus |
|---|---|---|
| Request-based SLI | Successful requests, eligible total requests, and explicit exclusions | Apply the defined treatment of retries, duplicate events, and missing records; state numerator and denominator |
| Time-based SLI | Expected observation time, available time, and missing intervals | Explain sampling and missing-data treatment; do not replace time with request counts |
| Failure timeline | Start/end, duration, recurrence, detection, and recovery records | State whether the window covers the target load; do not extrapolate long-term reliability from a short window |
If definitions, sources, or windows are incompatible, report them separately instead of merging them into one conclusion.
Audit inputs before you start
Before analyzing reliability evidence, classify the input into six evidence states. A gap is not automatically a failure, but it must not disappear inside the conclusion.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.
Keep the decision layers separate
In a reliability review, do not compress four different kinds of language into “recommended to pass”:
| Layer | How to write it | Application here |
|---|---|---|
| Fact | What the supplied material directly shows | Cite the source, version, input, or run record for the focus |
| Evidence-backed inference | What several facts support together | Show the inference chain and retain uncertainty |
| Recommendation | The smallest next action | Name the evidence, review, execution, or regression path |
| Human Decision | What an accountable person must decide | Leave scope, risk acceptance, resources, and release meaning to the owner |
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Availability objective | SLO or business objective, service boundary, SLI definition, and measurement window | Mark unsupported targets or definitions as unconfirmed |
| Failure modes | Queue delay, provider outage, storage saturation, or dependency timeout | Avoid generic system-down labels |
| Exposure window | Users, operations, regions, and duration affected | Mark unknown blast radius unassessed |
| Evidence readiness | Metrics, logs, traces, alerts, runbooks, and safe test environment | Separate design and execution evidence |
Use a request like this:
Use the reliability-testing Skill.
Task: Prepare a reliability review for a notification service depending on a queue and a third-party email provider.
Inputs: [SLO, dependency map, failure modes, traffic window, metrics, alerts, runbooks]
Scope: [authorized staging design and supplied/redacted production evidence review (read-only)]
Constraints: [no production fault injection, provider data masked]
Cover availability objectives, failure modes, exposure windows, and evidence readiness. State pending, blocked, or unassessed items and give the smallest isolated validation method.
Analysis
Use the input, matrix, and evidence state to form the judgment before writing the Finding; keep missing material as a gap.
Finding
Focused example: Review provider outage behavior without inventing availability
Illustrative scenario: assume the email-provider contract says requests may time out and the notification-service design mentions retries, but neither explains queue backpressure or user-visible status. Mark timeout and retry as supported by the supplied material, the queue and status behavior as missing, and request evidence for the queue, alert, and exposure window. One successful staging smoke test covers one run; it does not establish an availability percentage.
The review classifies objectives and readiness from supplied material. It cannot prove real availability, fault containment, or recovery behavior until authorized validation produces runtime evidence.
Example finding: turn one problem into a handoff
The field example below shows the recording pattern; it is not an execution result.
If the supplied material cannot show whether the stated reliability objective is met, write the finding like this. It does not invent the missing rule or turn missing evidence into a failure.
| Field | Example wording |
|---|---|
| Source and scope | Record the requirement, version, environment, and the specific availability objective |
| Finding | The objective’s condition or result is not yet traceable to evidence |
| Evidence state | missing / assumptions; use conflicting when sources disagree |
| Impact and priority | Name the affected user, journey, or delivery decision without inflating severity |
| Owner and Human decision | Ask the product, engineering, security, or test owner to confirm the rule and trade-off |
| Action and close condition | Add the smallest missing evidence; close only when source, judgment, and owner can be reviewed |
| Validation | Name one repeatable check, query, or run and retain the raw artifact |
The point is to let the next person walk from the finding back to the source and run an action that can change the decision.
Decision
The accountable owner confirms the decision question and risk trade-off; the Skill does not make that choice.
Validation
Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.
How a Finding enters the next stage
Handoff output
| Output field | Why it exists | Example status |
|---|---|---|
| reliability objective | Expected availability or degraded behavior | Source and measurement window |
| Failure-mode judgment | Expected behavior under one failure | Scenario and evidence state |
| Exposure analysis | Who or what is affected and for how long | Blast radius and unknowns |
| Evidence plan | Metric, trace, or exercise that reduces uncertainty | Owner and stop condition |
Every conclusion should point to a source, evidence state, and next action. If evidence is missing, use unconfirmed, BLOCKED, UNASSESSED, or NOT_SCORED instead of filling the gap with confidence.
Next-stage route
At minimum, hand off the source, evidence state, owner, close condition, and validation action; the next-stage conclusion remains bounded by the evidence state.
How to prepare better input
Provide the objective source, dependency and failure-mode map, exposure window, SLI definition and denominator, measurement window, alert and runbook references, environment, and data constraints. Without an approved objective, write an evidence gap rather than inventing an SLO.
A useful handoff includes input versions, scope, time window, evidence index, assumptions, Human decision boundary, and the smallest validation action.
Working with other Skills
- Recovery Testing: https://inaodeng.com/en/qaskills/recovery-testing/ — checks behavior after service or data restoration.
- Chaos Testing: https://inaodeng.com/en/qaskills/chaos-testing/ — designs controlled failure experiments.
- Observability Design Review: https://inaodeng.com/en/qaskills/observability-design-review/ — checks whether reliability evidence can be observed.
Common traps
- Treating uptime history as proof that unobserved dependency failures are handled.
- Describing a failure mode without user exposure, degraded behavior, or alert.
- Running broad fault injection before defining an isolated scenario and stop condition.
Install and invoke
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en --skill reliability-testing
Use the reliability-testing Skill.
Include objective, scope, versions, evidence paths, constraints, and decision boundary.
Audit inputs first, preserve evidence states, and finish with owner, close condition, and validation method.
FAQ
Can the Skill assign an SLO?
No. It exposes missing objectives, measurement windows, and validation paths. Service and business owners decide the target.
Is an incident-free period reliability evidence?
It is one observation with limited coverage. Keep its window and failure exposure visible; it does not prove failures that did not occur.
Does request success rate equal time-based availability?
No. Calculate each measure from its defined event, denominator, and observation window. If the definitions differ, report them separately instead of substituting one for the other.
The Skill is most useful when attached to one real project artifact and kept with its source evidence. Start narrow, validate the uncertain part, and expand only when evidence supports it.
References
Source Skill and execution contract
The complete execution contract lives in the Reliability Testing prompt: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/reliability-testing/prompts/reliability-testing.md. The source directory: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/reliability-testing may also contain evaluation cases and supporting material.
Static plans, file presence, and dry runs keep their evidence state. They do not become runtime proof, an all-passed claim, or release approval.