An error message can be technically accurate and still leave the user with no safe next action. A stack trace can help an operator and expose a secret at the same time.
Why this Skill is needed
Error Handling Design Review checks whether each designed failure path has an evidence-backed behavior before implementation: how it is classified, propagated, exposed to callers, observed, and recovered.
What the Skill does
Use this Skill before implementation to review failure taxonomy, caller action, retryability, fallback, observability, privacy, data consistency, and recovery across distinct failure modes. It prepares findings and validation actions; it does not run fault injection or prove runtime behavior. It is especially useful when inputs are incomplete or conflicting but a bounded review is still needed, or a handoff must identify what remains unverified.
Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills organizes Skills by language and testing stage. The series overview: https://inaodeng.com/en/blog/ai-testing/introduction_of_awesome_qa_skills/ explains the shared installation model; this guide stays with Error Handling Design Review.
A useful review follows one failure across three contracts: the caller gets a safe and stable result, the operator gets enough privacy-safe evidence to diagnose it, and the system preserves data consistency or names the required recovery decision. Apply it to paths such as validation rejection, authorization denial, timeout, dependency outage, rate limiting, duplicate requests, transaction conflict, partial success, or corrupted data.
What it does not do
The Error Handling Design Review Skill can organize failure-mode evidence and next actions. It does not replace:
- Confirmation of rules, scope, and risk by the accountable domain owner.
- A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
- Authorization for release, compliance, production actions, or residual-risk acceptance.
- A Human decision when supplied sources conflict.
What it checks
Connect each failure path to an observable input, a judgment, and a closeable next action. Start with a small matrix based on the source Skill’s output contract:
| Failure path | Review questions | Finding evidence |
|---|---|---|
| Validation, authorization, and rate limit | What triggers the path, what boundary applies, and what can the caller do? | Contracted rejection and caller-visible result |
| Timeout, dependency outage, and transaction conflict | Is retry safe, and can the original action have committed? | Retry/backoff or human-intervention rule, propagation, and data consistency |
| Duplicate request, partial success, and corruption | What state may already have changed? | Idempotency, compensation, fallback, or safe-stop decision |
| Breaker, degradation, and handoff | What stops, degrades, or moves to an operator? | Owner, close condition, and recovery evidence |
| Telemetry and privacy | What can diagnose the failure without exposing secrets or payloads? | Required log/metric/trace fields and privacy boundary |
If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.
Audit inputs before you start
Before analyzing a failure path, classify the input into six evidence states. A gap is not automatically a failure, but it must not disappear inside the conclusion.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.
Keep the decision layers separate
For each failure mode, separate facts, evidence-backed inferences, recommendations, and Human decisions:
| Layer | How to write it | Application here |
|---|---|---|
| Fact | What the supplied material directly shows | Cite the source, version, input, or run record for the focus |
| Evidence-backed Inference | What several facts support together | Show the inference chain and retain uncertainty |
| Recommendation | The smallest next action | Name the evidence, review, execution, or regression path |
| Human Decision | What an accountable person must decide | Leave scope, risk acceptance, resources, and release meaning to the owner |
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Failure scope | Validation, queue, worker, storage, download, and affected user state | Name included stages and their owner |
| Behavior contract | Trigger/boundary, retryability, propagation, caller result, and data effect | Keep undocumented behavior as a question |
| Evidence and recovery | Telemetry/privacy rules, retry/backoff, fallback, compensation, and handoff | Identify the smallest source or validation action |
Use a request like this:
Use the error-handling-design-review Skill.
Task: review the designed failure paths for a report export that can fail validation, queueing, generation, storage, or download before implementation
Inputs: [requirements, versions, links, logs, reports, or data paths]
Scope: [included and excluded objects]
Unknowns: [missing environment, accounts, data, or permissions]
Expected output: [risk-ordered findings, evidence status, and next actions]
Audit the inputs first. Separate facts, assumptions, and open questions. Do not claim execution without a run record.
Analysis
Start with the export stages and build one row per failure mode. For a storage write failure after generation, first establish the trigger and boundary, then classify the expected path as retryable, non-retryable, human intervention, or safe rejection from the supplied contract. Trace whether the report job may already exist, what the caller is told, which fallback or handoff applies, and which privacy-safe telemetry fields are required. If idempotency, commit state, retry policy, or caller wording is not supplied, leave it open; do not prescribe a retry or claim a fault was reproduced.
The handoff should preserve the input version, time window, evidence index, owner, and next validation action. The Skill can organize uncertainty; it cannot manufacture the missing artifact.
Finding
Example finding: turn one problem into a handoff
The field example below shows the recording pattern; it is not an execution result.
Use an EH-## record for one failure mode at one boundary. The example below leaves behavior unresolved where the source contract is absent; it is a design finding, not an incident or execution result.
| EH-01 field | Storage write fails after report generation |
|---|---|
| Trigger and boundary | Storage rejects or times out after generation; exact commit boundary is not supplied |
| Expected behavior type | unknown until the contract identifies retryable, non-retryable, human-intervention, or safe-rejection behavior |
| Propagation and fallback | Caller result, fallback, and handoff are missing; do not infer a retry |
| Data consistency | Determine whether the generated artifact/job already exists and whether repeating the action can duplicate work |
| Observable evidence | Confirm the required job/correlation identifier and error class; exclude file contents and secrets per the privacy contract |
| Source and evidence state | Requirement/API version not supplied: missing; no fault injection performed |
| Impact, priority, and owner | Describe user/job impact from evidence; export/API owner confirms priority and behavior |
| Validation and close condition | Review the source contract, then define a repeatable authorized check; close the design gap only when expected behavior, caller result, owner, and evidence are reviewable |
This handoff lets the next reviewer trace the failure path to its source and see which behavior or evidence remains unresolved.
Decision
The accountable owner confirms the decision question and risk trade-off; the Skill does not make that choice.
Validation
Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.
How a Finding enters the next stage
Handoff output
| Output | Required fields for each EH-## | Example status |
|---|---|---|
| Failure boundary | Mode, trigger, boundary, expected behavior type, and source | missing where contract is absent |
| Caller/system behavior | Propagation, fallback, retry rule, visible result, and data consistency | unassessed until source is reviewed |
| Evidence and ownership | Privacy-safe telemetry, impact/priority, owner, validation, and close condition | unverified until the stated check runs |
Do not write “passed” without the evidence required for that claim. A useful review asks the same failure to make sense at three levels: the caller sees a stable result, the operator sees a diagnosable privacy-safe event, and the system has an evidence-backed consistency or recovery path.
Next-stage route
At minimum, hand off the source, evidence state, owner, close condition, and validation action; the next-stage conclusion remains bounded by the evidence state.
How to prepare better input
If the first request contains only a one-line goal, keep the output limited. Add the source version, affected objects, environment, known defects, and decision owner to move from a plausible checklist to a useful review.
A richer input changes the answer here because failure taxonomy, user action, retryability, fallback, observability, privacy, and recovery must be tied to evidence rather than inferred from a familiar pattern.
Best practices
- Review one trigger and boundary at a time, then follow expected behavior through propagation, retries or fallback, state changes, and telemetry.
- Do not recommend a retry until the source establishes whether the original action committed and whether repeating it is safe.
- Prioritize high-impact gaps and request the smallest evidence or isolated check that can resolve them.
Working with other Skills
- API Error Contract Testing: https://inaodeng.com/en/qaskills/api-error-contract-testing/ — checks machine-readable failure responses.
- Root Cause Analysis: https://inaodeng.com/en/qaskills/root-cause-analysis/ — follows failures after the design review.
- Recovery Testing: https://inaodeng.com/en/qaskills/recovery-testing/ — verifies the designed recovery path.
Common traps
- One generic error for unrelated failure classes.
- Logging sensitive payloads for diagnosis.
- Retrying before deciding whether the original operation committed.
Install and invoke
Install the English Skill from its language directory. The series overview carries the shared installation instructions.
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en --skill error-handling-design-review
After installation, invoke it with the error-handling-design-review Skill and attach the real project material.
FAQ
Should users see technical error codes?
Expose a safe support reference when useful; keep internal details and secrets out of the message.
What if no fallback exists?
Say so explicitly, define the stop or escalation action, and avoid pretending the error is recoverable.
The example above is a design and review pattern, not an execution result. Keep static analysis, runtime evidence, human approval, and release acceptance separate.
References
Source Skill and execution contract
The complete execution contract lives in the Error Handling Design Review prompt: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/error-handling-design-review/prompts/error-handling-design-review.md. Read it before invoking the Skill; the prompt defines the detailed workflow and output contract. The source directory: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/error-handling-design-review contains the entry point and supporting assets where they exist.
The entry point centers failure taxonomy, user action, retryability, fallback, observability, privacy, and recovery. Keep its decision boundary visible and do not turn a static design into an execution claim.
Further reading
-
Awesome QA Skills GitHub: https://github.com/naodeng/awesome-qa-skills
-
Error Handling Design Review details: https://inaodeng.com/en/qaskills/error-handling-design-review/
More Resources
- Personal website: https://inaodeng.com/en/
- AI-generated test detection project: https://github.com/AI-Native-QA-Lab/ai-test-auditor
- DeepSeek QA Workbench: https://github.com/naodeng/dsh-qa
- More AI testing skills: https://inaodeng.com/en/qaskills/
- More testing wiki: https://inaodeng.com/en/wiki/
- More AI Wiki: https://inaodeng.com/en/AIWiki/