The successful response usually gets documented first. Then a consumer receives three different error shapes for validation, authorization, and dependency failure—and somebody calls that an API contract.
Why this Skill is needed
API Error Contract Testing checks whether failures have stable structure, useful classification, safe detail, and enough information for clients to choose the right next action.
What the Skill does
Build an operation-by-error comparison from supplied schemas, status conventions, code catalogs, authorization rules, logs, and redaction requirements. Match each response to its request and contract version, then separate violations of applicable rules from documented operation exceptions and unresolved rules.
Record candidates as AEC-## with the subject, preconditions, behavior of concern, source evidence, validation, impact/priority, owner role, and close condition. This is design review; candidate counts are not test coverage.
Awesome QA Skills organizes Skills by language and testing stage; the series overview explains the shared installation model.
What it does not do
The Skill does not execute requests, invent a status code or retry policy, require unrelated requests to share a correlation ID, or call every response difference a defect. A valid response control and an applicable contract rule are needed before a difference becomes a finding.
What it checks
| Contract dimension | Record from the source | Compare in the response |
|---|---|---|
| HTTP and business classification | Status mapping, business code catalog, and declared exceptions | Status and code pair for this operation and error |
| Required fields | Required keys, types, nullable rules, and field-path shape | Missing, mistyped, or incorrectly located fields |
| Correlation | How one request is linked across response and log | Traceability within the interaction; do not compare IDs from unrelated requests |
| Retry meaning | Which errors allow retry, correction, re-authentication, waiting, or stop | Whether the code and client action match the documented rule |
| Disclosure | Redaction rules for provider, database, and internal details | Whether response or log exposes restricted details |
Compare shared rules across operations while retaining sourced exceptions. A different shape is not automatically wrong; it becomes a candidate only when it contradicts a rule that applies to that operation.
Audit inputs before you start
Before analyzing Error finding, classify the input into six evidence states. A gap is not automatically a failure, but it must not disappear inside the conclusion.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
AEC-## finding contract
Each candidate should name the operation/error pair, applicable contract source and version, preconditions, observed concern, evidence state, impact/priority, owner role, close condition, and the smallest validation method. Keep facts, inferences, recommendations, and Human decisions distinct. A response difference without a sourced rule remains an open question.
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Error contract | Envelope, code, message, field path, request ID, and status mapping | Mark optional fields that clients currently rely on |
| Failure matrix | Validation, authentication, authorization, conflict, rate limit, and dependency failure | Do not collapse different actions into one generic case |
| Consumer behavior | Retry, display, logging, fallback, and support lookup | Trace each action to an error field |
Use a request like this:
Use the api-error-contract-testing Skill.
Task: validate errors from a payment API used by a checkout service and a mobile client
Inputs: [requirements, versions, links, logs, reports, or data paths]
Scope: [included and excluded objects]
Unknowns: [missing environment, accounts, data, or permissions]
Expected output: [risk-ordered findings, evidence status, and next actions]
Audit the inputs first. Separate facts, assumptions, and open questions. Do not claim execution without a run record.
Analysis
Start with a bounded pass—validate errors from a payment API used by a checkout service and a mobile client. Send an invalid card, an expired authorization token, and a provider timeout through the same checkout endpoint. Assert the envelope and then assert the client action: fix input, re-authenticate, or retry after a delay. Also check that provider details are redacted.
The useful handoff keeps the input version, time window, evidence index, owner, and next validation action together. The Skill can organize uncertainty; it cannot manufacture the missing artifact.
Finding
The following is a recording pattern, not an execution result:
| Field | Example |
|---|---|
| ID and subject | AEC-01 — provider-timeout retry meaning for the checkout operation |
| Preconditions and concern | The sourced contract labels the condition retryable, but the mobile client stops without the documented action |
| Evidence state | UNASSESSED until the contract version and matching client trace are reviewed |
| Impact/priority and owner | Explain consumer impact; name API contract and client owners; score only with the project scale |
| Close condition and validation | Align applicable contract and client behavior, then perform an isolated contract check and retain raw artifacts |
The source must establish that the rule applies before the difference can be called a defect.
Decision
The API contract owner confirms whether the rule applies to this operation; the client owner confirms the intended consumer action. Do not resolve conflicting sources by choosing the more familiar response.
Validation
Match the request, response, log, and contract version for one interaction. Compare a valid control and retain the raw artifacts; do not require correlation IDs from separate requests to match.
How a Finding enters the next stage
Handoff output
| Output field | Why it exists | Example status |
|---|---|---|
| Error finding | Shape, classification, field location, or security problem | Contract source and impact |
| Consumer action | Retry, correct input, re-authenticate, wait, or stop | Expected decision |
| Compatibility note | Change that breaks or weakens a client assumption | Migration or version action |
Do not write “passed” without a run record, query result, or source artifact. A useful test starts from the client decision, not the server implementation. If a mobile client cannot distinguish a temporary provider outage from an invalid card, the error contract has failed even when every response is valid JSON.
Next-stage route
Route unresolved code-catalog, retry, or redaction rules to their owners with the affected operations and source versions attached. Preserve operation-specific exceptions rather than flattening them into a global rule.
How to prepare better input
Provide the error schema and code catalog, operation list, contract versions, sample requests and responses, matching logs, client decisions, and redaction policy. If a source is missing, state which part of the comparison cannot yet be judged.
Working with other Skills
- API Contract Testing:covers the wider request and response contract.
- API Negative Testing:generates invalid and boundary inputs.
- API Security Testing:checks leakage and authorization failure handling.
Common traps
- Asserting only HTTP status.
- Letting human-readable messages become machine contracts.
- Returning provider or database details that expose internal state.
Install and invoke
Install the individual Skill. The series overview carries the longer installation explanation.
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/api-error-contract-testing -g
After installation, invoke it with the api-error-contract-testing Skill and attach the real project material.
FAQ
Should every error have a unique code?
Not necessarily, but each code must support a distinct consumer action or a deliberate shared policy.
Can snapshots prove the contract?
They can catch shape changes, but they need semantic assertions for retryability, privacy, and client behavior.
The example above is a design and review pattern, not an execution result. Keep static analysis, runtime evidence, human approval, and release acceptance separate.
References
Source Skill and execution contract
The complete execution contract lives in the API Error Contract Testing prompt. Read it before invoking the Skill; the prompt defines the detailed workflow and output contract. The source directory also contains the entry point and supporting assets where they exist.
The entry point centers error envelope, code taxonomy, field paths, retryability, redaction, and client handling. Read the supporting prompt before writing a conclusion, and keep the Skill’s decision boundary visible.
Reference links
-
API Error Contract Testing:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/api-error-contract-testing/prompts/api-error-contract-testing.md
-
Awesome QA Skills: API Error Contract Testing source:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/api-error-contract-testing
-
Awesome QA Skills GitHub:https://github.com/naodeng/awesome-qa-skills
-
API Error Contract Testing details:https://inaodeng.com/en/qaskills/api-error-contract-testing/