When an agent times out during payment authorization, the result may be committed, rejected, or still unknown. Recovery has to establish which state is true before it chooses another action or tells the user to try again.
Why this Skill is needed
Agent workflows span tools, services, and user-visible messages. A failure can happen after one step wrote state but before the next step completed. The Agent Failure Recovery Testing Skill helps teams trace that partial journey and decide whether to retry, reconcile, compensate, escalate, or stop.
What the Skill helps you do
Build a failure matrix from evidence the project actually provides:
| Failure | Confirmed write state | Retryability | Recovery action | Recovered state and evidence |
|---|---|---|---|---|
| Transient and no write confirmed | Authoritative sources and a complete query scope confirm no committed effect | Only if policy permits | Retry or poll with the original operation identity | Verify one final effect and user notice |
| Permanent rejection | Rejection is confirmed | Not by repeating unchanged input | Stop and explain or escalate | Preserve rejection and handoff record |
| Indeterminate timeout | Commit state is not established | Do not assume safe replay | Query by operation ID or idempotency key; then follow the contract | Keep unresolved until records reconcile |
| Partial workflow success | Some steps are committed | Depends on each remaining step | Reconcile completed and remaining steps; compensate only with protocol and authority | Verify every step and any compensation outcome |
Use AFR-## findings to connect the failure point, evidence, recovery choice, user-visible outcome, owner, and close condition. The matrix supports a review or test plan; it does not prove runtime recovery happened. Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills organizes Skills by language and testing stage; the series overview: https://inaodeng.com/en/blog/ai-testing/introduction_of_awesome_qa_skills/ explains the shared installation model.
Scope and limits
The Skill does not define a universal retry count, timeout, fallback, or compensation policy. It cannot treat a timeout as proof of no write, authorize a refund or rollback, or claim recovery from missing logs. The accountable system owner must confirm policy, and runtime claims need run records and state evidence.
What to inspect
| Recovery concern | Evidence question | Decision it supports |
|---|---|---|
| Failure certainty | Was the action rejected, never sent, accepted, or left indeterminate? | Classify as transient, permanent, or unknown |
| Prior writes | Which steps committed before the failure? | Prevent repeating completed side effects |
| Retry safety | Is the original operation ID or idempotency key reusable under the contract? | Retry, poll, or hold for reconciliation |
| Compensation | Does an approved protocol allow reversing the completed step? | Compensate only with authority; otherwise escalate |
| User communication | What state can the system prove to the user? | Say pending, complete, failed, or under review only when supported |
| Recovery evidence | Do logs, state transitions, attempts, and user notice agree? | Close as verified or retain an evidence gap |
A recovery action is not complete when a retry succeeds. Compare the before and after state, attempt count, side effects, resources, and user-visible outcome.
Audit inputs before designing recovery tests
| Needed input | Use in the review | Missing-input treatment |
|---|---|---|
| Workflow steps and failure injection points | Show where partial completion is possible | Bound the scenario to known steps |
| Operation ID, request ID, or idempotency key | Find the original attempt and committed effects | Do not replay an indeterminate write without a safe contract path |
| Retry, fallback, and compensation policy | Decide which actions are allowed | Leave the recovery choice open for its owner |
| State sources and user messages | Compare system truth with what the agent reports | Mark recovery unverified if logs or state are absent |
| Stop conditions and escalation owner | Prevent endless retries and unauthorized repair | Record who must take over and what blocks continuation |
Separate facts, assumptions, conflicts, stale records, and out-of-scope steps. Missing evidence limits the conclusion; it is not automatically a product failure.
Worked scenario: authorization timeout in an order agent
This scenario is a design example, not an executed payment test. The agent submits one authorization request, times out before receiving a result, and the order remains pending.
Recovery sequence
- Keep the original operation ID and idempotency key; record the timeout and attempt number.
- Within the authorized access scope, query the payment provider and internal ledger using those identifiers before issuing another write.
- If authorization is confirmed, reconcile the order state and continue the documented confirmation path. Do not charge again to make the order look complete.
- If the provider reports no operation, follow the contract’s retry rule with the original identity. A missing lookup result is not proof of failure if the source can be delayed or incomplete.
- If records conflict or remain indeterminate, stop automated replay and escalate with both records attached.
- Tell the user only what the evidence supports—for example, that confirmation is pending while the payment status is checked.
If the workflow partly succeeded, list completed and remaining steps separately. A refund or other compensation requires an approved protocol and an authorized owner; record the result and escalate if compensation itself fails.
Example finding
The fields below are a recording pattern, not a real incident or run result:
| Field | Example |
|---|---|
| ID and scenario | AFR-01 — payment authorization timed out before the order agent received a result |
| Confirmed write state | UNKNOWN until provider and internal records are reconciled |
| Retryability and action | UNASSESSED; query the original operation before any replay |
| User impact and priority | State the effect supported by evidence; use NOT_SCORED until the project scale can be applied |
| Owner and stop condition | Name the payment/workflow owner; keep automated replay stopped while commit state is indeterminate |
| Close condition | Attempt history, payment and order state, recovery action, and user notice agree and can be reviewed |
Do not turn an example, recommendation, or missing log into a confirmed failure or successful recovery.
Handoff to the next stage
| Handoff field | Include |
|---|---|
| Failure record | Failure point, classification, operation identity, timestamp, and attempt number |
| Write state | Confirmed completed, confirmed absent, partial, or still indeterminate—with source links |
| Recovery decision | Retry, poll, reconcile, compensate, escalate, or stop, plus its policy basis |
| User outcome | Exact notice shown and evidence that supports it |
| Finding | AFR-##, impact basis, owner, stop condition, and close condition |
| Validation | Before/after states, effect counts, logs, and compensation outcome if applicable |
A recovery is verified only when the system state and user-visible outcome agree with the evidence. If they do not, hand off the mismatch and keep the state open.
Related Skills
- AI Agent Testing: https://inaodeng.com/en/qaskills/ai-agent-testing/ reviews wider planning, memory, and goal behavior.
- Retry Testing: https://inaodeng.com/en/qaskills/retry-testing/ examines retry boundaries and duplicate execution.
- Production Verification: https://inaodeng.com/en/qaskills/production-verification/ connects recovery decisions to live signals.
Common traps
- Treating timeout as proof that no write occurred.
- Replaying before checking the original attempt’s committed effects.
- Retrying a permanent rejection with unchanged input.
- Recommending compensation without protocol and owner authority.
- Telling the user a task completed when only one workflow step is confirmed.
Install and invoke
Install the individual Skill and provide the workflow contract and real evidence sources:
npx skills add naodeng/awesome-qa-skills --skill agent-failure-recovery-testingUse the source prompt’s failure matrix and finding fields when requesting a review:
Use the agent-failure-recovery-testing Skill.
Task: assess an order agent that timed out during payment authorization
Inputs: [workflow contract, operation IDs, provider responses, logs, state queries]
Scope: [steps and effects included or excluded]
Unknowns: [missing policies, evidence sources, permissions, or owners]
Expected output: [risk-ordered AFR-## findings, recovery matrix, evidence states, and next actions]
Separate confirmed writes from assumptions. Do not call a recommendation an executed recovery.FAQ
Can recovery testing start without provider logs?
Yes, as a constrained design review. List what would distinguish no write, confirmed write, partial completion, and unknown state; leave runtime recovery unverified until the records are available.
When is retry unsafe?
When the original write may have committed and the contract offers no safe replay path. Check the original operation first, then follow its retry policy or escalate.
Should every partial failure be compensated?
No. Compensation can create another business effect. Use it only when an approved protocol and authorized owner permit it, and verify its result separately.
References
Source Skill and execution contract
The Agent Failure Recovery Testing prompt: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/agent-failure-recovery-testing/prompts/agent-failure-recovery-testing.md defines the workflow and output contract. The source directory: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/agent-failure-recovery-testing contains the entry point and supporting material.
- Agent Failure Recovery Testing details: https://inaodeng.com/en/qaskills/agent-failure-recovery-testing/
- Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills