A long-running job can look healthy for an hour and still lose critical state when a worker restarts. A progress bar only shows where work stands when it maps to records someone can verify.
Why this Skill is needed
Agent Long-Running Testing examines whether extended jobs make progress, preserve state, renew ownership, respond to cancellation, and leave enough evidence to explain a partial result.
What the Skill does
The Agent Long-Running Testing Skill gives teams a structured way to review progress checkpoints, lease renewal, restart recovery, cancellation, and partial-result rules.
Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills organizes Skills by language and testing stage. The series overview: https://inaodeng.com/en/blog/ai-testing/introduction_of_awesome_qa_skills/ explains the shared installation model; this guide stays with Agent Long-Running Testing.
Use it when sources disagree or a handoff needs to state what remains unverified. Tie each rule to a source and validation action; a design checklist supports a design review but does not establish a runtime pass.
What it does not do
The Agent Long-Running Testing Skill can organize material, evidence, and next actions for progress rules. It does not replace:
- Confirmation of rules, scope, and risk by the accountable domain owner.
- A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
- Authorization for release, compliance, production actions, or residual-risk acceptance.
- A Human decision when supplied sources conflict.
What it checks
The value of this Skill is not another keyword list. It connects each focus area to an observable input, a judgment, and a way to close the loop. Start with a small matrix based on the source Skill’s output contract:
| Focus | Question before analysis | Handoff output |
|---|---|---|
| Progress records | Stage and checkpoint meaning, and how counts map to completed work | A progress explanation that can be checked against the same rules |
| Heartbeat and liveness | Does a heartbeat only show that a worker responds, or is it tied to actual work progress? | Liveness evidence kept separate from checkpoints, committed steps, or item manifests |
| Interruption result | What survives restart or cancellation | Partial, cancelled, recovered, or unknown |
| Duplicate submission and operations | What retry rule applies to a task ID? How are lease, queue, or resource changes verified? | Idempotency rule, commit/recovery evidence, owner, and validation action |
If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.
Audit inputs before you start
Before analysis, classify the input into six evidence states. Missing material does not mean a runtime failure, but it must remain visible in the conclusion.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.
Keep the decision layers separate
Keep facts, inferences, recommendations, and Human decisions separate instead of compressing them into “recommended to pass”:
| Layer | How to write it | Application here |
|---|---|---|
| Fact | What the supplied material directly shows | Cite the source, version, input, or run record for the focus |
| Evidence-backed Inference | What several facts support together | Show the inference chain and retain uncertainty |
| Recommendation | The smallest next action | Name the evidence, review, execution, or regression path |
| Human Decision | What an accountable person must decide | Leave scope, risk acceptance, resources, and release meaning to the owner |
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Job contract | Ticket count, expected stages, completion definition, and partial-result policy | Do not infer completion from elapsed time |
| Runtime controls | Task IDs and duplicate-submission policy, worker lease, checkpoints/heartbeats, restart, cancellation, queue, and resource lifecycle | Mark unobserved transitions as unassessed |
| Evidence | Job ID, checkpoint records, processed counts, duplicate markers, and cancellation audit | Request a run record before making a reliability claim |
Use a request like this:
Use the agent-long-running-testing Skill.
Task: an agent spends two hours classifying 80,000 support tickets while workers can restart and users can cancel the job
Inputs: [requirements, versions, links, logs, reports, or data paths]
Scope: [included and excluded objects]
Runtime evidence: [task IDs, checkpoint/state versions, heartbeats, processed-item manifest, lease, cancellation records, and side effects]
Unknowns: [missing environment, accounts, data, or permissions]
Expected output: [risk-ordered findings, evidence status, and next actions]
Audit the inputs first. Separate facts, assumptions, and open questions. Do not claim execution without a run record.
Analysis
Use a bounded example: an agent spends two hours classifying 80,000 support tickets while workers can restart and users can cancel the job. Pause classification after checkpoint 37, restart the worker, and compare the resumed input set with the checkpoint manifest. Check which ticket IDs resumed, were skipped, were processed twice, or remain unclassified; a UI message that says “resumed” is not enough. This illustrates a test method and does not report a run on a real system.
Keep the input version, time window, evidence index, owner, and next validation action together in the handoff. The Skill can organize uncertainty, but it cannot supply a missing artifact.
Make recovery, lease, and cancellation checks observable
Using the same job, trigger one interruption condition at a time. First confirm that the job contract defines completion, cancellation, and partial completion; then check whether the run records support those states.
| Check | Condition | Records to inspect |
|---|---|---|
| Checkpoint recovery | Restart the worker after a checkpoint | Input range, state version, resume position, duplicates, and gaps |
| Lease takeover | Pause a lease holder until another worker takes over, then resume the old worker | Old and new owners, lease version or fencing token, and whether the old worker can still commit |
| Duplicate submission | Resubmit a task with the same task ID or request identity | Project idempotency/deduplication rule, committed effects, and processing result; do not assume exactly-once |
| Cancellation propagation | Cancel while a batch is in progress | Request and acknowledgment times, unfinished items, active child jobs, and user-visible terminal state |
| Partial result | End or pause the job mid-run or after a dependency failure | Completed and unfinished items, result freshness, downstream safety, and recovery or retry path |
If the system exposes no input manifest, commit identifier, state version, or lease record, record that as missing evidence. A heartbeat or progress number alone cannot show whether work was lost, duplicated, or completed safely.
Finding
Example finding: turn one problem into a handoff
The field example below shows the recording pattern; it is not an execution result.
If the supplied material cannot prove that the progress definition meets its contract, write the finding like this. It does not invent the missing rule or turn missing evidence into a failure.
| Field | Example wording |
|---|---|
| Source and scope | Record the requirement, version, environment, and the checkpoint rule under review |
| Finding | The relationship between a checkpoint and completed work is not yet traceable to evidence |
| Evidence state | missing / assumptions; use conflicting when sources disagree |
| Impact and priority | Name the affected user, journey, or delivery decision without inflating severity |
| Owner and Human decision | Ask the product, engineering, security, or test owner to confirm the rule and trade-off |
| Action and close condition | Add the smallest missing evidence; close only when source, judgment, and owner can be reviewed |
| Validation | Name one repeatable check, query, or run and retain the raw artifact |
The point is to let the next person walk from the finding back to the source and run an action that can change the decision.
Decision
The accountable owner confirms the decision question and risk trade-off; the Skill does not make that choice.
Validation
Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.
How a Finding enters the next stage
Handoff output
| Output field | Why it exists | Example status |
|---|---|---|
| Progress records | Stage and checkpoint meaning, and how counts map to completed work | A progress explanation that can be checked against the same rules |
| Interruption result | What survives restart or cancellation | Partial, cancelled, recovered, or unknown |
| Operational action | Lease, queue, or storage change to verify | Owner and evidence needed |
Do not write “passed” without a run record, query result, or source artifact. A checkpoint should show which work has been persisted; a heartbeat alone only shows that the system is responding. Also check whether a ticket can be processed twice, a lease can expire mid-stage, or work continues after cancellation.
Next-stage route
At minimum, hand off the source, evidence state, owner, close condition, and validation action; the next-stage conclusion remains bounded by the evidence state.
How to prepare better input
If the first request contains only a one-line goal, the output should stay limited. Add the source version, affected objects, environment, known defects, and decision owner to move from a plausible checklist to a useful review.
Working with other Skills
- Agent Failure Recovery Testing: https://inaodeng.com/en/qaskills/agent-failure-recovery-testing/ — covers failure and recovery decisions around the long job.
- Timeout Testing: https://inaodeng.com/en/qaskills/timeout-testing/ — focuses on time-bound interruption behavior.
- Production Verification: https://inaodeng.com/en/qaskills/production-verification/ — checks live progress and operational signals.
Common traps
- Using a percentage without defining which work is durable.
- Calling a cancelled job successful because its worker stopped.
- Ignoring lease expiry because the happy-path run finished.
Install and invoke
Install the individual Skill. The series overview carries the longer installation explanation.
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en --skill agent-long-running-testingAfter installation, invoke it with the agent-long-running-testing Skill and attach the real project material.
FAQ
What if the system only exposes a progress number?
Treat it as a claim to validate; request checkpoint or item-level evidence before trusting it.
Should partial results be published?
Only when the contract defines their completeness, staleness, and safe downstream use.
Does a heartbeat prove that a job is progressing?
Not by itself. A heartbeat usually shows that a process is responsive; verify work progress with checkpoints, committed steps, or item-level results. Interpret the terminal state against the project contract and run evidence.
The example above is a design and review pattern, not an execution result. Keep static analysis, runtime evidence, human approval, and release acceptance separate.
References
Source Skill and execution contract
The complete execution contract lives in the Agent Long-Running Testing prompt: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/agent-long-running-testing/prompts/agent-long-running-testing.md. Read it before invoking the Skill; the prompt defines the detailed workflow and output contract. The source directory: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/agent-long-running-testing also contains the entry point and supporting assets where they exist.
The entry point centers progress checkpoints, lease renewal, restart recovery, cancellation, and partial-result semantics. Read the supporting prompt before writing a conclusion, and keep the Skill’s decision boundary visible.
- Agent Long-Running Testing details: https://inaodeng.com/en/qaskills/agent-long-running-testing/
- Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills