A tool call can succeed while the task makes no progress. Agent Loop Testing compares each plan, action, and observation with the goal to see whether new evidence arrived, the state advanced, or the agent is repeating work.
Why this Skill is needed
A research agent may keep revisiting a blocked publisher page while its budget runs down. Without a per-iteration trace, a reviewer sees the final answer but cannot tell where the strategy stalled, whether a retry was allowed, or why the agent stopped.
What the Skill helps you evaluate
Use the Agent Loop Testing Skill to review the loop as a sequence of evidence-bearing steps:
- Record task, run, and step IDs; plan; tool and normalized arguments; observation identity or digest; goal-relevant state delta; and cumulative time or cost.
- Compare the new observation with the prior state. Progress requires new goal-relevant evidence or state advancement; a successful tool response or repeated prose alone does not count.
- Check the project-supplied no-progress limit, retry policy, time and cost budgets, and authorized stop, strategy-change, or escalation branch.
- Report the trigger step, termination class, remaining work, and evidence that no extra effects occurred.
Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills organizes Skills by language and testing stage; the series overview: https://inaodeng.com/en/blog/ai-testing/introduction_of_awesome_qa_skills/ explains the shared installation model.
Scope and limits
This Skill reviews loop behavior and its trace. It does not replace testing of a tool’s business contract, payment recovery, model quality, or human safety decisions. It cannot invent iteration limits, infer hidden state from the final answer, or turn a plan, static review, or dry run into execution evidence. If the project has not supplied a hard limit, keep the decision open for its owner instead of choosing a number.
Inputs to audit
| Input | What to establish | If it is absent |
|---|---|---|
| Objective and success criteria | What evidence or state would complete the task? | Ask the owner to define a checkable completion condition |
| Loop trace | Which plan, action, observation, and state delta belong to each step? | Do not reconstruct progress from the final response |
| Limits and policy | What no-progress, retry, time, and cost rules apply? | Mark the boundary missing; do not make up a threshold |
| Safety and cancellation rules | Which signals require an immediate stop? | Keep execution blocked until the authorized rule is known |
| Evidence handling | How are arguments normalized and sensitive values redacted? | Preserve deduplication evidence without exposing secrets |
Mark evidence as known, missing, conflicting, stale, out of scope, or assumed. For every assumption, state how it can be checked.
Build the loop trace
| Trace field | Review question |
|---|---|
| Plan and step ID | What was the agent trying to learn or change on this iteration? |
| Tool and normalized arguments | Is this materially the same action as before? Redact secrets but retain a safe comparison fingerprint. |
| Observation identity | Is this a new result, or the same blocked/empty result again? |
| State delta | Did goal-relevant evidence or business state advance? |
| Cumulative budget | How much time and cost have been used against supplied limits? |
| Decision | Continue, change strategy, stop, or escalate—and which rule authorizes it? |
Treat repeated action/observation with no goal-relevant advancement as a no-progress cycle. A retry is not automatically wrong: it needs a project-authorized reason, a bounded policy, and evidence that distinguishes it from the previous attempt.
Worked scenario: research agent revisits a blocked source
This is a design example, not an execution result. A research agent must collect three sources that meet the project’s authority criteria. One publisher page is blocked, and the trace shows the agent returning to it.
Scenario and decision gates
| Stage | Stimulus or precondition | Expected review decision | Evidence and stop condition |
|---|---|---|---|
| First blocked result | The publisher page returns a blocked observation | Record the result and state delta; do not call the task complete | Save step ID, normalized request, observation digest, and budget used |
| Repeated attempt | The same route returns no new source evidence | Count a no-progress cycle and apply the supplied attempt limit | Stop repeating when the configured limit is reached; if no limit exists, request one |
| Strategy change | The source remains blocked | Follow the authorized change-strategy branch, such as an official repository or registry | Record why the new source meets the stated authority criteria |
| Completion check | Candidate sources have been gathered | Verify each source against the objective; do not rely on the agent’s completion sentence | Stop only when the documented success rule is met or a hard budget/safety condition requires handoff |
| Unresolved outcome | Evidence or budget is insufficient | Stop or escalate under project policy | Report remaining work, unresolved evidence, termination class, and no-extra-effect evidence |
Example finding contract
The source prompt names findings ALT-##. This sample shows fields and evidence state only; it is not a measured failure:
| Field | Example |
|---|---|
| ID and trigger | ALT-01 — repeated the blocked publisher route without new source evidence |
| Expected concern | No goal-relevant state advancement across the compared steps |
| Evidence state | UNASSESSED until the run trace and project limit are available |
| Impact and priority | Explain decision cost or user impact; use NOT_SCORED until the project scale is supported |
| Owner and close condition | Name the agent/workflow owner; close when a repeatable trace check shows the authorized stop or strategy-change branch |
| Validation | Compare step IDs, normalized arguments, observation digests, state delta, budget, and termination record |
Handoff to the next stage
Include the objective and success rule, trace version, evidence states, no-progress and budget policy, triggering step, termination class, remaining work, owner, close condition, and smallest validation action. For tool-side effects, state what was checked and what remains unknown.
A downstream reviewer should be able to explain why the agent continued or stopped without guessing. If a hard limit or safety decision is missing, hand off that decision instead of embedding an invented default in the test.
Related Skills
- AI Agent Testing: https://inaodeng.com/en/qaskills/ai-agent-testing/ provides wider behavior and goal context.
- Timeout Testing: https://inaodeng.com/en/qaskills/timeout-testing/ examines time-bound stop and failure behavior.
- Agent Failure Recovery Testing: https://inaodeng.com/en/qaskills/agent-failure-recovery-testing/ handles recovery after a failed action; loop testing focuses on progress and termination across iterations.
Common traps
- Counting iterations without comparing goal-relevant state.
- Treating a successful tool call or fluent answer as proof of progress.
- Inventing a retry or budget limit when the project has not supplied one.
- Repeating a blocked action without recording the trigger, strategy change, or stop reason.
- Logging raw sensitive arguments instead of a safe normalized comparison.
Install and invoke
Install the individual Skill and provide the objective, loop policy, and trace evidence:
npx skills add naodeng/awesome-qa-skills --skill agent-loop-testingUse the agent-loop-testing Skill.
Task: assess a research agent that revisits a blocked publisher while collecting three authoritative sources
Inputs: [success criteria, loop policy, task/run/step trace, observations, time or cost record]
Scope: [loop and tools included or excluded]
Unknowns: [missing no-progress limit, evidence, authority rule, or permissions]
Expected output: [risk-ordered ALT-## findings, trace gaps, termination decisions, and validation steps]
Separate facts, inferences, recommendations, and Human decisions. Do not invent thresholds or claim a run that is not in the evidence.FAQ
Does every repeated tool call mean the loop is stuck?
No. Compare the observation and goal-relevant state. A retry can be valid if an authorized policy allows it and it has a reason; unchanged action and evidence without advancement is the signal to review.
What if the project has not set a no-progress limit?
Do not invent one. Record the missing rule, identify the owner who can define it, and keep the stop decision unassessed until the rule is confirmed.
What should the agent return when it cannot progress?
A bounded handoff: the blocking condition, attempts already made, evidence still missing, remaining work, and the authorized human or system action needed to continue.
References
Source Skill and execution contract
The Agent Loop Testing prompt: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/agent-loop-testing/prompts/agent-loop-testing.md defines the input audit, trace procedure, ALT-## finding contract, and output order. The source directory: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/agent-loop-testing contains the entry point and supporting material.
- Agent Loop Testing details: https://inaodeng.com/en/qaskills/agent-loop-testing/
- Awesome QA Skills: https://github.com/naodeng/awesome-qa-skills