Agent tool testing focuses on the boundary between an agent and the tools it can call. The review asks whether the agent chose the right tool, formed valid arguments, had permission for the action, handled failure safely, and left traceable evidence. A correct final answer does not prove that the call itself was safe.

Treat each call as a contract

Start from the tool schema, routing rules, authorization model, sandbox, and available tool-call traces. For every scenario, record preconditions, the triggering action, expected behavior, and the evidence that would confirm it.

BoundaryWhat to checkUseful evidence
SelectionThe chosen tool matches the requested action; ambiguity leads to a question or safe stopRequest, candidate tools, routing trace
ArgumentsRequired fields, types, ranges, and object identifiers match the schemaTool schema, arguments actually sent, validation result
AuthorizationThe actor may perform this action on this resource in this tenantRole and policy snapshot, allow or deny result
Retry and recoveryTimeouts, partial failures, and retries do not repeat a writeRequest IDs, retry trace, before-and-after state
Result handlingTool errors and partial results are not presented as completed workResponse payload, agent trace, user-facing answer

Exercise failure paths in an isolated environment

A refund workflow is an illustrative scenario, not a claim about a particular product. First, use a mock service or authorized sandbox to test an allowed request and a request outside the actor’s permissions. Confirm that a denied request leaves no state change.

Then check a timeout that occurs after the service may have accepted the first call. The mock can record the request as accepted before returning a timeout. Inspect the call trace under the project’s retry rule and evaluate side effects against the tool contract. Assert that no duplicate refund or notification occurs only if the contract defines idempotency; otherwise, leave retry semantics open.

For each test run, record the arguments, authorization decision, request ID and call ID, response, retry count, before-and-after state, and notification records. A timeout response without mock-state evidence cannot show whether the first call took effect; mark it “insufficient evidence.”

Use mocks, dry runs, or isolation by default. Never execute real writes without authorization. For each issue, rank impact and likelihood using the project’s scale; label unconfirmed causes as hypotheses. Missing schemas, policies, or run records are input gaps, not proof that a call passed.

Hand off an executable test plan

The result should let another tester act without reconstructing the conversation. Include the tool-contract matrix, normal and failure scenarios, assertions, evidence locations, risk priority, blockers, and residual risks. Give each open item an owner and a next check. The Skill can organize the test plan; it cannot manufacture authorization or runtime evidence.

Related guides:

Install and use

Install the English Skill with the Agent Skills CLI:

npx skills add naodeng/awesome-qa-skills --skill agent-tool-testing

Quick-start and troubleshooting guides:

After installation, describe the task in a tool that discovers Agent Skills automatically. If your tool supports explicit invocation, use its syntax to select agent-tool-testing. Invocation syntax varies by tool; do not assume every tool supports @skill.

Before analysis, gather the tool schema, Agent instructions and routing rules, authorization model, sandbox or simulator, tool-call traces, and the task scope, environment, constraints, and success criteria. Redact production data, personal information, and credentials according to your team’s rules.

For example:

Use agent-tool-testing to review this tool-call testing task.
Goal and success criteria: [fill in]
Scope, environment, and version: [fill in]
Materials: [attach the tool schema, Agent instructions and routing rules, authorization model, sandbox details, and call traces]
Constraints: [list prohibited actions; default to mocks, dry runs, or an isolated environment]
Separate confirmed facts, assumptions, and information gaps. Return risk priorities, executable scenarios, assertions, evidence, blockers, and next steps.

FAQ

  • What should I do if the tool does not load the Skill? Check the target tool’s Skills directory, language version, and installation method. If the tool supports explicit invocation, use its syntax to select the Skill.
  • How do I judge whether the output is ready for review? Check whether tool selection, normal and failure scenarios, assertions, evidence, and risk priorities match the task materials. Keep conclusions without supporting evidence marked as open questions or information gaps.
  • Will the Skill run tests or make real writes? No. It helps organize a test plan. Execution depends on the Agent, tools, and target environment. Use mocks, dry runs, or isolation by default; do not make real writes without authorization.
  • What if the schema, authorization policy, or run records are missing? You can still produce a limited plan, but label assumptions and input gaps. Do not mark a call as passed when evidence is insufficient.
  • Must the output be Markdown? Markdown is the default. Request a table, JSON, or ticket format by specifying the required format and fields.

References

ACP (Agent Client Protocol)Active MemoryAgent AuthorizationAgent CardAgent Evals
Share