An AI-generated test suite can contain fifty polished cases and still miss the one rule that refunds an expired coupon. Volume is a poor substitute for a source.
Why this Skill is needed
AI-Generated Test Review asks whether each generated test could detect an incorrect production behavior. It traces tests to rules and risks, inspects the code path and assertion, and looks for the smallest bug that could still pass.
What the Skill does
Use this Skill only for test types found in the supplied scope or named by the user. For each test, compare intended behavior with the production behavior exercised, the observable assertion, expected-value independence, mocking boundary, and failure modes that could escape.
Awesome QA Skills organizes Skills by language and testing stage. The series overview explains the shared installation model; this guide stays with AI-Generated Test Review.
Use it when AI-written tests look plausible or increase coverage but their regression protection, source support, data isolation, or maintenance risk still needs evidence.
What it does not do
The AI-Generated Test Review Skill can organize per-test verdicts and evidence gaps. It does not replace:
- Confirmation of rules, scope, and risk by the accountable domain owner.
- A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
- Authorization for release, compliance, production actions, or residual-risk acceptance.
- A Human decision when supplied sources conflict.
What it checks
The value of this Skill is not another keyword list. It connects each focus area to an observable input, a judgment, and a way to close the loop. Start with a small matrix based on the source Skill’s output contract:
| Review axis | Per-test question | Evidence to record |
|---|---|---|
| Scope and traceability | Which test type is actually present, and which rule/risk does this test serve? | Test location, source rule, version, or a scope gap |
| Behavior and oracle | Does production logic run? What observable result is asserted? Could a wrong implementation still pass? | Intended behavior, actual path, assertion, independent expected value, smallest escaping bug |
| False-green risks | Is the subject under test mocked? Are failures swallowed? Are setup, data, cleanup, waits, or parallel runs unsafe? | Concrete test/code evidence and impact |
| Per-test verdict | Does it provide strong regression protection, partial protection, or almost none? | STRONG, WEAK, or FAKE with confidence and reason |
| Repair and priority | What smallest change would make the relevant bug fail? | Evidence-backed P0/P1/P2 issue or P2 improvement and verification order |
If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.
Audit inputs before you start
Before judging test strength, identify the test types actually in scope and classify the available evidence. If no test code or reviewable test is present, report a scope gap instead of assessing a template or every rules file as a global checklist.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.
Keep the decision layers separate
For Traceability map, do not compress four different kinds of language into “recommended to pass”:
| Layer | How to write it | Application here |
|---|---|---|
| Fact | What the supplied material directly shows | Cite the source, version, input, or run record for the focus |
| Evidence-backed Inference | What several facts support together | Show the inference chain and retain uncertainty |
| Recommendation | The smallest next action | Name the evidence, review, execution, or regression path |
| Human Decision | What an accountable person must decide | Leave scope, risk acceptance, resources, and release meaning to the owner |
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Review scope | Test paths/types, test code, code under test, results, and framework | If no reviewable test is found, request a path or test type |
| Behavior sources | Requirements, acceptance criteria, API contract, risks, and version | Keep absent rules open; do not invent product defects |
| Test mechanics | Data, fixtures, mocks, expected values, cleanup, waits, and parallel execution | Inspect only relevant mechanisms in discovered test types |
Use a request like this:
Use the ai-generated-test-review Skill.
Task: review the supplied AI-generated API tests for a promotion engine that combines customer tier, coupon expiry, and cart thresholds
Inputs: [requirements, versions, links, logs, reports, or data paths]
Scope: [included and excluded objects]
Unknowns: [missing environment, accounts, data, or permissions]
Expected output: [risk-ordered findings, evidence status, and next actions]
Audit the inputs first. Separate facts, assumptions, and open questions. Do not claim execution without a run record.
Analysis
Use this as an illustrative review only; no real test suite is supplied here. Assume the source rule says an expired coupon cannot change the order total, and a generated API test only asserts HTTP 200. First confirm that the API implementation—not a mock of the subject—is exercised. If it is, the test may be WEAK: an expired discount could still be applied while the status remains 200. The smallest escaping bug is that unintended discount. Add source-backed assertions for discount rejection and the unchanged total. If the test only calls a mocked promotion subject and checks its preset response, it may be FAKE; confirm the boundary before classifying. If the rule or test code is missing in a real review, keep the verdict unassessed and report the scope gap.
The useful handoff keeps the input version, time window, evidence index, owner, and next validation action together. The Skill can organize uncertainty; it cannot manufacture the missing artifact.
Finding
Example finding: turn one problem into a handoff
The field example below shows the recording pattern; it is not an execution result.
For every reviewed test, record its intended behavior, actual assertion, and the smallest relevant bug that could escape. The example below is conditional on the illustrative rule and test above; it is not a finding about a real repository.
| Review field | Illustrative API test record |
|---|---|
| Scope and source | Expired-coupon API test; assume the supplied promotion rule says the expired coupon cannot change order total |
| Intended behavior and actual assertion | Reject expired discount and preserve total; generated test asserts only HTTP 200 |
| Escaping bug | Expired discount is applied while API still returns 200 |
| Verdict and confidence | WEAK only if production promotion logic runs; if the subject is mocked, inspect the boundary before deciding whether it is FAKE |
| Evidence gap and repair | Check test body, code path, and independent expected total; add source-backed business-state assertions |
| Priority, owner, and validation | Set P0/P1/P2 only after impact evidence; test owner confirms the minimal regression check and retains its run result |
The point is to let the next person walk from the finding back to the source and run an action that can change the decision.
Decision
The accountable owner confirms the decision question and risk trade-off; the Skill does not make that choice.
Validation
Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.
How a Finding enters the next stage
Handoff output
| Output | Required detail | Example status |
|---|---|---|
| Scope and evidence | Test types found, reviewed paths, source rules, versions, and open questions | scope gap if tests are absent |
| Per-test verdict | Name, STRONG/WEAK/FAKE, confidence, behavior, assertion, escaping bug, conclusion | Case-level, not suite-count based |
| Findings and repair | P0/P1 evidence-backed risks, P2 improvements, missing assertions, repair direction | No priority without impact evidence |
| Coverage and verification | Applicable paths, data/isolation/cleanup risks, residual risk, recommended verification order | unverified until evidence exists |
Do not promote pass counts or coverage percentages into quality evidence. A test is useful when the right defect changes an assertion: compare expected values independently, verify that relevant production logic runs, and check persistence or business invariants where the source contract requires them. Apply positive, failure, boundary, permission/state-transition, and recovery checks only to test types actually identified in scope.
Next-stage route
Hand off each test location, source rule, verdict rationale, escaping bug, evidence gap, owner, repair, and verification order. Keep test faults, product faults, and missing evidence separate.
How to prepare better input
If the first request contains only a one-line goal, the output should stay limited. Add the source version, affected objects, environment, known defects, and decision owner to move from a plausible checklist to a useful review.
Working with other Skills
- Test Case Reviewer:provides general case quality review.
- Requirement Traceability Analysis:connects cases back to source requirements.
- AI Feature Testing:covers the product behavior of AI features.
Common traps
- Treating natural language quality as requirement coverage.
- Accepting an expected result that cannot distinguish correct and incorrect behavior.
- Keeping duplicate cases because they use different wording.
Install and invoke
Install the individual Skill. The series overview carries the longer installation explanation.
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-generated-test-review -g
After installation, invoke it with the ai-generated-test-review Skill and attach the real project material.
FAQ
Can the reviewer use only the generated suite?
No. The source requirement is the authority; the suite is the object under review.
Should every generated case be repaired?
No. Remove unsupported cases when repair would require inventing product behavior.
The example above is a design and review pattern, not an execution result. Keep static analysis, runtime evidence, human approval, and release acceptance separate.
References
Source Skill and execution contract
The complete execution contract lives in the AI-Generated Test Review prompt. Read it before invoking the Skill; the prompt defines the detailed workflow and output contract. The source directory also contains the entry point and supporting assets where they exist.
The entry point centers source traceability, invented behavior, boundary coverage, oracle quality, and maintenance cost. Read the supporting prompt before writing a conclusion, and keep the Skill’s decision boundary visible.
Reference links
-
AI-Generated Test Review:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-generated-test-review/prompts/review-test.md
-
Awesome QA Skills: AI-Generated Test Review source:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-generated-test-review
-
Awesome QA Skills GitHub:https://github.com/naodeng/awesome-qa-skills
-
AI-Generated Test Review details:https://inaodeng.com/en/qaskills/ai-generated-test-review/