Automation Testing: Decide what to automate before choosing tools and frameworks
Automation is not moving manual cases into scripts. A journey that changes often, depends on human judgment, or runs in an unstable environment can cost more after automation. The worthwhile candidates are frequent, rule-based, stable, and useful in a delivery gate.
The Automation Testing Skill ranks candidates by value, maintenance cost, and risk, then helps choose the right layer, data strategy, and failure handling. It turns “should we automate this?” into an explicit trade-off rather than an assumption made after choosing a tool.
Starting with a manual regression pack, this guide shows how to select automation candidates, estimate maintenance cost, and place stable checks at the right delivery gate.
Awesome QA Skills organizes Skills by language and testing stage. The series overview covers repository structure and shared installation options; this guide stays with Automation Testing.
Read the source Skill first
The main prompt covers Quality Bar, Workflow, Core Constraints, Progressive Disclosure, Pre-delivery Checklist. Those headings are navigation; the project artifacts still provide the facts.
The source directory contains 2 references, 17 script entries. Start with Automation Testing supporting references, Template conversion script.
Work through one concrete task
Select maintainable automation candidates from a manual regression pack and place them in the delivery gates
Start with this compact input set.
Sources: current requirements, relevant pages or interfaces, known defects
Scope: journeys affected by this change
Unknowns: environment, accounts, data preparation
Expected output: risk order, coverage list, open questionsThis output fragment demonstrates the contract. It is not an execution result.
| Priority | Test point | Basis | Status |
|---|---|---|---|
| P0 | Core business journey | Acceptance criterion AC-01 | Not run |
| P1 | Error and recovery | Historical defect BUG-17 | Data missing |
| P1 | Boundary and state changes | Business rule BR-03 | Needs confirmation |
Do not write “passed” during static design. Runtime evidence comes later.
Move from coverage ideas to a decision
The task is Select maintainable automation candidates from a manual regression pack and place them in the delivery gates. Split by risk before deciding how many cases to write.
| Risk question | Test design | Evidence |
|---|---|---|
| Could the main journey block a transaction | End-to-end path with state assertions | Run record, order state, build ID |
| Could retry create duplicate data | Repeat submission and idempotency check | Request ID and data query |
| Can a user recover after failure | Timeout, refresh, and re-entry | UI state, logs, recovery result |
Grow the table after Quality Bar and Workflow has been checked against the change. For each test point, ask which requirement or risk supports it, which environment and data it needs, and what remains after failure. An unanswered item is still an idea, not an execution handoff.
A prompt you can adapt
Replace the bracketed fields with project facts. Specific material leaves less room for guessing.
Use the automation-testing Skill.
Task: Select maintainable automation candidates from a manual regression pack and place them in the delivery gates
Version and environment: [requirement / build / environment]
Inputs: [file paths or links]
Scope: [included and excluded journeys]
Constraints: [accounts, data, time, compliance]
Order coverage by business risk and distinguish design from execution status. Give each test point a basis, data need, and expected evidence.
Finish with open questions. Do not invent missing facts.Use the first pass to inspect structure and gaps. Supply missing material before asking for the handoff-ready artifact.
Advanced use, from one call to a maintained flow
Save the result of Select maintainable automation candidates from a manual regression pack and place them in the delivery gates as a baseline. After requirement or code changes, perform impact analysis and rerun affected journeys plus the fixed gate set.
A three-Skill chain
requirements-analysis → automation-testing → test-reporting
| Handoff | Payload | Receiver check |
|---|---|---|
| Upstream to automation-testing | Source versions, scope, risks, open questions | Automation Testing staleness and conflicts |
| automation-testing to downstream | Primary artifact, evidence index, unfinished work | Automation Testing executability and owners |
| Feedback to automation-testing | Runs, defects, new risks | Automation Testing baseline and regression update |
Do not paste three complete outputs into one large prompt. Give Automation Testing a structured summary and accessible source artifacts. It saves context and makes defects traceable.
Team gates
| Gate | Check | Failure action |
|---|---|---|
| automation-testing input | Version, environment, owner, accessible sources | Stop Automation Testing and list gaps |
| automation-testing artifact | Material claims carry basis and status | Return Automation Testing for evidence |
| automation-testing execution | Command, exit status, report are reproducible | Classify infrastructure or test failure |
| automation-testing decision | Residual risks have accepter and date | Do not enter the next stage |
Review Automation Testing adoption, human edit rate, unsupported claims, and failure-to-diagnosis time each sprint. Record a baseline for several cycles before setting targets.
What I would inspect during use
- Scope points to a real business journey instead of turning Automation Testing into an encyclopedia.
- Quality Bar is grounded in requirements, pages, interfaces, or defects.
- Happy paths, exceptions, boundaries, and recovery are ordered by risk.
- Status distinguishes designed, not run, passed, and failed.
Install and invoke
Install the individual Skill. The series overview carries the longer installation explanation.
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/automation-testing -g -a codex -yInvoke it with “Use the automation-testing Skill,” then attach the real artifacts.
Two practical questions
Can I use Automation Testing with incomplete material?
Yes, but the result should degrade to known facts, assumptions, and open questions. Missing environments or data cannot support an execution claim.
How do I know whether Workflow is sufficient?
Trace it back to requirements and risks. A handoff-ready result explains coverage, omissions, and the next action.
When is human review mandatory?
Require an accountable person for scope trade-offs, risk acceptance, release decisions, and source conflicts.
What should be archived?
Keep the input version, Skill output, human edits, and final evidence so the conclusion can be reconstructed.
Run Automation Testing against one real artifact and keep the input, output, and review notes. The fragments here establish structure; project evidence must still come from the project.
References
- Awesome QA Skills project:https://github.com/naodeng/awesome-qa-skills
- Awesome QA Skills series overview:https://inaodeng.com/en/blog/ai-testing/introduction_of_awesome_qa_skills/
- Automation Testing supporting references:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/automation-testing/references/local
- Automation Testing helper script: batch_convert_templates.py:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/automation-testing/scripts/batch_convert_templates.py
- Awesome QA Skills: Automation Testing Skill source:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/automation-testing
- Automation Testing Skill details:https://inaodeng.com/en/qaskills/automation-testing/