Automation ROI analysis compares a defined automation candidate with its manual alternative over the same time window and cost unit. A useful decision includes setup, execution, review, and change-driven maintenance; saved execution minutes alone do not show whether the investment pays back.
Why this Skill is needed
A quarterly workflow may be too infrequent to justify a brittle end-to-end suite, while a stable calculation check may still repay a smaller automation investment. The decision depends on supplied frequency, risk, effort, maintenance, and benefit evidence—not on a general preference for automation.
Define the candidate and cost boundary
First state what could be automated: a calculation invariant, data preparation, a full user journey, or a report comparison. Compare that candidate with alternatives such as simplifying the workflow, sampling, keeping specialist review manual, or deferring the investment.
| Cost or benefit | Record separately | Evidence to request |
|---|---|---|
| Build | One-time design, implementation, fixtures, and onboarding | Work estimate or recorded effort |
| Run | Machine occupancy and any human review | Run count, duration, review effort, and the unit used |
| Maintenance | Change-driven repair, data upkeep, diagnosis, and ownership | Change history and observed maintenance effort |
| Manual alternative | Preparation, execution, review, and rework | Time records or a bounded estimate with its source |
| Benefit | Risk reduction, feedback speed, or avoided decision delay | Defect history, exposure, or decision evidence |
Do not add human hours, machine occupancy, and money as if they were the same unit. Convert them only with project-supplied rates; without a monetary conversion, report labor separately from machine time.
Compare like with like
Fix the candidate, time window, and units before calculating. For a single window, the source Prompt defines:
net saving = runs × (manual cost per run − automated cost per run)
− maintenance cost − build costAll terms must use the same unit and time window. State the numerator and denominator if reporting an ROI ratio; the Skill does not prescribe a universal financial definition. If the inputs support labor hours but not money, report labor-hour savings rather than a dollar return.
For recurring net saving greater than zero, estimate break-even windows as the build cost divided by recurring net saving per window, rounded up to a whole window. If recurring net saving is zero or below, report that there is no break-even under those assumptions.
Then vary run frequency and change-driven maintenance using project-supplied low, base, and high cases. Show where the sign or recommendation changes. Missing rates or effort data mean formulas and conditional ranges, not a precise return or automatic adoption decision.
Audit the evidence before ranking
The source Prompt asks for known, missing, conflicting, stale, out_of_scope, and assumptions. For ROI work, comparability matters as much as completeness.
| State | Example in an automation decision | Treatment |
|---|---|---|
| known | A versioned release calendar and recorded manual effort | Cite the period and source |
| missing | Build effort, review time, or maintenance after product changes | Leave that term open; do not treat it as zero |
| conflicting | One estimate counts specialist review while another omits it | Show both definitions and resolve the unit |
| stale | Run frequency comes from a retired release cadence | Do not project it into the new horizon without confirmation |
| out_of_scope | Production automation or business risk acceptance is excluded | Keep the decision boundary explicit |
| assumptions | A first-pass estimate uses an owner-approved frequency range | State the validation method and which result depends on it |
Separate confirmed facts, evidence-backed inferences, candidate recommendations, and Human decisions. Do not infer that correlation between automation and fewer defects proves the automation caused the reduction.
Example: a quarterly specialist report
Consider an example request to assess a tax-report workflow that runs quarterly, changes often, and needs specialist data. Those details describe the scenario only; they provide no build estimate, run record, failure rate, review effort, maintenance history, or financial conversion.
A useful first pass would compare bounded candidates—for example, checking stable calculation rules versus automating the entire report journey—without deciding which to adopt. The comparison needs one agreed horizon, the manual and automated effort per run, specialist review, build work, change-driven maintenance, and any supplied rates. If the data are absent, the result remains unscored.
ARO-01 candidate: maintenance omitted from the comparison
| Finding field | Example record |
|---|---|
| Object and applicability | The proposed tax-report automation candidate, if its owner is comparing automation with a manual or sampled alternative |
| Trigger | A decision estimate is prepared without a time window or change-driven maintenance term |
| Expected concern | The comparison may overstate savings if specialist review, diagnosis, or repair after workflow changes is omitted |
| Source and evidence state | The example request gives cadence and change context only; cost and run evidence are missing, so mark UNASSESSED / NOT_SCORED |
| Impact, priority, and owner | No priority until business impact and evidence strength are reviewed; assign the automation owner with the finance or process owner as decision partner |
| Validation and close condition | Put each effort or cost in a sourced unit, calculate the same-window cases, and vary frequency and maintenance in supplied low/base/high scenarios. Close the analysis when another reviewer can reproduce the calculation and a Human has chosen among the options. |
This is a review candidate, not a calculated ROI or an instruction to automate. The Skill can prepare the comparison; it does not execute the workflow or approve the investment.
Turn the analysis into a decision
The output should show the candidate, alternatives, source inputs, time horizon, costs, benefits, sensitivity, evidence gaps, and residual risk. A recommendation may be to automate, simplify, sample, keep manual, or defer. State the conditions that would change the recommendation and set a review trigger with an owner.
Prepare better input
Provide a stable candidate definition, release or business cadence, a comparable observation period, manual and machine effort, review and maintenance records, benefit evidence, and project-approved conversion rates. Mask sensitive data and state whether numbers are observed, estimated, or assumed.
Work with related Skills
- Automation Testing turns an approved decision into an automation strategy.
- Test Maintenance Cost Analysis examines recurring maintenance effort.
- Test Effectiveness Analysis checks whether feedback finds useful problems.
Common mistakes
- Counting saved execution time as the entire benefit.
- Treating missing maintenance or review cost as zero.
- Combining labor, machine time, and money without a sourced conversion.
- Presenting a point estimate without its horizon or sensitivity.
Install and invoke
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/automation-roi-analysis -gInvoke automation-roi-analysis with the candidate, comparable time window, cost units, source evidence, and decision alternatives.
FAQ
Do I need exact financial numbers?
No. Use sourced ranges and explicit assumptions. If monetary conversion is unavailable, report labor separately and show what is still missing.
What if risk reduction cannot be measured directly?
Use a bounded proxy such as defect history, exposure, or decision delay, explain its source, and keep the causal claim unproven.
References
The Automation ROI Analysis Prompt defines the ARO-## finding fields, calculation boundary, and output order. The source directory contains the Skill entry point and supporting material.