For one login requirement, a tester can ask directly, use a Prompt, or invoke a Skill. A few sample responses cannot show which works better. Model versions, requirement materials, conversation history, and run settings all affect the output. When conditions differ, you cannot isolate the effect of the approach.
To keep the terms distinct, this article uses “direct chat” for a one-off natural-language request, “Prompt” for a reusable structured template, and “Skill” for reusable instructions, steps, and supporting material loaded by a host tool. Skill capabilities and available resources vary by tool.
The examples below show what to review in each approach. Measure actual performance by running the same samples under controlled conditions.
Getting started: a first comparison
For a first comparison, one requirement is enough. Prepare shared inputs and a reviewer-only reference, then follow these steps:
- Fix the task and requirement version. State what all three approaches must do, such as “identify key test scenarios and flag unspecified behavior.”
- Start clean sessions. Do not let one approach inherit another’s conversation. Record the tool, model, settings, and versions.
- Save every output. Keep follow-up questions, human input, edits, and time spent for every run.
- Review against one checklist. Hide the approach name where possible and check evidence, omissions, unsupported claims, and usability.
- Choose for the task. Direct chat may be enough for a one-off exploration; recurring work may justify maintaining a Prompt or Skill.
Prepare shared inputs and a review reference
For the phone-number login requirement, separate two kinds of material. Shared inputs include the requirement, confirmed rules, necessary context, and output requirements that all three approaches can see. A reviewer who knows the business prepares a separate reference list of evidence-backed critical risks, open behavior, and scoring rules. If the goal is to see whether the model can find gaps, do not give it the open points in the reference list. If the goal is to test execution against a known checklist, state that as a separate task.
Finalize the reference before reading outputs so reviewers do not mistake a model suggestion for the “correct answer.” It only needs to list the important, evidence-backed conditions for this task; it does not need to cover every possibility.
Redact real phone numbers, tokens, and other sensitive material before using requirements in the comparison.
The examples below use this exercise requirement. It illustrates the review method; it is not a real product rule or complete login test scope. “Given exercise rule” is shared input, while “still to confirm” is reviewer-only reference and is not given to the model:
| ID | Given exercise rule (shared input) | Still to confirm (review reference, not model input) |
|---|---|---|
| R-01 | An SMS verification code is valid for five minutes | Is it still valid at exactly 300 seconds? |
| R-02 | The phone number is locked for 15 minutes after five consecutive incorrect attempts | What happens after attempt five, at unlock, and when the counter resets? |
| R-03 | An account is created after the first successful verification | Can a failed path leave account or session state behind? |
Keep the requirement, context, and output requirements identical across the three groups. Record newly raised questions as findings for this round. If you update the model inputs for the next round, give every group the same version; if you update the review reference, give every reviewer the same version.
Control the inputs and change only the approach
Set up three groups:
- Direct chat: ask the same testing question in concise natural language.
- Prompt: use the structured template from the previous article, “Prompt-Based Software Testing: Make AI Outputs Fit Your Testing Needs.” Article: https://inaodeng.com/en/blog/ai-testing/ai-testing-getting-started/ai-testing-prompt-test-design/.
- Skill: invoke a reviewed requirement-analysis or test-design Skill, and record its name, source, version, supporting materials, and available tools.
Use the same requirement and context, model version, and parameters in all three groups. Keep the host tool and its settings the same where possible; vary only the direct-chat, Prompt, or Skill instructions. If the tool or settings differ, disclose that limitation: the result then compares tool configurations, not the method alone. Start each run in a clean session so one group does not inherit context from another.
Record tool and model names and versions, model settings, and the time from task input to a deliverable result. Where possible, run each approach at least three times and review all outputs for consistency. Three runs provide an initial stability check, not statistical significance or evidence about other requirements. If time or budget limits the run count, report that limit.
Record one-time setup and maintenance, per-run waiting time, and human follow-up, review, and revision separately. To estimate longer-term savings, state how many tasks you assume will share the setup and maintenance cost.
Keep the task fixed: for example, ask all three approaches to list key test scenarios and include the requirement source, preconditions, expected result, and open questions. Change only whether the instruction arrives as a one-off natural-language request, a reusable Prompt, or a Skill with fixed steps and output rules.
Illustrative outputs for the same requirement
These illustrative snippets show what a reviewer might examine. They are examples, not run results or a ranking:
| Approach | Illustrative output | What the reviewer should still check |
|---|---|---|
| Direct chat | “Test a valid and expired code, a lock after five wrong attempts, and account creation after first success.” | Check whether it links the themes to R-01–R-03, distinguishes attempt four from five, explains setup, and flags unknowns. |
| Prompt | “R-01: cover codes used before and after expiry; confirm whether a code is valid at exactly 300 seconds. R-02: distinguish the fourth and fifth incorrect attempts; confirm the response after attempt five and exactly when the 15-minute lock ends.” | Traceability is clearer; still check counter reset, failure side effects, required fields, and whether the real output follows the template. |
| Skill | “Input audit: lock response and counter-reset rules are missing. Use a resettable synthetic phone number to check attempt four and five; preserve source, state evidence, and open questions.” | It combines repeatable review steps, gap detection, and handoff. Check whether it actually uses references and produces auditable fields without constraining the task too much. |
These snippets show the kinds of information each approach may produce: direct chat takes little setup, a Prompt can fix output fields, and a Skill can carry repeatable steps and boundaries.
Do not score by test-case count
| Observation | What to record |
|---|---|
| Critical gaps found | How many pre-marked important unknowns were identified |
| Unsupported conclusions | How many unconfirmed rules were stated as facts |
| Risk coverage | Whether important states, boundaries, failure paths, and side effects were addressed |
| Executability | Whether preconditions, steps, and expected results can be used in the project |
| Format compliance | Whether the output meets agreed fields and traceability requirements |
| Consistency | Whether repeated runs find similar issues and how much scores vary |
| Human effort | Time spent on follow-up, review, and revision |
If you calculate a rate, define the denominator. For example, “critical-gap recall” can be the number of pre-marked critical gaps found divided by the total number on the reference list. It measures only that list; it does not prove that every unknown risk has been found.
If completeness also matters, agree on scoring rules before the experiment and have reviewers score independently. For example, 0 could mean “not identified,” 1 “mentioned without a source or action,” and 2 “supported by evidence with a next step.” Count unsupported conclusions separately so strong scores elsewhere cannot hide them. This scale is only an example for the experiment team to adapt; write the actual criteria down before running the comparison.
Best practices for a reviewable comparison
- Before reading model outputs, separately fix the shared inputs for all three groups and the reviewer-only reference list. Record requirement sources, confirmed rules, and open questions; do not change review criteria after seeing the outputs.
- Keep the task, inputs, output fields, and model settings aligned. Save the full instructions, Prompt template, and Skill source and version used. If the tools differ, limit the conclusion to the complete tool configurations.
- Save all runs, including failures and incomplete outputs; do not select one best run to represent an approach.
- Count unsupported claims separately so a format score or a large number of cases cannot hide them.
- Have reviewers score independently before discussing disagreements. Mark major unresolved disagreements instead of averaging them away.
- Include preparation, follow-up, review, and revision time in human cost; record versions and run counts.
- Use small samples only to make a decision about the current task, not to claim that one approach is universally better.
Interpret results in the context of the task
If direct chat quickly finds the important issues and a Prompt or Skill mainly improves formatting, a simple task may not justify the upkeep. If a Skill reduces omissions and rework, include its reference and maintenance costs in the comparison. When the scores are close, choose the lighter method the team can keep using.
Review bias matters too. Hide which approach produced each result where possible, and have two reviewers use the same criteria before discussing disagreements. Keep “looks professional” separate from “has evidence and can be executed.” Familiarity with a tool should not automatically earn it a higher score.
For disputed items, return to the requirement and reference list. Record the evidence each reviewer cites and clarify the scoring rules where needed. If reviewers still define a critical gap differently, revise the list or mark the result unresolved. Do not calculate an average score until the reviewers agree on what counts.
Example comparison: fields in the three illustrative snippets
The table below inventories only the content explicitly present in the three author-written snippets. It does not assess real-world performance.
The review reference has five open points: R-01, behavior at exactly 300 seconds; R-02, the response after attempt five, when the 15-minute lock ends, and when the error counter resets; R-03, whether a failed path leaves account or session state. A point appears in the table only when the snippet explicitly marks that behavior as unknown; naming a related test scenario is not enough. An unsupported claim is an unconfirmed behavior stated as a settled rule. The table also records whether each snippet gives specific content for the four agreed fields: requirement source, preconditions, expected result, and open questions.
| Approach | Open points explicitly flagged | Unsupported claims | Fields explicitly present | What is still missing |
|---|---|---|---|---|
| Direct chat | None explicitly flagged | None seen | None provided | Lists scenario names but gives no requirement source, preconditions, expected result, or open question. |
| Prompt | R-01: behavior at exactly 300 seconds; R-02: response after attempt five and when the lock ends | None seen | Requirement source (R-01, R-02); open questions | Gives no preconditions or expected result; does not mention counter reset or failed-path side effects. |
| Skill | R-02: lock response after attempt five and counter-reset rule | None seen | Preconditions (resettable synthetic phone number); open questions | Gives no requirement ID or expected result; does not mention the 300-second boundary, when the lock ends, or failed-path side effects. |
Do not infer method quality from the number of open points in each snippet: the Prompt example foregrounds unspecified behavior, while the Skill example shows a precondition, review steps, and evidence handoff. Measure consistency and human revision cost from repeated real runs; these snippets cannot provide those measures.
Focus: Put the comparison into a quality loop
Use each comparison to define a reviewable improvement for the next round:
- Freeze the baseline: save requirement versions, shared inputs, the reviewer-only reference list, samples, scoring rules, and model settings. Record known rules separately from critical unknowns.
- Run independently: use the same task and materials for direct chat, Prompt, and Skill. Start clean sessions and keep every raw output.
- Review blind: hide approach names and apply the same checks for traceability, omissions, unsupported claims, usability, and human revision time.
- Classify issues: decide whether each issue came from the requirement, input context, Prompt or Skill instructions, model variation, or review criteria. Keep it open when evidence is insufficient.
- Change one explainable factor: for example, add a boundary check, clarify an output field, or update a reference. Record the version and reason.
- Regress on the same samples: confirm the known issue is fixed without creating new false positives. Add a new, untuned sample to check for overfitting.
For each round, retain input and method versions, model and tool, raw output, human edits, findings, and review outcome. Reassess when the method, model, or business conditions change; old results describe only the conditions under which they were collected.
FAQ
Is a Prompt or Skill always better than direct chat?
No. Recurring tasks may benefit more from consistent structure and traceable steps. Direct chat may be cheaper for a one-off, low-risk exploration. Use task samples to decide instead of inferring quality from the method’s name.
Must I run each approach three times?
Three runs can be an initial check for consistency, but they are not a universal statistical standard. Record the actual run count and limit conclusions to what the samples, model, and settings support.
What if the scores are close?
Consider task frequency, omission risk, human review cost, and maintenance capacity. If quality differences are unclear, choose a method the team can keep using and maintain, while continuing to capture failures in the quality loop.