Generative AI weakens three assumptions that traditional test automation often relies on: we control the input, we know the expected output in advance, and unchanged code produces repeatable results. Real users paste long emails, make mistakes, mix languages, ask several questions at once, and change direction mid-conversation. Several answers may be valid, and a model, prompt, retrieval, or tool change can alter behaviour even when application code stays the same.

Testing now supports a different decision: whether the remaining risk is acceptable. No test can prove that risk is absent for every input, so the strategy gathers trustworthy evidence for that call. The Quality Duck organizes the work into five connected shifts: input, verification, safety, baseline, and observability.

Source: The Quality Duck, “How to build a strategy for testing AI,” published October 7, 2026. The four-week pilot and release checklist below are practical QA extensions, not a prescribed framework from the source.

1. Input shift: exercise more than the prompts we imagined

A handful of hand-written prompts mostly covers what the test team already expects. The input shift borrows personas from exploratory testing: describe different kinds of users, give each a goal, and let a model carry out multi-turn conversations with the product.

A useful persona records five things: context (what the user knows and has tried), style (language, vocabulary, typos, and verbosity), behaviour (patience, cooperation, or distraction), constraints (what the user does not know or will not disclose), and goal (the outcome they want).

For example, a multilingual, impatient customer may have already tried to request a refund, omit an order number, and ask for a human after three unsuccessful turns. A generator can produce hundreds or thousands of conversations across personas and goals, but volume alone does not prove useful coverage. Keep persona definitions stable, regenerate conversations as the product changes, and give each simulated user a goal and a stopping rule—such as reaching the goal or exhausting a fixed number of turns—so runs remain interpretable.

Coverage should be deliberate. Combine the user types, tasks, languages, difficulty, and tone that matter to the product. Include awkward cases: the confused newcomer, the expert who provides too much detail, the angry customer, and the person who gives only half the information. Seed personas with anonymised real examples where possible, then read a sample of generated conversations. Synthetic users often sound more coherent and polite than real ones. They broaden coverage; they do not replace production behaviour.

2. Verification shift: judge criteria, not one approved sentence

Exact-string assertions break down when several responses can be correct. An LLM judge can assess a response against a written rubric using the user request, the answer under test, and relevant source context. Criteria might concern accuracy, grounding in the supplied evidence, relevance, or tone. The judge returns a verdict and evidence that can be inspected or aggregated.

Keep each judge focused on one criterion. Ask whether a response made an unsupported claim, for example, rather than asking whether it was “good.” Prefer narrow questions, simple pass/fail or clearly defined scales, reasoning before the verdict, and structured output. A judge cannot verify a policy fact if the policy was never provided to it. Record the judge model, settings, rubric version, and output schema so results can be compared. The source contrasts revising a focused judge—which may take an afternoon as criteria change—with maintaining a fixed set of 2,000 expected responses. Lowering judge temperature can reduce run-to-run variation, though it cannot remove it.

{
  "criterion": "unsupported_claim",
  "passed": false,
  "evidence": "The answer promises a refund within 24 hours, but the policy gives no such timeframe.",
  "rubric_version": "v1.2"
}

Judges bring their own failure modes: they may reward longer answers, favor the first response in a comparison, prefer outputs from their own model family, or trust confident wording. For pairwise evaluation, run both response orders and average the results; where the risk warrants it, compare across model families; and make the rubric penalize irrelevant verbosity.

Human calibration is the part teams often skip. Have domain experts label a sample with the same rubric, run the judge on those cases, measure agreement, and inspect disagreements. Revise ambiguous criteria, then repeat. Recheck after a model, prompt, or product change. The judge is software under test, not an authority above review.

3. Safety shift: decide where people must review

Benchmarks and judges increase scale, but neither owns the consequences of a release. Legal, medical, financial, safety-related, and other high-impact outputs need an explicit human-oversight policy.

A sensible starting point is to review every issue a judge flags, randomly sample conversations it accepts, sample traffic that automated evaluation did not cover, and target cases raised by deterministic signals—an unusually long conversation, for example. Review rates should respond to evidence and risk. Increase sampling after a model, prompt, retrieval, tool, or feature change; lower it only when the evidence supports doing so.

Reviewers need the request, response, retrieved evidence, judge reasoning, and conversation history together. Give them examples and shared guidance, check agreement between reviewers, and rotate short review sessions to limit fatigue. Their work is broader than approving or rejecting: they assess usefulness and business fit, find failure modes the rubric did not anticipate, calibrate judges, and turn confirmed failures into regression cases. Name the person or role that can accept residual risk or block release.

4. Baseline shift: compare releases against reviewed cases

A new model or prompt can improve some answers and degrade others. A golden dataset gives the team a reviewed baseline: real production traces, support tickets, user feedback, expert-authored edge cases, and past failures, each paired with expected behaviour or an evaluation rubric.

Tag cases by task, language, risk, and failure type. Make sure the set represents core tasks, edge cases, known past failures, and out-of-scope requests. Version the dataset and scoring criteria; hold some cases back from tuning to reduce overfitting. Add every serious production failure to the regression set. A few hundred trusted cases are more useful than thousands of unchecked ones.

For cases with meaningful run-to-run variation, repeat the evaluation and compare pass rates and distributions, not a single result. Set release thresholds according to consequence: a 95% pass rate may be reasonable for tone, but it is not automatically acceptable when an error could cause harm. A critical regression should block an automatic release or trigger an explicit risk decision.

A release card makes that decision reviewable. Record model, prompt, and retrieval versions; the golden-dataset version; key pass rates and changes from baseline; high-risk failures and human findings; the monitoring and rollback plan; and the person who accepts residual risk. Scores can come from code, judges, people, or a combination—the method should be clear for each criterion.

Keep the evaluation outputs, human decisions, and unexpected failures as well as the summary scores. They become reusable evidence for improving the dataset, calibrating judges, and changing the product. Confidence is the main outcome; better evaluation data is a valuable by-product.

Keep the release card short enough to use:

  • Model / prompt / retrieval versions:
  • Golden dataset version:
  • Critical pass rates and baseline changes:
  • High-risk failures and reviewer decisions:
  • Production monitoring and rollback plan:
  • Residual-risk owner and sign-off:

5. Observability shift: follow the request after release

Pre-release testing cannot cover every real interaction. Users, data, models, and dependencies change, so quality work continues in production.

Tracing connects the steps in a request: the original input, assembled prompt, retrieved material and citations, model calls, tool calls and results, final answer, latency, token use, and cost. When an answer goes wrong, the trace helps locate whether the problem came from retrieval, orchestration, a tool, or generation. Connected spans, version information, user feedback such as ratings, edits, or abandonment, and online evaluation results make the record more useful. Traces can also feed sampled online judging and alerts, route cases to review queues, grow the evaluation dataset, and reveal drift over time.

OpenTelemetry’s GenAI semantic conventions and tools such as Langfuse, LangSmith, and Arize Phoenix are implementation options; the instrument matters less than being able to follow the evidence across the request. Traces may contain personal or confidential data, so decide on redaction, sampling, retention, access control, and legal constraints before collecting them.

These shifts fit the testing models teams already use

The original article places the five shifts within existing approaches. Personas and early evaluations reinforce shift-left work. Judges and baselines can run in a delivery pipeline, while tracing and sampled online evaluation continue after release. Together, these activities support holistic testing across the lifecycle. Teams can extend their current workflow to consider quality earlier, evaluate continuously, and gather evidence throughout the product lifecycle.

A practical starting order

The source suggests building visibility first, then a small baseline of roughly 50–100 real cases reviewed by someone familiar with the domain. Add one or two narrowly scoped judges and calibrate them against human labels. Write a risk-based human-review policy, use synthetic personas to address gaps the first evaluations expose, and add release gates and online evaluation when the measures are trustworthy. Bring human review forward for a high-risk customer-facing assistant; a lower-risk internal summarizer may start lighter.

Generating thousands of conversations before the team can judge their quality creates volume without evidence that coverage improved.

A four-week pilot for a QA team

This plan is an engineering extension of the source ideas.

Week 1 — Make one workflow observable. Choose a high-value user journey. Add correlated trace identifiers, record model and prompt versions and tool calls, define redaction, and collect real failures.

Week 2 — Create a baseline. Have QA, product, and domain experts review 50–100 cases. Record acceptable and unacceptable behaviour, risk and failure labels, and the dataset version.

Week 3 — Calibrate evaluation. Build one or two judges for checkable criteria such as unsupported claims or scope violations. Compare their decisions with human labels, prioritising dangerous false negatives.

Week 4 — Connect evidence to release. Add agreed regression thresholds to CI, prepare a release card, name the risk owner, and route sampled production failures into review and dataset updates.

Track quality (critical-criterion pass rates and severe production failures), evaluation reliability (human–judge and reviewer agreement, plus coverage), and operating cost (latency, tokens, and review effort). No single aggregate score should make the release decision.

Readiness checklist

Use each item to record a result, evidence link, and owner. “Mixed” is useful information; it should lead to a specific gap, not a padded score.

Input

  • Each persona describes its context, style, behaviour, constraints, and goal.
  • Tests include multiple languages, typos, changing goals, and difficult-to-communicate users.
  • Synthetic conversations are sampled by people and calibrated with real feedback.

Verification

  • Each judge evaluates one explicit criterion.
  • Rubrics, models, settings, and output formats are versioned.
  • Judge results are compared with human labels and disagreements are reviewed.

Safety

  • High-risk responses and all judge-flagged failures have a human-review path.
  • Accepted responses and traffic outside automated evaluation are sampled.
  • Release-blocking authority and residual-risk ownership are explicit.

Baseline

  • The golden dataset includes real, expert-authored, and historical failure cases.
  • Cases have useful labels; the dataset is versioned and includes held-out examples.
  • High-variance cases are repeated and compared by pass rate or distribution.
  • Critical thresholds appear in the release card or CI gate.

Observability and feedback

  • Traces connect input, prompt, retrieval, model, and tool activity.
  • Versions, latency, token use, cost, and user feedback are recorded where appropriate.
  • Redaction, access, retention, and sampling rules are documented.
  • Production failures can return to the dataset and trigger regression checks.

The most important next step is the largest evidence gap the team can reduce in the next two weeks. Put it on the release card with an owner and a way to verify the change:

Our largest evidence gap is: ______. The smallest meaningful action we can take in the next two weeks is: ______.

Confidence, not certainty

One green checkmark cannot prove that a generative AI product is safe or correct in every situation. A useful strategy can explain what was tested, how answers were judged, which conclusions people reviewed, what changed from the previous version, and how new failures will be detected. That is evidence a team can use to make—and revisit—an accountable decision about residual risk.

References

ACP (Agent Client Protocol)Active MemoryAgent AuthorizationAgent CardAgent Evals
Share