AI coding tools can now produce software and tests faster. The more useful question is whether they know why a system behaves as it does, which risks have been tested, and what failed before.

In the Ministry of Testing session From test cases to AI-ready quality data: why structure matters, Chris Faraglia, a lead solution architect working across Sembi’s TestRail and Ranorex brands, argues for a shift from feeding AI disconnected records toward providing structured, connected, historical quality context.

The session’s argument comes first. The schemas, governance practices, AI Native QA applications, and pilot plan are my recommendations—not results the speaker reported.

AI needs connected history, not another pile of records

Requirements live in Jira, test cases in a management platform or Markdown, automated results in CI, defects in issue trackers, release history in GitHub, and crucial decisions in Slack, Teams, or people’s heads.

Each record may be accurate, yet a standalone “test failed” says little about whether a requirement changed, which build and environment were involved, whether the failure is recurring, or whether the test itself is flaky.

Data volume is not context quality. The presenter says Sembi has conducted annual software-quality surveys of several thousand users, and that its first combined Software Quality Pulse report found respondents reporting 53% of code was AI-generated or AI-assisted. The public 2026 report describes nearly 4,000 respondents across QA, security, development, and engineering leadership. It reports that an average 53% of respondents’ code had been touched by AI, 61% saw moderate-to-dramatic increases in QA demand due to AI-generated code, and 17% said AI-driven testing had a significant impact. These are respondent-reported findings in a vendor-published survey—not a global census or proof that AI alone caused the changes. Faster software production makes trustworthy quality knowledge more important.

What makes a test result useful later?

The speaker compares quality data to healthcare records. Electronic health records capture structured diagnoses, lab results, and medication information, while clinical notes explain circumstances, observations, and reasoning.

Testing needs both:

Data typeTesting examplesValue
Structured factsTest IDs, requirement IDs, environment, build, outcome, defect IDSearchability, linking, aggregation
Unstructured narrativeWhy an assertion changed, reproduction notes, risk judgments, failure analysisExplanation and nuance

Purely structured records can lose engineering judgment. Pure free text is difficult to link reliably. AI-ready data does not mean eliminating Markdown; it means identifying important facts consistently and preserving the narrative that explains them.

Connect the result to its requirement and change.

Consider what these statements leave unanswered:

  • “500 tests passed.” Which requirements, workflows, and risks did they cover?
  • “The build failed.” Is this a new failure or a recurring pattern?
  • “The test changed.” Was that driven by a requirement, a bug fix, or maintenance?
  • “A defect was found.” In which environment and build, and against which release?

A useful evidence chain links a requirement or risk to a test obligation and case; the case to its execution, environment, and build or commit; a failure to its defect, fix, and pull request; and the resulting evidence to a release and decision.

The records do not have to live in one product. They need stable identifiers, explicit relationships, and inspectable evidence. Correlation must not be confused with causation: a code merge preceding a failure is a hypothesis, not proof of root cause.

Preserve enough history to explain recurrence.

The session uses a checkout example: a requirement changes, code is merged and deployed, previously passing tests start failing repeatedly, and a related defect is filed.

A single red test only reveals a symptom. Linked execution history, environments, builds, changes, and fixes make it possible to investigate recurring defects, configuration effects, and flaky tests.

This does not require retaining every raw log forever. Teams need retention rules, deduplication, timestamps, and version ownership. Large artifacts such as traces and screenshots can be stored externally with stable references and summaries rather than being copied wholesale into a model’s context window.

Build a quality-data lifecycle

The session proposes Structure → Connect → Preserve → Enrich → Expose. Here is how to apply them.

Structure: define consistent records.

A minimum test record should have a stable ID, purpose, preconditions, steps, expected results, requirement links, risk links, and lifecycle status. Automated tests also need code locations, IDs, and clear assertion intent.

For teams using BDD, scenario descriptions, step text, and reviewer comments can preserve business language and reasoning. They should still link to stable requirement and risk identifiers; prose alone is not a reliable relationship model.

id: TEST-CHECKOUT-017
requirement_ids: [REQ-CHECKOUT-004]
risk_ids: [RISK-PAYMENT-DUPLICATE]
level: api
preconditions: "A valid cart exists; payment gateway uses a controlled test double"
expected: "A repeated submission must not create a second payment"
owner: checkout-team
status: active

This is an illustrative schema, not a prescribed format from the session. The important property is consistent semantics: status must not mean execution outcome in one system and review state in another.

Connect: link quality entities.

Connect requirements, tests, executions, defects, changes, and releases using native IDs or permanent links rather than fuzzy title matching. Support many-to-many relationships: a business risk can require validation at multiple test levels.

Preserve: retain history and decisions.

Keep meaningful execution records, failure classifications, reasons for test changes, requirement changes, and release decisions. Retired tests should retain retirement reasons and impact, rather than disappearing from the historical record.

Enrich: add domain context.

Add environment, browser, geography, feature flags, test-data version, dependency versions, framework, and business criticality where useful. Every extra field should support a concrete decision; collecting metadata indiscriminately creates maintenance cost.

Expose: make context available safely.

Use REST APIs, integrations, or MCP to make quality context retrievable by AI assistants. MCP is an access and integration protocol; it does not automatically clean bad data or infer correct relationships. Apply least privilege, sensitive-data redaction, auditing, scoped queries, and source attribution.

Put connected data to work without assuming one platform

The session calls this approach Tests as Context. Testing assets can inform requirements, design, implementation, and review—not merely validate code after it is written.

For example, before changing a checkout API, an assistant could retrieve related requirements, existing tests, past regression defects, and high-risk assertions. A QA engineer could then identify new testing obligations without rebuilding the entire context manually.

Two failure modes remain: treating outdated tests as authoritative requirements, and promoting AI-generated test suggestions into “verified evidence” without execution. Context is evidence to examine, not a conclusion that becomes true automatically.

Choose where the quality data should live.

The speaker positions test management as a quality orchestration layer connecting CI, requirements, defects, releases, and manual and automated testing. That is consistent with the TestRail product perspective and should be understood as a vendor viewpoint.

For a small repository, Markdown, test code, CI artifacts, and a lightweight index may be sufficient. Larger cross-team and long-lived datasets may justify a dedicated management platform or relationship data layer. The selection criteria are more useful than a platform-versus-Markdown argument:

  • Can quality entities be identified consistently?
  • Can relationships and version history be maintained?
  • Can updates be synchronized without duplicate manual entry?
  • Can authorized queries return traceable sources?
  • Can maintenance cost and vendor lock-in be managed?

What the Q&A and reported outcomes tell us

The question-and-answer session sharpened several practical points.

Does model choice matter more than context? In the speaker’s experiments, relevant context often mattered more than switching between specific models. A more capable model without useful context could still perform poorly; retrieving focused material through MCP helped. Model capability, context quality, token usage, and task type all matter, so this is not a claim that model selection never matters.

Can teams connect multiple context sources? Yes. The speaker described combining test management with sources such as GitHub or GitLab and CI. The goal is not to connect everything indiscriminately, but to make important relationships useful and trustworthy. My practical recommendation: do not exclude a source merely because it is different, and do not treat access as a reason to retrieve every record.

How did the speaker’s team use the approach? He described manual and automated testing, testing early and often, and bringing testing data into developer workflows through MCP. In the Q&A, he said TestRail had been releasing almost monthly, alongside early-track releases for some customers. That is a product release cadence, not a universal team delivery recommendation.

Is this test management or quality management? The speaker described a move from managing a test library toward quality orchestration: connecting requirements, builds, executions, defects, releases, and decisions so that quality work is visible across the lifecycle.

How should context follow versions? Associate tests, requirements, defects, and change history with release milestones—for example, Jira fixed versions or Git release tags. The speaker preferred traceable history across releases over maintaining arbitrary isolated context for every branch.

Is AI augmenting work or merely offloading it? The speaker framed AI as a force multiplier for existing work, not simply a way to transfer tasks away. My extension of that point is that faster code generation without corresponding verification and governance can reduce production quality; focused context can reduce rework, but it does not remove human responsibility.

When is Markdown enough, and when might a platform help? An audience member described a large defense organization considering version-controlled Markdown for requirements and quality records. The speaker’s response emphasized integrations, structured data entities, and the ability to scale; he also acknowledged his possible TestRail bias. A small team can begin with Markdown, test code, CI artifacts, and a lightweight index. He also mentioned a client with 5 TB of test data as an anecdote, not a threshold for buying a platform. These are vendor-side examples, not proof that every team needs a dedicated platform. Evaluate scale, relationships, governance, integrations, maintenance cost, and lock-in.

Read the reported outcomes as case data.

The speaker reports Sembi internal results including 36% fewer AI tokens per feature and a 100% first-run pass rate versus 40% without Tests as Context. He also described fewer bugs, higher-quality code, and more defects prevented per feature, but gave no numerical estimate for those outcomes. These are presenter-reported internal case results, not independently validated industry benchmarks. The session does not provide enough detail on sample sizes, task distribution, model settings, or statistical methods to reproduce the comparison.

To evaluate the approach locally, compare similar tasks using first valid pass rate, human rework time, escaped defects, context retrieval precision, citation traceability, and token/cost usage. Record task difficulty, rather than equating faster generation with better quality.

Start with one AI Native QA workflow

Teams building QA Skills, agents, and test-quality tools can treat quality context as a shared capability instead of duplicating knowledge inside each agent:

  1. Define a common entity model: Requirement, Risk, Obligation, Test, Execution, Evidence, Defect, Change, Release.
  2. Build relationship indexes: trace changes to testing obligations and failures back to requirements and historical evidence.
  3. Classify evidence: distinguish user-provided information, tool-observed results, AI inferences, and human-confirmed findings.
  4. Provide controlled retrieval: filter by release, module, and environment before retrieving narrative details.
  5. Evaluate context quality: measure retrieval accuracy, relationship correctness, stale-data hits, fabricated citations, and impact on decisions.

Start with one business flow—requirement change, impact analysis, suggested tests, execution, defects, and release—rather than building a large platform upfront.

A four-week pilot.

WeekWorkAcceptance evidence
1Choose a high-risk workflow; inventory requirements, tests, defects, runs, releasesSource inventory, gaps, ID conventions
2Define minimum fields and links; import recent execution historyRequirement-to-test and failure-to-build/defect queries work
3Expose read-only API/MCP retrieval with citationsAI responses include verifiable IDs and versions
4Compare tasks with and without structured contextRecorded samples, failure cases, cost and quality metrics

Test cases should not become static lists that are forgotten after execution. Connected to requirements, risks, changes, results, and decisions, they can become quality assets that AI can use and humans can audit.

AI-ready quality data is not about letting AI read more. It is about making clear what each record means, what it connects to, when it applies, and why it should be trusted.

References

Source and attribution: This article is based on Chris Faraglia’s Ministry of Testing session, the supplied notes, and the supplied transcript. The AI Native QA application, pilot plan, and illustrative schema are additional recommendations. The survey and internal performance figures are attributed to the presenter; TestRail materials are vendor resources, not neutral comparisons.

ACP (Agent Client Protocol)Active MemoryAgent AuthorizationAgent CardAgent Evals
Share