A CI pipeline can be green and still be too slow to influence a developer’s next action. Optimization starts by finding which feedback arrives too late or cannot be trusted.

Why this Skill is needed

CI Test Optimization identifies bottlenecks, redundant work, flaky feedback, selection opportunities, and safe ordering changes without weakening meaningful coverage.

What the Skill does

The CI Test Optimization Skill is useful when a team needs a structured review of feedback time, test value, flakiness, dependency, parallelism, selection, and resource cost.

Awesome QA Skills organizes Skills by language and testing stage. The series overview explains the shared installation model; this guide stays with CI Test Optimization.

A practical standard for this Skill is simple: A good optimization preserves the reason a test exists. It moves expensive feedback to the decision where it matters, rather than deleting anything that makes the pipeline uncomfortable.

Use it when the team needs a repeatable review of feedback time, test value, flakiness, dependency, parallelism, selection, and resource cost—especially when several evidence sources disagree or a handoff must explain what remains unverified.

What it does not do

The CI Test Optimization Skill can organize material, evidence, and next actions for Bottleneck. It does not replace:

  • Confirmation of rules, scope, and risk by the accountable domain owner.
  • A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
  • Authorization for release, compliance, production actions, or residual-risk acceptance.
  • A Human decision when supplied sources conflict.

What it checks

The value of this Skill is not another keyword list. It connects each focus area to an observable input, a judgment, and a way to close the loop. Start with a small matrix based on the source Skill’s output contract:

FocusQuestion before analysisHandoff output
BottleneckQueue, setup, execution, retry, or artifact timeMeasured evidence
OptimizationSelection, shard, order, cache, or environment changeCoverage and risk guardrail
Follow-up metricTime, trust, failure detection, or costOwner and review date

If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.

Audit inputs before you start

Before analyzing Bottleneck, classify the input into six evidence states. A gap is not automatically a failure, but it must not disappear inside the conclusion.

StateMeaningHow this Skill should handle it
knownDirectly supported by the supplied materialKeep the source, version, and time with the judgment
missingNeeded for this pass but not suppliedName the smallest evidence action and limit the conclusion
conflictingSources disagreeShow both sources and route the conflict to an owner
stalePresent but outside the relevant version or time windowMark freshness; old evidence is not current proof
out_of_scopeRelated but excluded from this passKeep the boundary explicit
assumptionsTemporarily adopted to continue analysisState how and when the assumption will be checked

Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.

From problem to structured Finding

Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.

Keep the decision layers separate

For Bottleneck, do not compress four different kinds of language into “recommended to pass”:

LayerHow to write itApplication here
FactWhat the supplied material directly showsCite the source, version, input, or run record for the focus
Evidence-backed InferenceWhat several facts support togetherShow the inference chain and retain uncertainty
RecommendationThe smallest next actionName the evidence, review, execution, or regression path
Human DecisionWhat an accountable person must decideLeave scope, risk acceptance, resources, and release meaning to the owner

A complete case

This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.

Input

MaterialWhat to provideWhat to do when it is missing
Pipeline historyDuration, queue time, retry rate, failure type, and changed-file scopeUse a time window, not one slow run
Suite mapDependencies, tags, environment, data, and coverage intentDo not remove a test because it looks similar
Decision needPR gate, merge gate, nightly confidence, or release gateOptimize for the actual decision

Use a request like this:

Use the ci-test-optimization Skill.

Task: reduce pull-request feedback time for a monorepo whose browser, API, and contract suites share runners
Inputs: [requirements, versions, links, logs, reports, or data paths]
Scope: [included and excluded objects]
Unknowns: [missing environment, accounts, data, or permissions]
Expected output: [risk-ordered findings, evidence status, and next actions]

Audit the inputs first. Separate facts, assumptions, and open questions. Do not claim execution without a run record.

Analysis

Start with a bounded pass—reduce pull-request feedback time for a monorepo whose browser, API, and contract suites share runners. Compare PR runs over four weeks, then split browser tests by stable duration and dependency. Keep contract tests before UI tests when they catch interface breaks earlier, and measure whether the new shards reduce queue time without increasing reruns.

The handoff should preserve the input version, time window, evidence index, owner, and next validation action. The Skill can organize uncertainty; it cannot manufacture the missing artifact.

Finding

Example finding: turn one problem into a handoff

The field example below shows the recording pattern; it is not an execution result.

If the supplied material cannot prove that Bottleneck meets its contract, write the finding like this. It does not invent the missing rule or turn missing evidence into a failure.

FieldExample wording
Source and scopeRecord the requirement, version, environment, and the concrete object for Bottleneck
FindingThe condition or result for Bottleneck is not yet traceable to evidence
Evidence statemissing / assumptions; use conflicting when sources disagree
Impact and priorityName the affected user, journey, or delivery decision without inflating severity
Owner and Human decisionAsk the product, engineering, security, or test owner to confirm the rule and trade-off
Action and close conditionAdd the smallest missing evidence; close only when source, judgment, and owner can be reviewed
ValidationName one repeatable check, query, or run and retain the raw artifact

The point is to let the next person walk from the finding back to the source and run an action that can change the decision.

Decision

The accountable owner confirms the decision question and risk trade-off; the Skill does not make that choice.

Validation

Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.

How a Finding enters the next stage

Handoff output

Output fieldWhy it existsExample status
BottleneckQueue, setup, execution, retry, or artifact timeMeasured evidence
OptimizationSelection, shard, order, cache, or environment changeCoverage and risk guardrail
Follow-up metricTime, trust, failure detection, or costOwner and review date

Do not write “passed” without a run record, query result, or source artifact. A good optimization preserves the reason a test exists. It moves expensive feedback to the decision where it matters, rather than deleting anything that makes the pipeline uncomfortable.

Next-stage route

At minimum, hand off the source, evidence state, owner, close condition, and validation action; the next-stage conclusion remains bounded by the evidence state.

How to prepare better input

If the first request contains only a one-line goal, keep the output limited. Add the source version, affected objects, environment, known defects, and decision owner to move from a plausible checklist to a useful review.

A richer input changes the answer here because feedback time, test value, flakiness, dependency, parallelism, selection, and resource cost must be tied to evidence rather than inferred from a familiar pattern.

Working with other Skills

Common traps

  1. Optimizing average duration while p95 remains unusable.
  2. Treating retries as speed improvements.
  3. Selecting tests from file names without a coverage or dependency map.

Install and invoke

Install the individual Skill. The series overview carries the longer installation explanation.

npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ci-test-optimization -g

After installation, invoke it with the ci-test-optimization Skill and attach the real project material.

FAQ

Can a flaky test be removed from CI?

Remove it from a blocking gate only with an owner, replacement evidence, and a visible follow-up—not by hiding the failure.

Does parallelism always improve feedback?

No. Queue, shared data, runner contention, and setup overhead can make it slower or less trustworthy.

The example above is a design and review pattern, not an execution result. Keep static analysis, runtime evidence, human approval, and release acceptance separate.

References

Source Skill and execution contract

The complete execution contract lives in the CI Test Optimization prompt. Read it before invoking the Skill; the prompt defines the detailed workflow and output contract. The source directory contains the entry point and supporting assets where they exist.

The entry point centers feedback time, test value, flakiness, dependency, parallelism, selection, and resource cost. Keep its decision boundary visible and do not turn a static design into an execution claim.

ACP (Agent Client Protocol)Active MemoryAgent AuthorizationAgent CardAgent Evals
Share