A safe-looking answer can still cause harm when it omits uncertainty, exposes private context, or gives a user a shortcut around a control.
Why this Skill is needed
AI Safety Testing turns a versioned safety policy into reviewable scenarios for harmful-content categories, refusal and redirect behavior, privacy, escalation, and Human risk decisions. A checklist alone cannot approve a release.
What the Skill does
Use this Skill to compare policy-defined allow, restrict, refuse, or escalate branches against concrete prompts, outputs, tool actions, and evidence. Keep safe task completion and over-refusal in scope alongside harmful-response handling.
Awesome QA Skills organizes Skills by language and testing stage. The series overview explains the shared installation model; this guide stays with AI Safety Testing.
Use it when policy versions, model responses, privacy rules, or escalation ownership disagree, or when an evidence handoff must make open safety decisions visible.
What it does not do
The AI Safety Testing Skill can organize policy-backed scenarios, evidence, and next actions for each AIS-## finding. It does not replace:
- Confirmation of rules, scope, and risk by the accountable domain owner.
- A real environment, account, dataset, log, or run record; static analysis does not become runtime evidence by itself.
- Authorization for release, compliance, production actions, or residual-risk acceptance.
- A Human decision when supplied sources conflict.
What it checks
The value of this Skill is not another keyword list. It connects each focus area to an observable input, a judgment, and a way to close the loop. Start with a small matrix based on the source Skill’s output contract:
| Policy area | Map from the source | Evidence to review |
|---|---|---|
| Decision branch | Policy ID/version and audience → category/condition → allow, restrict, refuse, or escalate | Expected branch, allowed content, and prohibited effects |
| Harmful and benign neighbors | Pair policy-defined harmful prompts with nearby allowed requests | Harm handling, allowed-task completion, and over-refusal assessed separately |
| Refusal and redirect | Check the whole response and any tool action, not just its opening phrase | Content, action, classifier/reviewer evidence where applicable |
| Privacy and escalation | Sensitive context, disclosure limits, trigger, owner, pending Human decision | Privacy-safe evidence, stop condition, and handoff |
If a row has only a conventional expectation and no source or validation method, keep it open instead of turning it into a pass.
Audit inputs before you start
Before analyzing a safety decision, classify the input into six evidence states. A missing policy boundary remains unknown; it cannot become approval by assumption.
| State | Meaning | How this Skill should handle it |
|---|---|---|
| known | Directly supported by the supplied material | Keep the source, version, and time with the judgment |
| missing | Needed for this pass but not supplied | Name the smallest evidence action and limit the conclusion |
| conflicting | Sources disagree | Show both sources and route the conflict to an owner |
| stale | Present but outside the relevant version or time window | Mark freshness; old evidence is not current proof |
| out_of_scope | Related but excluded from this pass | Keep the boundary explicit |
| assumptions | Temporarily adopted to continue analysis | State how and when the assumption will be checked |
Keep the input version, scope, environment, evidence locations, and accountable owner together. Without a run record, deliver analysis, design, or a validation plan—not an execution pass.
From problem to structured Finding
Connect the source, scope, evidence state, analysis, owner, action, close condition, and validation before writing the conclusion. The case below keeps this Skill’s identifier and domain context.
Keep the decision layers separate
For Safety finding, do not compress four different kinds of language into “recommended to pass”:
| Layer | How to write it | Application here |
|---|---|---|
| Fact | What the supplied material directly shows | Cite the source, version, input, or run record for the focus |
| Evidence-backed Inference | What several facts support together | Show the inference chain and retain uncertainty |
| Recommendation | The smallest next action | Name the evidence, review, execution, or regression path |
| Human Decision | What an accountable person must decide | Leave scope, risk acceptance, resources, and release meaning to the owner |
A complete case
This case follows Input, Analysis, Finding, Decision, and Validation. When material is incomplete, keep missing, conflicting, or assumptions visible instead of turning them into a pass.
Input
| Material | What to provide | What to do when it is missing |
|---|---|---|
| Policy source | Policy ID/version, audience, categories, allowed/restricted/refused actions, privacy and escalation rules | Keep the expected branch unknown if the policy is missing or stale |
| Prompt/fixture set | Policy-defined harmful cases and neighboring benign requests, with sample version | Mask sensitive context; do not invent categories or labels |
| Observations | Full response, tool actions, policy/model version, classifier or reviewer result | A refusal prefix or unversioned sample cannot establish the branch |
Use a request like this:
Use the ai-safety-testing Skill.
Task: design a policy-grounded review for a financial assistant asked for a personalized high-risk investment recommendation with incomplete user data
Inputs: [requirements, versions, links, logs, reports, or data paths]
Scope: [included and excluded objects]
Unknowns: [missing environment, accounts, data, or permissions]
Expected output: [risk-ordered findings, evidence status, and next actions]
Audit the inputs first. Separate facts, assumptions, and open questions. Do not claim execution without a run record.
Analysis
For this example, first require the applicable policy ID/version and its rule for personalized high-risk advice with incomplete suitability data. If the policy defines a restricted or escalation branch, make that the expected criterion; do not invent a referral, refusal, or disclosure rule. Pair the policy-defined harmful request with a nearby allowed general-education question. Review the complete answer and any tool action, verify that sensitive input is not unnecessarily repeated, and record any escalation cause and responsible Human role. Without the policy, keep both expected branches UNKNOWN; any unrun scenario is NOT_RUN.
The useful handoff keeps the input version, time window, evidence index, owner, and next validation action together. The Skill can organize uncertainty; it cannot manufacture the missing artifact.
Finding
Example finding: turn one problem into a handoff
The field example below shows the recording pattern; it is not an execution result.
Each AIS-## should connect one policy object and trigger to its expected concern, evidence, owner, and close condition. This example leaves the policy itself missing and makes no safety judgment.
| AIS-01 field | Example record |
|---|---|
| Object and source | Personalized high-risk advice with incomplete suitability data; policy ID/version not supplied (missing) |
| Trigger and expected concern | Trigger is the incomplete profile; expected allow/restrict/refuse/escalate branch remains UNKNOWN until policy is supplied |
| Evidence state | No prompt/response, tool-action record, or approved policy excerpt supplied; no execution claimed |
| Impact and priority | Keep impact and P0/P1/P2/P3 NOT_SCORED until the affected user, likelihood, detectability, or decision cost are evidenced and reviewed by the safety owner |
| Owner and close condition | Safety/policy owner confirms audience, branch, allowed content, and escalation responsibility |
| Validation | Pair the policy-defined high-risk case with a nearby benign control; capture full outputs and tool actions, apply the policy’s stop condition, and retain NOT_RUN until executed |
The point is to let the next person walk from the finding back to the source and run an action that can change the decision.
Decision
The safety/policy owner confirms the expected branch and escalation responsibility. A Human alone accepts residual risk, authorizes an exception, approves rollback, or makes the final safety judgment.
Validation
Before closing the finding, run the stated validation and retain the raw artifact. Without an execution record, the status remains unverified.
How a Finding enters the next stage
Handoff output
| Output | Required evidence | Example status |
|---|---|---|
| Policy map | Policy ID/version, audience, category/condition, expected branch, allowed content, prohibited effects | UNKNOWN where policy is absent |
AIS-## finding | Object, source, trigger, rationale, evidence state, impact/priority, owner, close condition, validation | missing or unassessed as supported |
| Scenario and decision | Preconditions, stimulus, criterion, evidence, stop condition, residual risk, Human decision | NOT_RUN without execution evidence |
Do not treat refusal rate or a refusal prefix as a safety result. Compare the full harmful-path response and tool activity with policy, separately check whether the benign neighbor still completes its allowed task, and record privacy exposure and escalation ownership. Only the accountable Human can accept residual risk or approve an exception.
Next-stage route
Hand off policy version, fixture/sample version, expected and actual branch, evidence state, owner, stop condition, and the next validation action. Keep unrun scenarios NOT_RUN and unresolved policy edges UNKNOWN.
How to prepare better input
If the first request contains only a one-line goal, the output should stay limited. Add the source version, affected objects, environment, known defects, and decision owner to move from a plausible checklist to a useful review.
Working with other Skills
- Prompt Injection Testing:checks adversarial attempts to bypass controls.
- AI Feature Testing:covers feature behavior and degradation.
- Security Testing:connects safety findings to security risk.
Common traps
- Testing only prohibited keywords.
- Treating a long disclaimer as a safe answer.
- Ignoring privacy leakage because the output contains no obviously harmful instruction.
Install and invoke
Install the individual Skill. The series overview carries the longer installation explanation.
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-safety-testing -g
After installation, invoke it with the ai-safety-testing Skill and attach the real project material.
FAQ
Can a safety review prove the model is safe?
No. It can bound the tested risks and show remaining evidence gaps; safety is an ongoing control problem.
What if policy owners disagree?
Preserve both interpretations, identify the decision owner, and keep the case unresolved until the policy is explicit.
The example above is a design and review pattern, not an execution result. Keep static analysis, runtime evidence, human approval, and release acceptance separate.
References
Source Skill and execution contract
The complete execution contract lives in the AI Safety Testing prompt. Read it before invoking the Skill; the prompt defines the detailed workflow and output contract. The source directory also contains the entry point and supporting assets where they exist.
The entry point centers harm categories, refusal behavior, privacy boundary, misuse path, escalation, and monitoring. Read the supporting prompt before writing a conclusion, and keep the Skill’s decision boundary visible.
Reference links
-
AI Safety Testing:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-safety-testing/prompts/ai-safety-testing.md
-
Awesome QA Skills: AI Safety Testing source:https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/ai-safety-testing
-
Awesome QA Skills GitHub:https://github.com/naodeng/awesome-qa-skills
-
AI Safety Testing details:https://inaodeng.com/en/qaskills/ai-safety-testing/