Failover Testing examines whether an approved primary-to-standby transition moves requests to a ready path while preserving single-writer authority and business state. A route change alone does not prove that writes are safe or that users can finish the affected workflow.
Define the cutover contract
Start from the supplied architecture and business requirements. Record which node is primary, which is standby, what triggers automatic or manual action, how health is detected, who may write during each phase, and what “business ready” means.
| Phase | Evidence to compare |
|---|---|
| Detection and decision | Health signal, trigger rule, election or operator decision |
| Isolation and write authority | Whether the old primary is fenced before a new writer accepts commits |
| Route and reconnect | Request destination, stale routes, connection-pool recovery |
| Readiness and return | Business path health, replication lag, reconciliation, failback gate |
If the business recovery-time objective or accepted loss boundary is missing, keep it open. Do not infer zero loss or a universal threshold.
Follow one business record across the switch
Trace a supplied request or ledger item through the pre-switch, transition, and post-switch evidence. Compare request IDs, commit records, retries, webhook deliveries, and the resulting business state. Look for an old primary that can still write, a stale route that reaches it, duplicate processing, and an outcome that is unknown because acknowledgement was lost.
Assess availability and single-writer consistency separately. Traffic can reach the standby while two writers still accept commits. A later reconciliation or safe failback also needs its own evidence; success of the initial route change does not establish it.
Worked example: read write authority from a timeline
This is illustrative. Assume request req-example-01 was sent to the primary before cutover, then timed out at the caller and was retried against the standby. The review needs to join the original request, timeout record, old-primary fencing event, new-primary commit, and final business ledger.
| Evidence combination | Bounded interpretation | Still needed |
|---|---|---|
| Only the new-primary route is recorded; there is no old-primary fencing evidence | The request may have reached the standby, but single-writer authority is unproven | Old-primary fencing event, timestamps, and commit records |
| Both nodes accept writes for the same business intent | There is a split-write risk in that window; record the scoped discrepancy | Reconciliation, affected records, and an owner decision |
| The old primary is fenced and the retry reconciles to one record | This supports single-writer behavior for the supplied request and evidence scope | Broader scenarios still need the approved validation plan |
Distinguish failover from disaster recovery
Failover focuses on the live primary/standby transition, routing, write authority, and recovery of service. Disaster recovery also covers a disaster boundary, recovery objectives, backup evidence, and dependencies needed to restore the business path. A failover step may appear inside a recovery plan, but it does not prove that backups can be restored.
Record an FOV-## finding
Capture the affected object and rule, source, trigger or applicability, expected concern, evidence state, impact and priority, owner role, close condition, and validation method. Keep facts, inferences, candidate checks, and Human decisions separate. A timeline should identify nodes, request routes, and write authority at each phase.
Prepare a bounded validation
Use supplied diagrams, requirements, logs, and ledger evidence for a static review. Propose the smallest authorized, isolated, reversible check and its stop condition; do not inject a failure, access live dependencies, or describe a candidate as an executed test.
Related Skills
- Disaster Recovery Testing: https://inaodeng.com/en/qaskills/disaster-recovery-testing/ covers recovery objectives, backups, and restore paths.
- Resilience Testing: https://inaodeng.com/en/qaskills/resilience-testing/ reviews broader service behavior under disruption.
Install and invoke
Install the Skill:
npx skills add https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en --skill failover-testingInvoke failover-testing with the architecture, primary/standby roles, approved triggers, business readiness rule, and available route or ledger evidence:
Review this redacted primary/standby cutover timeline with failover-testing.
Architecture and rules: [nodes, trigger/election rule, write authority, business-ready and failback gates]
Records: [request IDs, routing, old-primary fencing, retries, commits, ledger reconciliation]
Judge availability and single-writer consistency separately. List candidate FOV-## findings, missing evidence, and the smallest isolated validation steps. Do not equate a successful route change with zero data loss or business recovery.FAQ
Does reaching the standby prove that failover was safe? No. Verify that the old primary was fenced and reconcile commits from both nodes with the final business state.
Are failover and disaster recovery the same thing? No. Failover covers the live primary/standby path; disaster recovery also needs evidence that backups, recovery data, and business dependencies can restore the service.
Source
Read the Failover Testing prompt: https://github.com/naodeng/awesome-qa-skills/tree/main/skills/en/testing-types/failover-testing/prompts/failover-testing.md. It defines the FOV-## contract and requires node, routing, and write-authority timelines.