Stress-test a canary percentage by proving that each expansion step can detect harm in time, abort cleanly, and avoid hiding failures inside sticky or uneven cohorts.
Canary plans often list percentages while leaving metric windows, cohort selection, and abort authority implicit. Those edges decide whether a bad change stops at 1% or rides to full traffic. The exercise should follow actual services, dashboards, and on-call paths rather than a clean rollout slide.
Inventory the expansion path
List every step with its percentage, duration, success metrics, abort thresholds, owners, and notification channels. Mark cohorts that are sticky by region, plan tier, or feature flag. Attach the last three canaries with raw scorecards and any waivers. Include the source of truth for each metric during the observation window.
Define the phases for shadow, limited canary, expand, observe, and full promote. Each phase needs an owner and an exit condition. Write the point after which rollback would require a different procedure, then review whether that action is still permitted. Capture maximum acceptable error and latency deltas in measurable units, including which customer cohorts are excluded from the gate and why.
Failure drills
- A minority cohort fails while the aggregate score stays inside the gate.
- Sticky sessions keep the same users on the canary longer than the plan assumes.
- The primary dashboard lags beyond the planned observation window.
- An operator expands early because a partner deadline is close.
- A dependency degrades only under the canary's traffic shape.
- Abort authority is unclear at 2 a.m. and the page lands on the wrong rotation.
For each drill, identify detection time, customer impact, containment, and the authority to pause. Require commands and dashboard links in the runbook. A statement that monitoring will catch it does not establish which alert fires or who receives it.
Prove expansion is reversible
Run the package in Pingpong with DevOps, SRE, product, and support seats. Ask product which user decision becomes unsafe first if the canary is wrong. Ask SRE whether origin load can absorb a forced rollback. Ask DevOps to show the exact gate query used as the exit condition.
Related reviews include the DevOps lead seat, a deploy freeze exception, and a feature flag rollout. Browse the war-game decisions hub for adjacent controls.
Authorize full promote only after a parity query across the planned window stays inside the documented threshold and a rollback drill restores the prior path without an undocumented manual step.