Reliability ops

Stress-test a pager rotation cut before soft coverage outlives the window

Stress-test a pager rotation cut by proving that each coverage window and hold rule can place on-call decisions inside the planned lane, survive override pressure, and avoid trapping operators inside a green coverage slide that hides long-lived gaps.

Pager rotation cuts often list a calendar while leaving restore behavior, override authority, and abort ownership implicit. Those edges decide whether a late coverage ask stops inside the cut packet or leaves a broken exception live under production load. The exercise should follow actual incident timelines, roster maps, and escalation paths rather than a clean staffing deck.

Map the coverage path

List every service class with its coverage window, waive rule, override behavior, owners, and notification channels. Mark paths that cannot reverse without a manual amendment. Attach the last three pager incidents with raw timelines and any waivers. Include the source of truth for open coverage counts during the observation window.

Define the phases for detect, hold, cut, abort, and communicate. Each phase needs an owner and an exit condition. Write the point after which a stuck coverage hole would require a different procedure, then review whether that action is still permitted. Capture maximum acceptable page friction in measurable units, including which cohorts are excluded from the cut and why.

Include the calendar of known events for the next two quarters: release freezes, holiday windows, and vendor cutovers that shrink the usable change window. A cut budget that ignores those dates will look calm until the week they land.

Break drills

  1. A minority high-volume service keeps a side-channel while the aggregate pager dashboard stays green.
  2. A restore has already left a customer surface without a trusted coverage state.
  3. The primary page dashboard lags beyond the planned observation window.
  4. An operator skips a hold gate because a release deadline is near.
  5. Automated and human escalations collide under the new coverage rule.
  6. Abort authority is unclear at week end and the page lands on the wrong amendment.

For each drill, identify detection time, customer impact, containment, and the authority to force a coverage rollback. Require commands and dashboard links in the runbook. A claim that monitoring will catch it does not establish which alert fires or who receives it.

Show timing and ownership

Run the package in Pingpong with reliability, on-call, support, and engineering seats. Ask reliability which coverage decision becomes unsafe first if pages still stick after the claimed window. Ask support whether ticket macros can absorb a forced override. Ask owners to show the exact roster version or export used as the exit condition.

Related reviews include an SLO breach response, a pager policy review, an on-call rotation stress test, and the reliability ops lead seat. Browse the war-game decisions hub for adjacent controls.

Authorize the published rotation cut only after a timed drill restores usable coverage hygiene inside the documented budget without an undocumented manual step.