Stress-test a secrets rotation drill by proving that each restore window and hold rule can place key swaps inside the planned lane, survive override pressure, and avoid trapping operators inside a green secrets slide that hides long-lived exceptions.
Secrets rotation drills often list a calendar while leaving restore behavior, override authority, and abort ownership implicit. Those edges decide whether a late key ask stops inside the policy or leaves a broken credential live under production load. The exercise should follow actual vault tools, rotation exports, and on-call paths rather than a clean compliance deck.
Inventory the rotation path
List every secret class with its hold window, restore rule, override behavior, owners, and notification channels. Mark paths that cannot reverse without a manual credential edit. Attach the last three rotation incidents with raw timelines and any waivers. Include the source of truth for open secret counts during the observation window.
Define the phases for detect, hold, rotate, abort, and communicate. Each phase needs an owner and an exit condition. Write the point after which a stuck rotation hold would require a different procedure, then review whether that action is still permitted. Capture maximum acceptable service friction in measurable units, including which cohorts are excluded from the hold and why.
Include the calendar of known events for the next two quarters: release peaks, audit windows, and partner cutovers that shrink the usable change window. A rotation budget that ignores those dates will look calm until the week they land.
Failure drills
- A minority high-volume service keeps a side-channel while the aggregate rotation dashboard stays green.
- A restore has already left a partner surface without a trusted secret state.
- The primary vault dashboard lags beyond the planned observation window.
- An operator skips a hold gate because a release is close.
- Automated and human holds collide under the new restore rule.
- Abort authority is unclear at week end and the page lands on the wrong rotation.
For each drill, identify detection time, service impact, containment, and the authority to force a secret rollback. Require commands and dashboard links in the runbook. A statement that monitoring will catch it does not establish which alert fires or who receives it.
Prove rotation changes are timed and owned
Run the package in Pingpong with security, SRE, platform, and finance seats. Ask SRE which service decision becomes unsafe first if secrets still stick after the claimed restore window. Ask platform whether capacity can absorb a forced override. Ask security to show the exact secret version or vault export used as the exit condition.
Related reviews include the security ops lead seat, an access review cadence review, and a security incident response stress test. Browse the war-game decisions hub for adjacent controls.
Authorize the published rotation response only after a timed drill restores usable secret hygiene inside the documented budget without an undocumented manual step.