Stress-test a feature store cutover by proving that online serving and offline training read compatible values, failures stay contained, and rollback remains available through the window.
Cutover plans often describe the new store topology while leaving backfills, point-in-time joins, cache warming, and long-running batch jobs implicit. Those edges decide whether models keep their meaning after the switch. The exercise should follow actual feature names, entity keys, and consumer jobs rather than a clean architecture slide.
Inventory the feature path
List every feature group with its entity key, freshness target, online and offline writers, authorized consumers, and owners. Mark features that require historical reconstruction. Attach parity reports, recent skew incidents, and results from the most recent shadow read. Include the source of truth for each field during the overlap period.
Define the phases for dual-write, shadow-read, cutover, observe, and decommission. Each phase needs an owner and an exit condition. Write the point after which rollback would require recovering the prior store, then review whether that action is still permitted. Capture the maximum acceptable skew for critical features in measurable units, not adjectives.
Failure drills
- Online serving reads the new store while a training job still materializes from the old offline path.
- A late event arrives after cutover and lands in only one store.
- Entity key formatting changes and silently orphans a high-traffic cohort.
- A consumer cache keeps stale feature vectors beyond the planned overlap.
- Telemetry for parity checks drops during the observation window.
- An operator discovers an undocumented batch job writing to the old store near decommission.
For each drill, identify detection time, model impact, containment, and the authority to pause. Require commands and dashboard links in the runbook. A statement that monitoring will catch it does not establish which alert fires or who receives it.
Prove the switch is reversible
Run the package in Pingpong with data engineering, ML ops, SRE, and a model owner seat. Ask the model owner which evaluation would fail first if online and offline values diverge. Ask SRE whether origin load can absorb a forced rollback. Ask data engineering to show the exact parity query used as the exit condition.
Related reviews include a schema migration, the data engineer seat, and the ML ops lead seat. Browse the war-game decisions hub for adjacent controls.
Authorize decommission only after a parity query across the full overlap window stays inside the documented threshold and a rollback drill restores the prior path without an undocumented manual step.