Reliability2026-02-10
Cutover rehearsals, and why we do six
A 412-step runbook took 19 hours on its first rehearsal against a 9-hour window. By the sixth it took 6 hours 40. The rehearsals were not practice — each one was designed to fail differently.
- Author
- Priya Raghunathan
- Published
- 10 FEB 2026
- Read
- 5 MIN
- Ref
- D610F9
The first time we ran Nordvik Bank's cutover runbook end to end, it took 19 hours. The agreed window was nine, of which two were reserved for the rollback decision. The runbook had 412 steps, had been reviewed by four teams, and was wrong in about thirty places.
That rehearsal was a success. Finding out in March that a plan does not fit its window costs a Saturday. Finding out on cutover night costs a regulatory incident, and quite possibly the migration itself.
Six rehearsals, six different failures
Rehearsals are not repetition. Each one is scoped to expose a class of problem the previous one could not, and each has an exit criterion written before it starts.
| # | Scope | Elapsed | What it actually found |
|---|---|---|---|
| R1 | Data load only, non-production hardware | 19h 10m | Index rebuild ran single-threaded; 6h recovered |
| R2 | Load plus full reconciliation | 12h 30m | Reconciliation had no defined pass threshold |
| R3 | Full technical cutover, no business validation | 9h 45m | Two steps had the same owner at the same clock time |
| R4 | Full cutover with business validation | 8h 05m | Branch staff had no route to raise a blocking defect |
| R5 | Rollback from the T+5h abort gate | 4h 20m | Rollback left CDC replication pointing the wrong way |
| R6 | Dress rehearsal: production hardware, real staff | 6h 40m | Two steps unowned because of an unlogged leave day |
R5 is the one organisations skip, and it is the one we refuse to drop. A rollback plan that has never been executed is a hypothesis. Ours was wrong in a way that would have left the legacy system live but no longer receiving changes — the worst available outcome, and entirely invisible on paper.
Write steps so a tired stranger can execute them
The runbook is read at 04:00 by someone who has been awake for twenty hours and is not the person who wrote it. Prose does not survive that. Every step carries an owner, a duration, a verification, and an abort action.
STEP 214 T+04:18 OWNER dba-oncall DUR 00:12 ROLLBACK 214R
ACTION Stop CDC replication LEGACY.CUST -> TARGET.CUSTOMER.
VERIFY select max(commit_ts) from target.cust_cdc_watermark;
must equal source high-water mark recorded at STEP 211.
ABORT IF lag is not zero after 15 minutes.
-> execute 214R, hold at GATE G3, page migration lead.
NOTE Do not proceed to 215 until GATE G3 is cleared verbally.
Three properties matter. Timings are relative (T+04:18), so a 40-minute late start does not require anyone to do arithmetic at 4am. Every step has exactly one owner — "the platform team" is not an owner. And the verification is a command with a stated expected result, not an instruction to check that things look fine.
Gates are where the decisions live
Between phases we place abort gates: named points where a single accountable person says continue or roll back, against criteria fixed in advance. Nordvik had four. G3, at T+5h, was the point of no return, because past it the legacy ledger stopped accepting writes.
Fixing the criteria in advance is the entire point. At 05:00, with a programme's reputation in the room, the pressure to press on with two failing reconciliations is enormous. A gate that says G3 requires zero unreconciled accounts and both payment rails confirmed by their named owner removes that conversation, which is a kindness to whoever is holding the pager.
What the rehearsals bought
Elapsed time fell from 19h 10m to 6h 40m, but the schedule was never the real product. The rehearsals produced 63 runbook corrections, a rollback procedure that had genuinely been executed, and eleven people who had each done their part more than once and did not need to read the surrounding pages to know what came next.
Cutover night ran 7h 15m against a nine-hour window. Two steps went long, both were absorbed by float, and the abort gates were never invoked. The recorded incident count was zero.
That is what six rehearsals buy: not a faster cutover, but an uneventful one. On the sixth rehearsal we found two steps with no owner because of a leave day nobody had logged. Sixth. If we had stopped at four — which is where the schedule pressure always argues for stopping — those steps would have been unowned at 03:00 on the night, and someone would have improvised. Improvisation is how migrations get rolled back.
