Lesson 21: The 45-Minute Drill That Saves You from a World of Trouble
Most operations teams think they’ve got incident response nailed-until a real mess blows up and suddenly, all the careful planning in the world isn’t enough. That’s where a simple drill comes in. It finds the blind spots, cements your team’s instincts, and makes the next emergency a lot less painful and way faster to handle.
Look, it’s easy to feel ready. You’ve mapped out every process, written detailed playbooks, pinned up checklists, and named your communications lead. On paper, it all looks solid. Everything makes sense during a quiet week, when nothing’s actually going wrong. But when things really hit the fan, surprises start flying in fast.
Maybe there’s a critical system nobody wrote down. The person supposed to answer calls in a crisis? Turns out, they changed teams ages ago and no one updated the list. Or maybe a recovery step depends on a tool that’s also out for the count. You grab a “ready” message template-only to discover it references a channel no one checks on Saturdays. Every gap feels small on its own, but together? They show up at once, while customers are waiting, and now you’re sinking time just untangling what’s broken.
The thing is, planning helps but can’t predict everything. Plans tell you how things should work. Drills show you how things actually work and where the fantasy parts are hiding.
So don’t sweat piling up even more paperwork. All you really need is a half-hour or maybe forty-five minutes, to actually run through what you would do in a real incident. That is where you uncover the stuff that textbooks never mention.
What Practice Actually Does
By now, you’ve probably built a pretty bulletproof system – documented, mapped out, full of backups and standard processes. You’ve thought through reversals, added redundancies, and covered routine procedures. But have you ever tested whether it all fits together smoothly, especially under a little time pressure or chaos?
Enter the tabletop incident drill. Not just reading scripts aloud. This is about putting people in a realistic scenario, making them actually follow the plays, and seeing what falls through the cracks before something goes truly sideways.
And honestly, the stuff you find is rarely what you expect. Suddenly, there’s an undocumented dependency nobody thought to mention. Or someone’s contact info is out of date, again. Maybe a rollback works fine solo, but if you do it during another recovery step, it breaks everything. “Perfect” message templates? Great-except they’re impossible to tweak under real pressure, and that eats up precious minutes.
All these fixes take maybe five minutes to address after a drill. If you only spot them mid-incident, you’ve already lost hours-and burned a ton of trust.
A Real-World Story: The Drill That Caught What the Real Crisis Missed
Here’s how this plays out if you skip the drill. There was a data sync issue-customers started seeing errors. The team hustled, got things patched up, but it took way too long. Only later did they realize what had happened:
Five systems relied on that syncing, but the team only thought of three at first. Some background job kept chewing through broken data, but nobody even remembered it was running. One “urgent” contact person-gone, replaced months before, but no one fixed the escalation list.
Not a single one of these failures needed to happen. They all would’ve shown up in a single structured run-through.
So, we ran a drill: The same failure, the same team, just as a tabletop exercise.
We got the owner, their backup, the on-call lead, communications owner-everyone you’d expect. No heads-up about the scenario. Just, “Hey, something’s broken-let’s go.”
Forty-five minutes later, we’d uncovered:
- The forgotten background job-a ghost process missing from every document.
- The outdated escalation list-the same hole that wasted hours before, fixed in five minutes now.
- A vague reversal step-the owner and backup didn’t even read it the same way. Would’ve led to clashing fixes if it were real.
After the drill? Easy fixes:
- A one-liner registry for background jobs-now attached to the main docs.
- Escalation list updated for real, with current names and contacts.
- The reversal procedure rewritten to make things unmistakably clear.
Next time the problem cropped up, resolution time dropped to a third of the original. The squad wasn’t magically smarter. They just didn’t stumble over the same old gaps.
How to Actually Run a Tabletop Drill
- Step one: Pick something real. Choose a failure mode that’s either actually happened or could easily happen-data gets messy, a delivery flakes out, a communication chain fails, you lose access to a key system. Forget theoretical disasters; go with something personal to your situation.
- Step two: Only tell the drill roster. Basically, the core team: main owner, their backup, the on-call/escalation point, and whoever handles external comms. Let them know a drill’s coming. But don’t share what the problem is ahead of time. Real incidents happen while you’re busy doing something else.
- Step three: Run the drill-30 to 45 minutes. The owner follows the documented process, live. Backup shadows, jotting down any confusion. Comms drafts messages as if customers were already asking questions. Track timestamps.
Whoever’s leading the drill sets up the scenario, watches the clock, pays attention to where people hesitate, argue, or hit question marks. Those are your findings.
- Step four: Pause and record.
– Every action people took, in order.
– Every confusion, missing piece, or unclear instruction you found.
– The single biggest improvement-just the most important fix. Let the rest wait.
- Step five: Share the lessons and make the main fix.
Draft a one-page summary: what broke, what you learned, what you’ll fix, and who’s doing it, by when. Don’t turn it into a novel. Get it to the team and relevant leads by week’s end.
Assign the top fix-the one that matters most-to someone specific, with a deadline. No catalogue of minor gotchas. Just the thing that will make life better for the next incident. Further drills can chew on the leftovers.
Why Teams Resist These Drills-and Why That’s a Mistake
People push back on dry runs all the time. They feel fake: “We all know this isn’t real, so why bother?” The energy isn’t there, nothing’s really at stake, so it’s easy to dismiss.
Makes sense, but it’s the wrong way to see it.
Yes, drills lack real pressure. That’s a feature, not a bug. When you’re genuinely panicking, you miss things. You make snap decisions, patch over cracks, and what you could’ve learned gets buried in the scramble. With a drill, you get the lesson without the pain. The team can stop, debate, ask why, and find missing steps-without someone yelling, or a customer waiting.
Those 45 minutes of somewhat artificial practice actually build real confidence. People know they’ve done the thing, not just read about it. When it’s for real, nobody freezes from uncertainty. Everything just moves faster.
So yeah, you “lose” an hour this week. But you get it back-many times over-the next time hell breaks loose.
The Real Lesson
Teams skip drills for the same reason they punt on any investment with a delayed payoff: the benefit is hidden; the price is right now. That’s time, energy, and focus you could spend somewhere else-especially during a busy week.
But if you want to reframe it, just look at your last firefight. The hours wasted, the headache, the customer pain-most of it, a quick drill would’ve cut down, or dodged entirely.
Any team that has ever suffered through a real incident and later ran a postmortem drill says the same thing: the drill would have saved them. They just did it too late.
So, get ahead. Run the drill before you have to. Root out what’s broken while it’s painless to fix. Give your people the experience while the stakes are low. That’s how you make the next crisis cheaper, faster, and a lot less terrifying. And the one after that-and the one after that, too.