Why we need to talk about contingency plans in software development

Trevor BrunerPosted by

The other day I read an article on why fighter pilots know that quick reactions are for losers. The gist of the article is that you need to respond to a situation, instead of waiting to react to it. The author explains that a pilot’s response is based on countless hours of experience, planning, and thinking through the all-important question of, “what will I do when X happens?”. The piece got me thinking about the insurance measures we take in enterprise software delivery, as well as inspiring three very different trains of thought.

Train one

It’s not that pilots simply have supernatural reflexes, it’s that they’ve trained to know what to do in most situations. As a guy who served for a few years underwater on a nuclear submarine, I can relate to that. We ran endless drills so we would know how to respond instead of react.  Although a submarine and a fighter jet are very different beasts, they’re both (relatively) small tubes operating in hostile environments where a small mistake can be fatal.

Train two

This article also got me thinking about the old idea of a court jester. A court jester is commonly thought of as an entertaining clown. Someone to make the king or queen laugh. Occasionally, they served as a counterpart to the yes-men of the royal court. A jester could deliver news no one else dared to. Imagine if we tasked someone to take on this role at a company. What would they say to the CEO? What sort of truths could they say that no one else wants to?

Train three

The same day I read the article, I received an insurance bill in the mail. Now that my car is paid for, I don’t have the bank telling me how to insure my car. However, I’m still carrying more insurance than legally required. Why? Well, the reason I have my current car is because the last one was totaled. I was driving on the highway in the exit lane. The exit was over a quarter of a mile away, but traffic had backed up quite a ways and had come to a stand-still.  I stopped in the line of cars, but the car behind me didn’t. I was sandwiched and my car was a complete loss. Luckily, I had insurance and was able to replace my car a few days later.

All three of these trains of thought are telling us about contingency and being realistic about what a business may face. That it’s vital to spend time, money, and effort into thinking and planning for things that we don’t want to think about.

You don’t want to think of an engine failure while flying; you don’t want employees to point out ugly truths that go unspoken; and you don’t want your car wrecked right after quitting work to go back to get an MBA.

I like to think of all these scenarios in terms of insurance. Insurance is an expense you are willingly to pay now in the hopes that you’ll never need it. But if you do, it will drastically reduce the negative impact of whatever happens. Often we think of insurance as a policy provided by another company. In reality, however, insurance is whatever we do to mitigate potential loss.

As an industry, we are very focused on maximizing gain. We prioritize features by their potential revenue versus effort. We spend our time on building the best features that will generate the most revenue. Very rarely do we stop and think about how much of our resources should go to minimizing losses.

We spend the majority of the time thinking and working towards making the most money, yet don’t spend enough time thinking about how to protect that money in the inevitable face of uncertainty. Insurance isn’t cheap. Paying for insurance shrinks your bottom line. The resources spent on insurance could easily be appropriated for more profits. And likely the biggest cause of avoiding the cost of insurance isn’t the actual costs, it’s the pain of thinking about bad things. Deciding to insure against a risk means evaluating unfortunate scenarios and really asking, “what should we do?”. That can be a scary thing.

It’s not a matter of if something unexpected will happen. It’s only a matter of when it will happen. And when it does, do you want to react to it? Or do you want to respond to it?

Think of these as the unknown-unknowns. Things that fall well outside of the typical feature complexity challenges we deal with.

Here are some scenarios to think through:

  • What if company machines get hacked and we lose customer data?
  • What do we do if a huge customer request comes in after we’ve committed to one course of action?
  • What do we do if we get sued?
  • What do we do if a key team member leaves?
  • What’s our response to a sexual harassment complaint?
  • What if our servers all die?
  • What if our task tracking tool dies? How do we keep working? Or do we?

I’m not in any way suggesting a majority of our time should be spent on these issues. I do suggest that some amount of time be set aside to work through these scenarios with a cross-functional team. The answers don’t have to be perfect, nor will the scenarios exactly match what happens in real life. But three things will happen:

  • Your organization may just have some answers before you need them
  • The act of practicing how to answer these hard questions will make it easier to do when you encounter something unexpected
  • Your organization will take a stand on what risks they’re willing to accept and that will need a contingency plan

The biggest drawback to this style of thinking is that it takes time away from the day-to-day push to move the company forward. Insurance is a cost, and it’s time-consuming. It’s also intellectually painful to think of what can go wrong. And the biggest drawback of all is that we don’t want all this time and effort to be wasted. We just blindly hope that everything goes according to plan.

And just like with real insurance, when the metaphorical scat hits the fan, you’ll be thanking past-you for going through the pain of creating a response instead of waiting to simply react. That way, like a fighter jet, your product delivery will continue to soar against unpredictable winds.