Disaster Recovery: Lessons Learned from the Colorado Floods

Scott Calonico

flooded street signEarly September 2013 saw the US state of Colorado hit by enormous floods that resulted in tragic deaths, towns cut off, and thousands of people displaced from their homes.

Meanwhile, the rain and the subsequent breaching of riverbanks caused serious outages in key infrastructure, including electricity supplies and telecoms infrastructures.

The Colorado flood occurred less than a year after “Superstorm” Sandy caused similar destruction in New York State.

Natural disasters such as these cause justifiable concern amongst technical professionals, making them wonder if the disaster recovery provisions they have in place would work adequately if they were caught up in a similar incident.

One key detail that has emerged in the aftermath of the Colorado disaster is the fact that not that many businesses suffered physical damage to their internal infrastructures. The main impact was on core public infrastructure. Internet connections died, power went off, and even the cellphone network failed to cope.

This all goes to show that the procedural planning for disaster recovery is just as important as the technical provisioning, if not more important. Different disasters have completely different profiles, and a localized flood in just one building would present a completely different set of challenges to a natural disaster.

Disaster Recovery Suggestions

So, if you are an MSP involved in disaster recovery planning for your customers, what lessons should you learn from the Colorado floods? We would suggest the following:

  • Always consider the fact that losing communications links might be the most significant issue following on from a disaster. Thinking about redundant connectivity is therefore worthwhile. Nothing is foolproof, however. In the aftermath of the flood, an “emergency” 3G access point would not have helped while the park bench in watercellular network was down.
  • Realise that planning is key to effective disaster recovery. A good DR plan should contain an action plan for a selection of different disasters: loss of access to a key building, flood, fire, virus attack….the list goes on.
  • Think carefully about what would happen during an extended power outage. Companies with on-premise infrastructures will quickly wish they had moved “to the cloud” or utilized virtualization if they are unable to relocate their center of operations.
  • Accept that sometimes disasters will be significant enough to impact the business for several days or even weeks. Identify which elements of the business are most crucial and focus on contingency plans for them. The most important thing is (usually) keeping customers happy.
  • Always test your plan! A good DR test will always identify issues you didn’t predict. Something as simple as an extended delay in repointing email can render a DR plan practically useless. You will never find out these shortcomings unless you perform a simulation.

Disaster recovery often falls too far down a client’s list of priorities. If a client fails to plan and ends up in a disaster situation, it’s sure to prove equally stressful for you as the MSP. Don’t let this happen!