What happens when a hurricane completely wipes out your data center? Not just any data center – one on an island, cut off from the mainland, with no power, limited communications, and even stolen generator fuel. This disaster recovery case study reveals what DR really looks like when everything goes wrong.
This blog post summarizes the main points of my latest podcast episode. If you’d like, you can listen to it or watch it at https://www.backupwrapup.com/. It’s also the continuation of last week’s blog about disaster recovery.
Basic Assumptions vs Reality in Disaster Recovery
The first lesson from this disaster recovery case study hits hard: your basic assumptions probably won’t survive contact with reality. The DR team discovered their Active Directory was tied to the mainland – making authentication impossible once they lost connection. Their satellite backup communications had a daily bandwidth cap that left them without service after noon. Even their physical infrastructure faced unexpected threats, with thieves stealing generator fuel and copper wire.
The Human Side of Disaster Recovery
Recovery teams don’t usually plan for living conditions during DR. In this case study, the team slept on air mattresses in conference rooms, worked 12-16 hour days, and ate chicken and rice for two weeks straight. The hurricane had stripped the island bare – even longtime employees didn’t recognize familiar routes without vegetation.
Documentation and Testing Failures
This disaster recovery case study exposed common DR planning weaknesses. Some servers weren’t being backed up because they weren’t properly documented. Their backup strategy relied on data owners identifying critical data rather than backing up everything – a choice that proved problematic when those data owners changed jobs over time.
Learning from Real DR Experiences
Key takeaways from this disaster recovery case study include:
- Test your assumptions – especially about infrastructure dependencies
- Document everything and keep that documentation current
- Back up more rather than less – storage is cheaper than data loss
- Plan for the human elements of DR
- Test your DR plan regularly and thoroughly
Breaking Down Communication Silos
The recovery effort revealed how communication barriers between IT teams can hamper DR. Different groups had different assumptions and dependencies that weren’t shared. These silos became major obstacles during recovery.
Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at S2|DATA, which helps companies manage their legacy data
