There’s nothing quite like hearing disaster recovery stories from someone who’s been in the trenches. A few years ago I spoke with a backup administrator who faced every IT pro’s nightmare – a hurricane that flooded their data center and their Iron Mountain facility simultaneously.
This blog post summarizes the main points of my latest podcast episode. If you’d like, you can listen to it or watch it at https://www.backupwrapup.com/disaster-recovery-stories-floodup.com/
When Disaster Recovery Gets Real
Our guest has been managing backups at the same company for almost 20 years, overseeing a petabyte-scale environment using NetBackup. They had two data centers on an island, with backups replicating between them and tape copies going to Iron Mountain. Sounds solid, right? Then the hurricane hit.
The Flood That Changed Everything
Water poured into one of their oldest data centers, forcing them to abandon the building entirely. But here’s the kicker – the Iron Mountain facility got hit too, making their offsite tape copies temporarily inaccessible. Talk about a worst-case scenario.
Disaster Recovery Stories: The Three-Week Marathon
Recovery took three weeks of intense work. While their replicated backups worked well, they discovered several challenges nobody had anticipated. Some systems weren’t backed up the way business owners thought they were. DNS, networking, and other infrastructure dependencies created unexpected hurdles.
Lessons From The Field
The experience highlighted several crucial points about disaster recovery:
- Testing is essential, but full data center recovery tests are rarely practical
- Infrastructure dependencies often cause more problems than the actual data recovery
- Replication between sites works well, but having copies off-island would have been better
- Business owners’ assumptions about backup coverage need verification
- Physical access and infrastructure issues can complicate recovery significantly
Moving Forward
Post-recovery, they consolidated operations into their newer data center and abandoned the damaged building. While they still use tape in some locations due to regulations, they’re gradually moving toward tapeless operations where possible, including cloud storage in regions where regulations permit.
Written by W. Curtis Preston (@wcpreston), four-time O'Reilly author, and host of The Backup Wrap-up podcast. I am now the Technology Evangelist at S2|DATA, which helps companies manage their legacy data
