IT disaster recovery / IT contingency planning
Tags: contingency planning, disaster recovery, network outage
After last week’s outage, we’re revisiting our DR plan. It’s actually pretty good, and the scheme to activate our externally hosted alternate website for the school worked. However, there are two things to change. I noted them in my comments to Friday’s post, but I’ll make them a little more public here:
1. DNS TTL: our emergency site had a value of 30 minutes. This is too long. I think 5 minutes for the initial value, assuming we could change this on the fly if we felt the emergency site would be up for longer. We would like the real site to come up as quickly as possible once the emergency site is deactivated. As it was, some folks thought our network was down longer than it actually was because of this value.
2. Adding a blog to our emergency site. WordPress or another blog would be a nice feature to add to our site so we can provide frequent updates to the community without having to deal with constant interruptions.
3. Communication difficulties. Campus network is down. How do we communicate that to the campus? It’s not like they can access the emergency site from within the network, nor can they access their email. We could initiate a calling tree, send a mass text message to those with cell phones, run around screaming, activate a loudspeaker system, or your option here. Those with smart phones can see the emergency website from the cellular network. Whatever option we choose won’t reach everyone, so the right answer is some combination of the above, recognizing that walking around and talking to people has to be involved in some way.
Your idea here…
Please note that I’m talking about IT DR, not general DR or general emergencies. The campus has a campus-wide emergency plan which pertains to true emergencies. Obviously, though, there’s overlap between the two areas.
Sukey said:
Mar 17, 08 at 5:40 pmI’m all for activating loudspeakers – especially if it involves a series of high pitched and/or wailing sirens indicating an announcement to follow.
kdghty said:
Mar 17, 08 at 6:34 pmwhat i finally thought of to do here was make a voice mail mailing list. sent out an email – a few times – asking anyone who wanted to be included to send me their voice mailbox number. all i have to do is log into voice mail, press 01, propagandize and send. downside – doesn’t reach students without office phones – but they’re usually in the labs, anyway, and we can spread the word from there. not quite as wonderfully annoying as a tannoy or the civil defense system preferred by sukey, but reasonably effective at persuading our users to leave us be so we can sort the problems out.
seth vidal said:
Mar 17, 08 at 9:29 pmI think your emergency message should say one thing:
“Communication disruption can only mean one thing: Invasion”….
http://www.youtube.com/watch?v=JfBnICcsMfY
Just to really bring the point home. 🙂
-sv
Gahlord said:
Mar 18, 08 at 5:47 amI think a dirigible connected to an emergency Twitter feed would be the sweet.
How about a turnaround on the voicemail idea: set up a voice mail box that always has an updated networks status message. That way whenever anyone anytime wanted to know the status of the network they could call the message line. This might also help lighten the load of people calling about the network being down when they are experiencing PEBKAC.
Then just publish the network status mail in all of the campus directories, newspaper, blimps and emergency loudspeaker systems. Maybe have people walking around with sandwich boards as well.
admin said:
Mar 18, 08 at 8:42 pmAgreed, Seth. That’s the perfect greeting to our voice mail box, which, it turns out we have–a dedicated network status number (that apparently no one ever uses.) So, while Gahlord’s idea is a good one, as well as the spiffy dirigible concept, no idea is good if we forget we’ve implemented it…. Hence, the birth of the Documentation of Emergency Action Duties project, or DEAD.