The CTO Project

Today a car ran into something or other and took out the network for a large chunk of the state. I’m pleased to say that we handled this outage better than our last one: the emergency website went up, though not as quickly as I would have liked, and then we took it down as soon as we could. Campus communications went pretty well too. But there are always lessons learned. Here are ours:

• remember time zones! what may seem like an early morning outage to you was a peak day outage to others. You need to get your … Continue Reading
  

After last week’s outage, we’re revisiting our DR plan. It’s actually pretty good, and the scheme to activate our externally hosted alternate website for the school worked. However, there are two things to change. I noted them in my comments to Friday’s post, but I’ll make them a little more public here:

1. DNS TTL: our emergency site had a value of 30 minutes. This is too long. I think 5 minutes for the initial value, assuming we could change this on the fly if we felt the emergency site would be up for longer. We would like the real site … Continue Reading

Last night the school’s network died. Just enough to be ugly.