Housekeeping: June 2004 Archives

Normally right after I apologize for a delay in posting, I follow that up by not posting for an even longer period. So I'll try to avoid that trend this time.

Over the weekend we had a test of our disaster recovery procedures. The SAN containing our mail data, desktop backups and file server went south and we spent a bit of time recovering from that.

Here's some thoughts (not all of it had to be learned the hard way)
1. Alerting is always good. You want to find out about these things as early as possible. I had a page from Unicenter that I interpreted as being about disk size but apparently it was trying to say "disk gone". If I hadn't wondered through work, I wonder how long that would have gone unnoticed.
2. You need a hard copy of your disaster recovery plan, or at least some people's home phone numbers. Its not good for it to be on the server that is down. (apparently there was some concern at providing us low level people with everyone's home information because we might go egg their houses. We're all in I.T.. We can probably figure out where you live already.)
3. Plan for the worst case scenario. It could happen.
4. Backup software is worthless if it cant restore the entire server.
5. In time of a disaster, everyone is needed. Sometimes even if you dont have knowledge about specific software, you can be a sounding board, or just run out for donuts.