Have you tested your database backups lately?
No you haven’t. I know you haven’t, because I was a DBA, and I hardly ever did it either. Now that I work for the makers of LiteSpeed, I hear the horror stories from our support folks. DBAs call up screaming in pain, and we can hear the boss loading the shotgun in the background. The DBAs wrote backups to tape and the tapes got hosed, or maybe the backups never actually made it to tape because the DBAs wrote the backups to disk but the network staff never backed up that disk share.
Do yourself a favor – pick your three most critical databases, the ones that’d really give you heartburn if they disappeared, and go try to restore them onto a testbed server that has decent performance. Time how long it takes, and mention it to your boss.
Mention it to your manager in casual conversation. “Hey, Pointy-Headed Boss, I tested the restores today just like I’ve been doing every week (cough) and they worked great. I never mentioned it to you before, but in case you’re curious, it’d probably take us about X minutes in order to recover in case the production server barfed.”
Your boss will probably say, “Why are you telling me this?”
You say, “I just thought you might want to know. If that server dies, I want you to kinda have a feeling for how long it’ll take me to restore, because in the heat of the moment, you know, people kinda stand around the desk waiting for it to happen. If the box explodes, I want you to be able to answer when your manager and your manager’s manager ask us how long it’s going to take.”
At that point, the fidgeting will start, and maybe they’ll ask you to make it happen faster.
Why am I telling you this? Because budgeting time is coming up, and that’s a good way to segue into a conversation about getting more storage performance for your servers or getting a faster, dedicated backup share.
Good luck. Let me know how it goes. (And start testing your backups regularly. Please.)
Update 10/2 – Funny Ha Ha – Jason Massie aka StatisticsIO.com posted a funny comic about testing your database backups.
October 2nd, 2008 at 3:12 pm
Another + is the fact that after you test that 500GB restore and you tell your Boss is will take XX hours, he may give you time and resources to implement partial database availability and\or a warm standby.
Sometime stuff like that doesn’t become a priority unless a real disaster recovery occurs. At that point, you are probably a member of the 24 or 48 hour club.
October 3rd, 2008 at 11:46 pm
When I was a contractor, my favorite sales pitch to CIOs was “Pick your most critical database. Got it? Ok…that server just spit its guts all over the server room, nothing to recover from. How long until you are back online? Don’t know? I can start Monday…”
90% success ratio selling at least a one or two day “review” engagement.
October 5th, 2008 at 7:34 am
Kevinic3 – I love that sales pitch. The next question I’d ask would be, “Quick – grab the checklist of what needs to be done in order to bring that server back online. What? You don’t have a checklist?”
June 7th, 2009 at 2:04 pm
[...] struggling for some comic fodder today. Like for at least 7 minutes. Then I remembered a blog post that BrentO wrote yesterday and blam. Humor(or lack of) aside, test your fracking [...]