I upgraded bitquabit to Ubuntu today. I learned a few valuable lessons:

  1. Untested backup scripts don’t count. This one I knew, but I didn’t fully process that “untested” really means “untested recently.” In particular, my backup script was backing up a database called wordpress. Unfortunately, I moved all the blogs hosted by bitquabit to a database called wp last fall. Result? The backups, though minutes old, were effectively from last October. I was lucky here: I happened to have a day-old WXR file, and a friend sent me the one missing post that was still on his screen in Google Reader. Another blog hosted on bitquabit was not nearly so lucky; it will basically have to start from scratch.
  2. Backups tested on systems substantially different from the deployed system are useless. I love Citadel; it’s an awesome groupware client. And my backups and restores worked quite well—when both ends were running Debian 4. Unfortunately, I just moved bitquabit to Ubuntu 7.10. You can guess the rest.
  3. Have multiple backup strategies. Although the Citadel “migration” didn’t exactly go as planned, I was saved there by paranoia: I sync my IMAP mailboxes to my local machine in mbox format. I was back up and running about a minute after the server came back up. No data loss there.
  4. Don’t migrate at odd hours if you don’t have to. Realistically, my last-minute checks would at least have caught the fact that the blog database was borked if my brain had been more awake. The “most recent” post was, after all, more than six months old, and had encoding errors that I’ve fixed before. Then again, I didn’t catch this in the dry run on Sunday, either, so I’m not quite sure how much I can meaningfully excuse my idiocy through fatigue.

On the bright side, everything, minus a blog, seems to be fully up and operational again. There are a couple of quirks—with the blog hell, I opted to do an impromptu migration to MovableType—but I’m confident I can get things straightened out again. Meanwhile, all I can do is live and learn and hope this is the last time I have a lossy server migration.