I’m probably saying nothing new to any seasoned Windows developer, but it was new to me: if you add a watch in VisualStudio for the value @ERR,hr (no quotes) you get the human-readable error string for the error code that would be returned by GetLastError() if you called it and then ran it through FormatMessage(). I was amazingly happy to discover this little tidbit of functionality, since previously I had thought I had to look up the value on the MSDN System Error pages.

I say nothing that should surprise diehard Mac users if I say that Mac OS X lacks decent power-user mail clients. Thunderbird suffers from a lot of the same problems as Firefox, Entourage is…well, Entourage, and Mail, pretty though it may be, is heavily underpowered in a lot of key areas. (E.g., you can’t even create nicely formatted lists, which is something I need to do quite frequently.) From this motley crew, I’ve traditionally opted to use Thunderbird. Yes, it’s ugly and doesn’t integrate with OS X, but it’s got the features I want, it runs quite fast, and I get to use the same app on both my Windows and my Mac machines.

Like a lot of people, I had been keeping a few thousand emails in my inbox to be sorted out at some point in the future (usually, “tomorrow”). Some of those were useless spam, and many were just trivial notes I should have deleted but didn’t, but a few were more important to me, like emails from my trip to Germany back in eleventh grade.

This morning, I brought my G5 out of sleep and noticed that Thunderbird had locked during the night. No biggie; force-quit and reopen. As I heated some tea and prepared my breakfast of burnt eggs cereal with spoiled milk instant oatmeal and a grocery list, I sat down to read my email, and noticed that Thunderbird was reporting that I had zero messages. It took my brain a few moments to fully process that Thunderbird was telling me that I had no messages, period. Thankfully, Thunderbird stores its emails in mbox format, and I actually do back my data up fairly regularly using rsync, so I didn’t worry too much, but I was fairly annoyed. The only other app that has ever lost my mail is Microsoft Outlook Express 4 for Mac—not good company to be in.

When I got home today and actually had a chance to look at the problem, though, things were weirder. The file for my inbox was not only fully in-tact, but Thunderbird was dutifully appending new messages to the end of the file. It simply refused to show me what it had downloaded. A few Google queries later, I had identified the problem: Thunderbird’s index file had gotten corrupted. The solution was to delete the index file.

This is dumb in so many ways I don’t even know where to begin.

1. My biggest complaint here by far is that Thunderbird has all the information it needs to fix this problem itself. It knows how many emails used to be in the account the last time the program ran. It knows how many emails the index file thinks there are now. If these numbers don’t match, rebuild the index.
2. It turns out that this is a known problem if your inbox gets big and you don’t compact it. What does compacting mean? It means actually deleting on disk what you deleted in Thunderbird. By default, Thunderbird simply marks certain messages as deleted, meaning that they won’t show in the GUI, but doesn’t actually expunge them from the file on disk, since that would require rewriting a large amount of data. Lots of programs use this shortcut for normal use, occasionally running a compacting cycle when the machine’s idle or at least not terribly busy. What’s unique to Thunderbird is that Thunderbird, by default, won’t compact, ever. That would be ludicrously stupid, but at least vaguely pardonable, if Thunderbird worked okay in this configuration, but the truth is that Thunderbird’s indices get corrupt if you don’t compact them often. To even discover this setting, let alone change it, you have to go to Tools:Options:Advanced, where it’s notably under “Disk Space” and not “Settings That Can Crash This App And Destroy Your Emails If You Don’t Change Them.”
3. Normal end-users are never, ever going to realize that they have any hope of recovering their data. To your normal end user, when the indices get corrupted because all that image-based spam they’re deleting doesn’t actually get deleted and corrupts the Thunderbird database, their email just vanished. Not what I’d expect from an application that bills itself as “a robust and easy to use client”.

At this point, I’m seriously considering reviving my earlier attempt to write a cross-platform email back-end in Squeak (or, these days, maybe Factor) with powerful threading and search, and then writing a nice GUI over it in Cocoa and .NET for normal use. In the meantime, I’m migrating back to Mail. It may be underpowered and it’s IMAP support may be an exercise in frustration, but at least its never deleted my email.

One of the things that continues to amaze me as computers become increasingly ubiquitous is how entire industries are beginning to vanish because their product has become superfluous. Nearly everyone following the tech sector knows that the decline of the media cartels has already begun: increasing numbers of bands are bypassing traditional recording labels and posting directly to iTunes and similar services; amateur directors are posting videos on sites like Google Video and YouTube rather than worrying about running in indie theaters; and would-be talk-show hosts are creating podcasts that are slowly beginning to challenge traditional radio. That whole area, though, makes me blasé; computers are by definition good at transmitting digital media, and, as all media becomes digital, it just makes sense that the studios would become yet another vestigial appendage of an older era.

What’s less intuitive to me, and what I follow with keen interest, is how computers are moving back into the traditional realms by revolutionizing physical media. Many people are familiar with CafePress, which allows nearly anyone to set up a store selling their original music or T-shirts, but more interesting to me are sites like Zazzle that let users make custom one-off designs for themselves. Of these, the one I’m excited about due to NaNoWriMo is Lulu, an extremely modernized on-demand publisher which will happily do one-off books at very reasonable prices ($9.50 and $12.50 for a 200- and 400-page paperback, respectively). The upshot is that, if I succeed at NaNoWriMo, no matter how poor the quality of what I produce, I can spend a pittance to get it in a professional form that I could sell to others.

No publisher. No editor. No agent. Just a PDF, a website, and few spare dollar bills.

Michael Gorsuch, our new sysadmin at Fog Creek, will be joining me in a month of NaNoWriMo misery beginning tomorrow. If we both win, my hope is to redact our novels just enough so that they don’t stink to high heaven, use my old newspaper background to typeset them and make them pretty, and then publish them in a joint volume through Lulu. That would’ve been impossible even two years ago, but now it’s ridiculously easy. If more people discover this technology, I fully expect that the publishing industry will end up reshaping itself more than ever as a purely marketing machine. Anyone who wants will get published; the publishers will be companies that simply trade a cut of the profits for free advertising. It’ll probably take a long time to get there, but it’ll be fun to watch the process.