OStatic

Like many people in the computer industry, I tend to operate on very little sleep. Over the years, I've gotten used to this state of affairs, although there are definitely times when I'm more tired than others. Just last night, I had an experience that showed not only that I should probably sleep a bit more, but also the importance of open standards in the software industry.

I pushed myself a bit harder than usual last night, trying to catch up on e-mail as well as empty the long list of tabs that had accumulated in my browser. I was a bit too tired, it seems, and found myself dozing off at the computer -- falling asleep, catching myself in the act, and then falling asleep again.

Unfortunately, my fingers didn't get the message until it was too late. And somehow, as I was dozing off, I managed to erase all of the mail in my inbox.

Now, I've been trying to work my way down to an empty inbox, as suggested by my college friend Mark Hurst, and also in an excellent video lecture ("Inbox Zero") given at Google. But my inbox typically contains about 20 messages at any given time. And of course, those messages are the ones to which I really need to respond, from contacts who are waiting to hear back from me.

So when I discovered that my inbox was now empty, I panicked. I thought that perhaps I could simply "undo" my most recent move or delete, but that wasn't effective. I thought that perhaps I wasn't seeing things correctly, and tried to refresh the screen. I looked through the "junk" and "trash" folders, as well as some other folders into which I had recently moved messages. Nothing seemed to help. And while I make regular backups of my home directory, reverting to the backup would undoubtedly result in the loss of some messages.

But then I remembered that I use Mozilla Thunderbird as my e-mail program, and that Thunderbird doesn't erase messages right away. Rather, it marks them as candidates for deletion, removing them only when you "compact" the folder. So long as I hadn't yet compacted my inbox, I would be able to recover the messages. A quick inspection of the Inbox on disk (as opposed to in Thunderbird) revealed that this was the case, that the messages were all still there.

A quick Google search later, and I had learned how Thunderbird identifies messages status, including the flag for deletion. I opened the inbox in Emacs, and changed teh X-Mozilla-Status header to indicate that the messages had been read, but were not deleted. I restarted Thunderbird, and -- voila! -- my inbox was back. Relieved to have saved myself from this disaster, I decided that it was time to go to sleep.

Now, beyond the fact that I need to rest more, what can we learn from this story?

Mostly, that it's sometimes crucial for people to have access to their data without going through the application that created it. This means that the format in which the data is stored should be open and available for anyone to learn, understand, and describe. Moreover, it should be possible for anyone to write programs that create and manipulate files in that format.

Without a documented, open standard, the application becomes the only way to get data into or out of a file. If the application is a closed-source, commercial program, then the user is at the software company's mercy, hoping that the program will continue to work, and that the format contains no serious bugs. The economics of proprietary software reward complicated and hard-to-understand file formats, because they ensure that users will continue to use the program.

With open-source sofware, the opposite is true: Programmers have an incentive to make the file format as open and readable as possible, and to encourage others to write programs that work with the same format. Format changes are documented and debated by a community of programmers and users, ensuring that the program strikes a good balance between backward compatibility and future features.

• thunderbird
• file formats