by Mike Shea on 8 September 2005
Here's a little bit of a scattered braindump today.
I found a neat little program on the web called Wikipad. It is a wiki-style notepad that lets you link words to child pages and spawn a great network of textually connected notes. You can search these notes, look back over previous days, view a history, export to a few varieties of HTML and XML, and all of the data are stored in text. It has a very simple interface and a lot of neat features.
I am reminded of Philip Greenspun's great description of a better operating system in his book, Philip and Alex's Guide to Web Publishing that included core database features such as rollback, autosaving, versioning, and object-based data management. The concept went something like this:
Why do we need to tell computers to put data from one type of memory to another? Why do I have to tell it "move stuff from ram onto the HD" when I don't tell it to move data from the processor cache to ram or from the HD cache to the HD itself. Why can't we, with our massive and powerful systems, finally have machines that know to save data for us automatically and offer us versions of our documents as we have progressed? I would give up all of the CPU and memory bloat of a powerpoint presentation to have auto-saved and versioned text files.
Why do I need a search engine to find crap on my computer? Why is my data stored in files that are already an arbitrary concept? Why can't I index, search, and cross-link my data regardless of its physical location, name, format, or anything else?
So what do we want in a good data operating system?
Versioning: I want to be able to restore to a previous state from a year ago. I want to see what stuff was like six weeks ago. I want to recover that phone number I deleted back in 1989.
State-less writing: I want to write when I write and know that what I write is permanent. I don't want to hit "save" and I don't want fifty files saved off that are all really the same thing.
Backup and Recovery: I want to be able to restore from catastrophic failure of either hardware, software, physical damage, or the worst possible form of data destruction: user error.
Interlinked data: Why can't I see everything I've written about D&D? Why can't I look at all of my Everquest stuff all in one place? Why can't I see what I wrote about Perl back 5 years ago? I should be able to bin and re-bin all of my information based on constantly changing topics. Also, metadata is crap so it should be all based around free-text searches.
Program, format, and OS independence: How do I know what I will use to write in 10 years? What did I used to write in ten years past? What programs will I use on Linux as opposed to Windows? All of my data should be transferable, writable, editable, and usable on cross-platform and cross-systems. The answer, my friends, is ASCII. ASCII is king, everything should be in ascii. Except binary stuff. Binaries shouldn't be ascii. Like Porn.
Archivability: If I wrote everything in grand leatherbound tomes of lore with my trusty fountain pen, archaeologists a thousand years from now should be able to decode it, read it, and understand it. This comes to a few things: physical durability, format longevity, proper organization so other people can understand your organization, data independence so it can work on its own without that thousand year old version of MS Word.
What can we do to ensure the above criteria are met? What applications work well in this regard? I don't have an answer for all of it, but I have a few things I have learned:
There are CDROMs that are durable to 100 years if well kept. The physical media isn't real strong though. I like Flash RAM a lot more. That BBC article about the durability of compact flash was very heartwarming, but I don't yet know how long data can survive in a flash drive although I hear its about 10 years - not very long. Hard drives are better than floppies but they also suffer from magnetic interference. Solid state is better. Right now the answer is to store it on a variety of media: CD-ROMs, redundant hard drives, and flash drives.
For data format, ASCII is king. Simplified HTML works nicely and works very well across platforms. Use as little markup as you can. Only mark up what you absolutely must mark up. The simpler your format the better. Ignore the UI elements, don't mark up for how it looks on your machine, mark it up to define elements of a document. Keep it Simple! Use platform independent and widely used formats to store data. ASCII and HTML for text documents, jpegs for images, mp3 for audio recordings, mpeg for video. Make sure the format is easily viewed, read, or listened to on a wide variety of machines. Don't pick obscure "standards", use what is used widely today as long as it's open and transferable.
There is a great discussion and set of articles about writing into one huge text file. One text file is very easy to move, compress, back up, restore, edit, read, and write on just about any computer system on earth. Since reading the articles, I started writing all of my notes into a huge text file which I store online. When I need it, I can just fetch it. When I want to write to it, I have a little input box I can load and type crap into. A perl script adds the current date to any post and saves the file locally. I've only used it for a short while, but its a great way to save draft documents, sets of URLs and interesting links, any interesting quotes or text that I might use later, or anything else you would normally write on a sticky note. I love it.
The end result of all of these thoughts leads to my life backup, lifebackup.tar.gz. The life backup is a full backup of everything I have written for the web including most of my websites, fictional writing, blog entries, some choice photos, my notes.txt file, and some of my dad's stuff including a full copy of his book, All Things Are Lights. The lifebackup.tar.gz file contains my entire work in static HTML files and jpeg images. It can be unpacked with winzip, 7zip, powerarc, or gzip. It can be hosted on a static webserver, indexed by a search engine, transferred from machine to machine, viewed on just about anything, and transformed into all new formats. It can be (and is) backed up to many devices including a website backup, three PCs at home and work, four USB thumb drives, a SD drive, and an external HD. I have disconnected backup copies in my car, on my person, and at my girl friend's house. It is backed up automatically from a variety machines every night. The only thing I don't have is versioning. I need better versioning. In the mean time, I am pretty sure that these data can outlive me.
Ok, I'm beginning to bore myself, not to mention you.
I ordered a new Apple 4GB iPod Nano last night. The 4gb solid state mp3 player with a screen should handle most of the music and audiobooks I could desire. I have about 13GB of music and books now, but I can easily trim it to the 4gb I would actually use on the go. The player is tiny and very cool. I can't wait to see it. I plan to use my iPod Shuffle in a contest for Caster's Realm or Mobhunter to generate money for the Red Cross Hurricane relief effort. I wasn't real happy with the iPod Shuffle. 1GB is an awful lot of space considering you don't have a screen. With 200 songs on it, you can't find that one song you wanted to hear and if you lose your place in an audiobook, you are completely screwed. The Nano solves this with a screen and the click wheel. $250 isn't cheap, though. Even better would be a 12 to 20gb solid state ipod but thats a few years off yet. I'm a sucker for solid state memory.
I received a bold-point nib for my Pilot Vanishing Point hoping that it would write better in my Renaissance Art journals. It didn't. The best pen I have used with the Renaissance Art journal is a Lamy Safari medium point. That seems to work best so that is what I will use.
I had the opportunity to prototype a new set of leather Moleskine covers from Renaissance Art over the past month. They're wonderful. I took a bunch of pictures on Flickr to show them off. They add the ancient element of a wrap-style leather journal with my favorite Moleskine pocket notebook. I love them.
Depths of Darkhollow, the tenth Everquest expansion, comes out in a week. I've been betatesting it and it has a lot of wild stuff. The best two features, by far, include spirit shrouds and monster missions. The Shrouds let you switch to a monster archetype of your level or lower so you can hunt with lower leveled friends in lower leveled areas and the monster missions let any group of six players go on a mission as monsters regardless of their character's level, class, or gear. It is the best way I have seen to get players together in a MMO and avoid the level gap problem.
I've been playing a bit of World of Warcraft as well. It is the best of the newer MMOGs, but the heavy focus on solo content really takes away from the fun of grouping with others. I argue with people all the time about their desire for solo content but my experience has been that the more solo content you have in a game, the less likely players will group together and make friends. Having fun with other players is the key purpose of a massive online game. I wrote a lot about the real strength of massive online gaming in my article Why I Play Everquest, 2005.
On the book front, I have received a lot of excellent copyediting from my friend Todd and the latest online version of Vrenna and the Red Stone and Other Tales should be a lot cleaner. Please give it a read and tell me what you think.
I have two more stories I hope to write, type, edit, and add to this book before I release it later this year. I am deep into the next one but I don't yet know where exactly it will go. The second one is more streamlined and I can't wait to get to work on that one.