by Mike Shea on 20 July 2006
Note: My backup plans have changed since I wrote this. You can read about my new procedures in the article Your Personal Digital Archive.
After a hard drive crash on Sunday due to a power failure, I learned the hard way that incremental xcopies are not the best way to back up files. Of six online copies of World of Warcraft that I had, three of them failed. One failed when the primary drive went down - two others failed due to file corruption issues on the backup drives. Luckily, the copy on my Western Digital My Book survived.
So I did more research, wasted a whole bunch of hours, and came out the other side with a better understanding of incremental copies, mirrors, and commercial-grade backup software. Of the commercial stuff, I liked Acronis True Image 9.1 the best but hearing horror stories about failed attempts to recover 20 gb backup files, I began to see the problem with a single file containing all of ones data.
I also did more work with Syncback but it took about 12 profiles to do what I was able to do with three batch files and it wouldn't let me do full copies, only incrementals. I decided that, due to file failures on drives, its best to do a full copy of every file about once a month. Incremental is fine, but a full copy will ensure that every file down to the byte is still good.
Xcopy has problems with filenames apparently, something to do with 8.3 vs full-character file names. I don't know the details but it makes me wonder if that was my problem with Warcraft. Either way, this steered me to Xxcopy, the program I'm using now.
Xxcopy is a bigger version of xcopy, a freeware with limits command-line app. it does backups, mirrors, recrusive, incremental, and full copies with validation.
So now, after all of that, I am down to the following backup routine:
Every day I run the following batch script:
del lifebackup.tar.gz wget http://mikeshea.net/lifebackup.tar.gz 7za.exe x lifebackup.tar.gz -y 7za.exe x lifebackup.tar -olifebackup -y copy lifebackup.tar.gz lifebackup del lifebackup.tar
xxcopy /E /YY /Z /V2 /BI "c:\My Documents\lifebackup" "G:\lifebackup" xxcopy /E /YY /Z /V2 /BI "c:\My Documents\lifebackup" "M:\lifebackup"
xxcopy /E /YY /Z /V2 /BI "c:\My Documents" "N:\My Documents" xxcopy /E /YY /Z /V2 /BI "c:\My Documents" "F:\My Documents"
dir N: /S > N:\file_manifest.txt chkdsk N: > N:\verify.txt
dir F: /S > F:\file_manifest.txt chkdsk F: > F:\verify.txt
This will pull down my lifebackup and unzip it to "My Documents".
The first two xxcopies will do incremental copying of the lifebackup to two USB thumb drives.
The second two xxcopies will do incremental copying of My Documents to the Western Digital My Book and the Lacie 80gb portable drive.
The last commands will build file manifests and do a chkdsk to verify that the disc is still ok. This is buggy at the moment since CHKDSK really wants a /F and you can't do that without dismounting the drive.
Every week I run the following script:
xxcopy /E /YY /Z /V2 /BI "c:\My Documents" "N:\My Documents Weekly Backup"
dir N: /S > N:\file_manifest.txt chkdsk N: > N:\verify.txt
This will copy "My Documents" to a "weekly backup" folder on the WD My Book drive. This way I have a weekly backup to help protect me from accidental deletes. Again, we build a file manifest and verify the disc. This also helps me get a timestamp of the last backup. I can just check the date on those files to see the last time the backup ran.
Here's a quick tutorial on the xxcopy commands I use:
/E - copy subdirectories even if they're empty. /YY - ignore any prompting with an automatic "yes" /V2 - Verifies after copy byte-by-byte check (DATMAN 2-pass). /BI - Incremental backup based on time and size. /Z - Deletes extra files or subdirectories in destination.
Xxcopy also has easier commands: /CLONE and /BACKUP for incremental backups or clones, but I want a bit more control so I use the direct commands.
Every month I run the following script:
xxcopy /E /YY /Z /V2 "c:\My Documents" "N:\My Documents" xxcopy /E /YY /V2 /Z "D:\Program Files\Everquest" "N:\Everquest" xxcopy /E /YY /V2 /Z "c:\World of Warcraft" "N:\World of Warcraft"
xxcopy /E /YY /Z /V2 /BI "c:\My Documents" "N:\My Documents Monthly Backup"
dir N: /S > N:\file_manifest.txt chkdsk N: > N:\verify.txt
xxcopy /E /YY /Z /V2 "c:\My Documents" "F:\My Documents" xxcopy /E /YY /V2 /Z "D:\Program Files\Everquest" "F:\Everquest" xxcopy /E /YY /V2 /Z "c:\World of Warcraft" "F:\World of Warcraft"
dir F: /S > F:\file_manifest.txt chkdsk F: > F:\verify.txt
This script will do a full copy of "My Documents", "World of Warcraft", and "Everquest" to both of my external discs. It does this non-incrementally, its a full on backup that will take the better part of two hours to run. The fourth xxcopy is a monthly copy of "My Documents" to once again protect from accidentally deleted files. This way, at any given time, I have four copies of "My Documents" at different time periods - right now, yesterday, last week, and last month.
Lastly, I have a script that runs the following command:
xxcopy /E /YY /Z /V2 "c:\My Documents" "N:\My Documents" xxcopy /E /YY /V2 /Z "D:\Program Files\Everquest" "N:\Everquest" xxcopy /E /YY /V2 /Z "c:\World of Warcraft" "N:\World of Warcraft"
This would copy all three of my important directories to another external drive. This is a full copy, not incremental. This drive would be pulled after the copy and stored off-site. Better yet would be another identical portable drive that I could rotate with the one I have now so I just yank the drive every month or so and store it off-site. Even better would be an automated push of my data to an off-site location. 20GB is a lot to move off-site, though.
For the time being, this looks to be a good backup solution. Only a true disaster will say for sure.