[helicopter34234]

I recently realized that optical disks like burned CD's and DVD's have a shelf life. Sometimes they can last for 30 years (depending on brand/quality, supposedly gold CD's are best) and sometimes they start losing bits after a couple of years. Since DVD backups was my primary form of backup, I became very worried. I didn't want to lose all my music or pictures which are invaluable to me. Optical disks are not the way to go. Also when your backup is physically next to your main HD, you could lose everything in a theft, fire, or flood.

In my opinion there are two options for permanent storage. First option would be an online backup servies. In terms of online backup, I heard that Online Backup, Data Backup & Remote Backup Solutions from Mozy.com – Welcome was the best option (unlimited storage for one registered computer) which would cost $100 for two years of backup. This isn't as expensive as it seems , it has automatic software to backup and might actually be cheaper than buying a new HD. The second option is keeping a redundant hardrive (one harddrive is just to mirror the main HD) with backup software constantly mirroring one drive to the other. Ideally this second drive would be a network drive that is physically in a seperate place than your main HD in case of fire/flood/theft (mine is a network drive I have set up at my work). Network drives tend to be expensive however. Also you can get a RAID configuration drive setup where you can have several drives hooked up which automatically have redundancies (depending on the setup). You can read more about RAID but I found it too expensive and difficult to setup for the average person who doesn't need really fast data access rates. If you aren't afraind of fire/flood/theft you can put the second drive in the same computer.

What if I choose to use a redundant hardrive?
It is definitely possible for old hardrives to start loosing bits or whole sectors. Also when copying files from one source to your hardrive you could drop bits in the process and be left with a corrupt file on your destination (apparently this is not true, data will not be lost during transfer). So lets say we go with the redundant hardrive backup option. How can we be sure that one hardrive (main or mirror) hasn't dropped data over time and now has files which would be unreadable. Well you can do a bit-by-bit comparison of all files on both drives (it should be a mirror so all files should be on both drives and both be identical). The best programs I found for doing this are TestPath 1.3 and Windiff (both free, both good). You can read about these elsewhere but basically they read each file on each drive and compare them bit by bit. If one drive has lost information then you will see a flag that that file is different. The other option if you have pretty static file libraries is to build an SFV file (Simple File Verification). You can highlight as many files or directories to add to this SFV file. The best programs I found for this are FlashSFV and QuickSFV (both free, both good). The SFV file that you create contains checksums and crc's (you can read about what these are elsewhere) for all the files you added to the SFV and when you click on the SFV file and say "verify" it checks to see if the files have changed at all since you created the SFV file. Therefore the files you find errors for are files that have lost bits. Once you identify these "dead" files you can delete them and copy a working version from the other hardrive.

Also programs like Ariolic Disk Scanner and Emsa Disk Check (both free, both good) will scan your harddrive and look for files that are unreadable and list them.

Hardrives aren't really expected to live any longer than 5 years so you will be swapping hardrives in and out every couple of years.

Also what I have found to work is to have one drive for all system files (which windows would be installed on and where all programs are installed to) and then another drive just for files (files, music, movies, other personal files). Its easier because you can reinstall windows without have to copy all of these file to something else in between. I have a relatively small high speed disk for the OS (I have a 10K rpm 36 gig) HD for my OS and then a 250 Gig extra drive for personal files. This 250 Gig drive is mirrored to an off site network drive every night.

 

[darkswordsman17]

You'll have to forgive me, I didn't read your whole post, but my recommendation for computer backup is redundancy and variety. Back it up in multiple places. I'd recommend burning some to discs (DVDs as data), maybe an external HDD (for more frequent/up to date backup), and in other ways as well. Online is a good way, but when it comes to backing up content such as music or movies you have to be concerned about the RIAA/MPAA.

I tend to have my main HDD partitioned, one relatively small for the OS, and then at least one for storing current stuff. Then I have an external drive. I also need to get in the habit of burning some things to discs as data.

With the low price of HDDs, I think it wouldn't be too bad to have it on 3 different ones. One on the current system where you actually use them, then on an external drive that you routinely update to stay current, and then a third one that you only update at extended intervals to keep modification to a minimum.

Even partial backups goes a long way to preserving sanity when you experience a crash.

 

[helicopter34234]

"Online is a good way, but when it comes to backing up content such as music or movies you have to be concerned about the RIAA/MPAA."

Online backups are encrypted.

"I tend to have my main HDD partitioned"

If you have two different drives then you can increase your read/write speeds.

I also forgot to mention that my c: drive where windows is installed is backup up and an image. After a fresh install of XP I made the disk image, therefore any time I want a clean slate I just reload the disk image.

 

[ericj]

Quote:

Originally Posted by helicopter34234 /img/forum/go_quote.gif Also when copying files from one source to your hardrive you could drop bits in the process and be left with a corrupt file on your destination.

Not really.

Verifying the integrity of written data is handled both at the OS level and at the hardware controller level. It's also very easy to manually verify after copying.

Over PATA, SATA, SCSI, and systems that emulate scsi over a different physical layer (like fiberchannel storage, or SBP-2 on firewire), data integrity between the device driver and the hardware controller on the drive is in fact guaranteed.

USB mass storage, not so much. Which is why usb2 mass storage is a touch faster than firewire. But like i said, you can manually verify the data very easily.

I prefer to back up with a harddrive that is offline.

A harddrive lasts a lot longer on the shelf than it does spinning in your computer.

I'm using a usb enclosure, but I'm also using rsync. All i have to do to verify the integrity of the transfer is flush the buffers and run the same rsync command again. It just takes a few minutes to compare the local md5sums to the remote md5sums.

I'm not exactly practicing law or anything, so what i need from a backup is a single snapshot of what i currently believe to be worthwhile data.

I'm protecting against disaster, not degradation.

My current data set is what's important. if i lose it, I'll be Very Annoyed. But I don't need any revision control.

Occasionally re-upping particular directories from particular machines should work very well.

There is the odd chance that a cosmic ray or something will damage a single file on a harddrive on the shelf, but a slightly degraded backup of all the important data is still almost all of the important data.

Certainly seems safer to me than entrusting my data to a third party that could run out of VC funding at the end of the quarter and dissolve their corporation to get out of their contractual obligations.

 

[helicopter34234]

"Not really."

I was just warning that if you were to say use a network hard drive (like I do)or usb hard drive it is possible have files corrupted on transfer. Off course you could create a checksum before copying and then verify it after the copy (or use a bit-by-bit comparison checker).

"A harddrive lasts a lot longer on the shelf than it does spinning in your computer."

I agree with this and I also had used an mostly offline usb drive for backup until switching to the my remote location network drive. I am not too worried about the life of the drives however because in 4 or 5 years when the drive dies im sure one with equivalent specs could be had for pennies. However, I have heard a few different times that drives can die faster when not used that often. It isn't initially clear why they would but I'm sure there are certain strange failure mechanisms that are more likely when you don't use the drive that often.

I also have checksums for files or directories that don't get updated often.

Your current setup is not protected from say flood or fire.

 

[ericj]

data integrity over network filesystems is also guaranteed. Why would every business with more than 5 employees in the developed world use network file servers if this were not true?

With proper tools - like rsync - you don't have to keep a static checksum somewhere if you still have the original data around. These are created on the fly by the software.

I still think flood and fire are less likely than some website just pulling the plug sending the employees home with the servers as severance pay.

Honestly how old are you? Were you not 'into' computers during the dotcom bust? We're going through a big economic crisis right now, and a lot of small companies that didn't exist 5 years ago won't exist in 2 years.

 

[ericj]

Alright, I'll grant you, EMC is a company I'd almost entrust with my off-site data.

But for $100 i can buy a 500gb drive and ask my buddy to keep it in his gun safe.

 

[helicopter34234]

Quote:

I still think flood and fire are less likely than some website just pulling the plug sending the employees home with the servers as severance pay.

Fair enough, I did not fully consider that aspect of it.

Quote:

... data integrity over network filesystems is also guaranteed.

Nothing is gaurenteed to work in a computer. All I know is that several files on the less than 6 month old network drive I am using to mirror back up my main storage HD have become corrupt. Whether they were corrupted during transmission or after I don't know.

 

[ericj]

Quote:

Originally Posted by helicopter34234 /img/forum/go_quote.gif Nothing is gaurenteed to work in a computer. All I know is that several files on the less than 6 month old network drive I am using to mirror back up my main storage HD have become corrupt. Whether they were corrupted during transmission or after I don't know.

Because on most all network technologies the delivery of any given packet is not assured, many layers of protection assure that the data eventually all gets from one end to the other.

This isn't like a pair of CB radios. It's like a team of 10,000 bicycle couriers running in circles between two companies.

Some of them will be delayed by traffic. Some of them will be hit by a bus and die on the spot. Some will wander into a bar and get drunk.

But bosses at both ends of the loop make sure that everything that needs to get from one place to another DOES get there, eventually, even if they have to retransmit it over and over.

 

[nburge]

Hey helicopter,

Thank you for the info i understand what you are saying. This is a good thread.

 

[EnOYiN]

Quote:

Originally Posted by ericj /img/forum/go_quote.gif Data integrity over network filesystems is also guaranteed.

This is without a doubt true. Try sending a file over a connection with huge amounts of packet loss. That file will get across perfectly as long as you are patient enough.

 

[jojoarmani]

Great thread!

i have my os with the installed programs, but my music and films are on a 500gig western digital. If i ever lost it, i think i would seriously lose it... It represents a lot of work and took 5 years to get to this point.

i have a friend who luckily also likes music and he has a copy of the same data on his external drive.

When are solid state hard drives coming out? at that point, i will move my data over onto it from the western digital.

Question: i dropped my external hard drive the other day. No big deal since it only fell 8 inches from the ground. Could this event have caused damage to the data (my 0s and 1s held together by a magnetic field).

i am listening to my records from it and all looks fine...

 

[helicopter34234]

Quote:

Could this event have caused damage to the data

You never know. My guess, if the disk wasn't actively accesing data when you dropped it then you would either experience a catostrophic failure (disk doesn't work at all due to major bearing failures or mechanics parts not functioning or the region it was accessing got damaged) or nothing at all. It is unlikely that dropping an HD would affect a specific sector of the disk (unless it was accessing that sector at the time). But to be careful you should keep a running backup.

You wouldn't believe how much you would end up paying to recover your data exactly how it was before it was destroyed. This is why data recovery business charge big bucks yet are still in business. Us OCD types spend many hours (and $) configuring our collections of data (music, pictures, ...) so we tend to value our data highly. That is why I started this thread, because if I wasn't aware of what I recently learned about the volatility of my current data backup scheme, I would want someone to tell me about it. To make an anaology to a different obession of mine (photography) I find it ironic that many people think they are doing themselves a service by backing up all of their photographs by digitally scanning in their negatives (or slides) for posteristy (you know, so the grand kids can see them) yet your negatives (or slides) will most likely outlive your digital version many times over.

So I can only offer my own personal wisdom. The best advice I can offer for maintaining your digital archives is to have redundant harddrives which are constantly updated and verified for readability (by comparing the mirrors). If you are uber paranoid (like me) your backup will be off site (can tend to be difficult to achieve). As far as I have found, no digital archiving system is foolproof and permanent (to any reseanable time span) so the next best thing (given how cheap hard drives are) is constantly monitoring and replacing dead HD's in your system.

 

[ericj]

If the drive heads were parked when the drive fell - powered off, or not actively accessing - it generally shouldn't corrupt the data. Might do mechanical damage though.

Your bits are 1's and 0's represented by the magnetic charge of a physical particle that is well adhered to what these days is usually a ceramic platter. The 'data' does not just fall off the disk.

Not like the old days, with token ring networks where the token could fall out and the IS crew would spend the rest of the day looking for it under a receptionist's desk. ethernet worked better, but then you were running a volatile ether around the building in little tubes . . .

 

[jojoarmani]

Sounds like i was lucky!

The external HD was turned off, so that is good news, and it seems to be fine since most sectors of my music are playing fine.

way back when i was backing **** onto plastic DVD, i am not exactly positive on the brand, the plastic on the dvd was cracking off! it hadn't even been 2 years i owned them!

Solid state hard drive will be the thing when they come out!!