Technical question about backing up for long term (hard drives)

[nobb]

This question just popped into my head today. My main music source is my laptop and everything is backed up to an external drive which I only turn on maybe once every 2 weeks to update files. So my laptop is the reference source and it is also the center of all my music data. After losing all my files once in the past, I just realize how priceless my data is to me.

Hard drives will eventually die, but what about bit errors? Most bit errors on hard drives should be corrected on the fly, but isnt there also the probability that some bit errors are not correctable? Over time, is there a possibility that files on a hard drive will become corrupt and unusable? I am thinking that even if the hard drive is still functioning correctly some files might be corrupt. Since my laptop is the reference source in which everything on my external drive is referenced to, a corrupt file on my laptop will correspond to a corrupt file on the external drive. I am using SyncToy with the "Echo" setting, so my important files on the laptop are mirrored to the external drive.

Is this even a valid concern? Or do I need to take further steps in protecting my data rather than simply echoing valuable files onto an external drive every 2 weeks? I am only 18, but thinking for the long term. I want to still be able to open up all my priceless archived memories when I am 80.

 

[Wodgy]

Quote:

Originally Posted by nobb /img/forum/go_quote.gif This question just popped into my head today. My main music source is my laptop and everything is backed up to an external drive which I only turn on maybe once every 2 weeks to update files. So my laptop is the reference source and it is also the center of all my music data. After losing all my files once in the past, I just realize how priceless my data is to me. Hard drives will eventually die, but what about bit errors? Most bit errors on hard drives should be corrected on the fly, but isnt there also the probability that some bit errors are not correctable? Over time, is there a possibility that files on a hard drive will become corrupt and unusable? I am thinking that even if the hard drive is still functioning correctly some files might be corrupt. Since my laptop is the reference source in which everything on my external drive is referenced to, a corrupt file on my laptop will correspond to a corrupt file on the external drive. I am using SyncToy with the "Echo" setting, so my important files on the laptop are mirrored to the external drive. Is this even a valid concern? Or do I need to take further steps in protecting my data rather than simply echoing valuable files onto an external drive every 2 weeks? I am only 18, but thinking for the long term. I want to still be able to open up all my priceless archived memories when I am 80.

This is actually a surprisingly valid concern that not many people think about. Modern hard drive densities are so high that physical read and bit drift failures happen. Not often, but they do happen. This phenomenon is called "bit rot."

To see how often this happens, look at the spec sheet for your drive. It will contain the line: "Nonrecoverable Read Errors per Bits Read." For desktop hard drives, it's about 1 per 10^14 bits, or one bit read wrong every 11641 Gb read. For laptop hard drives it's often higher. That doesn't sound like much, but over time these errors may add up.

New filesystems are being devised that can detect and recover from bit rot. Probably the most popular right now is ZFS, which stores a checksum along with every block, and duplicates blocks where possible, either through ditto blocks (copies filling up empty space) or via automatic drive mirroring or a new type of RAID that's impervious to the type of failures that brought down Head-Fi for weeks. This means it can always detect bit rot (and failing hard drives generally) and provided it's set up correctly, can also automatically self-repair.

Believe it or not, ZFS is free, but it only runs on Solaris, BSD, and OS X right now. It's also remarkably easy to configure (extraordinarily so compared to the usual junk you have to put up on UNIX).

Another option is the Drobo device you may have seen advertised. It does basically the same thing as ZFS, though its long-term reliability and underlying algorithms are unknown. It's also extremely expensive for what you get, given that ZFS is free and can run on any old computer that's too old for anything else.

Anyway, If you're looking towards serious long-term data storage, I'd put two hard drives together, run them as a ZFS mirrored pair, and then you don't have to worry about manually copying your data every so often. For prudence, you should periodically make a backup of your files on another drive that you put in a safe place, just in case you somehow accidentally delete the files in your ZFS pool.

If that sounds like too much work, at least store your music files as FLAC and keep doing the backups you're doing, and watch for signs of your drive failing. FLAC files have an embedded checksum that can test their integrity and if something starts to go wrong with part of your archive, you'll at least be able to tell. You won't be able to fix it unless your backup is clean (i.e. the error occurred after you made the last backup but before you made the next one), but at least you'll be able to get some idea that your archive is slowly losing integrity.

 

[nobb]

I wonder if doing a virus scan once a year (or something that will access every file on the drive) be enough to "refresh" the data and prevent bit rot. I plan on buying a new external drive every 5-10 years anyways, so would the process of taking all the data from the old drive and freshly writing it on the new drive be enough to prevent bit rot?

Most of my music is ripped at 320kbps mp3, so a few corrupt bits here and there probably shouldnt hurt the sound quality that much (compared to a mp3 ripped at 128kbps). Or am I wrong?

 

[sejarzo]

Quote:

Originally Posted by nobb /img/forum/go_quote.gif I am only 18, but thinking for the long term. I want to still be able to open up all my priceless archived memories when I am 80.

Advice from an old fart: Give some thought to rethinking that notion!

What musical data from 62 years ago is still readable by current state of the art technology? LP's were still 3 years in the future. That leaves you with 78's. Sure, some folks still listen to them, but not many...and who's building new hardware for that?

My first personal computer was a Kaypro in 1983. Like the original IBM PC, it didn't have a hard drive....just two 5.25" floppies, and 64k RAM. When hard drives first came out for PC's, they were commonly only 5 or 10 MB. Yes, MB. Entire hard drives had less than 1% of the capacity of current PC's RAM.

By the time you are 80, the technology will be entirely different......probably by four or five major generations of new storage technology.

And you will be lucky to get 10 years out of a hard drive, anyway.

 

[HFat]

In my experience, bad sectors are a bigger issue than bit rot, especially on laptop drives. But, to keep that in perspective, user or software error are bigger problems.
Dieing hard drives are also a more serious issues so, if you care about your data, back it up in more than one place! If you use hard drives, use at least two of them. If at all possible, have one sit in some other location than your other backups (to guard against theft, fire, and so on).

Considering your needs, I'm a bit skeptical about ZFS because I'm of the KISS (look it up) persuasion. Checking the checksums or whatever on the FLAC files sounds more like it. As to the files that don't have this feature, you can put them in archives.
But I wouldn't waste too much effort on this stuff anyway: user error and failing drives are the first concerns you should address. The most obvious solutions are keeping older backups around as well as current ones (there are filesystem and OS features that can help with that) and redundancy.

 

[HFat]

Quote:

Originally Posted by sejarzo /img/forum/go_quote.gif And you will be lucky to get 10 years out of a hard drive, anyway.

You'd be surprised. If you don't use them too often, they can last a long time.

But you probably won't want to keep them around that long anyway.

 

[sejarzo]

Quote:

Originally Posted by HFat /img/forum/go_quote.gif You'd be surprised. If you don't use them too often, they can last a long time. But you probably won't want to keep them around that long anyway.

I understand that.....but how many folks still have data that they need to keep sitting on 5 MB "Winchesters" from "only" 24 years ago?

My point is that the only solution is to upgrade your storage as technology allows, and do so over and over again.

 

[Wodgy]

Quote:

Originally Posted by nobb /img/forum/go_quote.gif I wonder if doing a virus scan once a year (or something that will access every file on the drive) be enough to "refresh" the data and prevent bit rot. I plan on buying a new external drive every 5-10 years anyways, so would the process of taking all the data from the old drive and freshly writing it on the new drive be enough to prevent bit rot? Most of my music is ripped at 320kbps mp3, so a few corrupt bits here and there probably shouldnt hurt the sound quality that much (compared to a mp3 ripped at 128kbps). Or am I wrong?

No, reading data periodically won't have any effect.

If you're just storing MP3s, I wouldn't bother with any of this. However, I would encourage you to start ripping future CDs in lossless... the cost of the storage is minor compared to the huge amount of time it takes to rip CDs with EAC or dbPowerAmp, and lossless means you don't have to waste that time again in the future. Transcoding to MP3 can be done later.

It doesn't sound like you have complicated needs. Your current backup scheme sounds fine.

 

[Wodgy]

Quote:

Originally Posted by HFat /img/forum/go_quote.gif In my experience, bad sectors are a bigger issue than bit rot, especially on laptop drives. But, to keep that in perspective, user or software error are bigger problems. Dieing hard drives are also a more serious issues so, if you care about your data, back it up in more than one place

Both of these are very true, though as storage densities increase, bit rot is becoming an issue as well.

Quote:

I'm a bit skeptical about ZFS because I'm of the KISS (look it up) persuasion. Checking the checksums or whatever on the FLAC files sounds more like it. As to the files that don't have this feature, you can put them in archives. But I wouldn't waste too much effort on this stuff anyway: user error and failing drives are the first concerns you should address.

The beauty of ZFS is it solves all the problems at once, from checksumming behind the scenes to prevent bit rot to detecting failing drives (the checksums get used again) to handling total drive failures. It all works together, and the filesystem is never inconsistent, no matter how the hardware fails. In a way, it is a very pure KISS system; it can't fall into the kind of chain of events that caused Head-Fi's data to be heavily damaged. Even if one discounts all the various data integrity features, being able to tell ahead of time that a drive may be about to fail is gold.

 

[HFat]

I should get into this ZFS thing. It sounds as if I'm really missing something. That said...

Quote:

Originally Posted by Wodgy /img/forum/go_quote.gif In a way, it is a very pure KISS system; it can't fall into the kind of chain of events that caused Head-Fi's data to be heavily damaged.

But in other ways (portability for one thing) it's not... not yet anyway.

Do you know what actually caused the head-fi outage by the way? Aside from the lack of backups of course...

Quote:

Originally Posted by Wodgy /img/forum/go_quote.gif being able to tell ahead of time that a drive may be about to fail is gold.

Come on... this isn't something ZFS can do.

 

[Wodgy]

Quote:

Originally Posted by HFat /img/forum/go_quote.gif But in other ways (portability for one thing) it's not... not yet anyway.

Yes, it's mainly for home-based data storage, though it may very well be the default filesystem for the next release of OS X, which would benefit laptop users. You don't get full reliability with a single drive, but it uses empty space for duplicates, so localized failures can be recovered and repaired automatically. All failures can be detected.

Quote:

Do you know what actually caused the head-fi outage by the way? Aside from the lack of backups of course...

One of the NAS disk controllers failed. It's not clear if it failed slowly (slowly corrupting the RAID) or failed in what's called the RAID write hole, where a single hardware failure can lead to corruption.

Quote:

Come on... this isn't something ZFS can do.

Because it stores checksums with every block and verifies them on every read, once the drive starts returning unreliable data occasionally, you know it immediately. This is a stronger indication that a drive is about to fail than SMART information (which some swear by, others, like the Google report, found to be next to useless). Not all failure modes can be detected this way -- for instance the drive heads flying off in the Seagate Momentus 5400.2 series couldn't be detected this way -- but most non-sudden failures can. Most cheap RAIDs until now don't do this, which makes them somewhat unsatisfactory... a person wants to know when his drives are in the process of failing!

The other neat thing that I forgot to mention is that it supports instant full-volume snapshots, on a block-by-block basis. So you can snapshot your archive every day or every hour if you like, and then if you do something accidental like delete half your music, you can pull the old data back from a snapshot. So you don't need rotating off-site backups, though it's worth keeping a single off-site backup in case something catastrophic happens. It's a nice KISS solution to most storage problems people are likely to encounter.

Veritas sells stuff like this to banks for a fortune, so it's really nice to get it free for ordinary folks.

 

[visualguy]

Quote:

Originally Posted by HFat /img/forum/go_quote.gif You'd be surprised. If you don't use them too often, they can last a long time. But you probably won't want to keep them around that long anyway.

Actually, not using them is also bad (worse than using them in some cases). For example, if you don't spin up a drive for a few years, there's a pretty high chance that it won't work.

 

[visualguy]

The best thing to do, probably, is to have multiple copies on recordable optical media, and create fresh copies every few years.

 

[HFat]

Quote:

Originally Posted by Wodgy /img/forum/go_quote.gif once the drive starts returning unreliable data occasionally, you know it immediately. This is a stronger indication that a drive is about to fail than SMART information (which some swear by, others, like the Google report, found to be next to useless). ... a person wants to know when his drives are in the process of failing!

Can you substantiate that? In particular, how many failures can you predict (as opposed to sudden deaths) and how many false positives do you get with whatever criteria you use to determine that a drive is dying? How much of an advance notice do these warnings give you?
In my experience, drives often die without warning and the ones that have occasional issues can keep working for a long time after the first error. I haven't experienced enough drive failures of late for my numbers to be meaningful though.

In any case, I don't see how you can compare that to SMART information which is a lot broader in scope.
Google (assuming you're talking about last february's paper) found that data to be useful but not predictive enough for most purposes. Note that this study didn't cover all indicators (such as "power on hours"). Is there a similar study to back up the usefulness of ZFS's data?

Without hard data, I'm afraid trusting ZFS would be as superstitious as trusting SMART.

Quote:

Originally Posted by Wodgy /img/forum/go_quote.gif The other neat thing that I forgot to mention is that it supports instant full-volume snapshots, on a block-by-block basis. ... So you don't need rotating off-site backups

LVM does the same thing, right? I understand that Windows has a similar feature now as well.

What's that got to do with off-site backups anyway?

 

[virtuoso]

dude.. hard disks are magnetic...
till the time their magnetic properties are intact..
they'l work fine!