Ask Slashdot: Best File System For the Ages?

New submitter Kormoran writes: After many, many years of internet, I have accumulated terabyte HDDs full of software, photos, videos, eBooks, articles, PDFs, music, etc. that I'd like to save forever. The problem is, my HDDs are fine, but some files are corrupting. Some videos show missing keyframes and some photos are ill-colored. RAID systems can protect online data (to a degree), but what about offline storage? Is there a software solution, like a file system or a file format, specifically tailored to avoid this kind of bit rot?

Comments

bit rot

[Anonymous Coward]

zfs

Re:bit rot

[tlhIngan]

It's pretty sad that in this day and age, only one person has highlighted the relevance of ZFS here, and they're an AC. Someone mod parent up. RAID is borderline necessary if you don't have multiple backups, (to recover from in the event of random corruption caused by gamma rays from outer space or a butterfly flapping their wings on another continent or whatever) but so far as I know, only ZFS has built-in checksumming to detect/prevent the data corruption in the first place.

No, RAID Is not sufficient to prevent bit-rot. In fact, RAID can accelerate it. You see, using a redundant mode like 1, 5, 6, most controllers (software and hardware) will only read enough disks to get the data, 1 drive in the case of RAID1, N-1 for RAID5 and N-2 for RAID6 (the non-parity ones, to save a parity calculation). But the drives can return bit errors - it's rare, but it does happen (there's a undetectable fault error rate, something along the lines of 1 in 10^20 bytes read or so will have an undetected error). And this the RAID controller will happily return to you since it didn't check the redundant drives to verify correctness. And it's possible it gets written back corrupted, thus causing corruption.

You really need something like ZFS which puts a checksum on every file and verifies it, so if it does get an error it can resolve it.

Re:bit rot

[__aaclcg7560]

You really need something like ZFS which puts a checksum on every file and verifies it, so if it does get an error it can resolve it.

ZFS also has its own flavors of RAID 1/5/6.

Re:bit rot

[MightyMartian]

Whose to say zfs will be around in a few decades?

The real solution here is relatively frequent backups, multiple copies in different filesystem and physical formats (ie. flash, hard drive, optical). Over time you just keep moving your file store to the new mediums. I have files that are over twenty five years old now, some of them coming from DOS and Windows 3.1, others from my old original Slackware 3 installs. Along the way some of those files have been on CD-Rs, DVDs, early USB thumb drives, various hard drives running everything from FAT, FAT32, ReiserFS, HPFS, NTFS, ext2 and ext3. And I'll keep on doing that until I drop dead, and I'll leave it up to my family to decide whether they want to keep any of the documents, pictures, music files, videos and so on that I've been collecting.

At no point do I ever assume a mere file system sitting on one physical and/or logical volume is ever going to do the job of keeping my files available over the long haul. RAID and file systems in all their glory are not intended for that. Multiple physical copies at multiple locations on multiple types of media, that's the only real way to assure your files remain accessible and safe over time.

Re:bit rot

[shellster_dude]

ZFSonLinux works just fine. I've got a huge Debian based NAS running it. Sure, you can't really boot to it, but who cares? I can reinstall the base OS, the irreplaceable part is my personal files which are on a ZFS RAIDZ2 with an encrypted cloud backup.

Re:bit rot

[Spazmania]

He said a filesystem for the ages. While it has wonderful features, ZFS isn't even a filesystem for this age, let along ages to come. FAT32 and ISOFS are your best bets for being readable 20 years from now.

Bear in mind that your hard disk checksums each block and returns an error if the block is uncorrectable upon read rather than give you bad data. So, if you're getting bit rot at all then you have a hardware problem.

With or without a hardware problem you want to be able to recover your data. The answer is par2, such as parchive or QuickPar. Par2 uses a Reed-Solomon code to take a set of source files and produce a set of recovery files such that the original files can be checked for correctness and up to N original files can be corrected where N is the number of recovery files created.

And that's your answer. A filesystem like FAT32 or ISOFS that's likely to still be implemented in future OSes and a recovery files which let you rebuild anything that suffers from bit rot.

Re:bit rot

[Anonymous Coward]

No it isn't. It's completely wrong on a number of things.

1. "ZFS isn't even a filesystem for this age" - WTF does that even mean? The software exists and is open source; it runs on BSD which is one of the most conservative OSes around and is used by many companies who take data storage seriously. It'll be around for decades.

2. "if you're getting bit rot at all then you have a hardware problem" - Wrong. Bit-rot is an issue inherent to any storage medium and will, with some low-but-not-low-enough-to-ignore-on-large-datasets probability occur on any hard drive. If you think a hard drive where bit-rot can occur is faulty then there isn't a functioning hard drive on the planet.

3."With or without a hardware problem you want to be able to recover your data. The answer is par2" - Or just use ZFS (or BTRFS) which has the same feature built into the filesystem so you don't have to manually generate recovery files every time your data changes.

Re:bit rot

[Spazmania]

"ZFS isn't even a filesystem for this age" - WTF does that even mean?

It means that even back when FAT was a johnny come lately it already had greater market penetration than ZFS. With decades behind it and broad market penetration today, there's good reason to believe it won't vanish with the advent of the next development in filesystem architecture. ZFS is likely to be a blip on the radar, a pause before the next innovation. Not what you want for an archival format.

Bit-rot is an issue inherent to any storage medium

Bit rot, aka corrupted data, is not inherent to correctly operating hardware. As implemented, you'll see tens of thousands of unreadable blocks on a hard disk before you see a single one in which data has been undetectably corrupted. Every single sector gets a checksum in hardware and if the checksum does not pass you get the famous Abort Retry Ignore. For most storage you get Forward Error Correction coding so that some number of bit errors can be corrected on read before having to throw an error.

When you see bit rot, the storage media is usually not at fault. More often the data passes through faulty non-parity ram, a noisy memory bus or an overheated controller and gets corrupted on its way to storage rather than getting corrupted at rest on the storage. It died when you used an overclocked piece of garbage to copy it from an old hard disk to a newer, bigger one.

Re:bit rot

[Spazmania]

Bit-rot is an issue inherent to any storage medium

Here's a quick article which explains how hard disks use error correcting codes so that the user-level experience is no bit rot but rather many many read failures before even a single block of undetectably corrupted data. Next time you can know what you're talking about.

http://www.pcguide.com/ref/hdd...

Re: Stone tablet and chisel

[Anonymous Coward]

Damn beat me to it!
Stone is secure!

Dude, if your hard drives were fine, your files wouldn't be corrupted. Keep RAID backups if you want a solution. The file system doesn't make a Fing difference.

RAID

[buchner.johannes]

RAID systems can protect online data (to a degree), but what about offline storage?

Still RAID is a good choice for your redundancy of choice.

Or paper: http://ollydbg.de/Paperbak/#1

Error correction codes. PAR2, btrfs, partitions,VM

[raymorris]

The magic phrase to Google is "error correction codes" (ECC).

PAR2 uses Reed-Solomon error correction. parchive is the ECC file format specification, for Linux you will want PyPar or par2tbb, and on Windows you use a GUI called QuickPar.

Btrfs can be set to use ECC on a single disk.

You can slice a single disk into partitions and then use RAID1 or LVM mirroring, or RAID5 or RAID6. LVM can alao be useful to divide (and combine) any number of drives into any number of volumes, then you can RAID across the volumes.

If you Google "ecc disk", "ecc backup", or "ecc archive" you'll find other options, with details about each option.

ZFS and lots of redundancy

[Chewbacon]

ZFS will guard against bit rot. That's not enough. RAID isn't enough. You need redundancy outside your home or office. Cloud maybe expensive for the amount of data you have, but Amazon S3 maybe the most affordable in that range. You could get S3 for maybe $15-20 a month if you have a terabyte of data. If that's cost prohibitive, rotate external drives regularly and keep one at work. You'll lose very little data since you're archiving things.

"some photos are ill-colored"

[hcs_$reboot]

That's a well known problem to photographers, photos colors are affected over time. Keep the photo negatives in a safe place!

BTRFS another option

[Anonymous Coward]

in addition to ZFS, BTRFS also handles bitrot. I'm running a 4 disk BTRFS RAID 10 in my closet, mounting to a development machine on my desk via NFS, it's been working fine for about a year, and I scheduled a scrub a couple times a month whose purpose is exactly this, to catch and correct bitrot. It does so by using a CRC32 check, and if it detects a problem on one slice it overwrites that slice from the data on the good slice.

Also I have offline and offsite backups of very important items.

When using BTRFS read the wiki and settle on a kernel version and btrfs tools version that is sufficiently up to date, it's stabilized sufficiently for these kinds of things, but only if you are careful to run an up to date version that isn't marked as buggy on the wiki

OFFLINE Storage, with FS Access

[williamyf]

That a job for Linear Tape FileSystem

https://en.wikipedia.org/wiki/...

Tape is (still) the best medium for Long Term Storge. Over the years tape (or more likely, the engineers) has agresively incorporated in the standards things like FEC codes (from reed-solomon to more exotic ones nowadays).

And since 2010, with LTFS, you can aceess the files with the convenience of a normal filesystem (but bear in mind, access is slow as hell).

Back up your data to tape (more than one set), and send it to specialized offline storage facilities (cimate controlled: ie. temperature/humidity/dust/light control) from different providers, in diferente geographical areas.

Since now there is only one true-tape standard (LTO-7 released in 2015, the tape business has been shrinking, so the proliferation os standards seems to be over now), so, if you use that today, chances are you will still find equipment to read it 50 years from now. Nonetheless, keep a few (as in two or more) SYSTEMS (Computer+Drive+SW) set up so that you can re-read. A cheapo micro formfactor mobo with an Atom Pocessor (but NOT the Atom C2000series PLEASE), linux, a 1Gbps nic and a tape drive should be more than enough. ....

Now, for Online, as other posters have said, ZFS WITH ECC memory (and therefore, a very expensive Xeon, or AMD server type mobo) and JBOD will do the trick.

Re:Error correction codes. PAR2, btrfs, partitions

[heypete]

QuickPar on Windows is long-obsolete. MultiPar is the more modern variant.

ZFS on Linux has software RAID.

[Futurepower(R)]

An Introduction to the Z File System (ZFS) for Linux.

Quote: "ZFS is capable of many different RAID levels, all while delivering performance thatâ(TM)s comparable to that of hardware RAID controllers."

That sounds good to me. I want to avoid hardware RAID because, when hardware RAID controllers fail, they are often difficult to replace.