Slashdot Mirror
Developer Shares A Recoverable Container Format That's File System Agnostic (github.com)
Long-time Slashdot reader MarcoPon writes: I created a thing: SeqBox. It's an archive/container format (and corresponding suite of tools) with some interesting and unique features. Basically an SBX file is composed of a series of sector-sized blocks with a small header with a recognizable signature, integrity check, info about the file they belong to, and a sequence number. The results of this encoding is the ability to recover an SBX container even if the file system is corrupted, completely lost or just unknown, no matter how much the file is fragmented.
There is some confusion as to what this is actually doing.
Most filesystems have use special structures to store the name and location of your files on the drive. Directories, cluster bitmaps, etc etc. The reason why it's difficult at best to recover files from a hard drive when parts of the filesystem have been damaged is that it's difficult to identify where on your hard drive the files are. Besides the special filesystem directories, no where else stores information on what is stored where. If you lose the directory it's hard to tell one file's data from another on your hard drive.
That is where SBX comes in. What it does is make sure that every physical sector that stores data for a particular file is labelled with a number that identifies that file, and a sequence number so you can reconstruct what order that piece is in the original file. Really, for the amount of overhead, something like that should be embedded into every filesystem. Basically a distributed backup of all the filesystem metadata.
Some people are criticizing this that is solves non problems. I disagree. While it isn't the solution to global warming, it is both simple and clever (and will thus suffer from a lot of people who will disparage it out of a "well anyone could have thought of that" attitude). It won't save you from a full hardware crash. It won't save you from physically bad sectors in that file. What it will save you from is accidental deletion and from loss of the filesystem's metadata structures. How often does this happen? Twice to me from failures of a whole-disc-encryption system driver.
I wouldn't use this for every file, but for critical ones, sure. Why not. The problem is, where it is most useful, for very volatile files that change a lot (databases etc) between backups, is where it can't really be used until/unless different applications start supporting it. So it unfortunately has limited use in the places where it would really help the most. Like I said above, this sort of thing really needs to get rolled into a filesystem. The amount of overhead it costs is meaningless in today's storage environment.
If you can meaningfully compact *anything* that's encrypted, the encryption was improperly implemented. You *always* want to compact files prior to encryption, and a well-encrypted compressed file should be statistically indistinguishable from random noise.