Slashdot Mirror
Developer Shares A Recoverable Container Format That's File System Agnostic (github.com)
Long-time Slashdot reader MarcoPon writes: I created a thing: SeqBox. It's an archive/container format (and corresponding suite of tools) with some interesting and unique features. Basically an SBX file is composed of a series of sector-sized blocks with a small header with a recognizable signature, integrity check, info about the file they belong to, and a sequence number. The results of this encoding is the ability to recover an SBX container even if the file system is corrupted, completely lost or just unknown, no matter how much the file is fragmented.
Thanks, looks interesting. I can see some applications for use in long term storage... it's better to get some data back rather than lose it all.
The Geek in Black
I know my BCD's (when I'm Sober)
That's an interesting property, but what's the use case?
How often does your filesystem get corrupt and instead of restoring from backups, you curse the fragmented tar file that can't be reassembled?
How practical is it to keep files in an sbx container rather than extracting them? Can apps read files inside an sbx container?
...but this is better than a backup, how, exactly?
What if your file system and/or hardware uses a different sector size? Didn't those change size over the last decades?
#DeleteFacebook
Doesn't GPG already compress the input data for improved input cleartext?
Ezekiel 23:20
I did a quick read of the code and see that it relies on a magic cookie in the first four bytes of every physical sector to identify a block. This may not work for files small enough to fit entirely within the MFT on NTFS since that data isn't guaranteed to be aligned on a physical sector. There are other filesystems that store small file segments in the metadata structures as well.
There's no way it can. LUKS is great but it wastes tons of disk space on vms.
It can! Just turn on discard (and have the system inside issue trim commands). This does have an impact on encryption, though, which might or might not be acceptable for you: it is possible to tell used from unused disk space, which leaks information about usage patterns inside the VM.
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
Yes. You could supply a password that is hashed and the the hash is XOR-ed against each block, signature included. It's not really strong encryption by any means (that could be implemented in a later version, at the moment I just wanted to keep it simple), but it's probably obfuscated enough to avoid detection. Especially if one didn't expect an SBX container being there, and/or isn't prepared to go trough a lot of data collection to find some blocks with the same first 4 bytes (different depending on the password used) sprayed around.
SeqBox
From what I understand his idea isn't to be hardware failure-proof, it's to be file system failure-proof.
#DeleteFacebook
Yes, exactly. And it's not just about data recovery. For example, I'm not sure how "generally useful" that would be, but I found myself using it for extracting/exchanging data from disk images formatted with a foreign file system.
SeqBox
I mean the chances of the filesystem being corrupted without the file itself also being corrupted seem small to none to me.
Unlike HDD controllers, SSD controller do wear-leveling, so there is no guarantee that your data will be written as as a contiguous block of memory (regardless of what the filesystem says), only that it will be in 4096 byte blocks. Recovering deleted data from a SSD is no simple task because it means you need to know or guess the controller behavior for wear-leveling in order to go back and find the order of previously written data. With this you would be able to just read the raw memory even after the controller has been reset and still be able to recover the data. I think it would be a nice option to have a filesystem be able to encode user files in something like this highly recoverable format. The only real problem is that the file has to be completely rewritten even if you only modify part it in order to differentiate the new version from the old version.
Anons need not reply. Questions end with a question mark.
It seems to me this would be a lot more useful if it directly incorporated forward error correction.
There is some confusion as to what this is actually doing.
Most filesystems have use special structures to store the name and location of your files on the drive. Directories, cluster bitmaps, etc etc. The reason why it's difficult at best to recover files from a hard drive when parts of the filesystem have been damaged is that it's difficult to identify where on your hard drive the files are. Besides the special filesystem directories, no where else stores information on what is stored where. If you lose the directory it's hard to tell one file's data from another on your hard drive.
That is where SBX comes in. What it does is make sure that every physical sector that stores data for a particular file is labelled with a number that identifies that file, and a sequence number so you can reconstruct what order that piece is in the original file. Really, for the amount of overhead, something like that should be embedded into every filesystem. Basically a distributed backup of all the filesystem metadata.
Some people are criticizing this that is solves non problems. I disagree. While it isn't the solution to global warming, it is both simple and clever (and will thus suffer from a lot of people who will disparage it out of a "well anyone could have thought of that" attitude). It won't save you from a full hardware crash. It won't save you from physically bad sectors in that file. What it will save you from is accidental deletion and from loss of the filesystem's metadata structures. How often does this happen? Twice to me from failures of a whole-disc-encryption system driver.
I wouldn't use this for every file, but for critical ones, sure. Why not. The problem is, where it is most useful, for very volatile files that change a lot (databases etc) between backups, is where it can't really be used until/unless different applications start supporting it. So it unfortunately has limited use in the places where it would really help the most. Like I said above, this sort of thing really needs to get rolled into a filesystem. The amount of overhead it costs is meaningless in today's storage environment.
If you can meaningfully compact *anything* that's encrypted, the encryption was improperly implemented. You *always* want to compact files prior to encryption, and a well-encrypted compressed file should be statistically indistinguishable from random noise.
You obviously tried to keep the per-block header small to minimize overhead. But that has caused questionable decisions that may make this format less useful than it could be.
It's surely a compromise, but I think it's pretty sensible for the present version (but some variations can surely be implemented as different versions, to better suite different scenarios).
Firstly, at 48 bits, the UID is a bit short. If UIDs are chosen randomly and with even distribution, there's a 1 in 1000 chance of a duplicate UID with just 750000 files.
That seems a bit off, 48bit assuming even distribution would give 281,474,976,710,656. But again, 750.000 files would seems an enormous number for the practical uses I was thinking about at the moment.
Secondly, the block sequence number is a 32bit value, so 4 billion blocks in a file max. With this format, files are limited to 2TB.
Yes, 2TB with the 512 block, or 16TB with 4K block. It's not good for everything but it's probably good for a lot of case. But again, it can be easily upgraded if needed.
Thirdly, the 2-byte checksum is too small.
A 2 byte CRC seems to be plenty for 512 bytes, and even for 4KB isn't bad. Here is not about detecting tampering, but just to distinguish a random block that just happen to have the right signature, with a specific UID & sequence number, from a real SBX block.
That said, again I agree that for applications that could envision a lot of files with far larger sizes, a different kind of header, with expanded fields (and some other things) would be better suited.
SeqBox
Compression before encryption often results in a padding oracle or other problems. If you're designing a system that is supposed to be secure, avoid compression until you fully understand the issues. Avoid compressing and encrypting chosen plaintext at all - you'll never be sure you understand all of the issues with that.
Hash the password AND block number through a key-stretching routine to get the encryption key. It is important to avoid using the same key for all blocks. If different blocks are XORed with the same key, I can still see your penguin:
https://blog.filippo.io/the-ec...
Could this be also be used when the file contents are deliberately separated? Eg, distribute the file pieces (sectors?) to different audiences / storage locations, such that one has to get cooperation from all piece-holders to retrieve the net results? Eg: nuclear launch codes, and other less dramatic scenarios.
The Lisa and early Macintosh drives supported 532-byte sectors. The extra sectors were used for "tags" - basically a less-sophisticated version of this scheme and without the "block 0."
For details on why "tags" were eliminated, see Macintosh Technote #94, "Tags," by Bryan Stearns, November 15, 1986.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Been struggling to understand this for the last 3 decades (not full time) ... :/
I know about that, but what does that have to do with the fact that there's apparently already an extra compression step in the pipeline above?
Ezekiel 23:20
My comment doesn't directly relate to having the *two* compression steps. I probably should have replied to the same person you replied to.
Thousands of images are often used in medical imaging for a single scan. I have a production filesystem with 2B images, over 200TB. 16TB file sizes aren't all that hard to come by either. Obviously there is also the birthday problem, ZFS alleviates it by using bit comparisons in combination with 128 bit checksums.
I'm baffled that 48 bit checksums is still considered good enough nowadays.
Custom electronics and digital signage for your business: www.evcircuits.com
There's no 48 bit checksum in the specs. The 48 bit field is a UID that's used to id a file, and it could be anything "unique enough" for the job. For example in the case of a digital camera that saves photo on an SD, that could be even just the progressive number used in the filenames. Or could be an hash, or just a random number. A giant file repository wasn't exactly the first thing on my mind when I developed the concept. 2TB/16TB could be a limit in some situations, undoubtedly, but again it's reasonable enough in a wide variety of other.
But note that the specs include a version number and so can be flexible. It's simple to add a new version with far larger limits, extending the size of the header, if needed, and that was the plan depending on the interest and usage that could come up.
SeqBox
I want to storage that is File System Atheist!
(And would that be like Write Only Memory?)
Tracy Johnson
Old fashioned text games hosted below:
http://empire.openmpe.com/
BT
I wrote three pieces of software:
Strongbox
Throttlebox
Clonebox
Then you chose Seqbox. :)