Slashdot Mirror
Long Block Data Standard Finalized
An anonymous reader writes "IDEMA has finally released the LBD (Long Block Data) standard. This standard, in work since 2000, increases the length of the data blocks of each sector from 512 bytes to 4,096 bytes. This is an update that has been requested for some time by the hard-drive industry and the development of new drives will start immediately. The new standard offers many advantages — improved reliability and higher transfer rates are the two most obvious. While some manufacturers say the reliability may increase as much as tenfold, the degree of performance improvement to be expected is a bit more elusive. Overall improvements include shorter time to format and more efficient data transfers due to smaller overhead per block during read and write operations."
How does larger block sizes result in better reliability? Intuitively, I would almost think the opposite, since a single byte corruption means a much larger block is now erroneous. I obviously am missing something though.
-dave
http://millionnumbers.com/ - own the number of your dreams
Is there a good reason why 4096 was chosen? Is that just an artifact of this being designed in 2000? At this point very few files on the average system would be smaller than this. It seems to me they could have quite safely chosen something like 16k which would have improved things more, future proofed them more, yet still have been small enough as to not waste a tremendous amount of space (like if they chose 512k).
Why not make it variable, in that each drive can have it's own value (limited to a power of 2, between 512 and say 512k)? That way one drives today could be 4k, with drives in a few years being more without requiring another 7 years for a new standard?
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Yeah why 4092 bytes? Why not 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 bytes? It seems to me to be the best option
> NOTHING is 512 bytes anymore.
Shortcuts can easily be 512 bytes long.
For example I've got a shortcut to a file on C:\, which is 391 bytes actual size, 4096 bytes on the disk.
Actually, they're going to take up eight times as much space... YOU FAIL IT! They will waste 3636b space unused in blocks, however, instead of only 112 bytes, so they'll be wasting over 32 times as much space. But then, won't ReiserFS already store multiple files in a single block in some cases?
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
NOTHING is 512 bytes anymore
Unless you've got a powerful fetish for ASCII pr0n
Saying 4096 was probably the easy part. Of course someone probably had to research what the largest (time efficient) and smallest (space efficient) block size would give the greatest advantage in space/time for current average files. But eventually you get into the issue of working with Hard Drive manufacturers who likely have to redesign some circuits and controls _from scratch_, BIOS developers who have to recode to detect and support two different standards, and OS/Driver developers who also have to deal with any low level changes...
You're talking about interacting with likely hundreds of companies trying to come up with a single standard that 1) they can all agree on and 2) won't make any of them lose money. Good luck.
-Rick
"Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
In addition, it doesn't matter whether the file is less than 512 or, in this case, 4096 bytes. What matters is if the 'size % block_size' is non-zero. I.e. let's say the file is 4090 bytes. It will fit just fine, and you'll only waste 6 bytes. Now the file is 4100 bytes, only 4 bytes over. Except now you need 2 blocks, and thus waste 4092 bytes.
Sure, on a multi-GB file that's not going to matter too much, as even on a TB drive you can only have a few hundred of those, and who's going to miss that 1MB?
However, there's plenty of other files that hover between 1k and 10k, 10k and 100k, 100k and 1MB where those tiny fractions do add up.
That said, GP is still right. Say you do have a TB drive.. unless you only have a few free MB left, you're not going to worry too much about the losses from block sizes.
Trying to fit an entire virus into 512 bytes was always a challenge.. but 4096 bytes? That's too easy!
How we know is more important than what we know.
These kinds of incremental standards are simply not forward-looking! I propose that the data block size be set to a minimum of 2^32 bytes.
Gamingmuseum.com: Give your 3D accelerator a rest.
Let's suppose you can fix one error per 512 byte block or 6 errors per 4096 byte block. Intuitively that might seem like a step back because 6/8 is smaller than 1, but that is not so. If you have 512-byte blocks and get two errors in a 512-byte sequence then that block is corrupt. However if instead you're using 4096 byte blocks then a 512-byte sequence within that block can have two errors since we can tolerate up to 6 errors in the whole block.
Or put another way, consider a 4 k sequence of data, represented by a sequence of digits dependent on the number of errors in each 512 bytes. 00000000 means no errors, 03010000 means 3 errors in the second block and 1 in the fourth block (ie a total of 4 errors in the whole 4096 bytes). With a scheme that can fix only one error per 512 bytes, the block with 3 errors cannot be corrected (because 3 > 1), but in the system which fixes up to 6 errors per 4096, the errors can be fixed because 4 6. This means that the ECC is far more reliable.
Engineering is the art of compromise.
Block size has absolutely nothing to do with how much redundancy you can build in, and I fail to see the logic in assuming so. Makes absolutely no sense. The 2048 bytes stored on a sector of a CD only refers to your data, and absolutely none of them have anything to do with the CD's error-correction mechanisms. They add lots of extra bits to make up their error-correction, over and above your 2048 bytes of data. But, the point is it doesn't matter how much space you reserve to hold user data, you can arbitrarily reserve any amount of space you want for error-correction bits. You can have 16-byte sectors with 16MB of error-correction. Now, *that* would be a lot of redundancy. But certainly something you could do if you want to, and there's not going to be very many people arguing that those 16-byte sectors weren't covered by much redundancy. I doubt anyone would ever use that much redundancy, obviously, but it's just an outrageous example to show that the amount of redundancy has absolutely nothing to do with how much user data is stored per sector.
...if you have Windows loaded.
Engineering is the art of compromise.
Did the space for the bootloader just increase to 4096 as well? For those unaware, the BIOS loads just the first sector of the disk into memory, the bootloader takes it from there. It would certainly let them get a lot more resilient, now they only barf if things are not as expected.
Live today, because you never know what tomorrow brings
With some probably minor inconvenience, you could fix that by using a Zipped archive. And someone will likely come up with a low-impact solution based on that.
I have to disagree with the whole premise here. I know that people always say that longer is better when it comes to hard drives, but I've never had any reliability problems with my smaller one. Not only that, but I've had very fast transfer rates under all sorts of strenuous loads.
Wait, we're talking about storage devices? Never mind...
Thank God for evolution.
So... 2 gigabytes?
FGD 135
If that kind of lossage bothers you, use a file system that can pack multiple file tails into the same block (reiserfs for sure, ext4 will too, I think). If you've got lots of small files, the impact can be surprising (my portage tree shrunk by about 100MB just by moving it from ext3 to reiserfs!). I've never noticed a difference anywhere else, however.
*sigh* back to work...
Creating new standards takes time. After some searching, I found the minutes from their annual meetings since they started in 2000.
2001 Chair: "How about we double it?" Vote: Nay
2002 Chair: "How about we triple it?" Vote: Nay
2003 Chair: "How about 4x?" Vote: Nay
2004 Chair: "How about 5x?" Vote: Nay
(minutes from intervening years were tragically lost)
2007 Chair: "How's about 8x?" Vote: Yay
Have you tried turning it off and on again?
Ask Wikipedia
It's in the table "Allocation and layout policies". Look at both tail packing and block suballocation.
There are a few others that do, but not many. (JFS, QFS, NWFS, and VMFS are marked yes; NTFS and ZFS are marked partial.)
It's about effing time!
512 bytes was good for floppy disks. I think we should have started upping the sector size around the same time as we hit the 528mb 1024-cylinder limit back in the early 90's. Considering that a modern hard drive has anywhere from one-half to two billion sectors, and that's some serious overhead for no reason. Error-correction is "easier" if it's spread over larger blocks. Why ? Because most files are quite large, and corrupting a 512 byte chunk is just as bad as corrupting a 4096 or 8192 byte chunk, because it's hosing the file either way. Might as well pool the ECC together and offer better protection for the large block, while still wasting less bits than the sum of all the small sectors' ECC. Even without the proposed ECC algorithm overhaul, larger blocks would allow more usable data per platter.
The downside is that we've had 512 byte sectors for so long, everyone's hardcoded the number in their apps and drivers. The biggest risk involved is to patch all that software... one little glitch could hose a ton of data.
-Billco, Fnarg.com
Let's say you have 4096 bytes arranged as 8x512-byte blocks and each block can correct one error. Now lets say that we RANDOMLY (ie statisticly independently) introduce, say, 4 errors into that set of 8 blocks. Sometimes the errors will fall so that there are at most one error per block. That is correctable. Sometimes the errors will fall so that there are more than one per block. In that case data will be lost.
However, if we can correct up to, say, 6 arbitrarily placed errors per 4096 bytes we can then have 4 errors anywhere in that block and we won't lose data. It does not matter whether they are spread out or clustered together we can always handle those errors.
This makes for stronger correction.
Engineering is the art of compromise.
Debian Finally Supports Long Block Data
What's so hard about that?
Go read the Linux Kernel mailing list, and you'll find interactions between the block layer and the virtual memory are one of the most difficult things to make work right in an operating system. The size of the block on the hard disk matters most to the driver, its mostly transparent to the rest of the operating system. The only thing it changes on actual file systems is the minimum filesystem block should be 4K minimum.
If it's already transparent, then what is the big deal? If what you say is true, they could make blocks/sectors as long as they want and we won't need to know (except the driver writers need to know what constraints exist in the interface to send the read and write commands to the drive).
Sorry to bust YOUR bubble, but I do know how the OS works, and how it's interface works. The issue depends on what blocksize the commands between driver and hardware require. If you cannot instruct the hardware at a finer resolution than 512 bytes at a time, then writing 256 bytes really does mean that the OS at some layer (maybe in the driver, maybe at a higher layer, depending on the OS) will read 512 bytes, update 256 of them, and write 512 back. If that interface is being changed to require 4096 bytes minimum per I/O operation, then it really does increase the needed transfer work going on between the driver and hardware.
My post was meant in part to be humorous, and in part to raise an issue. The issue is the transparency of the driver to hardware interface. I do know from random encounterings that for IDE it really does require the driver do I/O operations in multiples of 512. That really does affect the 256 byte I/O request from userspace, but it's not really a serious request due to the caching nature of modern operating systems. And there is no reason in the world why they cannot have been doing 4096 byte or longer blocks for years or decades. If they have been doing it, there's no news (but that's typical Slashdot, so how would we know). Given the I/O command interface is in units of 512 bytes, it's probably convenience that whatever long block size the drive uses be an exact whole multiple of 512 bytes, even that is not essential. A couple decades ago I wrote a "driver" for a mainframe OS that handled I/O requests in units of 4096 bytes, with an on-disk blocksize of 18432 bytes. If you do your arithmetic, you can see that is 4.5x. So some I/O request blocks end up spanning physical blocks on disk. No big deal.
Now I have not been following PC hardware technology very much, so I don't know how much has been added to the I/O interface capability. If they have added a new byte-level I/O command set, then fine. Do you even know if they have?
now we need to go OSS in diesel cars