Long Block Data Standard Finalized

An anonymous reader writes "IDEMA has finally released the LBD (Long Block Data) standard. This standard, in work since 2000, increases the length of the data blocks of each sector from 512 bytes to 4,096 bytes. This is an update that has been requested for some time by the hard-drive industry and the development of new drives will start immediately. The new standard offers many advantages — improved reliability and higher transfer rates are the two most obvious. While some manufacturers say the reliability may increase as much as tenfold, the degree of performance improvement to be expected is a bit more elusive. Overall improvements include shorter time to format and more efficient data transfers due to smaller overhead per block during read and write operations."

Comments

Thats a lot a bits

[rambag]

Yeah why 4092 bytes? Why not 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 bytes? It seems to me to be the best option

Re:Why 4096?

[42forty-two42]

Operating systems tend to use 4096-byte blocks already, as that's the size of a memory page on x86 and amd64. If you were to require 16kb transfers, then the block cache would have to start allocating contiguous four-page groups for DMA transfers and the like, which could be difficult if memory is fragmented; in comparison, pages are the basic allocation unit for RAM, so 4kb's easy to find.

Re:Discussed Since 2000?

[RingDev]

Saying 4096 was probably the easy part. Of course someone probably had to research what the largest (time efficient) and smallest (space efficient) block size would give the greatest advantage in space/time for current average files. But eventually you get into the issue of working with Hard Drive manufacturers who likely have to redesign some circuits and controls _from scratch_, BIOS developers who have to recode to detect and support two different standards, and OS/Driver developers who also have to deal with any low level changes...

You're talking about interacting with likely hundreds of companies trying to come up with a single standard that 1) they can all agree on and 2) won't make any of them lose money. Good luck.

-Rick

Re:Higher Reliability?

[5pp000]

The longer block sizes add reliability because the error correcting codes have more to work with at a time (more data bits, but also more ECC bits).

As for wasted space, that's under the filesystem's control, not the drive's.

Re:Why 4096?

[AKAImBatman]

Parent is correct. Pretty much every paging-capable microprocessor in existence uses 4K memory blocks, thus why they're the natural size for a hard disk. In the x86 world, the next step up is 4MB blocks. Burst performance of modern hard disks is quite good, but I have to wonder if 4MB blocks would be helpful or harmful to overall system performance? It might reduce the number of pages, that's for sure.

Re:Why 4096?

[geekoid]

yeah, sure. Give a logical AND knowledgable answer.
Way to ruin the curve.

What about the MBR?

[QuantumG]

Trying to fit an entire virus into 512 bytes was always a challenge.. but 4096 bytes? That's too easy!

Error correction better over larger blocks

[EmbeddedJanitor]

If you're working with a certain number of ecc bits per data bit, then the number of corrections you can perform increases with an increased data block size. Oversimplifying, just for explanation here:

Let's suppose you can fix one error per 512 byte block or 6 errors per 4096 byte block. Intuitively that might seem like a step back because 6/8 is smaller than 1, but that is not so. If you have 512-byte blocks and get two errors in a 512-byte sequence then that block is corrupt. However if instead you're using 4096 byte blocks then a 512-byte sequence within that block can have two errors since we can tolerate up to 6 errors in the whole block.

Or put another way, consider a 4 k sequence of data, represented by a sequence of digits dependent on the number of errors in each 512 bytes. 00000000 means no errors, 03010000 means 3 errors in the second block and 1 in the fourth block (ie a total of 4 errors in the whole 4096 bytes). With a scheme that can fix only one error per 512 bytes, the block with 3 errors cannot be corrected (because 3 > 1), but in the system which fixes up to 6 errors per 4096, the errors can be fixed because 4 6. This means that the ECC is far more reliable.

Longer != Better

[snoyberg]

I have to disagree with the whole premise here. I know that people always say that longer is better when it comes to hard drives, but I've never had any reliability problems with my smaller one. Not only that, but I've had very fast transfer rates under all sorts of strenuous loads.

Wait, we're talking about storage devices? Never mind...

Re:Oh great

[avxo]

Now when I want to update just 256 bytes, instead of reading 512 bytes, changing 256 of them, and writing 512 back, I now have to do this with 4096 bytes. So I end up transferring 3584 more bytes than I otherwise needed to.

So, your O/S requires that you issue all read and write operations using the hard drive's native block size? That must suck. What else must you do? Setup DMA manually in your app? Solder a microcontroller onto the board perhaps? Sarcasm aside, you seem to have a fundamental misunderstanding of what this change achieves, who it will affect, and how. Other posters have addressed those very issues eloquently, so I won't go into that.

They really could do this transparently. Let the driver write anything in any range.

Sorry to burst your bubble but it already is done transparently. The O/S lets you write anything -- from a single byte, to gigabytes -- transparently; all you do is tell the O/S read n bytes of file F so and so into buffer at x, or write m bytes from buffer at y into file F, which is the interface that 99% of programmers use. And after what you wrote above, I find it hard to believe that you are writing the specialized software, low-level drivers and/or controller microcode that could potentially be affected by this change.

Re:Sounds like a good idea to me.

[Dan Ost]

If that kind of lossage bothers you, use a file system that can pack multiple file tails into the same block (reiserfs for sure, ext4 will too, I think). If you've got lots of small files, the impact can be surprising (my portage tree shrunk by about 100MB just by moving it from ext3 to reiserfs!). I've never noticed a difference anywhere else, however.

blocks and clusters

[ceroklis]

To all the posters complaining about the loss of space when they will be forced to use 4096 instead of 512 bytes to store their 20 bytes file:

• The cluster size (unit of disk space allocation for files) need not be equal to the physical block size. It can be a multiple or even a fraction of the physical block size. It is fairly probable that you are already using 4K clusters (or bigger), so this will not change anything. This is for example the case if you have an NTFS filesystem bigger than 2GB.
• Not all filesystems waste space in this manner. Reiserfs or EXT3 can pack several small files in a "cluster" .

Re:Discussed Since 2000?

[HtR]

Creating new standards takes time. After some searching, I found the minutes from their annual meetings since they started in 2000.

2001 Chair: "How about we double it?" Vote: Nay

2002 Chair: "How about we triple it?" Vote: Nay

2003 Chair: "How about 4x?" Vote: Nay

2004 Chair: "How about 5x?" Vote: Nay

(minutes from intervening years were tragically lost)

2007 Chair: "How's about 8x?" Vote: Yay

Re:CD error recovery unrelated to block size

[ElecCham]

I can speak with some authority on this - I work for one of those aforementioned hard-drive manufacturers, and have been doing a small amount of work on this exact thing.

The easy answer is this: in order to do ECC-like data checking on a larger set of data (say, a group of eight 512-byte sectors), it means that if you want to write sector three of that eight, you end up having to re-read the whole thing before you do anything else - thus basically giving you 4,096-byte "sector" anyway.

The other half of that answer is this: do you know what the "real" storage capacity of a CD is, without all the error checking? It's a bit less than double. Even most of the enterprise folks wouldn't accept a 40% hit in data density in return for what works out to not that big an increase in reliability (data redundancy doesn't buy you that much unless that data is on different spindles). They'd just rather get the whole data space and do a RAID, especially since that's what they're going to do anyway.