Long Block Data Standard Finalized

An anonymous reader writes "IDEMA has finally released the LBD (Long Block Data) standard. This standard, in work since 2000, increases the length of the data blocks of each sector from 512 bytes to 4,096 bytes. This is an update that has been requested for some time by the hard-drive industry and the development of new drives will start immediately. The new standard offers many advantages — improved reliability and higher transfer rates are the two most obvious. While some manufacturers say the reliability may increase as much as tenfold, the degree of performance improvement to be expected is a bit more elusive. Overall improvements include shorter time to format and more efficient data transfers due to smaller overhead per block during read and write operations."

Comments

Higher Reliability?

[MankyD]

How does larger block sizes result in better reliability? Intuitively, I would almost think the opposite, since a single byte corruption means a much larger block is now erroneous. I obviously am missing something though.

Re:Higher Reliability?

[silas_moeckel]

I would think it has to do with the ability to have more bits for ecc type functions. Blocks would need to be terminated somehow so there is a fixed overhead per block. Reducing this overhead by a factor of 8 would leave more room for a larger parity type field and the more bits in there the larger failure that it can detect, fix and relocate. This would all assume they will not use the new space to push up sizing. Course this is all my rather speculative guesswork.

Re:Higher Reliability?

[5pp000]

The longer block sizes add reliability because the error correcting codes have more to work with at a time (more data bits, but also more ECC bits).

As for wasted space, that's under the filesystem's control, not the drive's.

Why 4096?

[MBCook]

Is there a good reason why 4096 was chosen? Is that just an artifact of this being designed in 2000? At this point very few files on the average system would be smaller than this. It seems to me they could have quite safely chosen something like 16k which would have improved things more, future proofed them more, yet still have been small enough as to not waste a tremendous amount of space (like if they chose 512k).

Why not make it variable, in that each drive can have it's own value (limited to a power of 2, between 512 and say 512k)? That way one drives today could be 4k, with drives in a few years being more without requiring another 7 years for a new standard?

Re:Why 4096?

[42forty-two42]

Operating systems tend to use 4096-byte blocks already, as that's the size of a memory page on x86 and amd64. If you were to require 16kb transfers, then the block cache would have to start allocating contiguous four-page groups for DMA transfers and the like, which could be difficult if memory is fragmented; in comparison, pages are the basic allocation unit for RAM, so 4kb's easy to find.

Re:Why 4096?

[AKAImBatman]

Parent is correct. Pretty much every paging-capable microprocessor in existence uses 4K memory blocks, thus why they're the natural size for a hard disk. In the x86 world, the next step up is 4MB blocks. Burst performance of modern hard disks is quite good, but I have to wonder if 4MB blocks would be helpful or harmful to overall system performance? It might reduce the number of pages, that's for sure.

Re:Why 4096?

[geekoid]

yeah, sure. Give a logical AND knowledgable answer.
Way to ruin the curve.

Re:Why 4096?

[42forty-two42]

Using 4MB blocks for everything would kill memory performance - and more specifically, mmap performance. Each library loaded in your system would require at least 4MB of ram - probably more, as they have code, data, and zeroed data segments. Additionally, each process would require another 4MB*n. There's no gain for doing this either, except under specialized circumstances, as the OS can already request a batch of sectors from the drive in one operation.

Re:Why 4096?

[Scott Wood]

No, x64 and ARM both use 4K pages (though ARM has 1K subpages that you can set permissions on individually). Alpha and sparc64 use 8K pages, though.

Thats a lot a bits

[rambag]

Yeah why 4092 bytes? Why not 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 bytes? It seems to me to be the best option

Re:Discussed Since 2000?

[RingDev]

Saying 4096 was probably the easy part. Of course someone probably had to research what the largest (time efficient) and smallest (space efficient) block size would give the greatest advantage in space/time for current average files. But eventually you get into the issue of working with Hard Drive manufacturers who likely have to redesign some circuits and controls _from scratch_, BIOS developers who have to recode to detect and support two different standards, and OS/Driver developers who also have to deal with any low level changes...

You're talking about interacting with likely hundreds of companies trying to come up with a single standard that 1) they can all agree on and 2) won't make any of them lose money. Good luck.

-Rick

What about the MBR?

[QuantumG]

Trying to fit an entire virus into 512 bytes was always a challenge.. but 4096 bytes? That's too easy!

Re:What about the MBR?

[Godji]

The parent raises a point though: now that we have bigger sectors, are we finally getting a standard for partition tables with more than 4 entries without using logical partitions?

Re:What about the MBR?

[ottffssent]

The word you're looking for is GPT. It has nothing to do with 4k hardware sectors, but it does support up to 128 partitions. Which ought to be enough for anybody (says the man with a 1 average number of partitions per disk in his household).

Error correction better over larger blocks

[EmbeddedJanitor]

If you're working with a certain number of ecc bits per data bit, then the number of corrections you can perform increases with an increased data block size. Oversimplifying, just for explanation here:

Let's suppose you can fix one error per 512 byte block or 6 errors per 4096 byte block. Intuitively that might seem like a step back because 6/8 is smaller than 1, but that is not so. If you have 512-byte blocks and get two errors in a 512-byte sequence then that block is corrupt. However if instead you're using 4096 byte blocks then a 512-byte sequence within that block can have two errors since we can tolerate up to 6 errors in the whole block.

Or put another way, consider a 4 k sequence of data, represented by a sequence of digits dependent on the number of errors in each 512 bytes. 00000000 means no errors, 03010000 means 3 errors in the second block and 1 in the fourth block (ie a total of 4 errors in the whole 4096 bytes). With a scheme that can fix only one error per 512 bytes, the block with 3 errors cannot be corrected (because 3 > 1), but in the system which fixes up to 6 errors per 4096, the errors can be fixed because 4 6. This means that the ECC is far more reliable.

Bootloader now 4096 bytes?

[Kjella]

Did the space for the bootloader just increase to 4096 as well? For those unaware, the BIOS loads just the first sector of the disk into memory, the bootloader takes it from there. It would certainly let them get a lot more resilient, now they only barf if things are not as expected.

Longer != Better

[snoyberg]

I have to disagree with the whole premise here. I know that people always say that longer is better when it comes to hard drives, but I've never had any reliability problems with my smaller one. Not only that, but I've had very fast transfer rates under all sorts of strenuous loads.

Wait, we're talking about storage devices? Never mind...

Re:Oh great

[avxo]

Now when I want to update just 256 bytes, instead of reading 512 bytes, changing 256 of them, and writing 512 back, I now have to do this with 4096 bytes. So I end up transferring 3584 more bytes than I otherwise needed to.

So, your O/S requires that you issue all read and write operations using the hard drive's native block size? That must suck. What else must you do? Setup DMA manually in your app? Solder a microcontroller onto the board perhaps? Sarcasm aside, you seem to have a fundamental misunderstanding of what this change achieves, who it will affect, and how. Other posters have addressed those very issues eloquently, so I won't go into that.

They really could do this transparently. Let the driver write anything in any range.

Sorry to burst your bubble but it already is done transparently. The O/S lets you write anything -- from a single byte, to gigabytes -- transparently; all you do is tell the O/S read n bytes of file F so and so into buffer at x, or write m bytes from buffer at y into file F, which is the interface that 99% of programmers use. And after what you wrote above, I find it hard to believe that you are writing the specialized software, low-level drivers and/or controller microcode that could potentially be affected by this change.

Re:Sounds like a good idea to me.

[Dan Ost]

If that kind of lossage bothers you, use a file system that can pack multiple file tails into the same block (reiserfs for sure, ext4 will too, I think). If you've got lots of small files, the impact can be surprising (my portage tree shrunk by about 100MB just by moving it from ext3 to reiserfs!). I've never noticed a difference anywhere else, however.

blocks and clusters

[ceroklis]

To all the posters complaining about the loss of space when they will be forced to use 4096 instead of 512 bytes to store their 20 bytes file:

• The cluster size (unit of disk space allocation for files) need not be equal to the physical block size. It can be a multiple or even a fraction of the physical block size. It is fairly probable that you are already using 4K clusters (or bigger), so this will not change anything. This is for example the case if you have an NTFS filesystem bigger than 2GB.
• Not all filesystems waste space in this manner. Reiserfs or EXT3 can pack several small files in a "cluster" .

Re:Discussed Since 2000?

[HtR]

Creating new standards takes time. After some searching, I found the minutes from their annual meetings since they started in 2000.

2001 Chair: "How about we double it?" Vote: Nay

2002 Chair: "How about we triple it?" Vote: Nay

2003 Chair: "How about 4x?" Vote: Nay

2004 Chair: "How about 5x?" Vote: Nay

(minutes from intervening years were tragically lost)

2007 Chair: "How's about 8x?" Vote: Yay

Re:CD error recovery unrelated to block size

[ElecCham]

I can speak with some authority on this - I work for one of those aforementioned hard-drive manufacturers, and have been doing a small amount of work on this exact thing.

The easy answer is this: in order to do ECC-like data checking on a larger set of data (say, a group of eight 512-byte sectors), it means that if you want to write sector three of that eight, you end up having to re-read the whole thing before you do anything else - thus basically giving you 4,096-byte "sector" anyway.

The other half of that answer is this: do you know what the "real" storage capacity of a CD is, without all the error checking? It's a bit less than double. Even most of the enterprise folks wouldn't accept a 40% hit in data density in return for what works out to not that big an increase in reliability (data redundancy doesn't buy you that much unless that data is on different spindles). They'd just rather get the whole data space and do a RAID, especially since that's what they're going to do anyway.