Slashdot Mirror
Exploring Advanced Format Hard Drive Technology
MojoKid writes "Hard drive capacities are sometimes broken down by the number of platters and the size of each. The first 1TB drives, for example, used five 200GB platters; current-generation 1TB drives use two 500GB platters. These values, however, only refer to the accessible storage capacity, not the total size of the platter itself. Invisible to the end-user, additional capacity is used to store positional information and for ECC. The latest Advanced Format hard drive technology changes a hard drive's sector size from 512 bytes to 4096 bytes. This allows the ECC data to be stored more efficiently. Advanced Format drives emulate a 512 byte sector size, to keep backwards compatibility intact, by mapping eight logical 512 byte sectors to a single physical sector. Unfortunately, this creates a problem for Windows XP users. The good news is, Western Digital has already solved the problem and HotHardware offers some insight into the technology and how it performs."
XP users do not need big hard drives to have problems.
...laura
When this issue came up a few weeks ago there was a problem with XP and with Linux. I see they tackled the XP issue pretty quick but what about Linux?
This place had something about it.
You want the sector size to be smaller than the average file size or you're going to waste a lot of space. If your average file size is large, and writes are sequential, you want the largest possible sector sizes.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
The filesystem's minimum allocation unit size doesn't necessarily need to have a strong relationship with the physical sector size. Some filesystems don't have the behavior of rounding up the consumed space for small files because they will store multiple small files inside a single allocation unit. (IIRC, Reiser is such an FS.)
Also, we are actually talking about 4 kilobyte sectors. TFS refers to it as 4096k, which would be a 4 megabyte sector. (Which is wildly wrong.) So, worst case for your example of a thousand 1k files is actually 4 megabytes, not 4 gigabytes as you suggest. And, really, if my 2 terabyte drive gets an extra 11% from the more efficient ECC with the 4k sectors, that gives me a free 220000 megabytes, which pretty adequately compensates for the 3 MB I theoretically lose in a worst case filesystem from your example thousand files.
If you read the article carefully, the new size is only 4K, not 4096K. The 4K size actually matches very well with most common files ystems.
Looks like they're not the only ones who miscalculated their block boundary.
https://www.eff.org/https-everywhere
No, that's totally wrong. The drive may well use 10 magnetic "cells" to store a byte (e.g. with 8b10b modulation or something similar), but that's an implementation detail. As far as everything else is concerned, bytes are 8 bits.
Depends on the drive. In recent electrical signalling (Gb ethernet, SATA/SAS, etc.) the 8b10b encoding scheme has been very popular; and is 10 bits to a byte. The extra bits are for recovering the clock signal. The HDD has to do the same, but the manufacturers don't have to adhere to any standards inside their case.
Now, if you're asking the question "how many bytes in a MB?" there is great debate. (The answer is, and has been from the first RAMAC*, 1,000,000. However, the binary bus people like to argue otherwise; and Microsoft Windows is one of the protagonists.)
* Okay, so technically the RAMAC was 5,000,000 words, where a word was 7 bits.
I can't grasp why all (these specific and most) benchmarks are so much obsessed with speed. Regarding HDs, I'd like to see results relevant to:
1. Number of Read/Write operations per task: Does the new format result in fewer head movements, therefore less wear on the hardware, thus increasing HD's life expectancy and MTBF?
2. Energy efficiency: Does the new format have lower power consumption, leading to lower operating temperature and better laptop/netbook battery autonomy?
3. Are there differences in sustained read/write performance? E.g. is the new format more suitable for video editing than the old one?
For me, the first issue is the more important than all, given that owning huge 2T disks is in fact like playing Russian roulette: without proper backup strategies, you risk all your data at once.
1 No one except LOSERS uses Windows XP.
Beck: I'm a loser, baby, 'cuz I'm usin' XP ...
2. What is Slashdot's commission on these shameful book plugs?
One free page from the book, randomly selected, until they've referred enough people to the publisher's site to receive the entire book. Unfortunately, it arrives as lose pages in no particular order. Cmdr Taco is never pleased with this.
Have a weekend, loozars.
Yours In Tashkent, K. Trout
Thanks, you too.
You can fix this on the filesystem level by using packed files. For the actual disk, tracking 512-byte sectors when most operating systems actually always read them in groups of 8 is just insane. (If you wish to access files by mapping them to memory, and you do, you must do so at the granularity of the virtual memory page size. Which, on all architectures worth talking about, is 4K.)
But as I understand it, byte == octet on all hardware that 1. allows the general public to develop applications and 2. is not discontinued.
Anandtech has a much better write up on this technology, complete with correct conversions from bits to bytes, knowledge of the difference between 4096 bytes and 4096 kilobytes, and no in-text ads.
You didn't dodge any bullet. Any file that has a size slightly over each 4096 border will take more space. For large amounts of larger files (such as an MP3 collection), you will, on average, have 2048 bytes of empty space in your drive's sectors. Lets say you have an archive which also uses some small files (e.g. playlists, small pictures) say that the overhead is about 3 KB per file, and the average file size is about 3MB. Since 3000000 / 3000 is about 1/1000 you could have a whopping 1 pro mille loss. That's for MP3's, for movies the percentage will be much lower still. Of course, if your FS uses a block size of 4096 already then you are already paying this 1 promille of overhead.
Personally I would not try and sue MS or WD over this issue...
No. Just no.
Never use the term 'KiB' for kiloBYTES ever again. Just don't do it. I don't CARE if it's "the new standard". Screw that, it's KB KiloBytes.
This "new" standard mandated by the IEC can eat me.
1024 bytes IS, and forever will be, 1 KiloByte (KB)
1000 bits IS, and forever will be, 1 KiloBit (Kb)
1999 and the IEC can DROP DEAD. I will never. EVER. Use the new """""""""""""standard"""""""""""".
That said, excellent job highlighting the dreadful editing, inaccuracies like that are so confusing to try and keep straight between what is written and what was MEANT. Thumps up for you!
This signature is lame.
Group code recording? http://en.wikipedia.org/wiki/Group_code_recording Back on my old Commodore Pet drive this was how they encoded data since too many zeros caused the head to lose it's place.
That article doesn't sound like fun at all. How are we supposed to mock it if they haven't made multiple errors, typos and other such blunders? We're smug, semi-knowledgeable 'first posters' with nothing better to do than critique articles that we were too lazy to read or too incompetent to write. I'm going to go wait on the homepage to refresh so I can jump into the next thread without a second thought.
IBM's GPFS is one, though it ain't free it does support Linux and Windows both mounting the same file system at the same time. They reckon the optimum block size for the file system is 1MB. I am not convinced of that myself, but always give my GPFS file systems 1MB block sizes.
Then there is XFS that for small files will put the data in with the metadata to save space. However unless you have millions of files forget about it. With modern drive sizes the loss of space is not important. If you have millions of files stop using the file system as a database.
From the wikipedia page you linked to: Btrfs
Thanks.
ReiserFS and UFS2 are stable
I was looking for file systems with killer features, not a killer maintainer ;-)
It was never KB and never will be. kB perhaps but not KB.
Unfortunately, it arrives as lose pages in no particular order. Cmdr Taco is never pleased with this.
Have a weekend, loozars.
For all intensive purposes, you're post should of exploded the heads of any grammar nazis as they read they're screen. Which begs the question of what more damage could possibly be done to effect there sensibilities? Honestly, I could care less.
Now that I think about it, I'm pretty sure everything I just said is completely wrong.
"it's 4 KiB or just 4096 bytes."
No. Just no.
Never use the term 'KiB' for kiloBYTES ever again.
"kiB" is for kibibytes, not kilobytes...
The introduction of those new units always kind of grated me, as it went against all the 20-odd years of experience I'd had with computers up to that point. But, I have to say, "kilobytes" and "megabytes" and "gigabytes" had always been ambiguously defined. Usually RAM would use the power-of-two definitions and disks would use the power-of-ten definitions... As someone who appreciates precise language, I think this effort to disambiguate the terminology is a good thing, even if it goes against what I learned about computers as a kid. I don't think making the opposite change (i.e. keeping "kilobyte" = 1024 bytes and making a new term for 1000 bytes) would have made any sense at all - the "kilo" in "kilobyte" goes against the normal definition of "kilo". I think it was always kind of sleazy that hard drive manufacturers could tell you they were giving you a megabyte of storage and it would be less than what the computer considers a "megabyte" - but the prefix has a definition that predates its use in computing, and from that perspective I think that usage, while problematic and misleading, was legitimate.
Bow-ties are cool.
Oh, how I wish I had mod points! :)
Whatever it is, it's notablog.
It's "for all intents and purposes" not "for all intensive purposes." When you say it you can get away with it wrong, but when you write it you just look dumb.
Indeed. Its a common mistake, but you're vigilance is dually noted. I'm just glad I didn't loose all credibility by making alot more mistakes.
Now that I think about it, I'm pretty sure everything I just said is completely wrong.
This is especially important for all you who manage a SAN. Learn it, love it, live it.
To learn why disk partition alignment can be important, please reference the following blog post: http://clariionblogs.blogspot.com/2008/02/disk-alignment.html
Instructions for Stripe Alignment/Partition Alignment within a Windows Operating Systems
Reference the following link for info on DiskPart, http://support.microsoft.com/kb/300415
1 - At a command prompt on a windows host type diskpart
2 -Type select disk X (X being the numbered disk within disk management that you want to align)
3 -Type create partition primary align=64
4 -You can then format the drive and assign a drive letter to it
NTFS uses a limited form of block suballocation: if the file is small enough, the file data can share a block with the metadata.
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
What this really means is that magnetomechanical media is dead.
When you're doing tricks like this to get a few extra bytes per block it means you have run out of physical media density technologies. It's kind of like when they moved the Earth, Moon and stars to get dial-up modems from 48.8Kbps to 56Kbps - redefining bps along the way. It's the End. It's an admission that we're out of magnetic media density improvements. There might be one more but after this but it's over and even now the density isn't even the important thing any more.
I warned about this here several years ago: the consolidation of server workloads leads to an I/O choke point. Next month AMD releases their 12-core Magny-Cours processor and Intel replies with a new processor technology - both of them increasing the memory channels and the amount of RAM that can be configured on a system to over a terabyte. It's on like Donkey Kong in terms of processing and RAM, but all of this tech will suffocate for lack of I/O.
The good news is that solid state technologies are here with sufficient capacity and doubling all of streaming bandwidth, IOPs and storage density at more than an acceptable rate. That they're greener is just bonus. And then there's the fact that the price per gigabyte - while still not competetive with consumer magneto-mechanical media - is coming down at an even better rate and already bests enterprise media (SAS and FC). There will be an accommodation period much like there was when we moved from analog modems to DSL and beyond - and this is a ripe field for the snakeoil salesmen. There will be wrenching pain as we realize that 8Gbps FC SAN doesn't even effectively serve a 5-pack of properly constructed third generation SSD-format drives, let alone an entire rack of them. The world will spin about us as multiplexed 4x SAS V2 (24Gbps) connections become the order of the day briefly, unless Intel makes a coup and figures a way to apply a heirarchical routing structure to LightPeak, which isn't even released yet and even so is obsolete. For sure electrical interconnects are right out - they don't have the bandwidth. We're going optical and I mean right now. 3.5" SAS drives will become the new tape. Tape has already been the new punchchard storage method for several years.
My guess: we'll find a new brand for "Enterprise storage" that uses RAID technologies to aggregate the bandwidth and improve the reliability of flash technologies in a way that doesn't rate-limit IOPs and in a way that provides reliable end-to-end performance and scales to terabits per second, until it becomes a static storage medium that actually reaches the performance of RAM. An interim solution may include huge RAM cache on SAS attached Flash drives backed by supercapacitors for guaranteed commited writes even if the power fails to preserve data integrity at the storage unit level. FC isn't the interconnect solution and SAS isn't it either - it'll likely be derived from external PCIe but be over optical media and probably multiple strands of it.
This is a big change - a revolutionary rather than an evolutionary change. A bigger change is coming. An extinction level event. When we've mastered the IOPs and the storage capacity of everything that everybody wants to store, then what? When every enterprise has consolidated their workloads down to three servers geographically separated for HA and DR, then what? What do we sell them then?
Friends the situation got dynamic. Good luck to you all.
Help stamp out iliturcy.
I thought the point was to have a small sector size. With large sectors, say 4096K, a 1K file will actually take up the full 4096K.
Most file system already use a cluster size of 4096 (clustering 8 sectors). The only file system I know of which used sector=cluster size was IBM's HPFS.
So NO, we don't use size. Still I am wary of this emulation stuff. First the 4096 byte sector is broken down to 8 512 byte "virtual" sectors and then those 8 virtual are clustered to one cluster. Would it not be better to use an intelligent file system which can handle 4096 bytes sectors natively? Any file system which can be formatted onto a DVD-RAM should do.
GPRS is a ridiculously fast os, probably the fastest in the world, when setup correctly. We used to use it for our cluster of 2000 cores.
And that kibi/gibi stuff only means that we have given up fighting and submitted to those marketing lies
Or maybe it just means that programmers have finally figured out the metric system?
In EVERY other discipline out there kilo means 1000. The reason it was defined as 1024 in IT wasn't out of some kind of brilliance, but rather shear laziness. Yes, I understand binary, and yes I understand why binary units are useful. So does the SI, which is why they invented the kibi prefix.
I could care less about marketers one way or another. I do care that a storage medium that stores one 1 MB per cm^2 does not store 10GB per m^2 if you use IT lingo. I don't see how that is in any way convenient for any engineer who actually works with anything in the physical world.