Slashdot Mirror
Exploring Advanced Format Hard Drive Technology
MojoKid writes "Hard drive capacities are sometimes broken down by the number of platters and the size of each. The first 1TB drives, for example, used five 200GB platters; current-generation 1TB drives use two 500GB platters. These values, however, only refer to the accessible storage capacity, not the total size of the platter itself. Invisible to the end-user, additional capacity is used to store positional information and for ECC. The latest Advanced Format hard drive technology changes a hard drive's sector size from 512 bytes to 4096 bytes. This allows the ECC data to be stored more efficiently. Advanced Format drives emulate a 512 byte sector size, to keep backwards compatibility intact, by mapping eight logical 512 byte sectors to a single physical sector. Unfortunately, this creates a problem for Windows XP users. The good news is, Western Digital has already solved the problem and HotHardware offers some insight into the technology and how it performs."
there is something more advanced than mkfs or format c:\ ?
I thought the point was to have a small sector size. With large sectors, say 4096K, a 1K file will actually take up the full 4096K. A 4097K file will take up 8194K. A thousand 1K files will end up taking up 4096000K. I understand that with larger HDD's that this becomes less of an issue, but unless you are dealing with a fewer number of large files, I don't see how this can be more efficient when the size of every file is rounded up to the next 4096K.
There is no "I disagree" mod for a reason. Flamebait, Troll, and Overrated are not substitutes.
ought to be enough for anyone
You mean 4096 bytes, not 4096k, right? Last time I checked, eight 512 byte sectors is considerably smaller than 4MB.
> The latest Advanced Format hard drive technology changes a hard drive's sector size from 512 bytes to 4096K.
I think not... try 4096 bytes.
Wait! Will this shorten or lengthen defrag times. Do file sizes under 4096K still exist?
XP users do not need big hard drives to have problems.
...laura
When this issue came up a few weeks ago there was a problem with XP and with Linux. I see they tackled the XP issue pretty quick but what about Linux?
This place had something about it.
It says 4096K, they mean 4096 bytes (4K). Error is in the original.
Stupid question- are bytes really 10 bits when talking hard drive capacity?
Is that some sort of checksum going on, or did the way computers store numbers change while I wasn't looking?
Guys, filesystem block/cluster size is not the same as hard drive sector size.
Jesus.
FTA:
"Each one of those ECC blocks is 40 bits wide; a 4096K block of data contains 320 bytes of ECC. Using Advanced Format's new 4096 sector size cuts the amount of ECC and Sync/DAM space significantly. According to WD, it needs just 100 bytes of ECC data per 4096K sector under the new scheme, a savings of 220 bytes."
For those not wanting to do the math...
220 extra bytes per 4096 bytes, in a 1 terabyte drive nets us 55GB more space.
From Google:
(220 / 4096) * 1 terabyte = 55 gigabytes
"where 1 byte = 10 bits."
as well as multiple prominent instances of "4096K"
Being off by 3 orders of magnitude is pretty hugely wrong, as is something as basic as how many bits are in a byte.
"The crows seemed to be calling his name, thought Caw."
I can't grasp why all (these specific and most) benchmarks are so much obsessed with speed. Regarding HDs, I'd like to see results relevant to:
1. Number of Read/Write operations per task: Does the new format result in fewer head movements, therefore less wear on the hardware, thus increasing HD's life expectancy and MTBF?
2. Energy efficiency: Does the new format have lower power consumption, leading to lower operating temperature and better laptop/netbook battery autonomy?
3. Are there differences in sustained read/write performance? E.g. is the new format more suitable for video editing than the old one?
For me, the first issue is the more important than all, given that owning huge 2T disks is in fact like playing Russian roulette: without proper backup strategies, you risk all your data at once.
1 No one except LOSERS uses Windows XP.
Beck: I'm a loser, baby, 'cuz I'm usin' XP ...
2. What is Slashdot's commission on these shameful book plugs?
One free page from the book, randomly selected, until they've referred enough people to the publisher's site to receive the entire book. Unfortunately, it arrives as lose pages in no particular order. Cmdr Taco is never pleased with this.
Have a weekend, loozars.
Yours In Tashkent, K. Trout
Thanks, you too.
Turns out NONE of my movies or MP3's are less than 4096 bytes so it looks like I dodged a bullet there.
But how big are script files and source code files and PNG icons?
Anandtech has a much better write up on this technology, complete with correct conversions from bits to bytes, knowledge of the difference between 4096 bytes and 4096 kilobytes, and no in-text ads.
That article doesn't sound like fun at all. How are we supposed to mock it if they haven't made multiple errors, typos and other such blunders? We're smug, semi-knowledgeable 'first posters' with nothing better to do than critique articles that we were too lazy to read or too incompetent to write. I'm going to go wait on the homepage to refresh so I can jump into the next thread without a second thought.
Name that filesystem. So far, I know it's not ext3, xfs, or jfs. All my torrent file chunks arrive in random order, and I guess Transmission doesn't pre-allocate the files, and stores 'em sparse instead. The result: pretty much worst-case fragmentation scenario. My filesystem is pessimized. I've had single files that take over 10 minutes just to delete (on xfs; jfs is waaay faster).
So I dedicate a partition to incoming torrents 'em and then copy 'em (or de-rar them) to my main storage when they're done, to defrag 'em. If I didn't have to do that, that would be sweet.
FTA:
"A WD10EARS and a WD10EADS have exactly the same unformatted capacity and Windows reports both drives offer 931GB of storage space."
TFA didn't say whether the advanced format drive is faster or not, more reliable or not, so as far as this reader can see is that users are paying more $$$ for less ECC bits and bigger sectors (which means less space for the same number of randomly-sized files).
Clever maybe for WD, but no obvious benefit.
From the wikipedia page you linked to: Btrfs
Thanks.
ReiserFS and UFS2 are stable
I was looking for file systems with killer features, not a killer maintainer ;-)
Unfortunately, it arrives as lose pages in no particular order. Cmdr Taco is never pleased with this.
Have a weekend, loozars.
For all intensive purposes, you're post should of exploded the heads of any grammar nazis as they read they're screen. Which begs the question of what more damage could possibly be done to effect there sensibilities? Honestly, I could care less.
Now that I think about it, I'm pretty sure everything I just said is completely wrong.
"it's 4 KiB or just 4096 bytes."
No. Just no.
Never use the term 'KiB' for kiloBYTES ever again.
"kiB" is for kibibytes, not kilobytes...
The introduction of those new units always kind of grated me, as it went against all the 20-odd years of experience I'd had with computers up to that point. But, I have to say, "kilobytes" and "megabytes" and "gigabytes" had always been ambiguously defined. Usually RAM would use the power-of-two definitions and disks would use the power-of-ten definitions... As someone who appreciates precise language, I think this effort to disambiguate the terminology is a good thing, even if it goes against what I learned about computers as a kid. I don't think making the opposite change (i.e. keeping "kilobyte" = 1024 bytes and making a new term for 1000 bytes) would have made any sense at all - the "kilo" in "kilobyte" goes against the normal definition of "kilo". I think it was always kind of sleazy that hard drive manufacturers could tell you they were giving you a megabyte of storage and it would be less than what the computer considers a "megabyte" - but the prefix has a definition that predates its use in computing, and from that perspective I think that usage, while problematic and misleading, was legitimate.
Bow-ties are cool.
Oh, how I wish I had mod points! :)
Whatever it is, it's notablog.
Unfortunately, this creates a problem for Windows XP users. The good news is, Western Digital has already solved the problem
Is there a particular reason that we should care that a new technology isn't backwards compatible with an obsolete technology? Especially in light that it actually is compatible?
I'm not even sure whether calendars based on the year of the Lord are more widespread than the 8-bit byte. Certainly, 8-bit bytes are in use in the Middle East, parts of which use the Islamic calendar and the Jewish calendar.
It's "for all intents and purposes" not "for all intensive purposes." When you say it you can get away with it wrong, but when you write it you just look dumb.
It's "for all intents and purposes" not "for all intensive purposes." When you say it you can get away with it wrong, but when you write it you just look dumb.
Indeed. Its a common mistake, but you're vigilance is dually noted. I'm just glad I didn't loose all credibility by making alot more mistakes.
Now that I think about it, I'm pretty sure everything I just said is completely wrong.
This is especially important for all you who manage a SAN. Learn it, love it, live it.
To learn why disk partition alignment can be important, please reference the following blog post: http://clariionblogs.blogspot.com/2008/02/disk-alignment.html
Instructions for Stripe Alignment/Partition Alignment within a Windows Operating Systems
Reference the following link for info on DiskPart, http://support.microsoft.com/kb/300415
1 - At a command prompt on a windows host type diskpart
2 -Type select disk X (X being the numbered disk within disk management that you want to align)
3 -Type create partition primary align=64
4 -You can then format the drive and assign a drive letter to it
Be careful, tzanger, I think Mr. Grumbine may be bating you!
The guy forgets an 'o' in 'loose' and you pull out your 'grammar ruler'? Schmuck. Maybe he had intensive purposes for that other 'o'?
For all intensive purposes, you're post should of exploded the heads of any grammar nazis as they read they're screen.
And you're certainly the one to point out grammatical folly. I bet after reading your post they're wishing that their screens would have exploded after reading the other one.
Which begs the question of what more damage could possibly be done to effect there sensibilities?
I'm pretty sure your post affected all of their senses even if it didn't have the intended effect on the rest of us. Well done.
I'm just glad I didn't loose all credibility by making alot more mistakes.
That could never happen to you even though it does seem to happen to others a lot.
The WD green AF acts like A when not aligned, and B when aligned...
and the WD black is better than both for just a little more.
Probably better to skip Subway for lunch, make your own sandwiches for a week and get the WD black (or go for 2 weeks and get the RE3).
Partitioning the right way deals with it. You can use fdisk in Linux to do the partitioning for both Linux and Windows.
First, find out exactly how large the drive is in units of 512 byte sectors. Divide that number by 8192 and round any fractions up. Remember that as the number of cylinders. In fdisk, use the "x" command to enter expert commands. Do "s" to enter the number of sectors per track as 32. Do "h" to enter the number of heads (tracks) per cylinder as 256 (not 255). Do "c" to enter the number of cylinders previously calculated. Do "r" to return from expert mode. Do "u" to change units to exact sectors.
Now allocate space on boundaries of a multiple of 8 sectors (with the "last" sector for any partition being one less than such a multiple). To squeeze a tiny bit more performance out of I/O scheduling and caching, allocate space on larger boundaries. I now allocate on 2048 sector boundaries, which is exactly 1048576 bytes, with the first partition beginning at sector 2048. That leaves plenty of room for big boot loaders.
I have not tested starting at sector 2048 with Windows, but I did test starting at sector 64 with Windows several years ago and that worked fine. I can't see why it would not work at 2048 if it worked at 64. But I do recommend letting Windows do the filesystem formatting for any Windows partitions.
now we need to go OSS in diesel cars
Those of us who work with RAID arrays have cared about partition alignment for a long time. If a write spans two RAID-5 stripes, the RAID controller has to work twice as hard to correctly update the parity information. Aligning partitions and filesystem structures on stripe boundaries is essential to obtaining good performance on certain types of RAID arrays.
What this really means is that magnetomechanical media is dead.
When you're doing tricks like this to get a few extra bytes per block it means you have run out of physical media density technologies. It's kind of like when they moved the Earth, Moon and stars to get dial-up modems from 48.8Kbps to 56Kbps - redefining bps along the way. It's the End. It's an admission that we're out of magnetic media density improvements. There might be one more but after this but it's over and even now the density isn't even the important thing any more.
I warned about this here several years ago: the consolidation of server workloads leads to an I/O choke point. Next month AMD releases their 12-core Magny-Cours processor and Intel replies with a new processor technology - both of them increasing the memory channels and the amount of RAM that can be configured on a system to over a terabyte. It's on like Donkey Kong in terms of processing and RAM, but all of this tech will suffocate for lack of I/O.
The good news is that solid state technologies are here with sufficient capacity and doubling all of streaming bandwidth, IOPs and storage density at more than an acceptable rate. That they're greener is just bonus. And then there's the fact that the price per gigabyte - while still not competetive with consumer magneto-mechanical media - is coming down at an even better rate and already bests enterprise media (SAS and FC). There will be an accommodation period much like there was when we moved from analog modems to DSL and beyond - and this is a ripe field for the snakeoil salesmen. There will be wrenching pain as we realize that 8Gbps FC SAN doesn't even effectively serve a 5-pack of properly constructed third generation SSD-format drives, let alone an entire rack of them. The world will spin about us as multiplexed 4x SAS V2 (24Gbps) connections become the order of the day briefly, unless Intel makes a coup and figures a way to apply a heirarchical routing structure to LightPeak, which isn't even released yet and even so is obsolete. For sure electrical interconnects are right out - they don't have the bandwidth. We're going optical and I mean right now. 3.5" SAS drives will become the new tape. Tape has already been the new punchchard storage method for several years.
My guess: we'll find a new brand for "Enterprise storage" that uses RAID technologies to aggregate the bandwidth and improve the reliability of flash technologies in a way that doesn't rate-limit IOPs and in a way that provides reliable end-to-end performance and scales to terabits per second, until it becomes a static storage medium that actually reaches the performance of RAM. An interim solution may include huge RAM cache on SAS attached Flash drives backed by supercapacitors for guaranteed commited writes even if the power fails to preserve data integrity at the storage unit level. FC isn't the interconnect solution and SAS isn't it either - it'll likely be derived from external PCIe but be over optical media and probably multiple strands of it.
This is a big change - a revolutionary rather than an evolutionary change. A bigger change is coming. An extinction level event. When we've mastered the IOPs and the storage capacity of everything that everybody wants to store, then what? When every enterprise has consolidated their workloads down to three servers geographically separated for HA and DR, then what? What do we sell them then?
Friends the situation got dynamic. Good luck to you all.
Help stamp out iliturcy.
And Melinda would say that ought to is the same thing as nought to: what is, is - and what should be isn't.
I'm just glad I didn't loose all credibility by making alot more mistakes.
You should of!
After the "linux doesn't handle it story" a couple of weeks ago http://hardware.slashdot.org/story/10/02/14/1541244/Linux-Not-Quite-Ready-For-New-4K-Sector-Drives I wondered if the mis-alignment was what was causing my poor postgres performance on the WD Caviar Green. After quite a lot of effot moving things around I didn't actually see any noticeable difference. Now I'm left wondering whether the mis-alignment effect is dwarfed by the effort of reading 3.5K of a 4K block for every random 0.5K block write. The fact that the disk is lying to the driver is the big deal here. Does anybody know how to force the linux sd driver to use 4k blocks regardless of the what the disk tells it about blocksize?
Look, I know the parent post is going to garner a boatload of hate from the FC SAN people who will protest for various reasons that their unicorns and rainbows magnify the effectiveness of the underlying storage until it's cheap and performant. I'm sorry, but you're all full of it (to be shkind). You need to find a new job.
When you figure the cost of FC storage, the network, the backup, the service contracts and whatnot, it's $30K-120K/TB. You guys got some cool stuff - I'll give you that. But it ain't worth 300x-1200x the price of consumer tech, especially when it lacks the density required for modern apps, and is limited in bandwidth to legacy tech, and can't serve the IOPs that VMHosts need. As it is your validation teams are slacking in approving modern density drives. For what you're asking we could do RAMDisk. Notwithstanding, 8Gbps is barely sufficient to feed one VMHost, let alone a blade chassis full of them.
And tape guys: seriously, it's time to give it up. Get a new job - please.
Help stamp out iliturcy.
It says (4K), they mean (4k). Error is in the correction.
Son, your InTerNationalUnits meant nothing back when they were inventing this stuff. They said KB and meant 8096 bits. That's 32x32x8, for those who are keeping track, or 2 x 8^4. It bites. That you expect us to keep track of your precious bits by the each only magnifies your fail. It's just not done that way any more. You might as well craft a rug with tweezers.
Help stamp out iliturcy.
From TFA: "Western Digital believes the technology will prove useful in the future and it's true that after thirty years, the 512 byte sector standard was creaking with age."
What does "creaking with age" really mean? I mean, the current format performs the same. The basic design is still the same, just with different magic numbers. I usually read "creaking with age" to mean that there's some kind of capacity or speed limit that we hit, but that's not the case. Is this more of a case of "why not" change it instead of "why"?
Were you there when MFM bowed to RLL? When the disaster that was RLE crashed? When that bifurcated into IDE and SCSI?
If you had been, you could not formulate this question. What do you know?
The underlying hardware of a spinning disc is that it is a disc that is spinning, and the data is rarely under the head when it needs to be read - and SSDs are solid state devices, any block of which can be read as quick as any other because it need not wait for a physical object to spin into position - random reads and writes are the same as sequential reads and writes - more than that: third generation drives leverage abstraction to stripe and spare their internal storage so as to deliver reliable storage within the context of traditional drives despite the fact that they are not even remotely similar.
Help stamp out iliturcy.
I thought the point was to have a small sector size. With large sectors, say 4096K, a 1K file will actually take up the full 4096K.
Most file system already use a cluster size of 4096 (clustering 8 sectors). The only file system I know of which used sector=cluster size was IBM's HPFS.
So NO, we don't use size. Still I am wary of this emulation stuff. First the 4096 byte sector is broken down to 8 512 byte "virtual" sectors and then those 8 virtual are clustered to one cluster. Would it not be better to use an intelligent file system which can handle 4096 bytes sectors natively? Any file system which can be formatted onto a DVD-RAM should do.
Usually RAM would use the power-of-two definitions and disks would use the power-of-ten definitions...
That would be disk larger then aprox. 1GB. Before disk hit the GB mark they too where measured power of 2 - which is the right way to do it - sector size if a power of 2 after all.
It is marketing which uses power-of-ten. And as an engineer this pisses me of big time. And that kibi/gibi stuff only means that we have given up fighting and submitted to those marketing lies.
Martin
Virtual XP machines perhaps ;-)
Actually it makes me wonder if the virtual 512 sector stuff can be switched off. XFS for example handles lagers sectors sizes gracefully.
Actually XFS supports true 4096 sector size as well. For example you can format XFS onto a DVD-RAM (sector size= 2048) without trouble. So best for Linux would be if you can tell the drive not to lie about the sector size.
You could have 11% more capacity - but for some unknown reason WD did not exploit that.
If the drive would not lie about the sector size then you would have a little speed gain as well - but for some unknown reason WD went for compatibly instead.
So yes: there is some potential for larger sector size - but is was not exploited.
Ha ha ha. I literally laughed my head off.
I do hope that while you are still learning the language's grammar, you also are developing a better, keener sense of humor, as the GP made it clear by the abundance of grammatical errors put together into a somewhat coherent message, that he was, without any doubt, making a mockery, a grotesque sneer if you will of the GGP post. You missed it on many more levels than one, only picking up on a single grammatical error and completely letting go of the actual context of the message. I sincerely wish you to grow in all your future endeavors, however I do suggest not to jump to foregone conclusions in such a haste as to make a fool of oneself.
You can't handle the truth.
I do care that a storage medium that stores one 1 MB per cm^2 does not store 10GB per m^2 if you use IT lingo.
That won't be true anyway - unless you would increase the sector size by 10'000 as well. In the early days we had single density and double density disk. Single density used 128 byte per sector and double density 256 byte per sector. That would equivalent to 90 kb and 128 kb (for 40 tracks). If you formatted a double density disk with 128 byte per sectors (as Atari did for compatibility) you only get 127 kb.
That's because of all the overhead as described in the original article. Or read it up here: http://en.wikipedia.org/wiki/Floppy_disk_format#Logical_formatting
I don't see how that is in any way convenient for any engineer who actually works with anything in the physical world.
Well, the address bus of a CPU is pretty physical. It's real copper lines connecting the CPU with it's memory. And the address bus has 2 (on and off) and not 10 states. So using the power of 2 is very convenient for engineers working in IT.
It is only logical to use half a page (128), a page (256) or two pages (512) for sector size on floppy disk and hard drives in the early days. Better performance, better memory usage. The advantages are clear for anybody who has any knowledge in OS design.
Modern computers have shifted page size from 2^8 to 2^12 - and now guess what the cluster size of modern file system is. It is just more efficient to store data in page size thunks.
And power of 10 is pretty useless because neither can 128, 256, 512 and 4096 be divided by 10 now is it easy to find multiples of those which are cleanly dividable by 10. There is no 1GB hard-drive - it is impossible to build (unless you accept the performance penalty of a sector size which does not align with the computers page size). The closes to get is either 999'976 or 1'000'488 byte. And that is not taking into account the platter and track counts.
Tom's Hardware has tackled this too. Just mentioning it for the sake of completeness.
Indeed. Its a common mistake, but you're vigilance is dually noted. I'm just glad I didn't loose all credibility by making alot more mistakes.
Dually noted? You noted it twice? Maybe you should have read it twice and duly noted that you can't write or spell, let alone judge someone else's grammar.
For someone who lives in a glass house you sure do have a lot of broken windows.
Where's a mod to mark this guy as an asshat?
Where's a mod to mark this guy as an asshat?
Did you actually, for real, miss the same joke twice in a row?
I'm impressed.
What the original post seems to be ignoring is the amount of 'data' stored with the block of data seen by the customer. It has been many years since I last looked into this so there may well be changes but:
A block of data consists of:
header/leader: this is alignment, block id, and other control information. at least 128 bytes
block of data: the data actually seen by the user
trailer: I don't remember the length, but probably close to the size of the header, but at least 64 bytes, probably 128. includes ecc and a second block id and other control information.
so assuming you have a track with a raw capacity of 1MB, just as an example:
512 byte blocks: 1,000,000/(128+512+128) = 1,000,000/768 = 1,302 blocks = 666,624 usable bytes out of 1,000,000 or 67%
4096 byte blocks: 1,000,000/(128+4096+128) = 1,000,000/4352 = 229 blocks = 937,984 usable bytes out of 1,000,000 or 94%
so the larger blocks make mush more efficient usage of the raw space. Even if the trailer becomes 512 bytes, the new utilization is 84%