Linux Not Quite Ready For New 4K-Sector Drives

Theovon writes "We've seen a few stories recently about the new Western Digital Green drives. According to WD, their new 4096-byte sector drives are problematic for Windows XP users but not Linux or most other OSes. Linux users should not be complacent about this, because not all the Linux tools like fdisk have caught up. The result is a reduction in write throughput by a factor of 3.3 across the board (a 230% overhead) when 4096-byte clusters are misaligned to 4096-byte physical sectors by one or more 512-byte logical sectors. The author does some benchmarks to demonstrate this. Also, from the comments on the article, it appears that even parted is not ready, since by default it aligns to 'cylinder' boundaries, which are not physical cylinder boundaries and are multiples of 63."

Comments

Set 32 sectors per track

[tchuladdiass]

The simple solution is to set you Sectors per Track to 32. This would make sure that everything is properly aligned (except the first partition, usually /boot, which is mis-aligned by one cylinder).

Re:Set 32 sectors per track

[Brian Gordon]

Sectors, blocks, clusters, cylinders... I hope that as we move to solid state drives, devs have the sense to exorcise these anachronisms from the kernel. We haven't been able to get rid of terminals in the 20 years since they've even existed.. this document is heart wrenching. Try reading it; it'll make you cry to see how deeply the now-irrelevant concept of a terminal runs in Linux.

Re:Set 32 sectors per track

[walshy007]

the now-irrelevant concept of a terminal

Speak for yourself sir, I for one like my rs-232 terminals to be handy for when ethernet is down and you can't ssh (and can't be assed hooking up keyboard and monitor). Seriously, anyone adept at the command line uses it far more than the gui to get things done, terminals will never disappear.

Re:Set 32 sectors per track

[rubycodez]

terminals are a very necessary and relevant part of Linux. That's how most server administration is done. That's how sending commands to many network appliances is done. That's how setting up high end computers is done (e.g. set up a midrange Integrity or Superdome and you'll start with terminal on the serial port, whether cu in linux or hyperterminal in windowws or a real terminal). Also how certain tasks are performed in GUI environments. It doesn't matter that the terminal is now mostly virtual, the cursor control and font attribute features make convenient applications possible. Even on the weekend here I am chatting via IRC to some tech friends with irssi in terminal under screen, and reading server status emails with mutt. the terminal, it's 21st century tool.

Re:Set 32 sectors per track

[Anonymous Coward]

terminals have nothing to do with the command line!
i think the op is complaining about the fact that things like
baud, stopbits and whatnot are deeply embedded in the
linux kernel. these concepts are not necessary to
have a command line. c.f. plan 9.

Re:Set 32 sectors per track

[amorsen]

Anyway - sooner or later we will have flash drives instead, and then this isn't a problem.

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Re:Set 32 sectors per track

[kimvette]

Oh, in addition, now that Windows Server (Core) has a real GUI-less mode and Powershell and UNIX environment shells on Windows finally have usable interfaces, shell prompts are becoming even more relevant even in large Window shops. So, even Microsoft has acknowledged that the UNIX-y way of doing things is key for automation and uptime in an enterprise environment. Now, most PCs won't boot with output to the serial port, but some enterprise server boards do have such options.

A GUI is great for basic tasks, but for repetitive tasks a command shell and scripting environment are key for efficiency, and reliable automation. VBS/Windows Scripting Host was an "acceptable" workaround for a while but in the past many Windows administrative tools required the box to not be headless, the workstation unlocked and the windows open for the GUI to be accessible for scripting - and even then it was iffy because not all GUI elements are accessible (especially third-party tools with custom controls).

Re:Set 32 sectors per track

[Anonymous Coward]

I don't think he was dissing command line interfaces.

I think his complaint was that even newfangled RS-232 terminals had to jump through hoops to remain compatible with computers that were hooked up to typewriters and line printers. The protocols and underlying software have idiosyncrasies built into them that just don't make sense any more. Instead of throwing away the cruft to make something better, everybody's hacking onto the same old outdated shit. It's limiting progress, in a way.

Open Source to the rescue

[bogaboga]

I am no kernel hacker but I can almost guarantee that some kernel hacker will provide a solution to this "short coming" fairly soon.

That's the beauty of Open Source.

I am aware though that "fairly soon" means many things to many people; which means that there could be a substantial delay before we get a working solution to this issue.

I am optimistic nevertheless.

Request to Western Digital: Provide all the information needed to develop a solution.

Re:Open Source to the rescue

[rsmith-mac]

On the contrary, this has (almost) nothing to do with Windows - it has everything to do with old OSes. The IDEMA didn't approve the 4K sector standard until 2006; it was only in the late 90's that the first meaningful research was begun by IBM on whether 512B sectors would be an issue.

As it turns out, yes, 512B sectors would be an issue, and drive manufacturers would be best served by moving to larger sectors (with some arguing over whether to go to 1K or 4K). So the IDEMA hashed this out over the first half of the decade, and finally in 2006 approved the 4K specification.

The point of all of this is that software written at the turn of the century was all done well before changing drive sector sizes was a serious discussion. WinXP was released in 2001, Mac OS X 10.0 was in 2001, and of course Linux 2.4 was also in 2001. None of those OSes know what to do with anything other than a 512B sector - the only reason Windows factors in to this equation is that WinXP just happens to be with us (no doubt trying to eat our brains) while the other two are dead. Anything circa 2005 or later such as WinVista, Linux 2.6, and Mac OS X 10.5 know full well what to do with a 4K drive.

But even that is beside the point. You don't just make major jumps like this, you have to do it in a transition so that you don't break old hardware and old software alike. Even if XP/Lin2.4/MacOSX knew what to do with 4K sectors, at some point you'd run in to hardware, 3rd party devices, etc that would not. A transition is necessary to let old hardware and software get flushed out of the ecosystem, and as such we're still years out from consumer drives offering native 4K access.

In short: drives are pretending to have 512-byte sectors because there's a lot of old stuff, including Windows XP that can't deal with 4K sectors.

Re:Open Source to the rescue

[DAldredge]

Drives have been lying to the OS for more than 20 years in regards to quite a few different things so why pick now to yell and get upset?

Re:Open Source to the rescue

[marcansoft]

I was referring to Windows XP - I should've mentioned it explicitly. Nonetheless,

There's a lot of old stuff, including Windows XP that can't deal with 4K sectors.

There's exactly one old thing used in any significant quantity and liable to have to work with these drives, and it's Windows XP. Everything else either isn't in any significant use any more, or will never be seeing one of these drives.

One potential concern would be USB enclosures which have to work with older OSes / devices. To that, I would say it should be the enclosure's job to do the 512-byte sector emulator, not the drive's.

Windows XP has been updated over the years (via service packs and the like) to handle hardware that didn't exist at the time of its release, including things like SATA. They could do the same for 4k sector disks, but they aren't going to do it because they want people to move to Win7. Therefore, Microsoft is still to blame for neither 1) providing a solution for XP, nor 2) providing enough compelling reasons to migrate to Win7. Heck, Vista was a trainwreck and it doesn't really count, so (proper) Windows is actually 3 years late to the 4k party, as Windows 7 was only released in 2009. Effectively, they've spent those three years scaling down support for XP while providing no viable alternative, and now the rest of the world has to deal with a significant amount of people still using a 9-year-old OS.

In short: the fact that people are sticking to 9-year-old XP and making hardware companies break or slow down improvements means Microsoft did something terribly wrong.

Re:Open Source to the rescue

[guruevi]

Ya know in the olden days, when I was young (love saying that) we didn't have hardware that configured itself so it would work on all platforms. We had to put in settings with jumpers and do low-level disk formats through the BIOS or a boot-floppy and WE LIKED IT (seriously).

These days all ya new-fangled hardware doesn't have to worry about being a master or a slave, getting 5V or 3.3V to the PCI bus or RAM modules, CPU multipliers on the motherboard.

I would simply do the same - get a jumper on the back of the drive that says 512 or 4k - we left it on 512 for ya because we assume you numbnuts still use Windows (XP) but if you want performance and use anything but DOS or WinDOS feel free to switch it. You can then reformat the drive.

Re:Interesting

[ScytheBlade1]

TFA disagrees with you.

$ time cp winxp.img /mnt/sdc # ALIGNED
real 5m9.360s
user 0m0.090s
sys 0m20.420s

$ time cp winxp.img /mnt/sdd # UNALIGNED
real 13m26.943s
user 0m0.110s
sys 0m19.350s

$ time cp -r Computer Architecture/ /mnt/sdc # ALIGNED
real 42m9.602s
user 0m0.680s
sys 1m59.070s

$ time cp -r Computer Architecture/ /mnt/sdd # UNALIGNED
real 138m54.610s
user 0m0.660s
sys 2m15.630s

The first two being a single file, the latter two being multiple files in a larger directory structure.

I would heartily disagree with you on the matter.

Oh slashdot..

[JeffSh]

Dear Slashdot,

I've been around for a while. Enough to understand, nay, love the fact that you are linux supporters and all that. But I remain an ardent supporter of truth and speaking in ways which are concise and leads the reader in the direction of truth. Nothing in this news story is inaccurate, but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.

Windows XP may not work with the new WD Green drives, but Vista and on have been perfectly comfortable with 4096 byte sectors. A lay reader may read this story and not "Read between the lines" as I have learned to do here. Their take away may be that Microsoft operating systems are broken in some way (which they are in a lot of ways), but not this one!

Re:Oh slashdot..

[Vellmont]

  but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.
.
.
Their take away may be that Microsoft operating systems are broken in some way (which they are in a lot of ways), but not this one!

It only takes about 3 brain cells to realize that Windows XP != All Microsoft Operating Systems. Even the average person has more than 3 brain cells.

For those people with less than 3 brain cells, Slashdot still has you covered since the article clearly says:

According to WD, their new 4096-byte sector drives are problematic for Windows XP users but not Linux or most other OSes.

(emphasis mine)
It only takes 2 brain cells to understand that "most other OSes" likely includes Vista and Windows 7.

For those unlucky few with only 1 brain cell, you're correct. Slashdot has certainly failed the 1 brain celled individual.

Re:So don't do that...

[marcansoft]

fdisk doesn't need to be fixed, it needs to be deprecated. DOS partition tables are a ridiculously bad artifact of the past. We won't be using them for much longer anyway; they're limited to 2TB for 512-byte-sector drives (or 4K drives with 512-byte emulation).

slashdot is not journalism

[SuperBanana]

should be an embarrassment of journalistic integrity.

Slashvertisements, basic English grammar and spelling problems, completely wrong summaries and titles...

...and you a)think that Slashdot is "journalism" and b)it's had integrity to lose in the first place?

I like Slashdot, but gimme a break...it's a user-driven blog which directs readers to existing stories (now often lagging behind the major news wires) with good categorization and semi-sophisticated commenting system, utilized by a larger commenter population. Not much more, and definitely not journalism.

Re:if vista/win7 really do support this correctly.

[walshy007]

The real problem is that it is lying about it's sector size, it's reporting 512 bytes when it's using 4k, if it told linux it was using 4k everything would be fine and dandy.

Why does it lie about it's sector size when it doesn't need to? because if it didn't the drives would not work on windows XP at all. Which would not bode well for sales.

Once drives with 4k sectors arrive its up the individual maintainers of each affected tool (fdisk, et. al.) to update their code.

Kernel handles sector sizes, and could handle 4k sectors ages ago, but when the hardware reports something it tends to trust it, which is now apparent it shouldn't. (512 byte sectors being implemented as an emulation layer of sorts on these drives.. and enabled by default)

Re:Interesting

[hedwards]

That's true, but it's also true that having hardware lie to the OS isn't a great situation to be in. At the very least there should be some way of forcing it to be honest for the benefit of OSes that can handle the reality. A lot of the gunk and instability in computing comes from hardware that does things that are more appropriately done by software and vice versa.

Forcing users to optimize isn't inherently wrong, it's just that they shouldn't need to do it for things which are somewhat standard as a work around for weird hardware designs. And yes, I realize that the 4096byte sectors aren't being implemented arbitrarily.

Re:Partitions are obsolete

[hedwards]

Which is nice if you're wanting to ensure that you've got the lowest possible reliability and safety for your data. While you're at it, make sure you're using a striped non-redundant array of disks as well, best use at least 4 in the array, otherwise you might get some of your data back.

You've got it exactly backwards, people shouldn't be partitioning disks into one huge partition. They should be able to split things up a bit to keep rapidly changing directories from mostly static ones and to manage the risk of filesystem corruption destroying important files.

Re:if vista/win7 really do support this correctly.

[ArghBlarg]

I see it rather as an indictment against closed-source OSes, if XP turns out to be incompatible with these new drives and MS never releases a patch to add support. People will need to upgrade for no good reason to one of MS's new operating systems. People should not have to deal with a complete upheaval of their tested and true systems due to a small hardware change such as this.

I can imagine MS is quietly chuckling with glee to itself, if this issue becomes a deal-breaker for machines still running XP.

Poorly researched article.

[Vellmont]

The article represents one data point, for one particular way to install a drive, on one (un-named) version of Gentoo, on one particular model of a WD drive that had a bugzilla entry entered by the author all of 2 days ago. So this is supposed to be an indictment of all of Linux?

The author even mentions that Ubuntu has an option on parted that accomplishes the task properly. I'd be much more interested in an article that talks about how the default installer handles this task rather than concentrating on one particular expert tool that does so. It's still good to know that fdisk on his un-named Gentoo distribution does the wrong thing.. but this hardly means we should fire up the klaxon and declare "Linux not fully prepared for 4096 sector hard drives!". It's certainly interesting, but I'll withhold judgment until we actually know more about the implications of this across the entire spectrum of Linux distributions and the various 4096 sector HDs.

Re:Poorly researched article.

[Vellmont]

I can't over-emphasize the importance of titles in communication, especially with complex technical subject where there's a lot of evidence presented to support a conclusion. Your title colors the rest of the article and creates expectations about what you're trying to say. When people read articles (especially on the web) they scan through them trying to find the important parts. That's been demonstrated through eye-tracking studies multiple times.

Your title was very broad, but the evidence to support it was very narrow (as you admit yourself). Since your article only referenced Gentoo, and installing a drive after installation using fdisk, why not: "fdisk not ready for 4096 byte sector drives?" If you wanted to cover all of "linux" (whatever that is), why not research other distributions and see how they handle the job, installing the OS flat out on several distro's and see how they perform. If you didn't want to do the research, you could have written in an inquisitive style, i.e. "Is linux ready for 4096 byte sector drives?" and then presented your evidence and talked about how fdisk didn't do this, but parted worked a bit better, and how the drives reported 512 sectors but weren't, etc, but you've only tried Gentoo, and maybe others should try other distributions with other tools.

I think the experimental aspects you did were great, and I'm glad to know that some of the tools out there don't fully work with this particular 4096 byte sector drive. I just don't think you've done enough research to say much more than the problem affecs Gentoo, on these specific WD drives.

Re:if vista/win7 really do support this correctly.

[thisissilly]

It seems these drives need a new "don't lie to me, I can handle it" command, so OSes that don't have a problem with 4k size sectors can get the real info.

Re:Partitions are obsolete

[HiThere]

And then when you install a different distribution, you blow away your home directory. Sorry, bad idea. /home should be in a separate partition from the rest of the stuff..

Also, since I usually have several distributions installed at the same time, I have several partitions...but that's a less common problem.

A better solution would be to have a boot partition snuggled up against the MBR that automatically adapts so that the boot + MBR is an appropriate size, say 32 MB. (My current boot directory is 14MB, so that shouldn't be a problem. These aren't, after all, small drives, so it doesn't hurt to allocate a bit of extra space. Maybe even make that 64MB.)

Perhaps one could rearrange the system tables a bit so that the MBR was counted as a part of the /boot partition, and so was the partition table. They'd need to be an a position guaranteed by the OS, but that's not a real problem.

Note that what I'm proposing is a major redesign, so there's about zero chance of it being adopted. But it's a better choice than scrapping partitions, and probably has a better chance of being adopted.

Re:Partitions are obsolete

[Anonymous Coward]

Splitting a disk into multiple pseudo-disks makes sense in many situations, but the clunky legacy partition tables are only good for inter-OS compatibility. Otherwise LVM beats partitions in every respect. Now if only we could get a LVM solution that works in multiple operating systems...