Linux Not Quite Ready For New 4K-Sector Drives

Theovon writes "We've seen a few stories recently about the new Western Digital Green drives. According to WD, their new 4096-byte sector drives are problematic for Windows XP users but not Linux or most other OSes. Linux users should not be complacent about this, because not all the Linux tools like fdisk have caught up. The result is a reduction in write throughput by a factor of 3.3 across the board (a 230% overhead) when 4096-byte clusters are misaligned to 4096-byte physical sectors by one or more 512-byte logical sectors. The author does some benchmarks to demonstrate this. Also, from the comments on the article, it appears that even parted is not ready, since by default it aligns to 'cylinder' boundaries, which are not physical cylinder boundaries and are multiples of 63."

Comments

first misaligned post

[Anonymous Coward]

damnit, obviously since this is not technically the 'first post', my web browser must be misaligned by a post

Parted / GPT

[Anonymous Coward]

I heard using parted and GPT labels instead of MSDOS will optimize it on 4096 byte sectors automatically. Any truth to it?

Re:Parted / GPT

[houstonbofh]

Even if that works for new partition, what about existing ones? These drives have been out a while, and people have been installing software to them... The best solution will be a program that moves and/or resizes existing partitions to align them properly. The question is will it be an updated gparted, or something new? What ever it is, I have no doubt that it will be out for Linux far before it is out for XP. :)

oh great.. i try to make a joke and ...

[decora]

the first time i have ever actually gotten 'first post'... it is when i try to make a joke about not having gotten first post. ya see my first post was supposed to come up like second or third.. it would have been HILARIOUS . .. but oh no in soviet russia, the fates mock you!!!!

Set 32 sectors per track

[tchuladdiass]

The simple solution is to set you Sectors per Track to 32. This would make sure that everything is properly aligned (except the first partition, usually /boot, which is mis-aligned by one cylinder).

Re:Set 32 sectors per track

[Brian Gordon]

Sectors, blocks, clusters, cylinders... I hope that as we move to solid state drives, devs have the sense to exorcise these anachronisms from the kernel. We haven't been able to get rid of terminals in the 20 years since they've even existed.. this document is heart wrenching. Try reading it; it'll make you cry to see how deeply the now-irrelevant concept of a terminal runs in Linux.

Re:Set 32 sectors per track

[walshy007]

the now-irrelevant concept of a terminal

Speak for yourself sir, I for one like my rs-232 terminals to be handy for when ethernet is down and you can't ssh (and can't be assed hooking up keyboard and monitor). Seriously, anyone adept at the command line uses it far more than the gui to get things done, terminals will never disappear.

Re:Set 32 sectors per track

[jedidiah]

Terminals are only irrelevant if you have a strictly Windows and DOS notion of computing.

The document you cited touches on this a little bit.

Re:Set 32 sectors per track

[rubycodez]

terminals are a very necessary and relevant part of Linux. That's how most server administration is done. That's how sending commands to many network appliances is done. That's how setting up high end computers is done (e.g. set up a midrange Integrity or Superdome and you'll start with terminal on the serial port, whether cu in linux or hyperterminal in windowws or a real terminal). Also how certain tasks are performed in GUI environments. It doesn't matter that the terminal is now mostly virtual, the cursor control and font attribute features make convenient applications possible. Even on the weekend here I am chatting via IRC to some tech friends with irssi in terminal under screen, and reading server status emails with mutt. the terminal, it's 21st century tool.

Re:Set 32 sectors per track

[Anonymous Coward]

terminals have nothing to do with the command line!
i think the op is complaining about the fact that things like
baud, stopbits and whatnot are deeply embedded in the
linux kernel. these concepts are not necessary to
have a command line. c.f. plan 9.

Re:Set 32 sectors per track

[Z00L00K]

Essentially we are back to the old problems of the ST412 interface where we had to figure out the best interleave for the drives as well when we were formatting them. Most drives then did have a fairly conservative interleave, but a reformat of them could improve the throughput considerably. A reformat could be done so that the whole track could be read in 2 rotations instead of 3, and what that does to performance is fairly easy to understand. C800:5 was a commonly used BIOS address where the low level format routine did reside.

But from what I understand this problem is an offset problem when the head steps from track to track, and that's also an issue to be considered. And today it's not common knowledge/practice to low level format hard drives.

And why stick at 4k sectors? Depending on the system you may want to use a different sector size. If you run Oracle on some systems the block size is 8k, and in that case you may want to have 8k disk blocks too since it would be good for performance.

Anyway - sooner or later we will have flash drives instead, and then this isn't a problem.

Re:Set 32 sectors per track

[Brian Gordon]

Yes, thank you. That's exactly what I mean.

Re:Set 32 sectors per track

[amorsen]

Anyway - sooner or later we will have flash drives instead, and then this isn't a problem.

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Re:Set 32 sectors per track

[blincoln]

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Couldn't this be addressed (at least in part) by a battery-backed write cache like better RAID controllers use? Set it up like SAN snapshots (so it just stores the diff between what's in the actual flash storage and what's been changed so far), and then write the changed blocks when it's most advantageous (e.g. when there's an entire block's worth of data, so it would all have to be erased by the flash storage anyway).
Maybe combine that with something like a disk defrag, except instead of storing frequently-sequentially-read data in physical sequence, store frequently-written data (regardless of if it's sequentially-read or not) in physical sequence.

Re:Set 32 sectors per track

[kimvette]

The terminal is not irrelevant. If your Cisco router is ever compromised (it happens) or if IOS becomes corrupt (or if you have an IOS install with a nasty bug where the password does not save correctly, or when an IOS upgrade goes badly) or someone fudges the configuration up, the only way you can recover it is often through the serial port. Serial ports are also very handy for integrating video surveillance with point-of-sales systems that are not IP-aware (or worse, antiquated DVR appliances which can't do POS integration over IP), for some smart switches, *NIX boxes that have been rooted (I've rescued a Solaris box through a serial connection in an enterprise environment where reinstall was not possible due to poor timing - week of finals - and backups were sabotaged by a disgruntled gradute student and logins through IP and at the console were blocked), and so forth. However, I'd rather see RS-485 or RS-422 take RS232's place, since RS-485 and RS-422 can work over much longer distances and you can hang multiple serial devices off of a single bus.

RS-232 might be absent from a lot of consumer motherboards, but it is far from dead and certainly not irrelevant, even now in 2010.

Re:Set 32 sectors per track

[kimvette]

Oh, in addition, now that Windows Server (Core) has a real GUI-less mode and Powershell and UNIX environment shells on Windows finally have usable interfaces, shell prompts are becoming even more relevant even in large Window shops. So, even Microsoft has acknowledged that the UNIX-y way of doing things is key for automation and uptime in an enterprise environment. Now, most PCs won't boot with output to the serial port, but some enterprise server boards do have such options.

A GUI is great for basic tasks, but for repetitive tasks a command shell and scripting environment are key for efficiency, and reliable automation. VBS/Windows Scripting Host was an "acceptable" workaround for a while but in the past many Windows administrative tools required the box to not be headless, the workstation unlocked and the windows open for the GUI to be accessible for scripting - and even then it was iffy because not all GUI elements are accessible (especially third-party tools with custom controls).

Re:Set 32 sectors per track

[bertok]

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Couldn't this be addressed (at least in part) by a battery-backed write cache like better RAID controllers use? Set it up like SAN snapshots (so it just stores the diff between what's in the actual flash storage and what's been changed so far), and then write the changed blocks when it's most advantageous (e.g. when there's an entire block's worth of data, so it would all have to be erased by the flash storage anyway).
Maybe combine that with something like a disk defrag, except instead of storing frequently-sequentially-read data in physical sequence, store frequently-written data (regardless of if it's sequentially-read or not) in physical sequence.

That's exactly what most SSD controllers do!

Some now come with 32 to 64MB of cache, and some of the new Sandforce controller based SSDs also come with a little ultracapacitor that acts like a mini UPS. The cache is used as scratch space for reordering writes and defragging blocks.

There was a firmware patch recently for the OCZ Vertex series of SSDs that enabled background defrag. If you let the drive site there for a few minutes, it would start getting faster until it returned to 'as new' speeds

Re:Set 32 sectors per track

[coolsnowmen]

In window's defense, they recognize thumbdrives just fine for almost a decade; it is the manufacturers who bottle their own autostart and drivers and non-standard thumbdrives so they can include backup software, encryption, or flat out spyware.

Re:Set 32 sectors per track

[orkysoft]

I recommend you visit Microsoft and have a look at their "Windows" operating system. The concept of a terminal doesn't run nearly as deep in it as it does in Linux. The same goes for the concept of security. Overall, it is kind of a poorly reinvented UNIX, but I think you might just like it. There are quite a few applications available for it nowadays, and it is gaining more and more marketshare and public recognition.

Re:Set 32 sectors per track

[MobileTatsu-NJG]

We both agreed that most of windows land involves emailing shit to yourself, and a lot of USB thumb drive use...

Explorer: \\ComputerName\c$\Documents and Settings\UserName\My Documents\

Permissions permitting, this is all you need to do. Or you just share folders.

(Of which I could fire off a good half-hour rant on how poorly windows handles mass storage devices. It's a USB THUMB DRIVE for gods sake. It's not a fucking printer! I want to plug it in, and transfer files to/from it. It doesn't need to be "installed", indexed, and have drivers downloaded for it. Just fucking open a file browser like any sane OS does. )

This is a 10 year old complaint.

I have a hard time working on windows, because I'm so much more efficient with a terminal. It's not that I can't use a gui - I'm just an order of magnitude faster using the terminal.

That and you're not using Windows properly.

Re:Set 32 sectors per track

[apoc.famine]

Damn, you really don't know what a terminal is, or what they do, do you?

Explorer: \\ComputerName\c$\Documents and Settings\UserName\My Documents\

Doesn't do anything I posted I did. I'm trying to put to words how ridiculously out-of-scope your "solution" is, but I really can't. You really need to learn what people do with terminals before posting on the internet, so you don't look like a complete dumbass. It doesn't help that I'm talking about transfers between machines which are not on the local network, and which don't run windows on either end.

And for the record, I AM using windows properly - I boot into it to play games. It's great for games, and not useful for anything else.

Re:Set 32 sectors per track

[MobileTatsu-NJG]

Ah, you're right, I missed the "between work and home" bit. I apologize. That's a little tougher because you need to get the two machines talking to each other, and that's what SSH does nicely. One way would just be to add a step in the beginning to set up a VPN. Alternatively I think Windows has a built in feature for connecting two machines' file systems remotely, but I haven't messed with it in years. There's probably a half dozen other ways to do it. Windows can share files over the net just fine, it's not some feature exclusive to *nix.

And for the record, I AM using windows properly - I boot into it to play games. It's great for games, and not useful for anything else.

Right. You're using it properly by being utterly unaware of how Windows networking works and using USB flash drives (that only seem to require drivers whenever you use it) and email to share files between two internet-connected machines. I wouldn't want to look like a dumbass for challenging you there.

Re:Set 32 sectors per track

[apoc.famine]

I'm sure that windows can share files over the net just fine. The issue is that it doesn't do it easily, cleanly, or by default. I could probably learn how to do it, but why bother?

>> scp file_location file_destination

That was my original rant - it doesn't matter if either location is on the same file system, a device connected to the computer, a computer on a local network, or running on something in space.

That was my point about ease and efficiency you completely ignored by offering a something that definitely wasn't a solution to my initial complaint.

As for your snarky reply about me not using windows properly, once again you don't seem to understand how that shit works. I wanted to quickly transfer 1gb of files off a windows laptop in a lab, so I slapped a USB thumb drive into it. As it was the first time it had been plugged in, I was treated to 2 minutes of installation and configuration. The same thumb drive, plugged into the OSX machine across the hall immediately showed up in Finder, and I started pulling files off it 2 seconds later. Sure, should I ever plug that same drive into that same laptop, it probably won't do that. However, it shouldn't have done that in the first place. It's a damn storage device.

If that laptop had any sort of useful networking on it, I'd have just transferred the files from it to the OSX machine. But windows doesn't come with anything that can talk to OSX by default.

Sorry, but when it comes down to it, Windows blows for networking and file transfers. I'm sure that with some windows-fu and some googling, and some third-party software you can get windows to transfer files securely across the internet to another machine. The real question is "Why would you bother?" Unix based systems just do it by default, pretty much as quickly and as efficient as possible.

If you're setting up a VPN to transfer files, you're not using windows properly. You should be playing games on it, and using a decent OS for networking.

Re:Set 32 sectors per track

[Anonymous Coward]

I don't think he was dissing command line interfaces.

I think his complaint was that even newfangled RS-232 terminals had to jump through hoops to remain compatible with computers that were hooked up to typewriters and line printers. The protocols and underlying software have idiosyncrasies built into them that just don't make sense any more. Instead of throwing away the cruft to make something better, everybody's hacking onto the same old outdated shit. It's limiting progress, in a way.

Re:Set 32 sectors per track

[hoggoth]

> . this document is heart wrenching

I don't think that word means what you think it means.

Re:Set 32 sectors per track

[Anpheus]

I don't know if an object oriented command line interface in which objects, not text, is piped from one command to another is quite the same thing as the "Unix way." It adds an enormous level of complexity.

Not that it isn't interesting or useful. And we'll see how well it takes off in the coming years as more Microsoft management tools become GUI interfaces that write Powershell scripts.

Re:Set 32 sectors per track

[mirix]

RS-422 is great. I've got a 4 port card in this machine, and have direct control over 4 machines (on a different floor no less) with it, whether the network is up or not.
Well, apart from a power switch.

Re:Set 32 sectors per track

[MobileTatsu-NJG]

The issue is that it doesn't do it easily, cleanly, or by default. I could probably learn how to do it, but why bother?

Because you had to learn how to do it anyway when you started with Unix. I'm sure there were a lot of things there that you couldn't find 'by default' that you ended up slapping your forehead about later.

That was my original rant - it doesn't matter if either location is on the same file system, a device connected to the computer, a computer on a local network, or running on something in space.

It doesn't matter in Windows, either. You have to have them talking to each other, just like you do in Unix.

As for your snarky reply about me not using windows properly, once again you don't seem to understand how that shit works. I wanted to quickly transfer 1gb of files off a windows laptop in a lab, so I slapped a USB thumb drive into it. As it was the first time it had been plugged in, I was treated to 2 minutes of installation and configuration. The same thumb drive, plugged into the OSX machine across the hall immediately showed up in Finder, and I started pulling files off it 2 seconds later. Sure, should I ever plug that same drive into that same laptop, it probably won't do that. However, it shouldn't have done that in the first place. It's a damn storage device.

Okay... so your complaint is that the first time around it needed to configure itself and it works every time after that. But you don't know for sure because you've only done it once... right? And this means you know everything about how Windows works? I just don't think if I tried to make the same complaint about Unix or Linux I'd be let off the hook very easily. In fact, I know this. I've gotten roasted pretty heavily before for not instantly knowing which configuration file to go into with a text editor and which line to flip values on.

If that laptop had any sort of useful networking on it, I'd have just transferred the files from it to the OSX machine. But windows doesn't come with anything that can talk to OSX by default.

Both OS's are configurable with network shares that the opposite OS can talk to. And when I mean 'congfigurable', I mean you don't need to install anything special.

Sorry, but when it comes down to it, Windows blows for networking and file transfers.

*nix is definitely much stronger at networking and remote administration than Windows. No argument from me, there. However, the problems you're ranting about were easily solvable.

If you're setting up a VPN to transfer files, you're not using windows properly.

Hee hee. Why do I think this line was written after a hasty Google search for "Windows VPN"?

You should be playing games on it, and using a decent OS for networking.

You should learn how to use whatever OS you're working on before you run around publicly criticizing it.

Re:Set 32 sectors per track

[hullabalucination]

All of the desktops in use at my local Chamber of Commerce (WinXP) can't show/mount a thumbdrive, apparently related to this issue:

www.howtogeek.com/howto/windows/find-your-missing-usb-drive-on-windows-xp

I pointed them to a Microsoft knowledgebase entry which supposedly fixes it, but they reported that it didn't work for them. I don't use Windows much so the issue is puzzling to me; the Chamber has a system builder on retainer but apparently nobody there has seen fit to rectify the problem (or just can't).

* * * * *

Be careful about reading health books. You may die of a misprint.
—Mark Twain

Re:Set 32 sectors per track

[quantumphaze]

I think he is talking about how Windows will give you a popup stating that it is installing a new device the first time you insert a USB stick who's specific model has never been used with that Windows install. I just tried it on our Vista machine and it has a popup saying something like "Installing New Device Software" then "Your device it ready to use".

It's an improvement on what XP did. XP would go though setting up the driver to work not only with each new model it saw, but for every time you used a different USB port on the same computer. It works the way it should if you plug an already used USB stick in a port it was in previously

Re:Set 32 sectors per track

[mikael]

'lynx', '(al)pine', bash, fdisk, parted, vim and ssh are still extremely popular. Command line isn't quiet dead yet.

Re:Set 32 sectors per track

[yuhong]

Yea, it was a big point in the Windows vs NetWare wars in the late 90s, as I remember

Re:Set 32 sectors per track

[yuhong]

So they have *started* to move to a server OS that doesn't have a ridiculous UI. I think it will be 2020 before we see a proper MS server operating system - where people who have no business being on the server are lost and don't know what to do.

BTW, Windows Server 2008 already has "Server Core" which is pretty close, if it is not already, I think.

Re:Set 32 sectors per track

[dargaud]

Yes, and the most funny thing is that the Mac, which was for a decade and a half a pure GUI system, now has a perfectly usable command line bash shell !

Re:Set 32 sectors per track

[cerberusss]

I recommend you visit Microsoft and have a look at their "Windows" operating system. [...] Overall, it is kind of a poorly reinvented UNIX, but I think you might just like it.

I've seen some people use it, then they told me you had to pay for it. I was flabbergasted -- why would anyone pay for an operating system?

Re:Set 32 sectors per track

[bemymonkey]

Decent USB sticks (i.e. no weird autostart firmware or things like that) take about one full second to go through the "Installing New Device Software" on Win7... Is this so much faster on Linux/OS-X?

It's never bothered me much, but hey, maybe I'm missing out on some fantastic USB-Stick superspeed ;)

Re:Set 32 sectors per track

[bemymonkey]

I'm sure that windows can share files over the net just fine. The issue is that it doesn't do it easily, cleanly, or by default. I could probably learn how to do it, but why bother?

What's not easy about right-clicking on a folder and setting a few permissions in the sharing tab?

>> scp file_location file_destination

That was my original rant - it doesn't matter if either location is on the same file system, a device connected to the computer, a computer on a local network, or running on something in space.

Care to elaborate? How is this easier on *nix? Replace "scp" with "copy" and it'll work just fine on Windows...

As for your snarky reply about me not using windows properly, once again you don't seem to understand how that shit works. I wanted to quickly transfer 1gb of files
off a windows laptop in a lab, so I slapped a USB thumb drive into it. As it was the first time it had been plugged in, I was treated to 2 minutes of installation and configuration. The same thumb drive, plugged into the OSX machine across the hall immediately showed up in Finder, and I started pulling files off it 2 seconds later. Sure, should I ever plug that same drive into that same laptop, it probably won't do that. However, it shouldn't have done that in the first place. It's a damn storage device.

So buy a storage device. Not a device that has "Automagical backup software" and "Virtual Optical Drive Emulation" built into its freakin firmware. Obviously this stuff doesn't work on *nix, because it's programmed for Windows, but you should be placing the blame on the manufacturers for ramming this crap down peoples' throats...

If that laptop had any sort of useful networking on it, I'd have just transferred the files from it to the OSX machine. But windows doesn't come with anything that can talk to OSX by default.

Isn't it also the other way around? i.e., OSX doesn't come with anything that can talk to Windows by default? I'm guessing there'd need to be compatibility on both sides...?

Sorry, but when it comes down to it, Windows blows for networking and file transfers. I'm sure that with some windows-fu and some googling, and some third-party software you can get windows to transfer files securely across the internet to another machine. The real question is "Why would you bother?" Unix based systems just do it by default, pretty much as quickly and as efficient as possible.

So you're saying that Windows is crappy for networking and file transfers because it doesn't have a built-in utility for securely transferring files across the internet?

Sure, I think a built-in facility would be useful, but the majority of Windows (home) users is going to be perfectly satisfied with opening "Network" in "Computer" and selecting the machine they want to transfer files to/from (this needs no setup, by the way - just leave your router with DHCP enabled and the machines will pretty much find each other without problems since XP [although back then it helped if the machines were all set to the same workgroup]).

Windows 7 has added "Homegroups" and all sorts of other crap you don't need (and would have to configure first), but simple file-based networking works perfectly fine without it... just specify which folders you want to share (read-only by default, set specific permissions if you want something else) on the remote machine and you've got access.

If you're setting up a VPN to transfer files, you're not using windows properly. You should be playing games on it, and using a decent OS for networking.

Hmmm, I use my VPN for opening files in place, not for file transfer. Using a VPN means you're effectively _inside_ the network you're VPN'd into, and you can use it as such. No need to keep the files locally at all, unless you have bandwidth constraints that make working in this way difficult...

Re:Set 32 sectors per track

[quantumphaze]

Linux isn't really fast with it. It takes about 4 seconds from insertion until KDE's device notifier pops up. Then again YMMV with different sticks. For all I know it may be programmed that way to let USB hard disks spin up fully before mounting.

Compare this to an SD card in my laptop's card reader which is near 0.3 seconds. Which is another thing I find Windows doesn't handle correctly with regards to unmounting. Windows Vista will present our desktop's built in multi-card reader (ones that take the old floppy drive slot and plug into the front USB pins on the motherboard) as persistent F:, G:, H:, I: drives and under the one entry in Safely Remove Devices.
Removing this will remove the actual reader from Windows, not just the cards. The reader is unusable until the machine is rebooted (there may be another way that doesn't involve replugging the reader in, please tell if there is) as the USB device is completely disabled. Ejecting the individual drive by r-clicking it in My Computer will unmount it without killing the card reader.
I don't know what Windows 7 does, not installed on that computer.

Linux seems to ignore the card reader and treats the cards as normal removable storage. I can even remount it again without having to replug it.

Re:Set 32 sectors per track

[bemymonkey]

Interesting, I've had this issue too but attributed it to the cheap USB card reader.

Re:Set 32 sectors per track

[jonadab]

> What's not easy about right-clicking on a folder
> and setting a few permissions in the sharing tab?

CIFS filesharing is one thing on a LAN, and something else again if you're transferring via an untrusted network, such as the internet. That's what scp is for.

The other user's main beef, near as I can tell, is that getting scp to work on Windows is an exercise in hunting down third-party software. Which is true.

But that's not really unique to scp. *Anything* you want to do on Windows is generally going to require having additional software installed that's not included with the OS. Even FreeBSD comes with more functionality "out of the box" than Windows, and I'm not talking about ports. Windows is basically just an OS (and a basic desktop environment), not a whole software distro. That's kind of the point: Microsoft supplies the OS, and you get the other software separately, usually from third parties (although Microsoft has some complementary products too, e.g., Word).

That's the way it's *supposed* to be on Windows; it's *designed* to be that way. With Windows, your computer is supposed to only have the software you've gone out of your way to install, so there's not a bunch of other stuff cluttering up your All Programs list. Windows is made for people who pretty much use the computer for the same few tasks all the time, so they only want the software they want and not a bunch of other (to them irrelevant) stuff. If that bothers him and isn't the way he likes things, if he wants a whole distro with software for a wide variety of purposes installed out of the box, why is he using Windows in the first place?

Re:Set 32 sectors per track

[jonadab]

> RS-232 might be absent from a lot of consumer motherboards

Yeah? Can you actually name a recent non-Apple desktop or workstation motherboard that doesn't have it? (Netbooks, yeah, sure, but those are supplemental portable devices, not intended to replace a desktop computer.)

Compaq experimented with selling "legacy-free" systems for a few months circa 1999. Their "iPaq" model of mini-tower system, for instance, was available in a USB-only "legacy-free" version, and then there was a "standard-ports" version for just a couple bucks more. When they discontinued the iPaq line in favor of the Evo a few months later, guess which way they went with it? Then HP bought them out and the whole thing become an obscure footnote for the computer history books.

I predict RS232 will outlast USB 2.0, because subsequent USB versions will replace 2.0, but RS232 isn't going anywhere.

While we're at it, PS/2 keyboards and mice don't seem to be going anywhere either (although a lot of them have become de-facto hot-pluggable, which may be at least partly due to the influence of USB).

In fact, the last major computer component interface I can think of that died out is ISA.

Re:Set 32 sectors per track

[bemymonkey]

So basically the gripe is, "I want everything I need built into the OS."?

Hmmmm, I'm just a dumb Windows user, but isn't that "everything including the kitchen sink" approach what many FOSS users refer to as bloat?

Re:Set 32 sectors per track

[TemporalBeing]

RS-232 might be absent from a lot of consumer motherboards

Yeah? Can you actually name a recent non-Apple desktop or workstation motherboard that doesn't have it?

It is actually very hard to find an RS-232 port on newer computers now. Almost all manufacturers have dropped them from laptops at the very least, if not their desktops as well.

Typically if you want to work with a serial port you now have to get a USB to Serial port interface.

RS-232/422/etc have gone away.

BTW - neither my Lenovo T61P nor my wife's HP laptop have a serial port of any kind on them. My 2003 era Dell D600 does, but most of the Dell's don't today (or even a few years ago).

Re:Set 32 sectors per track

[coolsnowmen]

Clearly you are not making this up, but I can't remember having this problem. Thanks for the link.

Re:Set 32 sectors per track

[jonadab]

> but isn't that "everything including the kitchen sink"
> approach what many FOSS users refer to as bloat?

Good point. There are two schools of thought on that. When you see people complaining about bloat and arguing that no extra features should be included in anything, you're seeing the opinion of a percentage of the open-source community. The most extreme of these people will argue with a straight face that the base OS install shouldn't include even basic utilities that many of us take for granted (e.g., egrep, bzip2), and furthermore that only very basic versions of essential utilities should be installed: ash or somesuch instead of bash, more instead of less, and so on. In fact, FreeBSD doesn't include bash, so if you do install it (via ports) it goes in /usr/local/bin instead of /bin.

To most Linux users, this extreme approach is unexpected and weird. Most of us expect such things as bash and bzip2 and egrep -- and perl and screen and make for that matter -- to be installed on all computers as a matter of course.

The other school of thought is, I don't want to fool with it, just install everything. Most Linux distributions offer you the option, at install time, to check and handful of boxes for broad categories (like Gnome, KDE, server stuff, and so on) and have a whole bunch of packages from those categories installed automatically. Just install it all, so I don't have to fool with selecting eight hundred packages individually. This point of view is just as common, if not more so. And this side of things also gets taken to extremes sometimes (e.g., there are people who literally install the ENTIRE contents of the CPAN on their computers).

Most of us are somewhere in between. I generally install all the major categories (Gnome, KDE, etc), because modern hard drives are big enough that I can, and it saves time selecting the individual packages I want. But yeah, there are still things that don't get installed, and so you have to select those for installation individually.

You can, however, get practically everything from the distro's repositories, so it's seldom necessary to hunt down anything from a third party, unless you insist on installing proprietary software (like Opera), or something so specialized or obscure that it's not packaged (like Evergreen/OpenILS).

Re:Set 32 sectors per track

[jonadab]

> It is actually very hard to find an RS-232 port on newer computers now.

Huh. That's... odd.

It's been years since I've seen a computer, new or otherwise, (other than laptops, as I noted earlier, and Macs) that doesn't have the serial and parallel ports.

> neither my Lenovo T61P nor my wife's HP laptop

Laptops, right. I already mentioned those as an exception, didn't I?

You don't get as many ports on a laptop, because nobody would use them anyhow, because nobody uses peripherals with a laptop anyway, because carrying them around everywhere would kind of defeat the whole portability thing, which is the main point of a laptop.

And Macs also don't have printer and serial ports -- even when they did, they weren't the same as the ones on PCs and so weren't compatible with non-Mac peripherals anyway, so no real change there.

But RS232 isn't going away at all. PCs (other than laptops, do I really have to keep saying that) all still have parallel and serial ports. And a lot of peripherals still use them (we just bought such a peripheral a6t work, so recently that we haven't even installed it yet).

Re:Set 32 sectors per track

[RivieraKid]

It's not so much a problem with track-to-track seeking, as it is a problem with the misaligned I/O itself.

If your on-disk structures do not align with the physical structures, then you experience a penalty for every host I/O to that disk. It's exactly the same issue with enterprise class storage arrays. the volume on the array is offset from the very beginning of the physical disk so is correctly aligned. However, every x86 style partition table laid down on those volumes consume the first 63K of the disk, meaning the data starts at the 64K mark - 1K before the start of the next disk track.

So, in a nutshell - for every host I/O, you will generate at least one additional I/O for the disk due to the badly aligned start of the partition table. Worst case, you'll generate two additional disk I/Os (for multi-track I/O) since the last track is also misaligned. For large block sequential I/O, you can pretty much ignore this effect, for small block random I/O - it's gonna kill performance. Align your partitions to 64K and the problem goes away - of course, this has to be done before you put any data on the partition and is not done by default with any OS I'm aware of today, which explains why nobody ever does it.

Note, since I mentioned storage arrays I'll clarify that this is not the same as the RAID write penalty.

Note also, this is usually only a problem for x86 style partitioned disks.

Re:Set 32 sectors per track

[fbjon]

What?...

At work, I have a PC that does not have an RS232 port, so I had to get a USB adapter. At home, under my desk, I have a computer just a few years old, also without RS232 (it does have parallel though).

Finally, looking through the motherboard selection at my local webshop, I see that RS232 is not just disappearing, it's gone. Not one board out of about 50 models of various sizes that I checked has it (I looked at the pictures of the connector side as well to make sure). The closest I can get is an old nForce3 board that takes DDR400 memory, on an auction site. That one has a serial port.

Conclusion: the serial port went away years ago.

Re:Set 32 sectors per track

[TemporalBeing]

You don't get as many ports on a laptop, because nobody would use them anyhow, because nobody uses peripherals with a laptop anyway, because carrying them around everywhere would kind of defeat the whole portability thing, which is the main point of a laptop.

Actually there are a few industries (e.g. trucking) where serial ports on equipment are very common place - so you do need them on portable devices. Unfortunately, the computer industry is more or less ignoring those industries so you have to get the converters instead.

Even if you did move to USB on the newer equipment, it'd still be years before the serial ports disappeared simply b/c of the slow turnover of the equipment.

So yes, there are good uses for RS-232/422 that don't defeat the purpose of portability.

And, FYI, USB was suppose to be the serial port killer, and the computer industry is treating it as such - so they've been dropping it more and more.

Re:Set 32 sectors per track

[jonadab]

> USB was suppose to be the serial port killer

Yeah, I heard that too, in the nineties. I also heard, around the same time, that Iomega drives were doing to be the floppy disk killer. They were starting to come out with internal models, see, and all new computers were going to come with the Iomega drives, and nobody would need floppies anymore. It never happened.

Instead, USB eventually turned out to be the floppy disk killer. CD-R made a decent start on killing floppies, but it couldn't quite finish the job (though it *did* kill off Iomega drives fairly effectively). Now, though, I'm seeing fewer and fewer floppies with every passing year, and I've seen more than a few new computers (*desktop* computers, not laptops) with no floppy drive. And it's not because of Iomega. It's because of flash-EEPROM-based USB 2.0 mass storage devices, the smaller capacity ones of which have become so cheap you basically get them in specially marked boxes of breakfast cereal these days. They cost less than PCI 10/100 ethernet cards or, more to the point, less than a small box of floppies. Apparently people must be using them, because I find more of them laying around (left behind by mistake) than I ever did floppy disks.

But as far as USB peripherals that formerly would have used the serial port, the only ones I've seen are digital cameras (and serial was never a good fit for them in the first place; SCSI would have made more sense, except that most computers didn't have it). I've seen devices that formerly would have been parallel and are USB, of course (printers mainly). And I've seen USB mice and keyboards occasionally. But peripherals that traditionally used a serial port, for the most part, still use a serial port. The camera is the only exception I know of.

Re:Set 32 sectors per track

[jonadab]

Where do you live, out of curiosity?

Like I said, all the new computers here have serial and parallel ports. Well, some of the laptops don't, but some do, and *all* of the regular computers do. I'm in central Ohio.

Re:Set 32 sectors per track

[TemporalBeing]

But peripherals that traditionally used a serial port, for the most part, still use a serial port. The camera is the only exception I know of.

Well, you also have scanners (though they were typically SCSI), external modems (who has a modem now?!), Computer-Enhanced Sewing Machines, and then there was the specialized equipment - like the Vistar Datacom MT 2000 that I worked with for asset tracking a few years back.

And of course, those specialized devices have very low turn-over, and don't get upgraded very much.

Either way, the computer industry itself has left serial behind, while the peripheral industries continue to use it. Thus the USBSerial device industry is doing quite well.

Re:Set 32 sectors per track

[yuhong]

even when they did, they weren't the same as the ones on PCs and so weren't compatible with non-Mac peripherals anyway

Ha? Apple designed their RS-422 serial ports from the beginning to be compatible with RS-232 through RS-423 provided you have the right cable..

Re:Set 32 sectors per track

[fbjon]

Finland.

I checked again, and there are some mini-ITX boards that do have serial ports. Other than that, the only non-used motherboard I have found with a serial port is an Asrock Socket A board, which is quite ancient by today's standards.

Re:Set 32 sectors per track

[jonadab]

> Apple designed their RS-422 serial ports from
> the beginning to be compatible with RS-232
> through RS-423 provided you have the right cable..

Yeah? Try to actually use PC serial peripherals with a Mac from the pre-USB era. The cable adaptor, as a rule, can only be counted on to work if the devices *comes with* said adaptor and is labeled as supporting the Mac as well as the PC. Otherwise, forget about it.

(In fairness, a lot of Mac peripherals don't work with the PC either.)

Granted, I don't know the technical details of the Mac serial port, so as far as I know it's possible that the problem isn't the serial port itself (other than the obvious need for an adaptor because the the physical plug is obviously dissimilar), but something else. (Drivers seem like a likely possibility.) All I know is, you couldn't use PC serial devices on Macs; it never worked.

Re:Set 32 sectors per track

[Xabraxas]

The OS supports it. Some tools do not yet properly support the new hard disks.

Good thread on this.

[Anonymous Coward]

http://www.osnews.com/thread?409281

Re:Good thread on this.

[darkpixel2k]

http://www.osnews.com/thread?409281

One of the comments in that thread suggests switching to GPT if you aren't using Windows.
I haven't used Windows at home since ~2001.

Can you just wipe/reinstall using GPT? I thought the BIOS was involved with the type of partition table and that I had to be using the msdos partition type because of the BIOS. Can a geek with deeper knowledge of partitions and and all things boot drop some knowledge?

Re:Good thread on this.

[Anonymous Coward]

The BIOS has no understanding of partition tables. It merely reads the first sector of the harddrive to 0x7C00 and then jumps to that location. The DOS partition table is used by convention for interoperability between operating systems. If you wanted to use a different partitioning scheme, there is no technical reason your operating system couldn't.

Re:Good thread on this.

[Anonymous Coward]

GPT wraps itself in a MBR partition map. At the very least the GPT is supposed to include an MBR map that claims the whole disk as used by GPT to avoid issues with old disk tools and the like. And if you've got a partition scheme that's compatible with the MBR scheme they can both contain the same information, assuming your disk tool supports this, so that MBR-only environments can still find your partitions.

It's also possible to format with GPT and then use an MBR-only tool (fdisk) to go back and manipulate the (fake) MBR to contain a partition that points to the same start/end points as the GPT boot partition -- GPT-aware systems will just ignore the MBR record, and non-GPT systems will at least be able to find the boot partition.

As to whether your motherboard/firmware supports GPT, it can be hard to say. Anything with EFI is required to support GPT. Some systems with a legacy BIOS pre-boot environment also have support for GPT, because it's the only way to support large disks. But I can't name particular firmware versions that do/don't support GPT.

Re:Good thread on this.

[marcansoft]

Unless your BIOS is trying to be too smart and peeking into your partitions instead of launching the MBR (sadly, some do), it won't matter. It's the MBR's job to boot your system after the BIOS hands off control to it, and on most Linux systems the bootloader is installed straight into the MBR.

Re:Good thread on this.

[Z34107]

Even if you are using Windows, Vista and up support GPT. It's handy for servers where you expect to have partitions larger than 2 TB.

But I guess if one were using a modern version of Windows, you wouldn't have the 4K alignment problems to begin with.

Re:Good thread on this.

[Z34107]

That's an interesting point - I assumed diskpart supporting it meant it could boot from it.

However, Vista and 7 (and the server editions) can boot from a GPT partition, but only the x64 bit versions. source

Re:Good thread on this.

[Anpheus]

I think Windows Vista & 7 requires UEFI + GPT simultaneously to boot.

IMO, it's good for the most popular OS to be really conservative on the issue of how booting is performed because it is so easy to mess up. I cannot count how many times, when experimenting with dual and triple booting, I've rendered a system disabled or nearly so. While recent tools (SuperGRUB, for example) make it easier, it's still a veritable minefield.

Re:Good thread on this.

[yuhong]

Yep, I wonder once we get beyond 2TB whether it is better to increase the logical sector size or just switch to GPT.

Re:Good thread on this.

[yuhong]

Which cause trouble with the Windows installer that always replace the MBR with the standard MBR. Luckily most bootloaders can be set to install into the partition boot sector, which avoid this issue.

Re:Good thread on this.

[RivieraKid]

Sorry, but up to Server 2008, Windows still doesn't handle partition alignment. Unless I'm completely mistaken, that's largely same kernel in Vista and Win7 too...

Interesting

[Murdoch5]

I actually have 2 of the these drives in my desktop right now. There is a slight decrease in performance compared to Windows 7 but nothing that it unacceptable or even a need for concern. If you need to worry about the performance lost with the 4k sectors then just go solid state.

Re:Interesting

[ScytheBlade1]

TFA disagrees with you.

$ time cp winxp.img /mnt/sdc # ALIGNED
real 5m9.360s
user 0m0.090s
sys 0m20.420s

$ time cp winxp.img /mnt/sdd # UNALIGNED
real 13m26.943s
user 0m0.110s
sys 0m19.350s

$ time cp -r Computer Architecture/ /mnt/sdc # ALIGNED
real 42m9.602s
user 0m0.680s
sys 1m59.070s

$ time cp -r Computer Architecture/ /mnt/sdd # UNALIGNED
real 138m54.610s
user 0m0.660s
sys 2m15.630s

The first two being a single file, the latter two being multiple files in a larger directory structure.

I would heartily disagree with you on the matter.

Re:Interesting

[markus_baertschi]

What do you disagree with ? There is a performance problem and the author thinks the kernel people should look into the matter. I don't think they can do much, after all the kernel just writes 4k block to the drive. The problem is with the drive microcode which should attenuate the problem in the case of sequential writes, like the first case. Obviously it does not do that at all. Markus

Re:Interesting

[Murdoch5]

That is more a distro problem then, I run a full stable Gentoo system and I see less then a 5% over head difference. Not saying your wrong because you posted output here but it can have a lot to do on how you have optimized your disk access and setup the kernel to interface the disks. Even the file system in use.

But then again that's just what I see on my system. Possibly running a different configuration would yield different results.

Re:Interesting

[ScytheBlade1]

I disagree with the "but nothing that it unacceptable or even a need for concern" part. If copying the same image to the same disk where the only difference is where the partition begins -- by one sector difference -- will amount to a 2.6x decrease in speed, and where copying a large directory with many subdirectories amounts to a 3.2x decrease in speed... that should qualify as an unacceptable decrease in speed that warrants concern.

While a kernel tweak may help alleviate the issue, it is primarily an issue with our current (userspace) disk partitioning and formatting utilities. I'd also disagree with you on the point where the problem is the drive microcode; drives should do what they are told, and not guess on behalf of the instructions they are given what to do. Admittedly, the microcode tweak would be minor and largely trivial, but I'd rather not fix (primarily) userspace software problems in the kernel, nor the device firmware.

Re:Interesting

[ScytheBlade1]

Any chance you can post drive model number, output of fdisk -l, and a few similar benchmarks?

Re:Interesting

[Murdoch5]

This problem isn't really kernel involved, the problem comes from the fact the write is not being issued in an effective manor. This more so leads to the fact that the system has not been fitted to use the 4k sectors. If you just put a little work into the install you can over come this problem.

By default yes you will likely see a big performance hit, but like any good system once it's tuned and fitted to it's job it will run great. People can post all the results they want from default installs and it means very little.

It's the same as if you bought a muscle car and never took the time to tune the engine, you can't try to make the claim that the car wasn't ready. you need to put the time in and get the car up and going.

The same thing is true of the install. You need to put the work in, tune it up and reap the benefits of a high performance install.

Re:Interesting

[markus_baertschi]

About the microcode part. The drive pretends to be a 512byte drive, but internally is using 4k sectors and and claims to 'translate transparently'. I can understand that in a random-access scenario it it has to read-modify-write 2 sectors each time and performance suffers (2 additional reads and one additional write). But in a sequential access scenario, the penalty should be once per sequence/file, not once per sector. Here the microcode fails completely to make the best out of the suboptimal situation.

Re:Interesting

[hedwards]

That's true, but it's also true that having hardware lie to the OS isn't a great situation to be in. At the very least there should be some way of forcing it to be honest for the benefit of OSes that can handle the reality. A lot of the gunk and instability in computing comes from hardware that does things that are more appropriately done by software and vice versa.

Forcing users to optimize isn't inherently wrong, it's just that they shouldn't need to do it for things which are somewhat standard as a work around for weird hardware designs. And yes, I realize that the 4096byte sectors aren't being implemented arbitrarily.

Re:Interesting

[maeka]

I hope you mean 70+ MB/s.

Re:Interesting

[bertok]

Wait.. wait.. what the fuck?

I was under the impression that mis-aligned partitions cause small random IOs to take up TWO small random IOs, basically halving random IO performance. It should have a negligible effect on any streaming operation, as only the first and last blocks will have an overhead, the others overlap with useful IO.

Just how BAD is the Linux 'cp' program? Is it doing writes one block at a time, or something insane like that?

On Windows, if you repeated the copy same test, you wouldn't see a measurable difference, unless the target partitions were horrifically fragmented. To get a real slowdown, you'd have to do something like work on a large database file (many small random reads and writes). Microsoft has a paper on tuning Exchange performance, where they demonstrate >100% overhead to unaligned partitions, but you get full speed for backup & recovery operations.

Re:Interesting

[coolsnowmen]

It is not a distro problem. On top of that, you run the same distro TFA does.

Evaluation setup and methodology:

        * Gentoo ~amd64 system with 2.6.31-gentoo-r5 kernel
        * fdisk version: fdisk (util-linux-ng 2.17)
        * The drives are identical, but I did not try swapping configurations to make sure that one drive isn't fundamentally slower than the other.
        * Core 2 Quad at 2.33GHz (Q9450), 8GiB of RAM
        * MSI X48 Platinum motherboard -- Intel X48 Express + ICH9R

Re:Interesting

[ScytheBlade1]

http://git.savannah.gnu.org/cgit/coreutils.git/tree/src

Of note, copy.c, copy.h, and cp.c.

Re:Interesting

[petermgreen]

You don't want a drive reporting different logical sector sizes under different situations, that's just a recipe for breakage with any format that works in terms of sector counts.

And having a 4096 byte logical sector size would cause major compatibility issues.

IMO the best compromise would be to have the drive allow reading of it's physical sector to logical sector size ratio somehow. Partitioning tools could then warn about misaligned partitions and offer to fix them.

Re:Interesting

[bertok]

http://git.savannah.gnu.org/cgit/coreutils.git/tree/src

Of note, copy.c, copy.h, and cp.c.

Wow.. it's like code that time forgot.

I love the way there's about a page of complex code that jumps through hoops to figure out a good buffer size, when 64KB would basically work well in all cases. From what I can see, there's a few situations in which 'cp' will pick tiny buffer sizes and waste a bunch of time on overhead. The default appears to be 32KB, which is too small these days. I've done experiments (admittedly on Windows, but same difference), demonstrating that using IO blocks as large as 1MB improves performance significantly. There's a reason Microsoft increased the default IO sizes on Windows Server from 64KB to 1MB.

A much better buffer selection algorithm would be to detect L2 or L3 cache sizes of the processor, and use a buffer slightly smaller than one of those. You'd have to play with benchmarks to see if staying within L2 is a benefit (I suspect that it isn't).

The biggest problem with 'cp' is that it's synchronous single-threaded. It does a read, then a write, then a read, then a write, etc... For comparison, Windows Vista and later does reads synchronously but then writes asynchronously so that the next read doesn't have to wait for the write to complete. Even with kernel buffering and lazy IO, 'cp' is probably much less efficient that it could be.

Re:Interesting

[walshy007]

And having a 4096 byte logical sector size would cause major compatibility issues.

Only with ancient os's, anything in the last five years should handle it just fine. I'm all for backwards compatibility etc.. except when it can gimp current performance through emulation layers etc like what the hardware does with this 512 byte sectors to 4096 byte business.

Open Source to the rescue

[bogaboga]

I am no kernel hacker but I can almost guarantee that some kernel hacker will provide a solution to this "short coming" fairly soon.

That's the beauty of Open Source.

I am aware though that "fairly soon" means many things to many people; which means that there could be a substantial delay before we get a working solution to this issue.

I am optimistic nevertheless.

Request to Western Digital: Provide all the information needed to develop a solution.

Re:Open Source to the rescue

[nomadic]

That's the beauty of Open Source.

And that couldn't possibly happen with closed source?

Re:Open Source to the rescue

[MoHaG]

And they pretend to be 512 bytes for "compatibility"... (Unlike CD-ROM / SD card sectors...)

Re:Open Source to the rescue

[bogaboga]

I guess you only read but did not understand! Key words in my piece are: "Fairly soon."

Re:Open Source to the rescue

[marcansoft]

Exactly. Drives are pretending to have 512-byte sectors because Windows can't deal with 4k sectors, and then silently reducing performance when you believe them and use 512-byte sector sizes. Had the drives reported 4k sector sizes, they'd work great under Linux and not at all under Windows.

This isn't a Linux problem, it's a drive problem caused by Windows. The solution is to implement yet another workaround for stupid devices, and start aligning partitions to 4k by default.

Nitpick: SDHC card sectors are always 512 bytes, and most SD card sectors are 512 bytes too. Flash memory would benefit from larger sector sizes too, but they've probably stuck to 512 bytes for Windows compatibility.

Re:Open Source to the rescue

[nomadic]

"Fairly soon."

Again, that's still possible with closed source.

Re:Open Source to the rescue

[jedidiah]

> I guess the beauty of Closed Source, then, is that the OS supports it out of the box

      Or not. The sheep-like closed source user is never likely to notice the problem.

      Although the real problem here seems to be that you can't consider a bit of hardware
as it presents itself. Now I wonder why a hard drive company feels the need to have it's
hardware LIE to the OS?

Re:Open Source to the rescue

[hedwards]

Trust me, it's right after adding GPT support to Windows XP home on their agenda. Shouldn't take more than, I don't know forever to add.

Re:Open Source to the rescue

[Pentium100]

Now I wonder why a hard drive company feels the need to have it's hardware LIE to the OS?

So the hardware is compatible with more software. For example, hard drives still report some number of cylinders, heads and sectors to the BIOS and the OS, but hard drives have been using ZBR for 20 years now (IIRC) so the sector number is meaningless.

But, as it is now, if my old system needs a new hard drive, I do not need to find an old drive to be compatible with my system (as long as it is IDE or SCSI, I don't know of any adapter from the newer interfaces to ESDI or ST-506, but they probably exist).

They could have made it a jumper setting set to 512B by default though. I assume the hard drive is faster using 4KB sectors instead of true 512B sectors, they could have made an option to reformat the drive to 512B (or maybe it's not possible with modern drives, I have an old 4GB SCSI drive that can be reformatted to a different sector size (I never tried it though)).

Re:Open Source to the rescue

[rsmith-mac]

On the contrary, this has (almost) nothing to do with Windows - it has everything to do with old OSes. The IDEMA didn't approve the 4K sector standard until 2006; it was only in the late 90's that the first meaningful research was begun by IBM on whether 512B sectors would be an issue.

As it turns out, yes, 512B sectors would be an issue, and drive manufacturers would be best served by moving to larger sectors (with some arguing over whether to go to 1K or 4K). So the IDEMA hashed this out over the first half of the decade, and finally in 2006 approved the 4K specification.

The point of all of this is that software written at the turn of the century was all done well before changing drive sector sizes was a serious discussion. WinXP was released in 2001, Mac OS X 10.0 was in 2001, and of course Linux 2.4 was also in 2001. None of those OSes know what to do with anything other than a 512B sector - the only reason Windows factors in to this equation is that WinXP just happens to be with us (no doubt trying to eat our brains) while the other two are dead. Anything circa 2005 or later such as WinVista, Linux 2.6, and Mac OS X 10.5 know full well what to do with a 4K drive.

But even that is beside the point. You don't just make major jumps like this, you have to do it in a transition so that you don't break old hardware and old software alike. Even if XP/Lin2.4/MacOSX knew what to do with 4K sectors, at some point you'd run in to hardware, 3rd party devices, etc that would not. A transition is necessary to let old hardware and software get flushed out of the ecosystem, and as such we're still years out from consumer drives offering native 4K access.

In short: drives are pretending to have 512-byte sectors because there's a lot of old stuff, including Windows XP that can't deal with 4K sectors.

Re:Open Source to the rescue

[TheKidWho]

My Hard drive with 4k sector sizes works perfectly under Windows 7?

Re:Open Source to the rescue

[DAldredge]

Drives have been lying to the OS for more than 20 years in regards to quite a few different things so why pick now to yell and get upset?

Re:Open Source to the rescue

[Blakey Rat]

Don't forget that this "beauty of open source" doesn't even get around to thinking about fixing the issue until a competing OS has had support for it for 3 entire years.

Re:Open Source to the rescue

[Blakey Rat]

Well, since Windows Vista and Windows 7 already support this, I'd say "fairly soon" is demonstrably false. Unless you're happy with "fairly soon" being "after everybody else has been doing it for several years."

Re:Open Source to the rescue

[marcansoft]

There are no hard drives with logical 4k sector sizes out there, as they all emulate 512-byte sectors for compatibility, so your point is moot. However, if they did exist, they would theoretically work with Windows 7. I was talking about Windows XP.

Re:Open Source to the rescue

[Miseph]

s/windows/windowsxp

Whatever you may think of Vista/7/Server2008, this isn't a problem for them. I agree that it's stupid for WD to make their drives lie for compatibility with what is effectively a legacy OS (by modernity, not use), but there's no call to tar newer releases for it.

Re:Open Source to the rescue

[marcansoft]

I was referring to Windows XP - I should've mentioned it explicitly. Nonetheless,

There's a lot of old stuff, including Windows XP that can't deal with 4K sectors.

There's exactly one old thing used in any significant quantity and liable to have to work with these drives, and it's Windows XP. Everything else either isn't in any significant use any more, or will never be seeing one of these drives.

One potential concern would be USB enclosures which have to work with older OSes / devices. To that, I would say it should be the enclosure's job to do the 512-byte sector emulator, not the drive's.

Windows XP has been updated over the years (via service packs and the like) to handle hardware that didn't exist at the time of its release, including things like SATA. They could do the same for 4k sector disks, but they aren't going to do it because they want people to move to Win7. Therefore, Microsoft is still to blame for neither 1) providing a solution for XP, nor 2) providing enough compelling reasons to migrate to Win7. Heck, Vista was a trainwreck and it doesn't really count, so (proper) Windows is actually 3 years late to the 4k party, as Windows 7 was only released in 2009. Effectively, they've spent those three years scaling down support for XP while providing no viable alternative, and now the rest of the world has to deal with a significant amount of people still using a 9-year-old OS.

In short: the fact that people are sticking to 9-year-old XP and making hardware companies break or slow down improvements means Microsoft did something terribly wrong.

Re:Open Source to the rescue

[pchan-]

Nitpick: SDHC card sectors are always 512 bytes, and most SD card sectors are 512 bytes too. Flash memory would benefit from larger sector sizes too, but they've probably stuck to 512 bytes for Windows compatibility.

This is no longer true. Most 2x NAND memory manufactured in the past year is 4KB block sizes with 8KB coming soon. That it pretends to be 512 bytes is a function of the SDIO MLC driver IC. Luckily for SD they come pre-partitioned so that the partitions are aligned properly.

Re:Open Source to the rescue

[marcansoft]

That was exactly my point - I was referring to SD and other Flash memory external interface standards, not the underlying NAND chips. At the SD interface level, sectors are 512b most of the time (or always, for SDHC), but the physical NAND devices are 2K/4K, so the interface would benefit from using the native sector size.

Re:Open Source to the rescue

[Blakey Rat]

Huh? What are you talking about? Was that intended as a reply to some other post?

The point I was making is that, since Windows has supported these drives for 3 years now, and Linux doesn't yet have the same level of support, obviously open source development isn't as fast this thread claims.

What the hell does the usage of Windows XP have anything to do with that point?

Re:Open Source to the rescue

[mcrbids]

Actually, the beauty of closed source is that the OS "supports" it out of the box, except that it's buggy as all get out, works very slowly for stuff that was much faster in 1995, and despite many users noticing and complaining about the problem both to the vendor and in various blogs and online forums, it doesn't get fixed for months or years while a qualified dev is finally diverted when the problem is so severe that even the US Army won't buy until it's fixed.

Re:Open Source to the rescue

[nhytefall]

Poster is exactly correct, please mod parent up, and mod GP down.

Re:Open Source to the rescue

[Sir_Lewk]

I'm asserting that for all intents and purposes, Microsoft has not solved this problem. If they had then the drives wouldn't need to fake 512 sectors.

Re:Open Source to the rescue

[guruevi]

Ya know in the olden days, when I was young (love saying that) we didn't have hardware that configured itself so it would work on all platforms. We had to put in settings with jumpers and do low-level disk formats through the BIOS or a boot-floppy and WE LIKED IT (seriously).

These days all ya new-fangled hardware doesn't have to worry about being a master or a slave, getting 5V or 3.3V to the PCI bus or RAM modules, CPU multipliers on the motherboard.

I would simply do the same - get a jumper on the back of the drive that says 512 or 4k - we left it on 512 for ya because we assume you numbnuts still use Windows (XP) but if you want performance and use anything but DOS or WinDOS feel free to switch it. You can then reformat the drive.

Re:Open Source to the rescue

[Blakey Rat]

But they have solved it.

They haven't invented a time machine to solve the problem in an OS that came out years before the problem existed. Is that what you're complaining about? The lack of a time machine?

Re:Open Source to the rescue

[Sir_Lewk]

Solving it is worthless if they can't convince anybody to actually use the fix. I'm complaining about their complete inability to get people to upgrade, so we don't have to make workarounds for a nearly decade old operating system anymore.

Re:Open Source to the rescue

[pchan-]

Oh, haha. I misunderstood your meaning. Good show, then.

Re:Open Source to the rescue

[bfields]

We had to put in settings with jumpers and do low-level disk formats through the BIOS or a boot-floppy and WE LIKED IT (seriously).

My memory agreed with yours up to that last bit....

Re:Open Source to the rescue

[argent]

Non-512-byte sectors first showed up with some early CD-R drives, which used 2k blocks. I also had to deal with Sun SCSI drives requiring non-512-byte blocks back in the '90s. Non-512-byte block size didn't magically become an issue in 2006.

Re:Open Source to the rescue

[IgnoramusMaximus]

... and of course Linux 2.4 was also in 2001. None of those OSes know what to do with anything other than a 512B sector -

And...

... Linux 2.6, and Mac OS X 10.5 know full well what to do with a 4K drive.

Total bullshit.

Linux kernel code had flexible block device sector size since the days of 1.x series of kernels. The "problem" is (and always was) with some of the user-space utilities for some of the file systems available under Linux, file systems specifically designed for ... compatibility with DOS and Windows (and through them with the original, ancient IBM PC XT BIOS).

Even then most of the same utilities have various override options that can be used to make them compliant with "unusual" (from the point of view of Windows) sector/block sizes and dive geometries, although it is not their default behavior. The very article you are responding to moans about this very thing, as if it were any news to long-time Linux users.

Windows apologists and their revisionist history are just pathetic.

Re:Open Source to the rescue

[eric-x]

very nice of you to ignore everything that is insignificant and tell them to drop dead when they want to replace the failing HDD in their antique system. Maybe you don't realize but not only PC's use HDDs. There's hardware out there running on obscure firmware and custom chips which cannot be simply replaced by a new box and a OS upgrade.

Re:Open Source to the rescue

[marcansoft]

An antique system is going to run into several other issues if you try to drop in a newer HDD, among them CHS->LBA, LBA->LBA48, and IDE->SATA. Not to mention that we're talking about new drives here - starting to manufacture 4KB drives doesn't mean ceasing to manufacture 512-byte drives. Right now the Windows XP issue means no one can get any true 4KB drives at all.

By the time replacement components are no longer available at all, the remaining die-hard users of antique hardware will be creating adapter hardware that can do the necessary conversion and 512-byte-sector emulation. And since this antique hardware will invariably be much slower than the drive anyway, the speed overhead will be negligible.

Re:Open Source to the rescue

[alantus]

They could always release a hotfix/patch/service pack to fix it.

Re:Open Source to the rescue

[MoHaG]

SD cards in the > 1GB - 4GB range have larger sectors... Not sure if that is visible to the OS:
http://en.wikipedia.org/wiki/Secure_Digital#Standard-SD_cards_.28non-SDHC.29_with_greater_than_1_GB_capacity...

Re:Open Source to the rescue

[marcansoft]

That's a nice idea. You know, some drives already do this for 128GiB limiting and SATA 1.5Gbps mode forcing, but even there some crappy Western Digital drives have ditched jumpers: they'll let you force 1.5Gbps SATA speed mode, but you have to use a program to change the mode! That means if you don't have a supported motherboard you need to find another computer, plug in the drive, boot into DOS from a CD, and hope their crappy DOS tool supports your SATA hardware.

I fully agree, I liked jumpers too.

Re:Open Source to the rescue

[marcansoft]

All CD-ROM drives use 2K sectors, but the issue is mitigated because people don't use regular HDD filesystems on CD-ROMs. DVD drives actually use 64K sectors or something along those lines, but they emulate 2K sectors for legacy reasons (though at least here you don't have read-modify-write issues).

Re:Open Source to the rescue

[marcansoft]

That's weird. I do know that non-SDHC cards can have larger sectors, but I didn't expect it to be a requirement for the 1GB-4GB range. Non-SDHC SD card addressing works on byte offsets (so effectively all block addressing it multiplied by the block size, even though it has to be block aligned anyway), so using larger blocks provides no benefit (it's not like ATA/SATA HDDs where blocks are addressed in units of, well, blocks). In fact, all SDHC did was change the addressing to block units and fix the block size at 512 bytes (besides some initialization sequence changes), so SDHC cards can handle up to 512B x 4G = 2TiB (they currently limit it at 32GiB per spec, but that's not a technical limit).

To answer your question, whether it's visible to the OS would definitely depend on the card reader. It will definitely be visible with raw SD Host Controllers (built-in PCI readers and the like), but USB readers can do all sorts of weird stuff.

Re:Open Source to the rescue

[yuhong]

the remaining die-hard users of antique hardware will be creating adapter hardware that can do the necessary conversion and 512-byte-sector emulation.

In fact, that is exactly what the WD Advanced Format drives are doing right now to be compatible.

Re:Open Source to the rescue

[marcansoft]

No, they're unconditionally crippling themselves in detriment to the OSes that actually could support the newer technology. What WD isn't doing isn't just backwards compatibility, it's decreasing the value of a new feature and increasing confusion by pretending not to have that feature while having hidden performance issues when OSes (rightly) treat it like a normal drive.

The Right Way to do it would be to have a jumper to enable and disable the emulation. Failing that, just default to 4KB mode and let the market come up with 4K -> 512byte emulation adapters if people truly demand them.

Re:Open Source to the rescue

[fuzzywig]

"Windows apologists and their revisionist history are just pathetic." I believe there's a saying about ignorance and malice and their relative occurrences which is of relevance here...

Re:Open Source to the rescue

[Freultwah]

Said drives already have a jumper setting for "XP compatibility". Just bridge pins 7-8 with a regular jumper and you're good to go. I keep wondering why no-one even has even bothered to google it.

Re:Open Source to the rescue

[jesup]

512 byte sectors were known to be a performance bottleneck LONG before the late 90's. At Commodore, I put full support in for any power-of-two HD sector size back around 92-ish (plus the FS supported allocation blocks of powers-of-two multiples of sectors, which most filesystems have for quite a while). Larger sector sizes was a win on performance, much the way larger allocation sizes in the FS was a big win, though more related to low-level overhead. Certainly on the platters larger sector sizes is a win - we experimented with that for floppies, and used a floppy format that got much of the benefit (removal of all inter-sector gaps, replaced with a single gap per track - you lost the ability to write single sectors, but got 10% more storage and faster transfer times - effectively one big sector, but with additional sync/etc headers that let you read data starting at any logical sector, to reduce rotational latency). Similar logic applies to HD platters - larger sectors lets you get more user bits on a track and higher transfer rates, but decreases the granularity for writing data.

Re:Open Source to the rescue

[MarcQuadra]

It's not even in the kernel, so it doesn't matter when the kernel was written. The problem is in userland, the partitioning utility in particular.

'util-linux' hasn't been updated since 2005. How much you want to be your Linux distro is using 'fdisk' from five years ago?

Luckily, someone got busy today and updated the util-linux-ng 'fdisk' to start partitions at 1MB, unless the device is super small and each megabyte matters.

http://git.kernel.org/?p=utils/util-linux-ng/util-linux-ng.git;a=shortlog;h=refs/heads/master

Now all we have to do is have the distros pick it up.

Problem solved. It probably took less than ten minutes of coding. Arguably, it should have been solved years ago. I know I've been aligning my partitions to 64K or 1MB bounds for four years (I got hip to this stuff when I started virtualizing things).

Re:Open Source to the rescue

[orngjce223]

Nope, they're using this issue (one among many, really) to get people to Upgrade(tm). (Can't use that cool new drive you bought? Sorry sucker, go upgrade to Windows UltraVista9 for just $599 a pop!) I'm sure you understand what I mean.

Re:Open Source to the rescue

[badkarmadayaccount]

There are EFI versions of XP you know.

Re:Open Source to the rescue

[badkarmadayaccount]

Different sector sizes? Way cool. What brand and model is it?

Re:Open Source to the rescue

[Pentium100]

Seagate ST34520N

Seatools Enterprise, the software from Seagate allows changing the sector size and changing capacity (to lower than original). Here is a screenshot: http://img40.imageshack.us/img40/2497/blocksize.png

The software also allows changing of various drive settings (automatic bad block reallocation on read and write, read and write caches, number of cache segments). Some settings are grayed out (performance mode desktop/server and acoustic management) presumably because my drive is too old and does not support them. The software only works with Seagate SCSI drives (if you select a non-Seagate or a very old drive it can only perform a drive test, basically verify every single sector).

Re:Open Source to the rescue

[badkarmadayaccount]

Thanks. That's some beautiful software. I wonder if there are modern versions or equivalents...

Re:Open Source to the rescue

[f8l_0e]

Setting multipliers on the motherboard wasn't too bad. If you forgot to set the split core voltage jumper on a socket 7 board when you had a pentium with mmx, now that was a problem.

I don't miss those days much. My life has been much better since I haven't had to worry about IRQ conflicts, setting I/O ports, or fscking with emm386.

So don't do that...

[russotto]

Author claims a massive performance drop if things aren't aligned right. Ubuntu already does it with parted and fdisk can do it manually. So, no big problem; fdisk ought to be fixed to have sane defaults with a 4096 byte block size, sure. That can't be all that difficult.

The author also seems to think that only a 30% increase in times for misaligned writes should be expected. I'm not sure why. In a naive implementation I'd expect a 100% increase in time (each block now needs to be written twice). Linux, obviously, doesn't use a naive implementation. It's expected that if the hardware violates the assumptions behind the techniques Linux uses to achieve high performance, that those techniques end up making things very slow instead.

Re:So don't do that...

[marcansoft]

fdisk doesn't need to be fixed, it needs to be deprecated. DOS partition tables are a ridiculously bad artifact of the past. We won't be using them for much longer anyway; they're limited to 2TB for 512-byte-sector drives (or 4K drives with 512-byte emulation).

Re:So don't do that...

[Dragoniz3r]

Methinks you're confused about something. fdisk doesn't have anything to do with DOS partition tables. fdisk manipulates the MBR, which is required to locate partitions (of any sort) on the disk, unless you're using an OS/BIOS combination that supports GPTs, which is hardly widespread at this point.

Re:So don't do that...

[marcansoft]

The MBR is the DOS partition table. The BIOS has nothing to do with partitions, as it just launches the code located in the MBR (which is in charge of deciding which partition to boot). GUID Partition Tables are supported by Linux, which is what we're talking about here, so there's no reason not to use them unless you also require a legacy OS.

You seem to think that DOS has some sort of underlying level of partitioning that we no longer use. That's not the case - the MBR partitions used today are still the same old craptacular DOS partitions (with LBA extensions tacked on), and they are managed by fdisk.

If you ever want to use a hard drive over 2TB, the DOS MBR partition table format will not work. You'll have to use GPT or you'll lose all capacity above 2TB. Well, maybe with some hacks (which no doubt will not be well supported by some/most OSes due to 32bit math issues) you'll be able to cram in two partitions of 2TB size each (maximum), but I doubt that'll catch on due to the aforementioned OS support problem, and there's no way of breaking the 4TB barrier with the current format.

Check with your distribution

[macemoneta]

I know that Fedora seems to have addressed this with parted 2.1.1 and util-linux-ng 2.1. Both are scheduled for Fedora 13, but can be pulled into Fedora 12 by those getting the hardware early.

Partitions are obsolete

[Anonymous Coward]

Easiest fix: stop dividing your disks into partitions.

Re:Partitions are obsolete

[jedidiah]

Nope. Read the original article. The first partition will be mis-aligned if you use the first available block rather than padding to the next 4K boundary.

You won't even know it unless you look at the partition table in expert mode.

Re:Partitions are obsolete

[hedwards]

Which is nice if you're wanting to ensure that you've got the lowest possible reliability and safety for your data. While you're at it, make sure you're using a striped non-redundant array of disks as well, best use at least 4 in the array, otherwise you might get some of your data back.

You've got it exactly backwards, people shouldn't be partitioning disks into one huge partition. They should be able to split things up a bit to keep rapidly changing directories from mostly static ones and to manage the risk of filesystem corruption destroying important files.

Re:Partitions are obsolete

[marcansoft]

What the GP is advocating is not using a partition table, instead of partitioning drives into one partition. This doesn't cause the issue mentioned in the article. This isn't a novel idea - there are some USB drives that do this already, and even Windows seems to cope fine.

Re:Partitions are obsolete

[HiThere]

And then when you install a different distribution, you blow away your home directory. Sorry, bad idea. /home should be in a separate partition from the rest of the stuff..

Also, since I usually have several distributions installed at the same time, I have several partitions...but that's a less common problem.

A better solution would be to have a boot partition snuggled up against the MBR that automatically adapts so that the boot + MBR is an appropriate size, say 32 MB. (My current boot directory is 14MB, so that shouldn't be a problem. These aren't, after all, small drives, so it doesn't hurt to allocate a bit of extra space. Maybe even make that 64MB.)

Perhaps one could rearrange the system tables a bit so that the MBR was counted as a part of the /boot partition, and so was the partition table. They'd need to be an a position guaranteed by the OS, but that's not a real problem.

Note that what I'm proposing is a major redesign, so there's about zero chance of it being adopted. But it's a better choice than scrapping partitions, and probably has a better chance of being adopted.

Re:Partitions are obsolete

[Anonymous Coward]

Splitting a disk into multiple pseudo-disks makes sense in many situations, but the clunky legacy partition tables are only good for inter-OS compatibility. Otherwise LVM beats partitions in every respect. Now if only we could get a LVM solution that works in multiple operating systems...

Re:Partitions are obsolete

[BitZtream]

If your worried about one big filesystem going south then I suggest you start using a modern filesystem without those concerns.

We've got well past the point where that should be an issue on any modern system.

Your acting like it's still the 70s, it's not, we've learned a few things since then.

Re:Partitions are obsolete

[Anonymous Coward]

I still do not use LVM on my special purpose workstations and servers because it does not provide the equivalent of MD over partitioned disks. With MD over partitioned disks, I am able to maintain multiple RAID5 or RAID6 volumes across the same disk set, and viably repair and upgrade the hosts with minimal downtime, including replacing disks with larger ones and/or resizing the RAID volumes as needed. For some reason LVM seems to be taking an incredible number of years to adopt any RAID levels beyond mirroring.

Not only does MD over partitions allow for LVM-like management of smaller single-purpose filesystems from a large pool, but it also allows different MD volumes on the same disk set to have different RAID modes, e.g. mirrored /boot, RAID5 or 6 / and /home, and RAID0 /scratch for high performance jobs.

Oh slashdot..

[JeffSh]

Dear Slashdot,

I've been around for a while. Enough to understand, nay, love the fact that you are linux supporters and all that. But I remain an ardent supporter of truth and speaking in ways which are concise and leads the reader in the direction of truth. Nothing in this news story is inaccurate, but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.

Windows XP may not work with the new WD Green drives, but Vista and on have been perfectly comfortable with 4096 byte sectors. A lay reader may read this story and not "Read between the lines" as I have learned to do here. Their take away may be that Microsoft operating systems are broken in some way (which they are in a lot of ways), but not this one!

Re:Oh slashdot..

[rubycodez]

nonsense, I guessed (correctly it turns out) from the description that Xp and older versions of windows would have problem. But those of us who keep older computer around for windows, with no interest in going beyond Xp, aren't fretting.

You have an imagined stereotype that isn't true. Here's my stereotyping, Less than 0.1% of readers of slashdot would be "lay readers" in matters of computers so you needn't worry.

Re:Oh slashdot..

[hduff]

Dear Slashdot,

Nothing in this news story is inaccurate, but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.

You are correct. It is a non-issue because nobody uses WindowsXP anymore and there are no legacy systems running it. Nor will there ever be a need to use a new drive in a WindowsXP stsem since there are no WindowsXP systems in use. You are correct.

Re:Oh slashdot..

[edxwelch]

Maybe Vista and 7 support it, but 70% of all PCs are still using Windows XP.

Re:Oh slashdot..

[Vellmont]

  but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.
.
.
Their take away may be that Microsoft operating systems are broken in some way (which they are in a lot of ways), but not this one!

It only takes about 3 brain cells to realize that Windows XP != All Microsoft Operating Systems. Even the average person has more than 3 brain cells.

For those people with less than 3 brain cells, Slashdot still has you covered since the article clearly says:

According to WD, their new 4096-byte sector drives are problematic for Windows XP users but not Linux or most other OSes.

(emphasis mine)
It only takes 2 brain cells to understand that "most other OSes" likely includes Vista and Windows 7.

For those unlucky few with only 1 brain cell, you're correct. Slashdot has certainly failed the 1 brain celled individual.

Re:Oh slashdot..

[maxume]

It isn't really even a problem under XP (as long as the drive is formatted with a tool that is aware of the new sector size, so that the logical sectors of the operating system line up with the physical sectors of the drive; Western Digital, the crazy fools, have a tool that will make sure this is the case...).

Re:Oh slashdot..

[migla]

When the summary explicitly mentions only XP, then obviously all others after that are fine.

Sure, if this was "Gramps weekly Gramps-thing" papyrus gazette, that for some reason had a story about these new drives, then yes, a clarification that Vista and so forth work ok (and an explanation about what the heck a hard drive is in the first place) would be in order.

Re:Oh slashdot..

[Bengie]

now xp is 62% and Vista(20%) and Win7(10%). So about 33% of Windows OS's are 4k aware.

Re:Oh slashdot..

[hduff]

Why does sarcasm get modded as flamebait? Or did it just go over your head?

I just bought one of these

[xhorder]

Can some kind soul tell me specfically what version of what utility I need to use for me to be OK? Or what settings?

My head hurts from trying to understand cylinders and sectors and drive geometry...

thanks!

Re:I just bought one of these

[King Kwame Kilpatric]

The problem is that WD doesn't tell the system about the sector size.
dev/sdd:

Model=WDC WD15EARS-00Z5B1, FwRev=80.00A80, SerialNo=
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=16

It looks to me that this should *really* be fixed by WD with a firmware update

.

Solution: Instead of fdisk, call it as fdisk -H 224 -S 56 as per Theodore Tso's blog.

Re:I just bought one of these

[King Kwame Kilpatric]

Also see this post.

slashdot is not journalism

[SuperBanana]

should be an embarrassment of journalistic integrity.

Slashvertisements, basic English grammar and spelling problems, completely wrong summaries and titles...

...and you a)think that Slashdot is "journalism" and b)it's had integrity to lose in the first place?

I like Slashdot, but gimme a break...it's a user-driven blog which directs readers to existing stories (now often lagging behind the major news wires) with good categorization and semi-sophisticated commenting system, utilized by a larger commenter population. Not much more, and definitely not journalism.

Re:slashdot is not journalism

[JeffSh]

Yeah, I don't disagree, maybe I'm even well aware of that, but if I had admitted that in my original post I'd have no room to indict the continued problems. I think they should aspire to be better even if Slashdot is everything you say it is.

Re:slashdot is not journalism

[Blakey Rat]

I'm with you, but on the other hand that doesn't mean they should just not give a shit about the quality of their end-product. We know from experience that they can edit and correct stories as corrections arise in the comments, but how often does that happen in practice? (Hardly ever.) Somewhere between a third and half of the stories posted here are either outright lies, or extremely misleading-- I may be exaggerating, but not by much-- and almost never are they corrected.

Look, any site that posts this article: http://tech.slashdot.org/article.pl?sid=09/02/16/2259257 without a single correct simply Does. Not. Give. A. Shit.

I don't think anybody's expecting the New York Times when they visit here, but some minimum level of competence would be nice. I don't fault anybody for complaining.

Re:if vista/win7 really do support this correctly.

[walshy007]

The real problem is that it is lying about it's sector size, it's reporting 512 bytes when it's using 4k, if it told linux it was using 4k everything would be fine and dandy.

Why does it lie about it's sector size when it doesn't need to? because if it didn't the drives would not work on windows XP at all. Which would not bode well for sales.

Once drives with 4k sectors arrive its up the individual maintainers of each affected tool (fdisk, et. al.) to update their code.

Kernel handles sector sizes, and could handle 4k sectors ages ago, but when the hardware reports something it tends to trust it, which is now apparent it shouldn't. (512 byte sectors being implemented as an emulation layer of sorts on these drives.. and enabled by default)

Drive lies and future fixes

[Sits]

There is an excellent thread talking about how recent (2.6.31+) linux kernels try to report the underlying hard drive architecture (found via the OSNews comments). Alas, it looks like some of these drives are not reporting this data correctly and thus automatic adjustment (at partitioning time) is not taking place. It looks like in the future rather than trying to do detection by reported capability fdisk (and hopefully gparted) will default to sectors of 1MiB if the topology can't be found by default (unless your media is small).

Additionally, I gather that recent Fedoras will try to adjust things like LVM to match larger sectors too. Hopefully whatever is laying out LVM will also be fixed too.

Coincidentally, it looks like Oracle have a very committed dev trying to make this stuff work by default...

This only effects the newer 1TB+ WD Green drives

[HouseOfMisterE]

It appears that this does not effect the older 1TB+ Western Digital Green drives such as the WDC10EADS. Those use 333GB platters and are native 512-byte sectors. The newer (newest) Western Green drives, like the WDC10EARS, use 500MB platters and have 4K sectors. One way to tell the drives apart with a quick glance is the old Green drives had 32MB of cache and the new ones have 64MB of cache.

Re:if vista/win7 really do support this correctly.

[ArghBlarg]

I see it rather as an indictment against closed-source OSes, if XP turns out to be incompatible with these new drives and MS never releases a patch to add support. People will need to upgrade for no good reason to one of MS's new operating systems. People should not have to deal with a complete upheaval of their tested and true systems due to a small hardware change such as this.

I can imagine MS is quietly chuckling with glee to itself, if this issue becomes a deal-breaker for machines still running XP.

A minor correction...

[HouseOfMisterE]

I posted that the newer WD Green drives use 500MB platters and I meant 500GB. 500MB platters would make for a very physically-large 1TB+ hard drive!

I was worried about this... and am still unclear

[bmajik]

I just got one of the 1TB 64mb WD drives that is known to be 4kb sector based.

Here is how it shows up in dmesg:
[ 3.420488] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)

and here's what hdparm -I says:
ATA device, with non-removable media
Model Number: WDC WD10EARS-00Y5B1
Serial Number: WD-WCAV55227529
Firmware Revision: 80.00A80
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 1953525168
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 953869 MBytes
device size with M = 1000*1000: 1000204 MBytes (1000 GB)
cache/buffer size = unknown
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 1
Recommended acoustic management value: 128, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_B

I have such a problem

[orkysoft]

I have a tiny 1.8" usb harddisk with 4096-byte sectors, and the Ubuntu installer crashes when it tries to read the partitioning information. Very annoying.

DragonFly's solution

[m.dillon]

We're adjusting our disklabel64 utility and kernel support to set the partition base offset such that it is physically aligned instead of slice-aligned, and we are using 32K alignment. That should fix the problem without having to mess around with fdisk.

The DragonFly 64-bit disklabel structure uses 64-bit byte offsets instead of sector addressing to specify everything. It ensures things are at least sector aligned but we wanted to make disk images more portable across devices with potentially different sector sizes. The HAMMER fs uses byte-granular addressing for the same reason, 16K aligned.

-Matt

Re:DragonFly's solution

[bertok]

We're adjusting our disklabel64 utility and kernel support to set the partition base offset such that it is physically aligned instead of slice-aligned, and we are using 32K alignment. That should fix the problem without having to mess around with fdisk.

The DragonFly 64-bit disklabel structure uses 64-bit byte offsets instead of sector addressing to specify everything. It ensures things are at least sector aligned but we wanted to make disk images more portable across devices with potentially different sector sizes. The HAMMER fs uses byte-granular addressing for the same reason, 16K aligned.

-Matt

You should use 64K alignment at a minimum, almost all RAID and SAN volumes are 64K aligned with IO sizes of 64K minimum.

You'll lose up to 50% performance on random IOs on any server class hardware.

As a comparison, Windows now aligns to the nearest 1MB.

Re:DragonFly's solution

[Just Some Guy]

We're adjusting our disklabel64 utility and kernel support to set the partition base offset such that it is physically aligned instead of slice-aligned, and we are using 32K alignment.

Darn bloated OS wasting 2% off the front of my floppy drive.

partition table

[gd2shoe]

There won't be a partition table with his suggestion. The boot sector set aside by the filesystem will be the very first sector of the disk.

Poorly researched article.

[Vellmont]

The article represents one data point, for one particular way to install a drive, on one (un-named) version of Gentoo, on one particular model of a WD drive that had a bugzilla entry entered by the author all of 2 days ago. So this is supposed to be an indictment of all of Linux?

The author even mentions that Ubuntu has an option on parted that accomplishes the task properly. I'd be much more interested in an article that talks about how the default installer handles this task rather than concentrating on one particular expert tool that does so. It's still good to know that fdisk on his un-named Gentoo distribution does the wrong thing.. but this hardly means we should fire up the klaxon and declare "Linux not fully prepared for 4096 sector hard drives!". It's certainly interesting, but I'll withhold judgment until we actually know more about the implications of this across the entire spectrum of Linux distributions and the various 4096 sector HDs.

Re:Poorly researched article.

[Radtoo]

I agree with the headlines being grossly misleading. Linux does support 4k block sizes just fine. But this is not a distro-specific issue, so you are wrong, too.
This is simply a matter of fdisk from that version of util-linux-ng (which is clearly named in the article) trusting the hardware vendor to specify correct block sizes. The vendor did not. Thus fdisk does not end up with 4k block sizes, as happens for many programs. And only(?) parted apparently contains a workaround that detects the correct block size.

Its not that you can't use parted on Gentoo, though, it is just that in the world of user choices that is Gentoo, not everyone will be using that program or that particular option.

Re:Poorly researched article.

[Vellmont]

But this is not a distro-specific issue, so you are wrong, too.

I never made any claims about this being a distro-specific issue, or not being a distro-specific issue. The only point I'm trying to get across is the article is extraordinarily narrow in what it's actually tested.

Re:Poorly researched article.

[Vellmont]

Nice to see you criticizing people for "poor" research when you hardly done any yourself.

Why is it some people just don't seem to understand the concept of arguing for not knowing something? I DID do research, I read the article quite carefully and determined the guy didn't take into account multiple different things, and that his claim was far too broad. If you can't understand that that involves research and analysis, I can't help you.

Gentoo has no versions. Nor cutesy names like Ubuntu. It is a source-based distro and everything is compiled on installation so doesn't need this careful versioning nonsense.

For a distribution that you claim doesn't have any versions, they sure do have a lot of announcements claiming new versions: (from distrowatch)

2009-10-05: Distribution Release: Gentoo Linux 10.0
    2009-09-27: Development Release: Gentoo Linux 10.0 Test
    2008-07-06: Distribution Release: Gentoo Linux 2008.0
    2008-04-29: Development Release: Gentoo Linux 2008.0 Beta 2
    2008-04-01: Development Release: Gentoo Linux 2008.0 Beta 1
    2007-05-07: Distribution Release: Gentoo Linux 2007.0

It is not just one version of fdisk on some backwater 15 year old distribution

Funny, I don't recall making any claims this was one version of fdisk. If I recall I only said the article referenced one version.

Re:Poorly researched article.

[Theovon]

I wrote the linked article.

I completely agree that the article is narrowly focused. VERY narrow. My objective was to demonstrate a problem and point out that Linux has not FULLY adapted. I didn't say Linux devs were idiots or that it would never be ready. I was trying to express the idea that Linux [distros in general but perhaps not all] is not QUITE ready for these drives, because not all the tools have fully adapted. Some tools make no mention of any problems in their man pages. Some (like parted's defaults) are even misleading if you mistakenly think that "track aligned" is a good thing.

And I was trying to do that in the very limited number of words I had available for a title.

Also, WD claimed that Linux is unaffected. Some distros probably are, but this could lead people to believe that the statement is universally true, which it isn't. Thus, my over-all objective is to educate people to the fact that if they don't know what they're doing, they can get this wrong. There are lots of mistakes I've made where I wished that someone had mentioned some critical fact on a how-to (like, don't use dmraid/fakeraid for RAID1 because reads aren't load-balanced; use mdraid instead). I've filed plenty of bug reports on such issues.

Re:Poorly researched article.

[Vellmont]

I can't over-emphasize the importance of titles in communication, especially with complex technical subject where there's a lot of evidence presented to support a conclusion. Your title colors the rest of the article and creates expectations about what you're trying to say. When people read articles (especially on the web) they scan through them trying to find the important parts. That's been demonstrated through eye-tracking studies multiple times.

Your title was very broad, but the evidence to support it was very narrow (as you admit yourself). Since your article only referenced Gentoo, and installing a drive after installation using fdisk, why not: "fdisk not ready for 4096 byte sector drives?" If you wanted to cover all of "linux" (whatever that is), why not research other distributions and see how they handle the job, installing the OS flat out on several distro's and see how they perform. If you didn't want to do the research, you could have written in an inquisitive style, i.e. "Is linux ready for 4096 byte sector drives?" and then presented your evidence and talked about how fdisk didn't do this, but parted worked a bit better, and how the drives reported 512 sectors but weren't, etc, but you've only tried Gentoo, and maybe others should try other distributions with other tools.

I think the experimental aspects you did were great, and I'm glad to know that some of the tools out there don't fully work with this particular 4096 byte sector drive. I just don't think you've done enough research to say much more than the problem affecs Gentoo, on these specific WD drives.

Re:Poorly researched article.

[jrumney]

one particular way to install a drive, on one (un-named) version of Gentoo, on one particular model of a WD drive that had a bugzilla entry entered by the author all of 2 days ago

It's the default way to install a drive on just about every Linux distribution out there. I bought one of these drives unsuspectingly last week to replace a drive which had died, and installed Ubuntu 9.10 on it. Sure enough, my root partition starts on sector 63, and every other partition is also on an odd boundary. So is there any utility for Linux that can do what the Windows utility alluded to in TFA does and shift all my painstakingly restored data by 512B, or am I stuck with reformatting again and restoring from backup yet again?

Simple Solution

[Khyber]

Don't partition the drive in XP - format the entire thing and don't split it apart. Get a secondary physical drive.

Re:if vista/win7 really do support this correctly.

[earthforce_1]

I wouldn't be too fond of the MS development model from what I hear from those who were on the inside:
http://www.nytimes.com/2010/02/04/opinion/04brass.html?pagewanted=all

Inside Microsoft, political infighting trumps common sense. If you really want to hold up a closed source development model as an example of "what works" take a look at Apple. They crank out far better products with a fraction of the resources.

Re:if vista/win7 really do support this correctly.

[thisissilly]

It seems these drives need a new "don't lie to me, I can handle it" command, so OSes that don't have a problem with 4k size sectors can get the real info.

Re:Other possible consequence of misalignment

[King Kwame Kilpatric]

I noticed both a performance hit AND stalling for a minute at a time when there was a lot of HDD activity, so I can confirm part of your experience. After going with 56 sectors per track as well, the freezing seems to be a thing of the past. The speed is definitely greatly improved.

Re:This only effects the newer 1TB+ WD Green drive

[King Kwame Kilpatric]

The affected drives are listed on Western Digital's site.

Re:Other possible consequence of misalignment

[King Kwame Kilpatric]

I'm happy to see that you saw the stalling disappear as well, it gives me confidence about the cause.

Time will tell if the the lock-ups were caused by the 512b partitioning (I wasn't able to repartition my drives until a few days ago), but it's a bit reassuring to know that I wasn't the only one experiencing these annoying as fsck freezes.

The affected drives on my end are WDC WD10EACS-00ZJB0 and WDC WD10EADS-00L5B

.

Re:I was worried about this... and am still unclea

[King Kwame Kilpatric]

(I think that) It's probably misaligned. LVM uses a 192k sector size for it's metadata. See Theodore Ts'o's post for more information.

linux is not ready because of fdisk?

[pizzap]

fdisk an elegant tool for a more civilized age.. no wait. fdisk is antiquated and we only use it, because we are afraid to leave the msdos partition table behind out of the irrational fear some other software would stop working.

not all 4K are created equal

[dltaylor]

There are four flavors of 4096 byte-sectored drives:

4096 physical/logical - the bookkeeping parts of the file system cause read/modify write cycles because they are nearly always less than 4096 bytes, but the performance hit is relatively small; parted is badly broken. If they're less than 2TiB, then you can use an MBR, otherwise the kernel is broken for partition sizes.

4096 physical/512 logical; LBA 0 aligned "off by one" with physical block 0 - created to deal with stupid BIOS (and Win XP, where some drivers rely on it), mostly work fine with the default tools, but still have the bookkeeping issues. Because Win Vista/7 and OS X use GPT AND don't worry about "track" boundaries, they work better than Linux.

4096 physical/512 logical; LBA 0 aligned with physical block 0 - works great with Win Vista/7 and OS X, but the Linux installers are still aligning on the bogus track boundary, and not asking the physical/logical alignment. Performance, without some very smart tweaking by the person doing the formatting REALLY stinks.

4096 physical/512 logical, but are reporting 512 physical (usually aligned 0 for 0) - again to deal with BIOS/Win XP. Basically, treat ALL drives produced starting with 2010 as having 4K sectors, aligned 0 for 0, unless they explicitly report otherwise, and use the same human-intervention-required layout as above.

Currently, the tools are the most pressing issue, since they are really broken in this respect, but there are kernel issues, as well, with drives larger that 2TiB and 4096-byte sectors.

All of a sudden?

[Waccoon]

I had never ever heard of drive alignment until I bought an SSD.

Not to be unhelpfully pessimistic, but... didn't it even occur to people that drive alignment might be important? Is 4K drive alignment just Y2K for hard drives? Why did people only start thinking about this now?

a firmware update isn't realistic

[YesIAmAScript]

I'll get to why in a second, but first:

RawCHS hasn't meant anything in a decade. The largest drive you can describe with CHS is 8GB.

Track size hasn't meant anything in even longer than that. When drives went to zone bit recording (ZBR), the number of sectors per track became variable. This happened in about 1989.

The sector size does mean something, but it is the actual sector size, not the sector "grouping" size. If the drive reported a sector size of 4K, then it would expect that the host understand that sectors are actually 4K in size, not 512B in size. But really no major OS supports this, they all expect 512B sectors. That's why these drives internally use one sector size and show another size to the host. And there is no way in the ATA specification for devices to indicate their internal sector size when they are presenting a different external sector size.

So this won't be fixed with a firmware update, unless Vista, 7 and every other major OS is fixed to actually support large sectors presented to the host. Then the drive could be firmware updated to report the large sector size to the host. And the drive would then be completely unusable under any earlier OS or with any USB or Fireware adapter.

whoops, one thing about RawCHS

[YesIAmAScript]

I forgot, there is one thing RawCHS nowadays. That is that there is no proper spec for how to know if a partition in an MBR (fdisk) partition table is a valid partition. So there are heuristics that are applied to the entries to guess if they are real or to be ignored as empty. One of the heuristics that some software uses is to ignore all partition entries that don't begin on a cylinder boundary. To be on a cylinder boundary, the partition has to start on a sector number that is a multiple of the number of sectors (S in CHS) in order to be valid. And since all drives 8GB or greater present an S of 63, that is why the first partition on an MBR disk has always started at sector 63, which makes it unaligned when the internal sector size is 4K (8 internal sectors).

Windows before 2000 checks the CHS alignment of MBR entries and ignores any partition entries that don't start on a multiple of S. So all disks out there are misaligned. With Windows 2000 or later, you can start the partition on any boundary you want.

Western Digital has a jumper you can put on the drive that adds 1 to all access requests, making all those misaligned first partitions aligned. But it'll also make any aligned partitions misaligned. So the real answer is just to layout your disk different. I would recommend using GUID disk partitioning instead of MBR anyway, because MBR doesn't work for >2TB drives. And GUID doesn't have any weird alignment requirements (and doesn't have any knowledge of CHS).

Re:whoops, one thing about RawCHS

[hpa]

The original reason for aligning to track boundaries (a track is a cylinder-head pair) is that the first four sectors of MS-DOS' IO.SYS (IBMBIO.SYS) had to be contiguous and on a single track.

Re:whoops, one thing about RawCHS

[yuhong]

Yep, in the old DOS versions there was even more restrictions.

C/H/S Needs to go

[LostMyBeaver]

Cylinders, head, sector addressing really needs to go anyway. MBR has been hacked so many times over the years to support this method of formatting but in reality it's a total waste. Old operating systems that were forced to work with this method of addressing had to translate to and from the CHS format since sector allocation has ALWAYS been linear. Even FAT-12 used linear addressing methods.

CHS is also misleading and CHS optimizations are wasteful. Since all modern drives (starting with the first Connor IDE 20 meg drive in the early days) supported some form of intelligent sector remapping that would keep spare sectors available for relocating data after the magnetic medium of a heavily used sector elsewhere began to fail.

Sector remapping makes it so that CHS optimizations are entirely irrelevant since even brand new drives, straight off the production line ship with bad sectors that have been remapped elsewhere. For better drive performance, algorithms space out the spare sectors across the drive so that when accessing a spare sector, the head doesn't have to slam to the inner or outer rings of the disc. But still, CHS doesn't apply to absolute positions anymore.

This is 2010 now, I wrote file systems which functioned on 4096 byte sectors on ESDI drives back in the 80's. Made my drives much bigger doing it too. It's time that we move to larger sector sizes again. Modern ECC isn't that much better than 80's grade, however the processing power available to us is so much more that performing ECC on larger blocks of data is achievable. Also, using RAID-5, 5EE or RAID6 makes it so that we can depend less on single drive redundancy. SCSI and IDE should be extended so that controller can inform the drive of bad sectors it finds when performing RAID XORing.

WD has a solution

[a-zA-Z0-9$_. !*'(),x]

http://www.wdc.com/en/products/advancedformat/

WD Align software aligns partitions on the Advanced Format drive to ensure it provides full performance for certain configurations.

Re:WD has a solution

[King Kwame Kilpatric]

http://www.wdc.com/en/products/advancedformat/

WD Align software aligns partitions on the Advanced Format drive to ensure it provides full performance for certain configurations.

Yes, certain configurations: XP. According to WD, Linux is not affected by this (but that's been proven to be false).

Re:if vista/win7 really do support this correctly.

[PsychicX]

Yep, right until the NEXT feature comes out that breaks the OS that says it can handle the truth. Now you need TWO "don't lie to me" commands. Repeat ad infinitum.

Doesn't need a kernel tweak at all.

[MarcQuadra]

All that needs to happen is a very minor update to the 'fdisk' utility to make it start partitions at 1MB. Microsoft saw this coming and changed the partitioner in Vista (and newer) to align on 1MB boundaries.

There don't have to be kernel tweaks. I've been doing this myself for years when I install Linux or Windows. Windows XP's 'diskpart' aligns by 32K, whereas the built-in format in setup from the CD aligns on 31.5K, so just format the drive from BartPE's diskpart utility first. In Linux you just drop into 'expert' mode in fdisk, hit 'x' and tell your partition to start at 2048 sectors (1MB).

Why 1MB instead of 32K or 64K? Because some RAID arrays have stripe sizes up to 1MB, and it just makes sense to 'waste' one meg in order to have alignment work on bounds of 4K, 16K, 32K, 64K, 128K, 256K, and 1MB (all common page or RAID stripe sizes).

Should I be concerned?

[busydoingnothing]

I just bought two of the WD Green 500GB drives to be used in a hardware RAID (Adaptec 2610SA, aka Dell CERC SATA1.5/6ch) on my Ubuntu-based server. I was going to format it in ext3. Will this problem affect me?

Re:Should I be concerned?

[King Kwame Kilpatric]

I just bought two of the WD Green 500GB drives to be used in a hardware RAID (Adaptec 2610SA, aka Dell CERC SATA1.5/6ch) on my Ubuntu-based server. I was going to format it in ext3. Will this problem affect me?

You should be immune. It seems that only >640GB drives have this problem. See the affected drives.