Linux Not Quite Ready For New 4K-Sector Drives

Theovon writes "We've seen a few stories recently about the new Western Digital Green drives. According to WD, their new 4096-byte sector drives are problematic for Windows XP users but not Linux or most other OSes. Linux users should not be complacent about this, because not all the Linux tools like fdisk have caught up. The result is a reduction in write throughput by a factor of 3.3 across the board (a 230% overhead) when 4096-byte clusters are misaligned to 4096-byte physical sectors by one or more 512-byte logical sectors. The author does some benchmarks to demonstrate this. Also, from the comments on the article, it appears that even parted is not ready, since by default it aligns to 'cylinder' boundaries, which are not physical cylinder boundaries and are multiples of 63."

Comments

first misaligned post

[Anonymous Coward]

damnit, obviously since this is not technically the 'first post', my web browser must be misaligned by a post

Parted / GPT

[Anonymous Coward]

I heard using parted and GPT labels instead of MSDOS will optimize it on 4096 byte sectors automatically. Any truth to it?

Re:Parted / GPT

[houstonbofh]

Even if that works for new partition, what about existing ones? These drives have been out a while, and people have been installing software to them... The best solution will be a program that moves and/or resizes existing partitions to align them properly. The question is will it be an updated gparted, or something new? What ever it is, I have no doubt that it will be out for Linux far before it is out for XP. :)

oh great.. i try to make a joke and ...

[decora]

the first time i have ever actually gotten 'first post'... it is when i try to make a joke about not having gotten first post. ya see my first post was supposed to come up like second or third.. it would have been HILARIOUS . .. but oh no in soviet russia, the fates mock you!!!!

Re:oh great.. i try to make a joke and ...

[Anonymous Coward]

And now you miss the "Post anonymously" box explaining it all!

Re:oh great.. i try to make a joke and ...

[ignavus]

the first time i have ever actually gotten 'first post'... it is when i try to make a joke about not having gotten first post.

ya see my first post was supposed to come up like second or third.. it would have been HILARIOUS . .. but oh no

in soviet russia, the fates mock you!!!!

No, what you failed to take into account was: in Soviet Russia, the first post got you.

Set 32 sectors per track

[tchuladdiass]

The simple solution is to set you Sectors per Track to 32. This would make sure that everything is properly aligned (except the first partition, usually /boot, which is mis-aligned by one cylinder).

Re:Set 32 sectors per track

[Brian Gordon]

Sectors, blocks, clusters, cylinders... I hope that as we move to solid state drives, devs have the sense to exorcise these anachronisms from the kernel. We haven't been able to get rid of terminals in the 20 years since they've even existed.. this document is heart wrenching. Try reading it; it'll make you cry to see how deeply the now-irrelevant concept of a terminal runs in Linux.

Re:Set 32 sectors per track

[walshy007]

the now-irrelevant concept of a terminal

Speak for yourself sir, I for one like my rs-232 terminals to be handy for when ethernet is down and you can't ssh (and can't be assed hooking up keyboard and monitor). Seriously, anyone adept at the command line uses it far more than the gui to get things done, terminals will never disappear.

Re:Set 32 sectors per track

[jedidiah]

Terminals are only irrelevant if you have a strictly Windows and DOS notion of computing.

The document you cited touches on this a little bit.

Re:Set 32 sectors per track

[rubycodez]

terminals are a very necessary and relevant part of Linux. That's how most server administration is done. That's how sending commands to many network appliances is done. That's how setting up high end computers is done (e.g. set up a midrange Integrity or Superdome and you'll start with terminal on the serial port, whether cu in linux or hyperterminal in windowws or a real terminal). Also how certain tasks are performed in GUI environments. It doesn't matter that the terminal is now mostly virtual, the cursor control and font attribute features make convenient applications possible. Even on the weekend here I am chatting via IRC to some tech friends with irssi in terminal under screen, and reading server status emails with mutt. the terminal, it's 21st century tool.

Re:Set 32 sectors per track

[tagno25]

the now-irrelevant concept of a terminal

Speak for yourself sir, I for one like my rs-232 terminals to be handy for when ethernet is down and you can't ssh (and can't be assed hooking up keyboard and monitor). Seriously, anyone adept at the command line uses it far more than the gui to get things done, terminals will never disappear.

or like on the SheevaPlug, Cobalt RaQ, or Cobalt Qube when you cannot hook up a keyboard and monitor

Re:Set 32 sectors per track

[rubycodez]

but in windows server 2008 and Exchange 2007 the command line and scripting (powershell) is back with a vengeance. Even Microsoft realized dah Powah of dah Toiminal!

Re:Set 32 sectors per track

[0100010001010011]

So why aren't more machines built like the SheevaPlug?

Yet is has console, but it's a USB connection. It has the USBSerial built in. I don't need anything for my Mac other than the cable that came with it.

Re:Set 32 sectors per track

[Anonymous Coward]

terminals have nothing to do with the command line!
i think the op is complaining about the fact that things like
baud, stopbits and whatnot are deeply embedded in the
linux kernel. these concepts are not necessary to
have a command line. c.f. plan 9.

Re:Set 32 sectors per track

[Z00L00K]

Essentially we are back to the old problems of the ST412 interface where we had to figure out the best interleave for the drives as well when we were formatting them. Most drives then did have a fairly conservative interleave, but a reformat of them could improve the throughput considerably. A reformat could be done so that the whole track could be read in 2 rotations instead of 3, and what that does to performance is fairly easy to understand. C800:5 was a commonly used BIOS address where the low level format routine did reside.

But from what I understand this problem is an offset problem when the head steps from track to track, and that's also an issue to be considered. And today it's not common knowledge/practice to low level format hard drives.

And why stick at 4k sectors? Depending on the system you may want to use a different sector size. If you run Oracle on some systems the block size is 8k, and in that case you may want to have 8k disk blocks too since it would be good for performance.

Anyway - sooner or later we will have flash drives instead, and then this isn't a problem.

Re:Set 32 sectors per track

[Brian Gordon]

Yes, thank you. That's exactly what I mean.

Re:Set 32 sectors per track

[amorsen]

Anyway - sooner or later we will have flash drives instead, and then this isn't a problem.

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Re:Set 32 sectors per track

[blincoln]

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Couldn't this be addressed (at least in part) by a battery-backed write cache like better RAID controllers use? Set it up like SAN snapshots (so it just stores the diff between what's in the actual flash storage and what's been changed so far), and then write the changed blocks when it's most advantageous (e.g. when there's an entire block's worth of data, so it would all have to be erased by the flash storage anyway).
Maybe combine that with something like a disk defrag, except instead of storing frequently-sequentially-read data in physical sequence, store frequently-written data (regardless of if it's sequentially-read or not) in physical sequence.

Re:Set 32 sectors per track

[apoc.famine]

Agreed. The GP is an idiot. I was just commenting to a friend of mine a few hours ago how I don't understand how windows users access data on multiple machines. On my unix based machines, I just ssh in, scp what I want, use ssh with X forwarding to access gui tools, etc. We both agreed that most of windows land involves emailing shit to yourself, and a lot of USB thumb drive use. (Of which I could fire off a good half-hour rant on how poorly windows handles mass storage devices. It's a USB THUMB DRIVE for gods sake. It's not a fucking printer! I want to plug it in, and transfer files to/from it. It doesn't need to be "installed", indexed, and have drivers downloaded for it. Just fucking open a file browser like any sane OS does. )

I hit a terminal, and fired off a one-line command to transfer all the stuff I needed from my work machine to this one this morning. Went and got breakfast, and it was here. When I'm done working, I'll fire off another one line command to transfer all changed files to my work machine. Could you do this outside a terminal? I'm sure you could. But I don't know why you'd even bother trying to get that set up. It's going to take as long or longer with a gui most of the time anyway, so why bother waste the time even setting the gui up?

I have a hard time working on windows, because I'm so much more efficient with a terminal. It's not that I can't use a gui - I'm just an order of magnitude faster using the terminal.

Re:Set 32 sectors per track

[kimvette]

The terminal is not irrelevant. If your Cisco router is ever compromised (it happens) or if IOS becomes corrupt (or if you have an IOS install with a nasty bug where the password does not save correctly, or when an IOS upgrade goes badly) or someone fudges the configuration up, the only way you can recover it is often through the serial port. Serial ports are also very handy for integrating video surveillance with point-of-sales systems that are not IP-aware (or worse, antiquated DVR appliances which can't do POS integration over IP), for some smart switches, *NIX boxes that have been rooted (I've rescued a Solaris box through a serial connection in an enterprise environment where reinstall was not possible due to poor timing - week of finals - and backups were sabotaged by a disgruntled gradute student and logins through IP and at the console were blocked), and so forth. However, I'd rather see RS-485 or RS-422 take RS232's place, since RS-485 and RS-422 can work over much longer distances and you can hang multiple serial devices off of a single bus.

RS-232 might be absent from a lot of consumer motherboards, but it is far from dead and certainly not irrelevant, even now in 2010.

Re:Set 32 sectors per track

[Anonymous Coward]

FINALLY, MS has realised that the CL is the tool of the power admin. So they have *started* to move to a server OS that doesn't have a ridiculous UI. I think it will be 2020 before we see a proper MS server operating system - where people who have no business being on the server are lost and don't know what to do. If you need a graphical interface to interact with a server system, then what are you doing there? After seeing lots of interns given admin rights on production machines, and the resulting rubbish (security holes, stability problems) that accumulates when "cheap" devs and admins are given access to machine, I dream of a day when MS will do what Apple did and opt for a *nix code base, and all the advantages that come with it!
A++
ps. Even in it's pitiful state, a Windows CL is still better for all operations that can be achieved with the CL... And not just because it's geekier, because it's FASTER.

Re:Set 32 sectors per track

[kimvette]

Oh, in addition, now that Windows Server (Core) has a real GUI-less mode and Powershell and UNIX environment shells on Windows finally have usable interfaces, shell prompts are becoming even more relevant even in large Window shops. So, even Microsoft has acknowledged that the UNIX-y way of doing things is key for automation and uptime in an enterprise environment. Now, most PCs won't boot with output to the serial port, but some enterprise server boards do have such options.

A GUI is great for basic tasks, but for repetitive tasks a command shell and scripting environment are key for efficiency, and reliable automation. VBS/Windows Scripting Host was an "acceptable" workaround for a while but in the past many Windows administrative tools required the box to not be headless, the workstation unlocked and the windows open for the GUI to be accessible for scripting - and even then it was iffy because not all GUI elements are accessible (especially third-party tools with custom controls).

Re:Set 32 sectors per track

[bertok]

Actually this problem is potentially much worse on SSD's. Erase blocks are huge, and read-modify-write really sucks on flash.

Couldn't this be addressed (at least in part) by a battery-backed write cache like better RAID controllers use? Set it up like SAN snapshots (so it just stores the diff between what's in the actual flash storage and what's been changed so far), and then write the changed blocks when it's most advantageous (e.g. when there's an entire block's worth of data, so it would all have to be erased by the flash storage anyway).
Maybe combine that with something like a disk defrag, except instead of storing frequently-sequentially-read data in physical sequence, store frequently-written data (regardless of if it's sequentially-read or not) in physical sequence.

That's exactly what most SSD controllers do!

Some now come with 32 to 64MB of cache, and some of the new Sandforce controller based SSDs also come with a little ultracapacitor that acts like a mini UPS. The cache is used as scratch space for reordering writes and defragging blocks.

There was a firmware patch recently for the OCZ Vertex series of SSDs that enabled background defrag. If you let the drive site there for a few minutes, it would start getting faster until it returned to 'as new' speeds

Re:Set 32 sectors per track

[coolsnowmen]

In window's defense, they recognize thumbdrives just fine for almost a decade; it is the manufacturers who bottle their own autostart and drivers and non-standard thumbdrives so they can include backup software, encryption, or flat out spyware.

Re:Set 32 sectors per track

[orkysoft]

I recommend you visit Microsoft and have a look at their "Windows" operating system. The concept of a terminal doesn't run nearly as deep in it as it does in Linux. The same goes for the concept of security. Overall, it is kind of a poorly reinvented UNIX, but I think you might just like it. There are quite a few applications available for it nowadays, and it is gaining more and more marketshare and public recognition.

Re:Set 32 sectors per track

[MobileTatsu-NJG]

We both agreed that most of windows land involves emailing shit to yourself, and a lot of USB thumb drive use...

Explorer: \\ComputerName\c$\Documents and Settings\UserName\My Documents\

Permissions permitting, this is all you need to do. Or you just share folders.

(Of which I could fire off a good half-hour rant on how poorly windows handles mass storage devices. It's a USB THUMB DRIVE for gods sake. It's not a fucking printer! I want to plug it in, and transfer files to/from it. It doesn't need to be "installed", indexed, and have drivers downloaded for it. Just fucking open a file browser like any sane OS does. )

This is a 10 year old complaint.

I have a hard time working on windows, because I'm so much more efficient with a terminal. It's not that I can't use a gui - I'm just an order of magnitude faster using the terminal.

That and you're not using Windows properly.

Re:Set 32 sectors per track

[apoc.famine]

Damn, you really don't know what a terminal is, or what they do, do you?

Explorer: \\ComputerName\c$\Documents and Settings\UserName\My Documents\

Doesn't do anything I posted I did. I'm trying to put to words how ridiculously out-of-scope your "solution" is, but I really can't. You really need to learn what people do with terminals before posting on the internet, so you don't look like a complete dumbass. It doesn't help that I'm talking about transfers between machines which are not on the local network, and which don't run windows on either end.

And for the record, I AM using windows properly - I boot into it to play games. It's great for games, and not useful for anything else.

Re:Set 32 sectors per track

[MobileTatsu-NJG]

Ah, you're right, I missed the "between work and home" bit. I apologize. That's a little tougher because you need to get the two machines talking to each other, and that's what SSH does nicely. One way would just be to add a step in the beginning to set up a VPN. Alternatively I think Windows has a built in feature for connecting two machines' file systems remotely, but I haven't messed with it in years. There's probably a half dozen other ways to do it. Windows can share files over the net just fine, it's not some feature exclusive to *nix.

And for the record, I AM using windows properly - I boot into it to play games. It's great for games, and not useful for anything else.

Right. You're using it properly by being utterly unaware of how Windows networking works and using USB flash drives (that only seem to require drivers whenever you use it) and email to share files between two internet-connected machines. I wouldn't want to look like a dumbass for challenging you there.

Re:Set 32 sectors per track

[apoc.famine]

I'm sure that windows can share files over the net just fine. The issue is that it doesn't do it easily, cleanly, or by default. I could probably learn how to do it, but why bother?

>> scp file_location file_destination

That was my original rant - it doesn't matter if either location is on the same file system, a device connected to the computer, a computer on a local network, or running on something in space.

That was my point about ease and efficiency you completely ignored by offering a something that definitely wasn't a solution to my initial complaint.

As for your snarky reply about me not using windows properly, once again you don't seem to understand how that shit works. I wanted to quickly transfer 1gb of files off a windows laptop in a lab, so I slapped a USB thumb drive into it. As it was the first time it had been plugged in, I was treated to 2 minutes of installation and configuration. The same thumb drive, plugged into the OSX machine across the hall immediately showed up in Finder, and I started pulling files off it 2 seconds later. Sure, should I ever plug that same drive into that same laptop, it probably won't do that. However, it shouldn't have done that in the first place. It's a damn storage device.

If that laptop had any sort of useful networking on it, I'd have just transferred the files from it to the OSX machine. But windows doesn't come with anything that can talk to OSX by default.

Sorry, but when it comes down to it, Windows blows for networking and file transfers. I'm sure that with some windows-fu and some googling, and some third-party software you can get windows to transfer files securely across the internet to another machine. The real question is "Why would you bother?" Unix based systems just do it by default, pretty much as quickly and as efficient as possible.

If you're setting up a VPN to transfer files, you're not using windows properly. You should be playing games on it, and using a decent OS for networking.

Re:Set 32 sectors per track

[Anonymous Coward]

I don't think he was dissing command line interfaces.

I think his complaint was that even newfangled RS-232 terminals had to jump through hoops to remain compatible with computers that were hooked up to typewriters and line printers. The protocols and underlying software have idiosyncrasies built into them that just don't make sense any more. Instead of throwing away the cruft to make something better, everybody's hacking onto the same old outdated shit. It's limiting progress, in a way.

Re:Set 32 sectors per track

[hoggoth]

> . this document is heart wrenching

I don't think that word means what you think it means.

Re:Set 32 sectors per track

[Anpheus]

I don't know if an object oriented command line interface in which objects, not text, is piped from one command to another is quite the same thing as the "Unix way." It adds an enormous level of complexity.

Not that it isn't interesting or useful. And we'll see how well it takes off in the coming years as more Microsoft management tools become GUI interfaces that write Powershell scripts.

Re:Set 32 sectors per track

[mirix]

RS-422 is great. I've got a 4 port card in this machine, and have direct control over 4 machines (on a different floor no less) with it, whether the network is up or not.
Well, apart from a power switch.

Re:Set 32 sectors per track

[MobileTatsu-NJG]

The issue is that it doesn't do it easily, cleanly, or by default. I could probably learn how to do it, but why bother?

Because you had to learn how to do it anyway when you started with Unix. I'm sure there were a lot of things there that you couldn't find 'by default' that you ended up slapping your forehead about later.

That was my original rant - it doesn't matter if either location is on the same file system, a device connected to the computer, a computer on a local network, or running on something in space.

It doesn't matter in Windows, either. You have to have them talking to each other, just like you do in Unix.

As for your snarky reply about me not using windows properly, once again you don't seem to understand how that shit works. I wanted to quickly transfer 1gb of files off a windows laptop in a lab, so I slapped a USB thumb drive into it. As it was the first time it had been plugged in, I was treated to 2 minutes of installation and configuration. The same thumb drive, plugged into the OSX machine across the hall immediately showed up in Finder, and I started pulling files off it 2 seconds later. Sure, should I ever plug that same drive into that same laptop, it probably won't do that. However, it shouldn't have done that in the first place. It's a damn storage device.

Okay... so your complaint is that the first time around it needed to configure itself and it works every time after that. But you don't know for sure because you've only done it once... right? And this means you know everything about how Windows works? I just don't think if I tried to make the same complaint about Unix or Linux I'd be let off the hook very easily. In fact, I know this. I've gotten roasted pretty heavily before for not instantly knowing which configuration file to go into with a text editor and which line to flip values on.

If that laptop had any sort of useful networking on it, I'd have just transferred the files from it to the OSX machine. But windows doesn't come with anything that can talk to OSX by default.

Both OS's are configurable with network shares that the opposite OS can talk to. And when I mean 'congfigurable', I mean you don't need to install anything special.

Sorry, but when it comes down to it, Windows blows for networking and file transfers.

*nix is definitely much stronger at networking and remote administration than Windows. No argument from me, there. However, the problems you're ranting about were easily solvable.

If you're setting up a VPN to transfer files, you're not using windows properly.

Hee hee. Why do I think this line was written after a hasty Google search for "Windows VPN"?

You should be playing games on it, and using a decent OS for networking.

You should learn how to use whatever OS you're working on before you run around publicly criticizing it.

Re:Set 32 sectors per track

[hullabalucination]

All of the desktops in use at my local Chamber of Commerce (WinXP) can't show/mount a thumbdrive, apparently related to this issue:

www.howtogeek.com/howto/windows/find-your-missing-usb-drive-on-windows-xp

I pointed them to a Microsoft knowledgebase entry which supposedly fixes it, but they reported that it didn't work for them. I don't use Windows much so the issue is puzzling to me; the Chamber has a system builder on retainer but apparently nobody there has seen fit to rectify the problem (or just can't).

* * * * *

Be careful about reading health books. You may die of a misprint.
—Mark Twain

Re:Set 32 sectors per track

[quantumphaze]

I think he is talking about how Windows will give you a popup stating that it is installing a new device the first time you insert a USB stick who's specific model has never been used with that Windows install. I just tried it on our Vista machine and it has a popup saying something like "Installing New Device Software" then "Your device it ready to use".

It's an improvement on what XP did. XP would go though setting up the driver to work not only with each new model it saw, but for every time you used a different USB port on the same computer. It works the way it should if you plug an already used USB stick in a port it was in previously

Re:Set 32 sectors per track

[mikael]

'lynx', '(al)pine', bash, fdisk, parted, vim and ssh are still extremely popular. Command line isn't quiet dead yet.

Re:Set 32 sectors per track

[webnut77]

Mod this feller up!

If you depend on a GUI you aren't an admin.

Re:Set 32 sectors per track

[yuhong]

Yea, it was a big point in the Windows vs NetWare wars in the late 90s, as I remember

Re:Set 32 sectors per track

[yuhong]

So they have *started* to move to a server OS that doesn't have a ridiculous UI. I think it will be 2020 before we see a proper MS server operating system - where people who have no business being on the server are lost and don't know what to do.

BTW, Windows Server 2008 already has "Server Core" which is pretty close, if it is not already, I think.

Re:Set 32 sectors per track

[dargaud]

Yes, and the most funny thing is that the Mac, which was for a decade and a half a pure GUI system, now has a perfectly usable command line bash shell !

Re:Set 32 sectors per track

[cerberusss]

I recommend you visit Microsoft and have a look at their "Windows" operating system. [...] Overall, it is kind of a poorly reinvented UNIX, but I think you might just like it.

I've seen some people use it, then they told me you had to pay for it. I was flabbergasted -- why would anyone pay for an operating system?

Re:Set 32 sectors per track

[bemymonkey]

Decent USB sticks (i.e. no weird autostart firmware or things like that) take about one full second to go through the "Installing New Device Software" on Win7... Is this so much faster on Linux/OS-X?

It's never bothered me much, but hey, maybe I'm missing out on some fantastic USB-Stick superspeed ;)

Re:Set 32 sectors per track

[bemymonkey]

I'm sure that windows can share files over the net just fine. The issue is that it doesn't do it easily, cleanly, or by default. I could probably learn how to do it, but why bother?

What's not easy about right-clicking on a folder and setting a few permissions in the sharing tab?

>> scp file_location file_destination

That was my original rant - it doesn't matter if either location is on the same file system, a device connected to the computer, a computer on a local network, or running on something in space.

Care to elaborate? How is this easier on *nix? Replace "scp" with "copy" and it'll work just fine on Windows...

As for your snarky reply about me not using windows properly, once again you don't seem to understand how that shit works. I wanted to quickly transfer 1gb of files
off a windows laptop in a lab, so I slapped a USB thumb drive into it. As it was the first time it had been plugged in, I was treated to 2 minutes of installation and configuration. The same thumb drive, plugged into the OSX machine across the hall immediately showed up in Finder, and I started pulling files off it 2 seconds later. Sure, should I ever plug that same drive into that same laptop, it probably won't do that. However, it shouldn't have done that in the first place. It's a damn storage device.

So buy a storage device. Not a device that has "Automagical backup software" and "Virtual Optical Drive Emulation" built into its freakin firmware. Obviously this stuff doesn't work on *nix, because it's programmed for Windows, but you should be placing the blame on the manufacturers for ramming this crap down peoples' throats...

If that laptop had any sort of useful networking on it, I'd have just transferred the files from it to the OSX machine. But windows doesn't come with anything that can talk to OSX by default.

Isn't it also the other way around? i.e., OSX doesn't come with anything that can talk to Windows by default? I'm guessing there'd need to be compatibility on both sides...?

Sorry, but when it comes down to it, Windows blows for networking and file transfers. I'm sure that with some windows-fu and some googling, and some third-party software you can get windows to transfer files securely across the internet to another machine. The real question is "Why would you bother?" Unix based systems just do it by default, pretty much as quickly and as efficient as possible.

So you're saying that Windows is crappy for networking and file transfers because it doesn't have a built-in utility for securely transferring files across the internet?

Sure, I think a built-in facility would be useful, but the majority of Windows (home) users is going to be perfectly satisfied with opening "Network" in "Computer" and selecting the machine they want to transfer files to/from (this needs no setup, by the way - just leave your router with DHCP enabled and the machines will pretty much find each other without problems since XP [although back then it helped if the machines were all set to the same workgroup]).

Windows 7 has added "Homegroups" and all sorts of other crap you don't need (and would have to configure first), but simple file-based networking works perfectly fine without it... just specify which folders you want to share (read-only by default, set specific permissions if you want something else) on the remote machine and you've got access.

If you're setting up a VPN to transfer files, you're not using windows properly. You should be playing games on it, and using a decent OS for networking.

Hmmm, I use my VPN for opening files in place, not for file transfer. Using a VPN means you're effectively _inside_ the network you're VPN'd into, and you can use it as such. No need to keep the files locally at all, unless you have bandwidth constraints that make working in this way difficult...

Re:Set 32 sectors per track

[quantumphaze]

Linux isn't really fast with it. It takes about 4 seconds from insertion until KDE's device notifier pops up. Then again YMMV with different sticks. For all I know it may be programmed that way to let USB hard disks spin up fully before mounting.

Compare this to an SD card in my laptop's card reader which is near 0.3 seconds. Which is another thing I find Windows doesn't handle correctly with regards to unmounting. Windows Vista will present our desktop's built in multi-card reader (ones that take the old floppy drive slot and plug into the front USB pins on the motherboard) as persistent F:, G:, H:, I: drives and under the one entry in Safely Remove Devices.
Removing this will remove the actual reader from Windows, not just the cards. The reader is unusable until the machine is rebooted (there may be another way that doesn't involve replugging the reader in, please tell if there is) as the USB device is completely disabled. Ejecting the individual drive by r-clicking it in My Computer will unmount it without killing the card reader.
I don't know what Windows 7 does, not installed on that computer.

Linux seems to ignore the card reader and treats the cards as normal removable storage. I can even remount it again without having to replug it.

Re:Set 32 sectors per track

[Anonymous Coward]

So he would not end up with a crappy OS that doesnt support hard disks properly.

Re:Set 32 sectors per track

[bemymonkey]

Interesting, I've had this issue too but attributed it to the cheap USB card reader.

Re:Set 32 sectors per track

[jonadab]

> What's not easy about right-clicking on a folder
> and setting a few permissions in the sharing tab?

CIFS filesharing is one thing on a LAN, and something else again if you're transferring via an untrusted network, such as the internet. That's what scp is for.

The other user's main beef, near as I can tell, is that getting scp to work on Windows is an exercise in hunting down third-party software. Which is true.

But that's not really unique to scp. *Anything* you want to do on Windows is generally going to require having additional software installed that's not included with the OS. Even FreeBSD comes with more functionality "out of the box" than Windows, and I'm not talking about ports. Windows is basically just an OS (and a basic desktop environment), not a whole software distro. That's kind of the point: Microsoft supplies the OS, and you get the other software separately, usually from third parties (although Microsoft has some complementary products too, e.g., Word).

That's the way it's *supposed* to be on Windows; it's *designed* to be that way. With Windows, your computer is supposed to only have the software you've gone out of your way to install, so there's not a bunch of other stuff cluttering up your All Programs list. Windows is made for people who pretty much use the computer for the same few tasks all the time, so they only want the software they want and not a bunch of other (to them irrelevant) stuff. If that bothers him and isn't the way he likes things, if he wants a whole distro with software for a wide variety of purposes installed out of the box, why is he using Windows in the first place?

Re:Set 32 sectors per track

[jonadab]

> RS-232 might be absent from a lot of consumer motherboards

Yeah? Can you actually name a recent non-Apple desktop or workstation motherboard that doesn't have it? (Netbooks, yeah, sure, but those are supplemental portable devices, not intended to replace a desktop computer.)

Compaq experimented with selling "legacy-free" systems for a few months circa 1999. Their "iPaq" model of mini-tower system, for instance, was available in a USB-only "legacy-free" version, and then there was a "standard-ports" version for just a couple bucks more. When they discontinued the iPaq line in favor of the Evo a few months later, guess which way they went with it? Then HP bought them out and the whole thing become an obscure footnote for the computer history books.

I predict RS232 will outlast USB 2.0, because subsequent USB versions will replace 2.0, but RS232 isn't going anywhere.

While we're at it, PS/2 keyboards and mice don't seem to be going anywhere either (although a lot of them have become de-facto hot-pluggable, which may be at least partly due to the influence of USB).

In fact, the last major computer component interface I can think of that died out is ISA.

Re:Set 32 sectors per track

[bemymonkey]

So basically the gripe is, "I want everything I need built into the OS."?

Hmmmm, I'm just a dumb Windows user, but isn't that "everything including the kitchen sink" approach what many FOSS users refer to as bloat?

Re:Set 32 sectors per track

[TemporalBeing]

RS-232 might be absent from a lot of consumer motherboards

Yeah? Can you actually name a recent non-Apple desktop or workstation motherboard that doesn't have it?

It is actually very hard to find an RS-232 port on newer computers now. Almost all manufacturers have dropped them from laptops at the very least, if not their desktops as well.

Typically if you want to work with a serial port you now have to get a USB to Serial port interface.

RS-232/422/etc have gone away.

BTW - neither my Lenovo T61P nor my wife's HP laptop have a serial port of any kind on them. My 2003 era Dell D600 does, but most of the Dell's don't today (or even a few years ago).

Re:Set 32 sectors per track

[coolsnowmen]

Clearly you are not making this up, but I can't remember having this problem. Thanks for the link.

Re:Set 32 sectors per track

[jonadab]

> but isn't that "everything including the kitchen sink"
> approach what many FOSS users refer to as bloat?

Good point. There are two schools of thought on that. When you see people complaining about bloat and arguing that no extra features should be included in anything, you're seeing the opinion of a percentage of the open-source community. The most extreme of these people will argue with a straight face that the base OS install shouldn't include even basic utilities that many of us take for granted (e.g., egrep, bzip2), and furthermore that only very basic versions of essential utilities should be installed: ash or somesuch instead of bash, more instead of less, and so on. In fact, FreeBSD doesn't include bash, so if you do install it (via ports) it goes in /usr/local/bin instead of /bin.

To most Linux users, this extreme approach is unexpected and weird. Most of us expect such things as bash and bzip2 and egrep -- and perl and screen and make for that matter -- to be installed on all computers as a matter of course.

The other school of thought is, I don't want to fool with it, just install everything. Most Linux distributions offer you the option, at install time, to check and handful of boxes for broad categories (like Gnome, KDE, server stuff, and so on) and have a whole bunch of packages from those categories installed automatically. Just install it all, so I don't have to fool with selecting eight hundred packages individually. This point of view is just as common, if not more so. And this side of things also gets taken to extremes sometimes (e.g., there are people who literally install the ENTIRE contents of the CPAN on their computers).

Most of us are somewhere in between. I generally install all the major categories (Gnome, KDE, etc), because modern hard drives are big enough that I can, and it saves time selecting the individual packages I want. But yeah, there are still things that don't get installed, and so you have to select those for installation individually.

You can, however, get practically everything from the distro's repositories, so it's seldom necessary to hunt down anything from a third party, unless you insist on installing proprietary software (like Opera), or something so specialized or obscure that it's not packaged (like Evergreen/OpenILS).

Re:Set 32 sectors per track

[jonadab]

> It is actually very hard to find an RS-232 port on newer computers now.

Huh. That's... odd.

It's been years since I've seen a computer, new or otherwise, (other than laptops, as I noted earlier, and Macs) that doesn't have the serial and parallel ports.

> neither my Lenovo T61P nor my wife's HP laptop

Laptops, right. I already mentioned those as an exception, didn't I?

You don't get as many ports on a laptop, because nobody would use them anyhow, because nobody uses peripherals with a laptop anyway, because carrying them around everywhere would kind of defeat the whole portability thing, which is the main point of a laptop.

And Macs also don't have printer and serial ports -- even when they did, they weren't the same as the ones on PCs and so weren't compatible with non-Mac peripherals anyway, so no real change there.

But RS232 isn't going away at all. PCs (other than laptops, do I really have to keep saying that) all still have parallel and serial ports. And a lot of peripherals still use them (we just bought such a peripheral a6t work, so recently that we haven't even installed it yet).

Re:Set 32 sectors per track

[RivieraKid]

It's not so much a problem with track-to-track seeking, as it is a problem with the misaligned I/O itself.

If your on-disk structures do not align with the physical structures, then you experience a penalty for every host I/O to that disk. It's exactly the same issue with enterprise class storage arrays. the volume on the array is offset from the very beginning of the physical disk so is correctly aligned. However, every x86 style partition table laid down on those volumes consume the first 63K of the disk, meaning the data starts at the 64K mark - 1K before the start of the next disk track.

So, in a nutshell - for every host I/O, you will generate at least one additional I/O for the disk due to the badly aligned start of the partition table. Worst case, you'll generate two additional disk I/Os (for multi-track I/O) since the last track is also misaligned. For large block sequential I/O, you can pretty much ignore this effect, for small block random I/O - it's gonna kill performance. Align your partitions to 64K and the problem goes away - of course, this has to be done before you put any data on the partition and is not done by default with any OS I'm aware of today, which explains why nobody ever does it.

Note, since I mentioned storage arrays I'll clarify that this is not the same as the RAID write penalty.

Note also, this is usually only a problem for x86 style partitioned disks.

Re:Set 32 sectors per track

[fbjon]

What?...

At work, I have a PC that does not have an RS232 port, so I had to get a USB adapter. At home, under my desk, I have a computer just a few years old, also without RS232 (it does have parallel though).

Finally, looking through the motherboard selection at my local webshop, I see that RS232 is not just disappearing, it's gone. Not one board out of about 50 models of various sizes that I checked has it (I looked at the pictures of the connector side as well to make sure). The closest I can get is an old nForce3 board that takes DDR400 memory, on an auction site. That one has a serial port.

Conclusion: the serial port went away years ago.

Re:Set 32 sectors per track

[TemporalBeing]

You don't get as many ports on a laptop, because nobody would use them anyhow, because nobody uses peripherals with a laptop anyway, because carrying them around everywhere would kind of defeat the whole portability thing, which is the main point of a laptop.

Actually there are a few industries (e.g. trucking) where serial ports on equipment are very common place - so you do need them on portable devices. Unfortunately, the computer industry is more or less ignoring those industries so you have to get the converters instead.

Even if you did move to USB on the newer equipment, it'd still be years before the serial ports disappeared simply b/c of the slow turnover of the equipment.

So yes, there are good uses for RS-232/422 that don't defeat the purpose of portability.

And, FYI, USB was suppose to be the serial port killer, and the computer industry is treating it as such - so they've been dropping it more and more.

Re:Set 32 sectors per track

[jonadab]

> USB was suppose to be the serial port killer

Yeah, I heard that too, in the nineties. I also heard, around the same time, that Iomega drives were doing to be the floppy disk killer. They were starting to come out with internal models, see, and all new computers were going to come with the Iomega drives, and nobody would need floppies anymore. It never happened.

Instead, USB eventually turned out to be the floppy disk killer. CD-R made a decent start on killing floppies, but it couldn't quite finish the job (though it *did* kill off Iomega drives fairly effectively). Now, though, I'm seeing fewer and fewer floppies with every passing year, and I've seen more than a few new computers (*desktop* computers, not laptops) with no floppy drive. And it's not because of Iomega. It's because of flash-EEPROM-based USB 2.0 mass storage devices, the smaller capacity ones of which have become so cheap you basically get them in specially marked boxes of breakfast cereal these days. They cost less than PCI 10/100 ethernet cards or, more to the point, less than a small box of floppies. Apparently people must be using them, because I find more of them laying around (left behind by mistake) than I ever did floppy disks.

But as far as USB peripherals that formerly would have used the serial port, the only ones I've seen are digital cameras (and serial was never a good fit for them in the first place; SCSI would have made more sense, except that most computers didn't have it). I've seen devices that formerly would have been parallel and are USB, of course (printers mainly). And I've seen USB mice and keyboards occasionally. But peripherals that traditionally used a serial port, for the most part, still use a serial port. The camera is the only exception I know of.

Re:Set 32 sectors per track

[jonadab]

Where do you live, out of curiosity?

Like I said, all the new computers here have serial and parallel ports. Well, some of the laptops don't, but some do, and *all* of the regular computers do. I'm in central Ohio.

Re:Set 32 sectors per track

[TemporalBeing]

But peripherals that traditionally used a serial port, for the most part, still use a serial port. The camera is the only exception I know of.

Well, you also have scanners (though they were typically SCSI), external modems (who has a modem now?!), Computer-Enhanced Sewing Machines, and then there was the specialized equipment - like the Vistar Datacom MT 2000 that I worked with for asset tracking a few years back.

And of course, those specialized devices have very low turn-over, and don't get upgraded very much.

Either way, the computer industry itself has left serial behind, while the peripheral industries continue to use it. Thus the USBSerial device industry is doing quite well.

Re:Set 32 sectors per track

[yuhong]

even when they did, they weren't the same as the ones on PCs and so weren't compatible with non-Mac peripherals anyway

Ha? Apple designed their RS-422 serial ports from the beginning to be compatible with RS-232 through RS-423 provided you have the right cable..

Re:Set 32 sectors per track

[fbjon]

Finland.

I checked again, and there are some mini-ITX boards that do have serial ports. Other than that, the only non-used motherboard I have found with a serial port is an Asrock Socket A board, which is quite ancient by today's standards.

Re:Set 32 sectors per track

[jonadab]

> Apple designed their RS-422 serial ports from
> the beginning to be compatible with RS-232
> through RS-423 provided you have the right cable..

Yeah? Try to actually use PC serial peripherals with a Mac from the pre-USB era. The cable adaptor, as a rule, can only be counted on to work if the devices *comes with* said adaptor and is labeled as supporting the Mac as well as the PC. Otherwise, forget about it.

(In fairness, a lot of Mac peripherals don't work with the PC either.)

Granted, I don't know the technical details of the Mac serial port, so as far as I know it's possible that the problem isn't the serial port itself (other than the obvious need for an adaptor because the the physical plug is obviously dissimilar), but something else. (Drivers seem like a likely possibility.) All I know is, you couldn't use PC serial devices on Macs; it never worked.

Re:Set 32 sectors per track

[Xabraxas]

The OS supports it. Some tools do not yet properly support the new hard disks.

Good thread on this.

[Anonymous Coward]

http://www.osnews.com/thread?409281

Re:Good thread on this.

[darkpixel2k]

http://www.osnews.com/thread?409281

One of the comments in that thread suggests switching to GPT if you aren't using Windows.
I haven't used Windows at home since ~2001.

Can you just wipe/reinstall using GPT? I thought the BIOS was involved with the type of partition table and that I had to be using the msdos partition type because of the BIOS. Can a geek with deeper knowledge of partitions and and all things boot drop some knowledge?

Re:Good thread on this.

[Anonymous Coward]

The BIOS has no understanding of partition tables. It merely reads the first sector of the harddrive to 0x7C00 and then jumps to that location. The DOS partition table is used by convention for interoperability between operating systems. If you wanted to use a different partitioning scheme, there is no technical reason your operating system couldn't.

Re:Good thread on this.

[Anonymous Coward]

GPT wraps itself in a MBR partition map. At the very least the GPT is supposed to include an MBR map that claims the whole disk as used by GPT to avoid issues with old disk tools and the like. And if you've got a partition scheme that's compatible with the MBR scheme they can both contain the same information, assuming your disk tool supports this, so that MBR-only environments can still find your partitions.

It's also possible to format with GPT and then use an MBR-only tool (fdisk) to go back and manipulate the (fake) MBR to contain a partition that points to the same start/end points as the GPT boot partition -- GPT-aware systems will just ignore the MBR record, and non-GPT systems will at least be able to find the boot partition.

As to whether your motherboard/firmware supports GPT, it can be hard to say. Anything with EFI is required to support GPT. Some systems with a legacy BIOS pre-boot environment also have support for GPT, because it's the only way to support large disks. But I can't name particular firmware versions that do/don't support GPT.

Re:Good thread on this.

[marcansoft]

Unless your BIOS is trying to be too smart and peeking into your partitions instead of launching the MBR (sadly, some do), it won't matter. It's the MBR's job to boot your system after the BIOS hands off control to it, and on most Linux systems the bootloader is installed straight into the MBR.

Re:Good thread on this.

[Z34107]

Even if you are using Windows, Vista and up support GPT. It's handy for servers where you expect to have partitions larger than 2 TB.

But I guess if one were using a modern version of Windows, you wouldn't have the 4K alignment problems to begin with.

Re:Good thread on this.

[Anonymous Coward]

Vista can't boot from a GPT disk. So while it does support it, you'll still need MBR for your OS disk. Don't know about Win7, but seeing it's the same kernel I won't hold my breath.

Re:Good thread on this.

[Z34107]

That's an interesting point - I assumed diskpart supporting it meant it could boot from it.

However, Vista and 7 (and the server editions) can boot from a GPT partition, but only the x64 bit versions. source

Re:Good thread on this.

[Anonymous Coward]

Right, but Windows can't boot from GPTs. So you are mostly out of luck.

Re:Good thread on this.

[Anpheus]

I think Windows Vista & 7 requires UEFI + GPT simultaneously to boot.

IMO, it's good for the most popular OS to be really conservative on the issue of how booting is performed because it is so easy to mess up. I cannot count how many times, when experimenting with dual and triple booting, I've rendered a system disabled or nearly so. While recent tools (SuperGRUB, for example) make it easier, it's still a veritable minefield.

Re:Good thread on this.

[yuhong]

Yep, I wonder once we get beyond 2TB whether it is better to increase the logical sector size or just switch to GPT.

Re:Good thread on this.

[yuhong]

Which cause trouble with the Windows installer that always replace the MBR with the standard MBR. Luckily most bootloaders can be set to install into the partition boot sector, which avoid this issue.

Re:Good thread on this.

[RivieraKid]

Sorry, but up to Server 2008, Windows still doesn't handle partition alignment. Unless I'm completely mistaken, that's largely same kernel in Vista and Win7 too...

Interesting

[Murdoch5]

I actually have 2 of the these drives in my desktop right now. There is a slight decrease in performance compared to Windows 7 but nothing that it unacceptable or even a need for concern. If you need to worry about the performance lost with the 4k sectors then just go solid state.

Re:Interesting

[ScytheBlade1]

TFA disagrees with you.

$ time cp winxp.img /mnt/sdc # ALIGNED
real 5m9.360s
user 0m0.090s
sys 0m20.420s

$ time cp winxp.img /mnt/sdd # UNALIGNED
real 13m26.943s
user 0m0.110s
sys 0m19.350s

$ time cp -r Computer Architecture/ /mnt/sdc # ALIGNED
real 42m9.602s
user 0m0.680s
sys 1m59.070s

$ time cp -r Computer Architecture/ /mnt/sdd # UNALIGNED
real 138m54.610s
user 0m0.660s
sys 2m15.630s

The first two being a single file, the latter two being multiple files in a larger directory structure.

I would heartily disagree with you on the matter.

Re:Interesting

[markus_baertschi]

What do you disagree with ? There is a performance problem and the author thinks the kernel people should look into the matter. I don't think they can do much, after all the kernel just writes 4k block to the drive. The problem is with the drive microcode which should attenuate the problem in the case of sequential writes, like the first case. Obviously it does not do that at all. Markus

Re:Interesting

[Murdoch5]

That is more a distro problem then, I run a full stable Gentoo system and I see less then a 5% over head difference. Not saying your wrong because you posted output here but it can have a lot to do on how you have optimized your disk access and setup the kernel to interface the disks. Even the file system in use.

But then again that's just what I see on my system. Possibly running a different configuration would yield different results.

Re:Interesting

[ScytheBlade1]

I disagree with the "but nothing that it unacceptable or even a need for concern" part. If copying the same image to the same disk where the only difference is where the partition begins -- by one sector difference -- will amount to a 2.6x decrease in speed, and where copying a large directory with many subdirectories amounts to a 3.2x decrease in speed... that should qualify as an unacceptable decrease in speed that warrants concern.

While a kernel tweak may help alleviate the issue, it is primarily an issue with our current (userspace) disk partitioning and formatting utilities. I'd also disagree with you on the point where the problem is the drive microcode; drives should do what they are told, and not guess on behalf of the instructions they are given what to do. Admittedly, the microcode tweak would be minor and largely trivial, but I'd rather not fix (primarily) userspace software problems in the kernel, nor the device firmware.

Re:Interesting

[ScytheBlade1]

Any chance you can post drive model number, output of fdisk -l, and a few similar benchmarks?

Re:Interesting

[Murdoch5]

This problem isn't really kernel involved, the problem comes from the fact the write is not being issued in an effective manor. This more so leads to the fact that the system has not been fitted to use the 4k sectors. If you just put a little work into the install you can over come this problem.

By default yes you will likely see a big performance hit, but like any good system once it's tuned and fitted to it's job it will run great. People can post all the results they want from default installs and it means very little.

It's the same as if you bought a muscle car and never took the time to tune the engine, you can't try to make the claim that the car wasn't ready. you need to put the time in and get the car up and going.

The same thing is true of the install. You need to put the work in, tune it up and reap the benefits of a high performance install.

Re:Interesting

[Murdoch5]

I will tomorrow. Right now I'm not on my desktop. I can easily get 70+ Mb/s write to them I think somewhere around 77.6 Mb/s. As for my partition scheme I used parted and had it properly align the disk.

Sorry that's the best I can do from where I'm, I'll make sure to update the numbers tomorrow. if it will help I'll even post the gparted version I used to partition with.

Until then have a good one.

Re:Interesting

[markus_baertschi]

About the microcode part. The drive pretends to be a 512byte drive, but internally is using 4k sectors and and claims to 'translate transparently'. I can understand that in a random-access scenario it it has to read-modify-write 2 sectors each time and performance suffers (2 additional reads and one additional write). But in a sequential access scenario, the penalty should be once per sequence/file, not once per sector. Here the microcode fails completely to make the best out of the suboptimal situation.

Re:Interesting

[hedwards]

That's true, but it's also true that having hardware lie to the OS isn't a great situation to be in. At the very least there should be some way of forcing it to be honest for the benefit of OSes that can handle the reality. A lot of the gunk and instability in computing comes from hardware that does things that are more appropriately done by software and vice versa.

Forcing users to optimize isn't inherently wrong, it's just that they shouldn't need to do it for things which are somewhat standard as a work around for weird hardware designs. And yes, I realize that the 4096byte sectors aren't being implemented arbitrarily.

Re:Interesting

[maeka]

I hope you mean 70+ MB/s.

Re:Interesting

[Murdoch5]

lol my bad ya MB/s

Re:Interesting

[Murdoch5]

I could not say it better my self, I'm going to show that post to my prof if it's okay with you. That might be the best way of explaining hardware and software interfacing I've ever heard.

Re:Interesting

[bertok]

Wait.. wait.. what the fuck?

I was under the impression that mis-aligned partitions cause small random IOs to take up TWO small random IOs, basically halving random IO performance. It should have a negligible effect on any streaming operation, as only the first and last blocks will have an overhead, the others overlap with useful IO.

Just how BAD is the Linux 'cp' program? Is it doing writes one block at a time, or something insane like that?

On Windows, if you repeated the copy same test, you wouldn't see a measurable difference, unless the target partitions were horrifically fragmented. To get a real slowdown, you'd have to do something like work on a large database file (many small random reads and writes). Microsoft has a paper on tuning Exchange performance, where they demonstrate >100% overhead to unaligned partitions, but you get full speed for backup & recovery operations.

Re:Interesting

[coolsnowmen]

It is not a distro problem. On top of that, you run the same distro TFA does.

Evaluation setup and methodology:

        * Gentoo ~amd64 system with 2.6.31-gentoo-r5 kernel
        * fdisk version: fdisk (util-linux-ng 2.17)
        * The drives are identical, but I did not try swapping configurations to make sure that one drive isn't fundamentally slower than the other.
        * Core 2 Quad at 2.33GHz (Q9450), 8GiB of RAM
        * MSI X48 Platinum motherboard -- Intel X48 Express + ICH9R

Re:Interesting

[Murdoch5]

I know I run the same distro but on that same note no two Gentoo installs are exactly the same.

"No two install are exactly the same"

that's right from the Gentoo handbook. If you tune the system and optimize it under the right parameters then your very very likely to get the same performance I do.

The full article even goes as far to say if you use parted to layout the partitions on the drive which I did and you align the partition correctly then your not going to be hit by the same performance problems, at least not all of the performance problems.

Lets also note his kernel is rather old as I'm running the 2.26.32 branch and I have taken the time to use the right linker flags on system compile.

So before you go saying it's not a distro problem, if your running a source distro it can always be a distro problem. There are so many tweaking variable and setting variables that can be different it's hard to compare them on a linear scale.

Re:Interesting

[ScytheBlade1]

http://git.savannah.gnu.org/cgit/coreutils.git/tree/src

Of note, copy.c, copy.h, and cp.c.

Re:Interesting

[petermgreen]

You don't want a drive reporting different logical sector sizes under different situations, that's just a recipe for breakage with any format that works in terms of sector counts.

And having a 4096 byte logical sector size would cause major compatibility issues.

IMO the best compromise would be to have the drive allow reading of it's physical sector to logical sector size ratio somehow. Partitioning tools could then warn about misaligned partitions and offer to fix them.

Re:Interesting

[bertok]

http://git.savannah.gnu.org/cgit/coreutils.git/tree/src

Of note, copy.c, copy.h, and cp.c.

Wow.. it's like code that time forgot.

I love the way there's about a page of complex code that jumps through hoops to figure out a good buffer size, when 64KB would basically work well in all cases. From what I can see, there's a few situations in which 'cp' will pick tiny buffer sizes and waste a bunch of time on overhead. The default appears to be 32KB, which is too small these days. I've done experiments (admittedly on Windows, but same difference), demonstrating that using IO blocks as large as 1MB improves performance significantly. There's a reason Microsoft increased the default IO sizes on Windows Server from 64KB to 1MB.

A much better buffer selection algorithm would be to detect L2 or L3 cache sizes of the processor, and use a buffer slightly smaller than one of those. You'd have to play with benchmarks to see if staying within L2 is a benefit (I suspect that it isn't).

The biggest problem with 'cp' is that it's synchronous single-threaded. It does a read, then a write, then a read, then a write, etc... For comparison, Windows Vista and later does reads synchronously but then writes asynchronously so that the next read doesn't have to wait for the write to complete. Even with kernel buffering and lazy IO, 'cp' is probably much less efficient that it could be.

Re:Interesting

[walshy007]

And having a 4096 byte logical sector size would cause major compatibility issues.

Only with ancient os's, anything in the last five years should handle it just fine. I'm all for backwards compatibility etc.. except when it can gimp current performance through emulation layers etc like what the hardware does with this 512 byte sectors to 4096 byte business.

Re:Interesting

[Anonymous Coward]

Indeed. Sorry for the late post, but I just received one of these drives (WD15EARS), and here is a simple test:

$ dd if=/dev/zero of=/dev/sda obs=512 count=8M
8388608+0 records in
8388608+0 records out
4294967296 bytes (4.3 GB) copied, 366.6 s, 11.7 MB/s
$ dd if=/dev/zero of=/dev/sda obs=4096 count=8M
8388608+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 47.4261 s, 90.6 MB/s

From this test, it is clear that the microcode does not write-combine sequential writes. In other words, every 512-byte sector write triggers a separate 4k read and 4k write operation. Eight sequential 512-byte writes will trigger eight separate read and write operations! That is completely in line with the factor 8 slowdown seen above, and is unforgivable.

WD should never have shipped these drives without write-combine optimization.

Open Source to the rescue

[bogaboga]

I am no kernel hacker but I can almost guarantee that some kernel hacker will provide a solution to this "short coming" fairly soon.

That's the beauty of Open Source.

I am aware though that "fairly soon" means many things to many people; which means that there could be a substantial delay before we get a working solution to this issue.

I am optimistic nevertheless.

Request to Western Digital: Provide all the information needed to develop a solution.

Re:Open Source to the rescue

[bhtooefr]

Here's the information: Sectors are 4096, not 512, bytes.

That's all that a developer needs to know, AFAIK.

Re:Open Source to the rescue

[nomadic]

That's the beauty of Open Source.

And that couldn't possibly happen with closed source?

Re:Open Source to the rescue

[Anonymous Coward]

That's the beauty of Open Source. And that couldn't possibly happen with closed source?

Not with Microsoft.

Re:Open Source to the rescue

[MoHaG]

And they pretend to be 512 bytes for "compatibility"... (Unlike CD-ROM / SD card sectors...)

Re:Open Source to the rescue

[bogaboga]

I guess you only read but did not understand! Key words in my piece are: "Fairly soon."

Re:Open Source to the rescue

[Anonymous Coward]

The beauty of Open Source is that you can code it by yourself, so why don't you start right now? Another history is distribution and maintability.

Re:Open Source to the rescue

[marcansoft]

Exactly. Drives are pretending to have 512-byte sectors because Windows can't deal with 4k sectors, and then silently reducing performance when you believe them and use 512-byte sector sizes. Had the drives reported 4k sector sizes, they'd work great under Linux and not at all under Windows.

This isn't a Linux problem, it's a drive problem caused by Windows. The solution is to implement yet another workaround for stupid devices, and start aligning partitions to 4k by default.

Nitpick: SDHC card sectors are always 512 bytes, and most SD card sectors are 512 bytes too. Flash memory would benefit from larger sector sizes too, but they've probably stuck to 512 bytes for Windows compatibility.

Re:Open Source to the rescue

[buddyglass]

That's the beauty of Open Source.

I guess the beauty of Closed Source, then, is that the OS supports it out of the box, without some user having to notice the problem, benchmark the performance hit, figure out (more or less) why its happening, make a big blog post, then wait for a qualified dev to fix the problem and for the major distros to pick up the fix?

Re:Open Source to the rescue

[nomadic]

"Fairly soon."

Again, that's still possible with closed source.

Re:Open Source to the rescue

[jedidiah]

> I guess the beauty of Closed Source, then, is that the OS supports it out of the box

      Or not. The sheep-like closed source user is never likely to notice the problem.

      Although the real problem here seems to be that you can't consider a bit of hardware
as it presents itself. Now I wonder why a hard drive company feels the need to have it's
hardware LIE to the OS?

Re:Open Source to the rescue

[hedwards]

Trust me, it's right after adding GPT support to Windows XP home on their agenda. Shouldn't take more than, I don't know forever to add.

Re:Open Source to the rescue

[marcansoft]

The beauty isn't Closed Source, it's Market Monopoly. You know, the one guaranteeing that device manufacturers make up for your failures by deviating from standards in order to make sure that their devices work out of the box with your broken OS.

Re:Open Source to the rescue

[Anonymous Coward]

ah yes, the "beauty" of open source: "someone else will do it."

Re:Open Source to the rescue

[Pentium100]

Now I wonder why a hard drive company feels the need to have it's hardware LIE to the OS?

So the hardware is compatible with more software. For example, hard drives still report some number of cylinders, heads and sectors to the BIOS and the OS, but hard drives have been using ZBR for 20 years now (IIRC) so the sector number is meaningless.

But, as it is now, if my old system needs a new hard drive, I do not need to find an old drive to be compatible with my system (as long as it is IDE or SCSI, I don't know of any adapter from the newer interfaces to ESDI or ST-506, but they probably exist).

They could have made it a jumper setting set to 512B by default though. I assume the hard drive is faster using 4KB sectors instead of true 512B sectors, they could have made an option to reformat the drive to 512B (or maybe it's not possible with modern drives, I have an old 4GB SCSI drive that can be reformatted to a different sector size (I never tried it though)).

Re:Open Source to the rescue

[rsmith-mac]

On the contrary, this has (almost) nothing to do with Windows - it has everything to do with old OSes. The IDEMA didn't approve the 4K sector standard until 2006; it was only in the late 90's that the first meaningful research was begun by IBM on whether 512B sectors would be an issue.

As it turns out, yes, 512B sectors would be an issue, and drive manufacturers would be best served by moving to larger sectors (with some arguing over whether to go to 1K or 4K). So the IDEMA hashed this out over the first half of the decade, and finally in 2006 approved the 4K specification.

The point of all of this is that software written at the turn of the century was all done well before changing drive sector sizes was a serious discussion. WinXP was released in 2001, Mac OS X 10.0 was in 2001, and of course Linux 2.4 was also in 2001. None of those OSes know what to do with anything other than a 512B sector - the only reason Windows factors in to this equation is that WinXP just happens to be with us (no doubt trying to eat our brains) while the other two are dead. Anything circa 2005 or later such as WinVista, Linux 2.6, and Mac OS X 10.5 know full well what to do with a 4K drive.

But even that is beside the point. You don't just make major jumps like this, you have to do it in a transition so that you don't break old hardware and old software alike. Even if XP/Lin2.4/MacOSX knew what to do with 4K sectors, at some point you'd run in to hardware, 3rd party devices, etc that would not. A transition is necessary to let old hardware and software get flushed out of the ecosystem, and as such we're still years out from consumer drives offering native 4K access.

In short: drives are pretending to have 512-byte sectors because there's a lot of old stuff, including Windows XP that can't deal with 4K sectors.

Re:Open Source to the rescue

[TheKidWho]

My Hard drive with 4k sector sizes works perfectly under Windows 7?

Re:Open Source to the rescue

[DAldredge]

Drives have been lying to the OS for more than 20 years in regards to quite a few different things so why pick now to yell and get upset?

Re:Open Source to the rescue

[Blakey Rat]

Don't forget that this "beauty of open source" doesn't even get around to thinking about fixing the issue until a competing OS has had support for it for 3 entire years.

Re:Open Source to the rescue

[Blakey Rat]

Well, since Windows Vista and Windows 7 already support this, I'd say "fairly soon" is demonstrably false. Unless you're happy with "fairly soon" being "after everybody else has been doing it for several years."

Re:Open Source to the rescue

[marcansoft]

There are no hard drives with logical 4k sector sizes out there, as they all emulate 512-byte sectors for compatibility, so your point is moot. However, if they did exist, they would theoretically work with Windows 7. I was talking about Windows XP.

Re:Open Source to the rescue

[Miseph]

s/windows/windowsxp

Whatever you may think of Vista/7/Server2008, this isn't a problem for them. I agree that it's stupid for WD to make their drives lie for compatibility with what is effectively a legacy OS (by modernity, not use), but there's no call to tar newer releases for it.

Re:Open Source to the rescue

[marcansoft]

I was referring to Windows XP - I should've mentioned it explicitly. Nonetheless,

There's a lot of old stuff, including Windows XP that can't deal with 4K sectors.

There's exactly one old thing used in any significant quantity and liable to have to work with these drives, and it's Windows XP. Everything else either isn't in any significant use any more, or will never be seeing one of these drives.

One potential concern would be USB enclosures which have to work with older OSes / devices. To that, I would say it should be the enclosure's job to do the 512-byte sector emulator, not the drive's.

Windows XP has been updated over the years (via service packs and the like) to handle hardware that didn't exist at the time of its release, including things like SATA. They could do the same for 4k sector disks, but they aren't going to do it because they want people to move to Win7. Therefore, Microsoft is still to blame for neither 1) providing a solution for XP, nor 2) providing enough compelling reasons to migrate to Win7. Heck, Vista was a trainwreck and it doesn't really count, so (proper) Windows is actually 3 years late to the 4k party, as Windows 7 was only released in 2009. Effectively, they've spent those three years scaling down support for XP while providing no viable alternative, and now the rest of the world has to deal with a significant amount of people still using a 9-year-old OS.

In short: the fact that people are sticking to 9-year-old XP and making hardware companies break or slow down improvements means Microsoft did something terribly wrong.

Re:Open Source to the rescue

[pchan-]

Nitpick: SDHC card sectors are always 512 bytes, and most SD card sectors are 512 bytes too. Flash memory would benefit from larger sector sizes too, but they've probably stuck to 512 bytes for Windows compatibility.

This is no longer true. Most 2x NAND memory manufactured in the past year is 4KB block sizes with 8KB coming soon. That it pretends to be 512 bytes is a function of the SDIO MLC driver IC. Luckily for SD they come pre-partitioned so that the partitions are aligned properly.

Re:Open Source to the rescue

[Sir_Lewk]

Whose fault is it that the majority of Windows users are still using Windows XP?

Nobody's but Microsofts. Microsoft is the reason this shit situation exists.

Re:Open Source to the rescue

[marcansoft]

That was exactly my point - I was referring to SD and other Flash memory external interface standards, not the underlying NAND chips. At the SD interface level, sectors are 512b most of the time (or always, for SDHC), but the physical NAND devices are 2K/4K, so the interface would benefit from using the native sector size.

Re:Open Source to the rescue

[Blakey Rat]

Huh? What are you talking about? Was that intended as a reply to some other post?

The point I was making is that, since Windows has supported these drives for 3 years now, and Linux doesn't yet have the same level of support, obviously open source development isn't as fast this thread claims.

What the hell does the usage of Windows XP have anything to do with that point?

Re:Open Source to the rescue

[mcrbids]

Actually, the beauty of closed source is that the OS "supports" it out of the box, except that it's buggy as all get out, works very slowly for stuff that was much faster in 1995, and despite many users noticing and complaining about the problem both to the vendor and in various blogs and online forums, it doesn't get fixed for months or years while a qualified dev is finally diverted when the problem is so severe that even the US Army won't buy until it's fixed.

Re:Open Source to the rescue

[nhytefall]

Poster is exactly correct, please mod parent up, and mod GP down.

Re:Open Source to the rescue

[Sir_Lewk]

I'm asserting that for all intents and purposes, Microsoft has not solved this problem. If they had then the drives wouldn't need to fake 512 sectors.

Re:Open Source to the rescue

[guruevi]

Ya know in the olden days, when I was young (love saying that) we didn't have hardware that configured itself so it would work on all platforms. We had to put in settings with jumpers and do low-level disk formats through the BIOS or a boot-floppy and WE LIKED IT (seriously).

These days all ya new-fangled hardware doesn't have to worry about being a master or a slave, getting 5V or 3.3V to the PCI bus or RAM modules, CPU multipliers on the motherboard.

I would simply do the same - get a jumper on the back of the drive that says 512 or 4k - we left it on 512 for ya because we assume you numbnuts still use Windows (XP) but if you want performance and use anything but DOS or WinDOS feel free to switch it. You can then reformat the drive.

Re:Open Source to the rescue

[Blakey Rat]

But they have solved it.

They haven't invented a time machine to solve the problem in an OS that came out years before the problem existed. Is that what you're complaining about? The lack of a time machine?

Re:Open Source to the rescue

[Sir_Lewk]

Solving it is worthless if they can't convince anybody to actually use the fix. I'm complaining about their complete inability to get people to upgrade, so we don't have to make workarounds for a nearly decade old operating system anymore.

Re:Open Source to the rescue

[pchan-]

Oh, haha. I misunderstood your meaning. Good show, then.

Re:Open Source to the rescue

[bfields]

We had to put in settings with jumpers and do low-level disk formats through the BIOS or a boot-floppy and WE LIKED IT (seriously).

My memory agreed with yours up to that last bit....

Re:Open Source to the rescue

[argent]

Non-512-byte sectors first showed up with some early CD-R drives, which used 2k blocks. I also had to deal with Sun SCSI drives requiring non-512-byte blocks back in the '90s. Non-512-byte block size didn't magically become an issue in 2006.

Re:Open Source to the rescue

[IgnoramusMaximus]

... and of course Linux 2.4 was also in 2001. None of those OSes know what to do with anything other than a 512B sector -

And...

... Linux 2.6, and Mac OS X 10.5 know full well what to do with a 4K drive.

Total bullshit.

Linux kernel code had flexible block device sector size since the days of 1.x series of kernels. The "problem" is (and always was) with some of the user-space utilities for some of the file systems available under Linux, file systems specifically designed for ... compatibility with DOS and Windows (and through them with the original, ancient IBM PC XT BIOS).

Even then most of the same utilities have various override options that can be used to make them compliant with "unusual" (from the point of view of Windows) sector/block sizes and dive geometries, although it is not their default behavior. The very article you are responding to moans about this very thing, as if it were any news to long-time Linux users.

Windows apologists and their revisionist history are just pathetic.

Re:Open Source to the rescue

[eric-x]

very nice of you to ignore everything that is insignificant and tell them to drop dead when they want to replace the failing HDD in their antique system. Maybe you don't realize but not only PC's use HDDs. There's hardware out there running on obscure firmware and custom chips which cannot be simply replaced by a new box and a OS upgrade.

Re:Open Source to the rescue

[marcansoft]

An antique system is going to run into several other issues if you try to drop in a newer HDD, among them CHS->LBA, LBA->LBA48, and IDE->SATA. Not to mention that we're talking about new drives here - starting to manufacture 4KB drives doesn't mean ceasing to manufacture 512-byte drives. Right now the Windows XP issue means no one can get any true 4KB drives at all.

By the time replacement components are no longer available at all, the remaining die-hard users of antique hardware will be creating adapter hardware that can do the necessary conversion and 512-byte-sector emulation. And since this antique hardware will invariably be much slower than the drive anyway, the speed overhead will be negligible.

Re:Open Source to the rescue

[Anonymous Coward]

Don't talk so fast. There isn't going to be a 17 year solution. An open source solution can be (and usually is) available as I type this. I've hacked kernels (and also on the kernel mailing list). And posted bugs. And with a senior systems engineer at Intel, who is also a Linux kernel hacker, came up with a solution in less than 6 hours for a nasty chipset problem, with an APIC not assigning interrupts properly. It only affected that chipset. Linus Torvalds found out about the problem by learning about the solution already in place. I think with the kernel, a lot of it is like that. Bits and bytes are not 'deep in the kernel'. Bits and bytes are in the driver, which is in Linux dynamically loaded (and also dynamically unloaded). If its one program (or two or three), and the change only involved three or four places in the code, the fix is already in.

Re:Open Source to the rescue

[alantus]

They could always release a hotfix/patch/service pack to fix it.

Re:Open Source to the rescue

[Anonymous Coward]

Because until now, the lies have been SMART.

Re:Open Source to the rescue

[MoHaG]

SD cards in the > 1GB - 4GB range have larger sectors... Not sure if that is visible to the OS:
http://en.wikipedia.org/wiki/Secure_Digital#Standard-SD_cards_.28non-SDHC.29_with_greater_than_1_GB_capacity...

Re:Open Source to the rescue

[marcansoft]

That's a nice idea. You know, some drives already do this for 128GiB limiting and SATA 1.5Gbps mode forcing, but even there some crappy Western Digital drives have ditched jumpers: they'll let you force 1.5Gbps SATA speed mode, but you have to use a program to change the mode! That means if you don't have a supported motherboard you need to find another computer, plug in the drive, boot into DOS from a CD, and hope their crappy DOS tool supports your SATA hardware.

I fully agree, I liked jumpers too.

Re:Open Source to the rescue

[marcansoft]

All CD-ROM drives use 2K sectors, but the issue is mitigated because people don't use regular HDD filesystems on CD-ROMs. DVD drives actually use 64K sectors or something along those lines, but they emulate 2K sectors for legacy reasons (though at least here you don't have read-modify-write issues).

Re:Open Source to the rescue

[marcansoft]

That's weird. I do know that non-SDHC cards can have larger sectors, but I didn't expect it to be a requirement for the 1GB-4GB range. Non-SDHC SD card addressing works on byte offsets (so effectively all block addressing it multiplied by the block size, even though it has to be block aligned anyway), so using larger blocks provides no benefit (it's not like ATA/SATA HDDs where blocks are addressed in units of, well, blocks). In fact, all SDHC did was change the addressing to block units and fix the block size at 512 bytes (besides some initialization sequence changes), so SDHC cards can handle up to 512B x 4G = 2TiB (they currently limit it at 32GiB per spec, but that's not a technical limit).

To answer your question, whether it's visible to the OS would definitely depend on the card reader. It will definitely be visible with raw SD Host Controllers (built-in PCI readers and the like), but USB readers can do all sorts of weird stuff.

Re:Open Source to the rescue

[yuhong]

the remaining die-hard users of antique hardware will be creating adapter hardware that can do the necessary conversion and 512-byte-sector emulation.

In fact, that is exactly what the WD Advanced Format drives are doing right now to be compatible.

Re:Open Source to the rescue

[marcansoft]

No, they're unconditionally crippling themselves in detriment to the OSes that actually could support the newer technology. What WD isn't doing isn't just backwards compatibility, it's decreasing the value of a new feature and increasing confusion by pretending not to have that feature while having hidden performance issues when OSes (rightly) treat it like a normal drive.

The Right Way to do it would be to have a jumper to enable and disable the emulation. Failing that, just default to 4KB mode and let the market come up with 4K -> 512byte emulation adapters if people truly demand them.

Re:Open Source to the rescue

[fuzzywig]

"Windows apologists and their revisionist history are just pathetic." I believe there's a saying about ignorance and malice and their relative occurrences which is of relevance here...

Re:Open Source to the rescue

[Freultwah]

Said drives already have a jumper setting for "XP compatibility". Just bridge pins 7-8 with a regular jumper and you're good to go. I keep wondering why no-one even has even bothered to google it.

Re:Open Source to the rescue

[jesup]

512 byte sectors were known to be a performance bottleneck LONG before the late 90's. At Commodore, I put full support in for any power-of-two HD sector size back around 92-ish (plus the FS supported allocation blocks of powers-of-two multiples of sectors, which most filesystems have for quite a while). Larger sector sizes was a win on performance, much the way larger allocation sizes in the FS was a big win, though more related to low-level overhead. Certainly on the platters larger sector sizes is a win - we experimented with that for floppies, and used a floppy format that got much of the benefit (removal of all inter-sector gaps, replaced with a single gap per track - you lost the ability to write single sectors, but got 10% more storage and faster transfer times - effectively one big sector, but with additional sync/etc headers that let you read data starting at any logical sector, to reduce rotational latency). Similar logic applies to HD platters - larger sectors lets you get more user bits on a track and higher transfer rates, but decreases the granularity for writing data.

Re:Open Source to the rescue

[MarcQuadra]

It's not even in the kernel, so it doesn't matter when the kernel was written. The problem is in userland, the partitioning utility in particular.

'util-linux' hasn't been updated since 2005. How much you want to be your Linux distro is using 'fdisk' from five years ago?

Luckily, someone got busy today and updated the util-linux-ng 'fdisk' to start partitions at 1MB, unless the device is super small and each megabyte matters.

http://git.kernel.org/?p=utils/util-linux-ng/util-linux-ng.git;a=shortlog;h=refs/heads/master

Now all we have to do is have the distros pick it up.

Problem solved. It probably took less than ten minutes of coding. Arguably, it should have been solved years ago. I know I've been aligning my partitions to 64K or 1MB bounds for four years (I got hip to this stuff when I started virtualizing things).

Re:Open Source to the rescue

[orngjce223]

Nope, they're using this issue (one among many, really) to get people to Upgrade(tm). (Can't use that cool new drive you bought? Sorry sucker, go upgrade to Windows UltraVista9 for just $599 a pop!) I'm sure you understand what I mean.

Re:Open Source to the rescue

[badkarmadayaccount]

There are EFI versions of XP you know.

Re:Open Source to the rescue

[badkarmadayaccount]

Different sector sizes? Way cool. What brand and model is it?

Re:Open Source to the rescue

[Pentium100]

Seagate ST34520N

Seatools Enterprise, the software from Seagate allows changing the sector size and changing capacity (to lower than original). Here is a screenshot: http://img40.imageshack.us/img40/2497/blocksize.png

The software also allows changing of various drive settings (automatic bad block reallocation on read and write, read and write caches, number of cache segments). Some settings are grayed out (performance mode desktop/server and acoustic management) presumably because my drive is too old and does not support them. The software only works with Seagate SCSI drives (if you select a non-Seagate or a very old drive it can only perform a drive test, basically verify every single sector).

Re:Open Source to the rescue

[badkarmadayaccount]

Thanks. That's some beautiful software. I wonder if there are modern versions or equivalents...

Re:Open Source to the rescue

[f8l_0e]

Setting multipliers on the motherboard wasn't too bad. If you forgot to set the split core voltage jumper on a socket 7 board when you had a pentium with mmx, now that was a problem.

I don't miss those days much. My life has been much better since I haven't had to worry about IRQ conflicts, setting I/O ports, or fscking with emm386.

So don't do that...

[russotto]

Author claims a massive performance drop if things aren't aligned right. Ubuntu already does it with parted and fdisk can do it manually. So, no big problem; fdisk ought to be fixed to have sane defaults with a 4096 byte block size, sure. That can't be all that difficult.

The author also seems to think that only a 30% increase in times for misaligned writes should be expected. I'm not sure why. In a naive implementation I'd expect a 100% increase in time (each block now needs to be written twice). Linux, obviously, doesn't use a naive implementation. It's expected that if the hardware violates the assumptions behind the techniques Linux uses to achieve high performance, that those techniques end up making things very slow instead.

Re:So don't do that...

[marcansoft]

fdisk doesn't need to be fixed, it needs to be deprecated. DOS partition tables are a ridiculously bad artifact of the past. We won't be using them for much longer anyway; they're limited to 2TB for 512-byte-sector drives (or 4K drives with 512-byte emulation).

Re:So don't do that...

[Anonymous Coward]

Fdisk is so retardedly out-of-date that 4k-block alignment is pretty low on the list of worries. It's fairly easy, so it might get done, but there are much more pressing matters for fdisk -- GPT support, not assuming MBR/DOS as the fall-through label, etc. -- that 4k-blocks are pretty low on the list.

Re:So don't do that...

[Dragoniz3r]

Methinks you're confused about something. fdisk doesn't have anything to do with DOS partition tables. fdisk manipulates the MBR, which is required to locate partitions (of any sort) on the disk, unless you're using an OS/BIOS combination that supports GPTs, which is hardly widespread at this point.

Re:So don't do that...

[marcansoft]

The MBR is the DOS partition table. The BIOS has nothing to do with partitions, as it just launches the code located in the MBR (which is in charge of deciding which partition to boot). GUID Partition Tables are supported by Linux, which is what we're talking about here, so there's no reason not to use them unless you also require a legacy OS.

You seem to think that DOS has some sort of underlying level of partitioning that we no longer use. That's not the case - the MBR partitions used today are still the same old craptacular DOS partitions (with LBA extensions tacked on), and they are managed by fdisk.

If you ever want to use a hard drive over 2TB, the DOS MBR partition table format will not work. You'll have to use GPT or you'll lose all capacity above 2TB. Well, maybe with some hacks (which no doubt will not be well supported by some/most OSes due to 32bit math issues) you'll be able to cram in two partitions of 2TB size each (maximum), but I doubt that'll catch on due to the aforementioned OS support problem, and there's no way of breaking the 4TB barrier with the current format.

Check with your distribution

[macemoneta]

I know that Fedora seems to have addressed this with parted 2.1.1 and util-linux-ng 2.1. Both are scheduled for Fedora 13, but can be pulled into Fedora 12 by those getting the hardware early.

Partitions are obsolete

[Anonymous Coward]

Easiest fix: stop dividing your disks into partitions.

Re:Partitions are obsolete

[Anonymous Coward]

yep just use the whole drive (sda not sda1)

Re:Partitions are obsolete

[jedidiah]

Nope. Read the original article. The first partition will be mis-aligned if you use the first available block rather than padding to the next 4K boundary.

You won't even know it unless you look at the partition table in expert mode.

Re:Partitions are obsolete

[hedwards]

Which is nice if you're wanting to ensure that you've got the lowest possible reliability and safety for your data. While you're at it, make sure you're using a striped non-redundant array of disks as well, best use at least 4 in the array, otherwise you might get some of your data back.

You've got it exactly backwards, people shouldn't be partitioning disks into one huge partition. They should be able to split things up a bit to keep rapidly changing directories from mostly static ones and to manage the risk of filesystem corruption destroying important files.

Re:Partitions are obsolete

[marcansoft]

What the GP is advocating is not using a partition table, instead of partitioning drives into one partition. This doesn't cause the issue mentioned in the article. This isn't a novel idea - there are some USB drives that do this already, and even Windows seems to cope fine.

Re:Partitions are obsolete

[HiThere]

And then when you install a different distribution, you blow away your home directory. Sorry, bad idea. /home should be in a separate partition from the rest of the stuff..

Also, since I usually have several distributions installed at the same time, I have several partitions...but that's a less common problem.

A better solution would be to have a boot partition snuggled up against the MBR that automatically adapts so that the boot + MBR is an appropriate size, say 32 MB. (My current boot directory is 14MB, so that shouldn't be a problem. These aren't, after all, small drives, so it doesn't hurt to allocate a bit of extra space. Maybe even make that 64MB.)

Perhaps one could rearrange the system tables a bit so that the MBR was counted as a part of the /boot partition, and so was the partition table. They'd need to be an a position guaranteed by the OS, but that's not a real problem.

Note that what I'm proposing is a major redesign, so there's about zero chance of it being adopted. But it's a better choice than scrapping partitions, and probably has a better chance of being adopted.

Re:Partitions are obsolete

[Anonymous Coward]

Splitting a disk into multiple pseudo-disks makes sense in many situations, but the clunky legacy partition tables are only good for inter-OS compatibility. Otherwise LVM beats partitions in every respect. Now if only we could get a LVM solution that works in multiple operating systems...

Re:Partitions are obsolete

[BitZtream]

If your worried about one big filesystem going south then I suggest you start using a modern filesystem without those concerns.

We've got well past the point where that should be an issue on any modern system.

Your acting like it's still the 70s, it's not, we've learned a few things since then.

Re:Partitions are obsolete

[Anonymous Coward]

I still do not use LVM on my special purpose workstations and servers because it does not provide the equivalent of MD over partitioned disks. With MD over partitioned disks, I am able to maintain multiple RAID5 or RAID6 volumes across the same disk set, and viably repair and upgrade the hosts with minimal downtime, including replacing disks with larger ones and/or resizing the RAID volumes as needed. For some reason LVM seems to be taking an incredible number of years to adopt any RAID levels beyond mirroring.

Not only does MD over partitions allow for LVM-like management of smaller single-purpose filesystems from a large pool, but it also allows different MD volumes on the same disk set to have different RAID modes, e.g. mirrored /boot, RAID5 or 6 / and /home, and RAID0 /scratch for high performance jobs.

Re:Partitions are obsolete

[Anonymous Coward]

Sorry, bad idea. /home should be in a separate partition from the rest of the stuff..

What if I don't know when I'm partitioning the drive how much space /home is going to need? What if I change my mind later?

Partitions are an ugly workaround to the fact that filesystems sometimes screw up. That's rare enough these days that if they do, you restore from backup - it's just one more low-probability threat like hard drive failure and velociraptor attack.

Re:Partitions are obsolete

[Anonymous Coward]

A better approach is to think how much everything that's NOT /home will need. The vast majority of the time, you can get away with 10GB for the base (Debian user here, YMMV). If you plan for the install to last years, make sure to allow one or two gigabytes more for /var (whether or not it's separate from /). The beauty of linuxs' architecture is that the more programs you have installed, the less space each successive program requires (since it's more likely that you have the requisite libraries already installed).

If you change your mind later, just boot into a live medium (knoppix or a modern distro install disc) or Hiren's Boot CD (questional legality) and resize the partition. You can even use gparted if you require GUI goodness (personally, I like to have as much confirmation as possible when doing something dangerous like resizing, and visual confirmation is pretty powerful).

As for implicitly comparing the likelihood of a hard drive failure to a velociraptor attack? It's always funny to see the hubris of those who haven't yet experienced hard disk failure (and you will experience it).

Oh slashdot..

[JeffSh]

Dear Slashdot,

I've been around for a while. Enough to understand, nay, love the fact that you are linux supporters and all that. But I remain an ardent supporter of truth and speaking in ways which are concise and leads the reader in the direction of truth. Nothing in this news story is inaccurate, but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.

Windows XP may not work with the new WD Green drives, but Vista and on have been perfectly comfortable with 4096 byte sectors. A lay reader may read this story and not "Read between the lines" as I have learned to do here. Their take away may be that Microsoft operating systems are broken in some way (which they are in a lot of ways), but not this one!

Re:Oh slashdot..

[Anonymous Coward]

I think it goes without saying that a major supplier of PC disk drives would make sure that they worked with the current edition of Windows, otherwise they stay in the labs and warehouses until the drivers are ready.

Re:Oh slashdot..

[rubycodez]

nonsense, I guessed (correctly it turns out) from the description that Xp and older versions of windows would have problem. But those of us who keep older computer around for windows, with no interest in going beyond Xp, aren't fretting.

You have an imagined stereotype that isn't true. Here's my stereotyping, Less than 0.1% of readers of slashdot would be "lay readers" in matters of computers so you needn't worry.

Re:Oh slashdot..

[Anonymous Coward]

I think you're nutso. I explicitly understood XP to mean that Vista and 7 were okay.

Re:Oh slashdot..

[Anonymous Coward]

Well, there are standards for web and yet the major browser out there ignores them forcing the web to be coerced into a de facto standard. So if a new hardware arrives and breaks the de facto OS... that is newsworthy, in that most people, who are still using it, will be screwed if they want to upgrade their machine with this new hard disk.

And guess who is going to have to "support" their cries...

Re:Oh slashdot..

[Anonymous Coward]

Some people don't understand the Windows Community is fragmented.

Re:Oh slashdot..

[hduff]

Dear Slashdot,

Nothing in this news story is inaccurate, but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.

You are correct. It is a non-issue because nobody uses WindowsXP anymore and there are no legacy systems running it. Nor will there ever be a need to use a new drive in a WindowsXP stsem since there are no WindowsXP systems in use. You are correct.

Re:Oh slashdot..

[edxwelch]

Maybe Vista and 7 support it, but 70% of all PCs are still using Windows XP.

Re:Oh slashdot..

[Vellmont]

  but to make it a point to say that Windows XP is incompatible with no mention of Vista and 7 being perfectly compatible should be an embarrassment of journalistic integrity.
.
.
Their take away may be that Microsoft operating systems are broken in some way (which they are in a lot of ways), but not this one!

It only takes about 3 brain cells to realize that Windows XP != All Microsoft Operating Systems. Even the average person has more than 3 brain cells.

For those people with less than 3 brain cells, Slashdot still has you covered since the article clearly says:

According to WD, their new 4096-byte sector drives are problematic for Windows XP users but not Linux or most other OSes.

(emphasis mine)
It only takes 2 brain cells to understand that "most other OSes" likely includes Vista and Windows 7.

For those unlucky few with only 1 brain cell, you're correct. Slashdot has certainly failed the 1 brain celled individual.

Re:Oh slashdot..

[maxume]

It isn't really even a problem under XP (as long as the drive is formatted with a tool that is aware of the new sector size, so that the logical sectors of the operating system line up with the physical sectors of the drive; Western Digital, the crazy fools, have a tool that will make sure this is the case...).

Re:Oh slashdot..

[migla]

When the summary explicitly mentions only XP, then obviously all others after that are fine.

Sure, if this was "Gramps weekly Gramps-thing" papyrus gazette, that for some reason had a story about these new drives, then yes, a clarification that Vista and so forth work ok (and an explanation about what the heck a hard drive is in the first place) would be in order.

Re:Oh slashdot..

[Bengie]

now xp is 62% and Vista(20%) and Win7(10%). So about 33% of Windows OS's are 4k aware.

Re:Oh slashdot..

[Anonymous Coward]

While right in the headline it says "_Linux_ not quite ready", which is plainly untrue, and you're complaining about some anti-Windows bias?? W. T. F.

Re:Oh slashdot..

[hduff]

Why does sarcasm get modded as flamebait? Or did it just go over your head?

I just bought one of these

[xhorder]

Can some kind soul tell me specfically what version of what utility I need to use for me to be OK? Or what settings?

My head hurts from trying to understand cylinders and sectors and drive geometry...

thanks!

Re:I just bought one of these

[King Kwame Kilpatric]

The problem is that WD doesn't tell the system about the sector size.
dev/sdd:

Model=WDC WD15EARS-00Z5B1, FwRev=80.00A80, SerialNo=
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=16

It looks to me that this should *really* be fixed by WD with a firmware update

.

Solution: Instead of fdisk, call it as fdisk -H 224 -S 56 as per Theodore Tso's blog.

Re:I just bought one of these

[King Kwame Kilpatric]

Also see this post.

Checking for alignment issues?

[Anonymous Coward]

How can one quickly check for alignment issues?

Re:Checking for alignment issues?

[Anonymous Coward]

Well, if you're lawful good and you come across an old lady and you slice her throat for her coin purse I would say you have an alignment issue.

slashdot is not journalism

[SuperBanana]

should be an embarrassment of journalistic integrity.

Slashvertisements, basic English grammar and spelling problems, completely wrong summaries and titles...

...and you a)think that Slashdot is "journalism" and b)it's had integrity to lose in the first place?

I like Slashdot, but gimme a break...it's a user-driven blog which directs readers to existing stories (now often lagging behind the major news wires) with good categorization and semi-sophisticated commenting system, utilized by a larger commenter population. Not much more, and definitely not journalism.

Re:slashdot is not journalism

[JeffSh]

Yeah, I don't disagree, maybe I'm even well aware of that, but if I had admitted that in my original post I'd have no room to indict the continued problems. I think they should aspire to be better even if Slashdot is everything you say it is.

Re:slashdot is not journalism

[Blakey Rat]

I'm with you, but on the other hand that doesn't mean they should just not give a shit about the quality of their end-product. We know from experience that they can edit and correct stories as corrections arise in the comments, but how often does that happen in practice? (Hardly ever.) Somewhere between a third and half of the stories posted here are either outright lies, or extremely misleading-- I may be exaggerating, but not by much-- and almost never are they corrected.

Look, any site that posts this article: http://tech.slashdot.org/article.pl?sid=09/02/16/2259257 without a single correct simply Does. Not. Give. A. Shit.

I don't think anybody's expecting the New York Times when they visit here, but some minimum level of competence would be nice. I don't fault anybody for complaining.

Re:slashdot is not journalism

[Anonymous Coward]

In you original post you write "But I remain an ardent supporter of truth and speaking in ways which are concise and leads the reader in the direction of truth". Then when someone points out how your whole point does not apply to slashdot you come back with "maybe I'm even well aware of that, but if I had admitted that in my original post".

So you won't admit a truth you are aware of if it gets in the way, while calling out slashdot for not having integrity for not mentioning something.

Re:slashdot is not journalism

[Anonymous Coward]

Any regular visitor must know that complains won't make Slashdot change, so anyone who bothers to complain must either be new here, be a hopeless optimist or insane*.

* As the saying goes: "the definition of insanity is repeating the same action and expecting a different outcome".

Re:if vista/win7 really do support this correctly.

[walshy007]

The real problem is that it is lying about it's sector size, it's reporting 512 bytes when it's using 4k, if it told linux it was using 4k everything would be fine and dandy.

Why does it lie about it's sector size when it doesn't need to? because if it didn't the drives would not work on windows XP at all. Which would not bode well for sales.

Once drives with 4k sectors arrive its up the individual maintainers of each affected tool (fdisk, et. al.) to update their code.

Kernel handles sector sizes, and could handle 4k sectors ages ago, but when the hardware reports something it tends to trust it, which is now apparent it shouldn't. (512 byte sectors being implemented as an emulation layer of sorts on these drives.. and enabled by default)

Drive lies and future fixes

[Sits]

There is an excellent thread talking about how recent (2.6.31+) linux kernels try to report the underlying hard drive architecture (found via the OSNews comments). Alas, it looks like some of these drives are not reporting this data correctly and thus automatic adjustment (at partitioning time) is not taking place. It looks like in the future rather than trying to do detection by reported capability fdisk (and hopefully gparted) will default to sectors of 1MiB if the topology can't be found by default (unless your media is small).

Additionally, I gather that recent Fedoras will try to adjust things like LVM to match larger sectors too. Hopefully whatever is laying out LVM will also be fixed too.

Coincidentally, it looks like Oracle have a very committed dev trying to make this stuff work by default...

This only effects the newer 1TB+ WD Green drives

[HouseOfMisterE]

It appears that this does not effect the older 1TB+ Western Digital Green drives such as the WDC10EADS. Those use 333GB platters and are native 512-byte sectors. The newer (newest) Western Green drives, like the WDC10EARS, use 500MB platters and have 4K sectors. One way to tell the drives apart with a quick glance is the old Green drives had 32MB of cache and the new ones have 64MB of cache.

Re:if vista/win7 really do support this correctly.

[ArghBlarg]

I see it rather as an indictment against closed-source OSes, if XP turns out to be incompatible with these new drives and MS never releases a patch to add support. People will need to upgrade for no good reason to one of MS's new operating systems. People should not have to deal with a complete upheaval of their tested and true systems due to a small hardware change such as this.

I can imagine MS is quietly chuckling with glee to itself, if this issue becomes a deal-breaker for machines still running XP.

A minor correction...

[HouseOfMisterE]

I posted that the newer WD Green drives use 500MB platters and I meant 500GB. 500MB platters would make for a very physically-large 1TB+ hard drive!

I was worried about this... and am still unclear

[bmajik]

I just got one of the 1TB 64mb WD drives that is known to be 4kb sector based.

Here is how it shows up in dmesg:
[ 3.420488] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)

and here's what hdparm -I says:
ATA device, with non-removable media
Model Number: WDC WD10EARS-00Y5B1
Serial Number: WD-WCAV55227529
Firmware Revision: 80.00A80
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 1953525168
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 953869 MBytes
device size with M = 1000*1000: 1000204 MBytes (1000 GB)
cache/buffer size = unknown
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 1
Recommended acoustic management value: 128, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_B

I have such a problem

[orkysoft]

I have a tiny 1.8" usb harddisk with 4096-byte sectors, and the Ubuntu installer crashes when it tries to read the partitioning information. Very annoying.

DragonFly's solution

[m.dillon]

We're adjusting our disklabel64 utility and kernel support to set the partition base offset such that it is physically aligned instead of slice-aligned, and we are using 32K alignment. That should fix the problem without having to mess around with fdisk.

The DragonFly 64-bit disklabel structure uses 64-bit byte offsets instead of sector addressing to specify everything. It ensures things are at least sector aligned but we wanted to make disk images more portable across devices with potentially different sector sizes. The HAMMER fs uses byte-granular addressing for the same reason, 16K aligned.

-Matt

Re:DragonFly's solution

[bertok]

We're adjusting our disklabel64 utility and kernel support to set the partition base offset such that it is physically aligned instead of slice-aligned, and we are using 32K alignment. That should fix the problem without having to mess around with fdisk.

The DragonFly 64-bit disklabel structure uses 64-bit byte offsets instead of sector addressing to specify everything. It ensures things are at least sector aligned but we wanted to make disk images more portable across devices with potentially different sector sizes. The HAMMER fs uses byte-granular addressing for the same reason, 16K aligned.

-Matt

You should use 64K alignment at a minimum, almost all RAID and SAN volumes are 64K aligned with IO sizes of 64K minimum.

You'll lose up to 50% performance on random IOs on any server class hardware.

As a comparison, Windows now aligns to the nearest 1MB.

Re:DragonFly's solution

[Just Some Guy]

We're adjusting our disklabel64 utility and kernel support to set the partition base offset such that it is physically aligned instead of slice-aligned, and we are using 32K alignment.

Darn bloated OS wasting 2% off the front of my floppy drive.

partition table

[gd2shoe]

There won't be a partition table with his suggestion. The boot sector set aside by the filesystem will be the very first sector of the disk.

Poorly researched article.

[Vellmont]

The article represents one data point, for one particular way to install a drive, on one (un-named) version of Gentoo, on one particular model of a WD drive that had a bugzilla entry entered by the author all of 2 days ago. So this is supposed to be an indictment of all of Linux?

The author even mentions that Ubuntu has an option on parted that accomplishes the task properly. I'd be much more interested in an article that talks about how the default installer handles this task rather than concentrating on one particular expert tool that does so. It's still good to know that fdisk on his un-named Gentoo distribution does the wrong thing.. but this hardly means we should fire up the klaxon and declare "Linux not fully prepared for 4096 sector hard drives!". It's certainly interesting, but I'll withhold judgment until we actually know more about the implications of this across the entire spectrum of Linux distributions and the various 4096 sector HDs.

Re:Poorly researched article.

[Radtoo]

I agree with the headlines being grossly misleading. Linux does support 4k block sizes just fine. But this is not a distro-specific issue, so you are wrong, too.
This is simply a matter of fdisk from that version of util-linux-ng (which is clearly named in the article) trusting the hardware vendor to specify correct block sizes. The vendor did not. Thus fdisk does not end up with 4k block sizes, as happens for many programs. And only(?) parted apparently contains a workaround that detects the correct block size.

Its not that you can't use parted on Gentoo, though, it is just that in the world of user choices that is Gentoo, not everyone will be using that program or that particular option.

Re:Poorly researched article.

[Vellmont]

But this is not a distro-specific issue, so you are wrong, too.

I never made any claims about this being a distro-specific issue, or not being a distro-specific issue. The only point I'm trying to get across is the article is extraordinarily narrow in what it's actually tested.

Re:Poorly researched article.

[Vellmont]

Nice to see you criticizing people for "poor" research when you hardly done any yourself.

Why is it some people just don't seem to understand the concept of arguing for not knowing something? I DID do research, I read the article quite carefully and determined the guy didn't take into account multiple different things, and that his claim was far too broad. If you can't understand that that involves research and analysis, I can't help you.

Gentoo has no versions. Nor cutesy names like Ubuntu. It is a source-based distro and everything is compiled on installation so doesn't need this careful versioning nonsense.

For a distribution that you claim doesn't have any versions, they sure do have a lot of announcements claiming new versions: (from distrowatch)

2009-10-05: Distribution Release: Gentoo Linux 10.0
    2009-09-27: Development Release: Gentoo Linux 10.0 Test
    2008-07-06: Distribution Release: Gentoo Linux 2008.0
    2008-04-29: Development Release: Gentoo Linux 2008.0 Beta 2
    2008-04-01: Development Release: Gentoo Linux 2008.0 Beta 1
    2007-05-07: Distribution Release: Gentoo Linux 2007.0

It is not just one version of fdisk on some backwater 15 year old distribution

Funny, I don't recall making any claims this was one version of fdisk. If I recall I only said the article referenced one version.

Re:Poorly researched article.

[Theovon]

I wrote the linked article.

I completely agree that the article is narrowly focused. VERY narrow. My objective was to demonstrate a problem and point out that Linux has not FULLY adapted. I didn't say Linux devs were idiots or that it would never be ready. I was trying to express the idea that Linux [distros in general but perhaps not all] is not QUITE ready for these drives, because not all the tools have fully adapted. Some tools make no mention of any problems in their man pages. Some (like parted's defaults) are even misleading if you mistakenly think that "track aligned" is a good thing.

And I was trying to do that in the very limited number of words I had available for a title.

Also, WD claimed that Linux is unaffected. Some distros probably are, but this could lead people to believe that the statement is universally true, which it isn't. Thus, my over-all objective is to educate people to the fact that if they don't know what they're doing, they can get this wrong. There are lots of mistakes I've made where I wished that someone had mentioned some critical fact on a how-to (like, don't use dmraid/fakeraid for RAID1 because reads aren't load-balanced; use mdraid instead). I've filed plenty of bug reports on such issues.

Re:Poorly researched article.

[Vellmont]

I can't over-emphasize the importance of titles in communication, especially with complex technical subject where there's a lot of evidence presented to support a conclusion. Your title colors the rest of the article and creates expectations about what you're trying to say. When people read articles (especially on the web) they scan through them trying to find the important parts. That's been demonstrated through eye-tracking studies multiple times.

Your title was very broad, but the evidence to support it was very narrow (as you admit yourself). Since your article only referenced Gentoo, and installing a drive after installation using fdisk, why not: "fdisk not ready for 4096 byte sector drives?" If you wanted to cover all of "linux" (whatever that is), why not research other distributions and see how they handle the job, installing the OS flat out on several distro's and see how they perform. If you didn't want to do the research, you could have written in an inquisitive style, i.e. "Is linux ready for 4096 byte sector drives?" and then presented your evidence and talked about how fdisk didn't do this, but parted worked a bit better, and how the drives reported 512 sectors but weren't, etc, but you've only tried Gentoo, and maybe others should try other distributions with other tools.

I think the experimental aspects you did were great, and I'm glad to know that some of the tools out there don't fully work with this particular 4096 byte sector drive. I just don't think you've done enough research to say much more than the problem affecs Gentoo, on these specific WD drives.

Re:Poorly researched article.

[jrumney]

one particular way to install a drive, on one (un-named) version of Gentoo, on one particular model of a WD drive that had a bugzilla entry entered by the author all of 2 days ago

It's the default way to install a drive on just about every Linux distribution out there. I bought one of these drives unsuspectingly last week to replace a drive which had died, and installed Ubuntu 9.10 on it. Sure enough, my root partition starts on sector 63, and every other partition is also on an odd boundary. So is there any utility for Linux that can do what the Windows utility alluded to in TFA does and shift all my painstakingly restored data by 512B, or am I stuck with reformatting again and restoring from backup yet again?

Re:Poorly researched article.

[dwinks616]

Gentoo has no versions. Perhaps in the future it would be beneficial to actually do some research before placing your foot in your mouth. The gentoo INSTALL DISC has versions...however, that said, it's completely feasible to install gentoo with the 2007 disc and emerge your system to the current stable versions of everything. You will of course have a faster, easier time using a newer installation disc, but gentoo, once installed, has NO VERSION.

Simple Solution

[Khyber]

Don't partition the drive in XP - format the entire thing and don't split it apart. Get a secondary physical drive.

Re:if vista/win7 really do support this correctly.

[earthforce_1]

I wouldn't be too fond of the MS development model from what I hear from those who were on the inside:
http://www.nytimes.com/2010/02/04/opinion/04brass.html?pagewanted=all

Inside Microsoft, political infighting trumps common sense. If you really want to hold up a closed source development model as an example of "what works" take a look at Apple. They crank out far better products with a fraction of the resources.

Re:if vista/win7 really do support this correctly.

[thisissilly]

It seems these drives need a new "don't lie to me, I can handle it" command, so OSes that don't have a problem with 4k size sectors can get the real info.

Other possible consequence of misalignment

[Anonymous Coward]

I didn't notice a performance drop in Linux when using such a 4k sector disk with misaligned partitions, but random stalls lasting up to about a minute. Some kernel threads (like md?_raid1, kdmflush) were in an uninterruptible sleep, and the hd led was continuously lit. It seemed like no data transfer happened at all. And after some time the disk seemed to work again as it should.

After aligning the partitions, by using 56 sectors per track, the disk seems to work flawlessly. Maybe it also works faster now, but I did not check it.

Re:Other possible consequence of misalignment

[King Kwame Kilpatric]

I noticed both a performance hit AND stalling for a minute at a time when there was a lot of HDD activity, so I can confirm part of your experience. After going with 56 sectors per track as well, the freezing seems to be a thing of the past. The speed is definitely greatly improved.

Re:Other possible consequence of misalignment

[Anonymous Coward]

The performance could've been hampered on my system due to the misalignment, but my concern was the stalls which gave me much more trouble. And maybe the performance hit was masked by using a raid1 setup, with a 4k and 512B sector disk.

Now the speed seems to be improved as well, but I did not measure. I'm happy to see that you saw the stalling disappear as well, it gives me confidence about the cause.

Re:Other possible consequence of misalignment

[King Kwame Kilpatric]

I'm happy to see that you saw the stalling disappear as well, it gives me confidence about the cause.

Time will tell if the the lock-ups were caused by the 512b partitioning (I wasn't able to repartition my drives until a few days ago), but it's a bit reassuring to know that I wasn't the only one experiencing these annoying as fsck freezes.

The affected drives on my end are WDC WD10EACS-00ZJB0 and WDC WD10EADS-00L5B

.

How do I align sectors on my drive

[Anonymous Coward]

I have two WD Green drives that each have one big ext4 partition on them, and they appear to be nonaligned (the partition starts on sector 63). Is there a simple way to align the partition to the 4k sectors?

Re:This only effects the newer 1TB+ WD Green drive

[King Kwame Kilpatric]

The affected drives are listed on Western Digital's site.

gdisk

[Anonymous Coward]

You shouldn't be using fdisk anyway; the MBR partition table is a dinosaur compared to GPT. Only Microsoft OSs still rely on the MBR, all others can handle GPT partitions without issue.

Its replacement (direct replacement), same interface, same feature set:
http://www.rodsbooks.com/gdisk/walkthrough.html

And yes, gdisk does handle sector alingment. It defaults to 4k-alignment on all drives larger than 800GB. The default can be changed, of course.

Re:I was worried about this... and am still unclea

[King Kwame Kilpatric]

(I think that) It's probably misaligned. LVM uses a 192k sector size for it's metadata. See Theodore Ts'o's post for more information.

I ain't your bitch, so fix it yourself

[Anonymous Coward]

For the record: I love my job (tinkering with Linux) but I hate my customers (Linuz zealots who do nothing practical but cheerlead on the sides).
Screw you, little fucks.

---
no-name kernel hacker.

parted solution for these drives

[Anonymous Coward]

Posted by Aleksander Adamowski to this thread on the util-linux mailing list: http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/2926 ... "So for any other owners of WD EARS drives, if these don't report
physical 4096-byte sectors to you, don't believe them and align your
partitions at the aforementioned sectors (a generally good idea is to
run the postmark benchark to compare performance on aligned and
non-aligned partitions).

Just in case anyone doesn't know how to align these partitions
(WARNING: the instructions below will likely destroy any data that's
on the given drive, only do this with drives you're intending to
erase):

# parted /dev/YOUR_DEVICE_NAME

(parted) mklabel gpt
# Here ^ I've chosen the GPT partition table format, but others may be
OK too - untested by me.

(parted) unit s
# Here ^ we're choosing sectors as units of measurement

(parted) mkpart primary ext2 40 -1
# Here ^ we're creating a partition that starts at sector 40, which is
divisible by 8.
# You can also try 48, 56, 64 and others - these should offer the same
high performance,
# but some space will go to waste - it's only some tiny kilobytes, though.

# Parted will likely complain about the end location of the ending sector:
Warning: You requested a partition from 40s to 2930277167s.
The closest location we can manage is 40s to 2930277134s.
Is this still acceptable to you?
Yes/No?
# Of course, we answer Yes.

(parted) quit

# After that, create a filesystem as usual, e.g:
# mkfs.ext4 -T largefile4 /dev/YOUR_DEVICE_NAME

This should get the optimum performance from your 4 kB physical sector
drives even when they report 512 B sectors only to the OS."

I had this issue bite me in the ass when building an OpenFiler-based home NAS this week. The real kicker is that the drive label specifically states a need to jumper two pins on the drive and align your partitions when using WinXP, but "all other" operating systems are good-to-go with no intervention required. Maybe that would be the case if the drive weren't misreporting its sector size to the kernel, but I should've known better regardless.

RTFM please

[Anonymous Coward]

I would be soooo embarrassed if it was me who wrote that...

It has been suggested that WD might internally offset block addresses by 1 so that LBA 63 maps to LBA 64. This way, Windows XP partitions would not really be misaligned. I performed a test that demonstrates that WD has not done this

Or you could read the label on the drive: "Windows XP, single partition: set jumpers 7-8 prior to installation".

For /dev/sdd, I used fdisk to add a Linux (0x83) primary partition, taking up the whole disk, using fdisk defaults. By default, the partition starts at LBA 63.

So... you're a technical writer, you're blatantly ignoring WD's recommended practices, and you're still blaming all of Linux based on the intentional misuse of one single tool? The author might have had a point if fdisk was the only tool to create partitions. The reality is that I don't know a single graphical installer that uses fdisk. He even acknowledges this by mentioning Ubuntu, but apparently did not care enough to do a little research into other installers:
openSuSE: Yast partitioning
Fedora: anaconda (libparted?)
Debian: partman (libparted)
Slackware: cfdisk (?)

So, to sum up: most mainstream distros use a tool from this century. The only places where you might find fdisk is in text-mode installers, and those are mostly used by skilled technical people (but also by bad article writers, apparently). Of course, I'm not saying that the libparted-based installers perform any better in this respect, but neither is he. However, that would be an article worthy to write.

Since this is one large file, and it can be written linearly to the disk, I expected that we would see a very slight performance hit. I think this is something that itself should be investigated. There's no reason for long contiguous writes to get hit this hard, and it's something that the kernel developers need to look into and fix.

How does the author know that it's a contiguous write? How full was the destination partition? What filesystem was used (extents-based, journalled)? What cache writeback mode was used? -1 for suggesting a kernel bug without giving enough detail to support that accusation.

Timothy Miller is a Ph.D. student at The Ohio State University, specializing in Computer Architecture, and Artificial Intelligence

I'm not used to looking for a hidden agenda among technical people... but I wonder what his angle is here.

linux is not ready because of fdisk?

[pizzap]

fdisk an elegant tool for a more civilized age.. no wait. fdisk is antiquated and we only use it, because we are afraid to leave the msdos partition table behind out of the irrational fear some other software would stop working.

not all 4K are created equal

[dltaylor]

There are four flavors of 4096 byte-sectored drives:

4096 physical/logical - the bookkeeping parts of the file system cause read/modify write cycles because they are nearly always less than 4096 bytes, but the performance hit is relatively small; parted is badly broken. If they're less than 2TiB, then you can use an MBR, otherwise the kernel is broken for partition sizes.

4096 physical/512 logical; LBA 0 aligned "off by one" with physical block 0 - created to deal with stupid BIOS (and Win XP, where some drivers rely on it), mostly work fine with the default tools, but still have the bookkeeping issues. Because Win Vista/7 and OS X use GPT AND don't worry about "track" boundaries, they work better than Linux.

4096 physical/512 logical; LBA 0 aligned with physical block 0 - works great with Win Vista/7 and OS X, but the Linux installers are still aligning on the bogus track boundary, and not asking the physical/logical alignment. Performance, without some very smart tweaking by the person doing the formatting REALLY stinks.

4096 physical/512 logical, but are reporting 512 physical (usually aligned 0 for 0) - again to deal with BIOS/Win XP. Basically, treat ALL drives produced starting with 2010 as having 4K sectors, aligned 0 for 0, unless they explicitly report otherwise, and use the same human-intervention-required layout as above.

Currently, the tools are the most pressing issue, since they are really broken in this respect, but there are kernel issues, as well, with drives larger that 2TiB and 4096-byte sectors.

All of a sudden?

[Waccoon]

I had never ever heard of drive alignment until I bought an SSD.

Not to be unhelpfully pessimistic, but... didn't it even occur to people that drive alignment might be important? Is 4K drive alignment just Y2K for hard drives? Why did people only start thinking about this now?

a firmware update isn't realistic

[YesIAmAScript]

I'll get to why in a second, but first:

RawCHS hasn't meant anything in a decade. The largest drive you can describe with CHS is 8GB.

Track size hasn't meant anything in even longer than that. When drives went to zone bit recording (ZBR), the number of sectors per track became variable. This happened in about 1989.

The sector size does mean something, but it is the actual sector size, not the sector "grouping" size. If the drive reported a sector size of 4K, then it would expect that the host understand that sectors are actually 4K in size, not 512B in size. But really no major OS supports this, they all expect 512B sectors. That's why these drives internally use one sector size and show another size to the host. And there is no way in the ATA specification for devices to indicate their internal sector size when they are presenting a different external sector size.

So this won't be fixed with a firmware update, unless Vista, 7 and every other major OS is fixed to actually support large sectors presented to the host. Then the drive could be firmware updated to report the large sector size to the host. And the drive would then be completely unusable under any earlier OS or with any USB or Fireware adapter.

whoops, one thing about RawCHS

[YesIAmAScript]

I forgot, there is one thing RawCHS nowadays. That is that there is no proper spec for how to know if a partition in an MBR (fdisk) partition table is a valid partition. So there are heuristics that are applied to the entries to guess if they are real or to be ignored as empty. One of the heuristics that some software uses is to ignore all partition entries that don't begin on a cylinder boundary. To be on a cylinder boundary, the partition has to start on a sector number that is a multiple of the number of sectors (S in CHS) in order to be valid. And since all drives 8GB or greater present an S of 63, that is why the first partition on an MBR disk has always started at sector 63, which makes it unaligned when the internal sector size is 4K (8 internal sectors).

Windows before 2000 checks the CHS alignment of MBR entries and ignores any partition entries that don't start on a multiple of S. So all disks out there are misaligned. With Windows 2000 or later, you can start the partition on any boundary you want.

Western Digital has a jumper you can put on the drive that adds 1 to all access requests, making all those misaligned first partitions aligned. But it'll also make any aligned partitions misaligned. So the real answer is just to layout your disk different. I would recommend using GUID disk partitioning instead of MBR anyway, because MBR doesn't work for >2TB drives. And GUID doesn't have any weird alignment requirements (and doesn't have any knowledge of CHS).

Re:whoops, one thing about RawCHS

[hpa]

The original reason for aligning to track boundaries (a track is a cylinder-head pair) is that the first four sectors of MS-DOS' IO.SYS (IBMBIO.SYS) had to be contiguous and on a single track.

Re:whoops, one thing about RawCHS

[yuhong]

Yep, in the old DOS versions there was even more restrictions.

Re:I was worried about this... and am still unclea

[Anonymous Coward]

192kB % 4kB == 0
So LVM/PV directly on a the full disk is fine for 4 kB disks as per original post, but will be misaligned for all SSDs (and USB flash).

SLC disks tend to have 128 kB erase block size, MLC 256 kB but some may have 1 MB. If you're really paranoid use some overkill alignment like 16 MB, it's not like you loose measurable amount of real space with it.

You can also run into the same issue with some RAID implementation, including Linux software RAID if you use superblock format 1.1 and 1.2, note that 1.1 was recently made the default! Good intentions (to avoid one specific issue) with bad results on at least SSD and possibly 4 kB disks too.

linux not quite ready for pseudo 512bsector drives

[Anonymous Coward]

clearly WD's fault.

anyone aware of 4k-sector _desktop_ drives, that have a real 4k interface?
(some new samsung 2,5" and 1,8" hdds are made like that according to heise's ct (german magazine))

C/H/S Needs to go

[LostMyBeaver]

Cylinders, head, sector addressing really needs to go anyway. MBR has been hacked so many times over the years to support this method of formatting but in reality it's a total waste. Old operating systems that were forced to work with this method of addressing had to translate to and from the CHS format since sector allocation has ALWAYS been linear. Even FAT-12 used linear addressing methods.

CHS is also misleading and CHS optimizations are wasteful. Since all modern drives (starting with the first Connor IDE 20 meg drive in the early days) supported some form of intelligent sector remapping that would keep spare sectors available for relocating data after the magnetic medium of a heavily used sector elsewhere began to fail.

Sector remapping makes it so that CHS optimizations are entirely irrelevant since even brand new drives, straight off the production line ship with bad sectors that have been remapped elsewhere. For better drive performance, algorithms space out the spare sectors across the drive so that when accessing a spare sector, the head doesn't have to slam to the inner or outer rings of the disc. But still, CHS doesn't apply to absolute positions anymore.

This is 2010 now, I wrote file systems which functioned on 4096 byte sectors on ESDI drives back in the 80's. Made my drives much bigger doing it too. It's time that we move to larger sector sizes again. Modern ECC isn't that much better than 80's grade, however the processing power available to us is so much more that performing ECC on larger blocks of data is achievable. Also, using RAID-5, 5EE or RAID6 makes it so that we can depend less on single drive redundancy. SCSI and IDE should be extended so that controller can inform the drive of bad sectors it finds when performing RAID XORing.

WD has a solution

[a-zA-Z0-9$_. !*'(),x]

http://www.wdc.com/en/products/advancedformat/

WD Align software aligns partitions on the Advanced Format drive to ensure it provides full performance for certain configurations.

Re:WD has a solution

[King Kwame Kilpatric]

http://www.wdc.com/en/products/advancedformat/

WD Align software aligns partitions on the Advanced Format drive to ensure it provides full performance for certain configurations.

Yes, certain configurations: XP. According to WD, Linux is not affected by this (but that's been proven to be false).

Doesn't it mean MS did something right?

[Anonymous Coward]

It's a classic problem in computer marketing: don't make your product too good (or good enough).

Re:if vista/win7 really do support this correctly.

[PsychicX]

Yep, right until the NEXT feature comes out that breaks the OS that says it can handle the truth. Now you need TWO "don't lie to me" commands. Repeat ad infinitum.

Re:I was worried about this... and am still unclea

[Anonymous Coward]

There is a jumper on the drive that you have to set to disable the 512 byte sector emulation.

Doesn't need a kernel tweak at all.

[MarcQuadra]

All that needs to happen is a very minor update to the 'fdisk' utility to make it start partitions at 1MB. Microsoft saw this coming and changed the partitioner in Vista (and newer) to align on 1MB boundaries.

There don't have to be kernel tweaks. I've been doing this myself for years when I install Linux or Windows. Windows XP's 'diskpart' aligns by 32K, whereas the built-in format in setup from the CD aligns on 31.5K, so just format the drive from BartPE's diskpart utility first. In Linux you just drop into 'expert' mode in fdisk, hit 'x' and tell your partition to start at 2048 sectors (1MB).

Why 1MB instead of 32K or 64K? Because some RAID arrays have stripe sizes up to 1MB, and it just makes sense to 'waste' one meg in order to have alignment work on bounds of 4K, 16K, 32K, 64K, 128K, 256K, and 1MB (all common page or RAID stripe sizes).

Should I be concerned?

[busydoingnothing]

I just bought two of the WD Green 500GB drives to be used in a hardware RAID (Adaptec 2610SA, aka Dell CERC SATA1.5/6ch) on my Ubuntu-based server. I was going to format it in ext3. Will this problem affect me?

Re:Should I be concerned?

[King Kwame Kilpatric]

I just bought two of the WD Green 500GB drives to be used in a hardware RAID (Adaptec 2610SA, aka Dell CERC SATA1.5/6ch) on my Ubuntu-based server. I was going to format it in ext3. Will this problem affect me?

You should be immune. It seems that only >640GB drives have this problem. See the affected drives.

Re:if vista/win7 really do support this correctly.

[Anonymous Coward]

It seems these drives need a new "don't lie to me, I can handle it" command, so OSes that don't have a problem with 4k size sectors can get the real info.

Actually not a command, but a jumper. But yes, I don't understand either why the drives do not have such a setting.

This is not the first time drives have had backward compatibility problems, and on earlier occasions the drives usually had a jumper to set the drive in a compatibility mode - for example to clip the capacity at 32 GB when installing in an old computer which could not handle drives larger than 32 GB.