Hope For Fixing Longstanding Linux I/O Wait Bug

DaGoodBoy writes "There has been a long standing performance bug in Linux since 2.6.18 that has been responsible for lagging interactivity and poor system performance across all architectures. It has been notoriously difficult to qualify and isolate, but in the last few days someone has finally gotten a repeatable test case! Turns out the problem may not even be disk related, since the test case triggers the bug only by transferring data either between two processes or threads. The test results are very revealing. The developer ran regressions all the way back to version 2.6.15 that demonstrate this bug has more than doubled the time to run the test in 2.6.28. Many, many people working at improving the desktop performance of Linux will be very happy to see this bug die. I know that I, personally, will find a way to send the guy that found this test case his beverage of choice in thanks. Please spread the word and bring some attention to this issue so we can get it fixed!"

Comments

Dang!!

[camperdave]

Dang! I was going for First Post, but my machine was stuck in some weird I/O wait state.

Is this bug currently affecting ....

[whoever57]

bugzilla.kernel.org?

Re:Is this bug currently affecting ....

[2Bits]

With the current response time, obviously, yes.

Re:Is this bug currently affecting ....

[ArsonSmith]

Given enough eyeballs, all bug tracking software is fragile

Longstanding...Since 2.6.18

[akpoff]

Right. I had to get up in the morning at ten o'clock at night, half an hour before I went to bed, eat a lump of cold poison, work twenty-nine hours a day down mill, and pay mill owner for permission to come to work, and when we got home, our Dad would kill us, and dance about on our graves singing "Hallelujah." --Monty Python: Four Yorkshiremen

Been waiting all of 2 years and change for your precious bug fix, 'ave you? You almost had my eyes tearing up there I tell ya: 25 Year Old BSD Bug.

Desktop???

[corychristison]

I'm not sure about anybody else here, but I was surprised to see that they mentioned that this will benefit 'Desktop' users.

I think that when it comes to the performance spectrum, Servers would be where this fix is the most needed. Admittedly if you are running a solid server, you should know to use older gen hardware and software that has been proven to be stable. However, some of this 'shiny new' tech coming out is appealing.

How about the Seagate 1500GB drive hang error? To my understanding Windows has been fixed, but the problem still persists in Linux. Could this potentially make a difference? I've been looking to build myself a nice NAS and those 1500GB drives are _cheap_. I can pick one up for about $160. I remember not too long ago that could only get me 80GB.

Re:Desktop???

[Anonymous Coward]

I believe the 1.5tb Seagate linux hang has been fixed. We're using a lot of them (100's) where I work on Ubuntu Hardy servers and haven't had hangs.

Re:Desktop???

[NormalVisual]

not HAL-9000 intelligence, which would be bad for data anyway

HAL-9K intelligence doesn't pose any problems to the data - it's the *operators* that need to be concerned, especially when giving the system instructions that could potentially conflict with each other.

Re:Desktop???

[cowbutt]

How about the Seagate 1500GB drive hang error? To my understanding Windows has been fixed, but the problem still persists in Linux.

The ST31500341AS requires a firmware update from Seagate to something newer than revision SD19 (more info). In the meantime, if you're using a drive which hasn't been updated to fixed firmware, there's a blacklist in the current development kernel to disable NCQ on affected models as a workaround.

Killing kernel.org server isn't very nice...

[Anonymous Coward]

I'm sure kernel.org appreciates these links. Now instead of fixing the bug they're putting out fires in the data center...great job slashdot.

Re:Killing kernel.org server isn't very nice...

[statusbar]

I'm sure kernel.org appreciates these links. Now instead of fixing the bug they're putting out fires in the data center...great job slashdot.

Well, maybe the kernel developers or bugzilla developers could use the practice in making a reliable scalable system out of the systems that they design.

--jeffk++

Windows Port?

[Al Al Cool J]

If this get resolved is there any chance the fix could get ported to Windows? I just had my Dad's XP laptop completely freeze after I plugged in a bog-basic USB thumbdrive. The desktop sprang to life only after I unplugged it. I wish some of the AC Windows fanboys who were hassling me here last week were around to see it. "Ready for the desktop" my ass.

Re:Just upgrade

[martinw89]

OS not fast enough? Just upgrade your hardware components, preferably to a new, top-of-the-line system.

Oh wait... that's the Windows way of doing things.

Yeah, exactly, that's why volunteers have been hard at work to find and fix the (published, admitted) bug. Just like Win... Oh, wait.

this is bad even for /.

[Harik]

wow, not just badsummary, utterly worthless summary. Here's the relevant discussion from LKML. Yes, this is all of it.

Peter Zijstra

Andrew Morton
In http://bugzilla.kernel.org/show_bug.cgi?id=12309 the reporters have
identified what appears to be a sched-related performance regression.
A fairly long-term one - post-2.6.18, perhaps.

Testcase code has been added today. Could someone please take a look
sometime?

There appear to be two different bug reports in there. One about iowait,
and one I'm not quite sure what it is about.

The second thing shows some numbers and a test case, but I fail to see
what the problem is with it.

This somewhat deflates the excitement evident in the OP. I mean, I know what he's talking about, these apparently random 1-2 second FREEZES while working, but if the guys in LKML arn't talking about it it's probably not being really worked on.

Re:this is bad even for /.

[bjourne]

If you haven't used Linux regularly within the last two years, you probably have not noticed that the system has gotten significantly slower with more recent releases. The probable symptom was discussed here. Many Ubuntu users, including me, have noticed that the latency of desktop operations got significantly larger around the time Gutsy was released, which coincides with the Completely Fair Scheduler and kernel upgrade from 2.6.18.

Since it is most likely a latency issue, the problem is extremely hard to diagnose. Alt-tabbing between programs seem a little slower, keyboard input might lag somewhat. You can't measure desktop latency easily.

Re:this is bad even for /.

[CAIMLAS]

Yep, this is a petty big problem - an easily reproducible one - and it's been around for a really long time. I don't remember when exactly it came about, but I moved from Debian Sid to Ubuntu 7.x about 8 months ago. I didn't have any problem under debian, and I'm uncertain whether the 7.x ubuntus had the problem, but I certainly noticed it in 8.x releases.

I do recall a bit of a somewhat gradual progression of desktop performance decreases, though, going all the way back to the later 2.0 kernels. Back then, the schedulers would all allow an at-the-time relatively slow machine run a fairly bloaty window manager (like E16) responsively while untarring an archive and running a kernel build at the same time - provided there was 100+Mb or so of RAM for the process, of course. Even still, if you were to dip into swap, the UI would remain pretty responsive. Not anymore.

The way things sit now, the Linux I/O scheduler results in desktop performance similar to Windows XP during I/O ops. That is completely unacceptable.

Part of me thinks this is due to a server-centric focus in development (being as the people doing kernel dev largely work for corporations who want server kernels), but I'm not really in the know. If that's the case, we really need to pull one of the old desktop schedulers out of retirement and use that instead of what we've got now, at least for the desktop, and maintain two different-focus schedulers within the kernel instead of just having a couple generally-suited schedulers.

Re:this is bad even for /.

[kwabbles]

"Many Ubuntu users, including me, have noticed that the latency of desktop operations got significantly larger around the time Gutsy was released, which coincides with the Completely Fair Scheduler and kernel upgrade from 2.6.18."

Uhh.. I didn't see anything in there about the Complete Fair Queuing - you just mentioned Completely Fair Scheduler, then kernel 2.6.18.

"Feisty had the 2.6.18 kernel and was quite responsive, so CFQ is in the clear. Gutsy featured 2.6.23 with CFS and was much slower which means it is a possible suspect."

This performance bug has been reported since 2.6.18.

Re:funny

[iluvcapra]

I trrrrrrrrrrrrrrranssssssssfer data betwwwwwwwwwwwwwwwwwwwwwween threads alllllll the time......

Re:Just upgrade

[El Lobo]

Sure, because every Windows developer is a lazy motherfucker that doesn't like his work and plays Solitaire the whole day long, and never ever work fixing things for the love of art. Hard working enthusiastic developers is a Linuzz monopoly.

I second this

[waslap]

I am overjoyed that my suspicions have finally been vindicated. I've been working 10+hours a day on linux for the last 13years and you tend to get in tune with your environment (i can still today recite my DOS bootup tune on my XT even though I haven't worked on it for 20 years:-) and some time ago after installing a new flavour of linux I immediately started complaining to fellow workers that something has gone wrong in the kernel but it was not annoying enough to really do something about it; you start living with it. It manifests sometimes when I compile - my system simply locks up for 20-30 seconds which is something I never experienced before. I'd say it happens once out of every 50 compiles of the same program with gcc. During such occurrences, I can't access anything on my desktop which annoyes me cause I typically switch to another kterm session to prepare to run the build whilst compiling (to keep up the productivity and all that). I have also seen strange ratios of i/o to cpu wait in 'top' nowadays but can probably ascribe that to CPU's that just became ridiculously fast and the way top calculates its scores. Nevertheless, I've mumbled over and lambasted i/o wait in Linux ever since a very specific time in the past and even though I haven't noted the exact date, I'm sure its related to this. Anyway, I found this intrigueing enough to create a slashdot account after years to share my joy that the bugs days are hopefully numbered now.

Problem is Real

[Anonymous Coward]

For what it is worth, the problem is real.

We have experienced massive negative effects with our MySQL server; downgrading to early linux kernel solves the problem. This has been very difficult to debug as we never guessed that the OS would be a factor... we figured it had to be something we were doing. Only by chance did we try another distro / kernel only to find that everything starts working fine when you downgrade.

Re:Problem is Real

[Harik]

If you can reproduce it, do a git-bisect. You'll find the change that caused it pretty quickly.

Re:funny

[BlackCreek]

Users notice this a ***lot***: https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/131094