Jens Axboe On Kernel Development

BlockHead writes "Kerneltrap.org is running an interview with Jens Axboe, 15 year Linux veteran and the maintainer of the linux kernel block layer, 'the piece of software that sits between the block device drivers (managing your hard drives, cdroms, etc) and the file systems.' The interview examines what's involved in maintaining this complex portion of the Linux kernel, and offers an accessible explanation of how IO schedulers work. Jens details his own CFQ, or Complete Fair Queue scheduler which is the default Linux IO scheduler. Finally, the article examines the current state of Linux kernel development, how it's changed over the years, and what's in store for the future."

Comments

Disagree with Mr. Axboe...

[isaac]

JA: In your opinion, with the increased rate of development happening on the 2.6 kernel, has it remained stable and reliable?

Jens Axboe: I think so. With the new development model, we have essentially pushed a good part of the serious stabilization work to the distros.

I respectfully disagree that the new development model works well from an end-user's perspective (an "end user" of many thousands of linux hosts, not a toy desktop environment). Minor point releases now contain major changes in e.g. schedulers. This makes for a lot of work for real Linux users, backporting the useful bugfixes while retaining older algorithsm for which workloads are optimized. Result: a severely splintered kernel and a lot more work for us.

If core changes of such magnitude are no longer sufficient to merit a dev branch or even a major point release, why bother with the "2.6" designation at all? Just pull a Solaris and call the next release "Linux 20" or "Linux XX."

-Isaac

Re:Disagree with Mr. Axboe...

[Kjella]

Well, on the other side distros were backporting *huge* amounts of patches from 2.5 to 2.4, so while plain vanilla 2.4 was stable, almost noone was running it. The 2.6 releases means the distros are shipping "stabilized unstables" instead of "destabilized stables", I guess that works out better for some and worse for some. Are RHEL, SLES, Debian stable kernels not good enough kernels to start out with, if stability is what you need? I feel there's quite a few things I see come which I find great that arrive in a timely fashion, not at the release of 2.8 in a few years. I think most that use a distro's kernel feel that way.

If you're the kind of kernel hacker who liked to get yours directly from kernel.org, yes then it sucks. But IMO the kernel has grown too big for just the core devs, think of it as an "extended" kernel team including the distros, where kernel.org releases are "internal betas". I think if you cut it back and expect just kernel.org to deliver stable kernels with the resources they have (which admittingly, they used to) then kernel development will slow way down.

Re:Disagree with Mr. Axboe...

[ComputerSlicer23]

Don't take this the wrong way, but your complaint sounds a lot like the story about a patient and a doctor:

"Doctor, when I do this, it hurts", and the doctor replies, "Well don't do that".

I mean, if you are following bleeding edge kernels, and complaining that they aren't as stable as you'd like. Why not just follow a vendors kernel? If you use or install "many thousands", you are either maintaining your own de-facto distribution or you are using someone else's distribution. Vendor's do exactly the work you want done on your behalf.

I patiently wait for my vendor kernel, which might be 10 point releases behind integrate bug fixes and then upgrade in a year or two to a much newer point release (I think RedHat has used 2.6.9 and/or 2.9.13 in recent memory)... Incrementing a different number wouldn't really make any difference anyways. At that point it's all semantics, if you know the rules of the game, it's not hard to tell what's dangerous as an upgrade and what's not.

It's not like 2.4.13 (or whatever one in the 2.4 series that introduced series disk corruption) was safe merely because it was a point release... They are safe because somebody took it out back and beat on the kernel for a while and it didn't cause any problems. If you upgrade without proper testing and it breaks, you get to keep the pieces.

Kirby

What about the process' priority?

[mi]

CFQ now uses a time slice concept for disk sharing, similar to what the process scheduler does. Classic work conserving IO schedulers tend to perform really poorly for shared workloads.

I wonder, if the originating process' priority is taken into account at all... It has always annoyed me, that the "nice" (and especially the idle-only) processes are still treated equally, when it comes to I/O...

CFQ not the default scheduler?

[rehabdoll]

Anticipatory is, according to my menuconfig:

The anticipatory I/O scheduler is the default disk scheduler. It is
generally a good choice for most environments, but is quite large and
complex when compared to the deadline I/O scheduler, it can also be
slower in some cases especially some database loads.*

Anticipatory is also preselected with a fresh .config

Re:CFQ not the default scheduler?

[zdzichu]

CFQ is default since 2.6.18, released back in September 2006.

No block devices = no disk scheduling?

[Kadin2048]

So how does that work?

At risk of starting a holy war, is there any reason why one approach would be superior? And do they lend themselves to different methods of scheduling? In TFA, Axboe talks about [1] the scheduling mechanism used in later versions of the 2.6 kernel series, which alleviates a problem that I (and most other people, probably) have run into before.

I'm curious, because although I don't use any of the 'real' BSDs very often -- I spend most of my time (at home, anyway) using either Mac OS X, which uses the Mach/XNU kernel (which is derived from 4.3BSD, although I don't know if the I/O scheduler has been rewritten since then), or Linux with the 2.6 kernel, and it seems to me that OS X's disk I/O leaves something to be desired compared to Linux's.

Does BSD handle I/O differently in some fundamental fashion than Linux? It sounds like, by eliminating block devices, that they basically remove the kernel from doing any re-ordering or caching of data, which makes things "safer" (in the event of a crash) but seems like it would have big performance penalties when using drives that aren't very smart, and don't do a lot of caching and optimization on their own. It seems like getting rid of I/O scheduling altogether is a stiff price to pay for "safety."

[1] (quoting because there doesn't seem to be anchors in TFA)

Classic work conserving IO schedulers tend to perform really poorly for shared workloads. A good example of that is trying to edit a file while some other process(es) are doing write back of dirty data. ... Even with a fairly small latency of a few seconds between each read, getting at the file you wish to edit can take tens of seconds. On an unloaded system, the same operation would take perhaps 100 milliseconds at most. By allowing a process priority access to the disk for small slices of time, that same operation will often complete in a few hundred milliseconds instead. A different example is having more two or more processes reading file data. A work conserving scheduler will seek back and forth between the processes continually, reducing a sequential workload to a completely seek bound workload. ...

Scheduling better than no scheduling?

[Kadin2048]

Are there any hard metrics on what the performance advantages are of various schedulers, under typical load conditions?

Reading TFA piqued my interest into I/O scheduling and I've been doing some reading on it, and it seems like there are several competing schools of thought, of which Axboe (and potentially the Linux kernel developers generally) are only one.

An alternative view, such as this from Justin Walker (a Darwin developer) on the darwin-kernel mailing list, holds that it's not worthwhile for the OS kernel to do much disk scheduling, since "the OS does not have a good idea of the actual disk geometry and other performance characteristics, and so we [kernel developers] leave that level of scheduling up to the controllers in the disk drive itself. I think, for example, that recent IBM drives have some variant of OS/2 running in the controller. Since the OS knows nothing about heads, tracks, cylinders for modern commodity disks, it's futile to try to schedule I/O for them." (written Mar 2003)

Axboe seems to acknowledge that this may sometimes be the case, because they do have the 'non-scheduling scheduler,' which he recommends only for use with very intelligent hardware. However, it seems like some people think that commodity drives are already 'smart enough' to do their own scheduling.

It seems like determining which approach was superior would be relatively straightforward, and yet I've never seen it done (although maybe I'm just not looking in the right places). Anecdotally, I'm tempted to agree with Axboe, since it seems like, when doing things where several processes are all thrashing the disk simultaneously, my Linux machine feels faster than my OS X one, but this is by no means scientific (they don't have the same drives in them, not working with the same datasets, etc.).

On what drives, and under what conditions, is it advantageous to have the OS kernel perform scheduling, and on which ones is it best just to pass stuff to the drive and let the controller do all the thinking?