Linux 2.6.37 Released

diegocg writes "Version 2.6.37 of the Linux kernel has been released. This version includes SMP scalability improvements for Ext4 and XFS, the removal of the Big Kernel Lock, support for per-cgroup IO throttling, a networking block device based on top of the Ceph clustered filesystem, several Btrfs improvements, more efficient static probes, perf support to probe modules, LZO compression in the hibernation image, PPP over IPv4 support, several networking microoptimizations and many other small changes, improvements and new drivers for devices like the Brocade BNA 10GB ethernet, Topcliff PCH gigabit, Atheros CARL9170, Atheros AR6003 and RealTek RTL8712U. The fanotify API has also been enabled. See the full changelog for more details."

Comments

Kernel locking

[iONiUM]

Well I'm glad they officially fixed the kernel lock. Out of curiosity, how long until Ubuntu or Debian sees this integrated into their line? A year? Not trolling, I only started using Ubuntu recently, so I'm curious.

Re:Kernel locking

[Anonymous Coward]

Ubuntu will ship with this kernel in their next release Natty in April

Re:Kernel locking

[Cyberax]

Ubuntu in about 6 months, 2.6.37 should be in the 11.04 release.

In Debian Stable - in about 2 years (in the next release).

Re:Kernel locking

[turbidostato]

"Well I'm glad they officially fixed the kernel lock. Out of curiosity, how long until Ubuntu or Debian sees this integrated into their line?"

Don't know about Ubuntu but since Debian is already "frozen" towards its next release (codename "squeeze") you can bet it will be about two years from now, more or less.

Of course, you will see it much sooner on their development lines, "Testing", "Unstable" and/or "Experimental".

Re:Kernel locking

[lederhosen]

In debian it will be available soon in unstable, the RC is already available in experimental.

Package linux-image-2.6.37-rc7-686

        * experimental (kernel): Linux 2.6.37-rc7 for modern PCs
            2.6.37~rc7-1~experimental.1: i386

Re:Kernel locking

[Anonymous Coward]

It's not a complete removal of the BKL. There are still some drivers (and a couple of filesystems I think) that have not been converted. Selecting those in a config as a built in or standalone will re-enable the BKL. IIRC, the plan to have those corrected or obsoleted is scheduled for .38

Re:Kernel locking

[inode_buddha]

Check the "rawhide" repositories. Fedora tends to track the -rc kernels fairly closely with near-daily builds. Therefore it is likely that fedora will have this in rawhide within a day if not already.

Re:Kernel locking

[korgitser]

And that would be the .38 special

Re:Kernel locking

[kbielefe]

It already is, for very liberal definitions of "integrated." :-)

Re:Kernel locking

[somenickname]

This kernel should trickle down to 10.04 LTS as well. One of the big complaints about the 8.04 LTS version was that hardware gets released so rapidly that a 3-5 year old kernel isn't going to support a lot of it. Even right now the Ubuntu 10.10 kernel (2.6.35) is in the 10.04 repos that are enabled by default.

Re:Kernel locking

[diegocg]

Note that the performance advantages of that change are zero. It's an aesthetical thing, so distros are not in eager of shipping it.

Re:Kernel locking

[Bootsy Collins]

Would someone mind explaining (for those of us who have some C experience, but aren't kernel hackers) what the Big Kernel Lock is? In particular, is this something that will impact the desktop user?

Re:Kernel locking

[bzipitidoo]

That's one reason I switched to Arch Linux. Updates make it in a lot faster. That's particularly nice for the browser.

Hoped I wouldn't experience downsides from that, but I have. I'd say Linux still isn't ready for the Year of Linux on the Desktop. These sorts of problems show the wisdom of holding off on kernel updates. I want Firefox updates right away. But kernel updates? Maybe not.

Kernel 2.6.34 and 35 had a problem in the e1000 driver, which seemed to affect only very specific motherboards, like in one of my computers. No networking was an especially inconvenient problem. I haven't stumbled over a neat way to downgrade in Arch, apart from a new installation from an old snapshot, so I simply used other computers until 2.6.36, and that fixed the issue. Going back to 2.6.33 or earlier wasn't a good option either-- those had other problems. Too many dependencies. Such as forcing an undo of a partial conversion from HAL to udev. I read that HAL is a mess, and will be abandoned for udev. Possibly the biggest change for version 1.9 of Xorg was a switch from HAL to udev.

Most of the nagging little problems are with the desktop. Don't know if reading from full sized CD-Rs has been fixed. Been fooling with lighter weight GUIs, specifically LXDE, and the automounting never has worked well. Why don't I just use KDE or Gnome? Don't like the sluggish feel of their interfaces. Shutdown on another of my computers has been random since around 2.6.30-- sometimes it works, and sometimes not. Used to always work. I have some ancient trackballs that connect to a 9 pin serial port. No luck getting Xorg to use such devices. Still can't ditch the proprietary NVidia driver for Nouveau. Am wondering when distros will offer btrfs as one of the options. Reiserfs is pretty old now, and dying, and ext4 hasn't thrilled me. But that age is nothing compared to the fact that FAT is still the easiest way to share files with Windows.

Re:Kernel locking

[Pr0xY]

It's a fairly simple idea. In any place that two threads of execution (be them real threads or interrupts or whatever) could possibly access the same resource at the same time, locking must be used to ensure data integrity and proper operation of the code. The "Big Kernel Lock" is a system wide "stop the world" lock. This is a very easy way to make the code "correct" in the sense that it will work and not corrupt data. But the downside is that while this lock is held... everything else must wait. So you better not hold it for very long and while it is easy to get correct, it has pretty bad performance implications.

A better solution is a fine grained lock just for that resource, so the only threads of execution which need to wait are ones that are actually contending for that resource. The downside here is that it is much more complicated to get correct. So when implementing this, you have to be *very* careful that you got it right.

The BLK has been in the process of being removed and has been phased out of the vast majority of the kernel for a while, this change is simply enabling a build in which it doesn't even exist if you don't build any of the older drivers which don't use more fine grained locking.

Re:Kernel locking

[phantomcircuit]

The Big Kernel Lock is a Symmetric Multi Processing (SMP) construct. In order to make kernel operations atomic you lock the entire kernel. This works as a good initial locking mechanism because it's relatively easy to implement, it avoids issues like deadlocking very well.

The problem with the BKL is that it locks the entire kernel, even if processes are calling functions totally isolated from each other only one can be in the kernel at a time.

In practice the BKL hasn't been a big deal for a while now since the important (performance wise) parts of the kernel have had finer grained locking.

So It's pretty unlikely to have much effect if any on desktop users.

Re:Kernel locking

[Yossarian45793]

For those that aren't aware, the BKL (big kernel lock) hasn't caused any issues except purist angst for a very long time now. All of the performance critical kernel code was fixed to use fine grained locking years ago. This change is just to satisfy the people who are offended by the architectural ugliness of the BKL. In terms of performance and everything else that matters, the removal of the BKL has absolutely no impact.

Re:Kernel locking

[Yossarian45793]

The BKL was a hack added in Linux 2.0 to support multiprocessor machines. It was ugly but expedient (like most engineering solutions). Over time, multiprocessor support in the kernel has gotten much better, and the BKL has become less important, up until now when it's so unimportant it can be removed entirely.

Nobody, especially not desktop users, will notice any change from its removal.

Re:Kernel locking

[AntiGenX]

Put simply, when you have many independent bits of code competing for finite shared resources/time within the kernel (this is different than code just running in user space), you have to put locks on them so that only 1 thread can access them at a time. Once a lock is released then the another thread gets a turn. With a big lock, only one lock exists for every resource. Although a thread may only need access to a single resource, all of the resources get locked.

The alternative is to implement more fine-grained locks on each resource or set of resources. This allows two threads that are using different kernel resources to potentially execute in parallel. The danger is it's more complex and requires careful coding to avoid deadlock or race conditions. That help?

http://blog.internetnews.com/skerner/2010/11/linux-2637-kills-the-big-kerne.html

Re:Kernel locking

[Korin43]

Arch does have an LTS kernel (although "long-term" in Arch is like 6 months), which you can use if the current version is broken. I used it for a while when wine + some kernel version caused World of Warcraft to not work. Hope this helps if you problems in the future.

pacman -S kernel26-lts

Re:Kernel locking

[deek]

Reiserfs is pretty old now, and dying, and ext4 hasn't thrilled me.

What features have you enabled in ext4? I'm running it on one of our servers, and I like the performance very much. It's even a debian stable machine, although I had to use a kernel from "testing" to properly enable ext4.

I've got extents, uninit_bg, and dir_index enabled, amongst other features. If you converted from ext3, then you probably don't have these options enabled. Even if you created the ext4 filesystem from scratch, some older versions of mke2fs won't enable these ext4 features by default.

Try giving ext4 another go. Maybe you'll like it more the second time around.

Re:Kernel locking

[TheRealGrogan]

It's a bit tricky in all of those so called "easy" distributions but not impossibly difficult. It's probably best to use the distributor's kernel sources and methods, but you don't have to if you make some changes to the system first. You have to be careful that you're not using features they've patched in to their kernels if you do that though.

I have Ubuntu 10.10 (actually Xubuntu with XFCE) on my netbook, and I have been using a regular kernel.org kernel (2.6.36.2 currently) without using an initrd or anything. This requires changes to fstab to use normal device nodes rather than the UUID. In Ubuntu I was previously using their sources and the alternate "Debian" method of building them.

https://help.ubuntu.com/community/Kernel/Compile

On redsplat distros (Fedora, RHEL) it won't boot without an initrd unless you manually create some hard device nodes in /dev first. (This is assuming it's still possible... I haven't tried one in a long time)

In all of those "easy" distros it sucks out of the box though... you have to install all the tools you'll need. (In *buntu that's usually just build-essential and libncurses5-dev for kernel)

Distros like Slackware, Gentoo, Arch and similar are pretty much meant for the user to roll their own kernels.

Re:Kernel locking

[Kjella]

And this is one example of why the kernel doesn't have a stable ABI. You can bet tons of unmaintained third party drivers would use the BKL, so you could never get rid of it. From what I've understood purging it from every driver has been a pretty big job and only possible because all the drivers are in the kernel tree.

Re:Kernel locking

[dsavi]

Actually I believe they will ship with 2.6.38.[1]

Re:Btrfs

[tibman]

It's running on my server at home, so i hope so ; )

Ceph is really cool

[Lemming Mark]

Ceph is a really cool bit of technology. It distributes storage redundantly across multiple machines, so you can store lots and lots of data and not lose any if one of the hard drives explodes. It should distribute the load of serving that data too. You can have a network filesystem based on this already, now they've added support for virtual block devices (i.e. remote disks) over it.

If you combine that with virtualisation (the Kernel Newbies changelog mentions that there's a patch for Qemu to directly use a Ceph-based block device) then you can do magic stuff. e.g. run all your services in virtual machines with their storage hosted by Ceph. Provide a cluster of virtualisation hosts to run those VMs. If a physical box needs maintenance, live-migrate your VMs off it without stopping them, then just yoink it from the cluster - the storage will failover magically. If a physical box explodes, just boot the VMs it was running on other nodes (or, combined with some of the hot-standby features that Xen, VMware, etc have started to offer, the VMs are already running seamlessly elsewhere as soon as the primary dies). If you need more storage or more processing, add more machines to the cluster, get Ceph to balance the storage and move some VMs around.

Not everyone is going to want to run Ceph on their home network but if you have a need for any of this sort of functionality (or even just an enthusiasm for it) then it's super cool. Oh yes and Ceph can do snapshotting as well, I believe. Ace.

Re:Btrfs

[Anonymous Coward]

I've been using it for nearly 2 years without any issues whatsoever (I haven't even had to fiddle with it to keep it going). It's been in the kernel for over 1 year.

They're well beyond "some sort of working version, even if it doesn't do much". Give it a try.

What's new

[Troll-Under-D'Bridge]

The link in the story just points to the list post announcing a new major version of the Linux kernel. Note that the changes listed in the post are for changes from the last release candidate (-rc8) and not from the last major kernel release (2.6.36). For an overview, it's better to head over to Kernel Newbies. It even has a section which summarizes the "cool stuff", major features that the new kernel brings.

Interestingly, the overview appears to overlook what I believe is a major feature introduced in 2.6.37: power management for USB 3. I may have to do some more digging through the actual kernel changelogs. Maybe the change was reverted during the last few candidate releases, but I remember reading about it in H-Online, particularly this part:

The XHCI driver for USB 3.0 controllers now offers power management support (1, 2, 3, 4); this makes it possible to suspend and resume without temporarily having to unload the driver.

(In the original, the parenthetical numbers are links to the kernel commits.)

Power management for USB 3 would have been the most important new feature for me. Without it, you have to resort to a number of ugly hacks to hibernate or suspend a laptop or a motherboard with USB 3 enabled. (Turning off USB 3 in the BIOS is a hardware hack that allows you to bypass the software hacks.)

Ooops

[Troll-Under-D'Bridge]

I missed the second link in the story, which does point to Kernel Newbies. (Blame it on my browser which doesn't color links in red or some other obscene color.) However, my comment about USB 3 still stands. I'm still trying to find a "news" source that highlights the new XHCI power management feature. Failure to hibernate/suspend because of non-working USB 3 power management is an issue that's been discussed in a number of forums.

Includes the "magic 200-line" task grouping patch?

[sagaciousb]

Looking through the changelog I couldn't find anything immediately evident about whether or not the "200-line kernel patch which does wonders" was included in this release or not. Here is the related original post. Anybody know if it will be in there for certain? I may have to remove my Ubuntu alternative workaround outlined in the subsequent article before doing an upgrade.

Re:Btrfs

[0100010001010011]

Before I committed ANY data to ZFS I sure as heck "played around with it" in virtual machines until I was comfortable doing about anything with it.

"Pull" one of the drives. What happens?
dd if=/dev/random of= to your disk in random places (skip/seek), what happens to your data.
Pull all of the drives and replace it with a larger one.

How are the user tools for btrfs? zpool & zfs are fairly well documented and have very simple short commands.

Does it automatically share over nfs/samba like you can with ZFS on Solaris?

Re:Includes the "magic 200-line" task grouping pat

[sciencewatcher]

That patch is scheduled for 2.6.38. This article details 2.6.37. This article is about the end of the pipeline, the article you linked to is about the beginning of the pipeline that kernel development is.

Re:Btrfs

[Korin43]

The problem with Btrfs isn't that it doesn't work (it works fine and has for years). The problem is that it's not very fast right now (most benchmarks I've seen show it slightly behind other major file systems in most tasks), and most things don't make use of the cool things it does.

Gurus: will this affect high-SMP workstations?

[isolationism]

I have dual-processor Xeon with six cores each, meaning there are effectively 24 threads (2 physical * 6 cores * 2 hyperthreading) and the system will lock up for SECONDS at a time during large IO operations. The file system is XFS over an 8-disc hardware RAID10 on 15K RPM drives. Seems to be most noticeable when copying to/from the network, although I'm not convinced the network is the problem here. For such a high-end machine these stalls are unbearable; I had (a lot) less difficulty with only 4 cores and less/slower drives in a hardware RAID 0.

Re:Gurus: will this affect high-SMP workstations?

[drinkypoo]

Unless it's a very expensive controller, performance will go down. OTOH, XFS has shown itself to be a bit of a bear... and problem-ridden. I just had it eat my superblock. Took more than an hour for xfs_repair to find the backup. Onward to ext4!

Re:Btrfs

[loxosceles]

Compared to zfs, which is the only other quasi-mainstream filesystem that has copy-on-write (which gives snapshots for free) and data checksums, btrfs is almost always faster. Most of btrfs's slowness is due to those two features. Comparing ext4 speed to btrfs speed is not fair unless you disable both with -o nodatacow.

http://www.phoronix.com/scan.php?page=article&item=btrfs_zfs_ssd&num=1
see page 4 for btrfs random write performance, which blows away both zfs and ext4 on hdds.

Without COW and checksumming, btrfs is closer to ext4. Even so, except for applications that are reading or writing massive amounts of data, I'd rather have data integrity and SSD wear leveling and free read-only snapshots instead of maximum speed.

ANOTHER version?

[Anonymous Coward]

Geez... it's a shame Lunis and the boyz couldn't be bothered to program it correctly the first time around.

Re:Btrfs

[profplump]

You could just turn on LVM, which has been stable (and even the stock config in some distros) for years now and gives you dynamic volume allocation, data striping and snapshots with any filesystem. There are reasons to like ZFS/btrfs/etc., but the things you're asking for are easily available with much older, better-supported, better-documented solutions.

Re:Versioning?

[shish]

Yeah, they abandoned that a few years ago, current plan is something along the lines of "six weeks development, two weeks bugfixes-only, release, repeat", incrementing the third part of the version number each time (ie there are no plans for the "2.6" part to ever change AFAIK)