Linux 2.6.17 Released

diegocgteleline.es writes "After almost three months, Linux 2.6.17 has been released. The changes include support for Sun Niagara CPUs, a new I/O mechanism called 'splice' which can improve the performance greatly for some applications, a scheduler domain optimized for multicore machines, driver for the widely used broadcom 43xx wifi chip (Apple's Airport Extreme and such), iptables support for the H.323 protocol, CCID2 support for DCCP, softmac layer for the wireless stack, block queue IO tracing, and many other changes listed at the changelog"

Comments

Re:Question for the masses.

[shird]

Modules... Only the modules (read: 'drivers') that are needed are loaded. It needs to be in the kernel because it accesses the hardware (the net card) at a fairly low level.

Re:Go Linux!

[freralqqvba]

sendfile(2) is now a call to splice() so programs that use the old syscall will benefit as well and without modificaiton.

some highlights from the changelog

[doti]

Some stuff I found interesting on the human-friendly changelog.

Block queue IO tracing support (blktrace). This allows users to see any traffic happening on a block device queue. In other words, you can get very detailed stadistics of what your disks are doing. User space support tools available in: git://brick.kernel.dk/data/git/blktrace.git

New /proc file /proc/self/mountstats, where mounted file systems can export information (configuration options, performance counters, and so on)

Introduce the splice(), tee() and vmsplice() system calls, a new I/O method.
The idea behind splice is the availability of a in-kernel buffer that the user has control over, where "splice()" moves data to/from the buffer from/to an arbitrary file descriptor, while "tee()" copies the data in one buffer to another, ie: it "duplicates" it. The in-buffer however is implemented as a set of reference-counted pointers which the kernel copies around without actually copying the data. So while tee() "duplicates" the in-kernel buffer, in practice it doesn't copy the data but increments the reference pointers, avoiding extra copies of the data. In the same way, splice() can move data from one end to another, but instead of bringing the data from the source to the process' memory and sending back to the destination it just moves it avoiding the extra copy. This new scheme can be used anywhere where a process needs to send something from one end to another, but it doesn't need to touch or even look at the data, just forward it: Avoiding extra copies of data means you don't waste time copying data around (huge performance improvement). For example, you could forward data that comes from a MPEG-4 hardware encoder, and tee() it to duplicate the stream, and write one of the streams to disk, and the other one to a socket for a real-time network broadcast. Again, all without actually physically copying it around in memory.

Microkernel anyone?

[argoff]

Why are the network drivers part of the kernel? It seems like this would make it more difficult to adopt newer hardware types. Also, since most computers have 1-2 NICs at the most, wouldn't that clog up the kernel with tons of drivers for hardware you'll never use?

This is the essence of the Microkernel debate. http://en.wikipedia.org/wiki/Microkernel/ The truth is that the Microkernel model probably is a better design, but in terms of when the Linux kernel was starting out - its implementation simply wasn't pratical. It didn't help that the people who thought they knew how to build a better kernel decided to try and intellectually brow-beat Linus into doing it instead of implementing it themselves and putting it under the GPL. This led to a lot of bitterness and resentment between the two camps. The HURD http://en.wikipedia.org/wiki/Hurd project is a GPL microkernel project, but it simply wasn't managed as well as Linus managed Linux.

I think over time, things eventually will move to a microkernel model even though there are other ways to emulate some of their security and flexability benefits - like xen http://en.wikipedia.org/wiki/Xen

Re:Video Editing?

[Anonymous Coward]

Insightful? How about Kino or Cinelerra or Lives or Mainactor?

not like that

[r00t]

This is really just a way for app code to manipulate data without needing to have it copied or memory-mapped.

Linus refused the FreeBSD-style zero-copy because it is often a lose on SMP and with modern hardware. Page table and TLB updates have huge costs on modern hardware.

If you do like the Microsoft way, use Red Hat's kernel. The in-kernel server works very well.

Re:support for the h.323 protocol, quite unlikely

[nick this]

I read that as ip conntracking to allow videoconferencing devices that follow the h.323 standard to be natted.

obtw: your pedant bit is apparently stuck high. just a fyi -- didn't know if you realized it. :)

Re:Go Linux!

[pavon]

Obviously though it is necessary to write new functions on occassion; for example when the new function is worse than the old function is under some circumstances.

That is exactly why it was done. More information about can be found at kerneltrap: here, and here. It was also previously on slashdot, although you would be best to skip that - it has more misinformation than the other kind.

In short, all the known ways of implementing zero-copy within the existing API's cause the most common usage cases of those API to be slower than they are now. Therefore, it made more sense to export this new API for the applications where speed is critical.

In the the first kernaltrap article, Linus also explains why splice is different from sendfile, contrary to the posts here claiming they are essentially the same.

Re:Missing driver?

[WhodoVoodoo]

try hitting '/' on make menuconfig, type ov511 hit enter. That's a hot tip that's saved me quite a bit of time...
It'll find it if it's there.

Re:Where is 2.7?

[x2A]

The stable/development branches might be a nice idea in theory, but in practice it doesn't work. Distros would ship, for example, a "stable" 2.4.xx kernel, except it wouldn't actually be that. They would spot nice features in the 2.5 kernel that they wanted to offer their users, and so back-port them... and any other nice patches floating around the net while they're at it. The result being that the kernels that ship with distros were so heavily modified, that stability (from one machine to another) went right out of the window. You couldn't go to kernel.org and download an updated kernel, as without all the patches, it wouldn't work. So you had to stick to the distro's kernels.

So instead, the 2.6 goal is to have development/stable parts of the cycle, rather than seperate branches. Roughtly: patches that could break things get submitted at the beginning of the cycle, and -pre1/-pre2 tarballs are released. If you want bleeding edge, you go here. Release candidates are released, where developers get chance to fix bugs etc in the code. Then, any code that's still [known to be] buggy gets dropped for the final release (eg, 2.6.17). The developer can work on it, and try add it again during subsequent cycles. When it works, it can be included in a final release.

During this cycle, security and other urgent bug fixes take place in the ultra-stable branch, with version such as 2.6.16.1, 2.6.16.2.

(This is the rough idea I believe, there could be some slight inaccuracies in how it actually takes place, I haven't followed it 100%, but this should be close enough to get the right idea).

Re:Where is 2.7?

[iabervon]

That was the theory. But in practice, if Y was even, the kernel was obsolete, while if Y was odd, the kernel was broken. Except, of course, 2.even.0, which was actually stable, but broke compatibility with the previous kernel that worked. And occasionally, 2.even was kept up-to-date because nobody could use 2.odd for development, because it didn't work at all. You could tell that the old model didn't actually work, because no distribution shipped any kernel that used that model; they all shipped 2.even with an arbitrary set of patches (generally hundreds) from 2.odd and elsewhere. With the new model, distros are shipping kernels with only a few patches, and those patches are getting merged upstream.

The stable kernels aren't remotely on the bleeding edge; they contain only features which have been tested over the past three months, after being filtered out of the bleeding-edge development as being things that have already stabilized and stand a good chance of being proven in three months. It's effectively very similar, except the development series isn't left known-broken and the stabilization process happens on a quick schedule, with stuff that isn't ready pushed off to the next cycle rather than delaying the current cycle. Also, the version numbers change by less (development gets -mm, -rc, or -git; stable series change the third digit by one instead of the second by two; and bugfix releases change the fourth digit instead of the third).

That and

[Sycraft-fu]

For kernel operations, you want everything pretty efficient. You want it as fast as possible and you don't want a lot of extra code hanging around. Unfortunately, the higher level a language you use, the more inefficency there is. For most programs it doesn't matter. They are either not the sort of thing that needs speed (like a word processor) or one where you can optimize the small part of the code that takes most of the time (like a game). However the kernel is a little different. Everything in there is time critical essentially.

C is the best compramise. While assembly might give you the theoritical best code, it'll big a giant mess to try and totally unmaintainable. Might actually be slower and larger for it. C is pretty good because it's easy enough to generate deceant code in, but it isn't much higher up the abstraction chain so it compiles quite efficient.

You have to remember that object orientation and such are all human creations. Processors don't think in objects, for that matter they don't really even think in functions. They think in memory locations, and jumps to those locations. Doing OO code means a whole messy layer the compiler has to go through to translate that in to something the processor actually understands.

Broadcom 43xx HOWTO:

[cbhacking]

Haven't tried the release of 2.6.17 yet, but rcX versions required extracting the firmware for your Broadcom card from a binary such as bcmwl5.sys (Windows driver). The tool bcm43xx-fwcutter does this.

I'm not an Ubuntu guy, but this reference might be useful to anybody trying to make the new Broadcom Wifi driver work in Linux. Very easy steps, and most non-Ubuntu users should find it easy to adapt for their specific distros.

Re:Go Linux!

[waveclaw]

The kernel is written in C, and so are those system calls. I don't believe you can overload a C function.

There is no overloading going on here. Overloading is to create a new function with the same name, but taking different parameters.

Ahem. The original function, sendfile(2), was rewritten to call splice() instead of doing something else.

Everybody that wrote code that used the old function now has to deal with splice() running instead of the old function's logic.

Just to hammer it home:
Old - app -> sendfile(2) -> some logic -> return to app
New - app -> sendfile(2) -> splice() -> splice's logic -> return to sendfile(2) -> return to app

With the Linux kernel, as this exepmlifies, you can improve the original code and get everyone (well, those to lazy to revert the changes) to use it. In this case you have a fixed API (sendfile(2) which is well known and published) so you don't just want to tell everybody to recompile with called to splice().

See the difference? Feel the difference.

The kernel is GPL and thus the actual source code used to compile the binary kernel you use is available to you. With a closed source kernel you might be able to purchase an SDK with linkable binaries and some (probably undocumetned) header files. Programmers in this situation need things like function overloading and class inheritence just to do anything. One way of looking at the history of languages like C++ is as a technical solution to the ethical problem of closed source programming. Those languages focus on extending on the outside. With OSS you can usually replace, fix and improve on the inside. BSD and GNU differ on a the point of GNU wanting everyone to share the source to those fixes if they share the resulting binaries. But I digress.

And I can't wait to see if this breaks something.

Re:module shotguns

[wertarbyte]

Many a linux distribution I've used (most noticeably Debian) applies the "shotgun" approach to module-loading because the hardware detection and hotplug methods are so convoluted and undependable. Kind of defeats the purpose of loadable modules if the distribution simply loads everything under the sun to see what sticks.

Obviously you haven't used Linux for a long time. Modules are not loaded to detect hardware, instead the hardware acquires the driver module: The kernel identifies hardware via PCI or USB device ids, which are also stored in the modules. So Hotplug (and newer versions of udev) can load the appropiate module once hardware is added to the system.

Worse, many modules aren't smart enough to determine "hey, I'm a driver for [some non-removable component]. If I can't find my hardware, maybe I should print an error to ksyslogd and unload myself."

The driver will not be loaded if there is no hardware, unless you explicitly tell your system to do so.

Re:module shotguns

[FireFury03]

Sure there is. There's just not a consistent ABI, and that's on purpose.

If you're contributing a driver, GREAT. It'll compile against the currently installed kernel just fine.

Untrue I'm afraid. If your modules aren't in-tree then they *will* break every so often because the kernel API is not stable. Especially under the 2.6 development model - under the previous 2.4/2.5 model you were pretty much guaranteed that API breakages would only be happening in the 2.5 tree, now they happen at any point in the 2.6 tree. (Yes, I do know this stuff - I work on out-of-tree kernel code).

There is some arguement that all drivers should be in-tree, and for common hardware it is definately a Good Thing to have the drivers in the tree - as the API changes then the person implementing the API change will fix up all the in-tree code that uses that API.

For very specialist and expensive hardware it poses a problem though: the person who does the API change won't have the hardware to test with, and probably all the people who use that hardware are using enterprise distributions so breakages to the module won't be spotted for a long time. It's hard for the hardware vendor to track these kinds of updates and perform the necessary regression testing.

There are valid uses for a GOTO

[Dan Ost]

I agree that using GOTO is a bad idea when another control structure is adaquate,
but, at least in C, there are times when using GOTO is the most natural and,
unequivically, the best choice.

Off the top of my head, I can think of two situations where using a GOTO is
the best solution:

1. breaking out of nested loops. In C, the break command can only break
out of a single loop level. If you need to break out of 2 or more loops, you
can play an ugly game of setting and checking state flags at each level
of looping or you can simply create a label at the exit point and use
GOTO to get there. (sometimes you can wrap your loops as a function call,
but that's often the ugliest solution)

2. shared cleanup code. In a function with multiple exit points, instead
of doing cleanup at each exit point, it is often clearer to set your
return value and then GOTO a label that handles all cleanup before
returning.

Be cautious when using GOTO, but don't be afraid of it. Learn to
recognize when GOTO is appropriate and when it should be avoided.

Re:There are valid uses for a GOTO

[Dan Ost]

1) If you can create a condition where a goto is to be placed, you can add that same condition to the top loop in the nest and let it exit out gracefully.

If that leads to clearer code, then in the cases where you can do that, fine. Do that.

However, there are situations when a condition doesn't make sense until you've already
entered the nested loops at least once (for example, when allocating lots of chuncks of memory,
you can't test to see if you've successfully allocated memory until after you've tried to
allocate memory). Also, if there are several conditions that might require a break, but
they can all be handled the same (at least until after you break out of your loops),
do you really want each one to be tested at every loop test? Think how big and confusing that
would make your continuation test for your outer loops.

2) Use a clean-up function. It will return to the correct place without all the spagetti code.

There's nothing wrong with using cleanup functions if they are convienent for your
particular purpose, but if you have to free 11 objects before returning, then you'll
need to pass all 11 to the cleanup function each time you call it. I don't know about
you, but I usually find functions with 5+ arguments to be ugly. I would rather simply have
a 'goto cleanup' that jumps to a label that does all the cleanup in place. An acceptable
compromise would be to define a macro that does the cleanup in place but hides it from casual
code inspection, thus keeping the code clear, but avoiding the use of GOTO.

Using GOTO in the manners I've described will not lead to speghetti code since the flow of control
will be clear and uni-directional (the antithesis of speghetti code). In case (1), the use
of GOTO is equivalent to raising an exception in Java, C++, or Python from within the loop and
capturing the exception outside the loop (idioms commonly accepted in all three communities).
In case (2), the use of GOTO maps multiple exit points to a single exit point. If you feel
that these techniques qualify as speghetti code, then I would suggest that you've never
seen real speghetti code.

When Djikstra wrote "Goto considered harmful", he was talking about using GOTO to jump outside
the scope of the current function, something not possible in with C's goto (C's goto can only
jump to a label within the current function). See BASIC and PASCAL (I think) for examples of GOTO that
can jump anywhere in the program.