Kernel Comparison: Web Serving On 2.4 And 2.6

An anonymous reader writes "Many improvements have been made in the Linux 2.6 kernel to favor enterprise applications. This article presents results from the IBM Linux Technology Center's Web serving testing efforts, comparing the Linux 2.4 and 2.6 kernels from various aspects. The highlights here are the key enhancements in the 2.6 kernel, the test methodologies, and the results of the tests themselves. Bottom line: the 2.6 kernel is much faster than 2.4 for serving Web pages, with no loss in reliability."

Comments

Thanks SCO!

[Anonymous Coward]

See, all that Unix code that IBM stole from SCO and inserted into the Linux kernel was worth it!

Re:The question is..

[pilot1]

This is not a troll, but a serious answer.
None.

Why upgrade?

[flikx]

Aside from a small performance boost, is there really any reason to update perfectly stable systems? My shop has been using a few boxen running RedHat 5.2 for almost seven years now. If everything is stable, why upgrade?

Re:Why upgrade?

[WasterDave]

In the case of RH5.2, security. I hope you've been doing the furious quantity of patching necessary to keep that secure. If not, I'd seriously consider moving to Debian. You still have to secure it, but it's really easy to do.

The small performance boost, according to this paper, is large. Huge, much the same as the scalability boosts that came in as a result of the mindcraft benchmarks. However, they are for the most part improvements in SMP. There are also some "responsiveness" improvements in the scheduler.

Should you move to 2.6? Probably not. As far as I can tell the gains are on big iron, really really small iron, and the desktop. I'm sure as hell not moving off 2.4 any time soon.

Dave

Re:Why upgrade?

[scishop]

A "small performance boost"? IT'S FIVE TIMES FASTER! Any smaller and it would be in hyperspace.

Re:Why upgrade?

[Sepper]

Aside from a small performance boost

Small...right... i'll refer you to the text of the article:

Conclusion
We've shown that, using a typical test scenario -- Apache/WPT on an 8-way SMP IBM xSeries system -- the Apache server has better scalability and performance on the 2.6 kernel compared to the 2.4 kernel. On the same system under the same workload, the Apache server with 2.6.0-test5 kernel more effectively used system resources and served 5 times more Web pages than the 2.4.18 kernel did. This real data demonstrates that a variety of features and changes have helped the 2.6 kernel offer better scalability and performance and become more mature for enterprise-level applications.

Re:Stop pushing. Its all linux

[MBCook]

Did you even read the SUMMARY? On the system they showed, 2.6 performed dramatically better in EVERY AREA. Now if you are running a 128k processor megacomputer with 12 terabytes of RAM and ueberbit ethernet connection, maybe you don't need the performance increase because your computer could serve pages to every other computer ever built without breaking a sweat, but for people with NORMAL comptuers, that isn't the case. Upgrading to 2.6 is basically getting a free performance boost.

You were having problems with your current webservers? They can't serve pages fast enough? You'll have to spend $50,000 to upgrade so you can handle it? Put 2.6 on! You might be able to hold off that upgrade for 6-12 months, by which time that $50,000 will buy you much more computer than it will today (not to mention you could invest that money and have more by then).

What do you call a FREE PERFORMANCE UPGRADE? You call it good!

Besides, it doesn't matter if it needs a "little while to iron out." If you just blindly deploy new kernels on production servers with no testing, you deserve the flack that will come you way if you get bit by a bug.

Re:Stop pushing. Its all linux

[Vilim]

At this point 2.6 is _far_ better than 2.4 at the same point in the development cycle. Linus actually ripped out Rik van Riel's VM code and replaced it with new VM code. At this point 2.6 is FAR more stable than 2.4 at a similar stage, there really is no reason for the average user not to upgrade

Enterprise applications?

[Fritz_the_Cat]

I don't know about you guys, but I'm not too sure that I would describe this article as an examination of an "Enterprise application" on Linux.
Enterprise applications are many things to many people, but rarely are they web servers.
Enterprise servers are generally run complex applications running many complex operations.
While I'm sure IBM's web server is very good, I don't think that it would be typical of an "Enterprise application".

My point is, while I'm sure this is a fantastic article examining performance improvements between Linux kernel versions, let's not pretend it's something that it isn't.

A marvelous understatement

[leifw]

From the article:
"O(1) scales well..."

No, really?

Consumes more RAM...

[gregfortune]

While the performance gains are impressive (about 5 times as many pages under 2.6) it also shows that 2.6 used 5.6 times more RAM to serve the increased number of pages. If RAM on the system isn't limited, the performance gain is insane. If the system is already overloaded w/respect to RAM, it likely won't help much and there's a *slight* chance it would actually perform worse.

Of course, this is just a benchmark ;o)

Re:Consumes more RAM...

[Anonymous Coward]

Isn't that just a function of having 5 times as many apache processes serving content?

Headline: Linux Makes Bad Code Look Better

[LunaticLeo]

The new linux kernel is great, but the reason the this particlular kernel results is better performance ("5 times better") is because the application framework it is testing is horrible.

All of the "enterprise" applications in this test have several performance cripling features in common: socket per thread connections, fundemental reliance on threads, and massive memory foot print. Apache has one thread/process (the diff is a stack) per connections. Java requires a sizable multiple of memory usage as most other application languages (C/C++ obviously, but also Perl, Python, and PHP). J2EE is an inherently thread driven programming framwork.

So yes, Linux 2.6 ameliorates the downsides of unnecessary use of threading. It makes thread creation and context switching even faster on the Linux platform.

And Yes, Linux 2.6 memory management is fundementally better. Reverse Page Table Entry mappings make finding victim pages better; and it is designed to avoid victimizing active pages better.

But could you all imagine if people were designing fundementally better application framworks? Event driven application architectures like TwistedPython and POE, or Event-thread hybrid systems like SEDA.

The performance stats given in that article are shit, complete utter shit. I know. In the proprietary world I work in, I code faster programs on the same Linux platform on a daily basis; orders of magnitude faster.

All the accomplishments of Linux 2.6 can be used for true performance programming. I plead with you all, stop using Threads until you know what they are good for. Stop using the stack to maintain your program state. Throw off the shackles and learn to program network servers.

Re:Headline: Linux Makes Bad Code Look Better

[Hard_Code]

Whatchoo talkin' 'bout Willis!? There are two fundamental approaches to high load: threaded blocking IO, or non-threaded async IO. Both are different abstractions for the same fundamental goal of adding concurrance to a system. In reality most sites will do just FINE by modelling concurrency with a threadpool and blocking IO. Sure you CAN use non-threaded (or "less-threaded") async IO, but you incur a complexity overhead (this is the "overhead" hidden by the simplicity of threads). Unless you are writing an eBay or Amazon.com or Yahoo.com, a threaded/blocking-IO approach will work fine. Since it IS such a popular model, I think it is definately legitimate to compare kernel performance on those grounds. Note that I keep saying "abstraction" and "model". In reality there is nothing mandating that the thing I call a "thread" must really be implemented by a 1:1 mapping to a kernel lieghtweight process. Many systems use "green" threads or some other sort of user-space threading, so the implementation distinction is not as important.

To be completely optimal you wouldn't want a pure implementation of EITHER model, but some sort of hybrid in which you have a some number of threads, which each do some async IO. It all depends on the CPU/IO ratio for your app.

Maintainability is consistently underrated. You should shoot for a performance goal, and then create the SIMPLEST POSSIBLE SOLUTION to meet that goal...not spend days, weeks, months in coding kung foo zen. By the time you get out of your optimization stupor, hardware and network advances will probably have rendered your solution unnecessary.

Re:Headline: Linux Makes Bad Code Look Better

[LunaticLeo]

what has your two memory management lines got to do with anything?

When the kernel needs to get free memory pages, it looks on some sort of free page list or it has to find a victim page. There are lots of strategies to find victim pages. The reverse page table mappings allow the kernel to scan only pages in memory for victims and not have scan the virtual mappings for pages in memory AND satisfy some "victimizable" criteria.

Secondly, reverse page mappings alow you to know more about the page, like it is shared by several processes. The quick access to additional information allows you to make better choices about which page to swap out

I don't pretend to know everything about virtual memory systems. However, I do read LKML and these are the arguments others have made which to some degree I am parroting here (but I do get the gist of it).

Re:Headline: Linux Makes Bad Code Look Better

[LunaticLeo]

Thats not informative, thats just dumb.

I think I'll disagree with you on that.

How do you think things like I/O completion are implemented?

I've heard that the have a thread waiting for each completion port, sorta like the aio_* implmentation on Irix. But, you might be thinking I am some Microsoft stuge, just so you know I don't do windows; sometimes FreeBSD (cuz it has some cool stuff), but nearly exclusively Linux for the last 10 years.

Crappy threading performance on Linux (and in unix in general) has historically been because of crappy threading libraries, and because process creation is relatively cheap so people tended to just fork children instead of spinning threads.

I think I have used those very words myself.

Just because you're doing it in kernel threads instead of userspace threads doesn't mean that it's not threaded.

I never suggested user space threads. They don't require kernel intervention, but they don't utilize multiple CPUs. And M:N threading libraries are out of fashion. Apparently all those smart people realized that two levels of schedulers were hard to make fair and fast at the same time. NGPT lost in favor of 1:1 in NPTL (LinuxThreads just blew chunks). Solaris9 now makes 1:1 threading library default, where Sun used to trumpet the glories of their then new, now old, M:N thread library (now if they would just shoot mtmalloc in the head and be done with it).

And I'm _really_ not sure why you're showing us a frigging Python framework as some sort of example of super-performant network programming. Pythons great and all but a performance monster it is not. "Yeah, boss, we use a runtime-compiled interperter for all our performance-critical code, but by God we avoid context switching!"

As I said in another post in this thread: scalability comes from design not from optimization. If Perl or Python or Java are a constant factor slower, but I use a better algorithm, I can beat C/C++ hands down. I saw someones sig that said something like "If it can scale, I can buy performance." So while scalability != performance there is a relationship. And with a bunch of cheap PCs running Linux, I can crush someone elses Apache/JBoss/Websphere "enterprise" app running on some Sun E15K monstrosity.

MPM Module(s) Used with Apache?

[Lexicon]

I would be interested in knowing which MPM module(s) they used with Apache in their testing. Whether they used worker, prefork, or something else could make a big difference in serving performance. It would also test different areas of kernel performance I would think.

TwistedPython has a severe limitation

[jgardn]

TwistedPython runs everything from a single thread. Even if you have multiple threads, only one thread can be running at a time due to Python's GIL (Global Interpreter Lock).

Diregard the fact that TwistedPython is still in its infancy and thus immature compared to its rivals.

The fundamental problem with it is that it will not scale well to multiple processors because all of the python threads must interact and share the same memory. It's not like Apache which has one master process that handles incoming connections, and several children distributed across the processors, using seperate sections of memory - it is only a single process with multiple Python threads, forced to run one after another thanks to the GIL.

Apache scales far better than TwistedPython. When you have a properly scaleable database backend, or some kind of application logic layer that can scale, then it behaves very well as the load increases and on more advanced hardware.

Understand that I'm not saying that "TwistedPython Sux!" I am saying that I won't be using it for an application server that must scale. Once Python overcomes the GIL problem (and offers shared memory for Python objects) then TwistedPython may begin to have some hope of actually scaling.

Event driven application frameworks are kludges

[Per Abrahamsen]

...around poorly designed operating systems where threads are slow. The cool thing about 2.6 is that there now is one less motivation for using such kludges.