OSS Web Stacks Outperformed by .Net?

Gimble writes "eWeek has an article up that looks at the performance of portals using open source stacks and comparing them to their MS equivalents. The article's conclusion is that .Net outperforms the open source stacks, mainly because of its tighter integration, but also notes that running the open source stacks on Windows (WAMP) delivered strong performance." From the article: "Based on our forays into user forums for many top open-source enterprise applications, there are many IT managers attempting to run open-source products on Windows servers--attracted, no doubt, to the benefits and efficiencies of using open source without having to become Linux administrators. The results of our WAMP stack tests indicate that these folks might be on to something."

Comments

Left out?

[meburke]

From the article: "The criticism we expect to hear most is of the stacks we left out--including commercial J2EE platforms, such as those available from BEA Systems, IBM, Oracle and Sun Microsystems, as well as the many other database and server platform permutations." I can't believe they came to this conclusion on such little data. They did, however, create a blog to disparate results can be shared.

WAMP vs LAMP

[Poromenos1]

I'm no system administrator, but I have a home box running WAMP (XAMPP on 2003) and it's good enough for my needs. Recently I tried out Ubuntu Server to see what it's about, and I'm tempted to buy a new pc just to run that. When I tried to run mod_python under WAMP it took a whole lot of debugging and configuration (apparently it didn't like the already installed python 2.4), but with Ubuntu it was as simple as apt-getting it.

I would very much like it if I could continue using Windows (because I run other programs that are not available on Linux) but it can't match the simplicity of Ubuntu.

I'm going to have to use the /. rule of thumb

[Clockwurk]

If Linux wins, its a fact.

If M$ wins, its fud and was paid for.

If apple wins, its because of Steve Jobs.

If OS/2 wins, we're trapped in a parallel universe.

Re:I'm going to have to use the /. rule of thumb

[JacksBrokenCode]

It isn't as if they haven't been caught buying studies before. So the distrust is well justified.

Justified perhaps, but automatically accurate... not necessarily.

Plain and simple fact is if Microsoft could compete with the usefulness of a solid Linux distro their product would speak for itself. In some cases this is true but in essentially all technical senses Microsoft is just a plain loser.

Ok, fanboy.

Most of Microsofts problems is that they don't listen to the customers. I mean sure they listen to Dell, RIAA, MPAA, maybe even IBM and other big wigs. But what about us users? What does WGA give me in terms of a useful feature? What does the bloat that is WMP give me over a simpler mplayer?

So when MS doesn't add new features they are slammed for not innovating enough, and when they do add new features they are slammed for contributing to bloat that you don't want. People bitched about IE6 not having tabs, etc. Firefox came out and MS finally realized it had to update IE so it added a lot of features people were asking for and the most-heard comment on Slashdot after IE7b2 was released was "it's ugly". Face it: Microsoft just can't win.

Why must they invent their own file formats [e.g. Office files, WMV, WMA] that are proprietary instead of using or establishing more open standards? etc, etc, etc. Everything MSFT does is to benefit the stock holders through locking the "customers" into their system. Use our OS, use our office suite, use our media tools, use our development tools. All the while they ignore any sense of established standards [ISO C99 anyone?] which make interoperability a bitch for Windows users. There is simply no reason why MSFT uses these awful platform dependent libraries.

You sorta' answered yourself there.

Take DirectX for instance. On any OTHER platform you combine Allegro with OpenGL and have essentially the same thing [just 1/10th the size and in C]. But no, we must use the DX "experience" because somehow the hype makes it shinier!

Not every product is a winner. MS historically doesn't release every single product as a beta and quietly stop promoting the ones that suck. Instead they release final versions and some fall on their face. No company has a perfect record.

I know what I'm saying is "no duh", but you seemed to be hinting that MSFT hatred is not warranted. Us "OSS" users don't hate MSFT because it's better. We hate it because it lulls people into a sense of superiority when all it does is move to separate them from their money. It creates nightmares for us who chose to chose.

The problem is not that criticism isn't warranted, it's that MS can't win no matter what. If they release a weak or buggy product they get slammed, but if they take too long to release they get slammed. If they don't add new features they get slammed, but if they add new features it's called bloat. If an MS product gets bad reviews the reviewers are being honest, but if they get good reviews the reviewers are obviously being paid. For years MS got slammed for security issues, and they beefed up SP2 and suddenly there were waves of "but it broke my application" complaints. The list goes on.

Microsoft has gotten so big that they are in the impossible position of trying to keep everyone happy. I'm not particularly a Microsoft "fan", but I hate this wanton "Micro$oft is teh suxors!1!" b.s. OSS fanboys need to grow up and realize that Microsoft can't go back in time and correct the sins of the past, and since it is a monopoly it can't just genuinely screw its customers and break every file/application by releasing a new version of Windows that corrects all the problems of the old versions but offers no legacy support. They have a tough balancing act to do and, while they're not perfect, they're getting better.

Worst... Benchmark.... Ever...

[ThinkFr33ly]

I'm still not exactly sure what they tested. They have vague terms like "Request per Second" and "Throughput", yet they don't actually say what each page that is being requested is actually doing.

For the .NET tests they say they used "Sharepoint". Huh? For what? Considering that Sharepoint is *extremely* complicated and has incredibly rich functionality they should be very clear as to what they used it for.

Not to mention the fact that using a portal application in your tests means that there is really very little way to isolate if it was a poorly written portal application or a crappy framework that the portal application was built on that's causing perf issues.

It is very difficult to test framework vs framework, but this is just about the worst way one could even attempt it.

At absolute best, this compares portal frameworks on various platforms. Even if they were trying to do that, they did a piss poor job.

Test components too variable

[IflyRC]

How can you test the performance of a stack and compare it to others when the back end database servers, portal software and web server software is different?

How is the statement that .NET stacks are faster true when it could be the implementation of SQL being faster than MySQL? This test just doesn't make sense to me.

"performance"?

[erroneus]

What was the set of measures? For me, "performance" has more to do with uptime, reliability and security. Those are the performance standards I care about.

The article is NOT that conclusive

[MK_CSGuy]

I've read the article before it hit /. and their conclusion is that there is no clear winner. .Net outperforms OSS solutions on some tests and vice-versa. The surprising(*) results are how good WAMP performed in some of the tests (if you really want specifics RTFA). Here is a direct link to the tests.

* - I've seen similar results in benchmarks of Mono & .Net, i.e. Mono apps with .Net framework vs pure .Net and pure Mono, so although there is no connection between JIT compilers and web servers performance, the trend is there.
Too bad the article haven't touched Mono.

MindCraft would be the "worst" so far.

[khasim]

But this is a very close second place.

#1. NO tuning was done on the LAMP stuff. None at all. They ran the stuff "out of the box".

#2. They didn't write their own app. That means they didn't test the SAME processes on each system.

#3. They didn't bother to find WHERE the differences were. Is it in the IP stack? Is it in the OS? Is it in the scripting language? Is it in the app?

How bad can "research" be and still be published in "eWeek"? There wasn't any research done for that article.

Microsoft has, in the past, taken various short-cuts when IIS was the server and IE was the browser. Is that the case in this "study"? Are the other "stacks" "slower" because they follow the protocols?

You won't know because they'd didn't LOOK for the REASON behind their "results".

At least MindCraft was paid to do poor research.

No wonder Linux sucked!

[Just Some Guy]

For our tests, we ran what is essentially a pure Zope/Plone implementation, with Plone running on a SUSE Enterprise Linux system.

In some benchmarks, Plone was an average performer, sticking close to the middle. This is actually better than we expected, given that the Plone documentation is very upfront about the fact that Plone shouldn't be used alone in a production environment and should be run behind other servers to improve performance.

So, they ran an outward-facing Zope server (after being explicitly told not to) and the performace was lackluster? Go figure. In the real world, they'd run Zope behind an Apache or Squid proxy (as per every installation recommendation I've ever seen) which would immediately boost throughput by an order of magnitude. In short, using Zope to dynamically generate static content instead of caching the results whenever possible is insane, and pretty much no one does it. They also apparently forgot about ZEO, although I'm not sure how you can be savvy enough to get Zope up and populated without knowing about it's built-in clustering.

Apparently they had no interest in any tuning whatsoever, to the point of de-tuning it by installing it in an explicitly unrecommended configuring. And then it lost. Go figure.