Slashdot Mirror

← Back to Stories (view on slashdot.org)

OSS Web Stacks Outperformed by .Net?

Posted by ryuzaki0 on from the sad-penguin dept.

Gimble writes "eWeek has an article up that looks at the performance of portals using open source stacks and comparing them to their MS equivalents. The article's conclusion is that .Net outperforms the open source stacks, mainly because of its tighter integration, but also notes that running the open source stacks on Windows (WAMP) delivered strong performance." From the article: "Based on our forays into user forums for many top open-source enterprise applications, there are many IT managers attempting to run open-source products on Windows servers--attracted, no doubt, to the benefits and efficiencies of using open source without having to become Linux administrators. The results of our WAMP stack tests indicate that these folks might be on to something."

19 of 349 comments (clear)

  1. Left out? by meburke · · Score: 5, Insightful

    From the article: "The criticism we expect to hear most is of the stacks we left out--including commercial J2EE platforms, such as those available from BEA Systems, IBM, Oracle and Sun Microsystems, as well as the many other database and server platform permutations." I can't believe they came to this conclusion on such little data. They did, however, create a blog to disparate results can be shared.

    --
    "The mind works quicker than you think!"
    1. Re:Left out? by happyemoticon · · Score: 4, Interesting

      They seem to have a generalized poverty of data. Their charts seem absurd to the point of being straw men. I mean, come on - I don't think there's anything seriously wrong enough with Linux that WAMP would have a score of 12 transactions/sec, competing with Windows, whereas LAMP would have a performance of 2. My experience with Windows vs. Linux has always been that they are similar in terms of speed from pure processing tasks to 3d games. Sometimes Windows does a little better, sometimes Linux is better. But they're usually in the same ballpark. The numbers are just too neat. It's like they put up a chart saying that Republicans, Germans, Koreans and Canadians have sex once a month, whereas Democrats, Brazilians, and the British have sex five million times per second.

      Moreover, the whole rest of the article is morass of poetic circumlocution. My gut feeling as somebody who works with words a lot is that they're trying to obfuscate something with a giant wall of banal text. I don't know exactly what that is, because I don't feel like reading all of it, but if I had to guess I'd say that the real thing to take away from this article is that anybody can set up .NET and a Windows box, but that it requires a little bit of patience and research to make Linux work properly - research that these people were not willing to do.

  2. WAMP vs LAMP by Poromenos1 · · Score: 5, Informative

    I'm no system administrator, but I have a home box running WAMP (XAMPP on 2003) and it's good enough for my needs. Recently I tried out Ubuntu Server to see what it's about, and I'm tempted to buy a new pc just to run that. When I tried to run mod_python under WAMP it took a whole lot of debugging and configuration (apparently it didn't like the already installed python 2.4), but with Ubuntu it was as simple as apt-getting it.

    I would very much like it if I could continue using Windows (because I run other programs that are not available on Linux) but it can't match the simplicity of Ubuntu.

    --
    Send email from the afterlife! Write your e-will at Dead Man's Switch.
  3. I'm going to have to use the /. rule of thumb by Clockwurk · · Score: 5, Funny

    If Linux wins, its a fact.

    If M$ wins, its fud and was paid for.

    If apple wins, its because of Steve Jobs.

    If OS/2 wins, we're trapped in a parallel universe.

    1. Re:I'm going to have to use the /. rule of thumb by JacksBrokenCode · · Score: 5, Insightful

      It isn't as if they haven't been caught buying studies before. So the distrust is well justified.

      Justified perhaps, but automatically accurate... not necessarily.

      Plain and simple fact is if Microsoft could compete with the usefulness of a solid Linux distro their product would speak for itself. In some cases this is true but in essentially all technical senses Microsoft is just a plain loser.

      Ok, fanboy.

      Most of Microsofts problems is that they don't listen to the customers. I mean sure they listen to Dell, RIAA, MPAA, maybe even IBM and other big wigs. But what about us users? What does WGA give me in terms of a useful feature? What does the bloat that is WMP give me over a simpler mplayer?

      So when MS doesn't add new features they are slammed for not innovating enough, and when they do add new features they are slammed for contributing to bloat that you don't want. People bitched about IE6 not having tabs, etc. Firefox came out and MS finally realized it had to update IE so it added a lot of features people were asking for and the most-heard comment on Slashdot after IE7b2 was released was "it's ugly". Face it: Microsoft just can't win.

      Why must they invent their own file formats [e.g. Office files, WMV, WMA] that are proprietary instead of using or establishing more open standards? etc, etc, etc. Everything MSFT does is to benefit the stock holders through locking the "customers" into their system. Use our OS, use our office suite, use our media tools, use our development tools. All the while they ignore any sense of established standards [ISO C99 anyone?] which make interoperability a bitch for Windows users. There is simply no reason why MSFT uses these awful platform dependent libraries.

      You sorta' answered yourself there.

      Take DirectX for instance. On any OTHER platform you combine Allegro with OpenGL and have essentially the same thing [just 1/10th the size and in C]. But no, we must use the DX "experience" because somehow the hype makes it shinier!

      Not every product is a winner. MS historically doesn't release every single product as a beta and quietly stop promoting the ones that suck. Instead they release final versions and some fall on their face. No company has a perfect record.

      I know what I'm saying is "no duh", but you seemed to be hinting that MSFT hatred is not warranted. Us "OSS" users don't hate MSFT because it's better. We hate it because it lulls people into a sense of superiority when all it does is move to separate them from their money. It creates nightmares for us who chose to chose.

      The problem is not that criticism isn't warranted, it's that MS can't win no matter what. If they release a weak or buggy product they get slammed, but if they take too long to release they get slammed. If they don't add new features they get slammed, but if they add new features it's called bloat. If an MS product gets bad reviews the reviewers are being honest, but if they get good reviews the reviewers are obviously being paid. For years MS got slammed for security issues, and they beefed up SP2 and suddenly there were waves of "but it broke my application" complaints. The list goes on.

      Microsoft has gotten so big that they are in the impossible position of trying to keep everyone happy. I'm not particularly a Microsoft "fan", but I hate this wanton "Micro$oft is teh suxors!1!" b.s. OSS fanboys need to grow up and realize that Microsoft can't go back in time and correct the sins of the past, and since it is a monopoly it can't just genuinely screw its customers and break every file/application by releasing a new version of Windows that corrects all the problems of the old versions but offers no legacy support. They have a tough balancing act to do and, while they're not perfect, they're getting better.

  4. Worst... Benchmark.... Ever... by ThinkFr33ly · · Score: 5, Insightful

    I'm still not exactly sure what they tested. They have vague terms like "Request per Second" and "Throughput", yet they don't actually say what each page that is being requested is actually doing.

    For the .NET tests they say they used "Sharepoint". Huh? For what? Considering that Sharepoint is *extremely* complicated and has incredibly rich functionality they should be very clear as to what they used it for.

    Not to mention the fact that using a portal application in your tests means that there is really very little way to isolate if it was a poorly written portal application or a crappy framework that the portal application was built on that's causing perf issues.

    It is very difficult to test framework vs framework, but this is just about the worst way one could even attempt it.

    At absolute best, this compares portal frameworks on various platforms. Even if they were trying to do that, they did a piss poor job.

  5. Test components too variable by IflyRC · · Score: 5, Insightful

    How can you test the performance of a stack and compare it to others when the back end database servers, portal software and web server software is different?

    How is the statement that .NET stacks are faster true when it could be the implementation of SQL being faster than MySQL? This test just doesn't make sense to me.

  6. ASP.Net is pretty nice... by aztracker1 · · Score: 4, Insightful

    Not to start a war here, but ASP.Net is a pretty damned nice environment to work under... I've used a lot of PHP, Cold Fusion, some JSP, and Classic ASP in the past. ASP.Net is my favorite.. I've been peeking in with Ruby/Rails but just haven't had the time to dive in much. of the above Ruby/Rails is probably the closest competitor on an ease of development/functionality level.

    --
    Michael J. Ryan - tracker1.info
  7. "performance"? by erroneus · · Score: 5, Insightful

    What was the set of measures? For me, "performance" has more to do with uptime, reliability and security. Those are the performance standards I care about.

  8. Holy Throughput! by blazerw11 · · Score: 4, Funny

    I'm wondering if the high throughput numbers for the .Net stack were caused by it deliving huge binary files to the client. Ya know, 17MB Active X controls. Anyway, I didn't randomly come up with this conclusion, the article didn't mention the transactions per second for .Net. So, I conclude from ALL of the data that it did one transaction of 17MB*.

    *No math was done to come up with the 17MB figure.
    Also, no animals were harmed during the writing of this comment.

    --
    A great many people think they are thinking when they are merely rearranging their prejudices. -- William James
  9. The article is NOT that conclusive by MK_CSGuy · · Score: 5, Informative

    I've read the article before it hit /. and their conclusion is that there is no clear winner. .Net outperforms OSS solutions on some tests and vice-versa. The surprising(*) results are how good WAMP performed in some of the tests (if you really want specifics RTFA). Here is a direct link to the tests.

    * - I've seen similar results in benchmarks of Mono & .Net, i.e. Mono apps with .Net framework vs pure .Net and pure Mono, so although there is no connection between JIT compilers and web servers performance, the trend is there.
    Too bad the article haven't touched Mono.

  10. MindCraft would be the "worst" so far. by khasim · · Score: 5, Interesting

    But this is a very close second place.

    #1. NO tuning was done on the LAMP stuff. None at all. They ran the stuff "out of the box".

    #2. They didn't write their own app. That means they didn't test the SAME processes on each system.

    #3. They didn't bother to find WHERE the differences were. Is it in the IP stack? Is it in the OS? Is it in the scripting language? Is it in the app?

    How bad can "research" be and still be published in "eWeek"? There wasn't any research done for that article.

    Microsoft has, in the past, taken various short-cuts when IIS was the server and IE was the browser. Is that the case in this "study"? Are the other "stacks" "slower" because they follow the protocols?

    You won't know because they'd didn't LOOK for the REASON behind their "results".

    At least MindCraft was paid to do poor research.

  11. Re:Linux still wins by corren · · Score: 4, Interesting
    It's actually much worse than $1500 a box. And I'm a windows guy, I admit it, however licensing for windows products REALLY sucks for a small business that wants to run legit.

    Here's the price breakdown for a SINGLE webserver that allows external connections to authenticate (non-domain, say a e-commerce site with user accounts) against a SINGLE SQL 2005 Database. Sql Express is free, however it's not licensed for unlimited users in a production environment.

    Web Server (Prices from CDW.com):
    • 1 Copy Windows 2003 R2 (5 CALs): $959.99
    • 1 External Connector License for Windows: $1,969.99
    • Total: $2,929.98
    SQL Server:
    • 1 Copy Windows 2003 R2 (5 CALs): $959.99
    • 1 Copy SQL Server 2005 Workgroup Edition 1 Processor License: $3,819.99
    • Total: $4,779.98
    Grand Total for a single web and sql box: $7,709.96.

    And don't forget that you'll need SOME hardware to run that OS. Even barebones boxes with no data protection will run you $500 a box.

    So, to start a basic e-commerce site on the legit, you're talking roughly $9,000 for windows and $1,000 for Linux/OSS.

    TOUGH sell for Microsoft for the little guy.
  12. No wonder Linux sucked! by Just+Some+Guy · · Score: 5, Insightful
    For our tests, we ran what is essentially a pure Zope/Plone implementation, with Plone running on a SUSE Enterprise Linux system.

    In some benchmarks, Plone was an average performer, sticking close to the middle. This is actually better than we expected, given that the Plone documentation is very upfront about the fact that Plone shouldn't be used alone in a production environment and should be run behind other servers to improve performance.

    So, they ran an outward-facing Zope server (after being explicitly told not to) and the performace was lackluster? Go figure. In the real world, they'd run Zope behind an Apache or Squid proxy (as per every installation recommendation I've ever seen) which would immediately boost throughput by an order of magnitude. In short, using Zope to dynamically generate static content instead of caching the results whenever possible is insane, and pretty much no one does it. They also apparently forgot about ZEO, although I'm not sure how you can be savvy enough to get Zope up and populated without knowing about it's built-in clustering.

    Apparently they had no interest in any tuning whatsoever, to the point of de-tuning it by installing it in an explicitly unrecommended configuring. And then it lost. Go figure.

    --
    Dewey, what part of this looks like authorities should be involved?
  13. If I may expand upon your post ... by khasim · · Score: 4, Interesting

    You raise some excellent points. If I may be allowed to expand upon them ...

    #1. Set a price limit. You can set multiple limits ($1,000 vs $5,000 vs $25,000 vs $100,000 vs $1,000,000+). The key concept here is that you get different characteristics as your budget increases/decreases. What characteristics does each "stack" offer in each price range? Yep, this does give the advantage to Free stacks (Free like speech, free like beer). Deal with it. In the Real World the bottom line is the bottom line. Each team gets to spend the money however they want to.

    #2. Get the "experts" to tune each stack. BUT they must document each modification they make, including WHY they made that modification (what testing did they run and how did those test results tell them what mod's to make) AND they are only allowed to make mod's that can be found via public websites (no secret tuning parameters that are only known to the organization writing that software) AND they aren't allowed to touch any source code. They get what everyone else gets.

    #3. The fun part. Each team gets to pull apart the work of the other teams. Even if your solution is faster for the specs given, how much wiggle room do you have? Is faster and fragile better than slower and stable? How much "slower" is acceptable for how much more "stable"? Can the other team defeat your security (network access only)?

    #4. Freeze those systems. Then, over the next year, patch them and re-test them. Do the patches break the "tuning" that was done?

    Now that would be an informative test process (and would result in lots of articles and interviews for the magazine publishing it).

    Yeah, you can run Linux / Apache / MySQL / perl on a single drive workstation and get damn good performance for less than $300.

    But that will be completely different from Oracle / Java on a cluster of Suns costing $10,000,000. And not just in the number of boxes you'd be running.

  14. Cost by MrCopilot · · Score: 4, Insightful
    To do a fair comparison I would like to see the Cost of the systems as set up.

    To test the .Net stack, we ran Windows Server 2003 R2, SQL Server 2005 and SharePoint Portal Server 2003. Across the board, this configuration performed very well, with the top overall average throughput (by far) at 4.59M bps.

    Quick check.....
    $2,792.00 (Froogle Directron) Microsoft Windows Server 2003 R2 Enterprise, 25 Clients
    $5,489.18 (Froogle Non Academic) SQL Server 2005 Complete
    $5,619.00 (MS Website Retail) SharePoint Portal Server 2003 Server License with 5 CALs

    $1,124.00 (Dell) Suse Enterprise Linux 9 With Server Hardware
    http://configure.us.dell.com/dellstore/config.aspx ?c=us&cs=555&l=en&oc=MLB1580&s=biz Couldn't find Suse Enterprise 10 Integrated LAMP Stackhttp://www.novell.com/products/linuxenterpris eserver/lamp.html

    Hmmm, Could train a couple of Windows Admins with $11,000. Better yet just Hire a good Linux Admin.

    To a large degree, we credit this strong showing to the high level of integration that exists among the components of this stack. While most of the open-source and Java systems are developed independently of each other, each of the .Net components is designed specifically to integrate and perform well together. Even if the .Net stack had bombed convincingly in these tests, it would probably still maintain popularity in many companies.

    Some people (PHBs) will never come around.

    But its strong showing should give companies confidence that the .Net stack will handle most high-level enterprise needs.

    For more than $12Grand it better blow away the Free Alternatives and configure itself and require zero admin.

    I know I will get slammed for not using TCO but I don't believe those numbers at all. In my experience it takes the same amount of time for day to day maintenance. And when there is a problem (and there will be, no matter which one you choose) It costs me less time and therefore money to bring back up the Linux box.

    Cost is not the only factor in a buying decision but is a factor, and if performance is arguably equal than it is a huge factor.

    --
    OSGGFG - Open Source Gamers Guide to Free Games
  15. Re:Short memories by plague3106 · · Score: 4, Insightful

    Even if that was true I'd still rather have a proper UNIX environment anyways. Not to mention a real multi-user OS without shelling out huge dollars. Oh and the development tools and ...

    Sorry, but with modern windows, what exactly is missing that dis-qualifies it from being a real multi-user OS? And FWIW, I've found VS2005 to be much better than any development studio I've tried for linux.

    If you buy Windows to run AMP servers ... you're wasting your money.

    So you think its better to have servers which don't integrate with your corporate network? $400 for Web edition isn't a whole lot to anyone running web farms.

    Linux and BSD are more efficient to work with in server contexts

    Your opinion, there's no fact to that. I replace my linux server at home with SBS 2003 because its easier to manage the network using SBS2003 than it was with Linux + SMB.

    work better with 64-bit processors [Win64 is a huge compatibility joke atm]

    I've heard quite differently; indeed, some people claim XP64 is the best desktop OS, even better than 32bit XP. I don't think the 64 bit servers are suffering huge problems either.

    don't require you to call India each time your HD breaks and you need a re-install.

    A bunch of FUD here; mearly replacing an HD doesn't require calling anyone, and alot of shops image the drives every night so that if there is a failure, they slap on the saved image and are up and running again in no time.

    Heck, in BSD/Linux doing a ghost of your system is as simple as burning a tarball to a DVD.

    You can easily save an image of a Windows installtion as well.

    No need for 3rd party ghosting tools and praying that Windows lets you "get away with" using your OS...

    Go ahead and spend hours and hours tarballing your server; the fact is that it would be faster to just create an image of the HD and burn that to a DVD. There's no praying involved; my former employer had great success doing such restores. And putting down a new image is faster than un-tarring a file to disk again.

  16. it's not "stacks", it's portals by m874t232 · · Score: 4, Interesting
    What they tested were "portals":

    We used portals we consider popular--Microsoft SharePoint Portal Server 2003 (built on ASP), XOOPS (PHP), Plone (Python), and Liferay and JBoss Portal (JSP).


    Now, I know that Plone is a dog, and XOOPS may be popular on Sourceforge, but I don't think it's the most obvious choice for building a high performance portal using PHP. So, using these two as the basis for testing is silly.

    The fact that JBoss Portal on Windows outpaces JBoss Portal on Linux has a simple reason: JBoss isn't fully open source; crucial parts of it (namely the Java runtime itself) are under Sun's control, and hell will freeze over before Sun bothers to do a good job implementing Java for their competitors' Linux systems.

    As for things generally running faster on Windows, that's implausible. Differences between raw Windows and Linux system performance are at most in the single digit percentages, so if they saw any significant differences between the same applications running on top of the two platforms, either the application vendor spends more time tuning for Windows (as in Sun Java), or the testing labs screwed up.

    In fact, the whole test is really ill conceived: none of the "portals" they compared provide the same functionality; it just doesn't make sense to test them against each other. Overall, this test mostly seems to test the competency of eWeek, and they aren't doing too well.
  17. Re:Short memories by Cal+Paterson · · Score: 4, Insightful

    Sorry, but with modern windows, what exactly is missing that dis-qualifies it from being a real multi-user OS? And FWIW, I've found VS2005 to be much better than any development studio I've tried for linux.

    I think the grandparents point is that though it is technically a multi-user OS, it isn't a very good one. The kernel level schedulers on Windows give a really poor response under multiple sources of medium to heavy load.

    I'm not aware of that many "development studios" for Unix. There's a couple, like eclipse and such, but many Unix writers tend to work a different way. In line with the Unix philosophy (rules 1, 3, 4, 5, 7 and 8 if you're interested) there's a lot of people who keep the editor, compiler, linker etc all seperate. If you're looking for a "development studio" for Unix, I think you're looking for the wrong thing. We don't have a Start Menu either. :)

    So you think its better to have servers which don't integrate with your corporate network? $400 for Web edition isn't a whole lot to anyone running web farms.

    Well, I'm sure anyone could have pointed out that large companies with large budgets can afford the $400 dollar cost. But that $400 dollars is a cost, and to a lot of companies, it's a big cost. It's an even bigger cost when many difference licences have to be purchased. (Obviously, Unixes have costs of their own, but the costs tend to increase in line with the use being made, or the complexity of the "solution" etc, and up-front licences payments (which are immoral, in my opinion) normally end up being a far less weildy solution than Free Unix is) And in my, and the grandparents opinion, it's an unjustified cost.

    Your opinion, there's no fact to that. I replace my linux server at home with SBS 2003 because its easier to manage the network using SBS2003 than it was with Linux + SMB.

    Of course it's opinion! The whole damn subject is opinion! As if a (very very poorly run) test on the speed of a server is the last word on the subject! There's little universal fact to your personal experience either. ;)

    I've heard quite differently; indeed, some people claim XP64 is the best desktop OS, even better than 32bit XP. I don't think the 64 bit servers are suffering huge problems either.

    Have you actually used it? WinXP 64bit really doesn't work very well at all. I wouldn't know about 64 bit servers, but 64 bit WinXP is really not something that works very well.

    Go ahead and spend hours and hours tarballing your server; the fact is that it would be faster to just create an image of the HD and burn that to a DVD. There's no praying involved; my former employer had great success doing such restores. And putting down a new image is faster than un-tarring a file to disk again.

    It's possible to use either image or tar with Unixes. Tar is useful because it's easy to use diffs and delta compression on backups then. Tar's also useful when you only want to backup a certain area; such as /etc or /var. It's not like you're stuck with tar either - disk imaging still works fine. When you can tarball up only the parts of the machine you need to keep, the idea of disk imaging seems redundant (for most cases).