Tomcat/Cocoon Performance on Production Sites?

Saqib Ali asks: "A Tomcat/Cocoon setup can be used as a framework for publishing XML content. I have this setup running in testing environment for about 5 months now, and would like to move it into production. Is anyone using a similar setup and can comment on Tomcat's ability in handling the load from a production environment, or should we look at other Java Servlet engines like BEA, Jetty or Resin? If Open Source solutions can't work under these kind of pressure what should we be looking at in terms of different commercial solutions for publishing XML content?" A similar question with respect to Tomcat and Jetty was discussed in this previous discussion. What effects does using Cocoon have on Tomcat-based environments?

Comments

Tomcat runs fine

[Tomah4wk]

Ive had multiple tomcat servers running on a sunfire E6800 (24 processor, 32GB ram etc..) under solaris using j2ee 1.3 and they have all been working fine, not one single server failure. One of which is a webapp server that has a variety of apps running on it, and the other is essentially an 'xml server' used for a varitey of applications (globus, grid computing etc). Theses are both based on tomcat 4.0, and there are some more running but i forget which version of tomcat they use

Testing?

[FreeLinux]

I have this setup running in testing environment for about 5 months now, and would like to move it into production.

I can't find the words to ask this without sounding like an ass but, I really don't mean to be.

I have to ask though, what have you been testing for the past 5 months. When testing a server, web, mail, database, whatever, one of the early tests that I perform is load testing. Have you not done load testing to determine if Cocoon will handle your anticipated load and then some?

Why waste 5 months development time on a product that you don't know will support the load? Suppose it doesn't handle it (which I doubt), are you going to chuck 5 months work to move to BEA, WebSphere, IIS????? What have you been testing?

Re:Testing?

[Pengo]

With MOST of the JAVA Based projects I have worked on, it is almost ambiguous which platform you work on as long as you follow good standards and don't step too deeply into proprietary land.

Most of the gotchas will be w/the database pooling which each vendor seems to handle w/their own style. As long as re-usable class components where used, and a solid MVC model is in place, you should be able to move from one vendor to another without loosing much time.

I have bounced from Resin to Jetty to Websphere to Tomcat and had few problems.

So, he probably won't loose 5 months worth of work if he moves to another platform, but maybe 5 days of reconfiguring and possible bug fixes that may have been hidden or exposed by one platform or another.

Re:Testing?

[stonebeat.org]

Oh we were testing few other solutions as well. It is not like we spent 5 mnths just on 1 solution. Plus we have that kind of buffer (in timeline), where we can spend time on testing for 5-10 mnths.

Caucho's Resin

[Pengo]

I have taken 2 production environments from Tomcat to Resin.. I have Resin to be more robust and easier to develop w/than Tomcat for various reasons. (First and formost the error message points you strait to the line of JSP rather than the error code linking back to the servlet. Resin does have some performance perks over Tomcat, but unless it's a VERY high volume site w/lots of load balanced servers, it's not going to probably matter much.

The thing I would say #1 is what you know. I know Resin inside and out, and when something goes wrong.. I know how to deal with it. Tomcat, I can't say the same thing. For this reason, it's been well worth the 500 dollars per server to deploy resin, as it's cheeper for us to maintain.

But, I have been working with resin for 3 some years now.

If tomcat works for you, go for it.. if it holds up under the stress tests , etc.. use it! :) You have had it in testing for a LONG time, at least by my experience..

Good luck either way, but if you do look to a commercial product, definately give Resin a serious look. http://www.caucho.com

load from a production environment?

[lorcha]

Well, that depends on whose production environment. What do you mean by load? 1req/min? 1req/sec? 1000req/sec? What does it take to fulfill a request?

No one on /. is going to be able to answer this question for you. You'll need to load-test your Tomcat configuration yourself and see whether or not it is performant.

Personally, I prefer resin, but have had successful JBoss/Tomcat configurations. But every situation is different, so you'll have to do some load testing.

Good luck!

Re:load from a production environment?

[stonebeat.org]

I know that load from from env differs from another env. However, my question was if anybody is "using" Tomcat+Cocoon in production env. And if so what is their experience?

Re:load from a production environment?

[Anonymous Coward]

yes. see below. that setup has been running fine since 1999 BTW. 24/7 since 1999 with an average sustained load.

performance numbers.

[Anonymous Coward]

e-business intranet site.
Server is a E450 with 4 x USii 450Mhz, 8 GB RAM and A3500 RAID-5 using FCAL SCSI 10K RPM drives.
40 megabytes data processed per request.
DB Server is Oracle 8i (Enterprise) with 6 GB RAM allocated to it. Servlet running on tomcat uses DB connection pooling and caching + the oracle JDBC driver. Approx. 40 megabytes of data is exchanged with oracle per request using simple (SELECT/INSERT) type statements with all the logic in the servlet and no SQL processing involved.
Web & Appserver is Tomcat 3.2.4 (the newer versions crashed and couldnt handle the load) with JDK 1.4.1 from Sun running in hotspot server mode with 1 GB of RAM dedicated in the JVM.
Server has 1 GB RAM free for Solaris 8 & TCP autotune settings are enabled. No swap usage even on heavy loads.
Tomcat cookies are enabled and session state maintained using cookies.
Performance :
4 requests/second sustained. 5 requests/sec max recorded. 3.45 requests/sec under heavy load (large Q).

I concur

[msuzio]

I've used both Tomcat and Resin (Tomcat much more back in the 3.x days, but a little 4.x too). I found Resin to be at least somewhat faster (definately the JSP engine was a lot faster), easier to configure, and generally well worth $500/server.

That said, Tomcat works fine for my needs, but I have not tested it under heavy loads. I *have* stress-tested Resin, and I can say it has never failed me :-). I find Resin + Apache a great combination (Resin handles application stuff, Apache handles static data + security needs).

Re:I concur

[Pengo]

Agreed, I have the image content serving from apache, and mod_caucho is quite smart. I knows not to push and proxy static content over to an application server.

I have not had much luck load balancing more than 3 or 4 servers with mod_caucho tho, anything bigger than that get a real load balancer. (Even Scott at caucho has mentioned that he doesn't know of any production sites running more than 3-4 servers in a single zone...) and to move to a hardward load balancer at that point.

jrockit

[galore]

one of the simplest tomcat performance improvements i've made is by switching from sun's vm to bea's jrockit. jrockit is free to use, and in my experience a much more robust server side environment. it's faster, has more flexible garbage collection options, and a nice gui management console. it helps to have a lot of memory (i think by default it will use 75% of the available system memory), but for some of the webapps i've built, i've seen tomcat+jrockit turn out 15-20% more pages/sec than tomcat+hotspot. jrockit 8 beta came out recently; it supports jdk-1.4 and has been solid for me (so far) running tomcat, jetty, and jboss. i've even been using jrockit on my personal workstation while developing with jboss because it cuts the startup time from about 1m:45s to 50s.

http://www.bea.com/products/weblogic/jrockit/ind ex .shtml

other than that, i've found a big tomcat performance problem to be that of spawning new threads. the default connector's configuration sets minProcessors=5 (ie: 5 threads in the threadpool available to handle new requests). i find that when load is high enough to warrant more than the number of initial threads, requests get dropped on the floor while more threads are added to the pool. so don't be shy about starting enough request processors from the beginning.

also, don't be afraid to run some load tests on your app while profiling. you'll need a fast machine to run tomcat+cocoon under a profiler, but the results are often very helpful in finding bottlenecks. in particular, you'll probably find a lot of time spent in StringBuffers, which is normal for an app like yours, but it's a good place to start when you're interested in shaving a few seconds here and there.

hope that helps a little.

Re:jrockit

[Pengo]

also, don't be afraid to run some load tests on your app while profiling. you'll need a fast machine to run tomcat+cocoon under a profiler, but the results are often very helpful in finding bottlenecks. in particular, you'll probably find a lot of time spent in StringBuffers, which is normal for an app like yours, but it's a good place to start when you're interested in shaving a few seconds here and there.

Don't you mean on Strings? Unless I am HUGELY mistaken, Strings are much higher overhead than StringBuffers and StringBuggers should be used unless features of String are needed.

Not to be nit-pickley, but just thought I would point that out.

Re:jrockit

[galore]

when i say that a lot of time is spent in StringBuffer i am referring to the concatenation of Strings (or primitive types converted to Strings). String concatenation is obviously a big part of servlet engine (just take a look at the .java file generated by your preferred jsp compiler). String concatenation with the + operator is equivalent to calling the StringBuffer.append method (thanks to the compiler). concatenating Strings across multiple statements is not equivalent to just calling .append multiple times. for each statement a new StringBuffer is allocated, which is costly, and then shortly gc'd, which is also costly.

for example:

String s = "one" + "two" + "three";
return s;

is roughly equivalent to:

StringBuffer sb = new StringBuffer("one").append("two").append("three");
return sb.toString();

whereas

String s = "one";
s += "two";
s += "three";
return s;

is roughly equivalent to:

StringBuffer sb = new StringBuffer("one").append("two");
String tmp = sb.toString();
sb = new StringBuffer(tmp).append("three");
return sb.toString();

the second set of statements requires more allocation and garbage than the first, and this is the optimization i'm referring to. if you have an idea of the target size of the finished StringBuffer, it's also helpful to declare the initial size of the internal buffer with:

new StringBuffer(int size)

obviously optimizations of StringBuffers or other similarly basic methods (equals, hashCode, etc) are important in gaining every last ounce of efficiency, but i'm also not saying this is the only way, or even a good place to start. architectural efficiencies will probably make the most crucial difference, so start there before profiling your StringBuffers :)

Re:jrockit

[dubl-u]

StringBuffers should be used unless features of String are needed

Better still, use the Writer or the OutputStream. If you're just using a StringBuffer to pile up output before sending it, it's better for your CPU and your RAM usage if you just write the stuff out.

AxKit

[Matts]

You might want to consider Apache's AxKit. While parts of AxKit are written in Perl to make it faster for us to write, the key parts that do the heavy lifting are written in C for performance reasons. Some fortune 500 companies are already running their site on it, and we get an awful lot of people coming to us from Cocoon for performance reasons.

It's a full apache project (under the XML umbrella), just like Cocoon is, and incorporates the same technologies (XSLT and XSP), but a lot of people skip over it because it's not Java. Maybe now is time to re-evaluate that decision.

Yes, for about a year now

[arthurs_sidekick]

The site in question gets about 60-655K hits/day running in a 2x1.13GHz PIII/1GB RAM running Linux ( 2.4 series) with JDK 1.4 using an Apache front-end for SSL. We run Tomcat 4 and Cocoon 2, making judicious use of caching (we serve pre-transformed HTML pages generated from Docbook XML sources), but every page is dynamically generated. Haven't yet seen anything that makes me think we'll need anything beefier in the near future.

to clarify

[arthurs_sidekick]

That's 60-65K hits/day. that oughta learn me not to preview.

My 2 p

[bobibleyboo]

I have been running tomcat on 2 production servers (1 = 4xXeon 700 with 2GB Ram, 2 = 2xAthalon 1.3Ghz 1GB Ram) for about 8 months and they have been pumping out the content with little to know problems (the xeon box came down twice but it turned out to be a bad proc). In addition to the stability I have found them to quite snappy even under a heavy load. The only other pice of advice is that they like their memory at least 1GB and to run lots of daemons by default they seem to take a sec to start. Good Luck!

peak to 5% cpu!

[hellojed]

We don't run a tomcat/cocoon solution exactly, but we borrowed many of the ideas from cocoon and developed our own MVC framework for tomcat (servlet 2.3) which uses a combination of xslt and xml for our website. We serve over 1 million pages a month on a pair of p2/800 machines. Neither machine peaks 5% cpu load on an average day! Now that's performance!

To see this in action take a look at: Point2 UsedIron

Re:JAVA Based projects

[bobv-pillars-net]

With MOST of the JAVA Based projects I have worked on, it is almost ambiguous which platform you work on as long as you follow good standards and don't step too deeply into proprietary land.

<RANT>

Please excuse the off-topicness, but you've just voiced my biggest gripe against the whole Java fad. I was there when Sun touted Java as the answer to computing's Holy Grail: Write-Once, Run-Everywhere. Except that it ain't necessarily so. Nearly every day, I encounter another Java app that runs in any OS you care to use, as long as it's Windows. GRRR!!!! Whenever I hear a new Java convert espousing its merits, the hypocrisy of the whole thing just about kills me. Every point where Java is supposed to shine is directly contradicted by my personal experience. It's not just a few exceptions that spoil the image. MOST of the JAVA apps I've been forced to deal with are slow, bloated, buggy, and worst of all, NON-PORTABLE!

</RANT>

Application, not infrastructure

[PinglePongle]

Most of the responses have concentrated on which of the servlet engines people have used.

I believe you need to focus more on the application - the performance of the best servlet engine can be crippled by a poorly designed application. Cocoon (and XML/XSLT in general) is not the fastest way to render a page.

I'm assuming you've invested enough time and effort into your cocoon development to be reluctant to throw it out. I would therefore suggest rigging your cocoon components to output performance logging information, put together a load testing infrastructure, and see what happens. Work out where your bottlenecks are (usually I find one or two bits of functionality which consume disproportionate amounts of time/resource), and see whether they can be optimised. Caching will yield significant benefits, esp. if you use relatively static information.

Only once you understand the performance characteristics of your application is it time to look at the infrastructure and see whether it addresses your application's requirements. For instance, improving the performance of your JVM is unlikely to yield any benefits if your application spends 80% of it's time doing XSL transformations.

FWIW, I think the Tomcat server is plenty fast enough for mainstream use; it's usually cheaper to add load-balanced hardware than pay for a commercial product.

Re:JAVA Based projects

[Pengo]

I don't want to start a flame war here, but I have ported some VERY large java applications to various OS's, and for the most part.. things went VERY well. I also don't consider myself a 'new convert' to JAVA either. I appreciate your rant as a Windows/Linux user.. but 90% of the coding work I do with java is in the COBOL sense of business middle-ware.

Usually the only thing that really bit-the-bot was someone hard coding paths into the code, rather than using a configuration file or something of the sort or referencing an external JNI call to a native library. This is VERY uncommon in the code that I have worked with.

Maybe with Swing/AWT applications, your right.. but on the server, which it doesn't sound like you have done much programming w/JAVA on, JAVA has lived up to it's promise. It's EXTREMELY portable. Over the last year I have switched my build managment to ANT, and if done correctly, will help even further in keeping things clean across multiple platforms.

As with ANY cross-platform tool .. (I run into this myself with my own Python code occasionally), will have problems w/cross platform if you don't think cross platform. Usually non-cross-platform JAVA code is either HIGHLY specialized or poorly written.

Nice karma whoring tho mate.

Re:JAVA Based projects (offtopic)

[bobv-pillars-net]

I don't want to start a flame war here, but...

No, for flame wars Usenet is a far better medium. (grin!) In all fairness, my gripes are mostly directed at Java in the user interface rather than Java on the server; your claims to its effectiveness are probably very well founded, as far as they go.

I appreciate your rant as a Windows/Linux user ... doesn't sound like you have done much programming w/JAVA ... will have problems w/cross platform if you don't think cross platform.

All of which is right on-track. No, I don't program in Java, though I'm forced to use it occasionally. From a programmer's perspective, Java may well be the cat's pajamas, but from a user's perspective (mine), Java sucks.

Nice karma whoring tho mate.

Harrumph! Posted too late in the discussion to win any mod points; if I were karma whoring I would have at least included some links. Moderators always like to see some links, even if they don't bother to follow them.

Re:JAVA Based projects (offtopic)

[Pengo]

All of which is right on-track. No, I don't program in Java, though I'm forced to use it occasionally. From a programmer's perspective, Java may well be the cat's pajamas, but from a user's perspective (mine), Java sucks.

Agreed. I haven't really found a java-based application that I love to use (from the user-interface standpoint) .. other than a few development environments. (JBuilder + IDEA)...

Really, I see JAVA as nothing more than a todays COBOL. It's GREAT for business based software, and really is quite easy to maintain and manage. But, middleware doesn't really have much to do with UserInterface. (There is probably hundreds of millions of lines of JAVA code sitting in the back-end of windows-based native applications, that you would never guess...)

Anyway, cheers!