Building a Stable and Clustered J2EE Environment?

royles asks: "I am working with several J2EE Application Servers and all seems fine until I start to scale and cluster. The goal is to ensure a web and EJB session is maintained across a pool of Application Servers. The level of complexity to achieve this straightforward requirement is proving too much. It quickly becomes evident that vendors claims of 'robust and scalable' go straight out the window in favour of extortionate support and consultant costs when the installation becomes large. Anybody else experienced this type of problem? What are other peoples experienceâ(TM)s in this area? Any recommendations on Application servers that âdo what they say on the tinâ(TM)? Any good online resources for achieving a highly scaled and robust solution?"

Comments

JBoss does this now

[GOD_ALMIGHTY]

First of all, do not store data on the application server. Secondly, make sure that all of your objects associated with a session in either the EJB layer or the HTTP session implement Serializable.

I'd recommend running JBoss + Tomcat on Linux with Apache frontending it with mod_jk2. Put all your static web content on Apache to free up Tomcat. The easiest way to do this is to use a load-balancing solution in front. Have it pass the sessions through to the Apache server which will offload any JSP/Servlet requests to Tomcat. Have the JBoss/Tomcat installation on each node of the cluster clustered using JBoss' JavaGroups based clustering. You can also use the JBoss Farm service to deploy the EAR file to one node in the cluster then have the other nodes pick it up from there.

If you use a seperate set of machines for the database cluster than the JBoss/Tomcat nodes can be set up as copies of each other with no data on board. Using Linux boxes, this is easy (dd the drives). If a node goes down for some reason, swap it with a spare. Any session info is already serialized and passed to the other nodes in the cluster, and the load-balancer w/ failover should handle the client end of that.

By the way this thing scales to at least 100 boxes from the stories I've heard. I've done it my self in small (5 box) installations. I've also used F5 equip for the load balancer/failover on the front. Everything worked well.

Buy the JBoss docs to help with the clustering. I've also got some docs I wrote on setting up JBoss clustering somewhere around here.

You'll be suprised how simple clustering is with JBoss once you understand a bit how JBoss works and is configured. You will also be amazed at how well it works.

The best part of course is that it's all Open Source, so no license fees. Hell, I'll set it up for you for 30% of the licenses and support you would have bought if you want. ;-)

Re:JBoss does this now

[oz_ko]

You really should read the Jboss clustering doco. Then read them another 2 times.

It took about 1 week for us to set up our cluster with hot deploy and session replication working properly, though I think the next time round it would be a lot quicker.

I don't think there are really any big issues in setting up a cluster though to set up a _good_ app server cluster you need to understand how the clustering implementation of your app server works and decide what is best for your solution.

Unfortunately there are no magical "click here to cluster" buttons.

Re:JBoss does this now

[danpat]

Actually, some appservers (WebSphere?) *do* have "click here to cluster" buttons.

The problem is that they don't work.

check out Resin

[ubiquitin]

Caucho's resin application clusters are a great way to get load balacing and distributed sessions into your appserver farm. Cost is around $1k per server, so real enterprise jocks will laugh at it, but unlike some of the "three engineers per server" solutions, it really works without having to spend all your time troubleshooting and hand-coding things.

Re:check out Resin

[Anonymous Coward]

first, let me say that I am a huge fan of Resin. I looked at the major servlet/JSP containers about a year ago and found Resin to be the top (mostly in terms of price/performance).

That said, Resin is *not* a "set and forget" solution, unless you are happy with the default configuration. It takes some trial and error for tuning, and the error messages it gives don't typically make the real problem obvious at all.

Their documentation is lackluster, but you can generally find the answer you need if you are willing to look hard enough.

Weblogic does this for us

[crstophr]

We are implementing a 3 tier high availability cluster using weblogic as the application layer. We have them clustered though a weblogic provided apache module on our load balanced web layer. It also survives the failover of the database layer as well. These things do still need consideration in the code you write (mainly the part about reconnecting after a database failover). Weblogic seems to handle the redundancy and load balancing fine if you let it maintain it's persistance by storing it's que and object data in the database layer. --Chris

one tip

[Anonymous Coward]

with clustering is to do setup pairs. sharing session cross three or more servers is tricky to configure and setup. It requires a lot of experience and care. It works best in pairs and you have to know what your doing. It's best to play with a stagging environment and do some serious load testing to see first hand how misconfiguration can create cascading problems. On one project, we had a BEA consultant come in to configure the clustering. It took him a week to do it and he had prior experience setting up Weblogic 5 in a clustered mode.

Also be care to only cluster stateful EJB. with stateless stuff, it's better to no cluster and simply load balance it.

"How To Scale" by Hardware Vendors

[villy]

"If you upgrade each of your current servers to the quad-processor GigaFLex, you'll have no problems."

"Do you have any perfomance metrics for that? How about a customer we could reference on that?"

"Ummm, no... those results are from our research lab."

(you get the idea)

Aren't these goals the rationale for EJB?

[MrBlack]

Isn't the purpose of EJB to meet these sort of goals without having to do anything special? Well, except architect your app. a certain way. If doing this is non-trivial then doesn't it mean that EJB has failed to a certain extent? If I have to hear about Data Transfer Object Factories and CMP entity beans again I think I'll puke....

Re:Aren't these goals the rationale for EJB?

[hargettp]

Yes, the EJB design pattern helps with clustering by abstracting out many bits of code that make clustering hard (e.g, direct database access); your code simply trusts the app server to manage those bits for you. However, EJB support in an application server of and by itself does not constitute clustering. That's an extra, layered feature that a good app server supports...and JBoss is definitely one of the good ones.

Clustering is hard, you can't avoid it

[mlq]

Clustering *is* hard. And even the best application server in the world can't reduce that complexity. It can just try to not add any complexity of it's own.

Front-end load balancing, internal load balancing policy, cache settings, fail-over settings, setting up the LAN correctly... these are all *extra* things you need to do under a clustered J2EE server that you don't need to worry about normally. Clustering is hard

Having said that, I've been involved in commercial projects using clustering under Weblogic and JBoss. We have been able to get them going without too much trouble.

Also, some things J2EE allows some things in non-clustered mode that it explicity warns won't work in clustered mode. For example, objects in Sessions must be Serializable when clustering.

=Matt

Re:Clustering is hard, you can't avoid it

[mlq]

Yep, you've hit upon the clincher.

How does the session know to replicate an object that's changing? For example, if during a request/response a mutator method is called on that object?

This is the problem. The session doesn't know, in this situation.

The only time a (key, object) pair in a session gets replicated is if Session.set...() has been called for that pair during this request/response cycle.

If you call a mutator on an object (where that object has been pulled out of the session), then that mutated copy of the object WILL NOT be replicated unless you call Session.set...() again!

This is something that can really kill you when you try to deploy an app that works well in a non-clustered environment into a cluster.

Check out this article by Bill Burke

[jawahar]

Chief architect of JbossGroup

http://www.onjava.com/lpt/a/1517

Cisco Load Balancing?

[pguerra1]

You can use a Cisco localdirector(s) at the front of the farm of your application servers, and just enable sticky cookies. This will cause the localdirector to keep the requests on the same application server, and reduces the complexity of having to implement software load balancing solutions in your code and on your servers.

Are EJB's really worth it?

[Anonymous Coward]

Excuse my ignorance, I'm just beginning to learn EJB. But as I learn more about them, I can't help thinking: "what's the deal?".

I mean, what's wrong with having your MVC webapp being based on JSP/Servlets, plus plain-old Java objects for the business-logic (accessing the DB via JDBC). Tomcat itself has support for load-balancing among several Tomcat servers; and as for the business-logic objects, these would be acting like client apps in a traditional C/S fashion, so they can be distributed without problems (and the transactional stuff should be taken care of by the DB, that's what RDBMS are for, no?)

So again, what's the big deal with EJB? Training wheels for programmers who don't grok databases and transactions; who think tha JDBC is too complicated??? Seriously, I'd like to be enlightened on this.

Re:Are EJB's really worth it?

[elemur]

EJBs are going to be in a guarenteed transactional state throughout the cluster, which simplifies all of your access. The containers should be caching some types of beans as well (CMP), so that you end up with a very fast transactionally complete solution across the environment. You can use JDO or other data solutions, add your own cache, manage it across the cluster.. and that may work well for you. Other beans are useful, such as the Message Driven Beans, for building a J2EE standard event driven component. Straight data mapping solutions don't provide this.

That said, personally I find many people don't need to worry as much about clustering, so as long as you are working on a good environment, thats probably more important. I use hibernate more these days with a good solid MVC design. I have servlets set to start at boot that become by queue listeners, and it works quite well. I can use XDoclet to generate my hibernate mappings if I want, and get up and running quickly.

The good thing is that you have alot of options. The bad thing is that you have a lot of options.

It comes down to picking the right technology for the job.. and your team. JDBC, JDO, OR-Mapping, EJB, etc all provide good tools to complete the job.

Re:Are EJB's really worth it?

[Anonymous Coward]

It comes down to picking the right technology for the job.. and your team. JDBC, JDO, OR-Mapping, EJB, etc all provide good tools to complete the job.

Agreed, but in my opinion any application which needs heavy multi-user data processing (think OLTP) is better served with the more "traditional" way of doing things: let the RDBMS do the transactions and cacheing, because they do it best. Call me a data modeler or SQL buff you like, but I still fail to see the advantages in letting the "application server" do these kind of things.

About the only thing I see useful in EJB (vs. POJO+JDBC) is that they give you the networking infrastructure to make them remotely callable, so that you don't have to mess with RMI or the like.

A good article I read recently by Martin Fowler, "Domain Logic and SQL" covers some of these points, and reinforces my opinion that neither EJB nor any other O/R mapping tools are suited for heavy OLTP apps. Or am I missing something?

incredibly easy

[MORTAR_COMBAT!]

You create a 'server group' and add clones to it with mouse clicks. Viola, instant-on, growable/shrinkable massively scalable J2EE container.

Maybe this guy is trying to use the "Free except the documentation" JBoss and getting lost in the API set?