Building a Stable and Clustered J2EE Environment?

royles asks: "I am working with several J2EE Application Servers and all seems fine until I start to scale and cluster. The goal is to ensure a web and EJB session is maintained across a pool of Application Servers. The level of complexity to achieve this straightforward requirement is proving too much. It quickly becomes evident that vendors claims of 'robust and scalable' go straight out the window in favour of extortionate support and consultant costs when the installation becomes large. Anybody else experienced this type of problem? What are other peoples experienceâ(TM)s in this area? Any recommendations on Application servers that âdo what they say on the tinâ(TM)? Any good online resources for achieving a highly scaled and robust solution?"

Comments

WebObjects

[Anonymous Coward]

WebObjects is inexpensive, stable and scalable plus it runs on pretty much any J2EE server environment:

Develop apps quickly

Provides a powerful integrated development environment

Allows you to create web services or three-tier Java server apps

Generates code-free prototypes in minutes

Build services dynamically

Allows you to create or use web services based on SOAP, XML and WSDL

Includes tools for code-free generation, configuration, and testing of web services from existing database assets

Integrates easily with services written in Java, AppleScript, Perl and .Net

Access data automatically

Reverse-engineers your database, modeling all tables, columns and relationships

Generates Java objects from JDBC and JNDI data sources, transparently handling data persistence

Offers database independence and simultaneous access to multiple databases

Generate pages seamlessly

Generates HTML, XML and SMIL from reusable templates

Builds data-driven web pages from visual development tools

Streamlines delivery of multi-lingual content

Deploy painlessly

Deploys to virtually any J2EE server or the WebObjects J2SE application server

Provides easy scalability with built-in clustering support and low-cost licensing

Supports Servlet/JSP, EJB, RMI-IIOP and JTA/JTS (J2EE technologies)

JBoss does this now

[GOD_ALMIGHTY]

First of all, do not store data on the application server. Secondly, make sure that all of your objects associated with a session in either the EJB layer or the HTTP session implement Serializable.

I'd recommend running JBoss + Tomcat on Linux with Apache frontending it with mod_jk2. Put all your static web content on Apache to free up Tomcat. The easiest way to do this is to use a load-balancing solution in front. Have it pass the sessions through to the Apache server which will offload any JSP/Servlet requests to Tomcat. Have the JBoss/Tomcat installation on each node of the cluster clustered using JBoss' JavaGroups based clustering. You can also use the JBoss Farm service to deploy the EAR file to one node in the cluster then have the other nodes pick it up from there.

If you use a seperate set of machines for the database cluster than the JBoss/Tomcat nodes can be set up as copies of each other with no data on board. Using Linux boxes, this is easy (dd the drives). If a node goes down for some reason, swap it with a spare. Any session info is already serialized and passed to the other nodes in the cluster, and the load-balancer w/ failover should handle the client end of that.

By the way this thing scales to at least 100 boxes from the stories I've heard. I've done it my self in small (5 box) installations. I've also used F5 equip for the load balancer/failover on the front. Everything worked well.

Buy the JBoss docs to help with the clustering. I've also got some docs I wrote on setting up JBoss clustering somewhere around here.

You'll be suprised how simple clustering is with JBoss once you understand a bit how JBoss works and is configured. You will also be amazed at how well it works.

The best part of course is that it's all Open Source, so no license fees. Hell, I'll set it up for you for 30% of the licenses and support you would have bought if you want. ;-)

Re:JBoss does this now

[oz_ko]

You really should read the Jboss clustering doco. Then read them another 2 times.

It took about 1 week for us to set up our cluster with hot deploy and session replication working properly, though I think the next time round it would be a lot quicker.

I don't think there are really any big issues in setting up a cluster though to set up a _good_ app server cluster you need to understand how the clustering implementation of your app server works and decide what is best for your solution.

Unfortunately there are no magical "click here to cluster" buttons.

Re:JBoss does this now

[danpat]

Actually, some appservers (WebSphere?) *do* have "click here to cluster" buttons.

The problem is that they don't work.

Re:JBoss does this now

[Anonymous Coward]

"insightful"?

Gimme a break.

The blind praising the blind. Yet, reading your "profile", you seem to have made intelligent comments in the past. Having a bad day, are we???

I work with WebSphere every day. How an enterprise product that's a company's *flagship* product "just doesn't work" --- and yet accounts for millions of dollars of sales --- is rather hard to fathom. It's a smashing success, in my experience - and my explorations of Bluestone and Sun One show that there's substance there, too.

Maybe you should read some books. Then talk to some folks who have the technical expertise to help you with your obvious appserver-related issues.

<soapbox>Enterprise software isn't a text editor. If you expect the infrastructure to just magically appear when you push a button .... well, wait until next year or the next when on-demand computing is mature enough to do it for you. Today, yes, you still have to do some of the work. </soapbox>

Weblogic, Oracle, Sun, Netscape (nee Bluestone), and many more, they all "work". If you are a technologist, I'd think you'd know how foolish it is to state that " just doesn't work"

Sheesh.

check out Resin

[ubiquitin]

Caucho's resin application clusters are a great way to get load balacing and distributed sessions into your appserver farm. Cost is around $1k per server, so real enterprise jocks will laugh at it, but unlike some of the "three engineers per server" solutions, it really works without having to spend all your time troubleshooting and hand-coding things.

Re:check out Resin

[Anonymous Coward]

first, let me say that I am a huge fan of Resin. I looked at the major servlet/JSP containers about a year ago and found Resin to be the top (mostly in terms of price/performance).

That said, Resin is *not* a "set and forget" solution, unless you are happy with the default configuration. It takes some trial and error for tuning, and the error messages it gives don't typically make the real problem obvious at all.

Their documentation is lackluster, but you can generally find the answer you need if you are willing to look hard enough.

Weblogic does this for us

[crstophr]

We are implementing a 3 tier high availability cluster using weblogic as the application layer. We have them clustered though a weblogic provided apache module on our load balanced web layer. It also survives the failover of the database layer as well. These things do still need consideration in the code you write (mainly the part about reconnecting after a database failover). Weblogic seems to handle the redundancy and load balancing fine if you let it maintain it's persistance by storing it's que and object data in the database layer. --Chris

one tip

[Anonymous Coward]

with clustering is to do setup pairs. sharing session cross three or more servers is tricky to configure and setup. It requires a lot of experience and care. It works best in pairs and you have to know what your doing. It's best to play with a stagging environment and do some serious load testing to see first hand how misconfiguration can create cascading problems. On one project, we had a BEA consultant come in to configure the clustering. It took him a week to do it and he had prior experience setting up Weblogic 5 in a clustered mode.

Also be care to only cluster stateful EJB. with stateless stuff, it's better to no cluster and simply load balance it.

Re:one tip

[Bill Privatus]

Sorry, this is horribly off the mark in the general case. If you're talking about some particular software, you should be more precise. Someone following your "advice" (rated informative, no less, by an Esteemed Moderator) might be led to believe many doors were closed, when in fact the field is wide open. Use many little servers, or get a few "big iron" boxes, but, geez, pairing will demand a much larger investment in infrastructure and ongoing administrative effort than having, say, 6, or even 12 instances!!

In no appserver technology I've ever worked with have I come across limitations such as you suggest. This problem you describe is, I suspect, either infrastructure design flaw related, or application design/implementation related.

For example, I've worked with 24 "instances" across 4 enterprise servers for a while now, and there are no problems with performance, availability, or reliability...this is vertical and horizontal "clustering", while you imply only horizontal.

Re:one tip

[alext]

Hmmmm.

WebLogic only replicates state between pairs, never between three or more.

It's not necessary to understand the details of this as it happens automatically - essentially WL picks a secondary server that it knows is on a different physical machine.

If your BEA consultant has only ever set up a cluster once, and that for a version of the product dating from four years ago (I assume you mean WL 5.1) it sounds like he was an impostor.

Apart from these little criticisms, I have some sympathy with your "load balance only" approach. One thing you can do when not using clustering is upgrade servers in rotation, which is admittedly pretty handy.

"How To Scale" by Hardware Vendors

[villy]

"If you upgrade each of your current servers to the quad-processor GigaFLex, you'll have no problems."

"Do you have any perfomance metrics for that? How about a customer we could reference on that?"

"Ummm, no... those results are from our research lab."

(you get the idea)

Aren't these goals the rationale for EJB?

[MrBlack]

Isn't the purpose of EJB to meet these sort of goals without having to do anything special? Well, except architect your app. a certain way. If doing this is non-trivial then doesn't it mean that EJB has failed to a certain extent? If I have to hear about Data Transfer Object Factories and CMP entity beans again I think I'll puke....

Re:Aren't these goals the rationale for EJB?

[hargettp]

Yes, the EJB design pattern helps with clustering by abstracting out many bits of code that make clustering hard (e.g, direct database access); your code simply trusts the app server to manage those bits for you. However, EJB support in an application server of and by itself does not constitute clustering. That's an extra, layered feature that a good app server supports...and JBoss is definitely one of the good ones.

WebSphere?

[Anonymous Coward]

Anyone want to comment on how easy it is to do this with IBM's WebSphere Application Server?

I have heard great thing about these people

[Plague_nz]

www.opencloud.co.nz

They create a SDK for creating appilcations that run over multiple systems and is extremly fault tolerant.

Clustering is hard, you can't avoid it

[mlq]

Clustering *is* hard. And even the best application server in the world can't reduce that complexity. It can just try to not add any complexity of it's own.

Front-end load balancing, internal load balancing policy, cache settings, fail-over settings, setting up the LAN correctly... these are all *extra* things you need to do under a clustered J2EE server that you don't need to worry about normally. Clustering is hard

Having said that, I've been involved in commercial projects using clustering under Weblogic and JBoss. We have been able to get them going without too much trouble.

Also, some things J2EE allows some things in non-clustered mode that it explicity warns won't work in clustered mode. For example, objects in Sessions must be Serializable when clustering.

=Matt

Re:Clustering is hard, you can't avoid it

[JediTrainer]

For example, objects in Sessions must be Serializable when clustering.

I have a question about that I've been wondering about for a while. How does the session know to replicate an object that's changing? For example, if during a request/response a mutator method is called on that object?

Say, for example, I have a Vector stored in the session. And each time a request is processed, something is added or removed from that Vector. If you never actually call set...() on the Session (just get the vector, then set something within it), does this actually get replicated correctly?

I'm a bit curious as to how that would work internally.

Re:Clustering is hard, you can't avoid it

[mlq]

Yep, you've hit upon the clincher.

How does the session know to replicate an object that's changing? For example, if during a request/response a mutator method is called on that object?

This is the problem. The session doesn't know, in this situation.

The only time a (key, object) pair in a session gets replicated is if Session.set...() has been called for that pair during this request/response cycle.

If you call a mutator on an object (where that object has been pulled out of the session), then that mutated copy of the object WILL NOT be replicated unless you call Session.set...() again!

This is something that can really kill you when you try to deploy an app that works well in a non-clustered environment into a cluster.

Re:Clustering is hard, you can't avoid it

[JediTrainer]

The only time a (key, object) pair in a session gets replicated is if Session.set...() has been called for that pair during this request/response cycle.

Hmm...

This seems impractical. You either lose your abstraction by having to make the session available to any part of the app which might change your object's state, or you lose efficiency by just having to assume that it's changed and force a 'set' every time it *might* have changed.

Perhaps the Java Servlet API needs to include another interface. Something along the lines of 'StateChangeNotifier' (which of course extends Serializable).

The Session.set() will accept only a StateChangeNotifier object (not just any Object). Basically, this interface defines a way for the session to register itself with that object, so that the session can know when that object's state has changed using some sort of event mechanism.

Obviously the person who implements that object would need to make sure that the event is thrown whenever something changes. But then again... what if that object contains other objects whose state might change? Arrrrgggh.

I guess you just have to force the replication if there's any chance it might have changed. Crap. This is a serious problem and an invitation for bugs with less-experienced developers, IMO.

Check out this article by Bill Burke

[jawahar]

Chief architect of JbossGroup

http://www.onjava.com/lpt/a/1517

Cisco Load Balancing?

[pguerra1]

You can use a Cisco localdirector(s) at the front of the farm of your application servers, and just enable sticky cookies. This will cause the localdirector to keep the requests on the same application server, and reduces the complexity of having to implement software load balancing solutions in your code and on your servers.

Re:Cisco Load Balancing?

[Bill Privatus]

This is a small performance tuning step only. You cannot presume that you have increased the availability or reliability of your site by using sticky sessions or session affinity or any other means of sending same-transaction/same-session traffic to the same set of servers.

If you ignore the remaining layers of an N-tier architecture, then when a server/node goes down, your user session/transaction is...gone.

This is usually unacceptable to users. And thus to site owners.

I'm not saying implement "in your code" the measures needed to provide performance, availability, and reliability services --- but your servers had better damn well provide and/or support them.

Unless, of course, you're running a "Britney Rocks!" or elementary school web site, that is...

Orion Application Server

[Anonymous Coward]

I heard good things about Orion's clustering capabilities (never tried it though).

They have some good documentation on their site.

Cheers

Are EJB's really worth it?

[Anonymous Coward]

Excuse my ignorance, I'm just beginning to learn EJB. But as I learn more about them, I can't help thinking: "what's the deal?".

I mean, what's wrong with having your MVC webapp being based on JSP/Servlets, plus plain-old Java objects for the business-logic (accessing the DB via JDBC). Tomcat itself has support for load-balancing among several Tomcat servers; and as for the business-logic objects, these would be acting like client apps in a traditional C/S fashion, so they can be distributed without problems (and the transactional stuff should be taken care of by the DB, that's what RDBMS are for, no?)

So again, what's the big deal with EJB? Training wheels for programmers who don't grok databases and transactions; who think tha JDBC is too complicated??? Seriously, I'd like to be enlightened on this.

Re:Are EJB's really worth it?

[elemur]

EJBs are going to be in a guarenteed transactional state throughout the cluster, which simplifies all of your access. The containers should be caching some types of beans as well (CMP), so that you end up with a very fast transactionally complete solution across the environment. You can use JDO or other data solutions, add your own cache, manage it across the cluster.. and that may work well for you. Other beans are useful, such as the Message Driven Beans, for building a J2EE standard event driven component. Straight data mapping solutions don't provide this.

That said, personally I find many people don't need to worry as much about clustering, so as long as you are working on a good environment, thats probably more important. I use hibernate more these days with a good solid MVC design. I have servlets set to start at boot that become by queue listeners, and it works quite well. I can use XDoclet to generate my hibernate mappings if I want, and get up and running quickly.

The good thing is that you have alot of options. The bad thing is that you have a lot of options.

It comes down to picking the right technology for the job.. and your team. JDBC, JDO, OR-Mapping, EJB, etc all provide good tools to complete the job.

Re:Are EJB's really worth it?

[Anonymous Coward]

It comes down to picking the right technology for the job.. and your team. JDBC, JDO, OR-Mapping, EJB, etc all provide good tools to complete the job.

Agreed, but in my opinion any application which needs heavy multi-user data processing (think OLTP) is better served with the more "traditional" way of doing things: let the RDBMS do the transactions and cacheing, because they do it best. Call me a data modeler or SQL buff you like, but I still fail to see the advantages in letting the "application server" do these kind of things.

About the only thing I see useful in EJB (vs. POJO+JDBC) is that they give you the networking infrastructure to make them remotely callable, so that you don't have to mess with RMI or the like.

A good article I read recently by Martin Fowler, "Domain Logic and SQL" covers some of these points, and reinforces my opinion that neither EJB nor any other O/R mapping tools are suited for heavy OLTP apps. Or am I missing something?

Re:Are EJB's really worth it?

[Bill Privatus]

Hey, what'd you post anon for? I would have missed this if I hadn't changed my filter settings for this thread. Sigh...

Poster is dead on. At the risk of regurgitating that which has been opined upon so much in the past, EJB is not a high-performance solution. EJB is not a "load-balancing" solution, intrinsically. It is a bringing together of data/function --- and a way of making that data/function available "anywhere" (i.e., from multiple disparate data sources or processors, to multiple disparate data sinks or processors).

A good, fast, universally-available (in a communications sense) RDBMS will kick the legs out from under EJB in a performance test nearly every time.

And, boy, does it provide bolshoyeh failover! Persistence without the horrible baggage that is "serialization". Don't get me started on that one :-/

Mod this parent anon post up, someone, so everyone with "no anon posts" filter settings can see it :-)

Re:Are EJB's really worth it?

[Anonymous Coward]

EJBs are going to be in a guarenteed transactional state throughout the cluster, which simplifies all of your access. The containers should be caching some types of beans as well (CMP), so that you end up with a very fast transactionally complete solution across the environment.

Yeah, but... what's wrong with a traditional RDBMS on a fail-over cluster (or if you can't tolerate even a few minutes of downtime, Oracle RAC)? Then just distribute your Tomcat app among several machines and use some kind of loadbalancing front-end. Granted, your servlet code has to deal with transactions in SQL (which I guess means using JDBC), but the transaction management and cacheing is most robust and performant when done by the RDBMS.

OK, maybe using EJB brings benefits under some circunstances, that's what I'm trying to figure out. To be specific, I would like to know:

• Could I use the EJB server's built-in transaction management guts to replace Oracle with a "dumb" storage system like MySQL?
• Does EJB really simplify programmer's life so much, when it comes to data access?
• Cacheing data at the EJB layer is appealing because it's distributed and potentially performant, BUT: can it be trusted in today's implementations?

I think these questions affect the bottom line, since if these were true, then that would mean (respectively) that EJB's would allow me to purchase a much cheaper database, build better code, and buy a database server with much less RAM.

Re:Are EJB's really worth it?

[elemur]

Well.. to me, there are certain operations that should be performed in the database, and some that don't *need* to be. If you are going to be processing alot of the rows in a table or multiple tables.. getting that logic as close to (inside) the database engine is the best bet. Keeping as much data off the wire is key with that.

When you get into other systems.. say a site like slashdot or similar, the forward caching that the EJB container gives you in a CMP world can bring a nice performance benefit to your system. If the EJB container is reasonably sure, based on your preferences, that the data is not stale, its not going back to the DB for anything.. its just pulling from local cache. There is your big speed advantage. If you have to hit the DB on every EJB request, rebuild the object, handle the locking, etc.. it will be dog slow.

I think the real problem is EJB is that its overly complex. Its the case of, do people need all of the complexity for 90% of what they do? Probably not. Things like XDoclet simplify that, but still not a trivial exercise. Then when you run into problems, debugging the EJB app is often difficult as well.

I personally prefer not to do the straight JDBC unless necessary, but will generally design with a DAO sort of model to abstract it anyway. Lately I've been using Hibernate2 with lots of luck. I just finished one site using 100% hibernate, and it was wonderful not mucking about with SQL. Especially singe Hibernate will build the right SQL for your target database.. makes portability a bit easier. Hibernate uses the JCS cache so you get some forward caching features without any of the headache of EJB. However, my site is not transactional in nature.. its 95% information viewing, so the traditional advantages of EJB don't apply to this project as much.

As an interesting note.. JBoss, in JBoss 4 seems to be going down more of a JDO supporting route. Thats a big shift for them.. well, and the aspect oriented programming they are doing.. which is telling.

Re:Are EJB's really worth it?

[Anonymous Coward]

Could I use the EJB server's built-in transaction management guts to replace Oracle with a "dumb" storage system like MySQL?

Perhaps not EJB specifically, but other J2EE App Server level technologies for dealing with similar issues do this. EJB may well but I don't know much about it. I have implemented apps with middle tier transaction mgmt etc., that let me replace oracle with mysql, which makes is one of the rare cases where programmer time might be cheaper than buying the damn software and hardware. You can just turn mysql into a row-saving program, it just provides persistence. All that other stuff is replicated in app level tools that are cheaper to buy and run than 9i.

incredibly easy

[MORTAR_COMBAT!]

You create a 'server group' and add clones to it with mouse clicks. Viola, instant-on, growable/shrinkable massively scalable J2EE container.

Maybe this guy is trying to use the "Free except the documentation" JBoss and getting lost in the API set?

Clustering on a Single Server

[rimu guy]

I'd second JBoss for clustering. The middle tier, EJBs and the like 'just works'.

The front end is tricker: e.g. setting up DNS round robin + virutal IP takeover in case a front end server goes down.

And the back end (database) is also fun. e.g. setting up a distributed file system. Or setting up database replication. Both doable, but hard to do right and get great performance.

I had one customer use our VPS servers to help them get clustering going.

They can set up several virtual servers (with their own file systems and IP addresses, etc) all on one physical server.

Once they have nailed they setup, they can them deploy on separate physical servers. Its a lot quicker, easier to manage (and cheaper) doing it that way.

Cheers, Peter RimuHosting - VPS/JBoss Hosting

Clustering todos --

[chicagozer]

If you plan to cluster, be sure to evaluate your use of session. If you intend to replicate your session objects across each instance of the cluster, they will need to be serializable so that the container can move them around. From past experience, you will need to consider the size of these objects as well -- if each session footprint is large, you may have some scalability limits. As an alternative, you can choose to not cluster the session objects and leave them on the original container. To do this, your front end router must ensure that the sessions return to the original container via a cookie or some other means. Also, consider eliminating any file-based storage on the cluster. Use JDBC for queues and logs. In this way, if you lose an instance of the cluster, your data will still be safe in your database (assuming you have a HA solution for your database)

Thanks

[royles]

Thanks for all the responses and detail. JBoss comes up a lot... I will investigate this further. Good points about sessions and DB usage.

I thought at this stage in J2EE development, clustering would be well understood and available 'out of the box'.

Thanks, Royles

Re:incredibly easy - Oh yeh???

[royles]

Actually 'this guy' is using the 'mouse click' features of Borland BES as well as BEA Weblogic.

Both of which 'cost and arm and a leg' for the documentation then rip you off totally when it comes to support.

Also the documentation on clustering for BES is very lightweight. YOu are looking at committing thousands to put together even the most basic clustered solution.

Will investigate your recommendations further though.

Re:incredibly easy - Oh yeh???

[MORTAR_COMBAT!]

If you really are interested in investigating WebSphere... let me know and I can probably get you in touch with some evaluation licensed software. (Note that however I do not work for IBM Sales!)

Try Tangosol Coherence

[ggleyzer]

If you are working with multiple servers, you should look at Coherence.

We do a lot more than clustering app servers. Our product serves as an in-memory data store in a clustered application tier. If you have experience with high-scale J2EE applications, you'll probably have experienced what I call the "single point of bottleneck": the database. For some of our customers, we've reduced their database usage by 99.9%+ and their overall infrastructure costs by over half. With the replicated cache, each application server has all the "cached" data locally, so access is instantaneous. With our concurrency features, the data integrity is maintained, whether on a single server or replicated across 100 servers. With our distributed cache, if you have 100 servers each caching a half GB of data, you have a 50GB cache!

Other features include support for grid computing, cache querys, and more. We've focused heavily on making sure that our clustering is fault tolerant and extremely reliable.

Check out how the TheServerSide created a single cluster out of multiple vendor's app servers using Coherence. You may not need it for your application, but it shows what you can accompish.

Gene Gleyzer
Tangosol, Inc

Re:JBoss does this now - Best Java Distro??

[Anonymous Coward]

I'm looking at a JBoss install - which Linux Distro(s) work best with Java - Slackware? Suse? Redhat?

Which have you used?

Your using Apache2 right? does 1.3.* also work?

thx

do you really need all that -- n tier architecture

[madvenu]

Why do you need such 'Application Clusters' anyway ?

http://philip.greenspun.com/wtr/application-serv er s.html

cherio
venu