Should A High-Profile Media Website Abandon Java?

newbroom asks: "The company I work for runs a large, high-profile web site with users all across the world and delivers them large amounts of streaming media content plus textual stories. You might guess therefore that this is a news website, frequently updated throughout the day, and delivering content 24x365. No names, or course, for obvious reasons. We have a big, custom, Java content management system (based on a framework from a proprietary vendor as it happens, but could just as well be EJB/J2EE for all that it matters in the context of this argument) and for deployment we run our website using Java app servers on Solaris behind Apache." If you were going to take such a site from 1000 users, to 10,000 users, would you be able to do it using this kind of setup?

"It is all hugely expensive to license and to run, and it's not very scalable. We'd like to up our userbase from several tens of thousands to ten times that number - but the cost of scaling the Java/Solaris infrastructure is not trivial, because the Java servlet architecture costs too much in memory and execution time (creating several 100Ks of in memory objects for each logon is expensive stuff!). On current hardware we can support only 1200-1500 concurrent logins and scaling up requires a new app server (eg 1 processor + 1GB RAM) and a $20K software license for each additional 600-750 concurrent logged in users. And in today's 'cost per active subscriber' economics it doesn't add up - we cannot justify the present cost structure, by any rational measure, even before we try to scale it up.

So we're thinking of chucking it out and replacing it with a largely static site that is generated (written out to cache) from a new, simpler content management system. The few dynamic elements would be assembled using simple PHP scripts, frontending our existing Oracle DB server. We reckon we could serve vastly higher numbers, ten to a hundred times as many, of users on the same (or cheaper!) hardware: and it would be simpler by far to build and maintain and support.

I, personally, believe that the benefits of the Java system (rapid prototyping, development) are not important when large scale deployment is the issue. I am (as a user) fed up with large, poorly performing Java-based websites. My beef is not about Java the language though - it's a question of appropriateness. Fifteen years ago we'd prototype in Smalltalk and then code for deployment in C, and I feel the same applies here. The economics of the noughties do NOT support spending massive amounts of money on web infrastructure, unless the transactional revenue justifies it. Of course, most businesses generally don't justify it, in my opinion.

Our outsourcing partner who supports and maintains the architecture thinks we are crazy. Putting their potential loss of revenue aside they are hugely concerned that we'll not be able to support what we create. They are seriously against this idea.

I remember, prior to Java & the like, supporting simple CGI websites with tens & hundreds of thousands of users off of cheap FreeBSD systems, and we didn't have to pay an outsourced partner to do it.

So what does Slashdot think? What would you do if you, were in the same boat?"

Comments

Wrong place man!

[Lally Singh]

Sorry dude, but you're going to get many of these:

1. Solutions pulled from the ass
   
2. Solutions that sound real good, but were pulled from the ass
   
3. Crap
   

For your case, I hope I'm wrong. Just out of curiousity, have you considered, I donno, profiling your application, to see where the time's being spent. Also, how about switching to a more cost-effecitve Java platform? I've heard good things about this little thing called Linux.

Note:My few comments certainly fall into Category 1 from above.

But, your current solution does work, right? How much exploration have you done in optimizing the application? Oracle can certainly scale, and you're already willing to strip down the site to mostly static pages. Why throw out all that proven and thoroughly tested code? And if you're outsourcer can't do that, maybe its time to switch partners, not platforms. You have a large investment here, there's a good chance you can save most of it.

Should you throw out the baby with the bathwater?

[Phouk]

No, you should not - throwing out a significant body of tested, working code in favour of "new, better!" code is not a good idea generally. A well-known article by Joel Spolsky eloquently explains some of the reasons.

Instead, try to improve your current solution an bring its cost down:

1. If you are scaling up by adding servers, see if you cannot add something like two Linux servers instead of one Sun (for a fraction of the price, naturally).
2. You haven't said which software packages bring up the license cost (besides Oracle, of course). But for most, there are open source alternatives. Sure, they might take more work to set up in some cases, but certainly less than rewriting the whole application, no?
3. You might even want to evaluate if you can replace your Oracle by a SAP DB instance if that is not your bottleneck (Hint: Caching! Caching!).
4. If, as you say, hundreds of KB are used up for every logged in user, then, in all likelihood, there are big inefficiencies in your code. You should profile it / have it profiled for both CPU and memory efficiency. Then tune the 5% of the code that use 95% of the resources, instead of throwing away 100%.
5. Are the outsourced programmers up to snuff? Maybe have the code checked by a third party (who could also do the profiling / tuning). Because a bad programmer can bring down any infrastructure, be it J2EE, .NET, PHP or whatever. It's the man, not the machine.

Good Luck!

Don't do it

[farnsworth]

You say that your 2 biggest problems with the current setup are cost and performance. A rewrite in a new language will not necasarily solve either problem. If you want to tackle cost, look at free/cheap app servers. If you want to tackle performance, look at the code. Why are you using hundreds of kilobytes of ram for each user? That's a huge red flag and an indication that perhaps your application is either not optimal, or legitimately doing some major leg work for each user. Either way, a rewrite won't fix those problems unless you understand what they are.

Think of the risks that a rewrite introduces:

you break existing business logic with the new implementation

you build a system that is slower than the existing one

you take way too long to finish it, all the while you have to pay your existing licences

Typically, the argument for a rewrite is that the cost of implementing new functional requirements is higher than the cost of just implementing them in a brand new system. Have you tried optimization? How are you maintaining session state? Do you know what it would take to get your app running in a free container? Have you looked at free/cheap caching APIs?

Further, java may not be the out-of-the-box fastest platform, but there is no reason whatever that it can't scale in an environment designed for it. Yes, you may need to have many smaller machines because of jvm memory issues, but that's exactly what you should want. The ideal situation is when you can say, 'if we need to support 10x the current users, we just need to drop in 10x more app servers.' It's called 'scaling linearly.'

Don't forget Developer skill sets

[Zachary Kessin]

It sounds like you have a shop full of good Java people. While you may want to change how things are run, I would not change languages. This is not based on any love for Java, but the fact that if you have a team of Java programers to get them to the point where they can write top flight code in something else will take time that you can better spend on something else.

But I would consider changing the architecture if that makes sense.

Let me get this straight...

[pi_rules]

You have a J2EE (or something like it) based application that is non-portable in both it's host OS and host application server and on top of that doesn't scale too well because of memory/CPU requirements?

Hmmm... somebody should be loosing their job. Either the consultants who built it or the person that approved such a thing.

If you had a true J2EE app that wasn't coded by a team of monkeys on a wild rampage this shouldn't a problem. The "porting" to a new app server should be trivial, if anything has to be done at all. You'd be able to keep the Sun hardware and whatever app server you use on it, while chucking in Linux machines with Bea, WebSphere or maybe JBoss on them. Slap a hardware based load balancer in front of it and viola, horizontal scalability.

I didn't see anybody else take offense to this, but 100k+ per user login memory usage? I might be showing my age, or rather my roots, but that seems excessive. My guess is your app's written like the app I now support. User logs in and everything about them is swallowed up into session (or application) wide collections immediately. The "lazy caching" thing just didn't cross these guy's minds. Of course, in my case neither did the "mark data dirty" thing but that's another matter.

Please, somebody show me 100k worth of data that you would really want on-demand from-memory on a user at any given instant. Just a C struct or something would suffice.

J2EE apps can be bleeding fast ultra lean sons of bitches if you do it up right. It can also be a dog-ass slow memory-hogging bastard. It just depends on who you had at the business end of the whiteboard I guess.

Going the PHP/static generation/caching route isn't neccessarily a bad idea either... but I don't think you should have to do this. I'm seeing the maintence of such a system as a big onus on the system administrators to make sure everything is up all the time... I know of no PHP frameworks out there that would let you drop sessions from one system to another. I've never tried pushing PHP that far though.

If your a system admin such a system might seem ideal... because while the systems and network might be a little "wonky" that's your domain and you feel comfortable supporting such a thing. I can't fault you for that; however I do think the onus is on the application development team. It is their job to make something scalable and construct it in a manner that it should fail over, recover, etc. from anything weird that may go on.

This isn't realistic, but you probably purchased a scalable application toted as portable because it's Java and you didn't get that. Demand that. If they can't deliver boot them out the door and take it inhouse if you must but I see many obstacles in your path on the system admin side alone... and certainly the re-development of it won't be cake walk.

Scalability problems are largely the development team's responsbilities, so long as such a requirement was put forth in the original development. Good system administration can help to reduce their errors along with a good helping of hardware but that's just a bandaid to the real solution.

Just my two cents.

Broadvision?

[jag164]

It almost sounds like you are stuck in a Boadvision nightmare....nortiously a lousy system with even worse scalibility issues. It'll scale alright....at an exponential rate that any normal J2EE solution will scale. Either start designing and coding a re-write with better CMS third party tools while keeping a few people on the BV maintainence side or be prepared to drop a few $100k into hardware per every 2000 concurrent users. *cringes* Good luck.

Also, anyone who got suckered into a Broadvision sales pitch for a enterprise solution should be shot^H^H^H^H fired on first site.

Re:look before you leap

[tigersha]

> Don't overlook maintainability as an asset.

Bingo. Right on the Money. You are my hero.

Maintainability is not an asset to consider, it is by far the most important asset (that and a good user interface.)

I sometimes wish all these Python/PHP/Ruby/whatever dudes would learn that strong typing is an ASSET not a problem. Simply because the compiler checks a lot of structural integrity BEFORE the damn thing runs highlighting load of errors right there.

I know the argument that a type system is only in the way and is not really needed when the program is debugged in any case.

But a type system makes a hell of a difference when you (or your poor successor) needs to change anything later because many (if not most) of the conflicts caused by a change are IMMEDIATELY nailed down by the typechecker. This thing is, typeing bugs are bugs. If you send a number to a thing that expects a database connection Python will moan just as much as JAva. the difference is JAva will moan before you run. PYthon and PHP will not.

Don't get me wrong, I write loads of tings in Ruby and Python. Most of them are small things that do a specific task, adminstrative scripts that sort of things. But for large complex systems, don't get me on a non-typed language.

That said, I wish Java would move into a type system that is much more based on type inference, such as HAskell where the language IS stronlgy typed but you pretty much don't have to worry about it since it will infer almost everything. The parts you have to specify are not too much and you get teh best of both worlds. Strong typing as well as non-cluttered code.

Not letting folk-wisdom trick you......

[BenTels0]

into seeing the problem in the wrong place. "[...] and for deployment we run our website using Java app servers on Solaris behind Apache."

This isn't the problem.

(creating several 100Ks of in memory objects for each logon

This is.

So what does Slashdot think? What would you do if you, were in the same boat?"

Don't know what /. thinks, but I think you need a serius rethink of your application's design. It sounds to me like you're throwing away exactly the advantage that servlets can give you by creating lots of objects for each logon.

Re:look before you leap

[E_elven]

>I sometimes wish all these Python/PHP/Ruby/whatever dudes would learn that strong typing is an ASSET not a problem. The problem are the people who design that poorly. If one writes one's software to take advantage of dynamic typing *while* making sure that the application logic is intact (e.g. calling DoSomething() only on classes that implement it), one is fine. Just because something can be used badly doesn't mean it will be.