Should A High-Profile Media Website Abandon Java?

newbroom asks: "The company I work for runs a large, high-profile web site with users all across the world and delivers them large amounts of streaming media content plus textual stories. You might guess therefore that this is a news website, frequently updated throughout the day, and delivering content 24x365. No names, or course, for obvious reasons. We have a big, custom, Java content management system (based on a framework from a proprietary vendor as it happens, but could just as well be EJB/J2EE for all that it matters in the context of this argument) and for deployment we run our website using Java app servers on Solaris behind Apache." If you were going to take such a site from 1000 users, to 10,000 users, would you be able to do it using this kind of setup?

"It is all hugely expensive to license and to run, and it's not very scalable. We'd like to up our userbase from several tens of thousands to ten times that number - but the cost of scaling the Java/Solaris infrastructure is not trivial, because the Java servlet architecture costs too much in memory and execution time (creating several 100Ks of in memory objects for each logon is expensive stuff!). On current hardware we can support only 1200-1500 concurrent logins and scaling up requires a new app server (eg 1 processor + 1GB RAM) and a $20K software license for each additional 600-750 concurrent logged in users. And in today's 'cost per active subscriber' economics it doesn't add up - we cannot justify the present cost structure, by any rational measure, even before we try to scale it up.

So we're thinking of chucking it out and replacing it with a largely static site that is generated (written out to cache) from a new, simpler content management system. The few dynamic elements would be assembled using simple PHP scripts, frontending our existing Oracle DB server. We reckon we could serve vastly higher numbers, ten to a hundred times as many, of users on the same (or cheaper!) hardware: and it would be simpler by far to build and maintain and support.

I, personally, believe that the benefits of the Java system (rapid prototyping, development) are not important when large scale deployment is the issue. I am (as a user) fed up with large, poorly performing Java-based websites. My beef is not about Java the language though - it's a question of appropriateness. Fifteen years ago we'd prototype in Smalltalk and then code for deployment in C, and I feel the same applies here. The economics of the noughties do NOT support spending massive amounts of money on web infrastructure, unless the transactional revenue justifies it. Of course, most businesses generally don't justify it, in my opinion.

Our outsourcing partner who supports and maintains the architecture thinks we are crazy. Putting their potential loss of revenue aside they are hugely concerned that we'll not be able to support what we create. They are seriously against this idea.

I remember, prior to Java & the like, supporting simple CGI websites with tens & hundreds of thousands of users off of cheap FreeBSD systems, and we didn't have to pay an outsourced partner to do it.

So what does Slashdot think? What would you do if you, were in the same boat?"

Comments

Your problem is architecture

[DevilM]

Seems like your problem is one of architecture and not the underlying platform. You suggest that you would move away from a dynamic site built with J2EE to a generated static site built with PHP. If you really feel having a generated static site is the best way to go then why not leverage your existing Java infrastructure and have it generate a static site instead of server a dynamic one? And if you can levarge your existing code base for that, then writing a new code base could still be done in Java, so I am not sure why you are pointing to Java as the problem.

With all the above being said, I don't know what is wrong with your system, but it isn't that hard to build a dynamic site in Java that meets your scabilitiy needs. All you need is a good caching strategy and you are set. Generally speaking, a good caching strategy coupled with a dynamic site can lead to as good as or better peformance than a static site.

You need to justify your choice of J2EE platform

[jsse]

On current hardware we can support only 1200-1500 concurrent logins and scaling up requires a new app server (eg 1 processor + 1GB RAM) and a $20K software license for each additional 600-750 concurrent logged in users

I'm afraid your company must seriously consider other J2EE platform, rather than root up your existing architecture.

First of all, fuck SUN. I'm biased, of course, because I'm here to pro-Linux in this case. SUN's J2EE app server is almost the most expensive among their competitors, not to mention the incremental maintenance cost incurred by expensive SUN hardware. Nowaday big corps like IBM and HP offers enterprise support for J2EE on Linux platforms, and their support are M3(24/7) with at least 3 9's maintenance

Also, you don't pay per user for large scale web deployment, you pay per server license. Fuck SUN's sales multiple timesfor not reminding you of better license terms for your new deployment.

I remember, prior to Java & the like, supporting simple CGI websites with tens & hundreds of thousands of users off of cheap FreeBSD systems, and we didn't have to pay an outsourced partner to do it.

You're just going backward in this case. Existance of J2EE platform is to solve various problems with CGI. One of our deployment just switch from CGI to J2EE due to the former behaved unstable when handling high volume requests. Of course, I've been told of many success with CGI, but J2EE seems to fit in in this case.

Besides, I don't understand why you've scale-up problem with J2EE. Scalability is the major advantage of J2EE. In our most current project, we decouple RDBMS(Oracle), Web-Tier(Apache), App-server(9iAS) and EJB containers(OC4J) into 4 seperated Linux cluster pool and one share storage of SCSI raw disks. We could easily scale up our architecture on various requirements.

Should you throw out the baby with the bathwater?

[Phouk]

No, you should not - throwing out a significant body of tested, working code in favour of "new, better!" code is not a good idea generally. A well-known article by Joel Spolsky eloquently explains some of the reasons.

Instead, try to improve your current solution an bring its cost down:

1. If you are scaling up by adding servers, see if you cannot add something like two Linux servers instead of one Sun (for a fraction of the price, naturally).
2. You haven't said which software packages bring up the license cost (besides Oracle, of course). But for most, there are open source alternatives. Sure, they might take more work to set up in some cases, but certainly less than rewriting the whole application, no?
3. You might even want to evaluate if you can replace your Oracle by a SAP DB instance if that is not your bottleneck (Hint: Caching! Caching!).
4. If, as you say, hundreds of KB are used up for every logged in user, then, in all likelihood, there are big inefficiencies in your code. You should profile it / have it profiled for both CPU and memory efficiency. Then tune the 5% of the code that use 95% of the resources, instead of throwing away 100%.
5. Are the outsourced programmers up to snuff? Maybe have the code checked by a third party (who could also do the profiling / tuning). Because a bad programmer can bring down any infrastructure, be it J2EE, .NET, PHP or whatever. It's the man, not the machine.

Good Luck!

yes, switch

[consumer]

It doesn't have to be to PHP, it could just be to an open source Java platform, but get off your expensive proprietary platform before it drives you into the ground. Java has good enough performance when done well, but most commercial Java frameowrks make it hard or impossible to write anything that isn't bloated, so ditch that thing. If you are good with PHP and your site is reasonably simple that should work fine. So would Python, Perl, simple Java servlets, etc. Static publishing (from a database) is a great idea if you can get away with it.

Ditch Solaris and go with Linux or FreeBSD on Intel hardware. Amazon and AOL did it and saved buckets of money, so you should feel confident that you can do it too.

Re:Python

[RevAaron]

That is the worst attempt at my-favorite-language-cheerleading I've ever seen. Ok, it's not, but it's still pretty bad.

1. Writing C for web apps is not a solution. The wrapping tools for Python aren't impossible to use, but they can't perform miracles. Yes, it is very easy to use an external C function for performing some repetitive math function, an FFT or something- but in a data-intensive web app, it really makes no sense. In the case of the poster's problem, he and his team would end up re-writing half of the framework their using in C, giving it Python interfaces. If they were having problem with just Java's raw execution speed, they could just as easily use Java's JNI to interface with C libraries.

2. No matter good it looks on paper, going from a big system written in Java for one particular framework to a system written half in Python and half in Java doesn't make all that much sense. They'll be dealing with the same bottlenecks, the same bloat- it's all running on the JVM. If anything else, they'd increase the footprint and slow the app down, as they're adding on yet another layer of complexity.

Yes, I am fully aware that Jython outputs Java bytecode itself, but Sun's Java compiler does a lot better generating efficient Java bytecode out of Java than Jython does. Nothing inherent to Python or Jython, but when you've got a multi-billion dollar project like Java, when you consider what Sun puts into it- then compare that to the miniscule (by comparison) project that is Jython, it'd be absurd to expect the same results.

I know it's easy to get a little jumpy when the dude mentions PHP and your favorite language is Python, or hell, anything that isn't PHP. You want to come in any say "hey, use my favorite language!" Believe me, I'm wanting to do the same thing, and I could substitute the word "Smalltalk" for "Python" throughout your post, and it'd be just as true; unfortunately, so would my points against it.

Python and Jython certainly have their places, no doubt. Depending on a couple factors, I may use Python to write my system intiailly, but simply having a language that spit out Java bytecode doesn't mean you have some non-trivial, seamless transition between two system.

Let me get this straight...

[pi_rules]

You have a J2EE (or something like it) based application that is non-portable in both it's host OS and host application server and on top of that doesn't scale too well because of memory/CPU requirements?

Hmmm... somebody should be loosing their job. Either the consultants who built it or the person that approved such a thing.

If you had a true J2EE app that wasn't coded by a team of monkeys on a wild rampage this shouldn't a problem. The "porting" to a new app server should be trivial, if anything has to be done at all. You'd be able to keep the Sun hardware and whatever app server you use on it, while chucking in Linux machines with Bea, WebSphere or maybe JBoss on them. Slap a hardware based load balancer in front of it and viola, horizontal scalability.

I didn't see anybody else take offense to this, but 100k+ per user login memory usage? I might be showing my age, or rather my roots, but that seems excessive. My guess is your app's written like the app I now support. User logs in and everything about them is swallowed up into session (or application) wide collections immediately. The "lazy caching" thing just didn't cross these guy's minds. Of course, in my case neither did the "mark data dirty" thing but that's another matter.

Please, somebody show me 100k worth of data that you would really want on-demand from-memory on a user at any given instant. Just a C struct or something would suffice.

J2EE apps can be bleeding fast ultra lean sons of bitches if you do it up right. It can also be a dog-ass slow memory-hogging bastard. It just depends on who you had at the business end of the whiteboard I guess.

Going the PHP/static generation/caching route isn't neccessarily a bad idea either... but I don't think you should have to do this. I'm seeing the maintence of such a system as a big onus on the system administrators to make sure everything is up all the time... I know of no PHP frameworks out there that would let you drop sessions from one system to another. I've never tried pushing PHP that far though.

If your a system admin such a system might seem ideal... because while the systems and network might be a little "wonky" that's your domain and you feel comfortable supporting such a thing. I can't fault you for that; however I do think the onus is on the application development team. It is their job to make something scalable and construct it in a manner that it should fail over, recover, etc. from anything weird that may go on.

This isn't realistic, but you probably purchased a scalable application toted as portable because it's Java and you didn't get that. Demand that. If they can't deliver boot them out the door and take it inhouse if you must but I see many obstacles in your path on the system admin side alone... and certainly the re-development of it won't be cake walk.

Scalability problems are largely the development team's responsbilities, so long as such a requirement was put forth in the original development. Good system administration can help to reduce their errors along with a good helping of hardware but that's just a bandaid to the real solution.

Just my two cents.