Slashdot Mirror
Should A High-Profile Media Website Abandon Java?
"It is all hugely expensive to license and to run, and it's not very scalable. We'd like to up our userbase from several tens of thousands to ten times that number - but the cost of scaling the Java/Solaris infrastructure is not trivial, because the Java servlet architecture costs too much in memory and execution time (creating several 100Ks of in memory objects for each logon is expensive stuff!). On current hardware we can support only 1200-1500 concurrent logins and scaling up requires a new app server (eg 1 processor + 1GB RAM) and a $20K software license for each additional 600-750 concurrent logged in users. And in today's 'cost per active subscriber' economics it doesn't add up - we cannot justify the present cost structure, by any rational measure, even before we try to scale it up.
So we're thinking of chucking it out and replacing it with a largely static site that is generated (written out to cache) from a new, simpler content management system. The few dynamic elements would be assembled using simple PHP scripts, frontending our existing Oracle DB server. We reckon we could serve vastly higher numbers, ten to a hundred times as many, of users on the same (or cheaper!) hardware: and it would be simpler by far to build and maintain and support.
I, personally, believe that the benefits of the Java system (rapid prototyping, development) are not important when large scale deployment is the issue. I am (as a user) fed up with large, poorly performing Java-based websites. My beef is not about Java the language though - it's a question of appropriateness. Fifteen years ago we'd prototype in Smalltalk and then code for deployment in C, and I feel the same applies here. The economics of the noughties do NOT support spending massive amounts of money on web infrastructure, unless the transactional revenue justifies it. Of course, most businesses generally don't justify it, in my opinion.
Our outsourcing partner who supports and maintains the architecture thinks we are crazy. Putting their potential loss of revenue aside they are hugely concerned that we'll not be able to support what we create. They are seriously against this idea.
I remember, prior to Java & the like, supporting simple CGI websites with tens & hundreds of thousands of users off of cheap FreeBSD systems, and we didn't have to pay an outsourced partner to do it.
So what does Slashdot think? What would you do if you, were in the same boat?"
Seems like your problem is one of architecture and not the underlying platform. You suggest that you would move away from a dynamic site built with J2EE to a generated static site built with PHP. If you really feel having a generated static site is the best way to go then why not leverage your existing Java infrastructure and have it generate a static site instead of server a dynamic one? And if you can levarge your existing code base for that, then writing a new code base could still be done in Java, so I am not sure why you are pointing to Java as the problem.
With all the above being said, I don't know what is wrong with your system, but it isn't that hard to build a dynamic site in Java that meets your scabilitiy needs. All you need is a good caching strategy and you are set. Generally speaking, a good caching strategy coupled with a dynamic site can lead to as good as or better peformance than a static site.
Sorry dude, but you're going to get many of these:
For your case, I hope I'm wrong. Just out of curiousity, have you considered, I donno, profiling your application, to see where the time's being spent. Also, how about switching to a more cost-effecitve Java platform? I've heard good things about this little thing called Linux.
But, your current solution does work, right? How much exploration have you done in optimizing the application? Oracle can certainly scale, and you're already willing to strip down the site to mostly static pages. Why throw out all that proven and thoroughly tested code? And if you're outsourcer can't do that, maybe its time to switch partners, not platforms. You have a large investment here, there's a good chance you can save most of it.
Care about electronic freedom? Consider donating to the EFF!
Python is a refactorer's dream. You can transition your Java application to Jython re-using your Java classes while ironing out the bugs and design of the Python code, implementing caching, static HTML generation and the like.
When you're done, swap the JVM out of Jython and run pure Python with debugged code. If Python gives you any performance trouble, write small C-based modules for your frequently used code and wrap it in Python (fairly easy to do).
You really will get a lot of ideas, grief, you name it. You're attacking a bunch of sacred cows with this question.
One problem I see. Who's the architect of this thing? Is s/he being held responsible for your scalability problems?
Another: Does your vendor/outsource/partner actually know what they're doing? Where are their references recommending them for building a highly scalable site?
The other thing is: tuning tuning tuning.
------- MacOS X, WebObjects, Apple (G5) hardware triply tied
You should be able to deal with a lot of your scalability issues by putting some kind of cache in front of your system, like Squid.
But it sounds like every page on your site is really dynamic. And thus uncachable.oy
But you want to replace it with a mostly static site, so obviously, not all that dynamic stuff is required.
Before you chuck the baby out with the bathwater:
- Can you revise your existing java site to serve most pages as essentially static?
- If so, will putting some cheap squid cache boxes in front of your main servers do the trick?
This technique really works, if you can do it.
---
I support spreading santorum
On current hardware we can support only 1200-1500 concurrent logins and scaling up requires a new app server (eg 1 processor + 1GB RAM) and a $20K software license for each additional 600-750 concurrent logged in users
I'm afraid your company must seriously consider other J2EE platform, rather than root up your existing architecture.
First of all, fuck SUN. I'm biased, of course, because I'm here to pro-Linux in this case. SUN's J2EE app server is almost the most expensive among their competitors, not to mention the incremental maintenance cost incurred by expensive SUN hardware. Nowaday big corps like IBM and HP offers enterprise support for J2EE on Linux platforms, and their support are M3(24/7) with at least 3 9's maintenance
Also, you don't pay per user for large scale web deployment, you pay per server license. Fuck SUN's sales multiple timesfor not reminding you of better license terms for your new deployment.
I remember, prior to Java & the like, supporting simple CGI websites with tens & hundreds of thousands of users off of cheap FreeBSD systems, and we didn't have to pay an outsourced partner to do it.
You're just going backward in this case. Existance of J2EE platform is to solve various problems with CGI. One of our deployment just switch from CGI to J2EE due to the former behaved unstable when handling high volume requests. Of course, I've been told of many success with CGI, but J2EE seems to fit in in this case.
Besides, I don't understand why you've scale-up problem with J2EE. Scalability is the major advantage of J2EE. In our most current project, we decouple RDBMS(Oracle), Web-Tier(Apache), App-server(9iAS) and EJB containers(OC4J) into 4 seperated Linux cluster pool and one share storage of SCSI raw disks. We could easily scale up our architecture on various requirements.
If you design a site in such a way that you're getting performance that bad. Well, it doesn't matter which language you use.
Php, perl, c, etc. Are not the panacea your looking for, good architecture is what you need. Languages have their pluses, but it sounds like you just need a better design.
Instead, try to improve your current solution an bring its cost down:
Good Luck!
Stupidity is mis-underestimated.
Think of the risks that a rewrite introduces:
you break existing business logic with the new implementation
you build a system that is slower than the existing one
you take way too long to finish it, all the while you have to pay your existing licences
Typically, the argument for a rewrite is that the cost of implementing new functional requirements is higher than the cost of just implementing them in a brand new system. Have you tried optimization? How are you maintaining session state? Do you know what it would take to get your app running in a free container? Have you looked at free/cheap caching APIs?
Further, java may not be the out-of-the-box fastest platform, but there is no reason whatever that it can't scale in an environment designed for it. Yes, you may need to have many smaller machines because of jvm memory issues, but that's exactly what you should want. The ideal situation is when you can say, 'if we need to support 10x the current users, we just need to drop in 10x more app servers.' It's called 'scaling linearly.'
There aint no pancake so thin it doesn't have two sides.
Met me out back in 10 minutes. Bring $100,000 in unmarked non-sequential bills. I'll have your system rewritten in C++ and running in record shape in no time.
-Bjarne
or shouldn't this sort of thing be under 'Ask Slashdot'... that's what that section is there for, is it not?
(\(\
(^.^)
(")")
*This is the cute bunny virus, please copy this into your sig so it can spread
So what does Slashdot think?
I'm so glad you asked. Given the scant details, this should be great fodder for a Java/PHP/whatever flame war. Seriously, you expect good advice based on that. Having said that, look at some 'real' big sites and see if they use Java: yahoo, google, slashdot, etc.
Ditch Solaris and go with Linux or FreeBSD on Intel hardware. Amazon and AOL did it and saved buckets of money, so you should feel confident that you can do it too.
I have to wonder, seriously, if this is the right forum for such a question. You're asking people to make a snap decision on very little information and in all seriousness, it's something you're getting paid to decide, not us. There's plenty of information out there on the web about doing this, and you're asking us to do your research. /end soapbox
To celebrate the occasion of my 1000th post, I will post no more forever on Slashdot. Goodbye.
Though Shalt Not Kill.
Though Shalt Not Steal.
Though Shalt Not Covet a Non-Java Solution for a Website.
Whatever.
Why does everyone automatically assume Java solutions are the most cost effective when both programmer time and hardware is considered? It's not always the case. If you need very high throughput and low memory overhead then using C++ (possibly supplemented by a lightweight scripting language) is a perfectly valid approach. It's obvious that this guy is not an idiot. He's probably profiled and tweaked the Java system to death. He also tells us up front that the cost of content management system licenses and Sun hardware is way too expensive. He knows what he's talking about. Since he already has a working live production "prototype" it would not likely take more than a couple of months to knock off a C++ equivalent. I've seen many projects over the past year migrate from Java to C++ for the very reasons this guy has mentioned. Perhaps most telling - the premier web site of the Internet, Google, uses C and C++ for the vast majority of its realtime systems.
So, please Slashdot, grant this poor sole your blessing and allow him to use C++ to deliver a responsive and inexpensive system without feeling uncool or dirty.
The Python vote is seconded. I got interested in this language a year ago because a co-worker was experimenting with it and loved it. I took 20 minutes to run through a tutorial - an hour I was proficient and had re-written our entire project's build and archival system in it. The langage is clean and logical - docs are awesome.
/worked there] I can't remember which, but it was one of the above.
:)
Now it seems like every test and tool application I write to support development of our major products - is in Python. It's just so much easier, faster - and stable - than the equivalent app in C would be given the same time constraints.
No web Python referal would be complete without mentioning the Zope framework. I've not used it, but I gather it's to web development as nuclear weapons are to hand grenades. So integral to the Python experience that the author of Python [created it / works there
Take 20 minutes to run through a Python tutorial as referenced from their main page, try making some simple scripts with it - and tell me you're not hooked.
Also: don't forget the Psyco just-in-time compiler which magically makes time-critical things faster without you doing anything.
Also 2: Bittorrent was written in Python and bundled into an executable with Py2exe! How's that for battle-tested! The user interface looks bad, but that's the author's fault.
Also 3: Check out the Win32All package for MFC wrapper libraries and the best free editor (Pythonwin).
Also 4: Activestate has a commercial Python implementation that may help (I've not used it, but someone must have a use for it).
Wow, that's quite an undertaking you're thinking about. Here's what I'd recommend (disclaimer: I have made similar decisions but never on a site quite that large).
First of all, you need to find out WHAT is wrong with the existing system. Where is the slowness? What's making 100k objects? What's wrong with it? Have you profiled/debugged it? What if you slapped a cache in front of the whole thing?
What if you re-wrote it in PHP and found out it takes 1 week to write out all the static content? Are you sure that your new system will beat the old one? Are you sure you just personally don't like PHP more than Java?
I happen to like PHP more than Java myself, but for your site I would actually choose Java over PHP for maintainability. In fact I think your statement here is completely backwards: I, personally, believe that the benefits of the Java system (rapid prototyping, development) are not important when large scale deployment is the issue.
Don't overlook maintainability as an asset. Java is a more maintainable language than PHP, simply because it is so strongly typed. You can write your unit tests and use your IDE and write much better code, faster, once you get going.
Consider also swapping out proprietary pieces. I personally wouldn't use anything but a farm of FreeBSD machines for a site like yours. The days of proprietary OS are over. I have lots of experience keeping large numbers of FreeBSD up and running and it sounds like you do too. If not, Red Hat will help you out with Linux.
Database and Java app server can remain proprietary as I haven't seen much free that's good. Caucho Resin is a fast Java server, have you looked at it?
It is a big undertaking to completely rewrite a system in another language with another architecture. If you were my client I would strongly advise against and would spend a few months inventorying and examining all the pieces of your existing system to see the "hot spots" that need to be fixed ASAP. Who knows, maybe you've got one little routine somewhere that is sucking down all resources when it runs. Maybe Sun's Java runtime is not appropriate for your system or needs more tuning. Maybe you just need to add indexes to your database. I've seen all of these things in sites that supposedly needed a complete rewrite. Unless you get hard numbers, you'll never know. You could spend a long time optimizing the wrong thing.
Yes sometimes rewrites are the right thing to do. My "trigger" for a complete rewrite is unmaintainable code. If the existing code is so badly written that it can't be broken down into objects, unit-tested, profiled, etc., then maybe you can toss it. But I'd never toss it for performance reasons alone, since I know that most systems today (Java, coldfusion, mod_perl, etc., etc) can all be made good performers on today's hardware.
So in summary, think before you act!
If you are looking at a rewrite of the proprietary CMS, it's probably not that cost effective if you are only paying 20k/server (one time or yearly?) + maybe a 3k/year maintenance/service cost. Figure you will need to hire at least two, possibly up to 4 developers to write the new CMS, including importing all of the existing content, user information, preferences, history, etc. Even then, you'll be responsible for any upgrades you need. Have you asked the vendor about the performance problem? You're probably paying them on a maintenance and support contract already.
Another area to consider is limiting the number of dynamic elements on the page to allow the front page to be cached. Not letting users personalize the page will let you scale a lot better - just publish a static version of the front page every minute to a URL, and serve that up to all of the users.
If you do personalize, instead of building the whole page on the fly from the database/CMS, try and figure out a strategy to cache as much of the page as possible in memory, and then add the personalized elements.
Can you serve the streaming media off of another server, without the CMS? Store the URL to the streaming media in the CMS, and then have the page build the links to point to the other server.
I don't think moving to PHP is going to solve your problem, especially if you keep the same level of dynamic page building. You'd be better off looking at caching solutions, like other posters have suggested.
Tips and Tricks for Mozilla
Most of the early CMS's built during the boom times they were large, very flexible and complex systems that tried to be all things for all people.
I think products like ATG Dynamo are great but they tend to be very over-engineered for small to medium size sites and maintaining them becomes a nightmare of inter-dependencies. It doesn't mean they won't perform and scale, just that you need very experienced people that knows all aspects of your site and CMS that can stay on top of 'expensive' function and minimise it's use. I find that a web site tends to go downhill when the experienced staff leave the project and amatuers start to hack stuff together without understanding potential problems. The fact is any web site on any platform is susceptible to this problem.
There are a lot of cheap/free products to ease your pain but you really need to research your options.
JSR 170 ( http://www.jcp.org/en/jsr/detail?id=147 ) seeks to standardise content management and could help you move forward and allow you to stay ahead on the CMS scene.
Good luck!
Oz
The economics of the noughties do NOT support spending massive amounts of money on web infrastructure, unless the transactional revenue justifies it.
Noughties? As in 200x years, or as in porn?
Grandparent post is full of poo poo.
It sounds like you have a shop full of good Java people. While you may want to change how things are run, I would not change languages. This is not based on any love for Java, but the fact that if you have a team of Java programers to get them to the point where they can write top flight code in something else will take time that you can better spend on something else.
But I would consider changing the architecture if that makes sense.
Erlang Developer and podcaster
Not to be prying or anything, but what's your role in your organization ?
CIO? Sysadmin? Architect ? Project manager? Consultant? Programmer?
Do you make architecture/infrastructure decisions ?
Use JBoss (www.jboss.org) if you need to stay with Java and want to save on licenses. It's free and works well but I have no experience on scaling it much.
Serving static content will always be faster than serving dynamic content. Perfomance will further increase if you can cache the static pages in RAM.
For the dynamic part of the site I would prefer JSP over PHP as JSP pages are faster since they're compiled in RAM and doesn't need to be interpreted for each request.
You have a J2EE (or something like it) based application that is non-portable in both it's host OS and host application server and on top of that doesn't scale too well because of memory/CPU requirements?
Hmmm... somebody should be loosing their job. Either the consultants who built it or the person that approved such a thing.
If you had a true J2EE app that wasn't coded by a team of monkeys on a wild rampage this shouldn't a problem. The "porting" to a new app server should be trivial, if anything has to be done at all. You'd be able to keep the Sun hardware and whatever app server you use on it, while chucking in Linux machines with Bea, WebSphere or maybe JBoss on them. Slap a hardware based load balancer in front of it and viola, horizontal scalability.
I didn't see anybody else take offense to this, but 100k+ per user login memory usage? I might be showing my age, or rather my roots, but that seems excessive. My guess is your app's written like the app I now support. User logs in and everything about them is swallowed up into session (or application) wide collections immediately. The "lazy caching" thing just didn't cross these guy's minds. Of course, in my case neither did the "mark data dirty" thing but that's another matter.
Please, somebody show me 100k worth of data that you would really want on-demand from-memory on a user at any given instant. Just a C struct or something would suffice.
J2EE apps can be bleeding fast ultra lean sons of bitches if you do it up right. It can also be a dog-ass slow memory-hogging bastard. It just depends on who you had at the business end of the whiteboard I guess.
Going the PHP/static generation/caching route isn't neccessarily a bad idea either... but I don't think you should have to do this. I'm seeing the maintence of such a system as a big onus on the system administrators to make sure everything is up all the time... I know of no PHP frameworks out there that would let you drop sessions from one system to another. I've never tried pushing PHP that far though.
If your a system admin such a system might seem ideal... because while the systems and network might be a little "wonky" that's your domain and you feel comfortable supporting such a thing. I can't fault you for that; however I do think the onus is on the application development team. It is their job to make something scalable and construct it in a manner that it should fail over, recover, etc. from anything weird that may go on.
This isn't realistic, but you probably purchased a scalable application toted as portable because it's Java and you didn't get that. Demand that. If they can't deliver boot them out the door and take it inhouse if you must but I see many obstacles in your path on the system admin side alone... and certainly the re-development of it won't be cake walk.
Scalability problems are largely the development team's responsbilities, so long as such a requirement was put forth in the original development. Good system administration can help to reduce their errors along with a good helping of hardware but that's just a bandaid to the real solution.
Just my two cents.
This is excellent advice (i.e. MOD UP).
It sounds like some areas of the Java solution are giving particular problems, in particular with respect to memory. If this functionality is provided by the application server, you need to look at alternatives (WebSphere, JBoss, etc). If it is provided by your custom-build CMS then profiling and limited refactoring to make use of object pools, etc, should take you a long way.
Most of the cost issues (apparently) relate to the application server and its host system; investigate the option to migrate to cheaper systems.
And a basic financial response: how much will it cost to scale the current solution? how much will it cost to develop the new solution (in particular bear in mind development infrastructure, salaries, and parallel deployment to prevent downtime in production)? how much will it cost to maintain the new solution as compared to the current solution? which solution best fits your business requirements (other than cost)?
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
Also, anyone who got suckered into a Broadvision sales pitch for a enterprise solution should be shot^H^H^H^H fired on first site.
He specifically said it was a non-J2EE proprietary Java app server. My guess is ATG Dynamo (a pre-J2EE version). I doubt Sun's sales had anything to do with recommending it.
The guy is obviously on some political kick where his group is fighting the Java guys, and he's looking for standard slashdot peanut gallery "Java Sucks" commentary to score some points at the next backbiting meeting. Boring.
(Too bad for him, it looks like he ain't getting it.)
My guess is that he's in the system admin group and he's getting fingers pointed his way asking why the Java app runs so slowly. So, he's deflecting by pointing to PHP ('Personal Home Page') which so aptly powers his 10 hit/day blog.
Sad thing, he's probably right -- some salesman sold his company some expensive piece of crap app server, and they only use 5% of functionality. Dumbass PHP scripts could probably easily blow it out of the water -- for how they are currently using it.
So the real question, assuming he gives a crap about solving the problem at all, is if they are going to make full use business of the CMS or if they are going to route around it. That's question is probably above our questioning friend. But, certainly a straight JSP solution would perform in the same ballpark as any PHP stuff and still be PHB approved.
Learning what you're blabbing on about. You seem to be a complete neophyte when it comes to actually doing something with Java. Are you a PHB by any chance (a particularly geeky type of PHB who reads slashdot...)?
It's just so typical for people who don't have a clue about web application development to make stupid claims like:
1) J2EE is for rapid development/prototyping of web apps. - J2EE is far slower than anything else I've used. The advantage of Java is that if you use it properly, it lets you create robust, scalable, maintainable applications. It is *precisely* for large applications that Java is king.
2) The site is slow and creates 100's of kilobytes of data per user - Well, duh, if you wrote it to do that it will do it. If your software engineer had his head so far up his ass that he couldn't design the application better, fire him right now and hire a better one! Don't blame design problems on Java, blame them on the designer! Java allows very, very high quality designs, but you still need a quality designer!
3) The licence costs are prohibitive. - What's that got to do with Java? Fire whoever bought such a stupid licence and get a better one. It's not like there isn't enough choice, ranging from free JBoss to the more expensive (but better documented and supported) alternatives. If you run PHP on a win2003 advanced server with a per-user cost of $whatever will you blame php too???
Just from the cost cut of firing these incompetent people you can probably pay your licence costs (before you upgrade them to something more appropriate).
If you crash your car into your garage wall, will you also blame the steel the car is made of, or the stupid driver??
'nuff said.
Daniel
Carpe Diem
I've worked on some very large sites with concurrent users running into the hundreds of thousands... these range from http://www.btopenworld.com/ through to the UK's Football League clubs and premium content video sites.
/., we all love it, and it's on Perl!
In my experience, Java was not the wisest choice, it was bloated, difficult to maintain (that's one that will rile the pro-Java camp), required too much focus on non-business focus areas (i.e. creating things like session pools and encryption when we should have been focusing on getting the actual business requirements fulfilled), created a object model bureacracy (pure OO with respect to encapsulation? or break the purity of the model because you know in advance that you want 27 objects and you could get them all from one piece of SQL, but this would have presumed knowledge on the internals of the object and thus have broken the rules of encapsulation).
All in all, Java proved to be the most substantial factor in late deliveries of projects, limited scaleability... and expense (you wouldn't run Java on Windows, and we were running it on some very sizeable Sun boxes). We had several major works at performance improvements, memory caching, singletons to persist seldom-modified data, re-working SQL, etc. But this didn't help dig us out of the hole that we were in.
As a comparison, we also ran some Windows boxes with ASP 3 code on it... used prolific file system caching, and because of poor OO support abandoned hope of properly creating encapsulation and objects purely... we did use re-usable components in DLL's, and we did do extensive work to cache page parts in both memory and on disk according to the predicted frequency of use.
Both systems were behind reverse proxy caches... but the Java had the benefit of all pages being cached (as authentication ran in an NSAPI plugin on the proxy), whereas the ASP did not have its pages cached (just the images, styles, etc) as authentication code ran in the pages (it had not moved behind the plugin when I had left the company).
Yet of these... the ASP consistently performed better on page generation times, concurrent users, etc... even though the ASP boxes were just cheap Compaq servers and the Java boxes were very over-specced Sun servers.
My experience of all of this led me to the following conclusions... which were ever obvious but merely got re-inforced.
1) Right tool for the right job. And at the moment that means considering things like PHP, Perl, ASP for web pages... not Java. String manipulation languages and those that are lower overhead are performing better on web sites.
2) If you do use Java, be prepared to dilute the purity of the object model you create to favour performance. DO NOT get caught in the trap that the object model purity is more important than total performance/maintenance... OO purity does not necessarily equate to maintenance increase... documentation and commenting achieves that more.
3) Cache everywhere! Parts of pages, generated pages, the images and styles used on pages, the queries in the database.
4) Control your cache flushing fiercely! Do not allow apps to ever flush anything that you are not sure has to be flushed... wild-card flushing should never occur. If you stay in Java, implement the Observer pattern and persist and serialise data everywhere.
Ultimately it comes down to architecture... but I have witnessed that Java encourages really strange architecture as everyone starts running after a holy grail of a pure object model.
I would generally favour not using Java and going for the re-write. Other languages encourage pure string manipulation and control of what you're doing at a far more approachable level.
Remember that you're only creating web pages:
1) Query database
2) Concat string
3) Echo string
4) ???
5) Profit!
It really isn't hard, and doesn't need rocket science. Look at
... we need highly publicised failures, to counter MS marketing.
From the parent post: "Anyway, slashdot is the wrong place to be looking for serious solutions to problems like yours."
Maybe so, but Slashdot is a great place to get ideas. Many times Slashdot readers have extremely useful comments because they have unique experiences and are willing to share them.
It sounds like you've got a site written around a very proprietary system, and that your scalability etc. is tied to what that proprietary system can do.
.com that used the same proprietary system. Buy the company for pennies on the dollar and assume their license portfolio.
The solution, therefore, is to get away from the proprietary system. But only if you think you can do better. Either find a better proprietary system or write your own. If you write your own then plan for 'scale out' on lots of servers running something cheap like *BSD or Gnu/Linux, Apache, Tomcat, JBOSS, posgres, mysql, etc.
If you _can't_ get away from the proprietary app, then perhaps you can 'wrap' it in something else. Use static pages, PHP/mod_perl/C++/Lisp/jsp/whatever and a cheap but good database (mysql, postgres). Use these for all of the 'custom' content. Then have them access your 'back end' and dumb down the back end to get rid of everything that is not essential to a data feed. If possible aggregate the 'php' users into a few categories for the CMS to deal with. E.g.: have a 'sports' profile with 10,000 php users accessing a single 'sports' user on CMS.
Try negotiating with the vendor. Perhaps you can present your 'success story' at a gartner symposium or somesuch. Complain about scalability. Demand a linux version. Get them to agree to some unlikely performance guarantee and use that to cut costs down (via penalties). Get some free consulting from them to help fix the problems. Make sure to wear a T-Shirt or use a pen from their major competitor whenever they are around -- much more fun that way.
Find a failed
Another approach is to update your resume and get the heck out of there.
"But actually trying to use m4 as a general-purpose langage would be deeply perverse" --ESR
Usually there are reasonable things that can be done to increase performance at a low cost. Optimize a query here, cache some data there, remove a big memory sink there, and next thing you know, your application is running a lot leaner.
On the other hand, sometimes it is worthwhile to just chuck out the whole implementation and start over again. In my case, we're thinking of doing that because we want to implement a bunch of new functionality that really changes the intent of our application. And after 7 years of crappy coding practices, the codebase is a mess.
However, as an IT manager, I rarely find it "sellable" to take an internally-ugly application, replace it with another application of identical functionality, and tell anyone that it's a success. Especially given the teething pains you will experience with a new implementation.
Your note goes on to suggest that your organization can replace hardware platform (Sun), vendors, application technology (Java), and the application itself. Is your organization really in a situation to make so many changes right now? So many changes sounds like a huge risk!
If the application can be fixed, and the cost isn't too high, and there are no other organizational reasons to switch out all this technology - then I say keep it where it is for now.
Later, you might want to build a long term vision for future development efforts.
There's a lot of comments here to this effect already, I'm just going to add my voice.
If you have 100s of K per login it almost certainly isn't the platforms fault, and it probably isn't the developer's fault either - all that memory must be going to customize content for the user, which means you can trace the performance problems back to the requirements. (your developers could be crap too, but profiling will tell you!)
If the user gets content which requires a massive amount of customisation on each and every page - and this a requirement - then performance will suck no matter what the platform, as that memory will still need to be used.
I've been through this before with a customer who demanded we try out every app server under the sun to resolve performance problems even though we showed him profiling figures that proved only 1% of the time per request was appserver overhead - 80-85% was in the DB, and the rest was the app code. Because the customer took a "religious viewpoint" that the appserver was wrong rather than believing at the profiling data, we wasted weeks.
You need to profile before you can state that java is the problem - and equally, you need to profile before you can state that it's not.
We'd like to up our userbase
"We upped our user base... so up yours!"
"Up" is not a verb. The word you're looking for here is "increase," or, if you prefer, the Anglo-Saxon monosyllable "raise."
This isn't the problem.
(creating several 100Ks of in memory objects for each logonThis is.
So what does Slashdot think? What would you do if you, were in the same boat?"Don't know what /. thinks, but I think you need a serius rethink of your application's design. It sounds to me like you're throwing away exactly the advantage that servlets can give you by creating lots of objects for each logon.
Looks like I need to bring Joel Spolsky's excellent article, Things You Should Never Do, Part I, to a new readership.
;^)
The article speaks for itself, but essentially Joel's point is, "If it ain't broke, it's going to take you a heck of a lot longer to rewrite something inferior than you could've ever expected." Old code has tons of lessons learned that you'll never tease out. New code is easy to read and can implement every buzz word you'll find on O'Reilly Net right now, but it won't be battle-tested.
If you're still able to even think about throwing out your old investment and moving to CGI and BSD, however, I'm thinking your site isn't doing much very fancy. If you don't have much customization invested in your propriatary system, what Joel and I are saying is moot, especially at the licensing fees you're mentioning.
I'd also point out the title is very misleading. It's not Java that's the issue -- it's your system's architecture. Java is just as capable as creating a, "largely static site that is generated (written out to cache) from a new, simpler content management system," as language X. This is quite similar to the discussion we had about whether Java is an SUV just a while back (if it is an SUV, btw, that's not a bad thing). Your programmers' skillset is what's most important. If they already have a familiarity with Java, why ditch it?
So, keeping true to the post that says the recommendations here come out our arse, here's another pulled from the same place:
I'd recommend trying to refactor your current codebase to do two things. First, try to implement your static page idea using your current system. Two, take out as much of the crappy, non-scalable system that happens to be written in Java as possible. You don't name the system, but the whole advantage of Java is that it doesn't need to be platform-specific (if done right). Ditch Solaris. Create a server-farm of cheap x86 hardware with Linux or BSD with a JVM installed. Reread your license -- if you have thirty "clients" (new Linux servers) making static pages from one legacy server's dynamic content, can you pay a lower fee?
PS -- Who said Java was good for prototyping? Visual Basic (and vbscript/ASP or *gulp* ColdFusion), sure. REALbasic, sure. Java? Are you folks mad?!!
It's all 0s and 1s. Or it's not.
Why not hire somebody to do the job using freely redeployable tools, and hold back a big chunk of the payment unless they hit specified performance targets?
Given the license fees you are going to save, a quarter million dollars might not be too much to get you off the Red Queen's racetrack.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
The problem is that a java-based solution is used to develop a highly scalable real-time system.
This is just not possible.
Whatever you Java-gurus would say, it won't scale, it won't run as fast as it could.
Yes, perhaps development time is somewhat shorter in java, but if you want to tweak your app to run 0.75 as fast as a properly developed C++ solution, you would spend 5 times as much time profiling and looking for bottlenecks, when, in fact, the bottleneck is the java architecture itself.
In fact, the benefits you get from the java *programming language* are highly overwhelmed by performance problems you get when the system is scaled.
I mean, is it THAT hard for a bunch of programmers to write an efficient server front end, that would accomodate thousands of concurrent users in C++ on a *cheap* server? (I mean, 1 GB of RAM for 600 users is ridiculous).
No ! It's a month's work in C++ for ONE programmer.
Today's programmers are lazy. They want garbage collection, they don't care if 100kb are used when only 4 kb are required, they don't want to mess with pointers and direct memory access, they don't want to squeeze their OS to the limits.
Perhaps, the only industry which properly understands this is the game development industry, although not all of them.
Don't get me wrong - I'm not against Java as a programming language, I'm against using inappropriate tools to develop a certain task.
Here's my suggestion:
- Drop the Java-based solution - it helped create the overall system architecture, now it's time to really implement it.
- Fire 90% of the development team; Hire 5 experienced C++ programmers and pay them accordingly.
- Use the OS at it's limits. Of course, it's nice to have a platform-independent source base, but do you really need it ?
- Use Linux to save on hardware/software costs
- Drop the web server and develop your own OR use the web/database servers as efficient as possible.
In several months you will notice an all-time minimum cpu and memory usage and your users will be more than happy to browse your responsive site.
I've done it with two projects, one was heavily overbloated with EJB, another one was a typical JSP thing. In both cases I've moved to Python+Zope and it was done pretty quickly and smoothly.
Well, I admit, I've done it without Jython, as I've found there was no need for old/new code temporary integration aside of transparent authentification (which was simple - through LDAP). And I've made sure that in the middle of the transiotion no need to share any session objects.
Performance has been improved (shut-up about that common myth that "Zope is slow"), and so has been a memory usage.
So, I know on practice - it's doable.
By the way, I've never found the situation, when you think about re-writing some Python function to C to accelerate your web-server AND Java was fast enough with the same (logically same) function. In a well designed web-system (including templates and database) a web server has no potentially bad issues. Plus, you can always cache something. And that is the same with Python and Java.
Less is more !
From JavaZone Norway 2003 ten days ago I can conclude that a major trend seems to be moving away from the lowest layers of the j2ee-platform. Having some kind of business logic or sessions beans, accessed through some kind of servlet or whatever seems to be still tolerated. After all, you need a service interface on the server (clients are by many not trusted to handle business logic). Whether or not it follows the requirements for sessions beans doesn't really matter. If you have sessions bean interfaces, keep using them.
Entity beans, however, are generally disregarded. There is a general agreement that they are too clumsy, too big, too many classes, and too slow. You do need some kind of data access service. JDOs like Hibernate (especially hibernate) were considered as improvements. Hibernate data objects (java sourcecode and deployment descriptor files) can be generated automatically in a matter of minutes from your database, using middlegen.
Of course, if you move away from from entity beans, you will also have to change the business logic implementation (session beans) to use the new classes. You can not expect JDOs and entity beans do coexist happily.
The alternative to JDOs might be using presistence-aspects (see aspectWerkz on sourceforge), but that is only for _very_ VERY experienced developers.
If you plan the transition, you should be able too keep (major parts of) the baby without being stuck with too much of the water.
1. Are you the programer?
2. Does the programming staff know PHP?
3. Could you easily change your site to be static pages?
You might have a lot of solutions available. Have you looked at JBOSS? it is a much lower cost java app server. You could do JBOSS on Linux and save a bundle comparied to Solaris.
PHP is a good a good solution to a lot of problems but it is not what I would choose to generate static pages. Perl or Python are the classic choices but you could get wild and use Java, Lisp, or even OCaml. You could even try the GNU Java compiler and use java comiled to native.
If the people that currently host your site think you are crazy maybe you are. They might have done a lot of research and found that this is the best solution to your problem. They might be wrong but they might also be correct.
Writing your own is not always the best solution.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
There are a number of dynamic cache products that are developed just to solve your problem - from WARP Solutions (warpsolutions.com) just to name one. It's a much lower risk solution than to wholesale drop all your technology investment and replace it that could be just as bad, or worse!
Lenny Primak PP-ASEL-IA,Heli
See Ace's Hardware articles on how they converted from PHP to Java/Servlets/JSP, it is a blow-by-blow walkthrough that reads like a HOW-TO:
Building a Better Webserver in the 21st Century
SPECmine - A Case Study on Optimization
Scaling Server Performance
It's 10 PM. Do you know if you're un-American?
Your problem is that you contracted this out to a company that grew up during the dot com haze and they gave you a dot com CMS. Now they're more than happy to manage and maintain the beast.
If you weren't using proprietary Java APIs (from BEA, ATG, Broadvision, etc.) you could get the same system running on inexpensive Intel-based machines with Linux or FreeBSD, Apache, JBOSS or JonAS (I can't believe nobody mentioned it yet) or even straight Tomcat with a Postgres DB on the back-end. All free/open source software.
Then refactor/re-architect the system. There have been plenty of comments and suggestions about that. No need to move to PHP unless you prefer maintaining pages of script language over a good MVC development approach.
Sounds like changes are infrequent... in which case a good caching strategy is your key.
.NET framework has built-in directives for kernel-level page caching and userspace-level page fragment caching.
My suggestion would be to build a proxy/caching server and specifically set the no-cache directive for dynamic content. If you have dynamic content within most pages that would take a slightly different approach; I would suggest hosting dynamic content on seperate pages on the server and have the proxy server stitch the two together. A proprietary HTML extension (similar to IFRAME) and slight modification of the proxy server is all that would be required...
Then you can run the proxy server on commodity hardware and when it's time to scale, go to a dynamic load-balanced environment, with your java app on the backend and the webcluster in front. Buy a ton of commodity servers and stack them to the roof in your data center.
If you want to get smarter about it, recode the proxy servers to replicate their caches to each other...
And I would have a cache invalidate command... so when you publish new content you just click a button and all the caches invalidate, guaranteeing that you're serving fresh content.
Of course if you want off the shelf, Microsoft's Content Management server can do all of this relatively cheaply.
And the
I am disrespectful to dirt! Can you see that I am serious?!
I'm no expert, and I'm strictly talking "off the top of my head" here, but I have a couple of ideas for you.
First of all, I never understood why people who were serving up something static like news articles would want to store them in a database, fetching them dynamically with Java. They usually say "to permit searching" but there are other ways to do that (i.e. store some metadata and URLs in the database, and the actual articles in the filesystem). I think most people who write database apps do it because "that's how you're supposed to do it"; they follow the herd and play it safe. Others are building whatever is going to look good on their consultant resumes.
Consider this: what if you were to store your text articles as xml flat files in a web-accessible filesystem on a server? The xml files would contain tagging for all of the metadata for the articles, plus the article content. You could work out a scheme in which the actual path to the article xml file was related to the article metadata, to ease searching and lookup (you could search based on metadata, article text, paths, or whatever). If you have media files related to the article, give it a "media" directory, and work out a tagging scheme for the xml file which allows you to place the media's URL in the file. You could use an XSLT stylesheet to convert the file for display in a browser, on the fly, when a user requests it. Maybe minimize your use of the database, using it more for user/permissions management than actual article management. Use Perl scripts and PHP web pages to actually interact with the system.
Benefits: You're spending a whole lot less money on your tools, you can back up your whole system by burning chunks of the flat-file directory to CD-Rom, Your scripts aren't going to be memory hogs because they'll probably only be a couple of K each, load balancing is going to be a piece of cake because you'll be mirroring your filesystem on each server (all the calls will be the same), you can use any inexpensive/free O/S you want (Linux, FreeBSD, NetBSD, OpenBSD, etc), you can go with a basic Apache system so it won't be that hard to administer...
I know it's old fashioned and totally unsexy, but it worked pretty well for the old-timers. Or, if you want it to be sexy, pick up some of those fru-fru Apple rack servers. They're pretty, and they seem easy enough to admin.
You've got lots of options, just get your own coders. If you hire consultants, they'll go for maxing out their billable hours and their expense sheet. In-house guys are generally on salary and don't have much in the way of conflicts of interest (umm... IF, that is, you don't hire any fanatics).
Anyway, like I said, I'm no expert but here are a few ideas to chew over. Free Advice, no warranty, YMMV, etc.
Farewell! It's been a fine buncha years!
My company is a large-market daily-US newspaper, and we are building CMS systems in Zope & Plone (using Python). There may be several advantages to using a scripting language, but a shift from Java to a non-OO scripting language like PHP is likely higher risk for you - Zope (and the Zope Content Management Framwork) may offer a better solution given it has a toolset of components to leverage out-of-the box, and a simple, component-oriented way of developing content management applications with a scripting language that is easier to use, but just as scalable as Java-based solutions.
Because it uses an object database for content repositories for digital asset management, you minimize the need to do object-relational serialization and marshalling between an OO system and a relational datastore. However, this isn't as complicated as it sounds; consider something like Archetypes, a schema-driven content type generation system that also has built-in relationship management for composition of media products from related assets.
Shameless plug: I'll be giving a talk on how we are doing much this at the Plone Conference. in October in New Orleans.
Scalability costs with this type of content-management solution will not be in licensing of yor apps, but in commodity hardware (scaling out). These costs would be greater than if you used something much more bare (i.e. PHP has no security model built-in, so pages might require less resources to render, but you get more limited flexibilty or need to implement such a layer anyway for your application, negating the performance difference), but performance and price would likely be on-par or more competitive than Java solutions.
If you are not in the market to build, but to buy the first 80% of your way into a solution, Zope Corporation has built a commercial CMS product on top of these open-source foundations (Zope/CMF, Squid+ESI) - it is called Zope4Media, initially developed for Viacom and Boston.com (one of the largest local media sites out there, which might speak for the performance characteristics of a well-designed Zope application).
You have already seen the problem, but do not want to acknowledge it. You are on a system that when it scales, costs lots of money.
Several others have suggested refactering/rearchitecting. If you have not done so in a while, you may wish to do just that. You have suggested the idea that Java may be killing you, but then you point out that you use pure java on Solaris. Your real costs is the system with the use of a pure dynamic environment.
If you truely have a significant amount of static or near static info(alters no more than say once a minute), you will probably find the cheapest bang for the buck is to simply cache the items.
Using squid (or some other caching) on multiple intel/low-costs systems with BSD or Linux to dish up static info is well worth the costs. Make it heavy on ram which is still cheap. You do not even have to refactor or move to a different language.
I prefer the "u" in honour as it seems to be missing these days.
Interesting. I've been learning J2EE this year, and I concluded pretty early on that Entity Beans seemed like a massive waste of time, a good way to increase complexity with very little gain over just encapsulating the SQL. Glad to hear I wasn't missing some fundamental benefit of entity beans.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
..and keep milking your job for all it's worth.
Generally the comments in here seem rabidly pro/anti Java, pro/anti Sun, ...
....
The reality is that the really big sites, the ones which scale, have thousands of simultaneous users, seem to be coded in mixtures of Perl, PHP, etc across multiple machines in a tiered architecture. The application is designed to scale, and break the functionality up into digestable bits that do not depend upon a specific app server.
Another reality (if you can filter the chaff from the wheat) that emerges is that even the rabid pro-java camp admit that it is hard to make Java apps scale on these systems, as you need someone who "really knows what they are doing." I take that to mean it is hard, and this jives with my, my team's, and my companies' experience. Yes, you can build a scalable application with Java. No, it is not easy. It is not trivial. It takes significant experience, and lots of trial by fire. Of course, such programmers are expensive as well. Real experience costs you real money. Luckily for employers, there are many such people out there looking for jobs (courtesy in part due to the massive bubble in the economy that has burst).
So what to do: You have a few options
1) re-engineer the hardware. No brainer, Sun equipment is slow, and should not be deployed where performance matters. Most folks who have been burned by this generally learn this. One of the reasons why Sun is and will continue to suffer.
2) re-engineer the software: Tweak/tune cycles. Not really a great idea, as you need to know what the problem is before you tune. You need a detailed understanding of what the code is doing to understand where it is spending the time. This requires profiling a complex application, where you have a tiny fraction of the source. I presume you do not have the app server source
3) re-write the software: In Java or in another language, using the app server or not using the app server. The disadvantage is the loss of investment in the original. The advantage is that you get to make real choices (unless the CIO/CTO is a twit and mandates a particular solution... I see far too much of this from my customers. It puts the cart before the horse).
If you have to make real choices, remember, Yahoo and other high volume sites use PHP. Slashdot uses mod_perl. A number of others use Bricolage (http://bricolage.cc) based upon HTML::Mason (http://www.masonhq.com), other good sites use Zope (http://www.zope.org).
The issue is application design, and how well the application language allows you to exploit scalable concepts in design. Check out a random selection of Java based websites, and a random selection of PHP based websites. Which is faster? It cannot be that there are many more PHP programmers out there than Java programmers, as the number of good Java programmers should swamp the total number of PHP programmers. Why is it that the Java sites seem so slow? Yeah, non-scientific. I would like to see a fast Java site. Seems they are just rare. Must be due to lots of bad Java programmers (the theme of many of these threads). Either that, or it is really hard to make it scale.
The SPARC platform is expensive, and you can get a much better value by using x86 hardware.
I wouldn't wish C++ on my enemies.
And yes, now that you asked, I really like C. But C++...
next thing you know, you'll be telling people that the STL "is really great".
It's true.
If you took the time to understand its principles - you'd agree.
But perhaps it's not your cup of tea - it's more for experienced programmers.
Today's programmers are lazy. They want garbage collection, they don't care if 100kb are used when only 4 kb are required, they don't want to mess with pointers and direct memory access, they don't want to squeeze their OS to the limits.
Where's the economy in spending $1,000,000 on java developers over the course of 2 years and buying big iron SMP Sun hardware at the cost of at least another million bucks?
Expert C++ programmers could have written the same code in the same amount of time for $1,000,000 and have it run on $50,000 worth of commodity hardware - for a savings of a million dollars.
I don't buy this "hardware is cheap" bullshit when it comes to large scale sites. It ain't true.
50% improvement in code speed and reduced memory usage results in HUGE hardware savings - easily saving the enterprise project more than development costs in just the first couple of years.
Yahoo (perhaps you've heard of it) is not exactly a small time operation.
They studied memory usage, stability and load and came to the conclusion that PHP was the right tool for the job.
Your advice is completely worthless.
100KBytes per logon is ridiculous, and shows that the architecture/design of the dynamic page was poor and didn't consider scalability, resource-caching and sharing etc. Also, java with free application servers on linux x86 boxes is pretty cost effective.
Where are we going and why are we in a handbasket?
When a module's new maintainer forgets to look before she leaps, she may break some of the contracts implicit in a module's behavior. Thus, there needs to be some way to automatically enforce these contracts. Many maintainers use test cases to enforce contracts after compilation, but test cases tend to fall subject to bit-rot. It'd be nicer if there existed facilities in the language for enforcing contracts. Static typing is a language feature that enforces one specific class of precondition, namely that the arguments to a function have specific types. Some languages such as Eiffel implement this Design By Contract(tm) paradigm even more deeply.
Will I retire or break 10K?
You mean "up" was not a transitive verb. As of October 2003, the American Heritage(tm) and WordNet(tm) dictionaries both define "to up" to mean "to raise."
yarr!
With a modern optimizing compiler STL vectors and lists become as efficient - or more efficient - space and speedwise over your own hand rolled code.
Is GCC 3.3.1 such a "modern optimizing compiler"? Or do C++ programmers have to shell out upwards of $6,000 per seat for the processor core vendor's own compiler in order to reap the benefits of STL?
Will I retire or break 10K?
Java seems to have the most hooks to the really cool database functions necessary for enterprise web. This isn't MySql land anymore. You absolutely need persistance and triggers and a DB that takes care of itself. Right now that's a terrible hack in PHP to even use that stuff, worse if you try to hack it on to MySql. It can be done, but even the writers of it will tell you not to put too much faith in it. It's not good or bad, just not the right tool for the job.
Frankly, IBM is where to be for enterprise web development. The stuff is designed to Just Work. You don't even have to use the expensive websphere products for everything, it's the fact that IBM has really embeded Java into their servers...in the same way Perl is woven into Linux. Java hook into all of their "marketspeak" products.
At the risk of sounding silly, look into IBM iSeries machines! [check out www.search400.com for iSeries happenings] Yes, they are BOOORRRRING, but in the enterprise server world boring is good! Seriously, they have almost every feature that a stock Linux box has.. out of the box. Sort of like BSD, a lot of stuff has to be turned on...lots of iSeries sit with LOTS of potential dark nowdays, IBM has really dropped the ball. The coolest feature is the Embeded DB2 that's built like a tank! All the replication, syncing, sharing machines is stock IBM code for you to use. Catch is they are REALLY, REALLY expensive to purchase....but REALLY cheap to operate! That make it a hard sell, but again, it's about the RIGHT tools for the job...an things go smoothly.
Fist of all, with JSP/Servlets you almost have the fastest solution currently possible running, given an underlying VM.
You have to have in mind that Java currently performs almost on the same levels as GCC generated C++ code.
PHP and Python currently are not in the same areas, same as perl, but the coming parrot VM might change this.
I think you should try following, you run an app server, which one, do you use EJB, if not, ditch the app server in favor of something leaner.
You don't need an EJB engine with 200 Megs overhead for serving servlets and JSP!
I have a site running with around 6 million hits per month running on two tomcats with an apache mod_redirect in front of it, the machine is not state of the art (I think a single Pentium 900 or so with 1 gig of ram, both tomcats, need arund 300 running around 15 webapps, the problem is the database on the same machine not the java app servers - I had to face the same problem, but more on a shoestring budget, I really praied for getting a better machine but the customer wanted to recycle this server)
The solution to your problem might be really downsizing, go for a static dynamic combo, use dynamics only where needed but don't ditch the perfectly working code here. If you see that the app server needs too much ram, go for something leaner like a plain servlet runner (resin or tomcat are very good with a low mem footprint)
Also look seriously into other JDKs, some JDKs go for a lower mem footprint and sacrifice some of the optimizations Hotspot does.
The other solution would be to move away from the Sun iron (sorry Sun I really like your machines but here is a problem you can't solve, price), to a cheap pc based solution, using multiple app servers delivering the content, on a pc cluster with an apache and mod_redirect in front to it.
Sounds like your architecture is fundamentally broken. I don't think java is the cause of the brokenness; I'm guessing that if you implemented the same architecture in PHP/ASP/PERL/COBOL it would suck.
I have personally been in this position - I managed a development team which had inherited a large piece of web architecture (not java !) from a consulting firm just prior to launch (yeah, I should have seen it coming), and it didn't scale. At all. On our big, expensive, 4-processor sun boxen, we could support a total of 12 concurrent interactive users. We were expecting a few million. DOH !
We had very little time, and looked at our options - we knew the architecture was fundamentally broken, we knew it would not support our business requirements, but we also had to go live in 3 months with 76 websites and millions of users.
We decided to put in place a program of short, medium and long-term refactorings. The short-term stuff was all about shoring up the application for launch - we used Squid to cache the static content, we rewrote a bunch of key bottlenecks, and generally did heroic stuff. We did this through "SWAT Teams" - each team was responsible for one key performance area we had identified. They had a great amount of latitude, and very clear, measurable performance targets.
The medium term improvements focused on improving the architecture to make it more flexible and extensible, so we could support our long-term objective : moving away from the architecture and re-building piece by piece in a new framework.
On the whole, this worked very well - the sites survived their launch, and regularly serve a million or so pages a day. We migrated key areas to our new framework (which was based on Java, natch), and performance went away as a key business risk. After only a few months, the business changed direction and we couldn't afford to continue our re-architecture plans and had to concentrate on short-term feature improvements instead, but that was sorta okay - our refactorings had given us the opportunity to do that and not worry too much about performance.
So, my advice to you is to look at the architecture you have now, what you want to end up with, and draw a roadmap. Make sure the roadmap contains numerous delivery points, because you may have to stop what you were doing halfway through, but if you've delivered at least some of your improvements, you can reap their benefits immediately.
Oh, and good luck.
It's all very well in practice, but it will never work in theory.
I think that for Multimedia streaming and such Java is going to do the job a good deal easier than PHP.
I love NetHack.
Profile your database Quest software has a lot of tools for this. I work on a huge n-tier system written in VB with an ORACLE backend and until I profiled out all the crappy SQL the thing ran like crap. It's extremely fast now users can find customers out of over a million records in seconds. Before profiling it took minutes. Things like adding,removing and changing indexes, making sure your SQL uses said indexes and redesigning querries pay off big time!
solutions pulled from my ass. I don't tend to think Java is your problem, and what you are suggesting with the static content/caching/PHP is not a step forward by any means. In fact it sounds like a suggestion from your sysadmin, who is probably a PHP fanatic and does not know any better. Don't know how many times I have had to smack our sysadmin at my company for saying: 'Why does it have to be so complex? just do the whole thing in PHP!' For most people complex means any thing but [insert the only language you know here] I for one think:
1. PHP is not likely to solve your performance problems. It is recompiled every time you hit the page, its not scalable, blah blah we all know the limitations.
2. As suggested somewhere above: Profile you app. Find out if your code just sucks, and fix it.
3. Chances are, you are not taking full advantage of J2EE platform. This is very likely.
4. Your hardware is probably old. Upgrade. I'm guessing you have a big ol' sun box with couple of 750 mhz processors or something old like that. Yes, its a nice big pretty box; yes, it cost you $126,000 and yes, its mostly filled with air. Good luck trying to pawn it off. You can max it out for a few hundred k more.
5. [licensing/costs] Sun/Solaris is Expensive as Hell, so too is your J2ee server. Try a bunch of cluster of intel boxes running JBoss on Linux. Can't get much cheaper than that. You could get a whole farm of these for the price of that sun box. (or you could have, but didnt.) And depending on which J2ee server you are currently using, you wont have to rewrite anything.
6. Oracle is pretty scalable, its relativly fast, but if your code sucks, its your fault.
Is java slow, or does your code suck? You might just want to make better use of what you have. I am a big advocate of Java and open source software of course. Its always worked for us. There is no reason you should'nt be able to do what you are trying to to with Java/J2ee. There are many reasons why other platforms won't cut it, mostly because they were not designed to.
TallGreen CMS hosting
I notice that you didn't specify how many *actual users* your projects supported. While I acknowledge that you could just make up any old number and I'd never be able to prove you wrong, I suspect that your solution wasn't viable for the number of users the submitter is looking to scale to. A Python+Zope combination might be great for a couple thousand concurrent users, but since he's looking to scale there's no way on Earth I'd deviate from an architecture someone has actually implementing successfully before. It's just to easy to get burned by trying to pioneer some obscure solution on a project that's on as large a scale as he's undertaking.
Before you start forking huge dollar amounts for an an app server or re-engineering the entire system, try moving to Jetty/JBOSS. I think you will be pleased with the results.
Obviously. I doubt he's just going to go through the Slashdot answers, print one and hand it to his boss. Perhaps he is 'asking us to do his researh'. On the other hand, perhaps he is asking us to have a discussion about CMS frameworks, which many people will be interested in.
Your financial response is pointless. He already gave you the numbers to figure the cost of scaling the existing solution. He has no idea how much it will cost to develop and maintain the new solution, because he doesn't have one yet. That's why he's asking.
We recently had a great deal of success converting a project similar to your own by having content re-generated every 6 hours but including way more JavaScript etc. in the site itself - I have to say that going client side with more complexity caused some unbelievable testing expenditure but it DID give us immense concurrency increases which saved amazing sums on the cost per subscriber. On top of that we became platform independent with the exception of the ISAPI filter we wrote for lightweight sessions, data submission and e-commerce requests. Even the C++ filter could be ported with no serious effort. This came into play when we just found out the thing will go over to Apache in six months (gotta love those suprises). I love Java but I hate to this how I could have implemented as cheap and effective solution in it that didn't bind us and the client permanently at the hip.
Bubble Pipe Life does not stop and start at your convenience Dude
If you're having problems like 100+K memory objects for a single user (have you got their whole family history in there?), then I suspect you're going to have a whole lot more problems when you try to do the same things in PHP.
Something sound awfully skewed in that application. I suspect that your site does the dotcom era idea of "MyNewsite" and stores loads of info on the specific users layout etc. While this is fine and dandy, one does ask oneself why all this is sitting in memory all the time. It sounds very much like an architecturural problem. There are surely 100 other ways of having that specific user have his own layout (seperating the content and templating logic out into the database or another server for instance) where that info doesn't have to overload the memory of the actual application server.
Redoing it in PHP would work only if you changed the architecture. If you used the same ideas in PHP your problems will only get worse as PHP's session management is defintely not at the same level of Java's.
I tend to agree. Sun hardware and Sun licenses are what will run you into the ground financially. Set things up on a free OS of your choice with x86 hardware and you can scale up a huge amount relatively cheaply. Just ask Google.
Linux is, of course a good choice simply because it has an official, up-to-date Java SDK and J2EE implementation available from Sun.
After reading this story, and all the comments, I've come to this conclusion:
Good programmers can write good/fast/efficient code in ANY language. Crappy programmers write crap/slow/inefficient code in EVERY language.
--
"What do you want me to do? Whack a guy? Off a guy? Whack off a guy? Cause I'm married."
Thanks 'newbroom'. This has been the most thoughtful discussion for some time on Slashdot.
Please post again.
As numerous people have pointed out, 100K of objects per session is insane.
There are dozens of tools out there you can use to track down the bottlenecks in your code.
http://www.quest.com/jprobe
http://www.quest.com/performasure
http://www.borland.com/optimizeit/index.html
http://www.wilytech.com
Any of these tools, along with a competent developer/analyst, can improve your app performance. I used to teach classes on Java performance tuning at one of these companies and I saw people get huge performance improvements from very simple changes - once they knew what to change.