Slashdot Mirror
On the Supercomputer Technology Crisis
scoobrs writes "Experts claim America has been eating our 'supercomputer feed corn' by developing clusters rather than new supercomputer processors and interconnects. Forbes says America is playing catch-up and that the new federal budget items are too little too late. Cray is laying people off due to decreased federal spending and claims lower margin products have forced them to create products based on commodity parts. Red Storm, one of their new Linux-based products, is being delayed to next year."
when you can build a top 5 supercomputer for under 6 million dollars, using off the shelf parts. Why spend the hundreds of millions of dollars?
This is an expected and predicted fallout from the recent rise in popularity of beowulf clusters. Slowly but surely managers are realizing, yes, it is possible to have a supercomputer on mass-market hardware, running a free OS.
Don't see this as bad news... it's a sign that we're winning.
+ Donald Gunth
+ Email: dgunth@quicktek.net
"Caffeine is the greatest lubricant ever created." -ESR
What most people don't seem to understand is that you don't need a supercomputer when a mesh of nodes on a network will do just as well. Just like most people don't understand that a 386 running Linux and Word Perfect 5.1 is just as good of a word processor as a 2.5Ghz Itanium running Windows and Word. Computer power has *usefull* limits as well as technological limits.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Of course people are going to cry that companies like Cray are falling by the wayside, but the truth is that their services simply aren't as needed as they were in years past.
If you have to ask, you'll never know.
I think that should have been "Seed Corn."
Free market sucess might lead to us actually having to pay for our own supercomputer research that we use in profit making ventures.
Random Array of Inexpensive Servers.
If the 'supercomputers' of today are increasing performance, does it really matter the design?
Maybe that is a signal that monolithic computer tasks are best handled in a hive mentality - have the Queen issue the big orders, have the warriors performing security, have the workers transporting the goodies (data), and have the requisite extra daughters and suitors to grow the hive and assure its viability (redundancy).
The fact that it is cost-effective is even better.
Don't see this as bad news... it's a sign that we're winning.
Right. The Cray folks have just realized that they are about to go the way of buggy whip and the slide rule. They don't like it one bit. They can only complain by making a lot of noise. But it won't work. When you're extinct, there is no coming back.
It's seed corn. Seed, as in, what you don't eat, but save to plant next year.
Kids these days.
With reasonable men I will reason; with humane men I will plead; but to tyrants I will give no quarter. -- William Lloyd
Clusters are not good for very chattery parallel processes, a shared memory supercomputer can still do much better for computational fluid dynamics.
Its the fact that clusters require higher skill to program efficiently for than do single processor systems. Plus you have all of the wasted processing power used for communication between the nodes. Granted, many problems lend themselves well to distributed computing (essentially what a cluster is, but the nodes are closer and communicate faster), but there are also problems that are handled better by a smaller amount of specialized hardware. The other point is that by using off the shelf parts, we are not really innovating in this space like we should be. We are allowing the commodity computer market determine the direction of the supercomputer market.
One of my professors (everybody has one of these it seems) is working on cluster computing research, extensions of MOSIX. He's a guy with networking and operating systems expertise. I wouldn't hire him to build a new generation of super computing interconnects or processors. As the Republicans have taught us, federal budgets are not a zero sum game. Why divert focus from one to the other when we could have both?
We have to be careful about measuring these things however. One of the goals of cluster computing was to lower the cost of computing. If the government is spending less and still meeting needs, thats not nessecarily an indicator of a problem. If that means that we aren't writing code to fit into a vector platform, so be it!
I Browse at +4 Flamebait
Open Source Sysadmin
Cray has been engaging in scare tactics about "America being dominated by overseas competitors" for a while, because they're terrified of losing the lucrative business contracts from government and big business, they'll pull out all the stops. They've come up in the IT press recently a couple of times.
Screw 'em. If there's a need, the market will provide. If it turns out that the important tasks can be parallelized and run on much less expensive clusters, then all that means is that we have a more efficient solution to the problem.
May we never see th
Poor little babies, now where will their executives and boardmembers get free money? Will they actually have to do something useful for a living, and for a change? It seems we are only a Darwinian capitalist economy when the little guy gets fucked. When the professional bullshitters at the top get it, it becomes some sort of strategic crisis that requires immediate injections of billions of dollars. Screw them. May Cray rest in peace.
La Supercomputadora is Dead! Viva la Supercomputadora!
If you really want a vector-processor supercomputer you can program in Fortran, get yourself a G5 and gcc. The PPC64 supports SIMD vector processing. For that matter, any problem which benefits from vector processing is trivial to parallelize with threads.
In the age of IP and patents it seems like it is very hard for companies to make major advances [in any field] without some other company cry foul and taking that company to court over patent/IP rights, especially if the alleged infringer is a smaller company (i.e. less lawyers). IBM and MS, among others, are filing dozen if not hundreds of patents a day. What we are seeing as an affect is that innovation is being stifled by litigation.
(pat pending)
"Look Lois, the two symbols of the Republican Party: an elephant, and a fat white guy who is threatened by change."
Technology first developed on the high end slowly works it's way down into the low end. What happens when the high end is no longer there.
Not that many people really need a race care, but advances in fuels, materials, engineering in race cars eventually leads to bette passenger car. And for raw performsnce, strapping together a bunch of Festivas will not get you the same as an Indy racer.
There seems to be some historical revisionism going on regarding the demise of the "supercomputer industry". People are coming out of the woodwork now saying that lack of government support caused the great supercomputer die off.
As Eugene Brooks predicted in his paper Attack of the Killer Micros, the supercomputer dieoff was caused by the increasing performance of microprocessor based systems. Many of us now own what used to be called supercomputers (e.g., 3GHz Pentinum processors, capable of hundreds of megaFLOPs).
The problem with supercomputers is that high performance codes must be specially designed for the supercomputer. This is very expensive. As people were able to fill their needs with high performance microprocessors they quit buying supercomputers.
Many people who need supercomputer levels of performance for specialized applications (e.g., rendering Finding Nemo or The Lord of the Rings) are able to use walls of processors or clusters.
There are, of course, groups where putting together off-the-shelf supercomputers will not suffice. But these groups are few and far between. As far as I can tell they consist of the government and a few corporations doing complex simulations. The problem is that this is not much of a market. Even if the government funds computer and interconnect architectural research, there does not seem to be a market to sustain the fruits of this research.
In the heyday of supercomputers there were those who argued that when cheap supercomptuers were available the market would develop. The problem is, again, programming. High performance supercomputer codes tend to be specialized for the architecture. Also, no supercomputer architecture is equally efficient for all applications. It is difficult to build a supercompter that is good at doing fluid flow calculations for Boeing and VLSI netlist simulation for Intel (the first applications tends to be SIMD, the second, MIMD). The end result of these problems tends to suppress any emerging supercomptuer market.
The reality right now seems to be that those who are doing massive computation must build specialized systems and throw a lot of talent into developing specialized codes.
If there truly is a demand for those kind of processors, then somebody will likely meet that demand. Right now, it seems that actual demand is so low that they have to drum up this legislation a as a sort of wellfare for vector processor manufacturers.
It's a simple cost tradeoff. If you can save millions in purchasing computers, it means more money to pay for people to run those computers and do the real work.
This sig has been temporarily disconnected or is no longer in service
So, what tasks still require a high-speed shared data memory? Answer that, and you'll understand where you can still sell a supercomputer.
Bruce
Bruce Perens.
And do interesting things. And try to keep in touch.
For things like weather forecasting, maybe big vector machines still have an edge, but I suspect that's changing as the weather guys get more experience in using machines with large numbers of micros. This seems to have already occurred, in fact; NCAR appears to have mostly IBM RS6000 and SGI computers these days, with nary a Cray in sight.
The most common term I used to hear in the early 90's was Killer Micros; I think the term dates back David Bailey in the 80's sometime. If you want more evidence that the death of the supercomputer has been going on for a long time, check out The Dead Supercomputer Society, which lists dozens of failed companies and projects over the years; this page was apparently last updated 6 years ago!
Have you read my blog lately?
Granted, it is more difficult to program something (from the ground up) that runs distributed, than it is to program something that runs on a giant 2048-way box.
Just like it's more difficult to write multithreaded code than it is to write single-threaded code.
That's where software, and platforms come in. There is a TON of research being done, which uses technologies like Infiniband and Myrinet as interconnects, and can make a cluster "look" like a big monolithic machine. If you as an end user write code that goes down into the TCP stack itself, you're working too hard, and you're going about it the wrong way.
Put it this way: In 5 years the odds are overwhelming that there will be a good software platform that can let you pick 5000 servers and run your app 10,000 threaded, with everything appearing just like a single process, and running "as it would on a Cray." It's easier to solve this stuff with software -- take your problem (distributed computing) and solve the problem with a different set of technologies (high performance/low latency interconnects, shared address space/DMA across machines, etc).
Apple's Xgrid is a step in this direction. It's missing a ton of "Supercomputer" functionality right now, but it's a nice cross-machine GUI scheduler. Right now this type of app can address maybe 20% of what supercomputer apps need... in the future maybe more like 98%.
And in other news today, buggy whip manufacturers demand increased government subsidies.
Jesus was all right but his disciples were thick and ordinary. -John Lennon
Forbes has been complaining that federal support of advanced computing is too little? If the government over-stimulates an industry that has too small of a market, it wil just delay the failure.
Of course the governent should continue in its current policy of funding a few leading-edge machines that are too costly to sell into the general market, but will test new technology. The governemnt itself is a customer will energy testing, weather modeling, medicine development, etc.
I've been in this field over 25 years, been in public position at a major lab now for 8.
If this was a simple issue, the HPC community would already have completely moved to clusters and never looked back 3 or 4 years ago. But it's not kiddies.
Want to run a physics projection for more than 1 microsecond? Takes real horsepower that clusters cannot provide even distributed. Just too much damn data. Chem codes that include REAL data for useable time slices? too slow for clustered memory. Every auto maker in the world (almost) has been whining about the lack of BIG horsepower for a few years now.(crash codes and FEA) I could go on forever. Sure, some problems work awesome on clusters, which is why we have them. But definately not all of them.
The problem is partly diminishing returns, partly the pathetic ammount of useable memory on a cluster and its joke for memory throughput, partly the growth in power of the low end and clustered networking, partly the ridiculously long development cycles invloved in High Performance Computing and the low $ returns,
One of the biggest things congress sees is that this country will more than likely NEVER again lead the world in computing power for defense and research.
And thats something we ought to do as the last real Superpower.
The national labs TRIED clusters, they don't get all the jobs done they wanted. (see testimony before congress, writings in HPC jounals, and the last couple RFPs from US gov. labs,heck every auto maker in the world) People in HPC _know_ it now, but having let what little there was of the supercomputer industry die out, there isn't mcuh of an industry left to turn to now. It just may be too darned late. HPC hasn't been a money making industry since the early 80s.
Heck, even Intel abandoned their clustered machine they custom built for the government.
Most folks in HPC will readily admit the Top500 is kind of a joke. The HPC-challenge #s are a little more realistic for the tests, but we really do need something that approximately real world applications, not just a 70s cpu benchmark.
For those that think this is a 'Linux wins' issue,
consider that mostly it was fast interconnect networks that allowed clustering, not the OS. Examine the history of clusters and you'll see this is true. Btw, the last few SC companies are already mostly moving to linux anyway.(nec,fujitsu,cray;ibm dabbles in hpc)
Hopefully the industry will survive long enough to allow for even better mergers of supercomputing power with low end cost, but at this point I doubt it. Cray has been on the ropes since 96, fujitsu's sc division is a loss leader, and NEC has been trying to get out of it for a while for something with a margin.
Ed -gov labs HPC research punk
-former Cray-on
-former CDC type
I've seen a lot of naive comments suggesting that supercomputers are being replaced by clusters. The truth is, anyone who can replace their supercomputer with a cluster didn't need a supercomputer in the first place:
- (compared to a supercomputer):
- The prime advantage of an x86-based server is that it is cheap, and it has a fast processor. It is only fast for applications in which the whole dataset resides in memory - and even then, it is still the slowest of the group.
- Clusters are a little better, but suffer from severe scalability problems when driving IO-bound processes. As with the x86 server, if you can't put the full dataset into memory, you might as well forget using a cluster. The node to node throughput is several orders of magnitude slower than the processor bus in multiple CPU systems. (6.4GB/s vs 17MB/s for regular ethernet, or 170MB/s for Gigabit)
- Multiple CPU servers do better, but still lack the massive storage capacity of the mainframe. They work better than clusters for parallel algorithms requiring frequent syncronization, but still suffer from a lack of overall data storage capacity and throughput.
- Mainframes, OTOH, possess relatively modest processors, but the combined effect of having several of them, and the massive IO capability makes them very good for data processing. However, their processors aren't fast at anything, and often run at 1/2 or 1/3 the speed of their desktop counterparts.
- Supercomputers combine the IO throughput of a mainframe with the fast processors typically associated with RISC architectures (if you can still consider anything RISC or CISC nowadays). They have faster processors, more memory, and much greater IO throughput than any other category.
It used to be that the prime reason for faster computers came from the scientific and business communities. But now that the internet has turned computers into glorified televisions, the challenges have gone from that of crunching numbers to serving content:As our economy has shifted away from a technological base to an entertainment one, the need for supercomputers has begun to evaporate. We outsource innovation overseas so that we can lounge around on the couch watching tv and drinking beer (or surfing the net and drinking beer). The primary purpose of technological innovation has shifted from that of discovering the universe to merely bringing us better entertainment.
The society for a thought-free internet welcomes you.
One technology that I work with is called Artificial Life and is basically large evolutionary software simulations. (This is not exactly the same thing as genetic programming, but it's close.) This is an example of something that just plain doesn't cluster well. Try to cluster one of these, and you will max out a gigabit switched LAN in less than a second (I've done it!). I've even maxed out a gigabit "star configuration" LAN with this stuff. It just doesn't cluster.
The problem is that these simulations involve many cells that must interact with each other in real-time. The cluster spends 90% of it's time waiting on other nodes no matter how you build the architecture.
There are lots of other problems like this that just don't cluster well.
Clustering only works for problems like protein folding or SETI that divide up into neat "work units" that can be shipped out and then returned. Millions has been spent, along with massive amounts of time by people like myself, and we're still no closer to being able to really cluster applications like this efficiently.
There is no super computer technology crisis, there is however a paradigm shift happening in the supercomputer market. Twenty years ago building your own supercomputer, even a loosely coupled cluster, was not a very viable option for most research institutions. Today this option is not only viable but often exercised.
Obviously the big SC vendors and designers seeing less business roll their way, why pay them tons of money when you can have grad students assemble your cluster for the price of some pizzas? That isn't to say SC clusters are the end-all be-all of computing but they're very useful and relatively inexpensive. Realistically they're simply an extension of what Cray started with their T3D supercomputer. The T3D was very impressive in its days but now the technology to build such systems is in the hands of just about everyone.
Taco: What the hell is up with the IT color scheme? This is even worse than the scheme for the Games section. I know the Slashdot editors don't actually read the site but other people try to and we're not all colorblind or reading from grayscale monitors.
I'm a loner Dottie, a Rebel.
Several engineering issues making this Ludicrously LSI operation possible have not been solved. One type of logic found in computers is called CMOS, the other called TTL. TTL has another special implementation that allows it to achieve high clock rates.
This special TTL that is used today has reached its peak at .09 nanometers in width. The semiconductors used today do not have enough "gas" (voltage) to power such a large system. Every transistor has a voltage drop within it, and adding more and more transistors only increases heat, power consumption, and probability of failure. If a single part of a huge new chip were to fail, the entire chip would have to be replaced.
Ever wonder why CPU's run hot? They dissipate voltage (and with it current, which is the True Power). At some point, the input voltage on the processor will be so close to the output that measuring HIGH or LOW (1's and 0's) becomes impossible. Until some scientist discovers a brand new material that can overcome current and voltage barriers, clustering will be the norm. Said scientist would be a billionare if they could patent a material like that.
Researchers are out looking for these new substances, mind you. Until they find one and build a semiconductor company, single computers with low clock speeds and high bandwidth will not be able to tackle what the human imagination has instore for them. There is also a limit involving physics and microwaves that I could get into, but i'm not a physicist. I'll leave that up to them :)
Two...I have two words for you!
Seriously, I don't see the problem, so long as companies like IBM and (dare I say it) Microsoft continue to do research in this area. That is the real value of companies that are committed to *real* research in revolutionary sciences and technology.
Of course, US companies don't have a hammerlock on this research. There is a lot of work being done internationally in the area, by corporations, and by educational/research institutions.
---anactofgod---
---anactofgod---
"Equal opportunity swindling - *that* is the true test of a sustainable democracy."
In particular, he wants a 2000s-version of an 1980s architecture running a 1960s language. For $1M, he could train his technology guys to use newer programming techniques. Yes, I realize that Fortran 90 is newer than Fortran 77 which is newer than Fortran IV which is newer than Fortran 1, and that the biggest CPU job these guys do is usually crunching big matrices of floating point numbers. That's a job for a subroutine you write once and feed with data and user interfaces that are written in languages that are more efficient for prototyping and user interface design.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Is it because there is a perceived zero-sum game being played between Linux-based clusters and supercomputers? Hey, let's take a reality check here, a lot of research is not directly applicable. In fact, I've read numerous discussions on /. railing against the MBAs and the Bush regime for not funding anything that doesn't turn a profit within about 18 months or have something to do with killing brown people.
/.'rs that the country as a whole subsidizes advanced computing? Isn't computer science all about seeing what can be computed? Letting supercomputing die because it's expensive seems like an extraordinarily short-sighted thing to do.
Letting supercomputing die may be harmless, after all, the US doesn't have to be the best at everything in the world and some other country will fund the research. But from some of the more coherent posts I've read, it seems like supercomputing has a definite niche in the natural sciences, something we should be pushing for a better society - learning for learning's sake - and paying for out of public coffers. My taxes go to a lot of shitty things I'd rather them not go to, like subsidizing Haliburton with no-bid contracts. Why is it so offensive to
Oh, and yes, I'm a Linux fanboy, but I was also reading comp.arch (remember Usenet?) back in the days when the Attack of The Killer Micros was starting to kill the minicomputer and mainframe industry ("careful with that Vax, Eugene!") and RISC vs. CISC was still a design issue, so I do have some perspective on the game.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
And there's no real reason why you can't do that with a cluster as well- archetecting both the hardware and the software for the cheapest possible construction.
Reminds me of my early programming days on the TI99/4A. The brilliant bit that made that computer more powerfull than most other micros on the market at the time (except maybe the Commodores and Ataris, but neither of those had this as much as the 99er did) was multiple specialized subprocessors. Most others had *maybe* a video processor and a sound processor, but the TI also had a memory manager and an I/O management chip and a speech synthesis chip as well. A good assembly level programmer had his own cluster supercomputer- even if it was only running at 3.44Mhz. Better yet, the PEB allowed you to build your own specialized cards for additional tasks- Disk I/O had it's own processor, as did Serial and Parallel communications.
We do this today with off the shelf consumer hardware thanks to USB, AGP graphics cards, and separate sound processors on sound cards. To truly take over the Supercomputer market, that's the paradigm we need to get back to.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
What does it matter if we don't develop single unit supercomputers. Clearly in a free market if these thing had value they would be persued. There is not predetory tax laws on supercomputer, or any other regulations on domestic use. The only reason development has slowed is there is not much market for the beasts.
There are many reasons for that too, for one other then in stealer, neculear, mathematic, and bio research feilds few industries need more computing power then can be had off the shelf any day of the week. That was not true yesterday it took all sorts of custom hardware to make CGI happen in films that can be done now in my basement in resonable time frames. So no more super computer market there the ROI is gone I am sure this plays out in all sorts of other engineering feilds as well.
Many places where you do need super computing power can be done with clusterd systems that are cheap to build and cheap to maintain.
At least people in the pure science and research fields have learned to be better thinkers and programers, they found ways to do things in parallel that were traditionally serial. Things that still are serial can be made to work on a cluster, sure it might take longer then a single computer considered to be equal FLOPSwise but considering I could either spend all the money I saved makeing my cluster bigger and more powerful so I can get back to equal time or on other profitable efforts while I wait there is again no ROI.
It so happens that may of the most interestin questions in math, physics and computer science such as quatum theory need massive amounts of parallel work, rather then serial so that works better on a cluster anyway.
If there is a real reason to do it people will build supercomputer, because there is nothing stopping them other then economics. No need to fear Supercomputers are not going away. Everyone else that needs that kinda proc-ing power will settle for clusters, as well they should. This is just another largly obsolete industry wanting someone to bail them out because they have failed to adapt to a changing market. If they are going to die we should let them, just like we should let the Universitys adapt or die, and the RIAA needs to adapt or die, we need to stop proping up obsolete undustries so new ones can replace them!
Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
Oh really. Don't blame me for not trusting a guy with that kind of potential bias.
Time makes more converts than reason
Yes, but they work ON supercomputers, they don't build or design them... Do they?
I was talking about a center whose purpose was the creation of ever-more-powerful supercomputers. The rental section would just be there to make use of the tech, and put it through its paces.
Farewell! It's been a fine buncha years!
How about a compromise like SGI's NUMA systems? Seems like a reasonable compromise between fast memory & lots of CPUs when memory access is important.
my sig's at the bottom of the page.
This reminds of a Quote from Cray. It goes something like, "Would you rather have 1024 chickens pulling a wagon or a big Ox."
New supercomputing advances on the way will radically redefine the industry.
I refer to a DARPA funded project created to fill in the performance gap between today's inadequate SC technology and tomorrow's (quantum, bio) still far in the future stages.
The project is called HPCS, which does not stand for High Performance Computing System, but rather High Productivity Computing System. The point is not to increase flops but increase value. The earth simulator, for example, is down for maintenance about 2/3 of the time and can only be reliably run in 8 hour chunks. The ASCII series may have high peak performance but averages only 5-10% of that. DARPA knows that if this is the state of the art, then there is work the do.
Three companies are currently doing research on proposals and are DARPA-funded through 2006. The three are Sun, IBM, and Cray. Two companies will continue from 2006. A working product will be delivered in 2010.
Many radical new technologies are in place, but as you can understand most of it is tightly under wraps. But do some reading on the DARPA page and you will find some interesting things.
Here is the link
- employee at one of the three aforementioned companies
There never really was a supercomputer market. There was a cold war, that subsidized the supercomputer market.
Then there is the cost. Companies stopped making SC because they were too expensive. If the guy from Ford wants to pay 1 billion for a supercomputer I am sure someone will build him one. The cost build a FAB is over 4 billion. Why do you think HP teamed with Intel. Why do you think there are so few processor families? You have to make a living in the commodity market where you can sell things in the millions because supercomputers even in their heyday were sold in the hundreds.
Then there is the problem that many problems are solvable on clusters. So those specialized problems can not depend on other parts of the HPC market to help subsidized their corner of the market. i.e. clusters make the really hard problems more expensive.
It is question of how much you want to pay to solve your problem? Simple economics actually. If the numbers don't work, the problem doesn't get solved. If the Gov. wants to solve some problems (and during the cold war they did) then they can step in and subsidize the market.
And don't cry about Japan and the Top500. When the top500 has price column then it will start to be meaningful.
HPC for Primates. Read Cluster Monkey
Actually you're only partially correct. The alpha processor despite popular opinion was headed towards the scrap heap before Compaq bought them. The reality is that the conventional 'supercomputer' markets were falling away from the Alpha because of the promise of the Itanium and Opterons (which were just poking their heads out of their holes at the time and announcing their existance). In additional benchmarks of the existing alphas were only marginally better than the Intel Pentiums of the time (pre-Itanium & AMD Opteron) despite the 64 bit processing. The stall wasn't because of Compaq, but because the core development teams for the Alpha were... well, getting old and retiring without suitable replacements. Plus I heard it rumored that Compaq purchased Alpha not for the processor, but for the architecture which it intended to 'reinvent' in a new image. Dunno what happened with that though...
I hear this type of FUD all the time from some of the older folks I work with -- all the hype about Cray systems, shared memory capabilities. There are several problems with the 'supercomputing' market though. First off if you go to any local University, they teach fortran as a basic intro course, but most professors will footnote their comments by saying things like "Fortran is no longer considered a marketable language"... so students think -- why waste time learning it (other than the basic programming capability?); this erodes the base support for vectorized programming support (putting aside arguements that fortran is not vector programming for now please). More importantly (and secondly), Cray architectures are MASSIVELY expensive in relation to where the standard desktop CPU is without enough benefit to make them worth it. People talk about the need to play "catch up" as if there is a real crisis at hand. The reality is that clusters are gaining in popularity because they're cheap, and per CPU tend to be more 'powerful'. Institutions that purchase Crays end up with a number of folks trying to run software and inevitably you run into time bottlenecks. What folks at these institutions realized is that by purchasing a 'state-of-the-art' 32/64 bit system with dual processors and a couple gig of memory, that by the time their jobs were starting on the Crays, they could already be running on their new desktops.... In the end I say why rent time on a big Cray when you can purchase your own system and run it into the ground with jobs? Despite some of the FUD I've seen splashed around by folks who are proponents of Cray systems, Clusters are relatively simple to setup, do not require multi-million dollar yearly contracts, and can generally be maintained by the purchaser without much effort. Most of the cost of clusters anymore is the cost of the system administration -- which you don't need a "staff" of SA's to administer. IMHO, you don't even need an SA anymore for designing, building, and running a cluster; with only a baseline of knowledge of computers, and a little reading just about anyone could build their own. When all is said and done, you end up with more compute power in a cluster at a much lower cost. Some folks would say that throughput is an issue; that's problem dependent, and can be rectified with a little problem solving. Some folks indicated that clusters aren't optimized for peak performance... WHO CARES?? If it takes me two minutes longer for a 5 week job to finish, I've still made out better for not purchasing a Cray. I've heard folks complain about not enough memory, or programming problems -- generally these tend to be older folks (what I like to call the obsolete-engineers; aka -- old fogies) These are generally the folks that complain about the 'new fangled' software they have to use, and simply don't want to have to reinvent their "marvelous software" in a new environment. I've run 64 gig memory jobs on clusters without a problem; again my problems allow for parallelization optimization... Eventually I end up telling folks who are steadfast supporters of the Cray to wake up, this is a brand new era about you. The days of the single supercomputer vendor filling all your needs are over. You now have choices, and with those choices you have to take the responsibility of defining how your new parallel supercomputer will function. Either learn and adapt, or become obsolete.
If you read the papers at the recent OLS (Ottawa linux simposium) you'll see that SGI is running linux images (specially tuned) on 64, 128, 256, and in 2 cases 512 cpus. Reading the paper is an interesting view into the problems of running kernels and OS's on such huge NUMA machines.
http://www.finux.org/proceedings/