On the Supercomputer Technology Crisis

scoobrs writes "Experts claim America has been eating our 'supercomputer feed corn' by developing clusters rather than new supercomputer processors and interconnects. Forbes says America is playing catch-up and that the new federal budget items are too little too late. Cray is laying people off due to decreased federal spending and claims lower margin products have forced them to create products based on commodity parts. Red Storm, one of their new Linux-based products, is being delayed to next year."

Comments

it makes sense

[dncsky1530]

when you can build a top 5 supercomputer for under 6 million dollars, using off the shelf parts. Why spend the hundreds of millions of dollars?

Re:it makes sense

[Otter]

If you RTFA, an administration panel on high-end computing claims that clusters are inappropriate for certain tasks. I don't necessarily trust the claims of what I assume is an industry-heavy panel, but then I don't necessarily trust the supercomputing expertise of a bunch of Lunix fanboys "administering a network" in their parents' basement either.

My inclination is to let the market sort itself out, although if supercomputer makers go under, they won't necessarily reappear the moment they're needed.

Re:it makes sense

[TedCheshireAcad]

It all depends on what you need the computer to do. If it's a special purpose machine for, let's say, primality testing for large Mersenne numbers, then standard x86 or PowerPC CPUs would be a waste of money. You'd want a bunch of chips that just do FFT, and do it quickly.

Take for example Deep Crack (luminaries, remember that one?). Perfect example of specialized hardware for a single job.

Although, you are probably right in that most of todays supercomputing needs can be met by clustering together off-the-shelf equipment. Although some of the heavy lifting (like solving giant systems of partial differential equations numerically, for, say, weather modelling) could be done faster with specialized hardware, it's probably more cost-effective just to but a thousand PowerMac G5's and rack them up.

Re:it makes sense

[gUmbi]

when you can build a top 5 supercomputer for under 6 million dollars, using off the shelf parts. Why spend the hundreds of millions of dollars?

Because instead of fundamentally advancing the science of computing, the industry is simply scaling commodity technology. The American supercomputer industry has gone from innovator to an assembly operation.

We're in need of a paradigm shift. Where's the next Seymour Cray?

Jason.

Re:it makes sense

[Rosco P. Coltrane]

Well, I don't have that kind of money, but if I did, I'd rather get me an older Cray than a cluster of beige-box PCs. If nothing else, Cray machines are classy and impressive, and when we make clients visit the premises, we can go "oh, and this is the Cray computer, crunching at numbers for you" and look at the customer being impressed, as opposed to "oh yeah, that pile of nondescript computer, that's the 1000-node beowulf cluster of AMD Cheaperon computers". I'm sure the extra value in marketting would be worth it...

Re:it makes sense

[badboy_tw2002]

Then trust the fact that not all problems are easily attacked from a parallel perspective. This means problems where working on one section of the dataset affects large amounts of data in other sections. There's a lot of locking and waiting for tasks in other parts of the system to be completed; and a lot of data transfer/need for shared memory, which if you're bussing between cluster components, its going to be slow.

This doesn't mean that clusters don't have some use in these regards, it just means that for these types of problems no one has figured out an efficient parallel algorithm to use on them.

Re:it makes sense

[sTalking_Goat]

but then I don't necessarily trust the supercomputing expertise of a bunch of Lunix fanboys "administering a network" in their parents' basement either.

Aren't the people running these top 6 clusters part of various Universities and Research Labs?

I would think they would know if their cluster is doing what they want it to do. And if they have something that does the job, why should they go out and hand Cray several million dollars to build them something that Cray says is better?

Re:it makes sense

[Marxist Hacker 42]

And there's absolutely no reason why you can't put a bunch of FFT chips on a specialized PCI card, mass produce it, and get a bunch of networked FFT enhanced supercomputers.

Re:it makes sense

[ch-chuck]

because, sometimes you need two strong oxen instead of 10240 chicken.

Re:it makes sense

[Rosco P. Coltrane]

The American supercomputer industry has gone from innovator to an assembly operation.

Yes and so what?

The American space industry is taking the same path with AirShipOne. Don't you think it's good?

If computers hadn't commoditized, you'd still be posting your comment from a terminal connected to a mainframe, while a line of people are waiting in line behind you at the library. But instead, you have a great computer at home, just for you, because you can afford it. Heck, if you wanted, you could afford a small Beowulf cluster too. Could you afford a Cray?

Oh yes and the other thing is, clustering is a technique that has progressed a lot thanks to commoditized hardware (yes, it's harder than "copy that little snippet of code 1000 times and collect the results).

Re:it makes sense

[DAldredge]

That is why you write an app that looks pretty and shows the state of the system on several large flat panel monitors that are flush with the wall. Just make sure you program in several 'test' modes that really look cool so you can 'test' while the VIP's are around.

Not that I have ever done anything like that :->

Re:it makes sense

[Rei]

I'll agree that it sounds like an industry heavy panel; furthermore, it sounds like a supercomputer-industry-heavy panel. Furthermore, it sounds like a *nostalgic* supercomputer industry heavy panel. What else could explain lines such as " In contrast, classic supercomputers that rely on very fast, specially designed vector processors "could be programmed in Fortran," Scarafino said. "They could be programmed in a language that mere mortals . . . could program in." ?

Yes, there are tasks where supercomputers are needed. Most tasks are not among these. If there is a single parallelizable task in a CPU-intensive process, odds are that a cluster is your best bet. For example, even if your core algorithm requires intensive memory locking and must be done in a completely serial manner, if you are going to be running that core algorithm over a range of possible inputs, a cluster will probably be your best choice.

Re:it makes sense

[aldoman]

Yes, but (for example) lets say a specialized CPU could do FFTs 10x faster per clock cycle than your Opertons - but usually you are looking at paying so much more, that buying 20x more CPUs (even if they are Opertons) and using them is cheaper, and you get more power.

Not only that, you can still use that cluster in the future for other jobs and not have to reinvest millions of dollars again.

Re:it makes sense

[mangu]

problems where working on one section of the dataset affects large amounts of data in other sections

Can you give a specific example? When I think about it, the most CPU-heavy problems that occur to me are highly parallelizable. Things like solving partial differential equations, for instance, which means physical simulations. Or, in the case of research centers, I suppose neural networks might be one heavy user.

I can't think of any super-computer application that doesn't involve lots of data being processed by the same (relatively simple) algorithm.

Re:it makes sense

[DAldredge]

Perhaps true, but if the need is so great why aren't the systems being bought?

Re:it makes sense

[Anonymous Coward]

Partial differential equations are NOT necessarily highly parallelisable. Linear ones, maybe. But the interesting ones that simulate Da Bombs are all nonlinear elliptic P.D.E.s.

Re:it makes sense

[ikeleib]

While you may doubt the real difference between a traditional supercomputer and a cluster, let me assure you that it is not industry bull. A cluster can efficiently deal with tasks that are easily parallelizable and don't require much communication between the nodes. There are problems that are not easily parallelizable and/or require lots of communication between nodes. A cluster will not do in this case. There is more to the super in supercomputer than just MFLOPs.

In previous years, clusters weren't nearly as cost effective, and it kept many commercial customers buying supercomputers. However, the current market presents very few commercial buyers. The majority buyers are research institutions and government. From a commercial perspective, the death of this expertise can be problematic. However, the US government has a national security interest in keeping the supercomputer industry alive. The government uses these computers for classified cryptographic and weapons work. It is widely believed that in the past, the NSA gave contracts out to Cray simply to keep them in business and perserve a viable American supercomputer industry.

Although one could simply argue that if that is the case, the government should undertake this effort on their own. However, it is unlikely to be cost effective. The costs involved may simply be too large for the government to handle alone.

Re:it makes sense

[Marxist Hacker 42]

Then trust the fact that not all problems are easily attacked from a parallel perspective. This means problems where working on one section of the dataset affects large amounts of data in other sections.

Possible bad example- all I'd do with this is create a virtual shared memory store on a gigabit network and use a reasonable data engine such as MySQL in SQL Server Mode to create a shared memory space. To make it really handy, put the whole thing on a terabyte ramdisk with battery backup.

The vector processors are harder, though- because it's not off the shelf parts.

Re:it makes sense

[grawk]

As someone who works for a supercomping center, I can say that some things work VERY well on cheap unix based clusters. I am the primary admin on a 5 TFLOP cluster. We've also got a Cray X1, and while it's only 2.6 TFLOPs, it will eat my IBM's lunch when it comes to some specificly tuned tasks. Much in the same way that we can outperform mac clusters that have significantly higher floating point performance because of the speeds of the interconnects. Supercomputing is about a LOT more than just raw CPU power.

Re:it makes sense

[fitten]

Clusters (talking about "Beowulf" types and such using stuff like Ethernet, no matter the speed) do OK at coarse grained problems. That is, problems where communication is seldom and typically when it happens, is a chunk of data like a WU or something where the consuming node can go off and compute on it a while then submit some result or intermediate result. All of the "Distributed" projects like SETI are of this type and are even a subclass of coarse grained problems called Embarassingly Parallel.

Fine grained problems typically require much more inter-node communication and are also typically much more sensitive to latency. Fine grained problems typically show lots of communication and (not always, but typically) smaller amounts of data transferred. For example, take a problem where each node will sum up an array of 16 integers passed to it and pass this (partial) sum back to the sender for it to use in its calculations. On 100Mb Ethernet, the latency of the transfer of that 16 bytes is *huge* compared to the full transfer time (overhead of packet size and slowness of the actual transfer) and the computational time for adding 16 integers together. A "Beowulf" cluster would be ill suited to this type of problem simply because Ethernet at those scales is extremely inefficient. However, there are architectures (some SuperComputers) that could do this algorithm quite well.

Another example of this is to think about a kernel (maybe a Linux one) that is multiprocessor. How well would a shared image Linux kernel perform if it were using Ethernet as its interconnect between nodes as opposed to shared memory on a dual/multi CPU motherboard? This is one reason why SGI, Cray, Sun, IBM, and others have developed NUMA architectures - basically the bus between processors is really a switched network that is extremely high bandwidth (GB/s) and extremely low latency (measured in microseconds at worst) in order to run single system images - in order to scale the number of CPUs up in a box that can work on the same problem in parallel.

So, parallel programs are written with a mind to the algorithms being used to solve the problem and the hardware on which it will run. For example, there are a number of algorithms for solving systems of equations in parallel. Different algorithms may be more latency critical than others (some may not be penalized by high latency networks such as Ethernet as others which require more communications). Not all problems have multiple "good" algorithmic solutions where the programmer can pick based on the hardware available. Some problems do not have "good" algorithmic solutions that are latency tollerant at all. Those algorithms/problems need low latency networks to solve and typically, the only solutions are super computers and very "special"/exotic networks (read: expensive) are the ones that supply very low latency interconnects - Myrinet, Giganet, RACE, etc.

Re:it makes sense

[Anonymous Coward]

I think you are missing at least some of the point, without supercomputer development around the area of transputers (google for it) we would never have had Intel MMX or AMD 3D Now! in our chips. Supercomputing invents new stuff at every turn of the handle and much of that genuinely new technology ends up inside mundane, commodity hardware like your Beo cluster of x86 boxes.

I see Japan as the next most likely source for this stuff, in the same way that they lead 3G mobile technology after missing out on the 2G-fest (see Qualcom).

Re:it makes sense

[Anonymous Coward]

this sounds like just another business whining. if it's so critical the gov. can give their tech's a job and they won't loose the expertise. it's naive to think if these companys go under no else will be able to build supercomputers.

Re:it makes sense

[Anonymous Coward]

I don't necessarily trust the supercomputing expertise of a bunch of Lunix fanboys "administering a network" in their parents' basement either.

With just a little more intelligence, you might be able to make a point without resorting to elitist putdowns!

I happen to be one of those "Lunix fanboys" administering a network in my own basement, thank you very much! I was also helping administer Novell networks in the early '90's, currently administer a company's network for a living and the cluster in my basement is to experience what classes of problems can and cannot be solved with a cluster.

My inclination is to let the market sort itself out...

I agree. Those on the top 500 list using Beowulf clusters for scientific applications are not doing so because clusters do not meet their needs. If Cray and others are going belly-up, it has to do with not providing what customers need at prices they are willing to pay.

Re:it makes sense

[Jeremy Erwin]

when you can build a top 5 supercomputer for under 6 million dollars, using off the shelf parts. Why spend the hundreds of millions of dollars?

Why waste time and energy trying to cobble something out of off the shelf parts, when you could be designing a system that could challenge the three-year-old Earth Simulator?

Re:it makes sense

[bit01]

I don't trust such a panel. I've seen many vector supercomputer installations that had very dubious justification. The closest to a real justification I've seen is single-threaded legacy applications that would be too expensive to reengineer. And that's a pretty bogus reason given the cost of supercomputers.

I can't think of a single, large scale software problem that doesn't have a straightfoward parallel solution. Can anybody suggest one?

---

It's wrong that an intellectual property creator should not be rewarded for their work.
It's equally wrong that an IP creator should be rewarded too many times for the one piece of work, for exactly the same reasons.
Reform IP law and stop the M$/RIAA abuse.

Re:it makes sense

[JonGretar]

But a lot more comes in. Reliability. The backbone speed. The coolness of owning a Cray that could kill you if you press the wrong button.

But who do you think the Army trusts more to build the most powerful machine in the world and shut up about it? IBM or the New York Linux User Group.

Things have to prove themselves for much longer than the cluster has done to be valid for military use.

Re:it makes sense

[fitten]

...and that is one of the problems... The benchmark used to make the Top500 list is an extremely simple one that doesn't reflect some of the harder problems to solve (it actually reflects a rather easy one to solve and one that is very latency tollerant). Those harder problems to solve tend to require at least as performant interconnect as the CPUs themselves.

Ethernet is a very high latency interconnect and is poorly suited to many types of fine grained parallelism. If Ethernet was the end-all, be-all of high performance computing, why did AMD use HyperTransport to interconnect Opterons instead of Ethernet? (or any other high speed NUMA switched network to interconnect processors by any other vendor)

Re:it makes sense

[Jeremy Erwin]

Yeah, yeah, you could spend your time hand optimizing loops-- and profiling, over and over. Or you cold spend your time actually running the simulation, and optimizing the underlying model.

Re:it makes sense

[thue]

Take for example Deep Crack (luminaries, remember that one?). Perfect example of specialized hardware for a single job.

Wikipedia has an article: http://en.wikipedia.org/wiki/Deep_Crack

Re:it makes sense

[Tony-A]

When I think about it, the most CPU-heavy problems that occur to me are highly parallelizable.

CPU-heavy implies that internal-internal interactions dominate internal-external interactions. This is the opposite of mainframe processing.
CPU-heavy implies that something internal must be highly parallelizable. It does not imply exactly what or how. This depends on the nature of the required internal-internal interactions.

The super-computer applications which are feasible (conceivable?) are determined by the tools and paradigms which are available. If these are too limited (are there is no way to know a priori) then you are cut off from much which should be feasible.

Re:it makes sense

[dingbatdr]

You make a mistake when you say "just rack them up." Modern supercomputers are (ususally) made of commodity chips connected by extremely fast networks. Solving large systems of equations, especially elliptic or parabolic systems, requires large amounts of communication. You can debug codes running smaller problems on a cluster. To do large production runs, real supercomputers are necessary.

Nobody in their right mind would use a supercomputer where it is not needed. They are difficult to get time on and painful to use.

dtg

Re:it makes sense

[mikefe]

If it's going to save you millions I think you can afford a few extra people optimizing loops.

Re:it makes sense

[Anonymous Coward]

Horse crap.

Re:it makes sense

[Anonymous Coward]

Name a top 5 supercomputer that fits your description. http://www.top500.org/list/2004/06/

Re:it makes sense

[chris_mahan]

Yeah, especially if the extra people are LFBs (Lunix fanboys) getting paid in soda and pizza (hold the vegis)

Re:it makes sense

[jafac]

If the world were full of nothing but 5-star restaurants, serving $100 dinners of steak and lobster, many people would just plain starve to death.

However, say, in this hypothetical world, that one day, a "McDonalds" arose to challenge the high-end dining monopoly. Everyone could now afford a meal. If someone had a large family, they could feed them for far less money. The 5-star restaurants would take a HUGE hit on marketshare. Would there still be 5-star restaurants at the end of that day? Certainly. There will always be people willing to pay $100 to not have to eat a mass-produced burger.

Now imagine a few days later, and all the 5-stars have gone out of business, simply because now that everyone's eating at "McDonalds" - the market for the 5-stars is far less than the basic requirements of infrastructure to support the 5-star restaurant industry costs to maintain. It collapses, and we're in a world with one choice. The Crappy Meal.

There would be some people who still crave a Steak. There would even be some people who could afford a steak. But there would be nobody left who could afford to make, and sell a steak.

Market Fundamentalists would say that this is a good thing. Even a NECESSARY thing. If there's insufficient market to support a product or service, such a product or service does not "merit" existence.

Until someone in another country who craved steak enough, paid for it on his Oil Revenues, grew strong on the real red meat, came and kicked the soyburger-eating asses of the narrow-minded fools who let themselves lose leadership in yet another security-critical industry.

Re:it makes sense

[Jeremy Erwin]

A few extra people? Do these few extra people get their own supercomputers?

The problem with clusters is that they don't scale well in all cases. Programming tricks may help, but profiling and testing on a sub cluster may not reveal bottlenecks in the full cluster.

Re:it makes sense

[Lumpy]

Ok you want an answer that will make sense??

NOAA - the highbrow science guys trying to predict weather patterns chose a linux cluster over a supercomputer.

NASA - Fricking rocket scientists, chose a linux cluster over a supercomputer.

MIT - trying to model fluid dynamics
Berkely - Messing with Astronomical calculations.

I think these people are the REAL users of massively powerful computing and they certianly see a performance acceptability in a cluster.

I'll listen to scientists before a bunch of salesmen any day.

If I can get the same or better computing performance from 1/10th the budget.... It's stupid to spend money for the sake of spending.

Re:it makes sense

[Tony-A]

As an example to see how this can be so, consider an old mainframe with a bunch of old green-eyed monsters. The screens have enough intelligence so that the entire screen can be filled in and submitted at one shot. Take the same system and have the mainframe react to each keypress/release and the mainframe will be crippled.
Now have each users keypress/release affect all other users concurrently.
Now scale it bigger.

Re:it makes sense

[rgmoore]

The American supercomputer industry has gone from innovator to an assembly operation.

So what? The supercomputer industry doesn't exist out of an abstract idea that supercomputers are a good thing. It exists because there's a market for supercomputers. If there is, great; the supercomputer companies should be able to keep in business- and keep innovating- by selling to it. If there isn't, then there's no point in keeping the business afloat artificially out of an abstract idea of protecting American innovation. Companies with a market for their products live. Companies without a market for their product die. That's the way that a free market works.

Re:it makes sense

[The Lynxpro]

"What else could explain lines such as " In contrast, classic supercomputers that rely on very fast, specially designed vector processors "could be programmed in Fortran," Scarafino said. "They could be programmed in a language that mere mortals . . . could program in." ?"

Sounds like they need an Atomic Vector Plotter. That was always a pain to acquire on the Vogon ship. :)

Re:it makes sense

[sharkey]

I don't necessarily trust the supercomputing expertise of a bunch of Lunix fanboys

Well, if you don't like Lunix, give Linux or BSD a try.

Re:it makes sense

[Minna Kirai]

virtual shared memory store on a gigabit network

Gigabit transmission is already fatally slow compared to the data rate between CPUs of an SMP box. (If you had meant gigabit ethernet, it'd be even worse)

Re:it makes sense

[rgmoore]

But who do you think the Army trusts more to build the most powerful machine in the world and shut up about it? IBM or the New York Linux User Group.

They trust IBM, of course. But do bear in mind that what they get is (gasp) a cluster of IBM macines running Linux, not an exotic vector computer.

Re:it makes sense

[agraupe]

Speaking of linux fanboys adminning home networks: why not? For a 15 y/o (which I am) it is a great first step to becoming a full-fledged admin. I am considering taking the LPI certification at the moment. It is a learning experience. IMO one of the great strengths of Linux/BSD is that it brings UNIX to a more accessible level. It will be an everyday interaction for us linux fanboys, and it will help take away the black hole of usability that has plagued it in the marketplace.

Re:it makes sense

[jcr]

if supercomputer makers go under, they won't necessarily reappear the moment they're needed.

Umm.. Nothing appears the moment it's needed. That need is called "demand", and it takes a while to meet it.

Frankly, I'm not at all worried about this.

-jcr

Re:it makes sense

[Rostin]

Computational fluid dynamics. The most common manifestation of this would be weather models, although it can also be used to model petroleum resevoirs, airplane wings, and the list goes on and on. The system is broken up into little chunks. What's going on in each little chunk depends on what goes on in all the the other chunks. And we are talking about millions and millions of chunks.

Re:it makes sense

[Anonymous Coward]

General Relativity is nonlinear; you cannot use superposition to calculate the field of each object at a point and simply add up the results like you can with Newtonian gravity.

Re:it makes sense

[rspress]

Exactly! Why spend millions and millions of dollars developing a processor that will be obsolete soon after it is created and has such a narrow use. If anyone wanted to spend 20 to 30 million of a bunch of clustered Xserves they would have the fastest supercomputer in the world.

And yes, the iPod is God!

Re:it makes sense

[Otter]

Sorry, I didn't mean to be putting you down -- obviously, what you're doing is great. I was just poking some fun at the people with your level of expertise (if that) who decide to hold forth on the merits of Linux clusters in supercomputing. Not that I have any pretense of being versed in the subject either...

Re:it makes sense

[vivian]

IANAM but what about say, calculating the Fibonacci ratio to two billion significant figures.
Each term depends on the sum of the previous two terms, and the ratio is the ratio between successive terms.

Re:it makes sense

[afidel]

In the worst case (very large messages) the latency between nodes in the VT cluster is about 3X worse than the latency in this SGI behemoth. Unless you are solely limited by message latency then you are usually better off with the VT style cluster since it is about half as much per flop. Btw basically ALL of the money going into researching problems that require traditional supercomputers is being done by governments, so all of the money is already coming from government work.

Re:it makes sense

[randomwalker]

Even if the application can not be parallelized, my experience working in these environments is that most times the same group needs to run the same application many times with different parameters. Putting the application on slower machines, but running different cases simultaneously also favors clusters.

Its very rare the high margin items ever maintain or increase their high margins. Anyone writing/using large resource intensive applications needs to plan for these type of things. crays are wonderful machines but if there is a better solution people should go with it.

Its also interesting to note that mainframes are still high margin and profitable for IBM. All the comptetion went away and lots of legacy code out there.

Re:it makes sense

[CoolGuySteve]

A lot of posters in this article decry commodity hardware for sucking but they're not considering the long term. The tasks commonly performed by commodity hardware are starting to be more and more suited to scientific applications. We can already see how vector optimizations like Altivec and SIMD have worked their way into every desktop chip around.

As 3D rendering and sophisticated media codecs are becoming the primary reasons for upgrading a home PC, the front side bus and CPU (especially the CPU) have gained a disproportianate speed advantage compared to the rest of the PC. And when you start throwing in high speed network compenents like blade interconnects and 10Gig E, you can create an extremely powerful, scalable computer. Vendors like IBM love this because it means they can sell one design to everyone, just changing the number and specs of the nodes to fit their customer.

Speaking as someone who worked support on a large academic cluster where thriftiness is king, not only is the hardware cheaper, but it's mundane specs mean it's less likely to fail, when it does, the nodes are replaceable like lightbulbs, and you can run down to the local computer store in the event of a faulty IDE drive or memory stick. The support costs and reliablility are far superior to typical high performance solutions because support can mostly be done in house by existing staff.

And that ComputerWorld article is mostly bunk, OpenMP supports Fortran. The people who are writing large scale simulations in the first place are not only technical but also extremely bright. While it is a hassle, adding networking to the already heavy code optimization that they do is not that big of a deal.

Though there are problems that require a shared memory system, $2 million can buy so many more networked Xeons than shared memory SGI processors that the scientists in question should really consider their needs. While it may take much longer, their problem set can be much, much larger since they'll have more collective RAM and HD space. The system can also easily be partitioned, rebuilt, and shared in such a way that it's always in use by one or several people. In many cases, the tradeoff is worth it. And even then, shared memory systems will probably still have a lower price/performance ratio compared to clustering a smaller number of high memory database servers. Say for example, a group of Itaniums with 64Gig of ram each vs their SGI Origin equivalent in terms of memory footprint.

I think as time goes on, interconnect speed will increase to the point where clusters are very similar to traditional supercomputers. We can already see baby steps in this direction with blade equipment and various motherboards with gigabit NICs built into their own bus to avoid choking in PCI land. Infiniband and 10G over copper are the first major steps but switching and motherboards still have to catch up it seems. Instead of whining, the typical supercomputing vendors should be looking into merging standardizing their designs with those of clustered systems. Of course, I could see how the entrenched traditional vendors would want to legislate as much as possible to avoid having to compete with dozens of smaller engineering companies with less overhead and better ideas.

Now that I think of it, a lot of the posts I've read here are probably skeptical of massively parallel machines because they don't realize that the performance is all in the network. I've seen two clusters first hand with an identical number of identical nodes. One was being used as a compute farm where several hundred dual Xeons were attached to a rediculously large and overly expensive IBM foundry switch. The other was being used as a massively parrallel simulation machine, with dual gigabit nics attached to a bunch of 'cheap' 24 port switches in a 2D mesh. The second design cost much less because it wasn't built by IBM while still being about twice as fast.

Re:it makes sense

[civilizedINTENSITY]

So for tasks where supercomputers are needed, that don't parallelize well...if you intend to run over a range of inputs, you suggest a "cluster" of supercomputers? Lets buy, say, 32 Cray X1s?

Re:it makes sense

[clem.dickey]

if supercomputer makers go under, they won't necessarily reappear the moment they're needed

Which is why we need Congressional legislation to form the National Supercomputing Corporation. Call it "Amcompute" for short. They can even use the old "pointless arrow" now that Amtrak has moved on to "three sheets to the wind".

Re:it makes sense

[Fulcrum of Evil]

Possible bad example- all I'd do with this is create a virtual shared memory store on a gigabit network and use a reasonable data engine such as MySQL in SQL Server Mode to create a shared memory space. To make it really handy, put the whole thing on a terabyte ramdisk with battery backup.

You haven't addressed anything. You're still limited to Gbit ethernet, which has crappy latency compared to a proper interconnect. Look for something on the order of 1-10Gbyte channels that scale to over 256 nodes.

Re:it makes sense

[FuzzyBad-Mofo]

when we make clients visit the premises

Interesting phrasing.. do you work for the Dept. of Corrections, perhaps? :)

Re:it makes sense

[CoreyG]

a plural noun that is actually SUPPOSED to end in -en? holy shit!

Re:it makes sense

[charnov]

You do realise that many of the current Crays are made from proprietary massively parallel AMD Opteron clusters, right?

BTW, Cray got its ass handed to it when Thinking Machines,et al. switched from large scale esoteric vector processor based systems to massively parallel ones.

The next big thing will be either analog elliptical, quantum, or multi-state based. Can't wait.

Re:it makes sense

[The Conductor]

No, it really is true. With non-linear PDE's the parallelization falls apart. You say, "ok smarty-pants, when modeling the copressible fluid around a car (ie, the air), how much can the flow around the bumper depend on the flow arond the mirror?" The answer is not much, but when you discretize the problem into a fine mesh, there are a gazillion boundary points between the mirror region and the bumper region, so the communication links between parallel processors bog down. It is faster to solve all the points with the same, really fast, processor.

Re:it makes sense

[Jason Pollock]

And yet, I'm told that WRF (the new weather model software) and MM5 both run happily on a beowulf clusters. :)

http://www.wrf-model.org/

Google for WRF beowulf
http://www.aspsys.com/clusters/beowulf/di sciplines /default.aspx/weather_modeling.aspx

Perhaps the communication is limited only to adjacent cells?

Jason Pollock

Re:it makes sense

[DaKrzyGuy]

Actually it is more likely a cluster of IBM PPC machines running AIX

Re:it makes sense

[kscguru]

Or you can sit down and throw some linear algebra at the problem, and come up with a formula for successive fibanacci numbers that ISN'T dependent on the previous numbers. Hint: it's a geometric series, with the slight catch that you have to use complex numbers (embedded in 2x2 matrices) instead of scalars, and thus is slightly more difficult to solve (it takes about a chalkboard of math because it can only be solved in the eigenbasis). A Fibonacci problem can be solved with a better algorithm :-).

The difficult large-scale problems have "chunky" parallel solutions - each chunk is parallel, but some chunks take longer than others (it's difficult to know how long beforehand) and the overhead of scheduling and balancing all those chunks begins to dominate the actual computation. Combining large, arbitrary, sparse matricies would do it - some multiplications will result in very little work, while some will have lots of collisions and require a lot of work.

Re:it makes sense

[PingPongBoy]

The point may be that supercomputers are merely rising to new depths. While 6 mil may buy more computing than ever before the top 5 is getting more and more crowded with systems of similar power. Surely something should have beaten the Earth Simulator by now, especially now that we see so many teams putting together clusters of PC class machines.

All the same people don't want to invest $50 million only to be overtaken in 3 or 4 years by a $5 million system. It's more economical to upgrade as new processors and technology become available. The incentive to spend big money depends on how soon results are desired. One may argue that supercomputers are used by people in a real hurry and thus it is simply paradoxical for these people to even think about saving money. On the other hand, if one plans to build a $50 million system, it will take quite some time to thrash out the design and in a few years a lower cost but simpler system can be built with the same speed.

Another incentive for expensive supercomputers is the proof of concept. Today's cheapo supercomputers are made possible by experience with really expensive initiatives. Such high end systems should be in the top 5.

Re:it makes sense

[HiThere]

You don't need to trust the claims. You could, instead, try to figure out how to do whatever task you are currently programming in a thousand (or so) stream process.

Some processes are inherently serial.

OTOH, an awful lot of them can be approached indirectly through parallelism. But the parallel approach is often thousands of times as inefficient. (Still, the brain has only a few areas that are truly serial.)

Re:it makes sense

[Anonymous Coward]

Because, when it comes down to it, the bean counters look at the cost of the two oxen vs. the cost of the flock of chickens and say: "Hey. Those chickens can do the job. Use them."

Re:it makes sense

[Anonymous Coward]

Uh, Cray YMP and XMP aer massively parallell computers - if they were no good for the calculaion of the weather forecast, the UK Met Office would not hve used them, correct...?

Re:it makes sense

[weiyuent]

Yeah but what we really want to know is how many VAXen are needed.

Re:it makes sense

[Grab]

But this *does* parallel up well. You work out for the current state of each little chunk relative to the others, rinse and repeat. Each processor handles a large number of chunks, and they then only have to communicate about the states of the chunks on the boundaries.

In physical problems like this, you *do* literally have a lot of little chunks with propagation delays between them, so the solver maps pretty accurately onto the problem. The only issue is getting the chunks small enough to be an accurate representation. The way air behaves over a city is different from elsewhere due to light reflection, heat emission from buildings, the thermal mass of the concrete, different levels of pollutants, etc.

Grab.

Re:it makes sense

[chthon]

It is all about making money.

A long time ago I read an article (I think it was in an IEEE pub.) about supercomputers, and their general conclusion about the current ones on the market then, was that most speed ratings giving where burst ratings (eg. Cray-1, was 80 Mflops burst), but most could only give only about 10% sustained performance of their burst performance.

The talk of the panel about parallellisation is the same for normal supercomputers. If you want high sustained performance, then you need an algorithm that is highly vectorisable, and if you look at how such a computing unit is built, it is also a highly parallel system.

What's the death of your high speed ? Many decision making tasks. It always comes down to the same, or you have a high speed integer system which is good at decision making, or you have one that can run continuus floating-point computations, but not both together.

Re:it makes sense

[bolind]

And while we're at it:

"Memory is like orgasms, it's a lot better when you don't have to fake it."

--Seymour Cray on virtual memory

Re:it makes sense

[trentblase]

Umm.. Some stuff already exists the moment it's needed. That stuff is called "supply" and is usually used to fill "demand". If I want a new laptop I can go to the store and buy one that they have on hand. But what if nobody makes laptops anymore? That's what the OP was getting at.

Re:it makes sense

[moodboom]

Or maybe the paradigm shift is toward more innovative "assembly operation". Or the algorithms that let you distribute the crunching better.

Re:it makes sense

[trixillion]

a chalkboard full of math????? There are three general approaches to solve the relevent difference equation. One can assume the form of the solution, and the solve a polynomial for the necessary constants (anologous to how most people solve ODE's.) One can apply the Z transform which is the discrete analogue of the Laplacian transform (this makes handling the boundary conditions particularly easy.) Finally there is the third method which is to rewrite the second-order difference equation as a system of first-order difference equations. The latter is a gross overkill, it is a much more powerful technique better left to much more difficult problems (e.g. coupled difference equations); in this case it reduces to the first method. It is unfortunate that your professor took the long route to solve this very easy problem, as it stunted your understanding of how best to solve difference equations.

Re:it makes sense

[gurps_npc]

He's figuring out some interesting new networking ides that will allow the cluster to become a Top 1 computer instead of a top 5.

Your arguement fails hideously.

Basically you are stating something along the line of: "No one is researching nuclear powered subway systems because so many people are using those cheap Automobiles."

And denigrating the "Automobile" research as "lesser advancements".

We have proven that the large scale stuff is NOT a good idea for most things. Yes we should do a little more research in them, but it is not a high priority, it will NEVER help us as much as the quick gains we can make by furthering the lesser advancements that you so dislike.

Eventaully we will make all the "easy" discoveries, and be forced to do a huge Paradigm shift. But I will bet you one thing. The new Paradigm will be a NEWParadigm, and therefore will NOT involve classic supercomputers.

Expected fallout from the Beowulf takeover

[beee]

This is an expected and predicted fallout from the recent rise in popularity of beowulf clusters. Slowly but surely managers are realizing, yes, it is possible to have a supercomputer on mass-market hardware, running a free OS.

Don't see this as bad news... it's a sign that we're winning.

Re:Expected fallout from the Beowulf takeover

[Cromac]

I agree, it's definately good news. Our tax dollars can be much better spent on other things than buying a Cray when the same level of performance can be had for much less by building a cluster.

Re:Expected fallout from the Beowulf takeover

[techsoldaten]

Dunno about that so much as we are supporting the status quo.

One thing about technology is that each generation feeds off the last. If we get into a cycle where our expections for hardware empasize quantity rather than superiority how will we every achieve the ultimate ends of computing: to... uh... accomplish, uh, something... um...

Maybe it's all headed somewhere.

M

Re:Expected fallout from the Beowulf takeover

[captain_craptacular]

If you RTFA you'll find out that "the same level of performance" CAN NOT be had by building a cluster. Clusters only help when a problem is easily paralelized, meaning it can be broken into many small parts which can easily be handled by their own low power processor. Other problems (many modeling applications) do not fit this description and require specialized hardware which can take a large, complex problem and deal with it in one massive chunk... A cluster will choke on these problems just like your workstation with the same processor would.

Re:Expected fallout from the Beowulf takeover

[Exiler]

Quite frankly, I prefer quanity. Lots and lots of shiney, blinkey things. I don't care how well the work, just so long as there are GOBS of em!

Re:Expected fallout from the Beowulf takeover

[Anonymous Coward]

In cases where either there is a prohibitive amount of communication between processors or the data is not parallelizable, clusters will not work. Unfortunately, many of these areas are not of immediate interest to industry. So, unless the federal government supports these companies, research that can *not* be accomplished with cluster computers will grind to a halt.

Re:Expected fallout from the Beowulf takeover

[susano_otter]

Since clusters are so much cheaper than mainframes, it's often the case that clusters still offer better performance for the money spent than a mainframe would, even if the cluster isn't really optimized the way the mainframe is, for the task at hand.

That being the case, wouldn't it make more sense to invest heavily in R&D to solve the cluster's problems and remove its limitations, than to invest heavily in R&D into next-gen mainframes?

Re:Expected fallout from the Beowulf takeover

[weiyuent]

This is an expected and predicted fallout from the recent rise in popularity of beowulf clusters. Slowly but surely managers are realizing, yes, it is possible to have a supercomputer on mass-market hardware, running a free OS.

Don't see this as bad news... it's a sign that we're winning.

Not necessarily. There are plenty of computational problems that, so far, do not lend themselves well to parallelized solutons.

The point of this post and the linked article is that the hype about Beowulf and similar cheap, cluster-based technology has caused mainstream technologists to overlook the need for traditional supercomputers like the Crays.

That indeed is bad news and a sign that computer science in general, while not necessarily losing, is not winning.

Re:Expected fallout from the Beowulf takeover

[Aadain2001]

Then let the people who have those problems pay for the hardware to solve those problems. For the people who are doing parallised work, clusters make perfect sense. I think the big guys (Cray) are just unhappy to see that most of their business is going away because their hardware isn't needed as much since people are figuring out how to use clusters to get the jobs done.

Re:Expected fallout from the Beowulf takeover

[janbjurstrom]

Perhaps one future problem could be, that the problem domains people will work in will change?

Complex stuff where mainframes would carry the research forward (but clusters would not), that - with only clusters available - won't be feasible and not get the attention it might deserve. Just an uninformed thought..

Re:Expected fallout from the Beowulf takeover

[eXtro]

If you spend the R&D dollars to solve the clusters problems you end up with architectures like sgi or cray.

Re:Expected fallout from the Beowulf takeover

[Tony-A]

wouldn't it make more sense to invest heavily in R&D to solve the cluster's problems and remove its limitations

Yes, like finding some new 2-digit numbers that come after 99.
The limitations of a cluster are because it is a cluster.

Of the problems that should be worth solving,
a few will be embarassingly parallel
many will be extremely parallel
many many will be moderately parallel
many many many will be somewhat parallel
too many will be highly convoluted.

Re:Expected fallout from the Beowulf takeover

[Veridium]

No kidding. What happened to the free market? I'm so sick of the anti-OSS crowd accusing us of socialism and the like. Here's a for profit company, that works for profit, trying to get itself access to a nice government tit. If it isn't profitable, perhaps this for-profit company ought to form a non-profit and reveal it's true nature as a model that is not viable in the free market.

Yeah. Who are the socialists again?

Re:Expected fallout from the Beowulf takeover

[demachina]

You pretty much hit the nail on the head.

The problem with the big investments in supercomputing by the U.S. government in the last decade is that they've been time after time to put one huge white elephant system after another in one national lab after another. Some problems:

- They take a long time to build from first RFP, to contract award to first delivery till they are fully deployed. By the time they are done they are usually starting to look long of tooth and the lab starts the whole process all over again.

- I don't know what their exact work load is but WAY to much money has been spent in the U.S. on simulating nuclear bombs. That is a computing work load with NO ECONOMIC benefit. It has a security benefit only if somebody decides to use one of the bloody things someday and that wont be a good day. I wager the Bush administration is using them to design new tactical nukes to bust bunkers and caves but if the U.S. starts using those as a matter of routine that will also be a bad day. Way to many of them are going in to the NSA to spy on communication. It occassionally has security benefit but it ain't worth it when traded against the invastion of privacy of everyone.

- The classification of these systems no doubt discourages their application for anything economically useful

- The same cast of characters line up to bid on them. Its just high tech pork for IBM, HP, Intel, SGI etc. They get to build leading edge one off systems that keep architects and chip designers entertained but again they are extremely poor in economic value.

If super computing is going to survive they need to do the same thing 3D graphics did, it used to be overpriced too. They need to design single chips that have:

- Really fast vector units and good scalar units
- Huge memory bandwidth (with supporting cache and memory systems)

They need to drive up volume by making these chips as accessible as possible to auto manufacturers, oil and gas exploration, chemistry etc. When they get the volume up, reliability up, ease of use up and price down they might sustain a viable market though I'm still not sure there really is one. Someone also needs to fund applications and visualization that turn them in to breakthrough technology that leads to huge advances in a range of technology fields.

Of course this is kind of SGI's business model and they, for whatever reason, can't make it work either.

I could see the U.S. government, in concert with private industry, putting seed money in to developing:

- very fast vector units in commodity processors
- Adapting GPU's to supercomputing apps
- Very large/fast cache/memory and driving down cost so everyone benefits
- Very fast I/O and driving down cost so everyone benefits
- Breakthrough applications applying the above

But if instead:

- government just keeps putting out contracts for another teraflop machine to be squandered in a national lab simulating nuclear bombs screw the whole damn concept

- If IBM, Cray, SGI, HP etc. are just looking for government handouts to help them build toys that are cool but no one else really wants to pay for, again screw the whole concept.

The cool thing about Japan's premier supercomputing facility is its dedicated to climate and weather simulation so it has a purpose.

If you want to have a viable supercomputing industry you need to identify applications that make it worth while and figure out how to best develop hardware to meet a need. The apps most likely to matter are ones that lead to breakthroughs in their respective industries especially basic science and R&D but the U.S. is pathetic at basic research these days which is another reason there is no demand for supercomputers. They don't factor in well on the quarterly profits.

Just saying you need to keep handing out hundreds of millions of dollars to the same old crowd to keep doing what we've always done, whether it works or not, is a pretty dubious way to spend money (though so was squandering $200 billion on Iraq).

Re:Expected fallout from the Beowulf takeover

[forkazoo]

A cluster has poor latency and bandwidth compared to moving data register to register on a CPU. Big fast CPU's have lots of local bandwidth. Clusters have less. How in frack's sake do you expect to "Fix" that? It is the inherent distinction of a cluster. Separate boxes, with IO connecting them can never be faster at comms than the CPU itself. a 486 has more on chip bandwidth and better latency than Gig-ethernet. Sure, it only has 8 registers... Not a huge range of problems that it can solve entirely on-chip... :)

Re:Expected fallout from the Beowulf takeover

[CoolGuySteve]

Easy, you can solve bigger problems (as in process/create more data) for the same amount of money. It may take forever because of the latency, but eventually you reach a point where the scale of the calculation far outweighs that factor.

Here's an example, I have one Pentium 3 with 256 megabytes of ram, it's pretty impressive and can solve 256 megabyte problem sets extremely fast. For the same amount of money, let's say $2500, I can buy 20 Pentium 1's with 48MB of ram at $125 a pop. Now I can solve a 960MB problem for the same price at a much slower speed.

Re:Expected fallout from the Beowulf takeover

[Anonymous Coward]

You're missing the point. The people who have the problems will pay for the solutions. The question is will the pay Cray (American) or NEC (Japanese)?

(Caveat: Don't assume I agree with or care about the issue. I'm a Canadian and we make jet engines, guitars, space arms, hi-fi speakers and Alexander Keith's India Pale Ale but no supercomputers. I'm just keeping conversation on-issue.)

Re:Expected fallout from the Beowulf takeover

[bug]

The root of the problem seems to be using commodity Ethernet networking as a message passing interconnect. It's both slow and somewhat complex to program for. How about this crazy idea for a "fix?" Develop a simple, mass-producable device that plugs into each node's AGP port. If I recall correctly, AGP 8x has a max theoretical bandwidth of about 2 GB/s, coupled with direct access to RAM. Combine this with some virtual memory hacks to keep nodes from tripping over each other or reaching across nodes unnecessarily, and you could theoretically have a cheap interconnect that would replace the ugly hack of using ethernet as an interconnect. I'd have to think that it would blow existing clustering interconnects out of the water, and probably give I/O performance approaching supercomputers. Any computer engineers out there who can tell me how crazy I am?

Re:Expected fallout from the Beowulf takeover

[Jeff DeMaagd]

IIRC, your suggestion takes a bit of tweaking.

Infiniband helps, but even infiniband on PCI-X slots doesn't keep up with Cray & SGI interconnects, VA-Tech is using such cards.

AGP 8x is silly to implement now, once they are done (it takes a year or two), then everyone will be using PCIe, which is better than AGP in terms of bandwidth and memory access. AGP will take several years to go away, but why not hit the most advanced off-the-shelf system slot right away?

The thing that isn't off-the shelf is making a switch, but that might be helped with multiple ports. Then you have to deal with latencies because of the chips between PCIe and the CPU and memory. What of the costs of the cabling between computers? Will that be standard CAT-6 / CAT-5e? If not, then .

I think in the end, you'd still want an interconnect that connects straight to the northbridge or the CPU, to cut the latencies to a minimum. Custom or changed northbridges, and custom motherboards with them is what would cost money.

Re:Expected fallout from the Beowulf takeover

[susano_otter]

Actually, that's what SGI and Cray get, when SGI and Cray spend R&D dollars to solve cluster problems.

And let's not forget that Cray is the canonical supercomputer company. It's probably unwise to look to Cray as a yardstick or guideline for the direction and success of future cluster R&D.

Re:Expected fallout from the Beowulf takeover

[gnuLNX]

Wow....that is a kick ass idea. If what you are saying could really work then you me friend might be on to a big money idea.

Re:Expected fallout from the Beowulf takeover

[forkazoo]

You may have issues with the speed of light kicking your ass. :) The solution is pretty elegant at first glance. Provide some simple means for large data dumps from one node to another, using AGPnet. (Or whatever you want to call it.)

If we assume that you can run at full AGP bandwidth, the simplest setup is probably some sort of bridge chip sitting on AGP which can do DMA, and transfer any requested data out to the requesting node. The bridge chip adds a bit of latency, but it shouldn't be too bad. Now, how many pins are on an AGP slot? At what clock speed? You are going to have that many wires going from the AGPnet card to the other AGPnet card, all at native frequency? Well, okay. Sounds like a plan. One problem... With that much wire, you may get into clock skew issues that are pretty substantial. Maintaining the signal integrity over 6 inches on a board is way easier than six feet of wire. Oh, and how far apart can the systems be? 3 feet? 100 feet? I can have Gig-E run pretty damned far before it is an issue. If I need farther, hubs and switches are well known, cheap, commodity hardware.

Also, can I have more than two nodes? What sort of network topology are you planning? Ring? Centrally switched? Switch adds latency. Ring adds latency when you need data from past the next node - or, you are all on a shared bus, with substantial comms overhead. (And this solution is for chatty comms oriented cluster tasks, right?)

Basically, all you have said is that we should put network cards in our AGP slot. That's all well and good, but you left out the minor detail of "the network" -- The slot itself isn't that big of a deal.

Inevitable

[Marxist Hacker 42]

What most people don't seem to understand is that you don't need a supercomputer when a mesh of nodes on a network will do just as well. Just like most people don't understand that a 386 running Linux and Word Perfect 5.1 is just as good of a word processor as a 2.5Ghz Itanium running Windows and Word. Computer power has *usefull* limits as well as technological limits.

Re:Inevitable

[Anonymous Coward]

The problem is, when managers budget for clusters, you can't afford a real SSI supercomputer when a cluster *won't* do just as well.

Clusters are great for problems that break up into lots of almost-independant pieces, but there are a lot of problems that don't.

Re:Inevitable

[mfago]

a mesh of nodes on a network will do just as well

In some cases.

Unfortunately, some problems are particularly unsuitable for clusters of commercial computers, and really benefit from specialized architectures such as shared memory or vector processors.

A while ago it was decided by the US government to essentially abandon such specializations, and buy COTS. It is certainly cheaper, but not necessarily effective.

Re:Inevitable

[Anonymous Coward]

Just like most people don't understand that a 386 running Linux and Word Perfect 5.1 is just as good of a word processor as a 2.5Ghz Itanium running Windows and Word.

"Just as good" if you don't know how to use Word, anyway. In any case, I tried WP 5.1 on my old Pentium 66/Red Hat 5.1 box years ago -- if you got it to work smoothly on a 386, I salute you. It ground Pentiums to a halt for all the rest of us.

Re:Inevitable

[Marxist Hacker 42]

"Just as good" if you don't know how to use Word, anyway. In any case, I tried WP 5.1 on my old Pentium 66/Red Hat 5.1 box years ago -- if you got it to work smoothly on a 386, I salute you. It ground Pentiums to a halt for all the rest of us.

Way back in the day, I had it working smoothly on a 386DX-40. This was actually *before* 486s came out, let alone Pentiums. Of course, that was under DOS.

As for the Just as Good point- 95% of people using Word Processors are just using them as a glorified typewriter- a text editor would do as good.

Re:Inevitable

[14erCleaner]

Unfortunately, some problems are particularly unsuitable for clusters of commercial computers, and really benefit from specialized architectures such as shared memory or vector processors.

My guess is that most of these problems could be done massively parallel, it's just harder to program (and thus hasn't been pursued yet). You can buy a lot of programmer-years for $10 million, though, and unlike a big vector mainframe purchase, you can share the results if you spend the money on software development instead.

Re:Inevitable

[Marxist Hacker 42]

If you can't break the program down into lots of almost-independant pieces, then what coding methodology do you use? Monolithic Spaghetti Code Batch Files? Anything that can be proceduralized can be object oriented- and once it's OO, it's broken down into almost independant pieces that can be worked on by networked computers.

Re:Inevitable

[Marxist Hacker 42]

Shared memory can easily be simulated on a gigabit network, but vector processors is slightly harder. To simulate vector processors on a network, you basically have to create an extra bit of hardware for every node on the network- mass produce a vector processor card, and distribute that. You'd still probably end up cheaper than a Cray.

Re:Inevitable

[Anonymous Coward]

Shameless copy from a news site....
---snip---
Thursday, 29 July, 2004

NASA to build 10,000-processor Linux computer

Robert McMillan, San Francisco

NASA has given the green light to a project that will build the largest ever supercomputer based on Silicon Graphics' 512-processor Altix computers.

Called Project Columbia, the 10,240-processor system will be used by researchers at the Advanced Supercomputing Facility at NASA's Ames Research Center in Moffett Field, California.

Scientists will use Columbia to design equipment, simulate future space missions and model weather patterns. A portion of the US$160 million system will also be made available to other government agencies and educational facilities, says Bill Thigpen, manager of Project Columbia. "We need to look at working with other agencies to provide them with access to this system because it is a unique system," he says.

What makes Project Columbia unique is the size of the multiprocessor Linux systems, or nodes, that it clusters together. It is common for supercomputers to be built of thousands of two-processor nodes, but the Ames system uses SGI's NUMAlink switching technology and ProPack Linux operating system enhancements to connect 512-processor nodes, each of which will have more than 1,000 gigabytes of memory.

"We use a very large single-system image," says Jeff Greenwald, senior director of server product marketing with SGI. "The other guys come with a very thin node cluster, and try to screw them all together."

The Altix nodes will use Intel's Itanium 2 microprocessors, and the entire 20-node system is expected to be fully assembled by year's end, he says.

SGI has used this large-node technology to build a number of smaller Altix systems with between 3,000 and 6,000 processors, but Project Columbia will be the largest to date, Greenwald says,

Columbia's large-node, shared-memory architecture works well for NASA's "tightly coupled" weather and space simulation applications, where a lot of inter-processor communication is required, Thigpen says. "These codes scale very well on this type of architecture."

The downside to the large-node architecture is that if a single processor fails, the entire 512-node system goes out of service, he says.

The first node of Project Columbia, named Kalpana after Columbia astronaut Kalpana Chawla, was built by Ames researchers last fall. Since then, two nodes have been added, and NASA and SGI will spend the next five months assembling the 17 more nodes.

With the next version of SGI's NumaLink technology, expected in the fall, Project Columbia will be able to share memory across 2,048 processors, Thigpen says.

Linux creator Linus Torvalds applauded the team's success at using Linux in such large nodes. The operating system typically is used in much smaller nodes of 2 to 8 processors.

"Scaling up to ... 512 CPU's is pretty damn studly," Torvalds said in an email interview. "Putting 20 of them in a cluster and making them be programmable as a single machine is pretty hot."

Re:Inevitable

[fitten]

Yes... but it depends on the overhead required for those almost-independent pieces to be transferred from one machine to another. There are lots of fine grained problems out there where All the computers in the world tied together with Ethernet couldn't solve as fast as one good supercomputer.

Basically it boils down to:

Compute the cost for communication of one work unit for your algorithm.
Compute the cost for processing that work unit.

If the cost for communication is significant compared to the computaional cost (say, 1%) then you probably will have a performance issue.

This is why SETI and the like do well. The cost of communication of one WU is insignificant/irrelevant compared to the computational cost of one WU.

However, there are a good number of problems that can't (or at least haven't yet) had algorithms thought out where the cost of the communication of one WU over a high latency interconnect like Ethernet is useful given the amount of computation of that one WU. I assure you, if it were easy to make those problems embarassingly parallel like SETI, someone would have done it already and if you can figure even some of them out, you'd be very wealthy. If you could figure out a way to do it all automatically and do it well, you'd be rich beyond the dreams of avarice!

Re:Inevitable

[Marxist Hacker 42]

Thanks for the idea of a killer app- but given that it'd be GPL anyway I'll post it here for somebody more skilled that I to take advantage of: A virtual networked memory driver, based on Ethernet MAC addresses rather than TCP, with a thin packet that can be scaled to the cache size of the Opteron processor. Such a beastie would make the latency problem almost disappear, because the packets would be so small in comparison to TCP. In other words- a cluster memory driver based on a single layer MAC addressing scheme, with no higher level envelopes. When the CPU requests a memory address, the terabyte shared ramdisk provides a memory packet starting at that address to fill the cache on the CPU- even if that memory address is in the cache already. This refreshes the cache in hopefully less time than it takes to use up the cache- and from the point of view of the programmer it looks like a single memory space process.

Re:Inevitable

[Kupek]

Eventually those individual pieces need to syncrhonize again, and sometimes that syncrhonization is more expensive than the computation itself.

Re:Inevitable

[Anonymous Coward]

You are so talking out your ass it's not even funny. "I'll take that challenge by not doing it but suggesting an unresearched, ill-informed method of doing it that has no basis in reality so someone else can do it for me."

Pure. Genius. I thank you. Humanity thanks you. Seymour Cray has seen the light and wants to be baptised into the Holy Church of the Commodity Cluster with Zero Latency.

Bah.

Re:Inevitable

[Performer Guy]

No the government didn't abandon these. Infact the government is one of the few remaining purchasers of this type of hardware. It just so happens that a lot of problems including the governments are solved by clusters.

It could be argued that at least *some* of the ASCI (Advanced SuperComputing Initiative) computers had specialized architectures with loads of bandwidth & low latency interconnect (in their day).

It's a bit of a joke complaining about a lack of vector computing when every Intel and AMD CPU sold today has floating point vector instruction set extensions with very interesting operators.

I'd argue that if you take those lamented early 90s supercomputers there's not a problem they can solve faster that a relatively small contemporary cluster or even a single desktop system. A standard 4 CPU single PC desktop system with the right architecture could also spank those legacy systems in memory bandwidth, shocking but true. It just didn't keep pace with the scale and cost reduction of small systems & clusters.

The real problem here is *relative* performance of supercomputers and commodity components, but as it takes hundreds of millions if not billions to develop a new competitive CPU & architecture and manufacture it, scientists pockets aren't deep enough to pay for those costs (and thank goodness because it's our tax dollars). It is rather pathetic to lament that supercomputers have been outpaced by clusters. The economics make it impossible for supercomputers sold in low numbers to keep pace. Or more reasonably stated, the economics of consumer PC systems makes powerful computing ubiquitous and affordable to the point where it no longer makes economic sense to pursue specialized processors and architectures to try to outperform them.

If anything is to be done it would be to increase the bandwidth and reduce the latency of cluster interconnect, and guess what, that's EXACTLY what smart people are working on right now.

As for eating America's seed corn, it is Intel and AMD that sell most CPUs used in clusters today. It is that competition and the pressure of increased development costs that makes custom hardware untennable.

It is just false to imply that supercomputing technologies fed lower end development. It is a romantic vision of trickledown technology but it is not actually how technological development works. Look at computer graphics, since the commodity PC graphics cards beat big iron from SGI there has been more innovation and development in graphics hardware, not less. There is competition and a willingness to experiment with new features. The same is true with CPUs from Intel and AMD and the architectures and innovations in memory bandwidth they constantly drive forward.

Re:Inevitable

[Anonymous Coward]

wtf? I'm pretty sure I was running WP 5.1 with DOS on a 286.

But-- come to think of it, there were two versions of 5.1 weren't there? A DOS version and a gui version?

WP 5.1 rocked.

Law of Diminishing Returns

[Billobob]

It appears to me as if we have reached the point where supercomputers aren't really as practical as they were before. Fewer and fewer industries need and prefer supercomputers to a cluster of cheap PCs, and the market is simply heading in that direction - nothing really unique happening here other than capitalism.

Of course people are going to cry that companies like Cray are falling by the wayside, but the truth is that their services simply aren't as needed as they were in years past.

Re:Law of Diminishing Returns

[susano_otter]

I suppose the counter-argument would go something like this:

It's true that supercomputers aren't really all that useful or necessary these days. However, it may be that a future computing problem shall arise, which requires a next-generation supercomputer to solve. So we'd be well-served to have a next-generation supercomputer fresh from R&D, to apply to the problem.

We may only encounter one or two more supercomputer-class problems, but they might be important ones. We should be prepared.

On the other hand, we may encounter a problem that can only be solved by horses. But we don't see a lot of buggy-whip subsidies these days...

Re:Law of Diminishing Returns

[cowscows]

When that need occurs, then people will be more willing to spend money on it. If some farsighted investor wants to get the jump on that need, they can take a long term approach and hope to make it big later. And maybe some research money can come from the government, but not industry-wide commercial welfare to try and keep all these companies afloat until the next supercomputer killer-app comes along.

"Feed' Corn?

[jmckinney]

I think that should have been "Seed Corn."

Re:"Feed' Corn?

[jmckinney]

Offtopic, my left testicle. You're SUPPOSED to eat feed corn. You save seed corn to plant.

Re:"Feed' Corn?

[Seanasy]

Actually COWS are SUPPOSED to eat feed corn. Seed corn is grown for seed. Feed corn is grown for animals.

Re:"Feed' Corn?

[Anonymous Coward]

welcome to slashdot land of botched articles.

Re:"Feed' Corn?

[scoobrs]

Umm, thanks. I knew that. It was 8AM in the morning and I impulsively hit Submit because it sounded right. As for the issue, I think scientists are concerned about it because large computers have taken more and not less of a role in several critical fields. Just how much of astronomy is done with a slide rule and telescope anymore? What percentage of NSF and DOE money goes to computers for research? On top of that, a year or two ago Japan beat America by 40x average on real problems (not Linpack numbers) with a traditional supercomputer based upon NEC vector processors called the Earth Simulator. It's still at the top. The concern is that our scientists will go to Japan instead, so national labs have been opening up new public supercomputer resources for scientific research that are superior to many university labs. The debate here is more how should we spend our science money than should we subsidize another industry. Commodity clusters are cheap partly because the margins to compete are too low to allow for much risk-taking in design. Cray appears to be hurting because they devoted all their resources to a new processor that required huge investments in design without offering a cheaper product as well.

Expert complains:

[Anonymous Coward]

Free market sucess might lead to us actually having to pay for our own supercomputer research that we use in profit making ventures.

I Need A RAIS

[grunt107]

Random Array of Inexpensive Servers.

If the 'supercomputers' of today are increasing performance, does it really matter the design?

Maybe that is a signal that monolithic computer tasks are best handled in a hive mentality - have the Queen issue the big orders, have the warriors performing security, have the workers transporting the goodies (data), and have the requisite extra daughters and suitors to grow the hive and assure its viability (redundancy).

The fact that it is cost-effective is even better.

Re:I Need A RAIS

[Anonymous Coward]

"Random" Array? You *teh* crypto man? ;)

(Try "Redundant".)

Re:I Need A RAIS

[Jeff DeMaagd]

There are fail-over clusters. I'm not sure if anything has reached the sophistication of what you want. Given that security is a specialized task, for computers configured differently and they need to be at certain points of the network (i.e. at the gateway), there may be little point in letting just any computer anywhere do security. If you want redundancy, just keep a spare next to it.

There are load balancers for web servers, although there may not be much intelligence in the design, I think they just cycle the work around.

It's bad news for Cray

[Louis Savain]

Don't see this as bad news... it's a sign that we're winning.

Right. The Cray folks have just realized that they are about to go the way of buggy whip and the slide rule. They don't like it one bit. They can only complain by making a lot of noise. But it won't work. When you're extinct, there is no coming back.

Re:It's bad news for Cray

[Saeed al-Sahaf]

They can only complain by making a lot of noise.

That's the was it's normally done.

Re:It's bad news for Cray

[DarkMan]

Uh, Cray have a backlog of orders. A backlog to the tune of $153 million, if I recall correctly.

That's not the sign of a dying buisness model. If they are having problems, it's down to the mangement, not lack of demand.

There are problems that don't work well on clusters, but rocket on a proper supercomputer. These include a lot of interesting areas, there will always be demand for a few pieces of big iron. At the risk of echoing the ghost of IBM CEO's past, I think somewhere around 20-30 serious top end supercomputers in the world [0]. Most of the rest of the jobs will do just fine on high end clusters.

If you read the article, there are no quotes from Cray people. What there are quotes from is the people who used to get to play with special hardware, who now admin those clusters.

It's toys for the boys, not a buggy whip issue.

[0] That's informed by being someone who uses high perfromance computing, both cluster and supercomputer.

Re:It's bad news for Cray

[Anonymous Coward]

The Cray folks have just realized that they are about to go the way of buggy whip and the slide rule. They don't like it one bit. They can only complain by making a lot of noise.

You forgot lobbying congress and getting Republican Rep. Judy Biggert to give them a handout.

Re:It's bad news for Cray

[Louis Savain]

There are problems that don't work well on clusters, but rocket on a proper supercomputer.

True. Highly sequential algorithms fall in this category. The problem is that the switching speed of processors will soon run against a natural barrier. On the other hand, we will continue to add new nodes to our parallel superclusters long after the barrier is reached and supercomputing is stopped right in its tracks.

Re:It's bad news for Cray

[Anonymous Coward]

Then you work on dropping the price of clustering the high-end supercomputers and use those.

A lost metaphor brings out my inner language nazi

[peacefinder]

It's seed corn. Seed, as in, what you don't eat, but save to plant next year.

Kids these days.

Re:A lost metaphor brings out my inner language na

[Anonymous Coward]

Either way you get to see it twice.

Re:Isn't this good?

[Anonymous Coward]

Clusters are not good for very chattery parallel processes, a shared memory supercomputer can still do much better for computational fluid dynamics.

Re:A lost metaphor brings out my inner language na

[Anonymous Coward]

Yes - feed corn being what you (or the cows) DO eat.

Re:A lost metaphor brings out my inner language na

[asland]

And that is what it says in TFA, the slashdot story has it quoted wrong.

You are all missing the point

[Anonymous Coward]

Its the fact that clusters require higher skill to program efficiently for than do single processor systems. Plus you have all of the wasted processing power used for communication between the nodes. Granted, many problems lend themselves well to distributed computing (essentially what a cluster is, but the nodes are closer and communicate faster), but there are also problems that are handled better by a smaller amount of specialized hardware. The other point is that by using off the shelf parts, we are not really innovating in this space like we should be. We are allowing the commodity computer market determine the direction of the supercomputer market.

Re:You are all missing the point

[sterno]

What's key here is the amount of processing power you get for a given dollar. Clusters of general purpose systems may not be as efficient as a vector system, but in the end, the price makes up for the inefficiencies.

If the cost of the system plus the cost of the geek to run it is cheaper per unit of work than it is for a vector machine then that's all there is to it.

We are innovating by squeezing more and more processing power into smaller and smaller spaces and by improving on the interfaces for intercommunication between nodes.

Re:You are all missing the point

[Tony-A]

What's key here is the amount of processing power you get for a given dollar. Clusters of general purpose systems may not be as efficient as a vector system, but in the end, the price makes up for the inefficiencies.

In general, no.
The inefficiencies are structural, orders of magnitude, not percentage.

Where, how, when you apply the processing power is what matters, and there are no valid rules of thumb.

Corps and Gov't will buy them if they need them

[Anonymous Coward]

...and in many cases, even if they *don't* need them. If you want to ensure the economic health of a nation, investing in basic, long-term research would be a good start. This gives a head start in the area industry is weakest in.

Trying to create some sort of supercomputing subsidy out of a misplaced fear that the U.S. will miss out is silly -- if the demand is there, the companies will be as well.

Software vs hardware

[xenocide2]

One of my professors (everybody has one of these it seems) is working on cluster computing research, extensions of MOSIX. He's a guy with networking and operating systems expertise. I wouldn't hire him to build a new generation of super computing interconnects or processors. As the Republicans have taught us, federal budgets are not a zero sum game. Why divert focus from one to the other when we could have both?

We have to be careful about measuring these things however. One of the goals of cluster computing was to lower the cost of computing. If the government is spending less and still meeting needs, thats not nessecarily an indicator of a problem. If that means that we aren't writing code to fit into a vector platform, so be it!

There is no crisis

[0x0d0a]

Cray has been engaging in scare tactics about "America being dominated by overseas competitors" for a while, because they're terrified of losing the lucrative business contracts from government and big business, they'll pull out all the stops. They've come up in the IT press recently a couple of times.

Screw 'em. If there's a need, the market will provide. If it turns out that the important tasks can be parallelized and run on much less expensive clusters, then all that means is that we have a more efficient solution to the problem.

Re:There is no crisis

[Anonymous Coward]

Sometimes there is no viable market and subsidies are necessary. For example, would hubble or chandra ever be built without subsidies? There are many areas of research where cluster do not work and government funding is necessary for vector computers.

Re:There is no crisis

[DerekLyons]

Cray has been engaging in scare tactics about "America being dominated by overseas competitors" for a while, because they're terrified of losing the lucrative business contracts from government and big business, they'll pull out all the stops. They've come up in the IT press recently a couple of times. Screw 'em. If there's a need, the market will provide.

That's the problem, in this instance the market won't necessarily provide. The market is absolutely wonderful at providing high demand items, but it sucks badly at providing low demand items, especially when that low demand item is something that takes years of expensive R&D to bring to market.

Re:There is no crisis

[Anonymous Coward]

Just like "the market" provided solutions to moon landings, artificial heart research, mars exploration, rural electrification, and all of the other big-spending U.S. government waste.

Pork Barrel, not Feed Corn

[Un pobre guey]

It isn't "feed corn" that's disappearing, it's the old Cold War-style Paranoia Pork Barrel. Companies that used to lap up obscene amounts of funding for exotic hardware now have to go face to face with fast and cheap clustered COTS hardware. 25+ years of commodity-scale engineering, in the case of commecial microprocessors, has vastly outstripped the achievements of specialty supercomputer technology by the metrics of bang for the buck and constancy of improvement through time.

Poor little babies, now where will their executives and boardmembers get free money? Will they actually have to do something useful for a living, and for a change? It seems we are only a Darwinian capitalist economy when the little guy gets fucked. When the professional bullshitters at the top get it, it becomes some sort of strategic crisis that requires immediate injections of billions of dollars. Screw them. May Cray rest in peace.

La Supercomputadora is Dead! Viva la Supercomputadora!

Re:Pork Barrel, not Feed Corn

[Anonymous Coward]

Have you ever worked on these machines? The performance of COTS hardware is abysmal on many problems. Further, some problems will not work efficiently with clusters. Allowing this sector to die is not in the interest of the wider research community.

Re:Pork Barrel, not Feed Corn

[Un pobre guey]

So what do you suggest, in dollars and cents, and in comparison with the numerous other demands for government subsidies?

Re:Pork Barrel, not Feed Corn

[Halfbaked Plan]

Quit with the class warfare drivel, dude.

Re:Pork Barrel, not Feed Corn

[Un pobre guey]

Sorry. Ran out of Vaseline last week. Still smarting.

Re:Pork Barrel, not Feed Corn

[dbIII]

Companies that used to lap up obscene amounts of funding for exotic hardware now have to go face to face with fast and cheap clustered COTS hardware.

That is a very good point - instaed of buying some more expensive Sun machines to process seismic data where I am the solution is to get a few computers that would not even be considered a leet gaming rig. On some things the Suns walk all over them (IO), but most of the time things run a lot better on cut-down gaming boxes.

Re:Pork Barrel, not Feed Corn

[Halfbaked Plan]

Well, go out and find a real girl.

Stop dwelling on the picture of Lenin in the flyleaf of 'On Contradiction' for your 'inspiration' in such matters.

The classic supercomputer is the modern desktop

[iabervon]

If you really want a vector-processor supercomputer you can program in Fortran, get yourself a G5 and gcc. The PPC64 supports SIMD vector processing. For that matter, any problem which benefits from vector processing is trivial to parallelize with threads.

Re:The classic supercomputer is the modern desktop

[Shinobi]

One, Altivec only supports single-precision floats. Big no-no in quite a few scientific areas. Also, to be blunt: GCC sucks. If you want something decent, at least use IBM's compiler.

Re:The classic supercomputer is the modern desktop

[oudzeeman]

"get yourself a G5"

I just got myself 512 G5 Processors : )

http://www.clusters.umaine.edu/
http://www.clusters.umaine.edu/gallery/xserve

Re:The classic supercomputer is the modern desktop

[Creepy Crawler]

Heh, Im stupid, but I gotta ask...

What is that screensaver on Cam1? From the cam, it looks really cool.

Re:The classic supercomputer is the modern desktop

[Un pobre guey]

is trivial to parallelize with threads.

Multithreading is not trivial. Parallelizing with threads is probably not trivial either. Perhaps you meant "practical," "feasible," or "do-able," but certainly not "trivial."

Re:The classic supercomputer is the modern desktop

[oudzeeman]

It's just one of the standard screen savers on OS X (it's running Panther). I don't recall what the name is.

Re:The classic supercomputer is the modern desktop

[Jeremy Erwin]

could be "flurry"

Re:The classic supercomputer is the modern desktop

[Creepy Crawler]

Heh, Im just too poor to own a mac (college student). Ive pieced my own computers together from junk from friends...

Though I would love to have a g4-altivec machine ;-)

Thanks for responding ;) Just I would like that screensaver for my linux/windows machine. It does look freally neat.

Re:The classic supercomputer is the modern desktop

[flaming-opus]

The G5 is not really a vector processor. It allows you to use a 128bit register as 4 32-bit registers, or 16 8-bit registers. This allows one to do more math with fewer instructions. This is cool for multimedia apps, but it's different from real vectors.

A real vector architecture is not done that way to do math more quickly (thought it's pretty good at that), but rather to isolate the CPU from the ridiculous latency of memory. Since the op V1 = V2 + V3 takes many many cycles to complete, the load that's necessary for the following operation has time to get something from memory. The vector guys found that throwing more transistors and more wires adds to the bandwidth, but it's a lot harder to reduce latency, so they hid it instead.

Re:The classic supercomputer is the modern desktop

[iabervon]

Multithreading is, in general, not trivial. But if you already have the problem broken up into an array of problems each of which can be solved independantly, which is what you need in order to use a vector processor, then it is really easy to multithread. If fact, you can do it with fork and wait.

Catchup == lawsuits?

[FerretFrottage]

In the age of IP and patents it seems like it is very hard for companies to make major advances [in any field] without some other company cry foul and taking that company to court over patent/IP rights, especially if the alleged infringer is a smaller company (i.e. less lawyers). IBM and MS, among others, are filing dozen if not hundreds of patents a day. What we are seeing as an affect is that innovation is being stifled by litigation.

(pat pending)

Re:Catchup == lawsuits?

[Anonymous Coward]

The thing to do is innovate anyway, perhaps in secret. Once you have overwhelming technological advantage, melt the flesh from the bones of the establishment with your death ray. Patents are for chumps.

Trickle Down

[Anonymous Coward]

Technology first developed on the high end slowly works it's way down into the low end. What happens when the high end is no longer there.

Not that many people really need a race care, but advances in fuels, materials, engineering in race cars eventually leads to bette passenger car. And for raw performsnce, strapping together a bunch of Festivas will not get you the same as an Indy racer.

Re:Trickle Down

[Belial6]

No, but if you want to do something useful like getting 100 people 50 miles down the road, the bunch on Festivas will beat the Indy car every time. Of course if your goal is to get a nice shiny trophy, go for the Indy car.

Re:Trickle Down

[argent]

I'll bet you a bunch of Festivas will get you further than an Indy car on any real road. :)

Re:Please make it stop!

[SoCalChris]

Quit complaining. At least they didn't use this color scheme. :)

And if you really dislike the color scheme that much, there are simple instructions in my journal about how to use Firfox to change the colors displayed on here.

Without a market you can't survive long term

[wintermute42]

There seems to be some historical revisionism going on regarding the demise of the "supercomputer industry". People are coming out of the woodwork now saying that lack of government support caused the great supercomputer die off.

As Eugene Brooks predicted in his paper Attack of the Killer Micros, the supercomputer dieoff was caused by the increasing performance of microprocessor based systems. Many of us now own what used to be called supercomputers (e.g., 3GHz Pentinum processors, capable of hundreds of megaFLOPs).

The problem with supercomputers is that high performance codes must be specially designed for the supercomputer. This is very expensive. As people were able to fill their needs with high performance microprocessors they quit buying supercomputers.

Many people who need supercomputer levels of performance for specialized applications (e.g., rendering Finding Nemo or The Lord of the Rings) are able to use walls of processors or clusters.

There are, of course, groups where putting together off-the-shelf supercomputers will not suffice. But these groups are few and far between. As far as I can tell they consist of the government and a few corporations doing complex simulations. The problem is that this is not much of a market. Even if the government funds computer and interconnect architectural research, there does not seem to be a market to sustain the fruits of this research.

In the heyday of supercomputers there were those who argued that when cheap supercomptuers were available the market would develop. The problem is, again, programming. High performance supercomputer codes tend to be specialized for the architecture. Also, no supercomputer architecture is equally efficient for all applications. It is difficult to build a supercompter that is good at doing fluid flow calculations for Boeing and VLSI netlist simulation for Intel (the first applications tends to be SIMD, the second, MIMD). The end result of these problems tends to suppress any emerging supercomptuer market.

The reality right now seems to be that those who are doing massive computation must build specialized systems and throw a lot of talent into developing specialized codes.

And if they really want it...

[sterno]

If there truly is a demand for those kind of processors, then somebody will likely meet that demand. Right now, it seems that actual demand is so low that they have to drum up this legislation a as a sort of wellfare for vector processor manufacturers.

It's a simple cost tradeoff. If you can save millions in purchasing computers, it means more money to pay for people to run those computers and do the real work.

Re:And if they really want it...

[keithu73]

Let's say you are a government who needs fighter planes. Fighter planes are really expensive because they are designed to make sharp turns at high speeds (among other things). Unfortunately, the commercial market will support Cessnas, but it won't quite support fighter planes. Will you let all of the fighter plane makers go out of business and just buy Cessnas? Such is the world of high performance computing. A lot of tasks work great on a cluster. In other cases, people who need computing capacity (as opposed to computing capability) buy clusters. Many of the governments most important applications, however, need computing capability. That is, I have a big job and I need it done NOW! On a 1000 node cluster, that application might run 200 times faster than a single computer (if you are lucky). On a properly designed 1000 node supercomputer, it might run 750 times faster than a single computer (without that much work). That factor of 3.75 is REALLY important when the job runs for 3 weeks on the supercomputer (almost 3 months on the cluster). The problem in the market is that the government is one of the primary user of supercomputers. (weather forecasting, climate forecasting, scientific research labs, weapons labs). If they let the industry falter, they will be the ones to pay the price.

Re:And if they really want it...

[sterno]

You cannot buy 10 or 50 or 100 cessnas and make a fighter plane. You can buy a cluster and make it as effective as a vector system.

Yes, that factor of 3.75 is a big deal, that's why you buy a cluster that's big enough to make up for that 3.75. It will still be cheaper than the 1000 node supercomputer.

Re:And if they really want it...

[keithu73]

Two important notes:

1) supercomputer != vector supercomputer

It's a popular misconception that they are the same, but it is just not true. Supercomputer features note present in a cluster computer include things like truly high performance networks, system management infrastructure, and reliability by design. You buy a cluster that's big enough to make up for that 3.75 difference and you won't be able to keep the whole thing running for more than a couple of hours. That's not long enough to get your job done. Worse, the larger the computer gets, the poorer it scales. Scalability curves have a tendency to roll over at some point. (i.e. more processors make it slower).

2) the cost differential MIGHT be 3.75 for the vector system, but it isn't for other supercomputers.

I say "might" on the vector system because, as I pointed out above, scalability isn't just a straight linear thing.

What tasks require high-speed interconnects?

[Bruce Perens]

One of the nice things about clusters is that they encourage people to consider how to decompose a problem so that it can work without a large high-speed shared data memory. Some of the older supercomputers were important because scientists hadn't done this work because there wasn't the economic incentive back then. Now there is one.

So, what tasks still require a high-speed shared data memory? Answer that, and you'll understand where you can still sell a supercomputer.

Bruce

Re:What tasks require high-speed interconnects?

[Anonymous Coward]

So, what tasks still require a high-speed shared data memory? And when you see how few shops there are in the world that have problems that can't be solved by clusters, plus can afford to pay $6 million for a computer, then you'll understand why we don't make them any more. Simply put, the economics of amortizing $100 million in development costs over only 10 paying customer just don't make sense. If the Japanese want to subsidize a program just to have bragging right to "the fastest computer in the world", more power to them! It all depends on your definition anyway -- isn't SETI@home really the fastest, in terms of raw CPU cycles?

Re:What tasks require high-speed interconnects?

[Rosco P. Coltrane]

So, what tasks still require a high-speed shared data memory?

A high-speed shared memory test program?

Re:What tasks require high-speed interconnects?

[Anonymous Coward]

Bruce,
Basically any simulation that involves a large number of fully interacting 3 dimensional components over very many timesteps.

Problems that fall into this category:
weather simulation
aerodynamic studies
molecular modelling
(to a lesser extent other fields in computational physics and chemistry)

While these can be done on clusters (and often are), the longest, highest resolution studies are done on big shared memory systems, where one user runs their whole simulation for weeks or months.

The problem is, only a handful of these are really necessary, and only a fraction of these have a compelling need and the resources to afford them:
the weather service (has), NASA and the aerospace industry (has), auto makers (has), academics in chemistry (no such resources, greater patience, cheap student programmers). There aren't many other potential buyers, so it's not surprising that the commercial groups are trying to get the government groups to pitch in more. No one is asking the academics to even try (they're all off trying to make their shared memory codes run on cheap Linux clusters.. it's not as if they'd have had the money to pay for a supercomputing industry anyway).

-one of those computational chemists

Re:What tasks require high-speed interconnects?

[Soong]

How about high resolution physical simulation (whether that be climate modeling or plasmas)? One great thing of a Supercomputer is that you can hold so much data at once in the active set. One node can only hold so much data, so the full simulation has to be distributed across the whole supercomputer. It is definitely not RC5 keys, the opposing end of the spectrum in this data/compute tradeoff.

Re:What tasks require high-speed interconnects?

[Bruce Perens]

Good answer, Anonymous.

A long time ago, at Pixar, I got an ARPA grant to work on an image-processing application for the feature film industry. The purpose of the grant was economic and military at the same time. I was to help create a market for multiprocessor computers (not really supercomputers) so that there would be U.S. manufacturers of them if when/if the Army needed them for military purposes. This is what often gets called corporate welfare, although I could see the defense purpose was valid. I don't know if ARPA still does this sort of grant. To do one would require an application that is interesting to more than just the folks on your list. And these days visual effects is much more of a solved problem.

Thanks

Bruce

Re:What tasks require high-speed interconnects?

[renoX]

Why do you call these problems "fully interacting"?

Usually a location will only interacts with its neighbours, so you can subdivide the mesh into chunks which runs on different nodes, of course the communications between nodes are still quite high for "frontier" communication, but you can't really say that it is "fully interacting"..

It could be "fully interacting" if your timesteps is bigger than the propagation time of course, but usually this is not the case..

Its about time to face the facts

[0racle]

America isn't going to be the best in everything, its just not possible. So what the fastest supercomputer in the world is a Japanese creation, other then some hurt pride among builders of these things, it doesn't mean anything. The reaction, "Oh my God someone else made something better, we better dump money on the problem for no reason,' doesn't do anything so give up already.

Re:Its about time to face the facts

[Anonymous Coward]

I thought DeVry WAS the short bus.

Former Cray Folks Move On

[lofi-rev]

And do interesting things. And try to keep in touch.

About time...

[14erCleaner]

The surprising thing about this is that there are still companies making big-iron vector supercomputers. I worked in this industry from about 1980 to 1995, and when I left it was dying already. Even then, the majority of scientific computer users would rather have their own mini or microcomputer than get a small share of some behemoth Cray mainframe. It provided them more flexibility, and if they can use it 24 hours per day it also was more effective.

For things like weather forecasting, maybe big vector machines still have an edge, but I suspect that's changing as the weather guys get more experience in using machines with large numbers of micros. This seems to have already occurred, in fact; NCAR appears to have mostly IBM RS6000 and SGI computers these days, with nary a Cray in sight.

The most common term I used to hear in the early 90's was Killer Micros; I think the term dates back David Bailey in the 80's sometime. If you want more evidence that the death of the supercomputer has been going on for a long time, check out The Dead Supercomputer Society, which lists dozens of failed companies and projects over the years; this page was apparently last updated 6 years ago!

Re:About time...

[sterno]

Yeah I went to NCAR a year or so ago and their new top of the line was racks and racks of IBM RS6000's. Sure, it doesn't look as sexy as the old Cray they now use as a couch in the museum, but it does appear to get the job done.

Re:About time...

[DerekLyons]

If you want more evidence that the death of the supercomputer has been going on for a long time, check out The Dead Supercomputer Society, which lists dozens of failed companies and projects over the years;

By that standard, there is not a single significant industry that is dying. (Aerospace? I can think of dozens of failed companies and projects. Food Service? Ditto. Automobiles? Ditto.)

Pride ? Marketing argument ?

[Knx]

These are actually the only main advantages I can see nowadays about a supercomputer versus some cheaper cluster with an equivalent computation power. I think it's also reasonable to say that a cluster is likely to be more reliable, even if a supercomputer actually includes some distributed and redundant architecture, internally.

Still ... in next-coming sci-fi movies, the supa-dupa-computer of the supa-dupa-mothership is likely to remain indeed some big entity rather than a beowulf cluster for the next coming years, right? I guess the part of 'dream' associated to a very big and very powerful computer is not negligible. I guess that's true for both sci-fi and real life

Re:Pride ? Marketing argument ?

[Anonymous Coward]

Skynet (in the latest Terminator movie) was a distributed supercomputer of sorts.

Re:A lost metaphor brings out my inner language na

[Anonymous Coward]

My inner metaphor nazi here: the metaphor would only make sense if every time you planted your seed corn your next crop grew 10% taller.

I guess it sounds better than, "if we don't get more budget, the terrorists will win!"

"Eating Our Feed Corn"

[smagruder]

Isn't this the M.O. of contemporary American domestic policy? The sleazebag politicians are leading us to this grand era of the service economy where the U.S. will be producing nothing of significance, and magically we all* get rich.

*Everyone's who's already rich.

Re:"Eating Our Feed Corn"

[Simkin1]

True Freedom = Absolute Freedom

Absolute Freedom = Absolute Anarchy
...just a thought.

Re:"Eating Our Feed Corn"

[smagruder]

True Freedom = Absolute Freedom

False thought.

Re:A lost metaphor brings out my inner language na

[Dynedain]

It's seed corn. Seed, as in, what you don't eat, but save to plant next year.

Kids these days.

Maybee they did mean feed corn. As in corn that's harvested specifically to be fed to cattle.

Geezers these days.

No, YOU are actually missing the point.

[CatOne]

Granted, it is more difficult to program something (from the ground up) that runs distributed, than it is to program something that runs on a giant 2048-way box.

Just like it's more difficult to write multithreaded code than it is to write single-threaded code.

That's where software, and platforms come in. There is a TON of research being done, which uses technologies like Infiniband and Myrinet as interconnects, and can make a cluster "look" like a big monolithic machine. If you as an end user write code that goes down into the TCP stack itself, you're working too hard, and you're going about it the wrong way.

Put it this way: In 5 years the odds are overwhelming that there will be a good software platform that can let you pick 5000 servers and run your app 10,000 threaded, with everything appearing just like a single process, and running "as it would on a Cray." It's easier to solve this stuff with software -- take your problem (distributed computing) and solve the problem with a different set of technologies (high performance/low latency interconnects, shared address space/DMA across machines, etc).

Apple's Xgrid is a step in this direction. It's missing a ton of "Supercomputer" functionality right now, but it's a nice cross-machine GUI scheduler. Right now this type of app can address maybe 20% of what supercomputer apps need... in the future maybe more like 98%.

Re:No, YOU are actually missing the point.

[Shinobi]

There's one thing you touch upon in passing:
The protocol stack. Many of the problems in speed when it comes to clusters is that people still use TCP(And it doesn't help that a bunch of idiots are trying to get people to use TCP/IP even at subnet level with Infiniband... Talk about crippling Infiniband by doing that..), with alll the performance hits that entails.

Re:No, YOU are actually missing the point.

[Marxist Hacker 42]

Maybe that's what Cray needs to do to survive- learn how to create Cray Supercomputer Simulators on clusters of cheap hardware.

Re:No, YOU are actually missing the point.

[Seltsam]

"Granted, it is more difficult to program something (from the ground up) that runs distributed, than it is to program something that runs on a giant 2048-way box." Not quite. You don't use threads on Crays. I've done some work at the AHPCRC and the parallel programming done there is some combination of the following: MPI, UPC, Co-Array Fortran, OpenMP. Writing an MPI app for 2 processors is as easy as writing one for 4096.

Re:No, YOU are actually missing the point.

[Seltsam]

Darn...forgot the newlines. "Granted, it is more difficult to program something (from the ground up) that runs distributed, than it is to program something that runs on a giant 2048-way box."

Not quite. You don't use threads on Crays. I've done some work at the AHPCRC and the parallel programming done there is some combination of the following: MPI, UPC, Co-Array Fortran, OpenMP. Writing an MPI app for 2 processors is as easy as writing one for 4096.

FWIW, the AHPCRC's Cray X1 is 152 in the Top 500.

Re:No, YOU are actually missing the point.

[graveyhead]

Aarg! Either:

1) Fix the quotation in your sig.
or
2) Change it entirely because I've been using that one for years and I have a lower ./ id than you :P

OK back on topic:

Cray r00lz! No, Beowulf r00lz! No, a Beowulf cluster of Crays r00l!!1

Re:Please make it stop!

[pavon]

I will third that comment, but also add that the slashdot team updates the slashcode on the live system on a weekly basis, so just because they haven't been fixed it yet doesn't necisarrily mean that they are not listening.

Maybe there is no market Need for a Supercomputer?

[Anonymous Coward]

Afterall, 640K of RAM is more than enough for anybody... ;)

Since IBM is adding Dual Cores to the G5 CPU, and when Apple pops 2 or more of those CPUs in one box, well you'll have a 4-way CPU system from the start.

Imagine a cluster of those!

Isn't it about time we started working on
Positronic CPUs now?

Better?

[sterno]

How do you define better? Sure an uber powerful supercomputer might be better in terms of raw efficiency, but if you can have twice the computer for half the price by going with something a little less efficient, who really cares?

Re:Better?

[Shinobi]

The problem with communications-heavy tasks is that you easily saturate a cluster, even if you use Myrinet, Quadrics or Infiniband. You also hit a point of diminishing returns.

That cluster with twice as many CPU's might just perform at 1/3(And that's optimistic) of the speed that the supercomputer does the job at when it comes to such tasks.

Re:Better?

[sterno]

Yes, that may be true. But if that cluster costs 1/3 the price it's break even and in reality they cost far less than that.

Re:Better?

[Shinobi]

Likelihood of it costing only 1/3? Pretty small. Good interconnect fabric is costly, as are the nodes themselves if you intend to do tasks that requires lots of communication, since if a node fails, it will cause stalls etc as the part it was working on needs to be computed again due to that data being needed for the computation to go on.

Realistically, for the tasks we're talking about, you might get 1/5th-1/4th of the performance, for maybe 50-60% of the price.

Re:Better?

[sterno]

If that were true then the market for these computers would be sustainable. People wouldn't pay more to get less. They are buying these clusters because they can get equal or superior processing capabilities at a fraction of the cost.

Re:Better?

[Shinobi]

No, they are buying them for use with tasks that are well-suited for clusters, or because they have no alternative. As a footnote: NSC in Linköping has a 200-node cluster, each node dual 2.2GHz Xeons, 2GB RAM, and SCI interconnect. They also h ave a SGI 3800 with 128 CPU's and 128GB RAM.

For some of the stuff I do, such as building airflow simulations, heat distribution etc, the 3800 outperformed the cluster, despite the CPU's being so much weaker. Simply because of the Shared Memory architecture.

Layoffs at Cray?

[TroubleMan]

...because they can't compete in a market where computer hardware has become a commodity? Oh, BOO HOO!!!

and in other news...

[gmhowell]

And in other news today, buggy whip manufacturers demand increased government subsidies.

Re:Please make it stop!

[Anonymous Coward]

AAAAAhhhhhhhhhh

My eyes!!!!

Forbes promoting socialism?

[peter303]

Forbes has been complaining that federal support of advanced computing is too little? If the government over-stimulates an industry that has too small of a market, it wil just delay the failure.
Of course the governent should continue in its current policy of funding a few leading-edge machines that are too costly to sell into the general market, but will test new technology. The governemnt itself is a customer will energy testing, weather modeling, medicine development, etc.

Re:Forbes promoting socialism?

[SunPin]

Are you surprised? These punks work for Steve Forbes, the poster child of corporatism and its mantra, "capitalism for the people, socialism for us."

It's so much Bullshit though

[Anonymous Coward]

The problem with Cray more than anything is that it cant compete with IBM. Things like Bluegene and Virtual Vector Architecture have nothing to do with commodity hardware, IBM is still putting big money in more exotic supercomputer architectures.

Vector computers arent difficult to build anyway, no knowledge is being lost by not following down that path. This is just about pork and nothing else, and poorly allocated pork at that ... better to spend the money on new compiler technology so poorly written fortran code written by mere mortals can run better on clusters.

GPUs + Beowulf clusters?

[niktesla]

I read some articles about using standard GPUs for matrix and mathematical operations. Does anyone know if this is being coupled with clustering? Seems that this would give you some of the power of vector processors, but thats just my $0.02.

Mangled analogy

[LionMage]

Yes, the person who posted this to Slashdot mangled the analogy in the Computerworld article. The quote from the article:

Scarafino compared it to eating one's seed corn.

Makes more sense this way. You eat feed corn (or rather, livestock does); you save your seed corn to plant next year's crop. Eating your seed corn is thus a very bad, short-sighted thing.

supercomputer research doesn't do me any good...

[Freedom Bug]

Money spent researching beowulf type systems advances the start of the art of Linux, communication systems and other stuff related to what I do. Supercomputer research only benefits me peripherally.

Sorry, I'm selfish, but I like the previous status quo.

Bryan

Re:A lost metaphor brings out my inner language na

[LionMage]

Um, no, read the article. The analogy used in the article was that of "eating one's own seed corn." As in, doing something stupid and short-sighted that will cost you in the long run.

Idiots these days.

Complex issues that have to be solved

[Anonymous Coward]

I've been in this field over 25 years, been in public position at a major lab now for 8.

If this was a simple issue, the HPC community would already have completely moved to clusters and never looked back 3 or 4 years ago. But it's not kiddies.

Want to run a physics projection for more than 1 microsecond? Takes real horsepower that clusters cannot provide even distributed. Just too much damn data. Chem codes that include REAL data for useable time slices? too slow for clustered memory. Every auto maker in the world (almost) has been whining about the lack of BIG horsepower for a few years now.(crash codes and FEA) I could go on forever. Sure, some problems work awesome on clusters, which is why we have them. But definately not all of them.

The problem is partly diminishing returns, partly the pathetic ammount of useable memory on a cluster and its joke for memory throughput, partly the growth in power of the low end and clustered networking, partly the ridiculously long development cycles invloved in High Performance Computing and the low $ returns,

One of the biggest things congress sees is that this country will more than likely NEVER again lead the world in computing power for defense and research.

And thats something we ought to do as the last real Superpower.

The national labs TRIED clusters, they don't get all the jobs done they wanted. (see testimony before congress, writings in HPC jounals, and the last couple RFPs from US gov. labs,heck every auto maker in the world) People in HPC _know_ it now, but having let what little there was of the supercomputer industry die out, there isn't mcuh of an industry left to turn to now. It just may be too darned late. HPC hasn't been a money making industry since the early 80s.
Heck, even Intel abandoned their clustered machine they custom built for the government.

Most folks in HPC will readily admit the Top500 is kind of a joke. The HPC-challenge #s are a little more realistic for the tests, but we really do need something that approximately real world applications, not just a 70s cpu benchmark.

For those that think this is a 'Linux wins' issue,
consider that mostly it was fast interconnect networks that allowed clustering, not the OS. Examine the history of clusters and you'll see this is true. Btw, the last few SC companies are already mostly moving to linux anyway.(nec,fujitsu,cray;ibm dabbles in hpc)

Hopefully the industry will survive long enough to allow for even better mergers of supercomputing power with low end cost, but at this point I doubt it. Cray has been on the ropes since 96, fujitsu's sc division is a loss leader, and NEC has been trying to get out of it for a while for something with a margin.

Ed -gov labs HPC research punk
-former Cray-on
-former CDC type

Re:Complex issues that have to be solved

[wintermute42]

What is interesting is that at least some of the government labs, while not using clusters, are using processor grids. As the parent post notes, these have problems too.

I attended an unclassified talk on the implementation of a neutron transport code at one of the Labs. While some of the mathematics of neutron transport was over my head, what impressed me was the effort and creativity that went into developing a massively scalable solution. The person who developed this code came up with a very elegant solution that tiles across massive processor arrays.

As far as I can tell, the modus for high performance computing seems to be to throw some very bright hardworking people at the problem and hope that they come up with a massively scalable solution.

Another course would be to develop software, languages and other tools to aid in parallel programming. Beyond some libraries to support parallel programming this kind of support does not really exist. And since there is very little market for massively parallel processors, there does not seem to be much prospect for the emergence of this software.

Re:Complex issues that have to be solved

[jsac]

Here's the problem. On codes which need lots of data interchange, communication speed becomes the bottleneck. I don't know of anyone running a serious fluid dynamics or weather code, which are this kind of data-interchange-limited application, who gets anything near peak performance on "real-world" problems using ASCI machines. Sure, ASCI White (a 10000-node cluster) was billed as a 10-Teraflops supercomputer. Who cares, when you get 10% of peak performance if you're lucky? NOAA wanted to buy a supercomputer in the mid-90s, for weather and climate simulations. They did the requirements analysis and decided that a Japanese vector supercomputer was what they needed -- nobody in the U.S. made them anymore. Seymour Cray flipped out -- a government organization buying foreign supercomputers? heresy! -- pulled a bunch of strings, and very soon thereafter Japanese supercomputers faced a stiff tariff because the Japanese were "dumping" their product on the U.S. market. Of course, that meant NOAA couldn't get their NEC. They ended up buying some American-made cluster and getting their piss-poor 5% of peak performance. Well, two years ago, Japan brought Earth Simulator online. It's cluster of 5000 vector processors; it boasted 30 Teraflops peak performance, which was 3 times as fast as the then-current number one machine, ASCI White. And a group from NOAA went over to Japan on invitation to check the machine out. They spent on the order of a week adapting some of their current codes to the ES architecture and fired them up. And got 66% of peak performance right off the bat. How'd that happen? Well, ES cost on the order of $100 million. (By the way, as a rule, if your 'supercomputer' cost less than $10 million, it's not really a supercomputer.) Of that, about $50 million went into developing the processor interconnect -- it's a 5000-way(!) crossbar, for you EE types. With an interconnect that big and fast, the communication bottleneck which dooms the big physics codes suddenly disappears. So, yeah, the U.S. supercomputer market at its own seed corn. To see Earth Simulator jump to the top of the Top 500 was something of a slap in the face; to see it get 20 Teraflops on real-world problems was a terrible blow to the prestige of the U.S. supercomputing community. And not one we're going to easily recover from.

Re:Complex issues that have to be solved

[Phaedra]

Look, a particular problem cost-justifies a superomputer or it doesn't. It would appear that for many problems, a cluster is a better solution. It would also appear that there are many more of those problems than problems that require a supercomputer-based solution. If these latter problems are important enough, companies/goverment will be willling to pay and someone will step up and provide a supercomputer solution. I don't think we need to bias the situation by artificially propping up the supercomputer industry.

Government/Industry will either accept the lower performance/throughput and drastically lower cost of a clustered solution (because it is good enough) or they will pony up the big bucks for the big iron (because they really need it). If the supercomputer industry needs to rationalize to survive in a market with lower demand, so be it. Welcome to the Real World(tm).

Just my $0.02

Re:Complex issues that have to be solved

[Anonymous Coward]

can I have some free money too, please?

Clusters and supercomputers...

[gillbates]

I've seen a lot of naive comments suggesting that supercomputers are being replaced by clusters. The truth is, anyone who can replace their supercomputer with a cluster didn't need a supercomputer in the first place:

1. (compared to a supercomputer):
2. The prime advantage of an x86-based server is that it is cheap, and it has a fast processor. It is only fast for applications in which the whole dataset resides in memory - and even then, it is still the slowest of the group.
3. Clusters are a little better, but suffer from severe scalability problems when driving IO-bound processes. As with the x86 server, if you can't put the full dataset into memory, you might as well forget using a cluster. The node to node throughput is several orders of magnitude slower than the processor bus in multiple CPU systems. (6.4GB/s vs 17MB/s for regular ethernet, or 170MB/s for Gigabit)
4. Multiple CPU servers do better, but still lack the massive storage capacity of the mainframe. They work better than clusters for parallel algorithms requiring frequent syncronization, but still suffer from a lack of overall data storage capacity and throughput.
5. Mainframes, OTOH, possess relatively modest processors, but the combined effect of having several of them, and the massive IO capability makes them very good for data processing. However, their processors aren't fast at anything, and often run at 1/2 or 1/3 the speed of their desktop counterparts.
6. Supercomputers combine the IO throughput of a mainframe with the fast processors typically associated with RISC architectures (if you can still consider anything RISC or CISC nowadays). They have faster processors, more memory, and much greater IO throughput than any other category.

It used to be that the prime reason for faster computers came from the scientific and business communities. But now that the internet has turned computers into glorified televisions, the challenges have gone from that of crunching numbers to serving content:

1. Clusters are great for serving read-only content, because there's very little active synchronization required between nodes, and the aggregate IO capacity scales well.
2. Mainframes reign when it comes to IO throughput - companies that formerly had use for a supercomputer are finding that their role is shifting to more of an information-provider role; faster processors are no longer as important as fast IO subsystems.
3. Scientists aren't being trained to use the computer as a tool; most think of a computer more or less as a means of verifying their hypothesis, rather than a means of discovering possible explanations. Their primary work is done with a calculator and pencil, and only later, when they need something to back up their ideas, do they turn to a computer simulation. The computer is a verification tool, not a means of discovery.

As our economy has shifted away from a technological base to an entertainment one, the need for supercomputers has begun to evaporate. We outsource innovation overseas so that we can lounge around on the couch watching tv and drinking beer (or surfing the net and drinking beer). The primary purpose of technological innovation has shifted from that of discovering the universe to merely bringing us better entertainment.

Re:Clusters and supercomputers...

[Wizzy Wig]

"...if you can't put the full dataset into memory, you might as well forget using a cluster. The node to node throughput is several orders of magnitude slower than the processor bus in multiple CPU systems. (6.4GB/s vs 17MB/s for regular ethernet, or 170MB/s for Gigabit)"

There's the argument in a nutshell. A cluster ain't worth shiite to a modeler who needs to move petabytes of contiguous data in his algorithms.

Re:Clusters and supercomputers...

[Anonymous Coward]

Hey, at least there are still a few of us who prefer Scotch.

Re:Clusters and supercomputers...

[Anonymous Coward]

Scientists aren't being trained to use the computer as a tool; most think of a computer more or less as a means of verifying their hypothesis, rather than a means of discovering possible explanations. Their primary work is done with a calculator and pencil, and only later, when they need something to back up their ideas, do they turn to a computer simulation. The computer is a verification tool, not a means of discovery.

As a scientist in training, I can say that this is absolutely NOT the case. Most scientists would kill for a HPC cluster. The problem is one of funding. Most universities don't have to funds to go out and buy every department a 64 node cluster. If you have ever done any sort of work at a National Lab, you'd know that most everyone (who does numerical stuff) has access to some sort of cluster or other. In fact, a lot of the people that I work with probably do more coding than they use calculator/pencil

Re:Clusters and supercomputers...

[Anonymous Coward]

"The truth is, anyone who can replace their supercomputer with a cluster didn't need a supercomputer in the first place"

Succintctly, and accurately, put.

"Scientists aren't being trained to use the computer as a tool; most think of a computer more or less as a means of verifying their hypothesis, rather than a means of discovering possible explanations. Their primary work is done with a calculator and pencil, and only later, when they need something to back up their ideas, do they turn to a computer simulation. The computer is a verification tool, not a means of discovery."

Even worse, if your putative scientist isn't a computational scientist, he/she either has an unrealistic expectation of the fidelity of any numerical simulation to reality OR has a deep-seated and pathological distrust of ANY large-scale calculation.

I agree-- clusters are limited

[Anonymous Coward]

One technology that I work with is called Artificial Life and is basically large evolutionary software simulations. (This is not exactly the same thing as genetic programming, but it's close.) This is an example of something that just plain doesn't cluster well. Try to cluster one of these, and you will max out a gigabit switched LAN in less than a second (I've done it!). I've even maxed out a gigabit "star configuration" LAN with this stuff. It just doesn't cluster.

The problem is that these simulations involve many cells that must interact with each other in real-time. The cluster spends 90% of it's time waiting on other nodes no matter how you build the architecture.

There are lots of other problems like this that just don't cluster well.

Clustering only works for problems like protein folding or SETI that divide up into neat "work units" that can be shipped out and then returned. Millions has been spent, along with massive amounts of time by people like myself, and we're still no closer to being able to really cluster applications like this efficiently.

Re:I agree-- clusters are limited

[Anonymous Coward]

bullshit. ALife is one application that really clusters well if you know how to do it.
its relatively easy to cluster an environment like a ALife world since each member of that world only references the local surroundings 90% of the time.
its the coder, not the application.

I have the next-gen supercomputer HW designed...

[anactofgod]

and close to being ready to go into production. But I got sidetracked digging these tunnels out from my basement.

Sorry.

Seymour

Cray-1 vs. Pentium 133 vs. nVidia graphics card.

[billstewart]

Sure, the Cray-1s were cool. But their CPU power was about the same as a Pentium-133. They did more I/O, but it was only about 100 MIPS. I forget how fast the Cray-2 and the X-MP and Y-MP were, but your basic this-year's graphics CPU from NVidia or their competitors has a fairly similar amount of CPU and memory bandwidth - the RAM's usually smaller, and most of the calculations are 32-bit rather than 64-bit or 80-bit, but the horsepower's pretty similar and you can think of the 3 GHz Pentium-4 as a channel processor fetching stuff from RAM and disk drives if you'd like.

The Cray-something (2? YMP?) I visited at Livermore Labs back in the late 80s was really cool, for a now 17-year-old machine. It had a ~$2000 AT&T 80286 PC on top as a console controller, and the terabyte of data storage was an 8-foot cube tape robot from StorageTek. Today you could put the $2000 PC on top as the main CPU, including a terabyte of disk drive.

Like a vector computer will do any better

[Anonymous Coward]

Vector computers require extremely regular code, which is almost trivial to parallelize ... yes they tend to have a bit more bandwith too, for twizzling, but they are not suited for hard to parallelize irregular problems either.

Besides, it has always been the Asians which were mad about SIMD ... and the Americans who were more into MIMD. These people dont want things to go back to how they were, they just want pork and they will say anything to get it.

That should be SImD, not SIMD

[mangu]

The PCC and Pentium4 do have "single instructions multiple data", but "multiple" == 4. That's not a remarkably large multiple. One possible research topic would be increasing the number of data elements that could be processed by the same instruction in the CPU. But note that to get a real gain from that, one would also need parallel access to memory, in order to load those data into the CPU, another good goal for a research project.

Hammertime

[Graymalkin]

There is no super computer technology crisis, there is however a paradigm shift happening in the supercomputer market. Twenty years ago building your own supercomputer, even a loosely coupled cluster, was not a very viable option for most research institutions. Today this option is not only viable but often exercised.

Obviously the big SC vendors and designers seeing less business roll their way, why pay them tons of money when you can have grad students assemble your cluster for the price of some pizzas? That isn't to say SC clusters are the end-all be-all of computing but they're very useful and relatively inexpensive. Realistically they're simply an extension of what Cray started with their T3D supercomputer. The T3D was very impressive in its days but now the technology to build such systems is in the hands of just about everyone.

Taco: What the hell is up with the IT color scheme? This is even worse than the scheme for the Games section. I know the Slashdot editors don't actually read the site but other people try to and we're not all colorblind or reading from grayscale monitors.

From an engineering perspective...

[mkiwi]

The real limit to super computing is the implementation of logic devices (NAND, NOR, etc.) at significant clock speed. The only way developers currently have to create a faster computer is analogous to Apple's "Megahertz Myth"- with a shorter pipeline and slower clockspeed.

Several engineering issues making this Ludicrously LSI operation possible have not been solved. One type of logic found in computers is called CMOS, the other called TTL. TTL has another special implementation that allows it to achieve high clock rates.

This special TTL that is used today has reached its peak at .09 nanometers in width. The semiconductors used today do not have enough "gas" (voltage) to power such a large system. Every transistor has a voltage drop within it, and adding more and more transistors only increases heat, power consumption, and probability of failure. If a single part of a huge new chip were to fail, the entire chip would have to be replaced.

Ever wonder why CPU's run hot? They dissipate voltage (and with it current, which is the True Power). At some point, the input voltage on the processor will be so close to the output that measuring HIGH or LOW (1's and 0's) becomes impossible. Until some scientist discovers a brand new material that can overcome current and voltage barriers, clustering will be the norm. Said scientist would be a billionare if they could patent a material like that.

Researchers are out looking for these new substances, mind you. Until they find one and build a semiconductor company, single computers with low clock speeds and high bandwidth will not be able to tackle what the human imagination has instore for them. There is also a limit involving physics and microwaves that I could get into, but i'm not a physicist. I'll leave that up to them :)

Re:From an engineering perspective...

[stevesliva]

This special TTL that is used today has reached its peak at .09 nanometers in width.

Wow, that's some pretty miraculous stuff, considering modern processors are CMOS, with a minimum feature size being gates that are 90 nanometers in length. And for chrissakes, it's not like Intel hasn't already sampled 65nm SRAMs (albeit small) or anything. CMOS has at least another decade left in it. An intimidating crazy decade with unforseen problems, but heck, it pays the bills.

Re:From an engineering perspective...

[beowulf2003]

have not been solved. One type of logic found in computers is called CMOS, the other called TTL. TTL has another special implementation that allows it to achieve high clock rates.
What are you talking about ? TTL is a bipolar logic family and has a handful (10-20) gates (NAND/NOR/INVERT) per chip. Each and every microprocessor or ASIC today uses the CMOS logic. And even though some folks are running around talking about the end of CMOS, these folks don't seem think so. They say we can go on till 2018 drawing gate lengths as small as 22nm (or 18nm effective) (Page 15 in the executive summary PDF) since just about everybody in the semiconductor industry is involved in coming up with this roadmap, I think it just be have some validity ...

I have one word for you...Quantum Computing!

[anactofgod]

Two...I have two words for you!

Seriously, I don't see the problem, so long as companies like IBM and (dare I say it) Microsoft continue to do research in this area. That is the real value of companies that are committed to *real* research in revolutionary sciences and technology.

Of course, US companies don't have a hammerlock on this research. There is a lot of work being done internationally in the area, by corporations, and by educational/research institutions.

---anactofgod---

Re:I have one word for you...Quantum Computing!

[Leers]

Yeah, but thats at least 20 years away from being a practical technology

Re:I have one word for you...Quantum Computing!

[rumblin'rabbit]

I agree, except that the applications for quantum computing are limited right now (eg, determining primes, secure computing). Fluid dynamics, a good candidate for super computing, is (so far as I know) not on that list. No doubt, however, that the applications for QC will expand with time.

By the way, I was told by an expert in the field that if someone claims they understand quantum computing, then they don't understand quantum computing.

So basically it is the interconnect ...

[Anonymous Coward]

The use of commodity hardware for interconnects is indeed the real problem ... what the government should promote is faster interconnect architectures. Fund people like Virginia-Tech who are applying InfiniBand to clusters ... and promote the development of even faster interconnects for inside systems, and between nodes. That would be a good use of money, not giving out pork to have full custom extremely simple vector computers build. That doesnt promote innovation, vector computers are a step backwards.

Earth Simulator is an example of what not to do, a quarter billion spend just to get on the top of the list and getting the ability to run a very limited set of applications very fast. If they had given Virginia-Tech 50 million last year then it wouldnt have had to top spot on the supercomputer list today.

BTW the one thing I think would be a worse idea than promoting vector computers is promoting NUMA BTW :) NUMA scales poorly, you have to throw a lot of hardware at it ... it is a good architecture for lazy programmers. Real parallel programming is still done with MPI and its descendants.

Re:So basically it is the interconnect ...

[fitten]

There are a number of low latency, high performance interconnects out there. The problem is that they are *still* expensive because they don't reach economies of scale that the commodity stuff (Ethernet) does, and won't reach those scales for some time. Not everyone is going to run out and buy Myrinet cards and switches for their home machines so the prices for those are going to be high... maybe not as high as a Cray T3E connecting the same number of processors, but still pretty high. And then... you are limited by the speed of the PCI bus (and PCI-X and PCI-E etc.) What we eventually need is something that is as fast as the processor with near the same latency to memory as the processor for some problems and guess what that turns into.... a form of NUMA (yeah, I read your NUMA comments below :)

BTW the one thing I think would be a worse idea than promoting vector computers is promoting NUMA BTW :) NUMA scales poorly, you have to throw a lot of hardware at it ... it is a good architecture for lazy programmers.

Well, NUMA allows folks to use the single system image and shared memory programming paradigms. Some problems map to those pretty well.

Real parallel programming is still done with MPI and its descendants.

Thanks :) I was involved with MPI in several ways some years back.

Re:So basically it is the interconnect ...

[Anonymous Coward]

Well Im sorry to hear you have lost your religion :) I see shared memory programming becoming the next C ... the biggest producer of unnecessary bugs of its time.

Shared memory programming with software managed access controls offers far too much control, very little of which is really needed. I think languages and architectures should be developed around the concept of having either 1 processor with access to a given piece of memory, or no processor having write access to it. Any other way of accessing memory is indicitave of a bug anyway.

I rather like Milo Martin's token coherence for that reason.

Solving the Wrong Problems with Other People's $$

[billstewart]

The Ford guy wants the Feds to cough up $200M in tax money to develop machines that solve problems that Ford and a couple of other people have, instead of either Ford spending the money themselves or the Feds spending money getting the industry to develop computers or software that solve problems that far more people have which could be much more benefit to industry as a whole. I'm not convinced that the Feds should be doing that either, but since the Weather Service and Nuclear Weapons Designers seem happy with big clusters, the opportunity cost of spending my money designing a computer for building better SUVs vs. a computer for better medical research or whatever seems an obvious bad choice.

In particular, he wants a 2000s-version of an 1980s architecture running a 1960s language. For $1M, he could train his technology guys to use newer programming techniques. Yes, I realize that Fortran 90 is newer than Fortran 77 which is newer than Fortran IV which is newer than Fortran 1, and that the biggest CPU job these guys do is usually crunching big matrices of floating point numbers. That's a job for a subroutine you write once and feed with data and user interfaces that are written in languages that are more efficient for prototyping and user interface design.

Since when is /. a capitalist playground?

[twigles]

Is it because there is a perceived zero-sum game being played between Linux-based clusters and supercomputers? Hey, let's take a reality check here, a lot of research is not directly applicable. In fact, I've read numerous discussions on /. railing against the MBAs and the Bush regime for not funding anything that doesn't turn a profit within about 18 months or have something to do with killing brown people.

Letting supercomputing die may be harmless, after all, the US doesn't have to be the best at everything in the world and some other country will fund the research. But from some of the more coherent posts I've read, it seems like supercomputing has a definite niche in the natural sciences, something we should be pushing for a better society - learning for learning's sake - and paying for out of public coffers. My taxes go to a lot of shitty things I'd rather them not go to, like subsidizing Haliburton with no-bid contracts. Why is it so offensive to /.'rs that the country as a whole subsidizes advanced computing? Isn't computer science all about seeing what can be computed? Letting supercomputing die because it's expensive seems like an extraordinarily short-sighted thing to do.

Re:Solving the Wrong Problems with Other People's

[billstewart]

Oh, and yes, I'm a Linux fanboy, but I was also reading comp.arch (remember Usenet?) back in the days when the Attack of The Killer Micros was starting to kill the minicomputer and mainframe industry ("careful with that Vax, Eugene!") and RISC vs. CISC was still a design issue, so I do have some perspective on the game.

Re:Please make it stop!

[Carnildo]

It might just be my Powerbook monitor, but I see the games section colorscheme as a very nice-looking blue gradient on white.

Moreover I think those industry panelists...

[Ayanami Rei]

...must have never heard of FPGAs. That's where supercomputing is REALLY headed. Vector processing is a specific application in that sense... as is "a bunch of FFTs" or maybe a dedicated MPEG2 block encoder.

That's the beauty. You architect the local hardware to better implement a large task.

Re:Moreover I think those industry panelists...

[Marxist Hacker 42]

And there's no real reason why you can't do that with a cluster as well- archetecting both the hardware and the software for the cheapest possible construction.

Reminds me of my early programming days on the TI99/4A. The brilliant bit that made that computer more powerfull than most other micros on the market at the time (except maybe the Commodores and Ataris, but neither of those had this as much as the 99er did) was multiple specialized subprocessors. Most others had *maybe* a video processor and a sound processor, but the TI also had a memory manager and an I/O management chip and a speech synthesis chip as well. A good assembly level programmer had his own cluster supercomputer- even if it was only running at 3.44Mhz. Better yet, the PEB allowed you to build your own specialized cards for additional tasks- Disk I/O had it's own processor, as did Serial and Parallel communications.

We do this today with off the shelf consumer hardware thanks to USB, AGP graphics cards, and separate sound processors on sound cards. To truly take over the Supercomputer market, that's the paradigm we need to get back to.

Re:Moreover I think those industry panelists...

[Detritus]

FPGAs are slow, expensive and inefficient when compared to normal chip designs. Their only saving grace is their programmability.

Re:Moreover I think those industry panelists...

[The Conductor]

For the more common SRAM and EEPROM types, yes. The anti-fuse type, however, are not too bad on price-performance: 500 MHz clock and 2 million equivalent gates on-chip. Still only about a third of 90 nm CMOS (not counting Intel processors that are specifically architechted for high clock frequency) , but if you have a supercomputer budget you can use more of them and architect for more instructions per clock. They never became popular because you can't erase them, so prototyping is a pain. Imagine trashing a $15 chip every time you recompiled software.

waste of computing power?

[Anonymous Coward]

Call me a lunatic, but I've grown to be skeptical of large parallel computing, mainly because the economics driving the "research" detracts from useful applications.

o optimiizing the shape of a potato chip (maybe that was tongue-in-cheek in the article?)
o optimizing the wind profiles of our heaveier, bulkier vehicles, which is IMO a round-about way to attempt to reduce our dependence on fossil fuels.
o predicting the weather at the level of 1 km so that you can make a weather forecast product to sell. Severe weather monitoring in the upper midwest always seems to err on the side of over-protectionism (or is that cover your a**?).

Give me a reason to want to support supercomputers.

It is a myth

[Anonymous Coward]

You really think a heavily subsidized high end market with revenues a couple of orders of magnitude lower than the mainstream can develop technology better? It is just a nice justification to give boys on top leeway to spend money on toys.

The next big step in computer architecture will be Cell, and you know what is funding that step? Consoles and consumer electronics.

It's also called "FPGAs". (nt)

[Ayanami Rei]

Memory latency matters. A lot.

[j. andrew rogers]

Shared memory can easily be simulated on a gigabit network

Not just no, but hell no. One of THE major points of having shared memory is that such systems have an order of magnitude (or two) lower memory latency. Seeing as how memory latency defines the performance bound for most apps that just happen to run well on shared memory systems, there is no way you can reasonably use gigabit network fabrics as a substitute. Unless you can afford greatly reduced performance to save a few bucks.

When gigabit networks have typical latencies like that of a ccNUMA system, this will be worth considering. But we are still pretty far from that today, and it isn't a trivial engineering problem to get there.

So what?

[carlos92]

Are they supposed to be immune to the risks associated with technological changes? I am not a citizen of the US, but if I was I woudn't want to be paying for a supercomputer when a cluster is good enough.

Moore's Law does that to you

[billstewart]

Scaling commodity technology is one way to fundamentally advance the science of computing - much of the science is learning about how different things scale and what ideas work or don't. While not all of the lessons we've learned in the last 25 years of computer development are present in the typical high-end graphics PC, Beowulf cluster, or Nintendo gamer box, many of them are. Even if you could afford a Cray 1, you probably wouldn't want something that slow and lame - a 1990 Cray 3 was about 4.5 GFLOPS, so that's still a bit bigger than what you can put on your desktop for 0.01% of the price, but not by much, and that Pentium-4 is all yours, as is the nVidia graphics processor on it which is really a lot faster for some tasks.

One big paradigm shift is that computers that used to cost enough money that they'd take dozens of people to manage and hundreds to feed them data and years to design database applications for are now commodities that sit in your pocket or on your desktop and can be used for problems that you'd never have applied that level of resource to back in the 80s or the 60s. That means we need to relearn how to interact with users so they can take advantage of the speed, and we're only starting to crack into applications that can use some of the CPU speed, like video processing.

Another big paradigm shift is the connectivity of the Internet - you've got more data bandwidth on that cheap cable modem than the first N years of the Arpanet, and basic communication is basically free. That links the world together, for good and bad - it creates opportunities for Nigerians to sit in cybercafes near their homes and scame untold wealth from suckers around the world, and lets people have freedom of speech in ways that their governments have serious difficulty controlling or preventing.

Mooo... Biocomputing is the way?

[MooseByte]

"We're in need of a paradigm shift. Where's the next Seymour Cray?"

Right here? :-)

Seriously, doesn't biocomputing seem like the next quantum leap in computational power? More than just storing data in cows, of course. Actual computational systems implemented in wetware. Why bother retreading the same old tired silicon, and instead focus research on the stuff that can leap us ahead an order of magnitude or so.

Two sides to this...

[crazyphilman]

Ok, on the one hand, "real" supercomputers cost a bunch and go obsolete on you periodically. So you could spend tens to hundreds of millions of dollars, build out a location to receive the supercomputer, hire a staff and security to babysit and control it, hire company consultants to come in and set it up, and then repeat the whole thing in a few years when it becomes obsolete. Or you could use a cluster, which you can upgrade gradually over time with additional units (or faster units), assemble and staff with "mere mortal" graduate students (in CS), and stash pretty much anywhere you've got a spare storeroom (I'm exaggerating, but only slightly). "They" say there are things you can't do with a cluster, but every problem has a solution (provided you're not driven by the profit motive, ha ha).

On the other hand, what if they're right and there are some tasks that can't be easily done by a cluster? Then it might make sense to subsidize at least one or two supercomputer companies to maintain that skill set.

Here's a thought: Let's say it's determined that most tasks work just fine with clusters. Supercomputer companies find that there isn't enough business to support them anymore. They start to fade.

What would stop the Federal government from coming in and hiring their staffs to work in a nationally-owned and operated supercomputer center? Pick a location, set up a prototyping factory, and build a research center whose sole purpose is putting out the fastest computer ever made. Sort of a comp-sci NASA. They could aim at creating a new system every five or ten years. Each current system is maintained and run by one team while another team works on the next one, and the feds lease the systems out to researchers able to pony up a few bucks to help support the setup. Old supercomputers put out to pasture can be rented out for more mundane tasks.

Isn't that kinda interesting? It gainfully employs all the supercomputer related computer scientists so they don't move to some other country out of frustration, it preserves their knowledge for our future use, and it lets everyone get back to working with their clusters, except in those cases where you don't want to use a cluster.

Re:Two sides to this...

[Animats]

What would stop the Federal government from coming in and hiring their staffs to work in a nationally-owned and operated supercomputer center?

We already have a whole chain of federally funded supercomputer centers. San Diego CA, Huntsville AL, Pittsburgh PA, Fairbanks AK., etc. San Diego is generally considered to do serious number-crunching work. Huntsville is installing firewalls for grade schools.

Re:Two sides to this...

[crazyphilman]

Yes, but they work ON supercomputers, they don't build or design them... Do they?

I was talking about a center whose purpose was the creation of ever-more-powerful supercomputers. The rental section would just be there to make use of the tech, and put it through its paces.

Re:Memory latency matters. A lot.

[Marxist Hacker 42]

You can defeat latency with extremely small packet size and by minimizing the network layer envelope. A supercomputer only runs one process- even if that process is multithreaded. To mimic the supercomputer's memory, your cluster should only use the network for a single process- the virtual shared memory driver. Don't bother with TCP/IP, it's way too wraper intensive, to send a single byte you need to wrap it up in 300 bytes of envelope info. No, go down to the hardware packet level instead, with a "Queen Bee" that only does memory managment serving cache-sized (and no larger) packets to individual CPUs. Then your network latency would be VERY low- because you're simply not using all of the bandwidth available. True, it'd still be slightly higher than pure memory- but the increased number of nodes will cover that up in a hurry, especially if you code for specialized machine processes rather than a single general processs.

Re:Maybe there is no market Need for a Supercomput

[mabhatter654]

that would be the POWER5 processor used in IBM big iron. They sell boxes scaled to 64 procs right now...unfortunately, those are multi-million dollar boxes already.

I'd agree that we need supercomputers, but that's not quite the same way the "industry" does. Let's face it, there are very few everyday uses for supercomputers anymore. It would be fun to see a new round of "supercomputers" but frankly all of them put together don't have the technology resources to beat companies like Intel & AMD...let's not forget that both companies have recently built 2 BILLION dollar fabrication plants that are only slated to be state-of-the-art for 3-5 years tops... The dollar expenditure to build top end chips is large it's not profitable unless you can sell millions to get back your money. I'd like to see makers like AMD release "cell" motherboards that unchain the chips from arbitrary restrictions of chipsets and allow end users to connect chips like Opterons in more "novel" maners...considering an "average" Opteron server board can support 4 processors and 16+ GB of ram, the issue isn't a technology lead, but rather getting "lego-like" processing components into the hands of truely creative people again...

The novelty of structure is what made Cray so fast, not necessarily the individual components...the "supercomputer" industry lost touch with that ideal segment a long time ago.

less lawyers

[robindmorris]

... is a smaller company (i.e. less lawyers)

"less" is used with continuous quantities ("less flour"); "fewer" is used with discrete quantities ("fewer marbles").

So "less lawyers" implies that lawyers are continuously divisible. I'll have 2.7223 lawyers, please. Hmm, maybe you have an idea here...

So what

[DarkOx]

What does it matter if we don't develop single unit supercomputers. Clearly in a free market if these thing had value they would be persued. There is not predetory tax laws on supercomputer, or any other regulations on domestic use. The only reason development has slowed is there is not much market for the beasts.

There are many reasons for that too, for one other then in stealer, neculear, mathematic, and bio research feilds few industries need more computing power then can be had off the shelf any day of the week. That was not true yesterday it took all sorts of custom hardware to make CGI happen in films that can be done now in my basement in resonable time frames. So no more super computer market there the ROI is gone I am sure this plays out in all sorts of other engineering feilds as well.

Many places where you do need super computing power can be done with clusterd systems that are cheap to build and cheap to maintain.

At least people in the pure science and research fields have learned to be better thinkers and programers, they found ways to do things in parallel that were traditionally serial. Things that still are serial can be made to work on a cluster, sure it might take longer then a single computer considered to be equal FLOPSwise but considering I could either spend all the money I saved makeing my cluster bigger and more powerful so I can get back to equal time or on other profitable efforts while I wait there is again no ROI.

It so happens that may of the most interestin questions in math, physics and computer science such as quatum theory need massive amounts of parallel work, rather then serial so that works better on a cluster anyway.

If there is a real reason to do it people will build supercomputer, because there is nothing stopping them other then economics. No need to fear Supercomputers are not going away. Everyone else that needs that kinda proc-ing power will settle for clusters, as well they should. This is just another largly obsolete industry wanting someone to bail them out because they have failed to adapt to a changing market. If they are going to die we should let them, just like we should let the Universitys adapt or die, and the RIAA needs to adapt or die, we need to stop proping up obsolete undustries so new ones can replace them!

Re:supercomputer research doesn't do me any good..

[Anonymous Coward]

Most of what passes for state-of-the-art desktop CPU design are techniques first implemented on high-end mainframes and supercomputers of the 60s, 70s, and early 80s. That fact that my sub $500 office PC performs so well is due to the improvements of the semiconductor process engineers over the decades. (And here's to the process guys! Great job!)

So what happens if the process guys can't keep delivering better performance for the same old CPU designs? Where are the new, high performance CPU architectures going to come from if not the supercomputer and big-iron mainframe folks?

I lament the US government spending itself farther and farther into debt. I would lament more the last US supercomputer company dying off, leaving Intel to have to hope for new architectural ideas from NEC. I mean, sure Itaniunm is junk, but at least Intel didn't blow a decade learning that you can't write an OS in Prolog.

NOT insightful!

[pla]

Its the fact that clusters require higher skill to program efficiently for than do single processor systems.

...vs This year's topology from Cray or SGI or the like. Yeah, whatever.

If I know MPI (which I do), I can code on any cluster out there, and it will perform similarly, with a linear time dependance on network speed. Including most supercomputing architectures. By comparison, if I know HPF, well, I can write code that will at least run on most supercomputer topologies, but run efficiently? All at the whim of the manufacturer, and whether or not they had my particular problem (or a similar variant) in mind when optimizing their compilers.


The other point is that by using off the shelf parts, we are not really innovating in this space like we should be.

Define "should"...

Customer X wants a supercomputer to perform a few set tasks. He can pay you to "innovate", or he can pay your competitor 3% of your costs for a setup that will wildly outperform your top-of-the-line machine. And you see a problem with using commodity parts?

An important part of "innovation" involves making the best use of what you have available today. Sure, tomorrow that may mean quantum coprocessors, and at the absolute limit, it may well mean coming scarily close to breaking the laws of physics.

For the present, however, it means using a cluster of Opterons.

Re:NOT insightful!

[javiercero]

SGI, CRAY or whatever offers you a single memory image, which is a VERY different thing than a separate image cluster of commodity parts. Two very different things.

Oh, and not every cluster out there uses MPI BTW.

But by all means, you know everything there is to about computer architecture, so I guess we should just constrain ourselves to clusters of Opterons... Jeez, I guess we should have to make due with a dual 486 and call it a day. Afterall, what is the point... right?

Re:NOT insightful!

[hxnwix]

The point is solving problems, not architecture fetishes.

Handouts

[BeagleBoi]

The entire story is contained in this line:

"The U.S. House of Representatives this month passed two supercomputing-related bills: HR 4218, the High-Performance Computing Revitalization Act of 2004, and HR 4516, which seeks about $200 million in funding for supercomputer development at the U.S. Department of Energy."

It's just people wanting more government handouts for their pet projects.

Re:Handouts

[Anonymous Coward]

i.e., the Reps' home districts. Take for example the bill's co-author Jim Johnson. His Illinois district includes the National Center for Supercomputing Applications. I presume the others are in corresponding situations.

Re:Handouts

[BCW2]

Just remember why there are two programs. The DOE project is for weapons research, nukes, star wars, whatever. The other one is for general use, geology, ocean currents, weather, .....

Not a single SuperComputer company has lasted

[rlglende]

All of the $Billions the FedGov has invested in this effort have not produced successful commercial products. The only possible exception has been Cray Computer, which started failing after Seymour Cray left.

The FedGov's funding has, however, prevented the success of clusters based upon cheaper systems.

Lew

Feed Corn?

[condition-label-red]

Hmmm...I believe the term is SEED corn.

Beware the intra-agency politics here.

[deanj]

Beware the politics behind all this talk. Some of this MAY have to do with cluster vs. big iron, but I'm not entirely sure that's the only thing about this. There's politics involved. I'm not talking about Democrat vs. Republican here. I'm talking intra-super-computer center politics.

The Ohio supercomputer Center and the Pittsburg center are not currently part of the NSF Supercomputer program. NCSA (in Illinois) and SDSC (San Diego) are. The other two WERE part of the program, and they're still pissed about getting cut from the original program.

You'll notice a few of things in the article:

First, NSF funds the current supercomputer program, not DOE. DOE will get the funding (and make the award) for this program, if it goes throught.

Second, a Illinois rep is gunning for this. Presumably that rep wants to give NCSA a good shot at the big iron. (Pork barrel poltics, the old fashioned way). Presumably this is to keep NCSA going in case NSF ever decides to cut funding. Whether they actually get a DOE program remains to be seen.

Third, the other supercomputer center mentioned in the article is Ohio. Again, currently not funded under the NSF program. I'm SURE they're gonna gun for the new stuff, so I'm not surprised they're complaining about clusters. Whether they believe it's the wrong way to go or not, I'm not sure. I think this is purely to get the money going so they have a shot at it.

It'll be interesting to see whether this actually gets through the funding stage or not. It'll be a LOOONG time before this money is awarded. If they passed it TODAY, it would take a year and a half, maybe two, to actually award the money.

Re:Beware the intra-agency politics here.

[Anonymous Coward]

yeah, well, you have some of it right.

NSF's supercomputing centers have resources consisting of both more traditional sc's (IBM p690's and Crays at San Diego and Texas) and clusters (NCSA's forte). Overwhelmingly, NCSA and SDSC "customers" are NOT really supercomputer users. They are individual PI's who need (want?) a bigger computer than they can afford to buy.

They ONLY real competition for this money is between Oak Ridge and NERSC (DOE's current supercomputer center at Berkeley). Users at both these sites are real supercomputer users: they REQUIRE large memory, high memory bandwidth, and scalable interconnects to get their problems done AT ALL.
For them, a cluster, no matter how big and geeky cool, is essentially useless.

NCSA will never be a DOE site. No matter what. NCSA will continue to be a place where computer science (as opposed to computational science) is center stage.

Re:Beware the intra-agency politics here.

[deanj]

I do have to wonder why a Illinois Representative pushed on this so hard though, if she didn't think that her state would be able to stake some claim to that money. I really have to wonder what the modivation there is.

NCSA, to my knowledge, pushes a lot more computational science (weather, cosmology, etc), than straight CS on it's supercomputers and (now) it's clusters. There may be groups that do traditional core CS type of work, but it's not the PIs using the clusters/supercomputers. Perhaps I misunderstand what you're saying here.

The two things NCSA is most known for Mosaic and NCSA telnet, but that work was done in the software development group was broken up a long time ago.

Re:Beware the intra-agency politics here.

[Simkin1]

"Users at both these sites are real supercomputer users: they REQUIRE large memory, high memory bandwidth, and scalable interconnects to get their problems done AT ALL." -- Please define these problem metrics for me -- what kinds of memory are we talking about, what kinds of bandwidth (memory/interconnects)?
Somewhat annoying to have someone throw out a statement like this without any definition of what they're doing specifically. What software? Without that information, without knowing the problems, you'll never know if a cluster will work or not. I've heard this same FUD from colleagues who swore their software would never run on a cluster -- key note here -- We retired the last onsite Cray almost two years ago and my colleagues are finding they're MUCH more productive running their software on (*surprise*) a cluster. They too require large memory, high memory bandwidth, and scalable interconnects to get their problems done...

I love it when folks say their problems are too big for us poor little cluster folks. I really love proving these folks wrong.

Chuck Moore's Forth Chips have lots of power

[randall_burns]

Chuck Moore's Forth Chips pack an amazing amount of power into a small package. They aren't super computers-but they are some of the more interesting architecture intensive computing projects I've seen recently.

What bother's me here is it seems like the Government is trying to pick technological winners using corporate welfare instead of fostering real competition among US companies. As far as national security goes, I think there is a lot bigger fish to fry than the loss of the supercomputer business. The entire US technological sector is a mess-and much sensitive data is now completely outside the regulation and protection of the US government(either outsourced or under control of guest workers).

Re:Chuck Moore's Forth Chips have lots of power

[Anonymous Coward]

I like Forth. Wish we would hear more about Forth.

Business needs are not the same as science

[hung_himself]

It is sort of silly to compare what Google or the Bank of America needs and do to what scientists do though there is some overlap. In science there is a need for both FLOPS and throughput

For example we use a Linux cluster for protein simulations to try to figure out how proteins fold (or as you would say - test out our ideas about how proteins fold...). We are very aware of the bandwidth and memory limitations. That's why the software is written so that it fits into memory and so that there is as little IO betwen nodes as possible and thus things pretty much scale linearly with the number of processors. None of the work would be feasible without the CPU horsepower.

Another example is the particle physics lab that my brother used to work in also which used a cluster to process their events. An old mainframe then handled and stored the huge amount of data fed to it by the workstations (this was a few years ago). There they needed both the computational power to evaluate the data and the throughput to store it

In both cases, I suspect that in a business environment, buying a supercomputer would have been referable to noodling with the algorithms. I'm sure some consultant would have been hired to deem it necessary. And indeed we could use the extra IO but it's just not worth the cost - I'd rather use the limited resources we have on more processors and work around the IO limitations.

BTW, the last calculator I used had LEDs though I do like to work out equations on paper still...

Re:Business needs are not the same as science

[gillbates]

For example we use a Linux cluster for protein simulations to try to figure out how proteins fold (or as you would say - test out our ideas about how proteins fold...).

I would much rather you:

1. Get the funding you need to buy a real supercomputer, and,
2. Let the computer figure out "how" the protein folds through exhaustive algorithm analysis.

I understand that there's a practical limit to computer simulation; however, rather than merely testing potential explanations, we should have the computer itself iterate over the possible "simple" algorithms in the problem space, and quit when it finds a match with the observation set.

I must admit that I don't know exactly what you're dealing with exactly, but this is exactly the kind of problem that computer scientists love to solve. If you haven't already partnered with a grad student or CS professor, may I suggest that you do?

1. A computer scientist will be able to refine your algorithm to be more efficient, or perhaps know of a better algorithm. In my professional work, I have improved runtimes by four orders of magnitude in some cases. The value of algorithm refinement cannot be overstated.
2. A computer scientist will be able to point out those sections of code that perform redundant work, are inefficient, or non-portable.
3. A good computer scientist will be able to optimize the most intensive processing sections in assembly language. Typically, this will reduce run times by a factor of two or three, even when using optimizing compilers.

Re:Business needs are not the same as science

[hung_himself]

I don't think you have an idea of the scope of the protein folding problem. We *do* use machine learning techniques - quite extensively actually but the algorithm space is much broader than you can imagine and takes a long time to explore - hence the human. A supercomputer would be only of minor help at this exploratory stage in the game since only a few approaches are memory bound at least for proto-typing.

I also think that you are a bit confused about biologists - it's not like we just took a Java course. This is computational biology and all of us are computer scientists as well as biologists and the demands of the problem put us at the cutting edge of both fields. The code is not as clean or optimized as it could be but that's because they are prototypes. We want to test as many approaches as possible as quickly as possible. If it works then we can clean up the code at our leisure. GROMACS is an example of a more mature technology which has been optimized. This is the nature of scientific programming...

Interesting...

[Xabraxas]

It's true these "parallel processing" machines can go fast--Virginia Tech built the third-fastest machine in the world for just $5.2 million with 1,100 G5 chips from Apple Computer (nasdaq: AAPL - news - people ). But they have proven "exceptionally difficult to program" and problematic at certain performance levels, according to a 2004 study by the President's High-End Computing Revitalization Task Force.

Oh really. Don't blame me for not trusting a guy with that kind of potential bias.

metaphor schmetaphor - r e a d t h e w o r d s

[jh7459]

That sound byte is the only remarkable pro-mainframe thing anyone's said.. and while it is a nice quote its not really an accurate metaphor.. like when arnold calls his opposition 'girlie men', just draws the ear from the fact its completely nonsensical. its clear that in reality NO BODY'S CONSUMING 'the seed corn' - it has nothing to do with anyone 'squandering the future of supercomputing' .. it doesnt even make any sence. compare the ammount of research done on single vs parallel systems.. there is MUCH more potential growth for one of them, while one has 60 years under its belt... not that mainframes dont have their purpose... in a parallel system.. for the individual processes that require their power. the simple fact is most of the supercomputing users are tied to the gov't, and its now seeing all its support go away for its legacy systems, and all the govt can really do is make laws and tax people.. its doing what it always does to maintain its status quo

Vector Machines

[Detritus]

That old Cray could still kick the Pentium-4's ass around the block a few times when it came to memory bandwidth and making efficient use of the floating point hardware with large data sets.

When you have a data set that doesn't fit in the machine's cache, it doesn't matter how fast the CPU is.

Re:Vector Machines

[Fulcrum of Evil]

That old Cray could still kick the Pentium-4's ass around the block a few times when it came to memory bandwidth and making efficient use of the floating point hardware with large data sets.

That's why you go buy a dual Opteron - SC-level memory busses, although it doesn't scale to 1024 nodes just yet.

Still wouldn't work

[scheme]

True, it'd still be slightly higher than pure memory- but the increased number of nodes will cover that up in a hurry, especially if you code for specialized machine processes rather than a single general processs

It would still be a lot slower than memory. Typical memory latencies are on the order of 100ns. Going to the network would still be a lot slower. Suppose network latencies where about a tenth of what it is typically. That puts the latency at about .1 ms (a ping to a system on my network takes about 1.5ms) which is still about a 1,000 times slower than memory. Even with an improvment to .01ms in the network latency, we're talking 100 time difference in latency.

That essential puts a lot of stuff in the difficult to work with area. So no, I don't think a network can effectively replace a shared memory machine on latency dependent work.

Re:Still wouldn't work

[afidel]

Uh, real latency numbers are about 3ns for modern ECC memory and about 15-60ms for Infiniband depending on message size, so not hundreds of times but rather 20 times worst case. That's still not good if you are message latency limited but you have to remember that crossbars and cross cabinet interconnects have latency as well.

Re:Still wouldn't work

[Shinobi]

Actually, Infiniband, Quadrics, Myrinet etc are all down to 15-60 _micro_seconds for useful payload sizes, not milliseconds. Gigabit and 10GbE are far slower though, and you'd have to write a software stack for handling remote DMA(Which Infiniband handles in hardware). And it's still not even near achieving what a true shared memory system can achieve.

Re:Still wouldn't work

[scheme]

Uh, real latency numbers are about 3ns for modern ECC memory and about 15-60ms for Infiniband depending on message size, so not hundreds of times but rather 20 times worst case. That's still not good if you are message latency limited but you have to remember that crossbars and cross cabinet interconnects have latency as well

You should check your figures. The Sunfire systems have latencies between 100ns-240ns depending on where the request originates and where the data is stored. The opterons have latencies of about 100ns.

Your 3ns latency might be just for responses on the memory chip. I'm referring to the latency from when the address is request to when the data reaches the cpu.

Re:Still wouldn't work

[afidel]

According to this site Infiniband is about 2-4x slower than local CPU to CPU copy on an Itanium 2 (for values that fit into CPU cache, for some values larger than the cache it is actually faster to go across the Infiniband link!) and about 10x faster than messaging over GigE.

Re:Isn't this good?

[ameoba]

How about a compromise like SGI's NUMA systems? Seems like a reasonable compromise between fast memory & lots of CPUs when memory access is important.

MPI works fine on clusters

[Anonymous Coward]

Since with MPI cache coherency is not an issue it will work fine on clusters, the only problem is interconnect.

Nowadays you can get PCI-E Infiniband controllers if you want to spend the money ... that gives you bandwith pretty close to a Cray RapidArray link (1.6 vs 2 GB/s ... close nuff).

Its not about if you need a supercomputer

[Anonymous Coward]

Granted most things these days can be done on commodity hardware. The issue is what comes next. Many hardware advances over the last couple of decades have been made in supercomputers, these way out techniques become tomorrows commodity parts.

If you start with commodity parts then where is the technology drive for tomorrow?

It may be that there is no market for them. Does that mean that computing has reached its limits? If no one pays for 'supercomputers' then how does the research into things like quantum computing become a commercial reality?

Perhaps they're just complaining of a reduction...

[pragone]

in their market share. True enough, some problems can be parallelized, some can't, but before clusters were feasable they all had to be solved with supercomputers.
Now, the problems that CAN be solved by a cluster will surely be solved be one (it's cheaper).
So perhaps they won't desapear... they will just loose some market share.

Hypercomputers?

[squaredot]

What about FPGA based supercomputers? Like these: http://www.starbridgesystems.com/

Bull

[Anonymous Coward]

One AC to another: All the simulations you are talking about will have to be re-run a lot of times with different combinations of input parameters anyhow. Therefore, these are trivially parallelizable on a cluster (just run one combination of input data on each of the CPUs).

- non-punk who actually saw how Chem simulations are run
- old enough to see CDC run

Re:Bull

[Anonymous Coward]

You're an idiot.

You don't do parameter studies with hero runs.

Re:Bull

[Anonymous Coward]

What ?!? Do you just wish the results up?

It's kindof hard to examine the effect of a
parameter differing without results from the
parameter differing.

By all means though - if you have solved this
problem, all the world wants to know. We are
waiting, moron.

1024 Chickens

[bombadillo]

This reminds of a Quote from Cray. It goes something like, "Would you rather have 1024 chickens pulling a wagon or a big Ox."

Re:1024 Chickens

[Simkin1]

Personally I'd rather have 1024 chickens.. assuming that each chicken is tethered to the wagon and can only pull at most 1 pound in any direction, I figure let the little birds go nuts and move all over the place... even if only a quarter of the chickens move in the direction you want to go, you're still better off... why?? simple... Even if you never make it to your destination, at least you'll be well fed!

All joking aside for a moment... I hope Cray realizes that the analogy in the question is wrong -- here's the corrected version: Would you rather have 1024 big Ox pulling your wagon or one big ox?

Re:A lost metaphor brings out my inner language na

[Anonymous Coward]

I don't think people eat feed corn-- doesn't it have low sugar content, high protein content, and taste like cardboard?

Ask Monsanto about it.

[MacDork]

They'll remind you that you DO NOT save seed corn, lest you be sued into oblivion.

Next-gen systems already under development

[Anonymous Coward]

New supercomputing advances on the way will radically redefine the industry.

I refer to a DARPA funded project created to fill in the performance gap between today's inadequate SC technology and tomorrow's (quantum, bio) still far in the future stages.

The project is called HPCS, which does not stand for High Performance Computing System, but rather High Productivity Computing System. The point is not to increase flops but increase value. The earth simulator, for example, is down for maintenance about 2/3 of the time and can only be reliably run in 8 hour chunks. The ASCII series may have high peak performance but averages only 5-10% of that. DARPA knows that if this is the state of the art, then there is work the do.

Three companies are currently doing research on proposals and are DARPA-funded through 2006. The three are Sun, IBM, and Cray. Two companies will continue from 2006. A working product will be delivered in 2010.

Many radical new technologies are in place, but as you can understand most of it is tightly under wraps. But do some reading on the DARPA page and you will find some interesting things.

Here is the link

- employee at one of the three aforementioned companies

pfft. Duh

[Frequanaut]

We'd all rather watch Brittany and Bomb arabs...cmon science??? yuh

Clusters are not perfect.

[zymano]

I agree with the comments in the article. Commodity clustering is NOT the answer. It consumes tons of energy also. Everything in the article seems dead on correct. Parallel processing is not the big nirvana everyone thinks. Some jobs can't be done that way. It's ironic that this was posted today because i was looking up info on google about superconducting supercomputer.

Pyramids

[panxerox]

It easier to make pyramids (clusters) using lots of people and well understood principles than it is to make art (new processor / architecture). Just a reflection of our whole socity - don't take chances. Thants how you end up with rocky 4.

It is about cost

[deadline]

Sigh...

There never really was a supercomputer market. There was a cold war, that subsidized the supercomputer market.

Then there is the cost. Companies stopped making SC because they were too expensive. If the guy from Ford wants to pay 1 billion for a supercomputer I am sure someone will build him one. The cost build a FAB is over 4 billion. Why do you think HP teamed with Intel. Why do you think there are so few processor families? You have to make a living in the commodity market where you can sell things in the millions because supercomputers even in their heyday were sold in the hundreds.

Then there is the problem that many problems are solvable on clusters. So those specialized problems can not depend on other parts of the HPC market to help subsidized their corner of the market. i.e. clusters make the really hard problems more expensive.

It is question of how much you want to pay to solve your problem? Simple economics actually. If the numbers don't work, the problem doesn't get solved. If the Gov. wants to solve some problems (and during the cold war they did) then they can step in and subsidize the market.

And don't cry about Japan and the Top500. When the top500 has price column then it will start to be meaningful.

Because if you don't...

[Anonymous Coward]

...the terrorists win??

Compaq killed it...

[evilviper]

The fact of the matter is, Compaq is the one that killed the supercomputer.

They were #1 in a BIG WAY after they bought DEC and sold Alpha-based mainframes. Alphas were something like 7 of the top 10 in the Top500 a year or so ago... Mainly because Alpha processors were great, and outperformed everything else by leaps and bounds.

Unfortunately, under Compaq, Alpha development stalled, and stalled, and stalled... Their latest processor wasn't a big step above x86 processors of the time, and since they only release a new processor once in a while (unlike the PC market where faster processors are released almost weekly) current PC processors are faster than Alphas for many operations. With Alphas being far more expensive than PC processors, is it any wonder they aren't posing much competition to clusters? It seems to be a common problem...

IBM's newest PowerPC processors looked like they might revitalize the supercomputer market, but it really just wasn't that much faster than everything PC processors either.

Is it possible that conventional processors just can't get much faster with better design? Is it possible that the sheer volume of processors AMD/Intel sell, allows them to stay ahead of the giants of computing?

Well, in any case, it's quite possible that the giants are just stuck in their ways... Each one wants to hang-on to it's propritary architecture, instead of buying-in to commodity computing. There's really nothing stopping IBM, Sun, SGI, et al., from building a supercomputer around Opteron processors, but they just don't want to... They want something that fits in with their market strategy of propritary lock-in.

Wonder if the up-take of Microsoft Windows ever got the federal government to subsidize propritary Unix OS development... Doubtful, but it's really the same thing.

Re:Compaq killed it...

[Simkin1]

Actually you're only partially correct. The alpha processor despite popular opinion was headed towards the scrap heap before Compaq bought them. The reality is that the conventional 'supercomputer' markets were falling away from the Alpha because of the promise of the Itanium and Opterons (which were just poking their heads out of their holes at the time and announcing their existance). In additional benchmarks of the existing alphas were only marginally better than the Intel Pentiums of the time (pre-Itanium & AMD Opteron) despite the 64 bit processing. The stall wasn't because of Compaq, but because the core development teams for the Alpha were... well, getting old and retiring without suitable replacements. Plus I heard it rumored that Compaq purchased Alpha not for the processor, but for the architecture which it intended to 'reinvent' in a new image. Dunno what happened with that though...

Imagine....

[Anonymous Coward]

... a beowulf cluster of these...

Re:Imagine....

[Tablizer]

Imagine a beowulf cluster of Cray's.

Here's a thought

[couch_warrior]

I know folks who did their PhD thesis on a Cray (~100MIP +15 years ago) who now get much more computing done on a Pentium III desktop workstation. It has more megaflops and more RAM. The point - most of the research poured into "supercomputers" has been a complete waste of time. The SIMD technology of vector machines is virtually useless for general computing, but the enormous fiancial engine behind commercial CPUs has driven their R&D to pass up the performance of former "super" computers typically within 5 years. The best advice would be to just shut down the useless waste of money that is supercomputing R&D, and wait for the commercial sector to do it.

Can you imagine...

[toddhisattva]

Can you imagine not imagining a Beowulf cluster?

Reality check..

[Simkin1]

I hear this type of FUD all the time from some of the older folks I work with -- all the hype about Cray systems, shared memory capabilities. There are several problems with the 'supercomputing' market though. First off if you go to any local University, they teach fortran as a basic intro course, but most professors will footnote their comments by saying things like "Fortran is no longer considered a marketable language"... so students think -- why waste time learning it (other than the basic programming capability?); this erodes the base support for vectorized programming support (putting aside arguements that fortran is not vector programming for now please). More importantly (and secondly), Cray architectures are MASSIVELY expensive in relation to where the standard desktop CPU is without enough benefit to make them worth it. People talk about the need to play "catch up" as if there is a real crisis at hand. The reality is that clusters are gaining in popularity because they're cheap, and per CPU tend to be more 'powerful'. Institutions that purchase Crays end up with a number of folks trying to run software and inevitably you run into time bottlenecks. What folks at these institutions realized is that by purchasing a 'state-of-the-art' 32/64 bit system with dual processors and a couple gig of memory, that by the time their jobs were starting on the Crays, they could already be running on their new desktops.... In the end I say why rent time on a big Cray when you can purchase your own system and run it into the ground with jobs? Despite some of the FUD I've seen splashed around by folks who are proponents of Cray systems, Clusters are relatively simple to setup, do not require multi-million dollar yearly contracts, and can generally be maintained by the purchaser without much effort. Most of the cost of clusters anymore is the cost of the system administration -- which you don't need a "staff" of SA's to administer. IMHO, you don't even need an SA anymore for designing, building, and running a cluster; with only a baseline of knowledge of computers, and a little reading just about anyone could build their own. When all is said and done, you end up with more compute power in a cluster at a much lower cost. Some folks would say that throughput is an issue; that's problem dependent, and can be rectified with a little problem solving. Some folks indicated that clusters aren't optimized for peak performance... WHO CARES?? If it takes me two minutes longer for a 5 week job to finish, I've still made out better for not purchasing a Cray. I've heard folks complain about not enough memory, or programming problems -- generally these tend to be older folks (what I like to call the obsolete-engineers; aka -- old fogies) These are generally the folks that complain about the 'new fangled' software they have to use, and simply don't want to have to reinvent their "marvelous software" in a new environment. I've run 64 gig memory jobs on clusters without a problem; again my problems allow for parallelization optimization... Eventually I end up telling folks who are steadfast supporters of the Cray to wake up, this is a brand new era about you. The days of the single supercomputer vendor filling all your needs are over. You now have choices, and with those choices you have to take the responsibility of defining how your new parallel supercomputer will function. Either learn and adapt, or become obsolete.

Free market: this Internet thing

[scoobrs]

Free market success might actually have lead to DARPA not funding any research in this thing called the Internet in the 60s or to more libertarians tooting their own revisionist history horns. Then again, the free market can do anything you say it did after the fact. That's why it's so free.

Why do supercomputers have to make business sense?

[Anonymous Coward]

I'll pretend someone will read this late post, but most of the arguments people are presenting against supercomputers are business arguments. Why do we need a business model to want to build supercomputers? Governments build a lot of stuff that makes horrible business sense. For example, particle accelerators. These things are damn expensive and I doubt profitable. But they do enable fundamental scientific research. The same can be said about supercomputers. It's going to be less than 20 years before we could start thinking about almost science-fiction type problems --- for example, molecular dynamics simulations of whole human cells. However, we're not going to get there by strapping all of our 2025 cell phones together and making a conference call (the 2025 idea of a cluster). Supercomputers are designed for fundamentally different scalability, reliability, compute performance, and bandwidth than consumer systems. While they might not be a good business idea, it hard to argue against the benfits to material science, physics, medicine, etc. that supercomputers could provide to society. In that light, $200M of government spending isn't about holding onto ideas of the past, it's about preparing for the future.

shameless plug

[igodard]

I work for a CPU startup (ootbcomp.com) that is exploring how our high-end chips would fit in a supercomputer. Our Mill architecture is a general-purpose CPU (yes, Linux) that looks like nothing you've ever seen when you get the hood up. Supercomputers are multinode machines, which does *not* imply that they are loosely coupled clusters in the VATech model. A node is one CPU chip, a local memory, and an interconnect to other nodes to provide system-wide global shared memory; node counts from 100 to 10,000 are common. Per node, our proposed system has up to 100Gflops, up to 960GB/s memory bandwidth (to any mix of local or remote memory), up to 4 GB of local memory, and burns under a kilowatt. With reasonable packing densities you can get 3 Earth Simulators worth of flops and up to 7TB memory per rack, using around 600kWatts. That's with global shared memory and a maximum memory latency from any CPU to any byte of under 100 clocks. We just do chips, but would be quite interested in someone who wanted to put our chips in their boxes and sell them in this market. Please don't kick the tires to satisfy your individual curiosity - if you don't know why those numbers are significant then we won't tell you, and if you are not with a present or potential market player then you'd be wasting time we need to spend getting it working :-) Ivan

Very *large* messages are the worst case????

[PaulBu]

In the worst case (very large messages) ...

The WORST case in the latency-limited problem are very *small* messages: "The value @0x67564987 has changed to 0xDEADMEAT, which value to look at next?" (note the implied waiting for the answer).

You do not gain anything when you pump up bandwidth while exchanging small messages (it helps for large, sure!), you are still limited by the latency.

Paul B.

Earth Simulator envy

[Animats]

If you read the congressional testimony, you can see the real theme - "The Japanese Are Ahead Of Us". The fastest single computer today is in Japan, and some people are having a major cow over this.

It's an impressive machine. 640 nodes, each of which is an 8-CPU shared memory multiprocessor. The unusual feature is that the machines are interconnected not by a network, but by a brute-force 640 x 640 crossbar switch. Each data path has 12.3GB/s, and 640x2 paths can be going at once. 83,000 separate cables. Aggregate bandwidth about 8TB/s.

It's a reasonable way to make a one-off machine, but a dead end as a commercial product.

It's about US vs Foreign dominance

[Michael Snoswell]

This isn't about clusters vs single memory architectures. The main issue as mentioned elsewhere in comments is that the market for high end computing is being partly filled by cheap clusters so *US* manufacturers are not focusing on those kinds of systems as it's less commercially viable.

Non-US copanies like NEC are still focusing on and winning the non-cluster high end market race. Notice how the US govt was so protectionistic a few years back (97 I think) when NEC won a major contract over a US supplier (SGI) because the NEC was cheaper. The US govt overturned the sale saying NEC unfairly discounted.

Well in this battle the US companies are slipping behind so now the US govt is injecting funds to help prop up these less competitive US companies. I'm sure this money will be spent on purchasing "classified" systems that non-US manufacturers wont be allowed to compete for "security reasons".

At the end of the day it's just another form of protectionism.

Re:A lost metaphor brings out my inner language na

[meringuoid]

Seed, as in, what you don't eat, but save to plant next year.

Isn't that illegal?

Everywhere the same trend...

[octogen]

"Hey, these Cray computers are really expensive; but I figured out, that we could just use personal computers instead, if we emulate the load balancing hardware, so let's just put together some 50,000 x-boxes. Ooooh! These x-boxes are even cheaper! We just have to emulate the PC software in order to be able to emulate the load balancing hardware on the X-box. LOOK AT THIS!!! We can even use Excel instead of our expensive mathematics software, we just need to put in some Visual Basic Script, connect it with some DOS-based C++-Code to the NT-based Java-Code, and send the results to the Linux-based Emulator, and everything should work!! This is even CHEAPER!!!11!!!111"

2 Weeks later

"Uhmm.. now what's that? It can't be zero!? Shouldn't it be something like 68 Billions and 45 Millions or something? .. oh no .. can I borrow your screwdriver? We should also exchange node #43091, I think, something's wrong with the memory.."

First, they exchange all the professional, mature and robust parts with the cheapest consumer-class devices available on the market, and then they start wondering and complaining, why nothing works reliably anymore...

SGI is running linux on a 512 CPU NUMA

[barneyfoo]

If you read the papers at the recent OLS (Ottawa linux simposium) you'll see that SGI is running linux images (specially tuned) on 64, 128, 256, and in 2 cases 512 cpus. Reading the paper is an interesting view into the problems of running kernels and OS's on such huge NUMA machines.

http://www.finux.org/proceedings/

OT (Re: Sig)

[balloonpup]

"Well I don't want Fop, dammit! I'm a Dapper Dan man!"

I can't remember for the life of me the name of that movie. Can you tell me what it is?

Communication

[Becquerel]

Perhaps the communication is limited only to adjacent cells?

Typically a Finite Element Analysis in parallel, analyses a problem by breaking it down into millions of elements. These can be an element of air/water (CFD), an element of metal (stress analysis), an element of soil (my own speciality), or combinations. At it's core FEA is basically the solution of a massive array of simultaneous equations

Taking the Weather analogy. Say you are processing a global model on 16 processors. Each would take charge of thousands of elements and would calculate all the flows of air/heat/moisture/water/etc within and between those elements. Communication between the processors is required at the edges of each processors domain, to know what is going on in the next domain.

As you can imagine this requires a lot of communication between processors for a solution to be converged upon. Remember back in high school when you had an array of 4 simultaneous equations to solve and how each time you eliminate a variable from one row you had to change all the others, well the communication involved in solving a billion equations gets quite large.

Organising the decomposition of the domain to optimise the inter processor communication is a field on it's own, needless to say though that a beowolf cluster of linux boxes, does a good (and very cheap) job of running the calculation, but is always limited by the speed of the communication between boxes. Whereas a custom machine can be built to reduce the latency of communication greatly. If you know pretty much what you are going to be processing then you can even set up the communication so it is much faster to 'local' processors to speed up communication more.

Anyway in short the answer to your question is, Yes (more or less)

Re:Communication

[Jason Pollock]

I think I understand (sorta). :)

Thanks!

Feh, stalin has spoken...obey

[Anonymous Coward]

"..DARPA not funding any research in this thing called the Internet.."
Yes, because only big govt projects can do anything...riiight? ...talk about revisionist history...bah

"Govt is like a guy that breaks your legs, takes your wallet and buys you crutches, saying 'see, without me you couldn't walk'"

NOT NOT insightful!

[Donny Smith]

Gee the article says "But they have proven "exceptionally difficult to program" and problematic at certain performance levels," (quote from a 2004 study by the President's High-End Computing Revitalization Task Force.)

Now you come up with some MPI theory negating a study by HPC experts.. WTF?
Anyway, your MPI coding skills are not a relevant to the main issue raised in the article.

>If I know MPI (which I do), I can code on any cluster out there, and it will perform similarly, with a linear time dependance on network speed.

Exactly - with a linear time dependEnce on network speed. Which means that for some tasks, as explained in the article, MPI-type clusters are not the right solution. They are the cheapest solution - or merely a "workaround".

And finally, any task with huge data sets that cannot be split in subsets doesn't lend itself to MPI-type processing.

The article's point is that instead of evolutionary progress (replacing Myrinet with Infiniband or Xeons with Opterons), the government should spend more to create revolutionary improvement.

Re:Please make it stop!

[strictnein]

And they changed it! :)

Um, Indy car *is* off the shelf technology

[Moderation abuser]

It's Formula 1 which race prototype machines.

It's quite an apt analogy actually, on any given track (except maybe those boring boring boring ovals) the F1 cars are seconds per lap faster than Indy cars. The problem is than even the cheapest F1 cars are ten times the price of an Indy car.

If you want the ultimate in performance you just bloody well have to pay for it. If average performance is acceptable then you can save a bundle.

Dual citizenship

[heroine]

If US can't keep up with the rest of the world, don't complain, just leave. Only through attrition will Americans be forced to decide whether they want the good stuff or they want to move aside.

Do the star players of the world all want to play for the worst team in the league? Do you watch a superbowl for the 2 worst teams in the league? So why keep rooting for the world's smallest economy? Thanks to globalization there are many countries with more to offer than US, yet opinions of what should be are hard to change.

You should sprint to the next Korean startup to develop terrahertz microprocessors. Sprint to the next Indian startup to develop back office suites. Run to the next Chinese startup to develop nanotechnology rockets.

Man, your parents have a BIG basement!

[brodin]

someone had to say it!

Re:Man, your parents have a BIG basement!

[grawk]

Dad was pissed when our last power bill came...

supercomputers work would be outsourced anyway

[AIXmaster]

Even if U.S. Congress were to allocate
$200 million or $800 million to the
supercomputer work, the companies
would outsource the work overseas and
pocket the savings among the CEOs,
shareholders, etc....

Desgin Better Potato Chip?

[Anonymous Coward]

"American supercomputer users, such as Procter & Gamble and Ford Motor, like legislation that bolsters high-end computing research and development because it will help them design better cars and potato chips."

I can understand designing better cars, but potato chip? lol, just think "I need a beowulf, no a cray to make better potato chips", we'll sell one of these to every idaho potato farmer.