Slashdot Mirror
Supercomputing: Raw Power vs. Massive Storage
securitas writes "The NY Times reports that a pair of Microsoft researchers are challenging the federal policy on funding supercomputers. Gordon Bell and Jim Gray argue that the money would be better spent on massive storage instead of ultra-fast computers because they believe today's supercomputing centers will be tomorrow's superdata centers. They advocate building cheap Linux-based Beowulf clusters (PCs in parallel) instead of supercomputers." NYTimes free reg blah blah.
Cluster computing really is the future. Supercomputers are expensive, run wierd OSes (sometimes), and have infrasructure requirements. A cluster (I prefer OpenMosix, but Beowulf if you like) just requires fast ethernet or fibre.
Plus, think of all the computers that go unused at night in places like school computer labs. All those free machines could, at night, join a cluster and do number crunching for researchers.
-- Bill "Houdini" Weiss
There are lots of reasons to have really good bulk storage technology. But what's the killer app that's going to get the $10^9/year in government spending? Can you say "Domestic Surviellance" boys and girls? I knew you could!
Look at the average Joe Schmoe, or even us uber-users, who really needs a 3+ GHz machine? Even some of the cornerstones of fast computing such as computational problem solving are being addressed by grid/cluster based solutions which typically don't use high end machines.
I'm perfectly happy with my P3 800MHz, but I run out of hard drive space everyday.
Cheap, YET RELIABLE high density storage solutions are still not readily available. I know we are now down to a $1 per Gig, but the average size of a user's file has increased now. Media (legal or otherwise), games, and other programs are chewing up hard drive space.
There needs to be more research into trustworthy, lowcost high volume storage mediums.
-"Those who fought today will die tommorow."-
As much as I hate conspiracy theories and Microsoft bashing, this may be an extremely clever move. As of now, mainframe and supercomputing worlds are still relatively safe from commiditization. Unlike Linux, which is still virtually ireelevant on the desktop, mainframes and supercomputers are much bigger a piece to swallow for Microsoft. By recommending Linux clusters, Microsoft may actually be trying to establish commodity hardware in the world of supercomputing. The keyword here is hardware. Once clusters become ubiquitous, Microsoft will start aggresively pushing Windows 200X Server Cluster Edition, fighting an enemy it has already much experience with.
I think they're advocating spending the big bucks on data storage rather than on big iron.
When they mention beowulfs, it's in the context that when researchers need the equivalent of a supercomputer, they can just build/use a beowulf cluster. What they can't do on their own is come up with petabyte storage facilities and the data in them.
So what they're really advocating is spending money on storage; it doesn't say in the article what form that storage should take.
The government may very well like this. They're going to need big data farms to support the TIA program. It takes a lot of space to remember what kind of toppings every person in the US likes on their pizza.
In a word, Bullshit.
These are not MS evangelists we're talking about here. Gordon Bell and Jim Gray know a 'thing or two' about high-performance computing.
If these guys weren't able to speak their minds on technical matters entirely without retribution from Bill and Steve, they wouldn't be at MS at all. They don't have to be. They CERTAINLY don't have to tow the party line and recommend the flavour of the week, because it messes with the latest Sun/IBM/HP/Linux/Mac threat.
Now if you actually look at the statement these guys are making and examine it based on their history, they've got a very good point. They're not talking even remotely about 'one-size-fits-all' systems--they're talking about the future of cutting-edge research.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
Research on building Mega beowulf clusters is a legit govt activity and so is building some. But the beauty of the beowulf cluster is that it is affordable to bussinesses, acadmeics and govt, plus its very adaptable to budgets and interconnection schema (fast, slow, grid, scavenger).
but beowulf clusters wont replace the need for super fast, super scalable, computers with well architected interconnects. there are lots of problems in this class, mostly physics simulation, that just cant be done well on beowulf clusters.
I should probably note that my own work involves large computer clusters. However my probelms (in biology) are in fact well suited for beowulf clsters. thus I'm happy to hear of more money for beowulf computing. but frankly I think that this should be in addition to the fast computers.
the flip side here is that it might be the case that money for fast computer resources is not being well spent as it could be at present. there seems to be too much emphasis on "landing the contract" for the computer center than on building a good design. congress via DOE tends to doll these things out in a political fashion making sure each big client gets funding for a center rather than letting the best center get the most contracts. as a result some of the so-called super computers may be just glofied too-expensive-per-cpu unscalable systems already that could be eclipsed by a comparable low cost beowulf system.
but that being said its still an area that the gov needs to fund since it wont drive itself commercially but its needed for lots of science and simulation.
Some drink at the fountain of knowledge. Others just gargle.
He also talked about CERN generating 10 PetaBytes a year when their new collider comes on line
Supercomputers are sexy, but are losing the technology war. If you start designing a new one today it will be years before it is ready. During those years Intel and AMD will crank up their clock speeds and negate much if not all of the CPU speed advantage you get from your fancy design. Why not go for parallelism from cheap machines?
No electrons were harmed creating this post, though some may have been subjected to electrical and/or magnetic fields.
they're making the case that clusters should be pursued over supercomputers for the data-intensive number crunching activities like nuclear explosion modeling, etc.
I doubt it. You can only use a "cluster" like a Beowulf if your problem can neatly be divided into small, completely independent work units. If you want to render a movie, then so long as you have all the scene data, each frame can be rendered completely independently of any other, then stitched together at the end.
Try using a Beowulf-style cluster for a CFD problem, and watch as all computation grinds to a halt as your processors and interconnects devote all their capacity to inter-node coherency and synchronization. You need a traditional supercomputer like an SGI Origin for jobs like that, because of its massive internal bandwidth.
There is absolutely no danger of Beowulfs killing off the supercomputer in the near future. In fact, the supercomputer market is looking pretty healthy.