Supercomputing: Raw Power vs. Massive Storage

securitas writes "The NY Times reports that a pair of Microsoft researchers are challenging the federal policy on funding supercomputers. Gordon Bell and Jim Gray argue that the money would be better spent on massive storage instead of ultra-fast computers because they believe today's supercomputing centers will be tomorrow's superdata centers. They advocate building cheap Linux-based Beowulf clusters (PCs in parallel) instead of supercomputers." NYTimes free reg blah blah.

Comments

Open to possibilities.

[thirty-seven]

Its nice to see some MS researchers going against the perceived stereotype and being open in their suggestions like this.

And I think they have a good point about massive memory being a very important part of computing advancement right now.

Re:Microsoft recommending Linux Beowolf cluster?

[TopShelf]

You're right, this could be more about impeding Sun and IBM than anything else, but I don't seem them recommending this as a one-size-fits-all deal - rather, they're making the case that clusters should be pursued over supercomputers for the data-intensive number crunching activities like nuclear explosion modeling, etc.

Hell, if we're going to talk all SERIOUS and shit

[Bold Marauder]

You have to wonder why, all things seriously being equal, they don't recommend a *BSD-based solution instead of a Linux-based one. Esp given the near-equivalent functionality of the *BSDs, and the fact that MS has publicly endorsed the BSD license in the past, citing it as an superior alternative to the GNU License.

They work in a Microsoft research facility...

[fruey]

But that doesn't mean they have to push Windows solutions, since they're really more into a full on research program looking at computing theory, a level up from the "let's bash MS about their desktop and server dominance strategies" - both these guys have long histories in the Internet and networking and clustering, worked on PDP-10s and their ancestors, etc.

From the MS site, the Bay Area Research Center is "... a small Microsoft Research group located in the San Francisco Bay Area. We've been working on two large projects with other universities, companies, other Microsoft Research groups, and with Microsoft product groups in Redmond and Cupertino. These projects are Scalable Servers and Media Presence. "

I can't see scalability involving commodity hardware with MS OSes. In spite of Microsoft's desktop domination strategies, and small business server dominance (arguably, at least for the moment) they know they won't be taken seriously about clustering Windows 2003 server, purely because there is no design AFAIK in the kernel for operating in clusters in the first place. This is supercomputing using commodity hardware, not supercrashing using commodity OSes. Linux is perfectly situated to be recommended by anyone because it is not a competitors product, per se.

The homepages of the two men can be seen here, if anyone is interested in some of the more interesting history of the two. Little of it has to do with Microsoft propaganda and the marketing machine:-

Gordon Bell
Jim Gray

Yes, but we need research into BOTH

[alispguru]

Raw speed will always be useful for problems that are hard to parallelize. Right now those problems (parts of crypto, some quantum physics calculations, etc.) are important scientifically, but away from the money.

Industry will spend R&D money on clustering for storage and reliability, without major government subsidy, because there's a crying need for it. How much government money went into Google/eBay/Amazon?

Government research is supposed to complement industry R&D - to be aimed at fields where the results are still important, but maybe not as profitable. This is why government should not abandon raw speed as a research goal.

Re:Microsoft recommending Linux Beowolf cluster?

[cvdwl]

"Try using a Beowulf-style cluster for a CFD problem, and watch as all computation grinds to a halt as your processors and interconnects devote all their capacity to inter-node coherency and synchronization."

B...S...; we use a small Beowulf (16 dual 1 GHz PIII boards with a fast ethernet backplane from PSSC) for oceanic numerical modeling and the problem scaled almost perfectly with number of processors.

Our models are 3-dimensional, but sudivision and message passing takes place only in the horizontal two-D direction. And message passing only needs to account for the boundary nodes.

Ease of use is a bit of a larger issue, however. For convenience sake I usually end up running at home on the dual Athlon and then doing big runs and batch jobs on the Beowulf.