Slashdot Mirror
IBM Creates New Fastest Beowulf Cluster
shawnb writes "It seems that IBM has created the world's fastest Linux cluster built from lots of small servers (64 IBM Netfinity 256 servers). The Netfinity servers are linked together using "special clustering software and high-speed networking hardware, which causes the separate units to act as one computer, delivering a processing speed of 375 gigaflops, or 375 billion operations per second." They also go on to say that this is the fastest Linux supercomputer, "it will only rank 24th on the list of the top 500 fastest supercomputers. " "
I have a friend at SGI that said SGI's bid for this was almost at cost and it was like $1 million. IBM is losing a lot of money on this deal but they get the publicity and the fact that all the geeks at UNM will see big blue's logo all over it.
Also, the cost of the boxes is almost unimportant with something like this, you have to take into account the actual construction of the network (usually the most time-consuming part of any super computer) and the main cost is the support contract. You have to have people available to fix this guy on a moment's notice whenever it breaks.
I guess that would depend largely on the software run, however I believe that SMP will be useful on the nodes; simple SMP systems are dirt cheap nowadays, and they cost actually less than two UP boxes (1 powser supply, 1 MB, 1 bus, 1 Network interface, etc ...)
How about something like Myrinet? Iffen you gots the $$$ ;)
Your Working Boy,
Also, it mentions the limitations of networking; can't you link together 3com (now defunct, I know) switches in a stack to make larger switches? If not, I'm pretty sure that we'll have larger switches in the next few years.
--
Because the joke is so damn obvious. Christ sakes my mother could have thought of that one.
What possessed someone to mark the above post as 'flamebait'?
I literally read it 3 times through to find the 'flamebait' there: nada. The only moderation down, which could have had a sliver of merit would have been 'overrated' but this is ridiculous. Hopefully this gets caught in meta-moderation...
Chris
San Francisco values: compassion, tolerance, respect, intelligence
Beowulf != any of these. Beowulf is the idea that one can take commodity, off the shelf (COTS) components and build a powerful machine at a price far less then a comparable commercial offering.
Codes run on Beowulf, and really any parallel machine, typically use MPI, PVM, or custom message passing libraries. The beowulf idea includes the use of MPI & PVM, among other freely available software packages. Codes that run on shared memory machines typicall uses the shared memory device of MPI, shared memory, or pthreads.
For CPU intensive tasks the Beowulf idea is great. Codes that perform lots of disk I/O suffer, as adding higher performance (i.e. SCSI) disks increases system cost greatly. Communication intensive tasks perform the worst on beowulf style clusters compared to commercial computers, as the interconnect on beowulf-style clusters can't compare. For a relatively large increase in cost, one can use Myrinet. With Myrinet bandwidth and latency begin to approach that of the switch found on the IBM SP series of machines.
With high bandwidth, low latency interconnect technologies that scale well (e.g. Myrinet), one can build a cluster that outperforms a comparable commercial offering at, say one quarter to one eigth the price. The difference at that point is software. There's really not alot out there to configure and administer beowulf-style clusters, and commercial implementations of some packages beat the pants off of their freely available counterparts (compilers, for example). Until the software situation changes there is still reason to buy your big iron from IBM, SGI, and Sun.
--Jason
Actually, I'm not making a category mistake (I even noted in my original post that the things I was using as examples were not necessarily interchangeable), but you've made my point for me.
As you note, the real power of distributed/parallel computing comes from the message passing libraries, most commonly MPI or PVM. Beowulf per se is almost nothing more than a label for the generic concept of distributed computing on Linux. The same thing can be done with any other reasonably modern networked computer you have lying around, even those running Windows - you can even mix OSes in a cluster, although this introduces new and interesting problems. (There are a serious lot of underutilzed cycles sitting out there on the corporate world's desktops if they're not running OpenGL screensavers...)
BTW: If the phenomenal success of Sun's E10000 Starfire has taught us anything at all, it's that where I/O is important, a big honkin' SMP box kicks cluster butt! Seriously, the interconnect technology between boxes just *can't* be fast enough to compete effectively with a huge multi-level crossbar packet switch like the ones in the E10K. Sun and the other SMP vendors can win here because they own the domain in which the simpler problem resides...
Don't assume by this that I'm against Beowulf clusters at all - they are a great and amazing thing, but there's more than one way to skin a cat, and Beowulf isn't the only path to Linux distributed computing.
"The future's good and the present is nothing to sneeze at." - Roblimo's last
The article doesn't say, but despite what you'd think by reading the rantings of the ill-informed 3l337 d00dZ on slashdot, Beowulf isn't even a very good clustering technology for most problems.
There are far more serious, industrial-strength solutions out there, things like MPI, PVM, LINDA,and IBM's own HACMP. (Note these cover a lot of ground and are not necessarily even comparable to one another.)
Beowulf (or any of the others listed above) is not automatically the correct distributed computing methodology. Selecting the proper solution for the job at hand is far more complex than you might imagine. There is a lot more developer activity on some of these than there is on Beowulf - MPI in particular is maturing rapidly and is used for solving big/tough problems in many of the largest companies in the world. (No particular MPI advocacy or bias, it just seems like I run into it more often than the others...)
"The future's good and the present is nothing to sneeze at." - Roblimo's last
If you need it, get Ultra3 SCSI with solid-state drives. Sure it'll cost you an entire lifetime's salary, but hey, they're great.
Really though, using a solid state drive as a cache for a disk subsystem is an easy way to enhance performance, and is already being sold. You perform a write - instant gratification, and wiht proper caching algorithms, you can get the same thing for reads. A multi-gig SS Drive can easily max out a bus. Multi-level caching is a necessity as speeds increase in systems.
In this sort of system, the interconnect fabric (as fast as it is) can still be a little bit of a bottleneck, too... A good cached RAID disk system on the one end can really keep things smoking, though.
"It's tough to be bilingual when you get hit in the head."
http://www.unm.edu /~paaffair/news/news%20releases/Mar21hpcc.html
"It's tough to be bilingual when you get hit in the head."
Just point him at http://www.beowulf.org
Of course, it's a little more technical than just "a bunch of computers hooked up via high speed links (i.e. fast/gigabit ethernet) to provide a parallel solution for complex calculations", but there is a lot of info...
"It's tough to be bilingual when you get hit in the head."
Darn barrell shifters are so expensive in hardware (and so rarely used, aside from some specialized apps). The processors are Xeons, nothing special there (aside from full speed L2). So the RC5 speed would be ~the same as any group of machines with this proc/speed (since it isn't a network intensive computing project). Grab and crunch... crunching takes far more time (several orders of magnitude if done right) than grabbing, so it's not really a situation where beowulf clustering even helps... a second or two to d/l a new keychunck, and, depending on the size of that chunk, anywhere from a few minutes to a few weeks to finish it (even with mighty Xeons)... nice as a distributed app, no gain from a cluster.
"It's tough to be bilingual when you get hit in the head."
True, but they are using 2-way boxen...
e s/Mar21hpcc.html
http://www.unm.edu/~paaffair/news/news%20releas
"The National Computational Science Alliance (Alliance) will take delivery of a 512-processor Linux supercluster within the next month - a move that will give this nationwide partnership the largest open production Linux supercluster aimed at the research community. The new supercluster, called LosLobos, will be located at the University of New Mexico's (UNM) Albuquerque High Performance Computing Center (AHPCC), one of the Alliance Partners for Advanced Computational Resources sites."
"It's tough to be bilingual when you get hit in the head."
Well, if you can keep most of your working set in the 2MB L2, you can stay off of the bus more often, so there won't be so much holdoff or contention with the other proc... but a switched fabric is better for SMP than a bus... far more costly, though. Can't wait for a nice dual athlon board my self... Where's the 70 from? Looks like you are referencing the Alphas there (which use the same bus as the Althlons)... the IBM cluster is 256 boxes, so if they are SMP, you have 512 procs, and like I said, if your software is designed properly, you shouldn't be heading to memory constantly anyway (though this *is* unavoidable sometimes).
"It's tough to be bilingual when you get hit in the head."
Huh? Finite differences are still very widely used; most (like 80%+) of CFD is done using finite differences. Actually, in any model that you're examining a finite volume of space, rather than an object, you're most likely using finite differences (i.e. fluid problems, including AFAIK weather simulation).
Finite elements are mainly used for objects/mechanisms, particularly structures analysis (incl. car simulation) and manufacturing (metal stamping).
engineers never lie; we just approximate the truth.
There are 4096-machine clusters, but the marginal performance gain per machine drops for many kinds of computation because of the 'segmented' archictecture. That is, the limited bandwidth inter-cluster relative to the bandwidth intra-cluster makes programming the whole cluster a problem of finding a 64-way very loosely-bound approach, each segment of which is 64-way loosely-bound (each segment of which is N-way tightly bound because of the possibility of SMP). Finding algorithms to split up problems this way is very difficult, and in some cases, impossible. For a given problem, there is a network width/speed for which the limiting factor is the processor speed (e.g. you're not losing performance to overhead); this is the case for many more problems at switched N bps than at shared N bps (in/out-side a cluster).
On the other hand, there are algorithmic techniques for masking (network) latency (e.g. time-skewing), so it's possible to make better use of 'loosely-coupled' (relative to the algorithm's interconnect requirements) compute elements (machines/clusters/etc) than you may think.
-_Quinn
Reality Maintenance Group, Silver City Construction Co., Ltd.
I really think that more attention needs to be payed to clustering technologies. I first got started with my cluster about 18 months ago. Beowulf was fairly new, and almost considered to be a black art. After reading and tinkering for a few weeks, I was amazed at how easy it actually is to get things running in parallel. Now, don't get me wrong, like many things its easy to do but hard to do *well*...nevertheless, I'd sure like to see more activity in this area. (IMHO, SMP is going to run out of steam quickly)
All you slashdotters with three or more systems in your basement! Go! Get them networked! Load MPICH or PVM! This should be your mantra:
"If SuperID can do it, then I certainly can"
So to answer your question, no. MS clustering is really more for failover then load balancing. The load balancing works but not nearly as well as you like. The basic problem is that Win2k/NT is not designed to cluster at its core. Linux can be made to do that and thus has a destinct advantage.
-- soldack
Interesting.
Were the Linux cluster users using gcc/g77? It is well known that (at least for most scientific codes) you can get 50-100% speedup by switching from the GNU compilers to commercial ones from Portland or elsewhere.
If there is still a difference, then the next thing to try is the latest dev kernels, which have better SMP (if SMP nodes are used), and significantly faster disk io through the elimination of double caching.
Since most scientific apps should spend most of their eating user CPU cycles, I wouldn't expect there to be very much difference between one OS and another, however node uptime and more established remote admin are points in Linux' favour for big clusters.
It doesn't seem like a particularly techincal article, so someone probably just glossed that particular detail over. Your average Joe won't remember it anyway and someone may have been trying to avoid confusing the reporter.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Why does the "Doesn't Scale Well" MS FUD persist in the face of stories like this? Is there an NT cluster anywhere in the top 500 much less the top 100?
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
I dunno... My guess, if any computer were to download all of the internet, and be able to categorize it, it would be cracked of all its porn.
Just one man's guess...
The inter-machine communication requirements for genetic programming are low. Basically each machine can operate independently, and at the end of each generation, transmit only the fitness of each individual evaluated back to the server - very little data has to be sent over the network. See http://www.genetic-programming.com/machine1000.htm l
In fact, they're using a very standard 100Mbps ethernet, while my impression is that the IBM supercomputer will be much more tightly coupled. The GP cluster is rate at 0.37 Thz, about the same as the IBM machine.
On a side note, genetic programming will be an ideal for distributed.net; it's disappointing that GP isn't being attempted there.
Now, these Netfinity machines at best are 550 MHz Xeons (according to the best model I saw in IBM's Homepage) so I seriously doubt they outrun the Beowulf cluster of 1000 Pentium 2 350MHz machines (and the controlling host) being used for Genetic Programming, were they in fact clustered together to achieve the type of speed benchmarks IBM was after, rather than being used for a useful purpose as these are. Do they even outrun the 70 machine cluster of 533MHz DEC Alpha's that had previously been used for Genetic Programming? I doubt it. I tried to submit the Genetic Programming thing as an article once a while back, but it was rejected for some reason or another, even though someone posted about it in a forum like this once long ago, and people seem to continuously forget about this amazing cluster and what possibilities it presents to the computing world. Imagine if you told it to try to create a better version of itself? Once we have the storage capacity (the Petabyte, theorized to be necessary to store the totality of a human consoiusness) what would happen if you give it a pipe to the internet and told to to absorb data, correlate it with data it already has, "remember" or "forget" the data as is considered relevant based on things it already "knows"? Anyway. that's beyond the point... Which is this IBM cluster isn't amazingly new or ground breaking at all, and I have to doubt IBM's claim as fastest.
"Inveniemus Viam Aut Faciemus" 'We will find a way... Or we will make one!' --Hannibal of Carthage
Come on, IBM! We want to know how fast this thing is in BogoMIPS.
Excuse me? "Darn barrell shifters are so expensive in hardware"? Have you ever even seen a hardware VLSI design tool? Have you even heard of Verilog? In terms of hardware cost, a barrel shifter takes much less space than a fast carry-save or carry-branch adder. Flinging bits around is something that hardware is very good at doing cheaply. Arbitrary permutations basically boil down to renaming the inputs by shifting the output positions. While this is not exactly easy to implement for the general case, shifting or rotating bits, especially if the size of the object being rotated == the natural word size of the processor, is absolutely trivial. Even the naive implementation (selection tree ~5 clocks, forward, permute, issue) takes only 8 clocks, is easily pipelined, and takes marginal space.
While I'm all in favor of breakfast cereal based supercomputers, particularly the linux driven variety, like everything else, there are many unforseen problems in implementing such a device. I've built seven such clusters myself (including a personal 60 box cluster of old 486 and P1s. It's mighty nifty), and each time I run into the same fscking problem.
Squirrels.
That's right, squirrels.
It has nothing to do with software conflicts, or processers overstepping each other. That can all be taken care of with a little bit of clever coding and hacks/workarounds. But that doesn't take care of the squirrel problem. Everytime I finally work around all the network tomfoolery, and get the power for umpteen boxes managable, like clockwork the squirrels come.
And they come not in single spies, they come in batallions.
It's never the same. Sometimes they just chew the wires. Sometimes they try and make off with a box or two. What the hell does a rodent need with a computer?!?!? Wait, I don't want to know the answer to that. My repeated attempts at hunting down and exterminating the wascally bastards are met with comic hijinxs and failure. And as far as I'm aware, there aren't any open sourced squirrel repellant systems. I can't trust a proprietary system to not conflict with the many many tweaks I've made to the system. But alas, I'm stuck with my ACME catalogue and a variety of clever devices which only fail and fail again, each one making me look successively worse.
So let's hope IBM can manage a good rodent-security system, and release it back into the community. God knows I've tried. I'm sure they will realize the importance of this issue after the first few attacks. This is a much overlooked problem, but we need a solution. And as soon as possible.
thankyoutheend.
A better account of what's going on -- complete with a description of how Beowulf is used -- can be found at http://www.linuxplanet.com/linuxplanet/reports/162 6/1/
it is on top500.org
You jest, but actually there is an NT supercluster within the National Computational Science Alliance. See here for more information. I was part of the original group which developed this clustering technology while it was a research project in the computer science department. Now that it has been deployed as a real computation resource, one of my projects is to make it available to the national computational grid which the Linux article discusses.
Deploying an NT cluster was certainly a challenge in some ways that would have been easier with Unix, but not impossible. Some of our collaborators have published results favorably comparing the performance of the NT supercluster to that of Linux clusters, so there seem to be good reasons to continue building at least some technology like this on NT.
Sorry couldn't resist.
More race stuff in one place,
than any one place on the net.
There is no mention of Beowulf in the article,
just "special clustering software"
Well to really drive this redundant topic into the gound. How about, maybe the moderator thought the beowulf cluster of beowulf clusters was kind of reapetedly redundant and then the repeated repeating of repeating the original repeating of the beowulf of beowulfs was being repeated and this was eyed, seen and viewed as redundant, then the moderating moderator moderated the repeated post outlining and stating in the verbage of the written post that in his/her point of view and from his/her mindset he/she thought that the repeated repeating of repeating the original repeating of the beowulf of beowulfs was being repeated and this was eyed, seen and viewed as redundant. So the moderating moderator moderated the repeated post outlining and stating in the verbage of the written post that in his/her point of view and from his/her mindset he/she thought that the repeated repeating of repeating the original repeating of the beowulf of beowulfs was being repeated and this was eyed, seen and viewed as redundant. This being said, the moderator more than likely tagged the repeatedly repeating repeat posts stating the redundancy of the redundancy was redundant.
More race stuff in one place,
than any one place on the net.
it will only rank 24th on the list of the top 500
I like the comment that its "only" 24th. As though being only the 24th richest person alive, or only having the worlds 24th fastest car would also be something to be sneezed at.
Anyway, is there a list of world supercomputer rankings?