Slashdot Mirror
IBM Sets Supercomputer Speed Record
T.Hobbes writes "IBM's BlueGene/L has set a new speed record at 36.01 TFlops, beating the Earth Simulator's 35.86 TFlops, according to internal IBM testing. 'This is notable because of the fixation everyone has had on the Earth Simulator,' said Dave Turek, I.B.M.'s vice president for the high-performance computing division. The AP story is here; the NY Times' story is here."
It's a sort of two layer system. The compute nodes (2 cpus per compute node) run a IBM proprietary, very small and simple, kernel. 64 compute nodes are managed by an i/o node running Linux.
A quick Google search has netted the following: OS - Linux, HPK (High Performance Kernel) Complilers - Fortran95, C99, C++ Math Library - a subset of ESSL If you would like to read the article, it can be found at http://www.llnl.gov/asci/platforms/bluegene/talks/ gupta.pdf
Here's a bit more: each node has 2 cpus and 4 fpus, custom non-preemptive kernel
application program has full control of all timing issues kernel and application share same address space
kernel is memory protected
kernel provides: program load / start / debug / termination file access all via message passing to IO nodes
I could go on and on but it's all on Blue Gene's site http://www.research.ibm.com/bluegene/index.html
I can't resist adding that GCC won't use the second FPU on each die...
Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.
"About IBM's Blue Gene Supercomputing Project Blue Gene is an IBM supercomputing project dedicated to building a new family of supercomputers optimized for bandwidth, scalability and the ability to handle large amounts of data while consuming a fraction of the power and floor space required by today's fastest systems. The full Blue Gene/L machine is being built for the Lawrence Livermore National Laboratory in California, and will have a peak speed of 360 teraflops. When completed in 2005, IBM expects Blue Gene/L to lead the Top500 supercomputer list. A second Blue Gene/L machine is planned for ASTRON, a leading astronomy organization in the Netherlands. IBM and its partners are currently exploring a growing list of applications including hydrodynamics, quantum chemistry, molecular dynamics, climate modeling and financial modeling."
-from the IBM website
Moderation: +1 pwnage
For a great deal of detail about this system surf over to this pdf
http://www.busyweather.com/
Off the shelf is used loosely here. The BlueGene processors are indeed custom, but they happen to be based on the PowerPC 440 processors. That is, if you go buy a machine with a PowerPC 440 cpu, it's not exactly the same as what's in BlueGene. It is mostly the same though. What is pretty interesting is that each of the cpus is pretty paltry (the DD1 chips run at 500MHz, and the DD2 chips run at 700MHz), but the overall architecture seems to scale pretty well.
TZ
I'll be very interested in seeing how well this thing performs on benchmarks other than linpack.
Blue Gene is a very interesting design in so much as it uses IBM's 32-bit powerpc cores, normally used for embeded applications. They put 2 cores on a die, and integrated a memory controller, as well as the 4 different interconnect networks. The cores are only clocked at about 800mhz, and are thus pretty wimpy individually. However, that can be good. Since the processor cores are quite modest, the ratio of memory bandwidth to CPU flops is quite high. Similarly the ratio of interconnect bandwidth to CPU flops is also very high. Thus the CPUs should run very efficiently on problems that will parallelize to thousands of cpus. Some problems, on the other hand, will perform terribly. I expect a lot of this system's performance depends on the scalability of the system software, and the compilers / libraries.
That said, the earth simulator is also really good at some applications, and not so good at others. Instead of 16,000 small CPUs, it uses 5000 massive vector CPUs. Each is clocked at only 500mhz, but has 8 parallel execution pipes, and about 50GBytes/sec of memory bandwidth. Problems that don't vectorize run through the very modest 500mhz scalar unit.
Earth simulator has realized a large percent of it's theoretical peak performance on real world simulations (often up to 50%) while most large systems approach (10%). I'm looking forward to see how well utilized Blue Gene is. Earth simulator was a direct descendant from NEC's sx-series supercomputers, which have a 20 year lineage. Blue Gene is a radical departure from IBM's regular HPC product offerings, and uses a new microkernel OS rather than clustered AIX nodes. I imagine there will be some stutter-steps in the early days of this new product, which will undoubtedly work themselves out over time.
Great work IBM.
http://foldoc.doc.ic.ac.uk/foldoc/foldoc.cgi?query =teraflop
Cruise TT
The peak of VTs System X cluster was about 17 Tflops, and the sustained rate was just over 10 (which rendered it the third place on the Top500 list). This peak/sustained ratio is significantly less that Earth Simulator's 90% efficiency, but compared to the cost it's extremely cost effective. ES cost 100 times more but have just 3 times higher sustained rate.
- Henrik
- when the Shadows descend -
Did they use infiniband? Or a proprietary interconnect, perhaps?
Proprietary. Actually, it has 3 networks, one mesh network for point-to-point communication, one tree network for collective communication and a service network for disk i/o, control, health monitoring etc. The service network is ethernet IIRC, the other two are custom.
I have no idea of what you mean by sustained.
He is refering to the fact that horsepower has a time componant. It's only in rare conditions that you're interested in the instantaneous force a horse can apply. What you want to know is how much work you can get out of it per day.
A cheetah may be able to sprint to 100 kph, but I'll out distance it in 10 minutes driving my car at only 80 kph.
Human hunters on foot can only run about 15 kph, but can chase down large prey that can run 65 kph, because the human can run at 15 kph all day, day after day, but the prey animal can only run at 65 kph for a very limited time before becoming exhausted and needing to stop.
The amount of time you are expected to perform some function in is a critical parameter.
Running a 4 minute mile is one thing, but there's a word for someone who can run 26 of them back to back:
Loser.
KFG
I've been down in the basement of the building, see a few of the towers 1/2 loaded (at that time), along with the massive cooling system that was added to the building to keep those racks workings. Lift up a raised floor panel and the 95 LBS of me will get lifted off the ground (or so it feels).
Sadly, all 64 racks will never be in Roch, just not enough space.
Actually, StarTribune has one (crappy) pic of some towers.
When in danger or in doubt, run in circles, scream and shout.
The top500 tecords are submited on an honor system. Most of the systems are thrown together with known processors and interconnects where the tesults should "make sense". Also, the systems teport their theoretical max performance and a measured tesult. It would be pretty hard to fudge a score for the top500 by much without many people questioning it. From this page the top500 people say:Its kinda like any tesearch field. Most people are honest, but anomolies can and do happen, and they are usually found out by others in the field. Two of the most tecent scientist scandles involved the guy from Bell labs, Hendrik Schön, who was found falsifying data, and he was fired, and I believe that he also lost his PhD. The other is from the US government funded tesearch on MDMA by George Ricaurte. Although I believe that nothing really happened in the Ricaurte case.
When you have a machine this fast the sampler cannot keep up. As a result what you see is a total distortion of reality. You may have seen this phenomenon with wagon wheel spokes rotating backward when the cart is really rolling. Thus when you read "speed record at 36.01 TFLOPS" your eyes view the letters going backwards and forwards.
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
Problems run on clusters have to be able to be broken down into small pieces that don't need to interact with the other pieces very much. This is so because of communication latency in such a system. Someone mentioned SETI, which runs great on a cluster, because looking at the signal from one piece of sky requires no information about the surrounding sky. However, something like a nuclear explosion, broken into pieces, requires lots of information about the surrounding environment. What's worse is that as you make the pieces smaller when simulating a nuclear explosion, your need for knowledge about the surrounding increases! Such simulations require a much more tightly coupled system, a 'traditional' super computer.
comes from building hardware for a specific task. Unfortunately most of you can't access this little bit of nerd heaven but some incredibly cool hardware architectures are being described at the High Performance Embedded Computing conference. Sky and Mercury have some of their hottest new designs here. How about a machine that can do a 256 mega-sample FFT in real time?, or a self configuring supercomputer on a chip? Of course most of these tricks will never escape the lab except for the speed-ups for rendering engines...one place where gamers and the DOD are driving technology in a dead heat race with lots of winners. Besides, in a few months, something will come along that will go even faster than blue gene.
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.
You deserve some credit for using "when", not "if" (IMHO)
The system is designed to work around failure. In the original protein folding simluation, the plan was (among other things) to checkpoint the system every hour in order to be able to restart if a failure occurred. In fact, the original expectation was that a processor would fail every few days (that presentation has since been taken down by IBM... was originally named "BlueGenePublic.pdf" ... I can't find it anywhere on the 'net anymore :( ). Failure detection is through a series of ECC bits attached to most of the data transfers and calculations. The software is also specifically written to check any points where conservation is true (meaning there is redundancy in the application and calculations are checked to ensure no errors). The mesh network that others have referred to also allows nodes which are not functioning to be worked around before they are replaced.
The processors cannot be replaced individually, but the boards (with 2 chips and memory) could be. As far as burning, the processor will often fail in a detectable way (meaning produce incorrect results) long before the device goes (literally) up in smoke. So I would expect the system would be able to disable the failed device long before its a problem. This may be part of what the Linux OS (which oversees 64 of the small processors) is doing. On a wider scale, the designers are smart enough that there are likely temperature sensors in the system that will slow (or shut down) the system if a heat problem is encountered.