Slashdot Mirror
IBM Sets Supercomputer Speed Record
T.Hobbes writes "IBM's BlueGene/L has set a new speed record at 36.01 TFlops, beating the Earth Simulator's 35.86 TFlops, according to internal IBM testing. 'This is notable because of the fixation everyone has had on the Earth Simulator,' said Dave Turek, I.B.M.'s vice president for the high-performance computing division. The AP story is here; the NY Times' story is here."
It's a sort of two layer system. The compute nodes (2 cpus per compute node) run a IBM proprietary, very small and simple, kernel. 64 compute nodes are managed by an i/o node running Linux.
A quick Google search has netted the following: OS - Linux, HPK (High Performance Kernel) Complilers - Fortran95, C99, C++ Math Library - a subset of ESSL If you would like to read the article, it can be found at http://www.llnl.gov/asci/platforms/bluegene/talks/ gupta.pdf
Here's a bit more: each node has 2 cpus and 4 fpus, custom non-preemptive kernel
application program has full control of all timing issues kernel and application share same address space
kernel is memory protected
kernel provides: program load / start / debug / termination file access all via message passing to IO nodes
I could go on and on but it's all on Blue Gene's site http://www.research.ibm.com/bluegene/index.html
I can't resist adding that GCC won't use the second FPU on each die...
Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.
"About IBM's Blue Gene Supercomputing Project Blue Gene is an IBM supercomputing project dedicated to building a new family of supercomputers optimized for bandwidth, scalability and the ability to handle large amounts of data while consuming a fraction of the power and floor space required by today's fastest systems. The full Blue Gene/L machine is being built for the Lawrence Livermore National Laboratory in California, and will have a peak speed of 360 teraflops. When completed in 2005, IBM expects Blue Gene/L to lead the Top500 supercomputer list. A second Blue Gene/L machine is planned for ASTRON, a leading astronomy organization in the Netherlands. IBM and its partners are currently exploring a growing list of applications including hydrodynamics, quantum chemistry, molecular dynamics, climate modeling and financial modeling."
-from the IBM website
Moderation: +1 pwnage
For a great deal of detail about this system surf over to this pdf
http://www.busyweather.com/
Off the shelf is used loosely here. The BlueGene processors are indeed custom, but they happen to be based on the PowerPC 440 processors. That is, if you go buy a machine with a PowerPC 440 cpu, it's not exactly the same as what's in BlueGene. It is mostly the same though. What is pretty interesting is that each of the cpus is pretty paltry (the DD1 chips run at 500MHz, and the DD2 chips run at 700MHz), but the overall architecture seems to scale pretty well.
TZ
I'll be very interested in seeing how well this thing performs on benchmarks other than linpack.
Blue Gene is a very interesting design in so much as it uses IBM's 32-bit powerpc cores, normally used for embeded applications. They put 2 cores on a die, and integrated a memory controller, as well as the 4 different interconnect networks. The cores are only clocked at about 800mhz, and are thus pretty wimpy individually. However, that can be good. Since the processor cores are quite modest, the ratio of memory bandwidth to CPU flops is quite high. Similarly the ratio of interconnect bandwidth to CPU flops is also very high. Thus the CPUs should run very efficiently on problems that will parallelize to thousands of cpus. Some problems, on the other hand, will perform terribly. I expect a lot of this system's performance depends on the scalability of the system software, and the compilers / libraries.
That said, the earth simulator is also really good at some applications, and not so good at others. Instead of 16,000 small CPUs, it uses 5000 massive vector CPUs. Each is clocked at only 500mhz, but has 8 parallel execution pipes, and about 50GBytes/sec of memory bandwidth. Problems that don't vectorize run through the very modest 500mhz scalar unit.
Earth simulator has realized a large percent of it's theoretical peak performance on real world simulations (often up to 50%) while most large systems approach (10%). I'm looking forward to see how well utilized Blue Gene is. Earth simulator was a direct descendant from NEC's sx-series supercomputers, which have a 20 year lineage. Blue Gene is a radical departure from IBM's regular HPC product offerings, and uses a new microkernel OS rather than clustered AIX nodes. I imagine there will be some stutter-steps in the early days of this new product, which will undoubtedly work themselves out over time.
Great work IBM.
The peak of VTs System X cluster was about 17 Tflops, and the sustained rate was just over 10 (which rendered it the third place on the Top500 list). This peak/sustained ratio is significantly less that Earth Simulator's 90% efficiency, but compared to the cost it's extremely cost effective. ES cost 100 times more but have just 3 times higher sustained rate.
- Henrik
- when the Shadows descend -
Did they use infiniband? Or a proprietary interconnect, perhaps?
Proprietary. Actually, it has 3 networks, one mesh network for point-to-point communication, one tree network for collective communication and a service network for disk i/o, control, health monitoring etc. The service network is ethernet IIRC, the other two are custom.
The top500 tecords are submited on an honor system. Most of the systems are thrown together with known processors and interconnects where the tesults should "make sense". Also, the systems teport their theoretical max performance and a measured tesult. It would be pretty hard to fudge a score for the top500 by much without many people questioning it. From this page the top500 people say:Its kinda like any tesearch field. Most people are honest, but anomolies can and do happen, and they are usually found out by others in the field. Two of the most tecent scientist scandles involved the guy from Bell labs, Hendrik Schön, who was found falsifying data, and he was fired, and I believe that he also lost his PhD. The other is from the US government funded tesearch on MDMA by George Ricaurte. Although I believe that nothing really happened in the Ricaurte case.
comes from building hardware for a specific task. Unfortunately most of you can't access this little bit of nerd heaven but some incredibly cool hardware architectures are being described at the High Performance Embedded Computing conference. Sky and Mercury have some of their hottest new designs here. How about a machine that can do a 256 mega-sample FFT in real time?, or a self configuring supercomputer on a chip? Of course most of these tricks will never escape the lab except for the speed-ups for rendering engines...one place where gamers and the DOD are driving technology in a dead heat race with lots of winners. Besides, in a few months, something will come along that will go even faster than blue gene.
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.