More Details Of IBM's Blue Gene/L

Bob Plankers writes "By now we've all heard about IBM's Blue Gene/L, LLNL's remarkable new supercomputer which is intended to be the fastest supercomputer on Earth when done (360 TeraFLOPS). IBM has released some new photos of the prototype, and renditions of the final cluster. Note that the racks are angled in order to permit hot air to escape vertically and reduce the need for powered cooling. The machine uses custom CPUs with dual PowerPC 440 processing cores, four FPUs (two per core), five network controllers, 4 MB of DRAM, and a memory controller onboard. The prototype has 512 CPUs running at 700 MHz, and when finished the entire machine will have 65536 dual-core CPUs running at 1 GHz or more. Stephen Shankland's ZDnet article also mentions that the system runs Linux, but not on everything: 'Linux actually resides on only a comparatively small number of processors; the bulk of the chips run a stripped-down operating system that lets it carry out the instructions of the Linux nodes.'"

Comments

Doom3?

[arcanumas]

Well, it may be able to play Doom3 when it is released.

Turing machine will be Turing machine

[incal]

no matter how many cpu's it will get. Maybe its better to invest time and resources in the David Deutsch research of quantum machines? http://www.qubit.org/people/david/David.html

Re:Turing machine will be Turing machine

[IWannaBeAnAC]

And what gave you the impression that this research was not being done?

On the other hand, it is nice to have a fast computer to play with now, not in 50 years time!

Infinite

[Raynach]

I'm really impressed with this computer. I think it's going to be the first computer that can finish an infinite loop in under an hour.

Re:Infinite

[Bronster]

If an infinite loop is infinite, how can it be finished?

That would be the sound of the joke wooshing around, and around, and around, and ... your head.

Travelling salesmen.

[Manywele]

This will be sure to boost the effeciency of travelling salesmen everywhere.

Re:Travelling salesmen.

[Vegard]

Actually, we better let the salesmen travel. It's a little known secret that the reason computers are so bad at solving the travelling salesmen problem is that those who design computers are technicians, and everyone knows that tech people hates salesmen, so the longer they spend travelling, the better for the techs.

"4MB of DRAM"

[vogon jeltz]

... those were the times. Ahhh, memories!

Re:it's all cool and everything...

[krumms]

...but can this computer actually run:

10 LET x = 1
20 LET y = 2
30 PRINT x + y

This seems to be a "does it run Linux?" joke gone horribly wrong.

Or is that kernel code you're posting? :P

4 MB DRAM

[Sensei_knight]

Holly shit where do I buy on of thoes!

Re:4 MB DRAM

[otis wildflower]

4MB per CPU, each with 2 processing cores, and an onboard memory controller.

Final version to have 65536 CPUs.

Smells like 256GB to me, which is pretty decent in _any_ book, especially if it lives on the same silicon as the CPU...

Re:4 MB DRAM

[javiercero]

Because they are used embedded DRAM, which although not as fast as SRAM it gives more storage in fewer transistors. This leads to a smaller die, and lower power/heat dissipation.

If your p1 runs at the same speed than your P4 for 90% of operations, then there is something wrong with your computer! The HDD is not the bottleneck for most modern computers, as they have enough memory to minimize page faults for most common home computing tasks.... Startup times however may be equal since both machines have to get the data/program from HDD... once stuffs are in memory, buubbie P1....

Subjective...

[Decameron81]

"The prototype has 512 CPUs running at 700 MHz, and when finished the entire machine will have 65536 dual-core CPUs running at 1 GHz or more."

Woah, this is the first time I think a box with 512 CPUs at 700 Mhz each one is crap.

Diego Rey

It's gorgeous...

[mOoZik]

I think I wet my pants.

Re:One should read 512 MB DRAM

[ezh]

It's gonna be 512 MB for BlueGene/L(ite) and 1Gb for proper BlueGene

I mean, per node :-)

AFAIK, 512 Mb is just too little for proper protein-folding calculations, while 1Gb provides enough capacity... And, of course, no swap is possible in this types of systems

Re:Only a PPC 440?

[shaitand]

No, It's a supercomputer. Those are RISC processors, a PPC 440 in reality gives better performance than a CISC processor like the PIII

What's new?

[LeoDV]

So, you mean they're going to build a computer that's going to be bigger, faster and with higher number stats than the current #1? Shocking!

Sorry about the sarcasm, I'm only asking to be proven wrong, but isn't Blue Gene just more of the same, only bigger? Big Mac was interesting because of how cheap it was and because it was the first of its kind to use Macs, the Earth Simulator was interesting because it brought back custom chips for supercomputing as opposed to off the shelf components, we've been reading about IBM's dishwasher-sized supercomputer, articles about efficient supercomputing, so what's new about Blue Gene, besides being newer and bigger?

Once again I'm not bashing, I haven't read much of anything but the /. blurbs, so I'm asking, is it just a bigger supercomputer, or does it have any "real" innovations?

Re:What's new?

[javiercero]

Well... Earth Simulator was bigger and faster... as was big mac. Every advance in computer design and fabrication has been about "bigger and faster." It may sound trivial to you, but that is because you got no idea about what is involved about making things "bigger and faster."

What is significant about blue gene is that is some sort of compromise between off the shelf parts (PPC based Processing elements vs. the Earth Simulator SX based custom vector PEs), and efficient interconnection (plain crappy cluster like the Big mac with a better interconnect at multiple layers starting with dual cores per die).

In the end it all leads to the same goal: tackling bigger problems faster. So it may sound trivial but there is a lot of research going into this baby.

Re:What's new?

[tigertiger]

Beowulf-style cluster are a big waste in terms of additional circuitry and hardware that you do not need in a supercomputer, from I/O busses to the power supply. Of course, since off-the-shelf components are so cheap, it is still cheaper to buy the stuff than to design your own tailor-made circuitry - up to a certain scale.

That is where IBM tries to go: BlueGene's design is based on a system-on-a-chip - everything (except memory) is integrated on a single chip. In the long run, this allows them to build systems much larger than you could with a Beowulf. They are basically aiming for a system where you can easily add computing power by simply putting in a few more chips, and the thing will scale. They are doing the same thing for storage with this brick

BlueGene is a also the first supercomputer marketed to the life sciences. It's interesting to see that it developed from a project at Columbia University called QCDOC for "Quantum Chromodynamics on-a-chip" which did research in computational high-energy physics, and QCDSP before, which used DSP processors to build a supercomputer about ten years ago. Both an instructive example how academic research in the long run becomes industrially relevant, and how science changes.

I'm surprised!

[Epsillon]

The standard of trolling has certainly fallen recently. Where's the SCO licence fee estimate for the finished 65536 processor SMP unit? You got a better class of idiot in those days... ;o)

Re:I'm surprised!

[Ianoo]

IBM better not forget to pay SCO their $45,809,664!!

The Racks Are Not Angled

[The Other White Meat]

If you actually look at the picture, closely, you will see that the racks themselves are NOT angled to reduce active cooling.

At the left side of the row of racks, there is an angled cover, which is either decorative, or being used to force cold air down the row of racks. Likely, its just decorative, and the cold air is being forced up from the raised flooring below.

Just like it is in every other enterprise-grade computer room...

Re:it's all cool and everything...

[gazbo]

Yes. The thought of a creature with two X chromosomes is horrifying.

See for yourself

[mangu]

save this as "test.c":

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <sys/time.h>
#include <unistd.h>

#define N 1000

struct timeval tv;
struct timezone tz;
long bu, bs, du, ds;

int main()
{
int i;
double t;

printf("Infinite loop test\n");

gettimeofday(&tv, &tz);
bs = tv.tv_sec;
bu = tv.tv_usec;

for (i = 0; i > -1; i++)
;

gettimeofday(&tv, &tz);
du = tv.tv_usec - bu;
ds = tv.tv_sec - bs;
t = (double)ds + (double)du / 1e6;
printf("executed in %13.6f seconds\n", t);
}

compile and link with:

gcc -g -o test test.c

run:

Infinite loop test
executed in 3.888419 seconds

Re:See for yourself

[uid8472]

My computer has 64-bit ints, you insensitive clod!

No, *Smaller* and Faster

[RalphBNumbers]

What's big about BlueGene/L is that it's small. That 512 processor prototype they mention in this article is the Dishwasher-sized computer you heard about.

BlueGene/L is about driving down the cost of supercomputing, not only in terms of money spent on hardware, but in terms of space, cooling, and maintanance, while at the same time improving scalability.

BlueGene/L is going to put 65,000+ processors in less space, using less power, and costing less, than many of todays >10,000 processor systems.
They do this with a minimalist approach, each processor is a SoaC (System on a Chip), with everything from the memory controller to internode networking to two cores and 4FPUs on the die, and the only other thing in a node besides the processors is a bit of RAM. This allows them to use much less power per node and gives them less heat per node to dissipate, which lets them pack the nodes much closer, which cuts down on internode latency, which increases scalability.

Linux nucleus for slaved compute nodes?

[Morgaine]

The part of the article that I found most interesting was:

Linux actually resides on only a comparatively small number of processors; the bulk of the chips run a stripped-down operating system that lets it carry out the instructions of the Linux nodes.

The "stripped down operating system" must be the distribution nucleus on the compute-only subnodes, presumably something that allow the Linux nodes to distribute the code and I/O of computations to them and to query or control their state during debugging, and to reaccquire lost processor control.

It's only a matter of time before those of us who already have sizeable LANs at home will have embedded compute-only clusters within them too. Those would differ substantially from the typical Linux clustering for high availability. Instead of a non-Linux nucleus on those subnodes though, I'd prefer to see a pretty ordinary Linux kernel running slaved to remote masters.

Is anyone already playing with something like this in their Linux clusters?

Re:Only a PPC 440?

[vicotnik]

Why did this get such a high score? Why not compare an old Sparc 4 against an Athlon 64 or a Pentium 4, the Sparc has a RISC processor so it should be faster, right? The PPC 440 might be faster for a number of reasons, and being RISC instead of CISC is hardly even among the most significant. That x86 has a crappy ISA doesn't mean CISC that has to be slower than RISC in general.

Re:Only a PPC 440?

[be-fan]

-1: Disinformative

RISC vs CISC means very little these days. Most current CPUs have a core even more minimal than RISC chips, but present a CISC (in the case of x86) or RISC (in the case of the G5) interface to the outside. They used the PPC 440 for different reasons:

1) IBM had to do significant custom engineering for it, and they own the PPC 440 core. That allowed them to use it to design an SoC.

2) They needed to add FPU hardware, which is easier to do on a design they own. The PIII only has one FPU, while this chip as 2 FPUs. IBM had to add this to the design, because the regular PPC-440 has no FPUs.

3) The PPC-440 was designed from the beginning to be an embedded CPU. At 1GHz, a stock PPC-440 consumes about 2.5W. Even a low voltage PIII consumes more than that.

OT: Hire a photographer (me ;)!

[MasTRE]

Besides the fact that their Nikon D100 has a stuck (hot) pixel, the pictures of people (first "set" on the page) are really bad quality-wise and there is not much creativity - i.e. two shots of the same geek (Hall) taking heatsink temperatures from slightly different angles aren't exciting even to fellow geeks.

Other than that, keep up the great work IBM!

Re:it's all cool and everything...

[42forty-two42]

Don't be silly, kernel code would be:

#include &ltlinux/config.h>
#include &ltlinux/module.h>
#include &ltlinux/kernel.h>
#include &ltlinux/types.h>

MODULE_LICENCE("GPL");

__asmlinkage inline unsigned int add_x_plus_y(unsigned int x, unsigned int y){
unsigned int ret;
spin_lock_irq(&current->arith->lock);
current->arith->accum = x;
current->arith->oprand = y;
__perform_add(&current->arith);
ret = current->arith->accum;
spin_unlock_irq(&current->arith->lock);
return ret;
}

Re:Dumpsterdiving seems a waste of time at IBM

[treat]

In that picture you can also see sprinklers! Oh my.

You're all Missing the point here...

[shrugwhaa]

I hear:

Oh wow, another technical marvel

Oh Gee, another super computer...

Morons...

The whole point here, is that it makes the simulation
of folding a complete gene in about a years time.

If THAT doesn't bowl you over, don't post.

p.s. I can hear the rest of you "umm... so?" people and I can't help you. Sorry. :)

Wouldn't it be great...

[mgeneral]

...If they ported over VMWare to run on this bad boy? Imagine the number of guest OS's you could run. This thing could be the data center of all data centers.
But otherwise, for all intents and purposes, its extremely proprietary and will ultimately run just a few specialized applications.
Never-the-less, with virtualized computing and beheamoth systems like these, the future of data centers is sure to change.