Software to Make Blue Gene Top 200 Teraflops

An anonymous reader writes "New Scientist has a story about the most intensive computer program ever created. It runs on IBM's big beast, Blue Gene/L, at Lawrence Livermore National Laboratory in California and carries out 207.3 teraflops (trillion cacluations per second). The program, called Qbox, performs very complex quantum calculations to simulate the behaviour of thousands of atoms in three dimensions. Wow."

Comments

Slight clarification

[LiquidCoooled]

It does not perform very complex quantum calculations, instead
It simulates interactions between 1000 molybdenum atoms under high pressure using equations that take the quantum behaviour of electrons into account.

Also, when its not being used to dynamically model atomic structures, the IRS uses it to calculate Bill gates's taxes.

Re:Slight clarification

[rolfwind]

And it almost makes the requirements for Vista!

Re:Slight clarification

[Memnos]

At the unfortunate risk of repeating myself on Slashdot (Oh, the Humanity!) you are correct. It is intrinsically impossible for a discrete-state system to model quantum mechanical events, unless you somehow sneaked under the Planck limit (There is no spoon..) So, they're faking it.. However, if it is a good model of "reality", then it is good science. If it can predict, it is useful.

Re:Slight clarification

[Bill Barth]

It's not "fake" so much as it's an approximation. I guarantee you the know by exactly how much they are in error (but not in what direction!). The Schroedinger Equation that is at the heart of this represents the probability (well its modulus does, at least) of something as a continuous function of space and time. These scientists make errors in that the equations that they use are discrete (in terms of mathematical degrees of freedom, strictly speaking, by discretizing space and time directly) models of the Schroedinger equation and in that the initial and boundary are not perfectly well known. That doesn't constitute "faking it" in my book. If they were faking it, they'd be making pretty pictures with no predictive value, and presuably their work makes good predictions, which, as you note, puts it in the category of "good science."

Re:Slight clarification

[mfago]

impossible for a discrete-state system to model quantum mechanical events
Huh? QM was a while ago, but I'm afraid you'll have to give a reference or two. You're saying that Density Functional Theory is impossible? The authors (of DFT) did win the Nobel proze a while ago, so I'm sure I'm missing something. Mind you, any implementation is only an approximation, but that's true of almost any computational method.

Re:Slight clarification

[Memnos]

Yes. I am am saying that a discrete-state-system, such as a Markov chain, cannot follow quantum mechanical events. QM state reduction is not beforehand deterministic because it it follows a wave function that be known beforehand in its full vector state (e.g. position and velocity.) If you wish references I would need to look them up, except for my remembrances of Richard Feynman and Stephen Hawking lecturing to me on this subject, and my own experiments. But I can find them. That neither obviates your point or mine. My background is in physics but I work in computer science, and good-enough computational approximations have a way of being "good enough". Perhaps our argument is about "truth", which we model only imperfectly. Or then, perhaps we have no actual argument at all. Oh, DFT is a good and very useful theorem and practical model, but it does does sidestep the issue of QSR in some cases, as does almost any model of QM. Thx.

Yeah, but...

[wiz31337]

Yeah, but can it beat Kasparov at chess?

Re:Yeah, but...

[elrous0]

It's so powerful, it can beat Kasparov in chess and monitor millions of phone calls for the NSA *at the same time*!

-Eric

Re:Yeah, but...

[Sabaki]

No, but it's already mapped his genome and is working on a clone that will be completely under its control.

Re:Yeah, but...

[elrous0]

Geez, I am sick of getting modded down for this. Can /. please stop giving the White House unlimited mod points?

-Eric

Re:Yeah, but...

[elrous0]

we aren't loud and obnoxious like you.

when you already control everything, you don't HAVE to be loud.

As for obnoxious, well, that's in the eye of the beholder. For example, I consider monitoring my phone calls and locking people up without due process to be pretty damn obnoxious. But that's just me.

-Eric

Re:Yeah, but...

[Doc Ruby]

The real sad truth is that the arrogance of Bush voters lets you think that putting that criminal in charge of the world, then lying us into the Iraq War, letting New Orleans drown, spending our surplus into $9-45 TRILLION debt, shredding American global credibility and an endless list of other stupid, evil and/or mindless acts is just a "disagreement".

That's your excuse, but turning your responsibility for electing that criminal into a "disagreement" with people who voted for a competent president, especially after 4 years of unmistakeable evidence, is just sad truth.

Too bad for Q-box...

[vishbar]

New Scientist has a story about the most intensive computer program ever created.

Too bad for Q-Box that their title will be stripped of them so soon. Vista's almost here.

Wait a minute, Vista? Nevermind...Q-box should have it for a long while.

Re:Too bad for Q-box...

[bl00d6789]

Since QBox's title is for requiring the most computing power to carry out its intended application, Vista may well unseat it. It's just that QBox's intended application is extremely complex quantum physics calculations, and Vista's intended application is letting people check their email. So... not quite a victory for Vista.

More importantly...

[batmn42]

"Wow."

More importantly, at what FPS does it play WoW?

Though I wouldn't be surprised if it needs a new graphics card for Crysis...

Re:Linux

[spun]

Only after it calculates Bill Gates' taxes and beats Kasparov at chess, apparently.

Specs

[neonprimetime]

Specs here and yes, Suse

Only the most intensive USEFUL program

[stratjakt]

I mean, I'm sure I could use up more than 200 teraflops with my "while (1);" program.

Re:Only the most intensive USEFUL program

[frank_adrian314159]

while(1); uses no FLOPS. OTOH, if you used while (1.0);...

(And for those of you who are humor-impaired, I do realize that neither would use any FLOPS because they would both be optimized into L1: jmp L1).

Re:Only the most intensive USEFUL program

[TheSkyIsPurple]

Sorry, but I imagine you'd keep one of the many processors very busy, with the rest left idling away.
Now, spawn a thread for each processor running this, and you might have something =-)

Re:Only the most intensive USEFUL program

[owlstead]

I would not use an optimizing compiler on that one either :)

...wow...

[sarlos]

So in essence, it takes about .2 teraflops per atom... And that was only after spending a lot of time condensing the algorithms. This makes me wonder two things. First, what do these equations look like such that it takes 200 gigaflops just to model one atom. Second, over what timeframe does this simulation take place? Are we talking real-time, calculating for 50 years, what?

Regardless, as a computer scientist, I say way to go to these guys, this is damn impressive.

Re:...wow...

[MustardMan]

So you're a computer scientist, but you apparantly don't understand Big-O notation or the concept that algorithms don't neccesarily scale linearly with the number of elements.

Re:...wow...

[mhore]

So in essence, it takes about .2 teraflops per atom... And that was only after spending a lot of time condensing the algorithms. This makes me wonder two things. First, what do these equations look like such that it takes 200 gigaflops just to model one atom. Second, over what timeframe does this simulation take place? Are we talking real-time, calculating for 50 years, what?

0.2 TFlops per atom, yes. But there are 1000 atoms, and it's molybdenum which has 42 eletrons... so that's 42,000 particles that all interact with each other. Still... that's not too many. But maybe they're considering interactions between nuclei, too. Who knows...

As for your question about what the equations look like? They're probably very nasty integrals of sines and cosines and what not to various odd (read: strange) powers and stuff. I do fairly computationally intensive simulations on some big IBM machines and just simple equations can amount to quite a bit of calculations. Nothing like what these guys are doing, though.

Finally... what time frame is the simulation over? I'd wager VERY SHORT times, maybe nanoseconds or something like that. Even casual "molecular dynamics" simulations can only probe very short timeframes. Their coarse-grained cousins can maybe do microseconds or milliseconds.

Mike.

Re:...wow...

[exp(pi*sqrt(163))]

In a classical physical system the time to compute what happens to N particles typically grows as a polynomial in N. The masses and positions of the particles form a 6N dimensional space (3 for velocity, 3 for position) and you're typically trying to trace a path through that 6n-dimensional space.

In quantum mechanics the state of the system is defined by a wavefunction on a 3N dimensional space. The state of a system is no longer a point, it's a *function* on a 3N dimensional space. That means that at any position in this space the function can take any value. So you need to compute the value of this function at every point in this 3N-dimensional space. Suppose we model this really crudely. Instead of considering a wavefunction that varies continuously through this 3N-dimensional space 'discretize' this space. consider just 10 points along each of the 3N axes rather than the infinite number required by quantum mechanics. We can then model the system by computing values of the wavefunction at 10^3N points. Suppose we're dealing with 1000 atoms. Let's model the atoms really crudely as one nucleus and one electron. That means 2000 particles and a 6000-dimensional space. So we need to compute the wavefunction at 100000000...000 points, where we have '1' followed by 6000 zeroes.

I know that physicists have a few tricks up their sleeves but it seems pretty obvious to me that these guys are actually cutting a lot of corners, and to accurately model this many atoms on a computer anything like what they have sounds pretty implausible to me.

There's a quick and easy way to look at this. When you combine two classical systems the work required to simulate the combination is typically the sum of the work required to simulate them separately (modulo a polynomial). When you combine two quantum systems you need to multiply the amount of work. Combining 1000 quantum systems borders on the insane...

Re:...wow...

[gardyloo]

Because those atoms do their things on VERY short timescales. There's no way you can probe what they're doing on short enough time (and length) scales, even with pump-probe laser experiments, and track movements. Possibly, in some very special circumstances, you can look at beginning and ending states, and then figure out intermediate states. However, in general, this isn't possible, and so we need such simulations to track in-between processes, especially in ergodic systems.

Re:...wow...

[poszi]

Why not just get 1000 Molybdenum atoms and watch what the fuck they do

Because they are apparently simulating them under extreme conditions that are present during nuclear explosions. And nuclear tests are banned.

Molest me not

[Weaselmancer]

The program, called Qbox, performs very complex quantum calculations to simulate the behaviour of thousands of atoms in three dimensions.

"Molest me not with this pocket calcualtor stuff."

How to test a nuke.. without testing one

[ScottLindner]

How do they know they got it right?

Re:How to test a nuke.. without testing one

[stratjakt]

They simply check the result by hand.

Call your broker, because it's a good time to invest in pencil and paper futures.

Re:How to test a nuke.. without testing one

[LiquidCoooled]

Thats actually quite simple.

If they are modelling everything without calibration from known experimental results then anything this machine can produce is as trustworthy as internet gossip.

For instance, if you were creating a weather prediction machine (easier to explain), you would feed it with all your historical data and allow the calculations to run from a set date in the past. If the results matched up with actual observed results for the following day/week/periods then you begin to build confidence in your algorythm.
You continue this and allow it to calculate longer and longer runs, most likely tweaking your code as you go along.

To put it into real perspective, heres the real version of the simulated experiment.

Smart, sure. But is it happy?

[fred_sanford]

Oblig. H2G2. "Here I am, brain the size of a planet and they ask me to take you down to the bridge. Call that job satisfaction? 'Cos I don't." - Marvin

Just wait...

[Raul654]

BlueGene/L has a sister project, Cyclops64 (formerly known as BlueGene/C) due out sometime late in 2006 or early 2007. My research group is (a) helping IBM do hardware verification on it. and (b) developing the systems software for it [esp. the compiler]. Cyclops64 could very well blow BlueGene/L out of the water.

Re:Just wait...

[Raul654]

C64 takes a totally different approach to high performance computing. Most supercomputer architectures are built around a moderate to large number of very, very fast (and power-hungry) processors. For example, Big Mac at Virgina Tech had something like 10,000 pentium 4 class processors. Cyclops64 is have an *enormous* number of processors (on the order of a million), but running only at 500 mhz, making them much easier to cool). The idea is to give the programmer more thread units than he knows what to do with, running very close together at a low level.

Re:Just wait...

[Raul654]

Cell was designed around one single objective - to get a clock rate as sickeningly high as possible, because clock speed cells. Trust me when I say that programmability was not (at all) a consideration (I should mention - my research group got one of the very first Cell processor's sent to the US. We are currently in the process of implimenting OpenMP on it to make it a little nicer to program).

As far as writing multi-threaded code, I've spent the last 5 months rewriting the NAS CG benchmark to work effeciently on Cyclops64, which will probably play some part of my PhD thesis. (A sidenote: All of NASA's NAS implimentations are written in Fortran (except Integer Sort), which would have necessitated me rewriting NAS-CG in C. Fortunately, I didn't have to start from scratch, because the Japanese had already done the hard part).

Re:Just wait...

[Raul654]

The compiler [the current version, at any rate] is based on gcc. So it sports the same out-of-order execution you would expect to get from compile-time optimization. I am not sure if it has hardware-based re-ordering. My guess would be that no, it does not, but without the Principles of Operation in front of me, I couldn't say (the advisor borrowed my paper copy for IPDPS 2006 and hasn't given it back yet).

Re:Just wait...

[Raul654]

What you are describing has already been done, and was done quite a while ago. Around 1990, NASA realized that the way we do parallel benchmarks sucks. The way most benchmarks (including hte parallel ones) work is that some organization posts the code, and people have to compile and run the code as-is. There's not much room there for optimization (other than tweaking the compiler flags, some trivial hardware settings, 'etc), which is essential to getting good parallel performance (because parallel machines vary so widely). So performance was tied very closely to the implimention over which nobody had any control.

NASA approached the problem differently. Their numerical analysis group put out a set of "paper and pencil" benchmarks (based on real world problems that one would encounter, for example, fluid dynamics). The actual implimentation was left up to the individual companies. This is what we know today as the NAS benchmark suite.

I suspect the answer ends up

[jhw539]

42.

it doesn't work like that

[tpjunkie]

It doesn't take .2 teraflops to model one atom, or even two atoms, even account for effects on the quantum level.. However, when you take into acount that each atom will more or less interact with every other atom, you have a massive amount of interactions to model. Thats what takes so much processing power.

Fill in Blank Please

[Frightening]

Imagine a _________ cluster of those.

Well done, you may now enter. Gaming room to the right, pron cubicles left, and crazy linux hardware center up ahead.
We hope you enjoy your stay at Geek Heaven.

HPCWire Interview

[multimediavt]

http://www.hpcwire.com/hpc/699401.html

There's some additional info about BlueGene and what Livermore thinks of it here. What this interview neglects to mention is the millions of dollars being spent on IBM and internal developers to get this code (and any others) working on BlueGene. I was briefed by the hardware and software teams that built BlueGene and I can tell you, it's no easy task to bring apps to that platform. Kuznezov seems to trivialize it in the interview and I'm gonna have to go back and review the process again. Maybe it has changed since my briefing in early 2004, but somehow I doubt it.

Screenshot here

[ultramk]

I wonder what the cubes represent?

Oh, wait. Qbox. Nevermind.

m-

Quantum Monte Carlo

[poszi]

First, what do these equations look like such that it takes 200 gigaflops just to model one atom.

The article is light on details but I suppose the only quantum algorithm that can handle 1000 atoms is Quantum Monte Carlo. The problem is that the algorithm is cubic with the number of particles (and has a huge prefactor). So in essence 1000 atoms is 1000^3=10^9 more time consuming than one. And I'm sure they still use dramatic simplifications, even though they have the most powerful computer. They probably do not consider all electrons, instead they use pseudopotentials. And the Quantum Monte Carlo is likely in a fixed-node variant which is approximate. How long does it take? It's hard to tell but probably a few hours or days each and they are performing several those with different conditions.

A few more iterations

[ender_]

Imagine, if you will, taking this super-computing ability out a few years. Can the U.S. justify the invasion of a country X because X successfully simulated an attack on the U.S? Or maybe they just had the computing power to simulate it.

To the UN: We'd like you to look at these satellite images that clearly show a super computer simulating the destruction of the U.S. We have to take out these terrorists and we're willing to go it alone.

Afterward: Well it turns out that they didn't have the computing power at all, the images we had were of a mobile home park.

Wow indeed

[mnmn]

Thousands of atoms. Shrodingers/Bohrs equations for all of them.

This has interesting consequences for the study of plastics, DNA, virii and other complex molecules.

Perhaps the program can run in a loop trying every possible atomic combination to produce the best of certain attributes, as in give me the hardest material or give me an easy to manufacture room temp superconductor. It bypasses the whole invention/discovery step.