World's Largest Working Computing Grid

fenimor writes "UK particle physicists claim that they will demonstrate the world's largest, working computing Grid with over 6,000 computers at 78 sites internationally. The Large Hadron Collider Computing Grid is built to deal with 15 Petabytes of data each year from the Large Hadron Collider (LHC), currently under construction at CERN in Geneva. 'This is a great achievement for particle physics and for e-Science,' says Professor Tony Doyle, leader of GridPP. 'Our next aim is to scale up the computing power available by a factor of ten'."

Comments

imagine

[denthijs]

the optimization flags on one of those,....

Re:imagine

[Anonymous Coward]

[80s]So funny I forgot to laugh[/80s]

People are mostly using "Scientific Linux", an in-CERN-house Redhat fork, but some sites are experimenting with other stuff - one of the computer science aspects of the grid is researching how to make good use of heterogenous systems, though different linux distros aren't amazingly heterogenous in the grand scheme of things, there are challenges.

And yes, there are people working on Gentoo, believe it or not.

And Debian and fedora core 2 and 3 and mandrake clic and suse.

And IRIX, AIX, Solaris and Mac OS X from the Real Unix camp.

And there's some poor fuckers somewhere working on windows too.

At last

[modest apricot]

Finally, something to run doom3 on. Though I may still have to turn shadows off...

Yeah, physics!

[-ing AnonymousCoward]

But let's talk about something serious: how many FPS in Doom III?



...



Mmmmm... That might be worth the upgrade then...

able to handle 15 petabytes a year?

[djfray]

finally something to deal with those pesky environmentalists.... :-P

Computing power

[nemexi]

Does anybody know facts about the computing power of the grid? How many teraflops will it be able to achieve?

Re:Computing power

[kristofme]

In Grid computing, which is not exactly the same as high-performance computing, the number of flops doesn't really matter that much, it's more about providing an environment for multiple users to address problems that can be solved by splitting it up in a huge number of smaller tasks.

Re:Computing power

[cmacb]

What I'd like to know is: who has the largest non-working computer grid. My guess is that it would be a lot bigger.

Grid vs. LHC@Home?

[Anonymous Coward]

What's the point of the Grid thingy if they've also setup this?

http://lhcathome.cern.ch/

Re:Grid vs. LHC@Home?

[David McBride]

The LCG resources have several different things that most home machines do not:

1) A Linux install with the requisite libraries for the already-written experiment analysis programs to run on.
2) Fast network interconnects, both to other LCG cluster nodes at the same site (using Myrinet, Infiniband, etc.) and large network connections to other participating sites (ie 100Mbit+).
3) Large amounts of reliable local storage, ie 1TB+.

SETI@Home-like distributed computing problems only work well for problems which do not require large amounts of communication between nodes before, during, and after an individual run. Many problems do not fall into this category.

Re:Grid vs. LHC@Home?

[Fallen Andy]

lhc at home is not for processing the data output, but helping them to position the magnets as they
*build* the LHC.

Analogy for the ADDers

[rokzy]

imagine a beowulf cluster of Half-Life 2 preloads! ;-)

PET-Animals jokes....

[djfray]

physicists enable torrent animals pedestrians engulf tree-nesting animals Polka Ensues from Trouncing Animals Police Escape from Tricky Animals Penguins for the Energetic Tazing of Animals

Images

[Limburgher]

I found a picture of the system here. You may have to zoom in a bit to see individual machines.

Re:Images

[rokzy]

your joke being funny not withstanding, that's a map of America, probably the least relevant place to show for this particular project.

CERN and Grid is European, notably Switzerland, France and UK.

the USA has plenty of great particle physics of its own (excitable New Yorkers beware - there's a particle accelerator on your doorstep - think of the children!) but this is not one of them.

Coordination

[erick99]

I wonder how they are coordinating the use of all of those computers? The article doesn't say that they will be exclusively for this project and, if they are not, then that is some task to have them all online and not otherwise busy. They must have some damned serious storage vaults as well if they are generating 15 Petabytes a year of data, which doesn't include the output from processing. Still, it must be something to have all of the "horsepower" at your command.

Cheers,

Erick

Re:Coordination

[MrNixon]

Because the Earth is a LOT brighter than the stars (because the stars are far away), and to properly expose the Earth onto whatever media is being used(film, CCD, whatever), less exposure is needed than would be necissary to pick up any stars (save the sun).

Just like pictures from the moon - you'll not see any stars in pictures taken of the moon on the moon (by Neil Armstron et al).

Hope that helps

Re:Coordination

[steve_l]

I have access to some of the machines; we donate idle systems to the project in exchange for low cost (read free) access to the superjanet network. When they arent doing UK NeSC grid stuff I can bring up vmware images of whatever distro I feel like, run whatever stuff we need -in my case usually distributed testing of distributed software.

That is how the grid works -it uses spare cycles on machines in the network. Unlike Seti@home, they are very fussy about bandwidth; you need a serious link to play. Most of the tier-1 sites are UK academic sites -rutherford labs, oxford, the london universities, etc. Tier-2 sites are us industrial sites with machines we let them play on.

Like you say, data storage is a big issue, and something the grid needs to work on. A lot of grid forum work is on data.

Incidentally, when the LHC comes on line, then the serious data starts to collect. Any collisions -events, they call them- generate vast amounts of data. There is some logic in the system to immediately split dull events from potentially interesting ones, but those interesting ones happen often enough you need to buffer it all up, then do more rigorous analysis to see if it is something new or something known about. The uk grid will be used for analysis of stuff the front end thinks could be interesting. Right now it can test the data generated by simulations (people can get phds writing good event simulations), so they can verify that the analysis code works.

One of the great fear in high energy physics is that the filter and detection logic is buggy and that nobel-grade events are missed because the data is simply too bizarre for the analysis code, and so gets missed out,..

Physics

[penguinoid]

Getting the physics right has been an important part of many of our favorite 3D games lately...

Largest?

[anethema]

I am not sure how they define largest...

Are these 6000 super computers? Or just other computers?

Distributed.net had around 330 thousand participants on the latest completed rc5 key.They had 15 thousand active on the last day of the challenge.

I would say this is much larger in computer numbers, but since they dont mention almost any usefull information in the article, I'm not sure if more computer power would be in the d.net.

However the line: By 2007, this Grid will have the equivalent of 100,000 of today's fastest computers working together to produce a 'virtual supercomputer', which can be expanded and developed as needed

So right now it isnt even 100 thousand computers, maybe not even close, so the computing power might be similar. (assuming 15 thousand active computers on d.net)

Either way, right now i highly doubt its the largest ;)

Re:Largest?

[Anonymous Coward]

So right now it isnt even 100 thousand computers, maybe not even close, so the computing power might be similar. (assuming 15 thousand active computers on d.net)


The point is not so much assembling all that computing power now (the LHC won't come online till 2007 or so anyway, and you don't really need the Grid to run Monte Carlo) so much as assembling the infrastructure so that when 100 universities go out and buy new analysis farms in 2007, they can get tied together and used efficiently.

10?

[real_smiff]

why a factor of 10? why not take it to.. eleven.

for when your particle collider needs that little push over the cliff..

Copycat writeup

[David McBride]

That writeup looks a lot like the one at The Register -- which came out a good two days early, the same day the results were actually announced at the AHM conference.

Large Hardon Collider?!

[Hitmen]

Erm, I think I read that wrong.

But...

[rune2]

Can it run Longhorn?

Seamlessly? I doubt it. Latency is a big problem

[Louis Savain]

Grid computing has been a target for IT developers and scientists for more than five years. It allows scientists to access computer power and data from around the world seamlessly, without needing to know where the computers are.

The key word here is "seamlessly." The problem with a world grid is the latency introduced by communication between nodes. If a computation is dependent on results from another computation happening half way around the world, I cannot see how a world grid can compete against a linux cluster. Besides, unless there is provision for redundancy (sorry, I did not read the entire article), a critical node may be down due to a power outage or something as mundane as the cleaning people turning off the computer. This would bring everything to a halt.

One tiny problem ...

[Anonymous Coward]

One of my buddies was an early numerical modeller. If I learned one thing from him it was that all the computer power in the world was no use if your model was even slightly defective. The models tended to 'blow up'. Imagine a hundred foot wall of water moving majestically down the estuary.

Typical of stories about these giant computers, they don't really describe the problems they intend to solve. In a way, that is the more interesting story. Mind you, that story is much harder to tell if you want your audience to understand it.

Super big "grid"!?

[ylikone]

Imagine if they had a super big worldwide "grid" of computers all connected via some common protocol! It would be amazing!

worlds largest working grid???

[jimmysays]

I was under the impression that the world's largest working grid was the United Devices grid.org project. They have over 2.5 million registered users and average over 300,000 work units returned every day. check out www.grid.org They are also doing real science.

Re:Seamlessly? I doubt it. Latency is a big proble

[tkittel]

True, and this makes it difficult for people who want to calculate protein folding or predict next weeks weather. But for particle physics computations we hardly need any communication between nodes at all. Rather, we need something simulated a huge amount of times (as in, "simulate this proton-proton collision 10 billion times") or "apply this fancy pattern recognition algorithm to each of these billions of events we took this week". Particle physics computations are to a large extent parallel in nature from the beginning.

The grid related problems faced in particle physics are of another nature, such as ensuring that the data is copied around the various grid facilities as needed and of ensuring that even if a given node fails to execute its job for some reason it is rerun elsewhere automatically - that sort of thing.

One future quote:

[tod_miller]

640 petabytes of memory should be enough for anyone, ever.

Although, if 640kb sounded anything like 640 petabytes does now, I'll have to rape moores law over a barrell and say I doubt we will ever have computers with 640petabytes of ram as standard.

Of course, I say that in jest, I would love for future people to read this post and laugh thier tits off (some futuristic velcro tits no doubt)