A Look At CERN's LHC Grid-Computing Architecture

blair1q writes "Using a four-tiered architecture (from CERN's central computer at Tier 0 to individual scientists' desk/lap/palmtops at Tier 3), CERN is distributing LHC data and computations across resources worldwide to achieve aggregate computational power unprecedented in high-energy physics research. As an example, 'researchers can sit at their laptops, write small programs or macros, submit the programs through the AliEn system, find the necessary ALICE data on AliEn servers, then run their jobs' on upper-tier systems. The full grid comprises small computers, supercomputers, computer clusters, and mass-storage data centers. This system allows 1,000 researchers at 130 organizations in 34 countries to crunch the data, which are disgorged at a rate of 1.25 GB per second from the LHC's detectors."

Comments

truly amazing

[Goldsmith]

I was having lunch with some CERN guys a couple weeks ago, and was asking them about the speed of their analogue to digital converters. I don't remember what the number was, but it seemed low to me, something like 200kHz. So, of course, I had to point out that *my* cheapo converters ran faster than theirs by more than an order of magnitude. They responded with "well, each of our converters does 200kHz on all of our 4000 channels at the same time, so we're really recording at..."

They won.

Data to crunch

[Lord Byron II]

As someone who worked on the processing of HEP experimental data for awhile, let me say that there is a ton of work to do. You have particles entering the detector every ~40ns and hundreds of different instruments making measurements, which leads to a ton of data very quickly. You then have to reconstruct the path of the particle based off of the detector information, but it's not straight-forward. The detector can have gaps in coverage; neutrinos (which are undetectable) can be created removing momentum; particles from the previous event can still be in the detector et cetera.

And all of the data crunching you do must be done in 40ns, so that you're ready for the next set. (Of course, you can do some processing offline, but if you don't maintain a 40ns average, then your data will start piling up.)

Re:Data to crunch

[klazek]

You have particles entering the detector every ~40ns and hundreds of different instruments making measurements, which leads to a ton of data very quickly.

Not exactly true. It's running at 40 MHz, so that's 25 ns bunch spacing. Further, you don't exactly have to 'crunch' the data as it comes in, there are multiple triggers that throw lots of data away based momentum cuts and other criteria before it ever makes out of the detectors.

In ATLAS, for example, there are ~ 10e+9 interactions/sec. The Level1 Trigger, consists of fast, custom electronics programmed in terms of adjustable parameters to control filtering algorithms. Input is from summing electronics in the EM and hadron calorimiters, and signals from the fast muon trigger chambers. The info is rather coarse at this point (transverse momentum cuts, narrow jet criteria, etc), and at level one the info rate is decreased in about ~2us (including communication time), from 40MHz to about 75KHz. Level2 now does a closer look, taking more time and focusing on specific regions of interest (RoIs). This process takes about 10ms, and data rate is reduced to about 1KHz for sending to the event filter. Here, the full granularity of the detector (the 'detector means all the bits - Inner detectors: Pixels, strips, Transition Radiation tracker - The calorimiters - The muon tubes at the outside radius) and runs whatever selections algorithms are in use. This takes a few seconds, and output is reduced to about 100Hz and written to disc for a gazillion grad students (like myself) to analyze endlessly and get our PhDs.

There is much more to it of course, but you can find info about it on line if you really are interested in the details. Have a look at the ATLAS Technical Design Report: http://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/TDR/TDR.html

Re:1.25GB/sec not that much.

[Alef]

A single 10gb ethernet connection can handle that quite easily.

Eh. A 10 Gb ethernet connection can't handle 1.25 GB/s at all, not to mention doing it reliably. Theoretically, 10 Gb is exactly 1.25 GB, but then you need to account for protocol overhead, packet loss and so on.

Re:I want more!

[Arathrael]

Have a look here: http://lcg.web.cern.ch/LCG/image.htm for Google Earth based dashboards showing WLCG live grid sites, links, data transfer and job activity.

Re:lhc infrastructure history?

[TheRaven64]

I mean: who could have guessed the processorspeed and diskspace we have now.

Gordon Moore?