Slashdot Mirror
Storing CERN's Search for God (Particles)
Chris Lindquist writes "Think your storage headaches are big? When it goes live in 2008, CERN's ALICE experiment will use 500 optical fiber links to feed particle collision data to hundreds of PCs at a rate of 1GB/second, every second, for a month. 'During this one month, we need a huge disk buffer,' says Pierre Vande Vyvre, CERN's project leader for data acquisition. One might call that an understatement. CIO.com's story has more details about the project and the SAN tasked with catching the flood of data."
Wow! Actually geeky science news, not enough of that here lately!
The network is one thing, but just processing that amount of data is incredible.
200 computer breaks the 1GB chink into more manageable 5MB/Sec chinks of data, but then they still need to handle the metadata that figures out how to put it all back together. On top of this they'll need to have some redundancy in case of data loss, and how the load is redistributed if a machine croaks.
These are good problems, it would be a fun system to work on.
Not only did the Slashdot editor not catch a spelling mistake, he apparently didn't catch the fact that the linked article is an advertisement from CXO Media, which, according to its web site, mixes articles and advertisements: "Through our integrated media and marketing programs we provide..."
From the linked article: "... the team is using Quantum's StorNext software as its file system..."
Question: Did a Slashdot editor get paid directly for running an advertisement disguised as an article? Or was someone in Slashdot's parent company paid "under the table"? Or did the parent company get paid?
Anyone wanting to read a real article from 2005 about CERN's data handling, data storage, and data processing can download this PDF file: Grid Computing: The European Data Grid Project.
Real articles begin this way: "The computing challenges for LHC are: * the massive computational capacity required for analysis of the data and * the volume of data to be processed."
Advertisements begin by talking about God and murder, this way (from the article linked by Slashdot): "CERN's Search for God (Particles)..."
and "Maybe you last read about CERN (the European Organization for Nuclear Research) and its massive particle accelerators in Angels & Demons by Dan Brown of The Da Vinci Code fame. In that book, the lead character travels to the cavernous research institute on the border of France and Switzerland to help investigate a murder."
I think the correct word, considering the meaning, is "caching".
No, I believe the word was catching. As in:
They're throwing all this data at me and I gotta catch it.
Your hair look like poop, Bob! - Wanker.
TFS makes a point about storing 1 GB (presumably GigaBYTE) of data per second, but THAT feat is already in widespread use, spefically for the digital manipulation of 4k film. The company that produces the systems that process this film data is called Baselight.
:P
Basically, 4k film, at a resolution of 4096x3112, requires approximately 50MB per frame @ 24 fps. That comes out to about 1.22GBps, and maninuplating the data doubles it to 2.44GBps. The systems[PDF] that Baselight sells run 8 nodes and 16 processors, and it's all built with commodity hardware and some flavor of Linux. Apparently they use 3ware RAID cards... and I found out about this by browsing 3ware's site when I was shopping for a RAID controller.
Either way, my point is, it's been done, and there's a real world application that requires that type of data storage bandwidth and has nothing to do with scientific data.
Boot Windows, Linux, and ESX over the network for free.
How much is a 500Gb drive worth nowadays? 150$? So your OVER NINE THOUSAND drives are worth about, hum....1.35M$. CERN has a budget of about 5B$. It's the speed at which data is coming that's a problem. Not the total amount of data.
You may think of it as product placement, but I use it. I even provide the occasional blog entry on it on Advanced Topics. I sat through a RedHat performance tuning class that was quite excellent. But when they came to the part about ext3 and tuning it, well, let's face it - ext3 just isn't going to scale. I started with Veritas' Filesystem which is pretty nice. If you're a small-time admin, then you never get beyond a local, 4U disk array. Once your group spends more than US$2million on servers though, it's obvious what the problem is: Storage - The Final Frontier. SAN and clustered filesystems allow a level of scalability completely unheard of before.
They also completely left out anything but a tagline of their multi-tiered solution. I wish they'd talked more about how CERN supports 500Gbit per second aggregate throughput to their disks (at least they implied that). 50GB/sec (or so) is probably the toughest I/O problem you've ever dealt with, or will deal with for a long time. Whose RAID controllers did they use? Did they focus on speed (ASIC and ISL minimization), availability (redundant fabrics), or both? Did each node get dual 4Gb links or just one?
If this had been an advertisement, they would have discussed some 3.0 features like LAN clients.
So, in short, it's easy to say it sounds like an advertisement. Quite possibly, Quantum (formerly ADIC) coerced them into getting the piece written. But if this had been an advertisement, there is so much more that is going on under the hood that would have been said. Large, fast, distributed filesystems are non-trivial and take an extreme amount of engineering and testing. StorNext really is good at what they claim to do.
If you want to read about some of the drawbacks though, I yak about them on my blog. Sorry for the plug.