Slashdot Mirror
Storing CERN's Search for God (Particles)
Chris Lindquist writes "Think your storage headaches are big? When it goes live in 2008, CERN's ALICE experiment will use 500 optical fiber links to feed particle collision data to hundreds of PCs at a rate of 1GB/second, every second, for a month. 'During this one month, we need a huge disk buffer,' says Pierre Vande Vyvre, CERN's project leader for data acquisition. One might call that an understatement. CIO.com's story has more details about the project and the SAN tasked with catching the flood of data."
Wow! Actually geeky science news, not enough of that here lately!
If only I could get porn that fast
there I said it, let's move on now.
I'm god, but it's a bit of a drag really...
Um...no. Actually, it's a product placement PR piece about Quantum's StorNext. (Read page 2...)
Actually their plan is to store all that data on Commodore 64 cassette tapes.
A standared dual CPU dual core HP server with Windows can keep a 4Gb FC pretty full if set up correctly. I work for a large bank, and we have many a Solaris box that can keep 4 or even 8 2Gb FC cards full into our FC and SATA disk arrays. Not to trivialize the extreme coolness of what they are doing at all, but a PB of data with a few PB of I/O in a day isn't what it used to be. I'm just glad to see they don't use Polyserve, it is worthless for clustering and has caused more downtime at work than it has ever prevented. If they really have that much data they should use 10Gb FC or Infiband. Even our stodgy old bank is implementing our first infiband system so we can move IO at 12Gb instead of the slow 4Gb links.
based on 1GB/sec * ((3600 * 24) * 31) means over 2.5 Petabytes.
Wow.
Something like 3000 of the current ITB drives.
How long until Exabyte level storage is required for some project or another?
Trying to associate Microsoft with "fun" is like trying to associate Satan with aromatherapy. -Tycho
2.6 Petabytes. The article says that they will be collecting petabytes of data. Also, the article clearly said GB. GB= Gigabyte Gb= Gigabit. The thing that I thought was "Wow that's ALOT of blinking lights!" Sweet!
Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
Actually, there really is a gigantic room at CERN full of commodity PCs that form the first level of computing for the different experiments. The data is then shipped off to sites around the world for further processing. There is a combination of 'locally' distributed computing and world-wide grid being used.
load"*",8,10 0 on the tapes?
Ok, who put California Games x 1000000000000000000000000000000000000000000000000
Nah it's just, spooky article submission at a distance.
The other article appeared because it knew this one would be submitted later in the future.
"Due for operation in May 2008, the LHC is a 27-kilometer-long device designed to accelerate subatomic particles to ridiculous speeds, smash them into each other and then record the results."
Next up ludicrous speed!!! Better fasten your seat belts...
Shakespeare poems - infinite monkeys with infinite time.Computer tech support - a few trained ones working from 9 to 5.
Hmm, lets see. ~2700 TB of data over one month. Let's store it on 500 GB drives. That's 5400 disk drives just to store the data. Add in the the extra drives for parity, and a few hundred hot spares, this thing could easily use OVER NINE THOUSAND drives.
With a God Particle generator, wouldn't you *generate* God? Wouldn't that be a hoot?!?
The network is one thing, but just processing that amount of data is incredible.
200 computer breaks the 1GB chink into more manageable 5MB/Sec chinks of data, but then they still need to handle the metadata that figures out how to put it all back together. On top of this they'll need to have some redundancy in case of data loss, and how the load is redistributed if a machine croaks.
These are good problems, it would be a fun system to work on.
Not only did the Slashdot editor not catch a spelling mistake, he apparently didn't catch the fact that the linked article is an advertisement from CXO Media, which, according to its web site, mixes articles and advertisements: "Through our integrated media and marketing programs we provide..."
From the linked article: "... the team is using Quantum's StorNext software as its file system..."
Question: Did a Slashdot editor get paid directly for running an advertisement disguised as an article? Or was someone in Slashdot's parent company paid "under the table"? Or did the parent company get paid?
Anyone wanting to read a real article from 2005 about CERN's data handling, data storage, and data processing can download this PDF file: Grid Computing: The European Data Grid Project.
Real articles begin this way: "The computing challenges for LHC are: * the massive computational capacity required for analysis of the data and * the volume of data to be processed."
Advertisements begin by talking about God and murder, this way (from the article linked by Slashdot): "CERN's Search for God (Particles)..."
and "Maybe you last read about CERN (the European Organization for Nuclear Research) and its massive particle accelerators in Angels & Demons by Dan Brown of The Da Vinci Code fame. In that book, the lead character travels to the cavernous research institute on the border of France and Switzerland to help investigate a murder."
The physicists don't really want to find god, it's just the only way they can get research funding under the bush administration.
I think the correct word, considering the meaning, is "caching".
No, I believe the word was catching. As in:
They're throwing all this data at me and I gotta catch it.
Your hair look like poop, Bob! - Wanker.
No. No, my friend; you do not grasp the scale of this project.
load"*",8,1 would load something from a diskette, not a cassette.
It's only 5x HD SDI single channel ~ 200MB/s. Any major studio could handle this with ease.
SDI is how the movie guys move their digital stuff around. A higher end digital camera will capture at 2x HD SDI for a 2K res, 4:4:4 colour space. A few of em' and you got your 1GB/s easy. Spools onto godlike RAID arrays.
Get em' to call up Warner Bros if they have problems.
Assuming a non-RAID 3x-replication tech solution (what Google do in their datacenters), using 500-GB disks (best $/GB ratio), they would need about 16 thousands disks:
Which would cost about $1.8M (disks alone):
15552 (disk) * 110 ($/disk) = $1710720
Packed in high-density chassis (48 disks in 4U, or 12 disks per rack unit), they could store this amount of data in about 30 racks:
15552 (disk) / 12 (disk/rack unit) / 42 (rack unit/rack) = 30.9 racks
Now for various reasons (vendors influence, inexperienced consultants, my experience in the IT world in general, etc), I have a feeling they are going to end up with a solution unnecessarily complex, much more expensive, and hard to maintain and expand... Damn, I would love to be this project leader !
"2,629,743 seconds in a month, so... 2,629,743 GB or 328,717 GB?"
If they were smart, they'd choose February. They could save ~172800 seconds and therefore some disk space!
Just e-mail it all to Google. By then gMail should be able to handle that much per user.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
Right now, the average event size for ATLAS is 1.6 MByte and the system is designed to keep around 200 events per second, or roughly 300 MByte. This isn't much of course, but you have to consider that the bunch crossing rate (i.e. the rate at which bunches of protons will collide and generate events) is 40 MHz.
So you have to design a system that boils this rate from 40 MHz down to 200 Hz and only keeps the interesting parts, while also buffering all the data in the meantime. For this reason, the first trigger level is entirely implemented in hardware right in the detector and reduces the rate down to 75 KHz with a latency of 2.5 s. The rest of the trigger works on clusters using Linux computers and has a latency of o(1s).
...all this data will be distributed to a handfull of TIER1 sites (CERN is TIER0) all over the world (about 10). At the TIER1 sites the data will be preprocessed. The TIER1 sites distribute their preprocessed data to TIER2 sites which are the places where the international scientists work. I work at a TIER1 site and we face a lot technical challenges with this project. At a TIER1 site as I mentioned, the data is preprocessed too, so we will need a compute cluster and the necesary bandwith internally to move the data around. With each new software release (about every six months), ALL raw data has to be reprocessed with the new software. All results have to be stored. So for every part of raw data we will have to store preprocessed data for every software release. Of course a lot of data will be stored on tape but we expect that the dataflow from CERN (for us 150MB/s to disk and 75 MB/s to tape) will be the least of our problems. Moving the data around and preprocessig the data is probably a bigger problem in the long run. An the fact that the machine will be running for about 15 years or so, this will be a very long run!
TFS makes a point about storing 1 GB (presumably GigaBYTE) of data per second, but THAT feat is already in widespread use, spefically for the digital manipulation of 4k film. The company that produces the systems that process this film data is called Baselight.
:P
Basically, 4k film, at a resolution of 4096x3112, requires approximately 50MB per frame @ 24 fps. That comes out to about 1.22GBps, and maninuplating the data doubles it to 2.44GBps. The systems[PDF] that Baselight sells run 8 nodes and 16 processors, and it's all built with commodity hardware and some flavor of Linux. Apparently they use 3ware RAID cards... and I found out about this by browsing 3ware's site when I was shopping for a RAID controller.
Either way, my point is, it's been done, and there's a real world application that requires that type of data storage bandwidth and has nothing to do with scientific data.
Boot Windows, Linux, and ESX over the network for free.
The ALICE experiment is actually concentrating on heavy ion collisions which is why they only worry mainly about one month/year, the rest of the time the machine is running protons for the other experiments, ATLAS and CMS, which will look for the Higgs. ALICE will hopefully study the quark gluon plasma but, as far as I know, has no plans to look for the Higgs.
According to a guy that I met yesterday on the street (he was talking to himself or somebody) the only way I could meet God (and hopefully His particles) was through his son. WTF? Can't even *God* get a good secretary these days?
Don't worry -- the products of particle accelerators only exist for a few picoseconds. If God is created during a collision event, he will wink out of existence so fast that we'll only become aware of his presence by the shower of Mormonions and PatRobertsonite particles impinging on the detection apparatus.
OK, we got a half way overview of CERN's decision, with some bold statements of questionable validity. I am submitting the criticism purely on the grounds of being really interested in large data storage, I don't work for any large storage vendor, but I am an architect of storage systems.
First of all, with the statement "and it's (StorNext) completely vendor independent": Lot's of other solutions provide flexibility about choosing the hardware vendor from a theoretical perspective. The theory says that if vendor A makes a SAN, vendor B makes a RAID controller, C a disk cabinet and D offers a clustered FS, and all comply to the relevant standards, you can plug them together and expect them to function. However, imperfections in the standards, hidden proprietary optimizations, always dictate certain configs and combinations for optimum performance. There is a lot of work to be done in the StorNext and other similar products, until they claim full flexibility. My experience in deploying a StorNext based solution on a 1200 node setup says so and to keep the post short, I shall exclude at this stage vendor details, but if someone is interested, I am happy to go over the details. There is vendor dependence if you wish optimum performance. Not to mention that if you mix and match the RAID and SAN cards in the setup, any unfortunate issue might end up in a multi-headache, even if you have solution support (A blaims B, B accusses A, and the game of ping-pong begins). You can never exclude vendor dependence in such a large setup, you have to deal with it.
Then you have the "Clustered file systems are still an evolving category, she says, but enterprise IT is warming up to it.". I can imagine what the author classes as enterprise IT here, but I think there is a bit of an orientation issue. CERN is not exactly the classical enterprise IT environment, is it? Not in terms of their requirements for resilience and capacity. These FAR EXCEED enteprise IT requirements. CERN is a research setup. And the mentality of a research setup (that incubated the WWW after all) is (or should be) that of innovation and playing with some of the latest and the greatest. In fact, some US based research setups have long experimented with other cluster FSes. They are not warming up. CIO claims that StorNext is scalable. It is. But to what extent? Have they excluded for example things such as Lustre? http://wiki.lustre.org/index.php?title=Main_Page If yes, why?
- Printed hardcopy. Many authorities recommend this as you do not need to worry about changes in data formats over time. For exact calculation, we would need to know the font they were planning to use and the character encoding. However, let's take a working assumption that they can cram 10KB of data onto an A4 sheet. That implies 259,200,000,000,000 pages. They will probably not want to use an inkjet printer if they use this solution and may, indeed, choose to acquire multiple printers and split the load. A single printer at 10 ppm would take approximately 50,000 years to complete the backup. On 70gm paper, it would weigh a little over two million tons. At any rate, this would certainly produce reams of output.
- Diskettes. This was good enough for nearly everyone 15 years ago. It is curious that such a tried and trusted technique is no longer in fashion. I assume regular 3.5" 1.44MB diskettes, generally recognised as easier to handle than 5.25". We shall need around 1,800,000,000 diskettes. One drawback is the person changing the diskettes as each one filled up might become a little bored after a while. On the positive side, the backup will be quite a lot faster than the printed solution. Assuming about one diskette per minute, inclusive of changing disks, the backup could be complete in less than 3,500 years.
- Now considered somewhat old fashioned, punch cards were once a mainstay of every programmer's personal backups. Like printed hardcopy, anyone familiar with the character encoding used, could read the data without needing any access to a computer. If we assume 80 column cards, we would need 32,400,000,000,000 cards. I would be somewhat concerned about the problem of getting this stack of cards back in the correct order if I dropped it. With a weight of about 30 million tons and stretching perhaps 6 million miles end to end, handling certainly would be challenging and an accident very possible.
- Paper (punched) tape was the only alternative on the first computer I used, a basic early model Elliott 803 without the optional magnetic tape. If I recall correctly, you could manage about 10 characters per inch, so you would need a paper tape over 4,000,000,000 miles long. Hmmm, that would be silly. The other solutions are clearly better.
I am sure other options will be considered, but I just wanted to bring these up in case CERN had failed to consider themImagine how deep the personality problems must run in a person who gets all hot because of someone's DNA sequences!
"Imagine how deep the personality problems must run in a person who gets all hot because of someone's DNA sequences!"
You must be new here.
I think you're thinking of that guy who got nailed to the cross (Jesus). Noah was born about 5,000 years ago.
Ben Hocking
Need a professional organizer?