How a Frozen Neutrino Observatory Grapples With Staggering Amounts of Data (vice.com)

citadrianne writes: Deep beneath the Antarctic ice sheet, sensors buried in a billion tons of ice—a cubic kilometer of frozen H2O—are searching for neutrinos. "We collect...one neutrino from the atmosphere every ~10 minutes that we sort of care about, and one neutrino per month that comes from an astrophysical sources that we care about a very great deal," researcher Nathan Whitehorn said. "Each particle interaction takes about 4 microseconds, so we have to sift through data to find the 50 microseconds a year of data we actually care about." Computing facilities manager Gonzalo Merino added, "If the filtered data from the Pole amounts to ~36TB/year, the processed data amounts to near 100TB/year." Because IceCube can't see satellites in geosynchronous orbit from the pole, internet coverage only lasts for six hours a day, Whitehorn explained. The raw data is stored on tape at the pole, and a 400-core cluster makes a first pass at the data to cut it down to around 100GB/day. A 4000-CPU dedicated local cluster crunches the numbers. Their storage system has to handle typical loads of "1-5GB/sec of sustained transfer levels, with thousands of connections in parallel," Merino explained.

Comments

Perhaps they could buy a station wagon and

[xxxJonBoyxxx]

Perhaps they could buy a station wagon, load it up with tapes and send it with the next dogsled. (I kid.)

It's not like they are using real-time data from this thing - it's more like a traditional particle smashing experiment where most of the analysis is done months and years after the data is collected.