Slashdot Mirror

← Back to Stories (view on slashdot.org)

CERN Releases 300TB of Large Hadron Collider Data Into Open Access (techcrunch.com)

Posted by BeauHD on from the free-for-all dept.

An anonymous reader writes: The European Organization for Nuclear Research, known as CERN, has released 300 terabytes of collider data to the public. "Once we've exhausted our exploration of the data, we see no reason not to make them available publicly," said Kati Lassila-Perini, a physicist who works on the Compact Muon Solenoid detector. "The benefits are numerous, from inspiring high school students to the training of the particle physicists of tomorrow. And personally, as CMS's data preservation coordinator, this is a crucial part of ensuring the long-term availability of our research data," she said in a news release accompanying the data. Much of the data is from 2011, and much of it is from protons colliding at 7 TeV (teraelectronvolts). The 300 terabytes of data includes both raw data from the detectors and "derived" datasets. CERN is providing tools to work with the data which is handy.

38 of 60 comments (clear)

  1. Pseudoscientists of the world, unite! by Lisandro · · Score: 5, Insightful

    I just can visualize a horde of crackpots using this data to fuel fringe theories, find messages from God and prove the existence of aliens.

    That being said, this is awfully cool from CERN. The raw data will be really useful in academic environments, and the Linux visualization tools are great.

    1. Re:Pseudoscientists of the world, unite! by Anonymous Coward · · Score: 2, Funny

      I just can visualize a horde of crackpots using this data to fuel fringe theories

      I heard from good authority that the LHC breached a planar dimension, and one of its red/white striped inhabitants escaped into the LHC data stream.
      So now they're releasing the data into the public in the hopes that someone will find this wimpy alien lifeform data object (waldo)...

    2. Re:Pseudoscientists of the world, unite! by TheRealHocusLocus · · Score: 1

      "Once we've exhausted our exploration of the data, we see no reason not to make them available publicly,"

      Actually they're just sifting and patching out the winning lottery numbers first. In these ~300TB dregs you'd be lucky to find a Pick 3. Best suggestion is to make a list of numbers absent from the data and play those.

      There's also a lot of Quantum Space Spam in it, such as embedded 3D jpgs meant to be projected into 4D space showing reproductive attachments for higher dimensional beings.

      --
      <blink>down the rabbit hole</blink>
    3. Re:Pseudoscientists of the world, unite! by starless · · Score: 3, Interesting

      Data from most NASA astronomy satellites is available after a specified amount of time.
      e.g. Hubble Space Telescope data are available after one year, and Fermi gamma-ray space telescope data are available as soon as it's processed (within one day).
      Software tools are also publicly available along with software support.

      Nice to see particle physicists catching up with astronomers on data release!

    4. Re: Pseudoscientists of the world, unite! by Anonymous Coward · · Score: 1

      CERN has been releasing a lot of data since the 90s. The hard part is not deciding to release it and just copy paste the data to some webserver, but the process of documenting it and making sure there are tools to work with the data. Both the particle physics and astronomy fields have put a lot of time and money into developing these tools for decades. It is now at the point other smaller, data heavy projects can take advantage of the same the same tools with a lot less investment of manpower. But it still often requires a dedicated person on smaller projects, who may not have that to spare.

    5. Re:Pseudoscientists of the world, unite! by Altrag · · Score: 1

      horde of crackpots using this data to fuel fringe theories

      To be fair, most crackpots would manage to fuel their fringe theories just as well without this data.

    6. Re:Pseudoscientists of the world, unite! by RockDoctor · · Score: 1
      But this saves the from the delays of writing a random-number generator to invent their supporting data with.

      I'm still expecting to see the crackpots doing the logical equivalent of reading from a book held upside down.

      --
      Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
  2. No reason not to make them available publicly ? by x0ra · · Score: 5, Insightful

    If I'm not mistaken, the LHC has been publicly funded, so these data should have been public to start with. Anything else is bs.

    1. Re:No reason not to make them available publicly ? by religionofpeas · · Score: 2

      It wasn't publicly funded by the entire world, though, so it makes sense to restrict the data sharing to the scientists of the countries that helped funding it.

    2. Re:No reason not to make them available publicly ? by BitterOak · · Score: 4, Insightful

      If I'm not mistaken, the LHC has been publicly funded, so these data should have been public to start with. Anything else is bs.

      It's standard practice in experimental particle physics to give those who put the time and effort into designing, building, and running the experiment the first chance to analyze the data and publish results. After that, it's not unusual to release the raw data publicly. Otherwise, there'd really be no incentive to do the work, since someone else could swoop in and publish results without having contributed to producing the data.

      --
      If I can be modded down for being a troll, can I be modded up for being an orc, or a balrog?
    3. Re:No reason not to make them available publicly ? by 110010001000 · · Score: 2

      You mean the taxpayers of the countries funding it. After all, not only scientists were paying for it out of their taxes.

    4. Re:No reason not to make them available publicly ? by joe_frisch · · Score: 1

      Unfortunately its not simple. Scientists and the organization that they work for are judged based on their publications. So a lab that spent a lot of money to build a new experiment need to show a lot of publications from that experiment or they won't get future funding.

      Its not a great system, but it is what is in place and its not obvious how to do better.

  3. 300TB by symes · · Score: 1

    I understand there are tools to work with these data, but even so, 300TB is a lot. Wouldn't it be better, assuming they want to encourage future generations of particle physicists, to open source the tools and provide better instruction on how one should manage these data? That seems like half the problem. No way will anyone in high school download 300TBs to play with. Even if they could, what would they use to play with it?

    1. Re:300TB by religionofpeas · · Score: 1

      I assume you don't need all the data if you just wanted it for education purposes.

    2. Re: 300TB by prefec2 · · Score: 1

      Nobody will use it in high school where people have problems with calculus. It might be helpful in college and university.

    3. Re:300TB by NotInHere · · Score: 1

      Yeah certainly, building the largest particle collider in the world is way more easier than copying 300 TB of data. And it will be way more fun, too!

    4. Re:300TB by Jumunquo · · Score: 1

      You manage it on a thumb drive, DUH! Make sure it's USB 3.0! Load the data into Excel, and you can make pretty graphs.

    5. Re:300TB by hackertourist · · Score: 2

      Congratulations on not following TFLinks. They did open-source the tools and provide instructions.
      You also don't need to download the entire 300 TB, the data is divided into batches.

      Available on the CERN Open Data Portal - which is built in collaboration with members of CERN's IT Department and Scientific Information Service - the collision data are released into the public domain under the CC0 waiver and come in types: The so-called 'primary datasets' are in the same format used by the CMS Collaboration to perform research. The 'derived datasets' on the other hand require a lot less computing power and can be readily analysed by university or high-school students, and CMS has provided a limited number of datasets in this format.

      Notably, CMS is also providing the simulated data generated with the same software version that should be used to analyse the primary datasets. Simulations play a crucial role in particle-physics research and CMS is also making available the protocols for generating the simulations that are provided. The data release is accompanied by analysis tools and code examples tailored to the datasets. A virtual-machine image based on CernVM, which comes preloaded with the software environment needed to analyse the CMS data, can also be downloaded from the portal.

  4. Library of Congresses? by Macdude · · Score: 1

    300 TB?
    How many Libraries of Congress is that?

    --
    "Grab them by the pussy" -- President of the United States of America
    1. Re:Library of Congresses? by Megol · · Score: 2

      US or metric LoCs?

  5. Re:Download cap? by dsmatthews9379 · · Score: 1

    Yeah you really need to upgrade from Telex to something a bit more modern.

  6. Re: No reason not to make them available publicly by prefec2 · · Score: 2

    It was available to all scientists of the funding and visiting countries. Now as the scientists are through with it you can have a look too.

  7. Re: No reason not to make them available publicly by x0ra · · Score: 1

    It should have been available to the whole population...

  8. Re:Download cap? by NotInHere · · Score: 2

    By the time you have downloaded the 300 TB, they'll have built another, bigger, particle collider, and released an even bigger tarball about that one.

  9. El Psy Congaroo by SeaFox · · Score: 1

    Maybe now, we can unlock the mysteries of Steins Gate! Mwahaha!

  10. Re: No reason not to make them available publicly by prefec2 · · Score: 1

    It is now. Before that the people who developed the experiments got first access. I personally understand that perfectly. They invested decades of their lives.

  11. Re:More useful delivery by armanox · · Score: 1

    Its not like we have 300TB SANs in our homes or schools.

    Not yet, anyway, but in the next five to ten years that might not be a problem any longer. Plus I would hope the data is in a more manageable form then just one giant tarball (is there any file system that allows for an individual file that big anyway?)

    --
    I'm starting to think GNU is the problem with "GNU/Linux" these days.
  12. After reading through 300TB, you'll be at... by Jumunquo · · Score: 1

    human is dead, mismatch.

  13. Re: No reason not to make them available publicly by AmiMoJo · · Score: 1

    Cool. Where's the torrent? It's not in TPB yet.

    --
    const int one = 65536; (Silvermoon, Texture.cs)
    SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
  14. Re:More useful delivery by Anonymous Coward · · Score: 1

    The place I work for has a 374TB SAN that's EoL. We got some quotes for re-sale and best offer we were given was £3000.

    So a bit of a chunk of cash but not crazy.

  15. Raises the bar by SkyratesPlayer · · Score: 3, Funny

    Before this, the largest collection of collision data was the Russian dash-cam footage on YouTube

  16. 300 TB. How many floppies? by edxwelch · · Score: 1

    Just curious how many floppy disks would it take to store 300 TB?

    1. Re:300 TB. How many floppies? by SkyratesPlayer · · Score: 1

      Let's assume 3.5" form factor. Using HD floppies, you need 20 million. The one on my desk is 3.25 mm thick, so they'd make a stack 65 km tall. In the spirit of Randall Munroe's What If, it would of course collapse and kill you long before it got that high.

    2. Re:300 TB. How many floppies? by HiThere · · Score: 1

      Why not use the 8 inch hard sectored floppy disks? Of course there's nowhere you could either read or write them... but they were a bit thinner, and I think they stored all of 100KB.

      --

      I think we've pushed this "anyone can grow up to be president" thing too far.
    3. Re:300 TB. How many floppies? by SkyratesPlayer · · Score: 1

      Why not use the 8 inch hard sectored floppy disks? Of course there's nowhere you could either read or write them... but they were a bit thinner, and I think they stored all of 100KB.

      Because I am willing to bet there were more 3.5" floppies made than all other types of removable media put together. I have a DEC RX-02 somewhere, but haven't had to use it since 1998... ISTR they held about 500kB.

  17. There sure are a lot... by flargleblarg · · Score: 1

    Sure are a lot of articles about the Large Hardon Collider lately.

  18. Download Link by Anonymous Coward · · Score: 1

    http://opendata.cern.ch/about/CMS

  19. Re: No reason not to make them available publicly by Baloroth · · Score: 2

    Why? What interest does the general population have in access to the LHC data? They've already release a subset of the data for educational purposes, in addition to this considerable data dump. It serves no public interest to make the whole data set available to everyone, and in fact would run contrary to the public interest: the data set is absolutely massive (the LHC produces petabytes of data per day), and the costs associated with making that data available to the public would be non-negligible.

    If a specific individual is interested in access to the data, they're certainly free to email their local (or not even necessarily local) university department associated with the LHC and ask for it, and they could probably get access to a subset of it, if they've shown genuine interest. And by "genuine interest", I mean have already downloaded, processed, examined, and understand much of the already publicly available data, to the point where they are capable of performing actual scientific research on the data, and aren't simply interested in wasting already-precious scientific research money and time in making some kind of political or philosophical point.

    --
    "None can love freedom heartily, but good men; the rest love not freedom, but license." --John Milton