disCERNing Data Analysis

technodummy writes: "Wired is reporting how CERN is driving the Linux-based, EU funded, DataGRID project. And no, they say, it's nothing like Seti@Home. The description on the site of the project is: ' The objective is to enable next generation scientific exploration which requires intensive computation and analysis of shared large-scale databases, from hundreds of TeraBytes to PetaBytes, across widely distributed scientific communities.'" If you're interested in this, check out the Fermi Lab work with LinuxNetworkX data as well as the all-powerful Google search on the Fermi Collider Linux project. As jamie points out, "Colliders produce *amazing* amounts of data in *amazingly* short time periods... on the order of "here's a gigabyte, you have 10 milliseconds to pull whatever's valuable out of it before the next gigabyte arrives".

Comments

shear quantity of data

[Alien54]

This first reaction I have is that I wonder if a distributed model of computing would even be able to make a dent if the amount of data is that big.

Does anyone have a idea on how much data Seti at home has processed? This would certainly be useful as a yard stick of sorts.

distributed computing

[sam@caveman.org]

here's a gigabyte, you have 10 milliseconds to pull whatever's valuable out of it before the next gigabyte arrives.

let's see. 1 GB in 10 ms works out to 100 GB per second. how recently did GB ethernet come about? and what would the average bandwidth of users be? i would guess much less, but let us assume 100KB per second.

so you have 107374182400 bytes of data per second. your users can take 102400 bytes per second each. even if everyone was connected directly to your network (no delays or bottlenecks... ha!) you would still require 1048576 users (that is over 1 million).

and this is not taking into effect sending any data BACK to the source or actual computation time on the users.

-sam

Re:EU funding

[pubjames]

Has anyone actually seen an IT related EU project that achieved something? The company I work for has been involved in two EU project proposals so far, and nothing came of either of them -- though they both consumed a large ammount of resources from universities to get through the three failed applications each.

Perhaps you are expecting the wrong results.

I have been involved in a couple of large EU funded projects, and have spoken to the project managers about the aims and motives of the projects.

One principal point is that just because a new successful product/standard/format whatever does not arise from a project, does not mean that it has been a failure.

The EU is made up of lots of different countries with lots of different types of people speaking different languages and with different working mentalities. This is a major competitive disadvantage for us compared to a country like the US. If a company in San Francisco wants to work with a company in New York, there aren't many barriers to them doing that. In the EU, there are lots of barriers. One of the main aims of EU funded projects (and the EU in general) is to break down these barriers by getting different companies and universities working together across the EU. If new technologies come our of these projects, so much the better, but that's not necessarily the principal aim.

Virtual science

[Sloppy]

This reminds me of an astronomy-related story I saw yesterday. Some projects are generating more data than the people doing the projects can handle.