Virtual Grid Supercomputer Goes (Partly) Online

hotsauce writes "The BBC is reporting that CERN (the guys who invented the Web) are working on a virtual supercomputer called the Grid. The Grid taps computing power from 12 countries to process data from a new supercollider that will simulate parts of the Big Bang. Phase One of the Grid just went online."

Comments

Over-hyped

[TwistedSquare]

I have heard from some people involved in the grid that it is a triumph of PR over substance, and that it is not going to be as well-used and participated in as the press releases suggest...

Re:Over-hyped

[grid geek]

I'm doing my PhD on it at the moment. While there is a lot of hype there is a lot more substance to it than there was 2 years ago.

For example we now have a single sign on system spread over more than a dozen countries (never easy when the gov wants to know why other countries scientists are using their machines.)

Loads of work has been done on integrating data resources into the network, large scale systems configuration, data discover etc.

As ever the BBC is a bit behind on the news - this is just a new releases of the software. And there had better be substance when LHC comes on line in 2007/8 as it will be generating 15, 000 TB of data to be stored a year and we need some way to deal with this.

Re:Over-hyped

[cperciva]

I'd go further than that. I've heard the EU DataGrid described as a perfect example of how not to run a large software project.

People working in half a dozen different languages (natural languages, that is, not programming languages!), a complete lack of chain-of-command, software being signed off on because a document detailing critical bugs had been produced (even though the critical bugs weren't fixed!), progress reports being sent back for revision because "they weren't positive enough"...

Take the worst elements of decentralized open-source programming, combine them with the worst elements of government beaurocracy -- and then replace half of the programmers with physicists who have less programming experience than most second year CS majors -- and you'll get somewhere close to what I've been told about the EU Datagrid.

Of course, this is all second-hand information. Maybe the person who was telling me all of this is just a burnt-out coder who would be unhappy with whatever project he was working on. But I doubt it.

Re:Over-hyped

[pubjames]

People working in half a dozen different languages (natural languages, that is, not programming languages!),

Welcome to reality. Reality for most of the rest of the world, that is.

In Europe we speak lots of languages. That's the reality of the situation. Of course this isn't as efficient as just speaking one language, but it is the reality. I know, as an American you're probably thinking "well why don't you all just speak English?"

I had a funny situation on the outskirts of San Francisco once. I asked a guy sweeping up outside a MacDonalds some directions. I replied in very stilted English that he couldn't understand me because he didn't speak English. I noticed his accent so I asked him my question in Spanish. The look of suprise and joy on his face was comical. He told me that most Americans can't speak another language and expect you to learn English, but not speaking another language themselves don't realise that learning another language isn't easy. And they treat you like crap and think you're stupid if you can't speak English. I don't know if what he said is true or not, but unfortunately I don't find it hard to believe.

Re:Over-hyped

[cperciva]

Sure, there are lots of languages in Europe. But do you really need to have half a dozen of them in a project which only has a hundred people?

Any company which was running a project like this would have picked one location, hired a bunch of people, and had them all working in the same building, speaking the same language. It's only because of EU politics -- the requirement that the EU Datagrid be a multinational consortium -- that there are so many languages and locations involved.

Re:Over-hyped

[Znork]

"Any company which was running a project like this would have picked one location, hired a bunch of people, and had them all working in the same building, speaking the same language."

And then tried to launch the grid project at multiple corporate sites, whereupon they discover the nice little problem that corporate politics and economics result in them not being able to run that project anywhere but at the site where the project was developed - your project, we're not contributing a cents worth of spare capacity, go buy your own machines.

If you want a project like this to have the faintest chance to succeed you get all the people who are expected to contribute anything out of their budget involved. Or you'll get sunk. Even in a corporation. And for a multinational, that means you have people with different native languages. This is usually not a problem in a multinational because the people involved in such projects usually have at least one language in common, and often more.

When will we do this ourselves?

[Max Romantschuk]

Distributed computing has been a long time coming. Sure, grids are cool, but when can we download a safe piece of software which to use for distributed calculations? When I'm not it need of doing stuff myself it would use my idle time for other people's calculations, and vice versa.

Sure it doesn't work for any type of calculation, but there's still a huge potential.

Distributed backups is another thing I'd like to have now, rather than tomorrow...

Re:When will we do this ourselves?

[grid geek]

No, seti is a distributed application - 3 million instances of the same program. The grid is a distributed environment - an operating system if you like, which programmers can write their own applications to run on top of.

Like the OS for your desktop the Grid's middleware software deals with things like I/O, resource allocation, security etc.

So, seti@home could run on the grid, but is not a grid in its own right.

Re:When will we do this ourselves?

[sql*kitten]

Sure, grids are cool, but when can we download a safe piece of software which to use for distributed calculations? When I'm not it need of doing stuff myself it would use my idle time for other people's calculations, and vice versa.

You can get it here along with some case studies of how it's used in production.

Distributed backups is another thing I'd like to have now, rather than tomorrow...

Uuencode, split, and post to Usenet...

Datagrid homepage

[maharg]

http://eu-datagrid.web.cern.ch/eu-datagrid/

Re:Datagrid homepage

[henben]

Jesus, look at this.

Here's their "intranet", which is publically accessible:

http://eu-datagrid.web.cern.ch/eu-datagrid/intrane t_home.htm

You can't access all the reports, but you can get lots of design documents and stuff. It's done in FrontPage and riddled with spelling errors. Not promising.

Don't diss it

[wodon]

I know it sounds like the world domination scheme from a bond film, but I've seen what it can do.

A friend of mine is working on part of the grid for his PHD in the uk
and once you have watched him transfer 700 MB files from the uk to
Switzerland in under a minute you realise that they aren't joking.

I guess their next step is running quake over it!

Power to the Grid!

[richie2000]

The Grid taps computing power from 12 countries

And electrical power from how many?

(
I for one welcome our new Grid ove *thwap*
But, just imagine a Beow *thud*
In Soviet Russia, the Gridski compu *wham*
)

What they do at CERN

[hkon]

Is it just me, or is referring to CERN as "the guys who invented the web" a bit like referring to Paul McCartney as "the guy from Wings"? To a lot of people, CERN is probably better known as "the guys who know more than anyone about particle physics".

Re:What they do at CERN

[AlecC]

Nasa wasted millions developing a zero-G ballpoint, whereas the Russians used pencils.

Urban Legend

Under-hyped

[pubjames]

I think one thing that distinguishes the USA from Europe (and in fact many other countries) is that, in the USA, you're really good at hyping stuff.

Over here in Europe we're crap at it, or rather, it's not seen as being so important - which I believe is a mistake.

I was once knew a marketing person at CERN who said that as they had invented the web, they were thinking of putting up some good web pages about it to let the world know what they had achieved. Later I found out that she had been told that it had been decided that it wasn't a priority and so no budget was allocated to it. So outside of techy fields most people in the world don't know that the web was invented in Europe. (Yes, I know the Internet network was an American invention).

This is true of many other achievements in Europe. For instance, everyone in the world knows lots about NASA and yet very little about the achievements of the European Space Agency, which has also done some really impressive things.

The USA also has a good way of claiming innovations as its own, by "Americanising" everything. So, for instance, most people - Americans and non-Americans - think that most technological innovation during the last century has been American, whereas in actual fact much of it was just "absorbed" by America. Example - landing on the moon is seen as a purely American achievement but was actually significantly based on German work. Another example - Einstein was of course European but many people believe he was American.

Don't get me wrong. I'm not really criticising the USA, I'm actually criticising Europe for not promoting its achievements or those of its citizens. And of course one of the great things about the USA is that it has the money and drive to make things happen, which is why many projects initiated in other places end up taking off in the USA. But it is frustrating to meet Americans that believe that everything is invented in America.

Re:Under-hyped

[pubjames]

Einstein, of course, was an American, and like many other Americans, he was born somewhere else, and got here as quickly as he could.

Einstein was born in 1879 and moved to America in 1931 at the age of 52.

Re:Under-hyped

[pubjames]

I think the responses I am getting to my post just demonstrate the point I am trying to make.

Einstein moved to America when he was 52. He had done nearly all his most important work by that time.

You say that this is because Americans have a different attitude to imigration. However, when Americans move abroad you don't seem to consider them no longer American.

I understand Madonna now lives in London. Should I now start referring to her as British? Are all her achievements now British ones? Is Gore Vidal Italian? He's been living there for years. Are his books now Italian ones?

You're just proving the point I am making.

Re:Under-hyped

[Aglassis]

You said: "(Of course then, you have to wonder why the US lags behind some other countries in science education....)"

You have to be really careful to say exactly what you mean when you make general statements like this. While it is true that US K-12 (or in particular 8-12) science education falls behind much of Europe, that is not true for college education or other types of education not normally considered (thinktanks, museums, libraries, private education, etc.,). You can't always compare one countries science education against another because they use different methods of implementation in their education process. For example, some countries give a test in about the 8th grade. Depending on the student's score, he or she is trained in vocational or scientific disciplines for the equivalent of his or her high school period. These vocational students are not tested for science when compared against other countries.

Mmm, supercollider

[Spunk]

I'm getting a hadron just thinking about it.

I used to work there...

[mt-biker]

I used to work in the computer centre at CERN, and they've been using distributed computing (read "clusters") for a long time (at least 10 years) now. By the time I left, there were already some 500 2 CPU Linux PCs in the computer centre, and a serious amount of thought was being given to building a mezzanine level within the computer centre to create more floor space for PCs. CERNs problem was always one of scale.

Now it seems they want to buy floor space at other institutions around the world. :) I hope it all works out. What I'm wondering is what sort of network they have connecting the sites - the work load of these machines is very simple - but mostly IO bound. What sort of bandwidth do they need to make 15,000 TB available all over the world?

More fun facts - at the time I left they had 5 STK Powderhorn silos, holding their current data. Prediction for LHC requirements (including better tape storage densities) was that they would need another 40 silos. If you've seen an STK Powderhorn, then you know just how ig the things are. So another building was to be built just for these silos. :)

Oh, and as someone pointed out, the 15,000 TBs a year is just the data that gets kept - the live data from the detectors is preprocessed in the computer centre and "thinned out". The data rates coming into the computer centre are truly mind-boggling.