Mass Grid Computing Around the Corner?

zoglmannk asks: "I've become interested in grid computing. A lot has happened since the last time that I looked at it several years ago during the SETI@home heyday. Now several public supported grid applications are coming to fruit: climate modeling, cancer research, protein folding, smallpox therapies, fighting bioterrorism, mersenne prime search, evolution, SETI, and others. All of these have public interest to make a better world. Is mass adoption of public interest grid computing just around the corner? Is there really a need for a majority of those spare CPU cycles? Or is there more computing power than can reasonably be used for the types of problems that can be distributed to home and educational PCs? What is needed to bring grid computing to the masses? More education, advertisement, prizes, reimbursement?"

Comments

No spare cycles

[rice_web]

I can assure you that there are presently no spare cycles..... of the people that even know that these projects exist. Okay, fine, I myself could run Folding @ Home on another two machines, and I'm sure the case is true in even many geeks homes. However, geeks represent a small number of people, and many that I've talked to have no idea that this is even possible.

The mass grid computing market just got saturated.

Cluster maybe, in a couple of years

[Kris_J]

I don't know about grid computing, but the little Dell PCs we have in our computer labs will start coming out of lease in a couple of year's time. I know they're "identical" because I manage them with Ghost. I'm seriously tempted to get 10 or so then and try to build my own cluster. They even have Gigabit ethernet built-in. Some cat 6 cable, a 12-port switch, a few powerboards, one master with a screen... I just need something worth doing with them.

Re:Cluster maybe, in a couple of years

[Blittzed]

We have already just done the same thing with a dozen PIII 866 machines out of a computer lab. We are running a rocks distro on it. This thing has got some grunt! I would seriously recommend having a go at it. A good switch is essential, especially if you are running something like Povray: this generates heaps of network traffic. A switch with a massive backplane is nice to have :) For other apps like John the Ripper, wihc we are currently running, the network isn't a bottleneck.

What is needed?

[Kardamon]

What is needed to bring grid computing to the masses?

We need more hype.

An easy reward...

[WordODD]

Most of these projects allow you to join teams and obtain stats based on your computer's performance and the performance of your team as a whole. The collection of these stats seems to be a successful method of motivating and rewarding the participants and best of all it takes minimal time, effort and almost no other resources away from the main project.

Distributed != Grid

[Anonymous Coward]

Grid computing is not the same thing as distributed computing. You are talking about distributed computing.

Here is a pdf describing what grid computing is.

Heating systems

[ballpoint]

With the ever increasing energy prices and power consumption of CPUs it might make sense to develop a electric heating apparatus (a radiator) that, instead of resistors, contains 32 *iums or *ons inside for a 'cool' 3KW of heating power.

Link them to a fast internet connection, pay a fraction of the heatees' power bills and you're in business for CPU intensive, network extensive grid applications.

Re:Heating systems

[Councilor Hart]

1) Build a small closet or room in you house with an air conditioner.
2) Stuff it with computers
3) sell your spare cpu cycels
4) use the energy extracted from to room to heat your house, thereby keeping the computerroom cool

hmmm, I forgot the ??? step. I knew something was wrong with this business model.

Distributed, Grid and how they differ

[basking2]

Most (all?) of the cited examples are ditributed computation projects. Most have agents that call up a server and signout work to do.

The vision of grid computing is to treat computation power like electrical power. Where there is demand, the computation power "flows" and "fills" that demand.

For example, a simple example is that you have a webserver cluster, a database cluster, and a network area storage cluster, all made up of off-of-the-shelf PCs. If demand for database logic goes up and stays up, one of the computers being used for redundant storage could STOP replicating data and START being a database server. If the web requests go up, one of the database computers could stop handling database requests and start handling web requests.

Obviously, there are hard limits that must be met. We must have at least 1 web server, 1 database server, and enough storage machines to keep our data safe. We also have to detect failure or excess load and have a transition path for the generic computer to change what computational task it helps with.

Amazing stuff, and if you ponder the details of it, it is a bit of a challenge!! As another poster mentioned before, this does relate to self-healing technology and it also does relate to the distributed computing projects you mentioned in your post.

Hope this is helpful.

Re:Distributed, Grid and how they differ

[Myself]

And I was just about to make a post suggesting that we optimize communications to make the bandwidth consumption of these projects easier to manage.

Most folks try to live near their jobs, or at least in the same state. If one spare cycle is the same as another spare cycle, why not have everyone in a particular geographic area work on the project whose servers are in that area? Then if a new project pops up, or if a grid computing task comes along, nearby resources could simply be reassigned.

A universal client, which would run whatever cycle-sucker was appropriate, could make this easy.

Re:Distributed, Grid and how they differ

[Anonymous Coward]

The vision of grid computing is to treat computation power like electrical power. Where there is demand, the computation power "flows" and "fills" that demand.

...

Amazing stuff, and if you ponder the details of it, it is a bit of a challenge!!

That's an understatement. I have the misfortune of working on grid computing and it's grossly immature and completely unsuitable for production. Grid computing is layer upon layer of software, most of which is alpha quality, none of which is any good at propogating errors, running a mix of sites administrated by different people who do slightly different thing. We're nowhere near the vision. It's just research that may pay off in the future. Unfortunately it's being touted as a mature technology, so when people deploy it it inevitably fails and people get angry and us peons have to deal with the crisis of the moment.

Re:Distributed, Grid and how they differ

[Anonymous Coward]

I don't think the power grid metaphor is apt. With electricity, control is centralized. You have a single producer of electrical resources and many consumers. In the distributed computing projects, you have many producers and a single consumer. It seems to me that grid computing envisions many producers and many consumers of computing resources. I have very little experience or knowledge of the area, so this may not be accurate, but it is my impression based on the types of systems I hear being described.

Check out OGSA

[jonasmit]

I agree with what others have said Grid Computing is distinct from other forms of distributed computing.
Check out OGSA - the Open Grid Services Architecture - and learn what is and is not a Grid. This is the de facto standard for building Grids.
Even new products that are *sold* with Grid in their title aren't necessarly THE GRID though they might be A GRID.

semantic grid

[fraccy]

Traditional grid computing stemmed from the science community which had specific needs to both collaborate across heterogeneous administrative domains, and to process increasingly large amounts of data (often at a budget). This vision has been increased to incorporate the ultimate goal of distributed computing, in which total global processing, storage, and computing resources in general can be seen as effectively a single entity. The problems in addressing this are huge and a major focus of software research. Grid computing systems as they are today tend to be only usable by relative experts, and it is still sometime before the average user will find themselves able to regularly harness such power - something which must happen before widespread use. Supporting such complexity requires increasing automation in the self-management of systems, which requires better representation of data and processes in terms of semantics. Ultimately the future of the web and grid computing are likely to collide into what some term as "ubiquitous computing". See www.semanticgrid.org for some more info.

exploit

[lcde]

What is needed to bring grid computing to the masses?

A new microsoft exploit :D

Apple QMaster, XGrid

[sakusha]

Like I've been saying for quite a while, Apple finally announced that all their Pro AV products will support distributed processing with QMaster. Currently Shake, Compressor, and Maya are supported, soon Final Cut Pro support will be added. Everyone will go nuts once they see the performance of FCP with distributed processing.
And then there's XGrid, designed for cooperative processing in scientific tasks. I haven't used it so I can't judge its utility.
Yep, Apple is leading the way with this sort of Grid Computing software. This is what you can do with a bit of your own proprietary software built on top of Open Source unix. And I don't see any reason why people couldn't extend QMaster and XGrid support to other platforms, creating a compatible OSS version.

Re:Apple QMaster, XGrid

[Anonymous Coward]

And then there's XGrid, designed for cooperative processing in scientific tasks. I haven't used it so I can't judge its utility.

It blows. Seriously.

Re:Apple QMaster, XGrid

[sakusha]

Well, be fair, XGrid is just a demonstration, it's hardly even beta. But QMaster rocks. I wish I had a 5U rack of XServes to run it.

distributed.net ?

[scythian]

Did you forget to mention distributed.net ?
They have been doing this for years ... OGR, DES, RC5, CSC ... dare I say the biggest one ever, except for the ueber-d.net-geeks who had to choose SETI. :)

money

[bcrowell]

I expect that fairly soon there will be an efficient market for CPU cycles. That might help to offset the cost of a new machine, but it will also probably have the effect of killing off the donation-based projects like SETI@home.

Is there really a need for a majority of those spare CPU cycles? Or is there more computing power than can reasonably be used for the types of problems that can be distributed to home and educational PCs?
It doesn't take much creativity to think of ways you could use up ridiculously large amounts of cpu power. In the relatively near future, I can imagine:

• spammers paying people to compute hashcash postage
• online gamers paying for rendering of Pixar-quality animation

For the more distant future, a lot of really difficult AI problems might become more tractable if you could throw enough CPU power at them. Neurobiologists can already simulate the nervous systems of very simply organisms; maybe at some point it will become possible to simulate a human brain directly.

Re:money

[ear1grey]

The processing potential of beyond-petascale grids is still insignificant compared to the atomic-level complexity of your own intestine. Therefore companies that can afford to pay more for resources will help to keep the cost relatively high - for example a pharmacutical company may wish to use myGrid to perform some particularly extensive in-silico experiments regarding intestinal chemistry - consequently the cost of supercomputing in the future will still be beyond most people.

That's not to say the the amount of processing potential for end users will not increse - it just won't increase as a significant percentage of the global total, so "money" will limit grid access for the commons.

Spare cycles no longer exist....

[decep]

Most modern CPUs no longer "waste" spare CPU cycles like they once did.

If your CPU runs at 100%, you are using more power and therefore makes your electric bill increase. Therefore, when you run distributed applications, you are actually paying $$$ for what you are giving away.

In recent history, laptop CPUs have started throttling themselves and using even less power, and desktop CPUs will start doing the same before long.

Not that this has that much to do with grid computing...

Grid computings goal is usually greater utilization of your own resources...distriuted computing usually utilizes someone elses resources. :-) (I know, true and not true at the same time)

Re:Spare cycles no longer exist....

[excaliber19]

Most of the electricity is hogged by the monitor. Without your monitor on, the net sum of your electricity consumption is actually rather small. Anyhow, its alot easier for a million people to pay an extra 10 dollars a year to cure cancer than 10 people a million dollars a year. Think about it.

The masses.

[ear1grey]

zoglmannk asked:

What is needed to bring grid computing to the masses? More education, advertisement, prizes, reimbursement?

Grid computing is not aimed at "the masses". Most of the research is concentrating on building systems for solving problems which normal people have no interest in.

That's not to say that we plebs won't benefit from a "cure for X" or "lower oil prices due to better flow models within pipelines" or even "more efficient cars desgined in simulated air tunnels"; we are just very unlikely to be the imediate user of the computing power.

Grid Grammar

[ear1grey]

The (appropriate) capitalisation of the term "GRID" in the parent is for emphasis; for those reading this and just learning about Grid Computing please note that the term "grid" is not an acronym so capialisation is inappropriate.

Similar to the distinction between "internets", "an internet", "internet technology" and "the Internet" when writing about grid computing it is appropriate to discuss "grids", "a grid" and "the Grid". With the capitalised form being used when referring a definite article such as a unified grid of grids.

always had this question about clusters...

[BigGerman]

so maybe someone will enlighten me.
If you build a cluster, among other things, you pay premium for:
high density server components;
major cooling and power.

Now if I use gigabit for the transport, can't I just distribute my boxes throughout the building so they can breathe easily?

Re:always had this question about clusters...

[rmkent]

One word.... Latency.

Or to put it another way:

Would you go to Italy for coffee when the coffee maker in your office works?

Costs beyond electricity?

[SecState]

Slightly off-topic, but I've long wondered whether running one of these programs decreases the lifespan of a computer. Do the excess heat and months of constant usage take a toll on the processor and other components? My P3/900 laptop, for one, sure gets hot when I run Folding@Home.

Re:Costs beyond electricity?

[Bandrew]

All non-mechanical parts are generally safe when running destributed computing (or just leaving your computer on for that sake). The thing you need to worry about is your darddrive, science their tendency to fail will increace ar the parts gets worn from the mechanical stress. Your cpu-cooler could also fail, but in that case your computer will most likely shut down before any dammage is done, sience the bios is monitoring the cpu temparature, and atleast modern bioses will shut down automaticly when a critical temperature level is reached.

The Bill Gates factor

[Anonymous Coward]

What is needed to bring grid computing to the masses?

If you ask me, it's for the man himself to incorporate it into the next version of Windows. Before long, a Distributed Computing application would be on most PCs around the world. You would only have to scroll down and select the project you're interested in. (I'm sure there would be a default mode set for Bill's pet project)

It's not unlike what Google did with their toolbar. They incorporated the Folding@Home software into it, so anyone who downloaded the Google Toolbar would automatically have FAH on their system. (i'm not sure if it defaults to working mode or if you have to turn it on, nor am I sure if they're still doing this)