Grid Computing: Conceptual Flyover For Developers

So what exactly is "grid computing"? by evil_one666 · 2004-11-01 01:09 · Score: 4, Insightful

So what is exactly "Grid Computing"? From the article...

Sometimes it's easier to start defining Grid computing by telling you what it isn't. For instance, it's not artificial intelligence, and it's not some kind of advanced networking technology. It's also not some kind of science-fictional panacea to cure all of our technology ailments.

If you can think of the Internet as a network of communication, then Grid computing is a network of computation: tools and protocols for coordinated resource sharing and problem solving among pooled assets. These pooled assets are known as virtual organizations. They can be distributed across the globe; they're heterogeneous (some PCs, some servers, maybe mainframes and supercomputers); somewhat autonomous (a Grid can potentially access resources in different organizations); and temporary.

Nope- that still does not tell me what "grid computing" is. This vague, loosely defined definition can describe just about every "next big thing" since the mainframe.

Re:So what exactly is "grid computing"? by carnivore302 · 2004-11-01 01:17 · Score: 4, Informative

Hmm. It's not that hard, but I agree: the author seems to obfuscate a fairly simple to explain principle.

In a few words: grid computing is the use of many connected computers for one task.

Or, you might want to think about it as multithreading, but spread out over multiple machines.

The author is making a case for a standardization of how this should be handled.

--
Please login to access my lawn
Re:So what exactly is "grid computing"? by lachlan76 · 2004-11-01 01:20 · Score: 4, Interesting

Keep going:

Consider this: most IT departments are being forced to do more with less. Budgets are tight, resources are thin, and skilled human resources can be scarce or expensive. To top it off, most corporate managers know that they have a super-abundance of idle computing power. It's well known in industry circles that most desktop machines only use 5% to 10% of their capacity, and most servers barely peak out at 20%. No surprise then that many of the big money people in corporate America balk at the thought of purchasing more equipment to get the job done. What these companies need is not more horsepower, but more efficient use of existing horsepower. They need a way to tie all of these idle machines together into a pool of potential labor, manage those resources, and provide secure and reliable access to the number-crunching muscle. Imagine if a corporation or organization could use all of its idle desktop PCs at night to run memory- and processor-intensive tasks? They would get more work done faster, possibly get to market faster, and at the same time cut down their IT expenses.

The idea seems to be to turn the whole network into a cluster. "Why buy more servers when you can gove some of the load to your desktops?" is a short summary.
Re: So what exactly is "grid computing"? by Alwin+Henseler · 2004-11-01 01:50 · Score: 4, Informative

"In a few words: grid computing is the use of many connected computers for one task. Or, you might want to think about it as multithreading, but spread out over multiple machines."
In a way it's a matter of taste, but I'd define it this way: "parallel" -> many CPU's, but quite close in 1 place, like in a SMP desktop. "distributed" -> with network in between, as in Beowulf cluster (possibly over the internet).
What would make "distributed" a Grid? The fact that it's 'everywhere', always working/available somewhere, like P2P networks. You can take your equipment off the network, but the network (ehh, grid) goes happily on doing its thing.
This becomes really useful when it's a easy to use and commonplace as the internet today. Send out some software, it grabs a piece of data here, grabs a program there, finds a server to do the computation, and reports back to you with the result. Got some cycles to spare? Put some in the Grid, earn money. Just wait and see, some day computing power will be supplied and consumed the way electric power is today.
Re:So what exactly is "grid computing"? by Liquid-Gecka · 2004-11-01 01:55 · Score: 2, Informative

Grid computing is the concept of using distributed resources as one big resource. For example, Boise State currently uses all of the computers in the Engineering labs as a super computer when classes are not in session. Micron Technology uses all of its desktop systems as one big super computer.

Todd Tannebaum just gave an exec lent keynote at Boise State's HPC Workshop. He explained that while computing power has increased on a system by system basis, the total available computing power to a single person has actually decreased. For example, if you wanted all the computing resources in the 70's you simply logged into the department computer. Now you can't do that. You get a fraction of the total power in the department. Grid computing is attempting to fix that by using the desktop systems together as a big super computer.

For more information on Grid Computing check out Condor. It is a super powerfull grid computing environment.
Re:So what exactly is "grid computing"? by Little+Hamster · 2004-11-01 02:00 · Score: 2, Informative

The idea seems to be to turn the whole network into a cluster.
This is answered in the FAQ at gridcomputing.com:
The key distinction between clusters and grids mainly lies in the way resources are managed. In case of clusters, the resource allocation is performed by a centralised resource manager and all nodes cooperatively work together as a single unified resource. In case of Grids, each node has its own resource manager and don't aim for providing a single system view.
Re:So what exactly is "grid computing"? by ColdGrits · 2004-11-01 02:16 · Score: 2, Informative

Grid Computing - simple.

As far as the user is concerned, you have 1 Black Box system containing everything.

The physical implementation could be a single supercomputer, or a whole host of different systems spread all over teh place. But the the end user, it's just a single computer that handles all their stuff for them.

That's a grid. Check out http://wwws.sun.com/software/n1gridsystem/ for a good overview of how this can be implemented.

Remember, a true grid system is more than just raw CPUs, it's all about services, processor, disk resource, the whole thing.

--
People should not be afraid of their governments - Governments should be afraid of their people.
Re:So what exactly is "grid computing"? by Anonymous Coward · 2004-11-01 02:30 · Score: 2, Informative

Think of grids as a distributed set of clusters. Instead of using distributed networks of single PCs you connect clusters.
The aim is get rid of the usual limitation to clusters (homogenious hardware/os) and make use of all the clusters in a virtual organization (think university network, distributed.net, seti@home protein folding and similar research institutions). You can then use a resource broker to specifiy you resource, say all big-endian machines with more than 1 gb of memory or all pentium machines.

Having played a little with Globus toolkit 2 the main limitation was that it required similar hardware/os to run as it was a C sdk, requiring a separate binary for all platforms. GT3 is written in Java to get rid of that limitation.
Basicly you write your program, compile it, place it on a resource visible to the grid and specifiy what kind of hardware you want to run the program on (number of cpus, memory requirements etc) and submit the job. Then wait for the results to pop out.
Re:So what exactly is "grid computing"? by kcm · 2004-11-01 03:15 · Score: 2, Funny

Nope- that still does not tell me what "grid computing" is. This vague, loosely defined definition can describe just about every "next big thing" since the mainframe.

I like to think of "grid computing" as the idea of building technology with the global use in mind: pluggable security models, standard protocols for job management/data transfer/etc., and so on. how to build services with the future (where the future is sharing those services) in mind.

grid computing MAY take the form of: shared supercomputers; grid services; cycles for sale; cluster-of-workstations; distributed focused problem solving; etc. .. but these are all just applications of the above.

yes, I know this is still a very abstract idea, but I never promised it would be as good as the glossies promised. :)

Change can be hard by millahtime · 2004-11-01 01:10 · Score: 4, Interesting

Recently I saw a similar design for a network and some "old timers" said it was no good to do it this way. It wouldn't satisfy the needs.

One thing I have noticed is that for many "old timers" there is the feeling of we have always done it the old way, why change. Any thoughts of how we drag that old donkey into the new methods when they don't want to go?

--
Evolution or ID?

Re:Change can be hard by Timesprout · 2004-11-01 01:18 · Score: 4, Insightful

Well this is the thing. Many old timers are not adverse to change when clear benefits can be shown from adopting the new technology.

Unfortunately for Grid Computing its still in the stage where people are struggling to explain what it is, nevermind what it does or how it can improve life. Thats always going to be a hard sell to me anyway. If its function is not obvious it makes you sceptical just how necessary it is.

--
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
Re:Change can be hard by the_duke_of_hazzard · 2004-11-01 01:41 · Score: 2, Insightful

It's going to be painful and resisted until it "just works" (TM), and then everyone will want to use it. So the trailblazers will most likely be techies who become suits, and want to prove themselves by saving a business loads of money by doing this. Probably banks, non?
The overheads will be enormous though, at least initially. Security issues, data issues, even logging issues have to be thought through. Any system that implements this will be *fucking* complicated to work through.
In any case, the "old timers" are often right, and call the younger guys on their technical wet dreams. In between the two camps lies progress...

Why is slashdot pushing grid computing so much? by Tedium+Unleased · 2004-11-01 01:12 · Score: 4, Informative

The linked article is written in May 2003 yet it's new now?

Unready Hype by Anonymous Coward · 2004-11-01 01:12 · Score: 5, Interesting

As part of a university group that adopted Grid computing about a year ago, the Grid is mostly over-hyped material that isn't ready for prime time. The basic idea (see e.g. Legion) worked more than a decade ago, but what I've seen of today's Grid software is fragile, overcoupled, underdocumented, and doesn't yet deliver on all the promises.

We were taught that the test of research software is whether a full professor (or corporate executive or other obscenely busy person worth >> $100/hour) finds it useful enough that they take time to learn it - the uses I've seen for the Grid don't pass that threshold yet.

There are some exceptions: tightly-integrated applications put together in a couple of the hard sciences that really just do supercomputing with a friendlier face. There's enough payoff there for a physicist to be happy with the software.

For a geek, however, even there, most "grid UI research" is simplistic, derivative, and uninspired.

Apologies to my first-ever-advisor who is now a Grid bigwig. :)

Re:Unready Hype by Anonymous Coward · 2004-11-01 05:26 · Score: 2, Informative

As former Director of Operations for a large Grid facility, I agree with you, and I think most of my colleagues would as well.

It has been standard practice over the past couple of years to overhype Grid, a practice which I suppose was intended to bootstrap interest but which instead just tends to leave people feeling confused and vaguely betrayed as they discover that what was presented as a production capability turns out to be a research project in its early stages. The article is typical of the approach.

There's a huge conceptual gap between "Grid as utility computing" and "Grid as a usable set of application services" which no amount of hype will close. Those of us who actually have to talk to upset users about this are left in a very embarassing position.

Yes, the basic Grid vision is right, and the Grid Services Architecture appears conceptually sound, even if implementations are not yet complete, let alone fully interoperable. No, in practice it's nothing remotely like plugging an appliance into a power outlet to toast your bread. Will it ever be? I'd say that at this point in history, we can't know.

Apart from the requirement of functional completeness over a very ambitious domain of capabilities, the Grid computational model must also achieve an ambitious degree of interoperability, that is, if our goal is truly to capture unused compute cycles rather than to justify the development of new computing infrastructure. Amd on the subject of interoperability, Gordon Bell has a few cautionary words:

Standards should be based on real experience, not on committee designs. Perhaps an even better way of putting this would be: "If you haven't actually lived with the design proposed as a standard, don't adopt it." The best way to establish whether a specification is real or not is to implement several alternative interfaces. In fact, the IETF has set just such a rule for itself, holding that no standard can be created unless there already are at least two interoperating implementations. Similarly, computer users who hope to use a particular standard to leverage their buying power should always take care to test their systems on two separate implementations before deciding whether to link their fates to that standard. The Grid community in particular would be well advised to adopt just such a discipline before wedding itself to standards that define its future. Unfortunately, the Grid software is being done in a monopolistic fashion by a few government labs and not in the fashion of IETF.

Gordon Bell is being somewhat unkind here, because he knows better than to treat the hype at face value. Your perspective is more realistic. Supercomputing is a relatively natural platform for implementing Grid applications, but adoption is proving to be nontrivial even there. That's the place to watch for it. Meanwhile, my advice would be to ignore the hype, but by all means read Foster and Kessleman. It's interesting stuff.

Grid by cuteseal · 2004-11-01 01:14 · Score: 2, Funny

"What is the matrix?"
"No Neo, try again"
"What is grid computing?"
"Bingo."

There's certainly alot of info to devour there, but I guess if companies like Google and Dreamworks are using it, then it has to be a Good Thing.

--

The friendliest digital photography forums on the net!

Security in Grids by ifoxtrot · 2004-11-01 01:15 · Score: 5, Interesting

I have had some experience with grids and the overwhelming difficulties I've come across have been in the areas of security.

First and foremost, grids are designed to run in a distributed environment which makes security design and administration that much more complex.

Second, grids are currently in their infancy and there is little prior art to the types of attacks and problems that will affect them. Despite this, they are very juicy targets with the kind of storage and bandwidth that would make even a hard-bitten cracker weep for joy. (i apologise for the imagery)

Third, in my book security has to be a top-down approach - i.e. the guys on top lead the way and then everyone else follows. Grids have no tops or bottoms which makes this a bit tough to apply. In short there is no security hierarchy in a default grid environment. Responsibility HAS to be established explicitly. A simple example is who is responsible for the data held on one of the nodes? Is it the person who wrote the application, the person who owns the application, the person who owns the hardware?

Grids are fascinating in their security requirements (and those who think these are solved by web services have another thing coming! People are a huge aspect of the security of a system, and distributed system like grids have a very complex task of ensuring that people behave the way they should).

Grid Computing is a buzzword by joib · 2004-11-01 01:20 · Score: 5, Insightful

Like, if you fit in "grid computing" in your grant proposal, the probability that you'll get funding increases. Now, if in addition to "grid" you manage to fit in "nanotechnology", "bio-informatics" and "paradigm" you'll be funded with a probability very close to 100 %!

Well it's ONE view on Grid Computing by joelethan · 2004-11-01 01:33 · Score: 4, Informative

This is meant to be a primer, and it just about "primes" the debate on Grid Computing.

The grid discussed here seems only to be built on the OGSA and Globus Toolkit, and Globus has not really covered itself in glory with their poor UIs etc.

Grid seems to address occasional demand for "much more power" from your computing resource, but does not really provide a consistent flexible computing resource.

The academic world uses External Grids to pool resources but private Enterprise has little to gain from these External Grids in exchange for a HUGE security problem.

And Internal Grids? These are so immature as to beggar belief. Why risk investing in these configurations when bang per buck is so uninviting.

/joelethan

Re:Hey by B1ackDragon · 2004-11-01 01:34 · Score: 2, Insightful

You know we're talking about a dead meme when the first comment is modded redundant.

The beowulf cluster joke is dead, long live the beowulf cluster joke!!

--
The snow doesn't give a soft white damn whom it touches. -- ee cummings

Re:The name sucks by ifoxtrot · 2004-11-01 01:51 · Score: 2, Interesting

Actually I always thought the name came from the concept of power grids. i.e. plug in your application to the computing grid, get it to run your computations, and get the results without having to worry about how the computing power got to your home...
Kind of similar to a power grid no? plug toaster, insert bread, get toast - no need to worry about coal/oil/nuclear fuel burning, transfomers or megawatts...

Practical application... by BobRooney · 2004-11-01 02:43 · Score: 2, Interesting

As an "boots on the ground" IT professional it would be nice to have a consumer grade "grid computing" solution to offer some small business customers as an alternative to buying a server farm for the two days a month they actually put strain on it.

If there were an easy way to cluster their workstations they wouldnt need to invest in an underutilized server farm. They could just schedule their processor/disk intensive reports and processes for off hours or rely on grid load balancing to take the extra cycles from the computer of the CSO (Chief Solitaire Officer) so that the impact would be imperceptible to the average user.

The current problem with the concept of grid computing is the lack of an easy way to deploy it in a standard business environment. What the article and its links are driving at is coming up with a cheap and easily implimented mechanism to turn every office, and chain of offices into a grid.

In theory, you could sell your unused processor cycles the same way people who generate their own power sell power back to their power companies. You ISP could actually, someday become a processor cycle reseller and you could operate on a minimal set of hardare in the typical office enviroment becuase you can always pick up extra cycles from your ISP when you need them.

Ah, the pipe dream.

If grids worked, hosting companies would sell them by Animats · 2004-11-01 05:28 · Score: 2, Interesting

As I keep pointing out, if grid computing was good for anything, it would be a service that hosting companies sold to keep their machines busy during off-peak hours.

Hosting companies have large numbers of identical machines with high bandwidth interconnects. That's just what you want for "grid computing". They're already set up to allow customers to run applications on their machines, and are able to deal with the security problems. Load is very low during off-peak hours. The machines stay up; they don't suddenly get disconnected from the net because somebody turned their desktop off. They're all loaded with the same base software. It's the ideal situation for commercial "grid computing".

So why is nobody selling this? Because there's no market for it. There's no real commercial market for supercomputer time, distributed or otherwise. Once upon a time, from about 1960 to 1980, there were engineering computer service centers, where you bought time-sharing service on big mainframes. Control Data and UNIVAC were the preferred machines for this. But that business is dead. CPU time became too cheap.

A well-known commercial grid was Gateway Processing on Demand, announced in late 2002 with great fanfare. Gateway offered "grid computing" on thousands of Gateway-owned machines. They quietly dropped that service some time last spring. Their former CEO admitted that it generated "not a lot" of revenue. Basically, it was an attempt to generate some revenue from Gateway's unsold inventory of machines.

Grid computing is one of those schemes where all the interest is on the sell side. Nobody wants to buy it. "Micropayments" and "portals" are like that. They didn't sell either.

Slashdot Mirror

Grid Computing: Conceptual Flyover For Developers

23 of 79 comments (clear)