Slashdot Mirror
Grid Computing Coming Of Age
ravenousbugblatter writes "The New York Times online has an article discussing grid computing and recent advances made by Dr. Ian Foster, among others. The article compares the state of grid computing over the internet to where the internet was in 1994, which was soon after the development of the software for the use of URL's, HTML, and HTTP. Predictions are made in the article that in the near future the massive power of grid computing will be available to anyone with an internet connection, not just to big companies that can afford to hire HP and Sun to run a grid project for them."
Its the 4th one, and getting better every year.
please consider setting up Grid Computing section! ...so that I can finally filter it!
Thanks in advance.
Welley Corporation - SLM Scammers
The article from NYTimes requires (free) registration.
Here is the registration free URL
Please use news.google.com for finding article links.
As a coder who works with things like md5 cracking programs (like the thingy in my sig) and various assundry other programs, I can honestly say: the crackers do NOT need any more processing power!
====
Crudely Drawn Games
>> Imagine a Beowulf cluster of these!
Wow! Imagine your karma going down!
Can anyone see another player apart from Microsoft having the market penetration required to make themselves the defacto distributed computing platform??
Go Google I say - let microsoft get someone else to beta test their software.
Q.
Insert Signature Here
Sure this is great if you're doing simulations or animating/rendering stuff . But for Joe Schmoe who surfs the web and reads his e-mail, what's the big deal? How will this affect network security?
Q.
Insert Signature Here
...that bastard Scott is right?
The network really is the computer?
Where'd I put that mousepad......
"It was the Woodstock of the grid -- everyone not sleeping for three days, running around naked and shagging in a kind of scientific performance art," said Dr. Larry Smarr, director of the California Institute for Telecommunications and Information Technology, who was the program chairman for the conference.
no wonder it took so long to develop.
I am a leaf on the wind
Teaching Computers to Work in Unison
By STEVE LOHR
Computers do wondrous things, but computer science itself is largely a discipline of step-by-step progress as a steady stream of innovations in hardware, software and networking pile up. It is an engineering science whose frontiers are pushed ahead by people building new tools rendered in silicon and programming code rather than the breathtaking epiphanies and grand unifying theories of mathematics or physics.
Advertisement
Yet computer science does have its revelatory moments, typically when several advances come together to create a new computing experience. One of those memorable episodes took place in December 1995 at a supercomputing conference in San Diego. For three days, a prototype project, called I-Way, linked more than a dozen big computer centers in the United States to work as if a single machine on computationally daunting simulations, like the collision of neutron stars and the movement of cloud patterns around the globe.
There were glitches and bugs. Only about half of the 60 scientific computer simulations over the I-Way worked. But the participants recall those few days as the first glimpse of what many computer scientists now regard as the next big evolutionary step in the development of the Internet, known as grid computing.
"It was the Woodstock of the grid -- everyone not sleeping for three days, running around and engaged in a kind of scientific performance art," said Dr. Larry Smarr, director of the California Institute for Telecommunications and Information Technology, who was the program chairman for the conference.
The idea of lashing computers together to tackle computing chores for users who tap in as needed -- almost as if a utility -- has been around since the 1960's. But to move the concept of distributed computing utilities, or grids, toward practical reality has taken years of continuous improvement in computer processing speeds, data storage and network capacity. Perhaps the biggest challenge, however, has been to design software able to juggle and link all the computing resources across far-flung sites, and deliver them on demand.
The creation of this basic software -- the DNA of grid computing -- has been led by Dr. Ian Foster, a senior scientist at the Argonne National Laboratory and a professor of computer science at the University of Chicago, and Dr. Carl Kesselman, director of the center for grid technologies at the University of Southern California's Information Sciences Institute.
They have worked together for more than a decade and, a year after the San Diego supercomputing conference, they founded the Globus Project to develop grid software. It is supported mainly by the government, with financing from the Department of Energy, the National Science Foundation, NASA and the Defense Advanced Research Projects Agency.
There has been a flurry of grid projects in the last few years in the United States, Europe and Japan, most of them collaborations among scientific researchers at national laboratories and universities on projects like climate modeling, high-energy physics, genetic research, earthquake simulations and brain research. More recently, computer companies including IBM, Platform Computing, Sun Microsystems, Hewlett-Packard and Microsoft have become increasingly interested in grid technology, and some of the early commercial applications include financial risk analysis, oil exploration and drug research.
This month, grid computing moved further toward the commercial mainstream when the Globus Project released new software tools that blend the grid standards with a programming technology called Web services, developed mainly in corporate labs, for automated computer-to-computer communications.
Enthusiasm for grid computing is also broadening among scientists. A report this year by a National Science Foundation panel, "Revolutionizing Science and Engineering Through Cyberinfrastructure," called for new financing of $1 billion a year to mak
It's about time grid computing become of age. I've been waiting to hit that for years.
Sincerely,
A dirty old mainframe
Consensual sex is boring.
We did study on size of infrastructure of Internet for demographic studies and thanks to partners at Stanford, found that Google setup now is bigger than the whole Internet in 1995 in terms of machines and total bandwidths. Grid computing definitely works..
-- Dr. Fu Ling-Yu, Internal Technology Consult; Tongji University, People Republic of China.
What would be the difference between Grid Computing and then a "psduedo-cluster" networked together with VPN Tunnels?
From the article they seem to be basically the same thing. Or am I wrong?
Error 407 - No creative sig found
no more upgrades every 3 years !
This is great if you think it's great. Grid computing is a technology without a cause right now. It's preposterous to think that the average joe, or even the average joe company, will have any use for grid computing in the forseeable future. Most of us can't keep our load average above 0.1 (that's 10% for you Windows-users) doing anything useful as it is!
Heck, look back over the grid computing stories we've seen here on /. Whose name keeps popping up?
SKYNET LIVES!!!!
Tsk tsk...what a blatant example of a violation of DMCA right here on /. This so misrepresents the law-abiding nature of the /. population ;-)
h-t-t-p-colon-slash-slash-slash-dot-dot-org
For more info on porgramming for grid computing try MPI or LSF/NQE/PBSPro
Rus
Cheap UK and US VPS
With the upcoming PS3 carrying out grid computing, there's no stopping for this technology reaching out to the masses, even those who don't know it!
h-t-t-p-colon-slash-slash-slash-dot-dot-org
The computation performed by Seti@Home is what Grid researchers refer to as an "embarassingly parallel".
Among many other things, Grid folks hope to solve problems that aren't quite so amenable to divide-and-conquer. But then they had to go base their protocols on the bloated Web services stack, implying a relatively high granularity per compute unit. So we'll see how well that works out!
I prefer the term distributed computing, why did distributed computing turn in to grid computing?
Oh, my god...least insightful post, ever. Do you have any idea what grid computing applications are? I've actually met Ian Foster, as I did some work on Globus for a semester. They're talking about moving around petabytes of data for analysis and storage. You have no idea the complexity of these issues. For instance, there's a program called the Digital Sky Survey...i'm not going to bother looking for links because right now I'm using my trolling account and I don't feel like logging in as my "serious" persona and I don't care about karma. Anyway, massive telescopes around the world pan every miliarcsecond of the sky, record their images, and then put them in the survey for view by researchers all over the world. Each image of each fraction of an arcsecond of the sky takes up megabytes. Combined, the map of the sky is petabytes in size. That's 10^15, bucko. You try just storing that data, let alone analyzing it, searching it, and recalling it, using fucking seti@home style tech.
There were several other applications under development...I remember a paricle collider of some kind that produced some obscene amount of data for each event. Something like a terabyte milisecond or something, and it was almost impossible just to gather the data...again, let alone store, analyze, and recall.
Seti@home? I scoff. I hope you choke on your ignorance.
I just recently heard that here at Virginia Tech we are getting a (massive?) grid comprised of some of the first dual-processor G5s rolling off the assembly line at Apple. The number of machines? I believe it was in the 1100 range.
Thus, if you like grid computing and want to do some research as a grad student or whatever, this might be the place for you.
I thought 1960s timesharing was nothing more than job submission and process load balancing run on supercomputers. Grid computing can involve 1000s of distributed nodes with millisecond (ethernet) latencies between them, that's not timesharing as far I know. The wheel isn't being reinvented here.
If you wanted to do this right now, you could cut a deal with a mid-range ISP. Buy an account on every server for use only during off-peak periods, run standard clustering software, and crunch all night. Run on a server farm with large numbers of identical machines interconnected with massive bandwidth. A true Beowulf cluster application.
Nobody does this. That's an indication there's no market for commercial "grid computing". Clusters, yes; reselling computer time, no.
Remember "push technology"? "Micropayments"? "Grid computing" will go the same way.
As for "peer to peer" systems, bear in mind that without copyright problems, music distribution would be trivial and cheap. Just put each new song out on Netnews. Netnews is far more efficient than any of the peer-to-peer systems. The music industry only generates a few tens of megabytes of new data per day, after all.
Timesharing is multiple people/jobs connecting to one mainframe or computer and "sharing" the useage of that single computer. Grid computing (aka distributed computing) is sharing one job over multiple computers. Totally different concept, completely different target audience.
now how did it go again? Ah, thats right....
*points and laughs* hahahahahahahahahahahaha
(I have mod points, but there is no -1 WRONG.... so I decided to take the piss instead
IBM, as well as other companies and organizations, is working on the Globus project and there are different scenarios out there. One of which is online gaming (butterfly.net). There are others, but right now, it's mainly scientific based. For the people who know, the Globus toolkit just reached version 3. This is important to know because this version is OGSA/OGSI (in draft) compliant which is an open standard describing the communication between the grid nodes (WSDL, WSDD, XML, etc). The grid is different from clusters and the grid is different from p2p computing. One view on the grid is use of remote resources. For instance, you can use en electron microscope remotely. Perhaps even with the DaVinci surgical machine, it would be possible to perform (minor) surgery remotely. The advantages of this are obvious. A specialist can help more people since it cuts on travel time. In my view, the grid cannot be applied to a certain solution meaning the grid isn't supposed to be for a certain problem, but rather, a new avenue to do things. With this in mind, the grid will grow according to how we think of using it. The Internet is an example of this type of growth. Furthermore, the grid is probably geared towards businesses and other backend operations. Later, perhaps, it'll become more of on online service directory in which you can find resources to do you work; printing facility, specialized resource use (super/quantum computers), and other things. Again, the grid will grow will become what we need it to be (even if we cannot predict it).
All aside, it's exciting technology, not to mention that the Globus toolkit was named on the of the top 10 techs that will change the world.
For those interested in security, the Globus toolkit involves an asynchronous certificate signing method initially and then move onto a synchronous method for better performance. The Globus books and papres call this a PKI scneario. (I'm not a security guy)
Also Globus is not the only grid tech out there. Seti at home is one (and their derivatives). There are also distributed storage methods in which you can send data onto the grid and it'll be there (somewhere safely tucked away).
fun stuff!
Also, Globus is open source and the services are written in Java. I just love this stuff.
I really think the people complaining about not personally having a use for grid computing are completely missing the point. As long as enough people have a use for it, it will be useful. Having done a good number of calculations on a few different supercomputers, I can think of nothing that the grid currently offers to me...but I'm sure the people who run many-hundred processor jobs on a regular basis have a different perspective. For a while, the grid might be the plaything of big scientific and industrial computational projects, but has any technological advancement like this ever not caught on. Eventually, someone will figure out a new idea, only possible on a grid, which involves porn, gaming, or the ability to transfer media files in a manner of questionable legality, and soon kids will be asking what life was like without it back in the dark ages. A little patience, people, give the geniuses and madmen (not necessarily mutually exclusive) a little time to work...
I have been watching the developement of one such application: Gled , "a hierarchic server-proxy-client-viewer model written in C++ and offering a mixture of object oriented framework and toolkit" (says the project homepage) and I can say that it looks a lot more like a Quake window to a programmable scene made of very complex object collections, running on multiple systems (and with multiple users) with a GUI to its underlying cluster systems, than a Seti@Home screensaver.
My personal favourites are the autogenerated code, the autogenerated GUI and the object brokering facilities over the clusters.
The trend of Grid and Grid-like cluster computing is, IMHO, going in the direction of better viewing facilities, more interactive software and higher-level interfaces, where the underlying grid can be thought of as a piece of iron, a strange dynamic multiprocessor arhitecture with impossible latencies.
Links:
-Kvorg
Clustering is to Grid (or distributed)
as
LAN is to WAN
or am I just fooling myself?
He tried to kill me with a forklift!
Just go to this site http://www.majcher.com/nytview.html save the page to your desktop open up the saved page and click the register and go icon (I remember hereing that NYT.com wised up and blocked traffic from this link) the links is also here
"Grid" computing has been around for decades. People have been doing useful work with arrays of systems for a long time, both in the commercial and academic areas.
Government of the people, by corporate executives, for corporate profits.
I've been working in the Grid Computing area for the last two and a half years, and would like to make a stand for all of us who aren't just worried about bigger supercomputers.
Supercomputers are great, but the number of big computing problems that can handle being run on distributed groups of supercomputers is small. That's why things such as the Earth Simulator and the ASCI programme still exist - sometimes it's just better to build a bigger box!
Where Grid Computing might take off in the science and business mainstream is collaboration and sharing of resources. In particular, I work on producing middleware to try and share and unify data resources. In the astronomy community for instance, they have spent many years standardising the naming schemes for their databases and as a result, projects such as Skyserver and SkyQuery are becoming possible. Now consider the bioinformatics field: hundreds of competing standards for naming things as simple as gene expression ids. Grid computing should provide some of the tools to make knowledge extraction from the many disparate scientific databases possible.
This has applications in business, and it's something we're already seeing in the uptake of Web Services. One recent Grid Computing initiative - Grid Services - is pushing the boundaries of Web Services, and extending them to standardise functionality such as state and lifetime management which should make them more useful for the kinds of collaborative problems which are cropping up in both business and science.
For instance: a car manufacturer has an agreement with different suppliers of airbags - obviously information exchange must take place to ensure safety of the passengers, but both the car manufacturer and airbag supplier will not necessarily want the other to be able to see all data for their parts, just use it. As suppliers change, the manufacturer must ensure that data is properly traced and expired. This is not much different from scientific collaborations, financial collaborations or even network gaming where we have a huge number of swiftly changing, transient resources.
It is these problems of dynamic collaboration and maintenance of resources that Grid Computing may eventually solve.
Use those unused CPU cycles in the search for disappearing African Uranium!
when is bandwidth going to be so available/cheap that this is going to matter?
Just in case someone's not familiar with parallel programming terminology, embarassingly parallel means that each operation is independant of any other, so it can scale virtually perfect. The classic example:
1 women * 9 months = 1 baby
9 women * 9 months = 9 babies
9 women * 1 month != 1 baby
Putting 9 women on the baby-making task for 9 months is scales the baby-making operation, but since it's embarassingly parallel, it doesn't speed up any one operation (each baby still takes 9 months, but in those 9 months, you can now get 9 babies).
Another example (I made up the 10 factor):
1 worker * 10 days = 1 car
10 workers * 10 days = 10 cars
10 workers * 1 day = 1 car, because they can work on seperate things at the same time.
Of course, in reality, they can't quite work on things at the same time (steering wheel can't be installed before the body is welded together), so you have operations that block, leading to the steering wheel guy sitting around doing nothing, but you get the idea. This, of course, is where pipelining comes in...
-- Is "Sig" copyrighted by www.sig.com?
I fully agree with you 100 %. And tell me about it. And what do you think of this site . Random NYTimes.com Registration Generator http://www.majcher.com/nytview.html
You go here and it fills the NYTimes registration page with random characters. Maybe it might come handy some day, but, oh god, you must be pretty angry about it. Yeah. And tell me about it. I agree with you fully 100%.
To see a world in a grain of sand, and then to step back and see the beach where the sand lies
As a student I have my disk space regulated, am I soon to have my CPU time regulated (this used to happen). I don't know whether I would end up with more CPU time available to me for a project or less?
Mmm, your example is incomplete, no ammount of woman can produce babies by themselves...
/.'er * 9 months = 1 harrasment suit
1 woman + 1 man * 9 months = 1 baby
1 woman + 1
etc.
"Semper in excretum set alta variant"
The "Grid" prescibes a standard way of connecting distributed computers for large-scale computing. Sort of like the InterNet didnt really take off until more than 20 years after its inception, when http and browsers became standards.
Yes, the Grid can provide true parallelism, but it doesn't just do that by itself; the programmer has to divide the job up into tasks and distribute them. What the Grid provides is the framework to make that possible, and the security infrastructure to make it more likely that the owners of compute servers will allow other people to use their resources.
The computing model really is the mainframe one: you submit jobs to a queue and they get run and come back to you. You don't care or know when or where they get run. This model may be ancient, but it's how much of "business computing" is done.
Yes yes it's off topic, but after having seen T3 it has been bugging the hell out of me that at the end of the movie John said that there was no central computer for Skynet, but instead it had spread into thousands of machines on the internet. Isn't that fantastic!
And then I thought, well, if Skynet is actually a distributed computing app running on computers on the internet, then, when Skynet decided to blow up all of the cities, wouldn't it also be destroying the majority of its compute nodes and thus massively diminish its capacity. It would, after all, be living in the same cities as you and I. Not to mention that computers rely on the same infrastructure (power grids, telecom) as we do.
Ok, so I am anal, but a plot hole large enough to launch the shuttle through does catch my attention and drive me crazy for a while.
Mmm, your example is incomplete, no ammount of woman can produce babies by themselves...
1 woman + 1 man * 9 months = 1 baby
1 woman + 1
Ah, true. You see, very few people on
-- Is "Sig" copyrighted by www.sig.com?
I don't get it...the man sits around for 9 months and then a woman comes along and poof a baby is born?
(1 woman + 1 man) * 9 months = 1 baby
Order of operations man! Gotta get those parenthesis right.
Trolling account? If you didn't care about karma you would not have posted as anonymous coward.
I can't believe you're too chicken to flame without your "real" account. The fact that you can't do it with your trolling account either is just pathetic.
I'd have to remind you that statistically, 36 is a very small number for a strategist. Any military leader would consider it minimal acceptable losses.
That comes from the cold side of me. The half that occupies my heart says that while I agree with you that it is a tragedy, your trolling is amatuer and smallish. Reminding people of the circumstances and results of war does nothing to make anyone's grief easier and places no strain on the leaders that make such horrible decisions. Your words are better spent lobbying and writing for your congressman. Not on Slashdot with offtopic agendas.
It's rather disconcerting to see someone with good intent waste it on such obviously poor application and execution. Atleast when I'm off-topic, I try to elicit a discussion. But I do so love flamebaite. It taste so good in the morning.
"Last one in is a rotten goblin!" - Kepp
If you are modeling weather and divided your target regions into small sectors, the variables in one sector does affect the next. As calculations are made, they need to be stored and shared with all computers in the grid if they need it. That's alot more problematic.
Well, there's spam egg sausage and spam, that's not got much spam in it.
While I do understand the baby making process quite well. I am not actively pursuing it as an option right now.
Thus far I have quite happilly stuck it out at step 1.
Sadly it seems that's about to end. My girlfriend of two years left me this morning....
life's a bitch.
"Semper in excretum set alta variant"
The car example is interesting, in that 1 worker will take
much much longer than 10 days to make a car, although
10 workers can easily make a car in 1 day, because no
one worker has the easy facility with all 10 of the distinct
production roles in producing a car. This is counter to
the argument of the skeptic of parallelism who claims that
Amdahl's law makes parallel computation inherently
inefficient, and points to what may be the single most
appealling aspect of the "Grid" paradigm for distributed
computing, to wit:
As opposed to clustering or even p2p, "Grid" is (albeit in a
vague and unfocussed manner) begining to amalgamate
diverse and disparate resources which can be used to
construct productive pipelines in which each contributing
system's peculiar strengths are more efficiently utilized
than is possible in a homogenous system, where the
imbalance in the resource uses of a particular application
almost *never* matches the imbalance in the resource
supply: The Grid is subsetted so that the global system
implementing a production pipeline is efficiently utilized,
or, to flip the coin, the intrinsic underutilization of
an inefficient pipeline construction does nothing to
harm the grid -- it only means that the remaining
unutilized capacity is available to other dynamic subsets,
to be utilized in other production pipelines.
-I like my women like I like my tea: green-
Man is optional.
-I like my women like I like my tea: green-
I don't get it...the man sits around for 9 months and then a woman comes along and poof a baby is born?
(1 woman + 1 man) * 9 months = 1 baby
Order of operations man! Gotta get those parenthesis right.
Not exactly, because you really only need the man for a few seconds (though I pity the guy who's that quick). So perhaps:
(1 woman + 1 man) * 30 seconds + (1 woman * 9 months) = 1 baby?
Who knew sex could be made so boring?
-- Is "Sig" copyrighted by www.sig.com?
You could have read Ian Foster writing about grid computing in April's Scientific American. If you can't find a copy, try sciam.com