Slashdot Mirror
Grid Computing at a Glance
An anonymous reader writes "Grid computing is the "next big thing," and this article's goal is to provide a "10,000-foot view" of key concepts. This article relates many Grid computing concepts to known quantities for developers, such as object-oriented programming, XML, and Web services. The author offers a reading list of white papers, articles, and books where you can find out more about Grid computing."
And with this change in computing comes another challenge. Not every company has applications that would benefit from distributed computing, but many do. The challenge is making a secure environment that will allow Company A to send their data *and* the software to process that data down the pipe to Company B for processing, meter the usage, and charge back the service. From what I have seen, no farm is really ever utilized 100% of the time, but there are crunch periods where something has to be simulated within a certain timeframe and the existing throughput on hand is not enough. It is those crunch times where you could really use a few trillion spare cycles.
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
1. e-mails with "EARN $$$ DOING NOTHING" ...
2. spyware that not only spies but also hijacks your CPU cycles for remote computation
3. dubious companies selling "grid computing" service pop up all over the place
4.
5. Profit?
It may look funny, but what if the next version of Windows comes embedded with this kind of thing? All it would take would be some marketing genius to convince enough people. (disclaimer: yes this is slightly paranoid, it's not intended to be MS bashing, just an example on how this technology could be misused).
The ENIAC Demo Competition
It really could be one of the next big things, considering the advent of Object oriented methods of handeling information it realistically could be a viable Object oriented model.
With The realitivly recent move of object oriented Programming, you could think of this as just the next level of abstraction, abstracting your objects out to a broader system level as apposed to an implemenatation level.
In Any event it would be a good scheme for many things that need distributed systems....such as cryptographic reaseach and other things that need to be distributed
Sun is heavily involved in Grid computing. They provide free multiplatform grid software (including for Linux), case studies, white papers, etc.
They also host an open source project Grid Engine for the software. The software used to be commercial, but Sun bought it and open sourced it, like they did with Open Office.
sounds a lot like good 'ole fashioned SMP to me, with a lot more disk space. As we all here know, not all computer-related tasks work well in a multi-processor platform, and as someone who has played with SMP programming, it certainly adds an order-of-magnitude level of complexity to try to harness the full power of SMP in your code. Compilers help, but not much...
never bring a twinkie to a food fight.
6 Years ago we were clamoring to use the desktops on the trading floor to run some of our financial models at night. We tried MPI and CORBA, and kludged together a workable (although lacking) solution. I can definitely see where a hodge podge solution like that needs to be improved, and it looks like the grid concept is looking to fill that gap, but at the same time the desktop is evolving. The Net PC seems to have gone the way of the DoDo, and while it is true that there are plenty of idle desktops during the evening, I would say most of the servers are well utilized by the departments.
The real benefit would be pulling in all that desktop power, but I do not believe desktops will remain as they are. With mobile work forces, and reshaped IT departments, workers are more likely to move about the company and form resource pools. In order to do that, they will need to be productive as soon as they set up shop with a new team. Current infrastructure makes that difficult to manage. The more successful implementations have those workers using laptops, which go home at night. Goodbye spare cycles. Future concepts seem to be brewing where you leave the peripherals and just carry around a small PDA size CPU+storage and plug that in at any station and your set to go. In that scenario the spare cycles are walking around with you. I only see limited use for consolidating existing server processing. There are already plenty of technologies that address that need. Not sure I buy this as "the next thing", but I guess the next short term thing maybe.
Anonymous Coward
I have given a lot of thought to this concept in the past and, although I think it has a lot of merit I also think it will require a different underlying software architecture than any of those we use today.
Currently for distributed computing we have Thin-Client/Fat-Server, Client/Server, N-Tier and Shared-Node architectures. I think most people are expecting a Shared-Node or Client/Server for Grid Computing because that is how existing implementations work. The issue with either of those is the size of the work unit. If the work unit is small than the nodes/clients must sychronize often. If the work unit is large then you are more likley to have nodes/clients in a wait state because required processing is not completed.
Using a network style architecture (distributed Shared-Node) raises more issues because of message routing. Interestingly, this is the 'web-service' model! For example a web site must verify a customer, charge her credit card, initiate a shipping action and order from a factory in a single transaction. So you get four sub-transactions. Let's say that each of those initiates two sub-transactions of its own and each of those initiates one sub transaction of its own. We now have a total of twenty transactions in a hierarchy that is three deep. Let's also assume that we only have one dependancy (the verification) before launching all other transactions asychronously.
The problem here is response times, they add up. if the average response time is 500 ms, then three transactions deep gives us 1500 ms. The dependacy, at a minimum, doubles this. So it takes three full seconds to commit the transaction. Something a user might be willing to live with until a netstorm occurs and the response time drops to thirty seconds or more. (Note: Isn't it funny how you never see this math done in the whitepapers pusing web services?) But three seconds is far too long for sychronizing between nodes of a distributed computing grid unless you only have to do it every once in a great while, pushing us towards large work units and idle nodes!
So the Internet itself imposes costs on a distributed model that wouldn't exist on, say, a Beowulf cluster because that cluster would have a dedicated high-speed network. Client/Server architectures work better for the Internet, but require dedicated servers and a lot of bandwidth to and from them.
I believe the real answer lies in what I call a Cell architecture. This would require servers, but their job would be to hook up nodes into computing 'cells' consisting of one to N (where N is less than 256?) nodes. Each node would download a work-unit from the server appropriately sized to the cell, along with net addresses of the other nodes in the cell. Communication would occur between the nodes until the computation is complete and then the result would be sent back to the server. When a node completes its work unit (even if all computation for the cell is not complete) it detaches and contacts the server for another cell assignment.
By reducing cross-talk to direct contact between nodes within the cell we allow smaller work units. By using a server to coordinate nodes into cells we are allowed to treat the cells as larger virtual work units.
Comments?
- -
Are you an SF Fan? Are you a Tru-Fan?
First off, this stuff has been completely mainstream for over 30 years now. The only thing new is that it keeps getting renamed, This year it's called GRID. I remember when it was called timesharing, and Time magazine had cartoons depicting it is 1973.
The entire GRID standard actually only covers the data transfer and login. Becasue that's the only thing standard about the different types of hardware. You still need to write the software specific to the hardware. Even with tools like MPI programming for Sun big iron is nothing at all like IBM big iron. And you dont exactly use Java. The value is not in the software - that's why it's getting standardized and is given away for free. The value, as always, is in owning a huge pool of computing power and renting it out, or even better, selling it in racks full.
The only people benefiting financially are the people that make the hardware - IBM, HP, Sun, Fujitsu, etc. Just like 30 years ago. Open Source has completely devalued the software - why pay for that, money is better spent on more hardware.
Then there is the cost of transporting the terabytes of data involved in the types of problems you do with these systems. Transport costs are more then the computing costs in many cases - another reason that part got standardized.
Hardware costs are falling FAST. Blade mounted and racked CPU are running about $500/Ghz ($7k for the same from IBM). That means for about 1 million you can get something like 2K CPUs and 2Thz of power, running Linux and all the tools you need. Thats a lot of FLOPS.
For those kinds of costs, outsourcing it at seems silly. You still have to do all software development, data transport, post-processing, and research yourself anyway, and those costs DWARF the hardware/electricity/HVAC costs of owning the hardware and having exclusive access 24/7 until the next updgrade.
- Adam L. Beberg - The Cosm Project - http://www.mithral.com/
I've seen entirely too many articles (such as recently appeared in SciAm, and now this one appearing on /.'s FP) giving the
"10,000-foot view" of grid computing.
I've seen a few articles giving the 10-micron view, describing CPU architectures making use of a grid topology.
I've seen a few small demos of massively distributed clusters. I've heard hype about the idea of a service provider and service consumer oriented topology. I've heard about self-healing networks. I've heard about the PS3 making use of a grid-based system.
I have not heard any of the "step 2s", the means by which we transition from individual PCs accessing a network, to a single shared "grid computer" actually composed of the network. At least, nothing that would make the resulting network noticeably different than the current internet.
For individual systems (ala the PS3), grid computing seems like possibly the next big thing, sort-of an evolution of SMP to a scale larger than dual/quad CPU systems. The rest of it, the over-hyped massive "revolution" in how people use computing resources in general? Pure marketing hot-air, and nothing more. The closest we'll get to the idea of a worldwide "grid" will consist of an XBox-like home media console with anything-on-demand (for a fee).
Here is a large Grid project that I'm working on.
from what I understand they did this at squaresoft during the makeing of the final fanstasy movie. indle cycles were used by rendering during the day when the cpu was not floored at 100% useage. Atleast from what i understood from the articles that i read about it. As well as workstations were tied to the dedicated render farm at night. Since many of the artists had more than one machine at there desk it proably worked out well both at night and during the day.
How 'bout this - build in a system whereby users who have downloaded a file can mod its quality up or down. Then while searching the network, you also get MD5s for the files, and the associate rating is accumulated from others who you search. This way crappy files float to the bottom.
There are a thousand forms of subversion, but few can equal the convenience and immediacy of a cream pie -Noel Godin
A lot of these concepts are what we are waiting for. We have a server farm that is metered by resources available from license servers, but the data is geographically separated so licenses available at one site may not be available at others. Technologies that allow reliable data transfer (NFSv4?) might enable this, but it also needs to be calculated whether the amount of time it takes to transfer the data will be longer than it takes to process the data. Not all our sites have multiple T1s so it may be cheaper to just boost the size of those lines vs buying $100k licenses. Total grid computing sounds very possible when you talk about breaking the job down to "Add register 1 and register 2" but the task of breaking the job down to that size and transferring to a remote system canh take so much longer than actually doing that locally. Some larger granularity will be needed to make it efficient to transfer a job remotely.
As I mentioned in my post, "secure" handling would be the first requirement. Security and encryption must go without saying. In order to further ensure security, a job must be as widely spread as possible. If I split an aerodynamic simulation in encrypted fashion across 100 compute farm services I am much more secure than if I did the same thing with a single service.
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
I've been writing (partially for my CompSci Masters thesis) a new Grid-oriented application that may be of interest. It's called GridShell, and aims to provide a Free/OSS interface to any and all Grid technologies. Currently, GridShell's skin-able web-based UI (WebUI) is almost completed, able to provide the equivalent of an expert Grid administrator/user through a very "clicky-clicky" frontend.
Oh, and it's all 100% Object-Oriented Perl, for those of you who care about clean code.
More really crazy GridShell modules are down the road, so check it out!
http://www.gridshell.org
-Will the Chill
Creator of RPerl, Scouter, Juggler, Mormon, Perl Monger, Serial Entrepreneur, Aspiring Astrophysicist, Community Organiz
Grid computing is pretty interesting, if anybody wants to find out more I have compiled a comprehensive list of references on the subject. As well as providing a brief (20 page or so) overview of the available Grid solutions. www.netinvasions.com/files/GRID/grid-paper.htm