Slashdot Mirror
Grid Computing at a Glance
An anonymous reader writes "Grid computing is the "next big thing," and this article's goal is to provide a "10,000-foot view" of key concepts. This article relates many Grid computing concepts to known quantities for developers, such as object-oriented programming, XML, and Web services. The author offers a reading list of white papers, articles, and books where you can find out more about Grid computing."
And with this change in computing comes another challenge. Not every company has applications that would benefit from distributed computing, but many do. The challenge is making a secure environment that will allow Company A to send their data *and* the software to process that data down the pipe to Company B for processing, meter the usage, and charge back the service. From what I have seen, no farm is really ever utilized 100% of the time, but there are crunch periods where something has to be simulated within a certain timeframe and the existing throughput on hand is not enough. It is those crunch times where you could really use a few trillion spare cycles.
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
Grid computing is not about making a giant computing farm out of a bunch of distributed machines.
...), and ideas are the important part here. having these "Grid" concepts built into every new technology (filesystems: NFSv4, security: Globus GSI, etc.) will allow these linkups, data transfer, and whatever we may awnt to do, to happen much more efficiently in the future.
see, that's the major fallacy of the hype behind "The Grid". yes, one of the benefits can be seen in the supercomputing realm, where you can link up many different machines (we haven't gotten to doing this between architectures yet, mind you) to make a gianto-machine.
however, the key in *all* of this is the technologies that allow for that to happen, along with the data transfer, authentication, and authorization, et al, that have to happen.
as far as cycles go, no, we probably won't see a dynamically created, scheduled, and allocated meta-supercomputer anytime soon. most companies will use these technologies to make static or mostly-static links between a few select sites and partners for now.
however, these protocols (GridFTP, ack), standards (OGSA,
to wit: the killer app in "The Grid" is not to make a giant supercomputer. it's to develop a lot of different ideas and technologies which allow for resource sharing (at the general level, among other things) to occur in a standardized, efficient, and logical fashion in the future. noone will use all of them, but the key is to use what you need from what "The Grid" encompasses. that's why it's referred to as "The Third Wave of Computing"!
You obviously didn't get the memo
I happen to know that beowulf clusters of quantum iPods, built by nanobots, running social software, using a Post-OOP paradigm and a journaled filesystem over a wireless IPv6 network to make profit with a subscription-based publishing model will be the next big thing.
Sun is heavily involved in Grid computing. They provide free multiplatform grid software (including for Linux), case studies, white papers, etc.
They also host an open source project Grid Engine for the software. The software used to be commercial, but Sun bought it and open sourced it, like they did with Open Office.
This is just an inverted version of the "network computing" universe where we all use thin clients that use a central server to do work. It can never become mainstream due to the physical limitations, not the technology ones. Suppose I am a corporation and I need a new big-iron system to process daily orders from our web site. Let's try grid computing: all 1000 employees in the company install a piece of software on their PC so we can use each PC to process an order, based on availability. The number of problems with this, as compared to using a central server, is incredible.
1) Still need a central server for storage/backup
2) One server needs one UPS, 1000 workstations...
3) Worsktations are flaky: They reboot, crash, play video games, etc. The distributed software can handle this, but the inefficiency involved is painstaking. I hope everybody doesn't run Windows Update all at once, or all the PCs could go down.
4) The corporate network is now a bottleneck.
I rattled off this list in about 30 seconds, so I'm sure there are lots more. Since these are physical limitations, not technology limitations, they aren't going away.
As we discovered early on in MIMD parallel computing, MIMD (aka grid computing) parallelism can only really help processes that are CPU bound in the first place.
Most of the processes that require 'big iron' are memory bound and I/O bound--e.g. databases that are hundreds of gigabytes to terabytes in size. This is why so many CPUs are '90% idle' in the first place, and this is why system designers devote more attention to bit-striping their disks, a good RAID controller, bus speeds, disk seek time and so forth.
Problems that require brute-force computation on small amounts of data, and produce small results, are simply few and far between -- and the people addressing those problems have been onto MIMD for decades. For instance, my first publication, in 1987 to the USENIX UNIX on Supercomputers proceedings, involved putting ODE solvers wrapped in Sun RPC, so that hundreds of servers could work on a different part of initial condition and boundary condition space, to provide a complete picture of the properties of certain nonlinear ordinary differential equations. Cryptanalysis and protein folding problems are already being addressed in a similar manner, and the tools to distribute these services as well as the required communications standards have been around for more than a decade.
Furthermore, if you've already got a marginally communications-bound domain decomposition of a parallel problem, and you want to cut down the communications overhead in order to take advantage of MIMD parallelism, the last communications protocol you're going to use is a high-overhead one such as CORBA, or a text-based message protocol such as XML. Both XDR and MPI are faster, more stable and better established in the scientific computing community than Yet Another layer of MIMD middleware--which is all Grid Computing is.