Slashdot Mirror
Xgrid Clustering Software and Demo
no_demons writes "Along with a selection of other goodies, Apple also unveiled their Xgrid clustering technology from their advanced computation group today. Xgrid can turn a number of networked Macs into a supercomputer, detects nodes automagically via Rendezvous, and can run in or out of a screensaver mode. You can download a technology demo (including a BLAST test app) here."
Does this work on the small scale as well, like OpenMOSIX? We have a few G4's at school that could benifit from clustering.
You can't judge a book by the way it wears its hair.
What's the cost? I can't find any info on pricing, etc. Sounds like a great product if it works as advertised.
But for me, the model I want is a broker model. I want to sell my processor time to a broker who will resell it on a day to day basis to whoever is the highest bidder. E-bay of grid computing, ya know. I don't want to pick projects, download clients, etc. I just want to pariticipate (i.e. make money) from whoever is willing to pay the most at any given moment.
And when I feel like it, I'll volunteer x% to non-commericial stuff like SETI@home.
"If you want to improve, be content to be thought foolish and stupid." - Epictetus
Why limit this to Macs only? Wouldn't this work even better if it were cross-platform -- like many other distributed computing solutions such as SETI, distributed.net, and the UD Cancer projects.
when the G4 was out the concept of a supercomputer was still the old how many gigaflops it ran, at that point the G4 was considered a supercomputer because it had just broken the old gigaflop barrier.
but with that Apple while advertising it was a supercomputer secretly wanted that concept to change cause it ment they couldnt sell the computer to a LOT of different countries due to export laws and supercomputers (the same thing happpened with Japan and the PS2 I think, or it might have been a urban legend in that case) Intel and AMD where also in on it since their chips ALSO where breaking the gigaflop barrier and where technically classified as "supercomputers" based on old 70's classifications.
Since then the rulling was changed and it will be a long time before any of the makers will reach the current mark with a one or two prosessor machine.
Now in Virginias case, it really IS a supercomputer, actually the third fastest at this point and THE fastest educational supercomputer
before I think the G4 was like 6 or 7 gigaherz slower than the closest "true" supercomputer
"Slashdot, where telling the truth is overrated but lying is insightful."
It's been a while since I played with Beowulf clusters, but I've been thinking of using a Beo-class cluster in an enterprise app. The problem there is that you want the IT people to have to do as little "deep magic" work as possible when it comes to grids. So, auto-detection of new nodes, transfer of jobs out of failing/crashed nodes, etc. are important features. Is there anything close to that for Linux? a few different solutions that integrate well together isn't out of the question either...
Oh good, you mean I can actually talk about XGrid now after signing an NDA over 6 months ago? :)
;)
We had the second installation of XGrid, the only other group using it at the time was NASA. I haven't had much time to play with it personally but we had our coop do some genetic sequence analysis using it and he was quite impressed. Plus the speedometer-like gauge measuring performance just looks soooo cool.
I don't know much about clustering, but maybe someone can fill me in. Would it be possible to run something like Postgres on a cluster of XServes? Would the clustering be transparent at the application level, so that any program could take advantage of the clustered resources or would specific distributed algorithm apps need to be written?
I'm hoping that I could just get a stack of XServes and run an OSS db on it for free (as in no extra effort required), but I'm guessing that's not the way it works.
I feel like a complete ass right now, but it's worth saying ...
I wrote VERY similar software last year (before I even knew XGrid was planned) that used Rendezvous/zeroconf to do distributed computing/grid computing that worked on FreeBSD and (at the very least) compiled cleanly on Linux. I was convinced at the time that nobody would ever use it, because things like shared library dependencies make normal computing such a bitch that trying to solve those issues in these types of scenarios would be just silly... so I stopped working on it short of actually releasing the code.
If I find the code in the next few hours, I'll post another comment. I fear that it's entirely lost, due to unrelated stupidity.
Here is another wonderful example of Apple beating Microsoft on more than one level.
.......... and now Beowolf clustering out of the BOX!!!
1. Over all Coolness AND Geek Factors.
2. Usefulness of coolness factor.
3. Ease of use of Geek factor.
Microsoft hasn't done anything remotely like this in thier existance, rarely do they push the tech envelope beyond the what has become ordinary and benign. And when they do, they end up with over analized and engineered products nobody want (Bob?).
Apple of the otherhand is OFTEN making stuff WAY before the curve. Newton/PDA, USB/Firewire, CDROMs, Floppylessness (is that a word), standard networking
If Apple is smart with this, they could play against the new model of "ON DEMAND" services from companies like IBM.
Imagine a corporation that could automatically timeshare, timeshift computing resources based on such technology using the workstations they currently own/purchase. Peak periods of processing could be syphoned from little or unused desktops "On Demand", back filling any need left over from the DP center.
If this is NOT revolutionary it will be at least evolutionary.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
The whole point of the ban on export of supercomputers was that they could be used to simulate nuclear explosions
Not exactly, it was also their usage in brute-force decryption. The former point was indeed emphasized in the days of the old Cray supercomputer, but the latter was stressed as the most important factor in mid 1990's, when the aging cold war COCOM was replaced by the Wassenaar Agreement.
I wonder how long it will take someone at MIT or some other tech university to figure out if I put all the Macs located on or around a college campus I could or University could make it's self a mini-Super Computer for very few dollars.
:)
We did this 6 months ago when we were first beta testing XGrid. I never had the opportunity to personally play with it, but a grid of a hundred EMacs was a pretty good start to a mini-supercomputer I was told.
Isn't this just a batch queueing system, much like Sun's Grid computing, Platform's LSF and the like.
If it is, it is different in nature than the often joked about beowulf clusters and the mosix systems.
I think its great that Apple has one too, the more ideas thrown into the pot, the better all of them become.
-Tim
-I just work here... how am I supposed to know?
To the rest of the bunch when one of the apples goes bad? *rimshot*
Seriously though, I would have seen this as bigger news if it would be possible to afford more than one apple on my wages. Anyway, I figure this would be an appropriate thread to rant about my friend. He's always hounding me about how hot the macs are in the studio, and that they use protools on these mac servers and trying to explain to me what "better quality" audio they produce, and how all the studios are using macs now.....
You should check out Pooch from Dean Dauger. He has adopted OS X for obvious reasons but older versions of Pooch ran on MacOS 9.
Pooch was the original easy to use and implement solution for clustering Macs used originally at UCLA for physics modeling.
Visit Jonesblog and say hello.
A code snippet from here(!) reads:
/* disable conditional is constant warning */
#ifdef _WIN32
#pragma warning( disable : 4127 4706 )
#endif
Why would developers at Apple ifdef for Win32?!
Are they? A G5 server starts at $3000. A high powered Linux server is about $1000, and you can get a low end Linux server for a couple hundred dollars.
It's pretty well accepted that Apple charges a premium for their products. They're well designed, so sometimes it's worth it, but it seems hard to justify it for something like a server farm where design doesn't matter much.
From Apples Website: http://www.apple.com/acg/ Xgrid Xgrid clustering software is intended primarily for scientific researchers who have a set of networked computers that are not yet being used to full collective CPU potential. The software provides a remote execution environment and file staging abilities that coordinate the running of tasks on distributed computing resources and ensure that each computer has access to all of the files necessary to execute the tasks, thus freeing both user and developer of such resources from this tedious work. Xgrid does not solve all clustering problems. It does not replace clustering software such as MPI or clustering hardware such as InfiniBand. Xgrid does not accelerate or "grid-enable" existing applications on your computer. For an application to take advantage of the Xgrid technology, you must update it to use the Xgrid APIs. However, if you currently run long computations in your Terminal windows using, say, an already-compiled executable, you should be able to use Xgrid out of the box to run batch jobs for this executable on your clusters. In this way Xgrid offers high-throughput computing with minimal development work on your part.
Considering the kernel for which Xgrid was written, though, is it unreasonable to consider that a person might be able to work with a custom, stripped down compile of BSD? Granted, there would have to be a few necessary X modules, but what I'm wondering is if a person would have to go your route (writing the trick bios, etc.) or if it would be easier just to start with BSD and build from scratch?
Damon,
http://actionPlant.com
The guys at Genentech must be having heart spasms today. They do a lot of Altivec-optimised calculations, and between XGrid and the G5 XServes they're in hog heaven right now. Any Mac-equipped research facility is now sitting on an extra supercomputer they don't have to pay for.
Imagine that, an extra supercomputer.
Lockheed and Boeing are excited. Everyone in biotech is going nuts. Everybody's IT department should be thinking: what could we do with a crapload of extra CPU ?
Veteran, Bermuda Triangle Expeditionary Force, 1992-1951
But to do it campus-wide makes many admins very very nervous; the cost of maintainance and upkeep may in fact rise, and even if the secretaries aren't using their P4's with hyperthreading most of the time, the admins don't want the hassle of keeping them running for the secretaries and running the distributed software at the same time (I've seen proposals to do this, and seen the results). I personally think it makes a great idea, but usually--since those proposing it tend not to be the first and second-tier support providers--it gets shot down.
Coincidentally, I was working on setting up Sun Grid Engine on a couple of G5s here at work yesterday when I heard the news about XGrid. I dropped my work on SGE immediately.
It was a little confusing at first to set up, but I eventually got 3 machines total configured as part of my grid - 2x2GHz G5s + 1x867MHz G4. Perhaps if I read the instructions, I would've better understood some of the terminology (agent v. controller v. client, etc.).
The tachometer is sort of flaky. Sometimes it's stuck at zero on one machine while it it is actively moving around on others. Other times it's stuck at some non-zero position. Opening up 2 tachs on the same machine (XGridBlast has its own tach) will show different speeds. Though I should have in theory 8.867 GHz total speed, I could never get it to go over the default 8 "red line"- I was curious if it would rescale once it exceeded 8.
The XGrid client (where you submit jobs) has some default demo type things (I've mostly been testing the Mandelbrot one as it runs in a continuous loop), but it also has a way to build "custom plug-ins" which allow you to submit arbitrary jobs. In other words, executables don't have to be modified per se. Of course for any kind of parallel execution, they do, but if you need to to run 1000 iterations of the same command with slightly different arguments, then it should be able to distribute that "run" pretty well. The GUI for building up such a run is pretty easy to use.
One potentially big issue I've noticed is that once you submit a job, you have to keep the XGrid app open until it is done. For a run that may take days or weeks, I think this could be a problem. I'd like for you to be able to submit the job, quit, and log out (or shutdown the client even) then come back later to check your results.
Also, there doesn't seem to be a queue manager where you can see a list of jobs and their states.
I think that for any file-dependent commands, you need local copies of the files on each Agent node. At least that's how it appeared from perusing the documents for XGrid Blast - each node needs a copy of the database.
FWIW, we're using SGE successfully on a bunch of RedHat based servers, but though OSX maybe *nix, the installation and config was turning to be a pain.
Anyway, those are my initial impressions. I'm sure some of these things will change in the "release" version. In the meantime, I'll have to get working on getting some real work stuff to try out (I work in a university bio department).
-h3