Slashdot Mirror
Swarm — a New Approach To Distributed Computation
An anonymous reader writes "Ian Clarke, creator of Freenet, has been working on a new open source project called Swarm. The concept is to allow a computer program to be distributed across multiple computers in a manner almost completely transparent to the programmer. The system observes the program executing and figures out how the workload should be distributed for maximum efficiency. Swarm is implemented in Scala. Its at an early-prototype stage, and Ian has created a good 36 minute video explaining the concept and the current implementation."
Just saying it outloud so we can start working on countermeasures.
.. was Mosix http://www.mosix.org/
It allowed mosix-running linux computers to distribute their loads over a connected other mosix-running linux computers.
Processes migrate to other nodes transparently. No programming changes were needed.
At first I thought they were talking about Swarm, a "attempt to gather up many different kinds of models that go under the heading of "agent-based modeling" and create a common language and programming approach." that I've worked with before. I'm surprised they went with the name of an established toolkit in another aspect of programming. Still, looks like a cool tool, another layer of abstraction to make distributed computing easier might make it more attractive to those that don't use it much at the moment.
"goodbye and hello, as always" ~Prince Corwin, from Zelazny's Amber series
Imagine a Beowulf cluster of... err. Oh.
It's so much cleaner, faster, higher quality and better to use than Youtube. Now if only they followed the trend and wrapped that Flash object in a <video> tag.
The thing that's always killed this idea (along with automatic parallelization even on the same machine) is that the overhead of figuring out what's worth distributing, and the additional overhead from mistakes (accidentally distribute trivial computations), often swamps the gains from the multiple processors banging away on it simultaneously. Determining statically what's worth distributing is very hard, since solving it properly is undecidable (basically equivalent to the halting problem), and even solving it in a significant enough subset of cases to be useful has proved difficult. It looks like this project is monitoring dynamically to determine what to distribute, which seems likely to be more fruitful, although historically that approach has suffered from the overhead of the monitoring (like always running your code with debugging instrumentation turned on).
I certainly hope he has a breakthrough vs. past approaches, or it could just be that advances in a lot of areas of technology have given him a better substrate on which to build things that naturally mitigates lots of the problems these things used to have (automatic parallelization research started probably ahead of its time, back in the 1970s, so that most academic stuff was killed off by the 1990s after no really knock-down results emerged). It's not entirely clear to me what the killer advance is, though. The particular variety of portable continuations? A good way of easily monitoring computations? Something that makes the data-dependency analysis particularly easy?
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
It sounds like a good idea, but I don't think the project is far enough along in this video to warrant a posting. Maybe he was using too much of a trivial example to be appreciated in the video, but his explicitly offloading the task to another computer doesn't appear to be very far beyond standard client server models. If it were already automatically transporting processing between different nodes, it'd be much cooler, but that is not a trivial problem to solve. Deciding what should and what shouldn't be distributed at the application level will be extremely hard I imagine. If the project were farther along in its maturity I'd be much more interested.
In Ian Clarke's Swarm, World "Hellos" you!
AT&ROFLMAO
If I understand what he says correctly, it is something like this: Distributing computation is hard, really hard. It's so hard that nobody ever did it properly. But Swarm will change this! How? Well, we don't know yet, there are so many interresting problems we have to solve first. And you can help!
http://fox.eti.pg.gda.pl/~pczarnul/DAMPVM.html (Dynamic Allocation and Migration Parallel Virtual Machine ?
So instead of R-ing TFA, I have to WTFV? Sigh.
The CB App. What's your 20?
The system observes the program executing and figures out how the workload should be distributed for maximum efficiency.
... this is unsolvable.
We cant even do this (efficiently) for an app running on a single core. How do they expect to do it when latencies are higher and throughput is lower?
Can they determine if the program will terminate or loop forever too?
Computer 1: MOV AL...what? No more? MOV AL what? I need a value! WTF am I supposed to do with that!?
Computer 2: 09? Nine? Who gave me nine on its own. That doesn't make any sense! Jeez! Hey, anyone out there missing some data?
Computer 3: Not me, I'm pushing the registers onto the stack
Computer 4: Nope, I've got an INT
Computer 5: Oh, hey, it could be me - does NOP have a value. No? Sorry, my bad!
Computer 1: Nine - yeah, nine - Well, I could stick that in AL if no-one else wants it!?
Computer 3: Oh, heck, give it to 1. I've just got a POP instruction so I am going to obliterate it anyway.....
AT&ROFLMAO
Did anyone else read the summary and think of Michael Crichton's Prey, and how this might apply (plus the use of the word 'swarm')?
I do respect Ian, but cant we do this with the existing language infrastructure and just extend it?
---- Booth was a patriot ----
Isn't that what the new vSphere or some up-and-coming release from VMware supposed to do?
-m
http://www.invisik.com
I think he should fix the monstrosity that is Freenet before he jumps onto other things.
Arash Partow's Philosophy: Be a person who knows what they don't know, and not a person who doesn't know.
Erlang apparently gets it right. Scales smoothly from single core to multi-core to multi-server in a near linear fashion. Astonishingly reliable, having achieved nine nines of uptime - much less than a second of downtime - in a year. Purposely designed to mitigate shared memory problems. Built for hot-switchover - you can upgrade Erlang problems without closing them first!
In just about every conceivable way, Erlang is the right choice for high-end multi-core multi-system clustered application development. I have a large-stack, clustered application written in PHP. While it works well, there are limits to what we can do within a single process - a problem that's likely to become worse over time as needs continue to scale up. If I were to do it all over again, I'd take a good, hard, look at Erlang.
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Freenet is a horrible failure. The only thing on there is porn which loads slower than using a dial-up connection, and the "websites" which actually *will* load are random. Freenet is a horrible piece of trash, nobody should care what other garbage this guy creates. I've asked him to please stop with this shit, but he wouldn't listen to me.
I read that as "distributed copulation" for some reason. I need more sleep.
"Ian Clark of the Freenet fame".. Actually, practically no claim about Freenet came true. The authors advertised "anonymity" etc. etc. at the same time as university professors published studies of statistics about the snooped connections: to any node present on the network for some time it is elementary to collect IPs.
It was painful to see so many users completely duped by the untrue claims, which their authors knew pretty well were untrue (and of which fact one-word admissions can be found buried somewhere in the wiki on their site).
Ian Clark and his collaborator knew nothing of the concept of the Small World (a type of graph that naturally grows in case of such networks), and therefore were not aware of the conditions (i.e. parameters that have to get set for the connecting nodes in their network) needed to make this network self-sustaining, and when pointed to the concept, they chose models by Newman, a prolific publisher of those arising from computer-simulated abstractions, rather than Barabashi (I'm afraid I misspelt his name), who offers much more realistic and practical ideas about this kind of topological network structure.
People do not change, really.
So what I'd expect from this announcement is a repetition of the story with Freenet: a real and interesting problem, inflated claims, and no actual solution, just claims and "development" for years to come.
So I remain a pessimist.
Is there a potential to use this on a GPU? The current problem with GPU programming seems to be solved with swarm.
Maybe the article needs correction because "swarm" on its own is fairly mature (as in old) concept in distributed computing.
I used SWARM an year ago and I was impressed by the possibilities it offers. It was also pretty stable. I'm sure it would have achieved a very reliable level of stability by now.
The largest prime factor of my UID is 263267.
way way back, IBM did some stuff with Java and agents... http://wapedia.mobi/en/Aglets
Did no one read "Prey". Give multiple computers the ability to work together today and tomorrow we'll have a bunch of computers trying to take over the world. As if grandma doesn't have enough problems with the computer when it really does start acting on its own.
Its the fricken $NtServicePacks in C:\WINDOWS that slows the whole damn thing down. But don't delete them manually, rather run "Disk Cleanup" and let it get sync the corresponding registry entries, so that windows can revert to its last known good configuration ... 3.1 IMO
RTFM is not a radio station.
I've not RTFA'd yet, but on first blush, this sounds suspiciously like the Amoeba project/work/files. The net result of Amoeba was that you'd end up with a large virtual machine, comprised of many individual machines scattered across different sites. How is this different?