Slashdot Mirror
Swarm — a New Approach To Distributed Computation
An anonymous reader writes "Ian Clarke, creator of Freenet, has been working on a new open source project called Swarm. The concept is to allow a computer program to be distributed across multiple computers in a manner almost completely transparent to the programmer. The system observes the program executing and figures out how the workload should be distributed for maximum efficiency. Swarm is implemented in Scala. Its at an early-prototype stage, and Ian has created a good 36 minute video explaining the concept and the current implementation."
Just saying it outloud so we can start working on countermeasures.
The thing that's always killed this idea (along with automatic parallelization even on the same machine) is that the overhead of figuring out what's worth distributing, and the additional overhead from mistakes (accidentally distribute trivial computations), often swamps the gains from the multiple processors banging away on it simultaneously. Determining statically what's worth distributing is very hard, since solving it properly is undecidable (basically equivalent to the halting problem), and even solving it in a significant enough subset of cases to be useful has proved difficult. It looks like this project is monitoring dynamically to determine what to distribute, which seems likely to be more fruitful, although historically that approach has suffered from the overhead of the monitoring (like always running your code with debugging instrumentation turned on).
I certainly hope he has a breakthrough vs. past approaches, or it could just be that advances in a lot of areas of technology have given him a better substrate on which to build things that naturally mitigates lots of the problems these things used to have (automatic parallelization research started probably ahead of its time, back in the 1970s, so that most academic stuff was killed off by the 1990s after no really knock-down results emerged). It's not entirely clear to me what the killer advance is, though. The particular variety of portable continuations? A good way of easily monitoring computations? Something that makes the data-dependency analysis particularly easy?
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
I'm just getting into Agent Based Modeling myself and I had exactly the same thought... why would they use the name of an established tool; especially when there are similarities in the concepts. This seems like a recipe for confusion.
A good first check when starting an open project is to check propesedprojectname.org and see if there's anything active there. Or even just Google it - if another project shows up near the top with the same name, it's probably a good idea to pick another name.
I'm sure there are plenty of synonyms for "swarm" that capture the idea, if not an alternate spelling.
But like you said, it does sound like an interesting project.
Mod parent up. This is exactly what Ian did with Freenet.
He cobbled together an overly-simplistic prototype to address a set of very difficult unsolved problems in anonymous communication and then farmed out the actual real-world legwork on those problems to interested open source developers while Ian himself effectively abandoned Freenet for other (paying) gigs. To this day he is credited, somewhat ironically, as "the creator of Freenet," and a decade later the Freenet project still hasn't solved the problems it set out to solve, even after changing the fundamental network architecture several times.
Great career strategy though. Get credit for the shiny things and pass the shame of failure off on others. He's CEO material all the way.
Clarke never said he doesn't know how to solve the remaining problems (of which, he freely admits in the video, there are many). Would you prefer that no open source project was released to the world until it was 100% finished? Good luck with that.