Slashdot Mirror
Swarm — a New Approach To Distributed Computation
An anonymous reader writes "Ian Clarke, creator of Freenet, has been working on a new open source project called Swarm. The concept is to allow a computer program to be distributed across multiple computers in a manner almost completely transparent to the programmer. The system observes the program executing and figures out how the workload should be distributed for maximum efficiency. Swarm is implemented in Scala. Its at an early-prototype stage, and Ian has created a good 36 minute video explaining the concept and the current implementation."
.. was Mosix http://www.mosix.org/
It allowed mosix-running linux computers to distribute their loads over a connected other mosix-running linux computers.
Processes migrate to other nodes transparently. No programming changes were needed.
At first I thought they were talking about Swarm, a "attempt to gather up many different kinds of models that go under the heading of "agent-based modeling" and create a common language and programming approach." that I've worked with before. I'm surprised they went with the name of an established toolkit in another aspect of programming. Still, looks like a cool tool, another layer of abstraction to make distributed computing easier might make it more attractive to those that don't use it much at the moment.
"goodbye and hello, as always" ~Prince Corwin, from Zelazny's Amber series
It sounds like a good idea, but I don't think the project is far enough along in this video to warrant a posting. Maybe he was using too much of a trivial example to be appreciated in the video, but his explicitly offloading the task to another computer doesn't appear to be very far beyond standard client server models. If it were already automatically transporting processing between different nodes, it'd be much cooler, but that is not a trivial problem to solve. Deciding what should and what shouldn't be distributed at the application level will be extremely hard I imagine. If the project were farther along in its maturity I'd be much more interested.
Yeah. Visit a website with an applet and you get the JVM startup and it stays up and running even after you leave the website and visit websites that don't have applets. In other words, I probably wouldn't notice at first either and I'd be chugging along until I restarted my machine and saw the JVM pop-up again for no apparent reason. Other folks who have no idea wtf a JVM is would never notice.
It's NOT me! It's the meds! I'm on 1000mg of Fukitol.
> The thing that's always killed this idea (along with automatic parallelization even on the same machine) is that the overhead of figuring out what's worth distributing
That kind of thinking is so 90's. Brute force data mining, as an example means harvest it all and let target groups sort out what they want. It is a waste of time to 'decide'. That's like stopping to inspect every shovel full of ore as it comes out of the ground. All or nothing has been the default for some time now, and this is just another example.
A friend of mine did a system like this about ten years ago --- hi, Iain! --- called Flit. It had a number of the same features, although using a custom language; it had some rather interesting concepts, such as asynchronous function calls that would return immediately, spawning a new thread, but return a future: a value whose value was not known yet. Accessing the value would cause the thread to be waited upon.
Unfortunately the killer problem that sunk Flit was that of distributed garbage collection. Collecting data over multiple machines is really, really hard, and he never found a usable approach to make it work. I was very disappointed to see that Swarm's garbage collection is still on the to-do list --- he doesn't appear to have started to think about it yet.
I hope he can make Swarm work --- it's something that we could all definitely use. But there are fundamental theoretical problems that have to be solved first...
That's true to some degree. But computers do slow down as they age. Components damaged by the constant heating cause more errors and therefore require retransmission or error correction, slowing things down.
No, not really. PCs are nowhere near that sophisticated. A high-speed CPU bus is not like a DSL connection. Pretty much it has to work near-perfectly, or it's blue-screen city.
... still as fast as it ever was (faster, actually ... I have it running a stripped-down version of XP.) If you have a motherboard or PC that is getting errors due to heating what you're going to see are crashes and lockups, not slowdowns. Personal computers are not mainframes or minicomputers: even with ECC memory they are not fault tolerant to any significant degree, and frankly I think it's a wonder they work as well as they do (Windows issues aside.) When a component starts generating errors your average PC just breaks ... if you're lucky it's just the faulty subsystem, but if you're not the machine is toast.
For example, I have a couple of Athlon 1.4 ghz machines that are running just as fast as the day I built them, and they've never been turned off. Also have an old Thinkpad R41
People's machines slow down because a. they never defrag their hard drives and b. they get infected. It just takes a single badly written piece of malware to turn an otherwise decent machine into a 386, yet users frequently blame the hardware for being too old, as if that somehow explains poor performance. Many people are completely amazed when I clean up their system for them and pack the hard disk. "Wow, it's like a whole new computer!" No, dimbulb, it's the same computer you've always had, you were just too lazy to give it even minimum maintenance. I'm glad I'm not in IT: it's a lot like being a doctor. You have to deal with people who have no ability to think rationally about their problems, and even when you give them good advice they never follow it anyway.
The higher the technology, the sharper that two-edged sword.
I don't think this characterization is fair, and I think you would have a hard time finding somebody who actually worked on Freenet to agree with you. Ian's orginal technical ideas for Freenet - as well as his vision - are very much still a big part of the architecture, and he could never be said to have abandoned it. In fact, time has vindicated many of his ideas to a far greater extent than I expected when we started working with them. You are right that the project has not yet solved the problems it set out to solve - but since it has wildly high ambitions, that should hardly be surprising. I think it has made a positive contribution all the same, if only to our understanding of many of the issues involved.
It is true that the press has had a tendency to paint Ian as the lone father of the project, but that is just the way to press works, and I have never seen Ian taking credit for other peoples work. And, to be honest, after you have done it a few times, you start realizing that dealing with the press isn't nearly as fun as it is cracked up to be, and that Ian has a knack for communication that most nerds, myself included, do not. I think Freenet has been very well served by Ian's ability to effectively communicate it's goals and gain attention -- among other things it has allowed several coders, of whom I was the first but not the last, to work full time for the project for certain periods. That said, I was a bit disappointed when the NYTimes ran a cover story on a presentation Ian and I held at Defcon and forgot to mention me at all, but I got over it :-).