Slashdot Mirror
Collaborative Map-Reduce In the Browser
igrigorik writes "The generality and simplicity of Google's Map-Reduce is what makes it such a powerful tool. However, what if instead of using proprietary protocols we could crowd-source the CPU power of millions of users online every day? Javascript is the most widely deployed language — every browser can run it — and we could use it to push the job to the client. Then, all we would need is a browser and an HTTP server to power our self-assembling supercomputer (proof of concept + code). Imagine if all it took to join a compute job was to open a URL."
Two comments:
1. He places the map/emit/reduce functions in the page itself. This is unnecessary. Since Javascript can easily be passed around in text form, the packet that initializes the job can pass a map/emit/reduce function to run. e.g.:
In fact, the entire architecture would work more smoothly using AJAX with either JSON or XML rather than passing the data around as HTML content. As a bonus, new types of jobs can be injected into the compute cluster at any time.
2. Both Gears and HTML5 have background threads for this sort of thing. Since abusing the primary thread tends to lock the browser, it's much better to make use of one of these facilities whenever possible. Especially since multithreading appears to be well supported by the next batch of browser releases.
(As an aside, I realize this is just a proof of concept. I'm merely adding my 2 cents worth on a realistic implementation. ;-))
Javascript + Nintendo DSi = DSiCade
Imagine how much *spam* you could send using this approach.
No, wait...
Save your wrists today - switch to Dvorak
We already have that. See botnets.
If you were really interested enough to donate your CPU cycles, is it really that much harder to install BOINC, and get a job running?
Plus then you can run native code instead of having to run in [shudder]Javascript[/shudder].
Convert FLACs to a portable format with FlacSquisher
Progress is running less JavaScript, not more.
You could also use this to index the MP3 files on everybody's hard drives, then share the music just by visiting a URL!! ... oh wait...
Javascript really isn't suited for this kind of thing, even with worker threads, for two reasons I can think of. First, web clients are transient... they'd have to report back often in case the user clicks away.
But more importantly, Javascript just isn't a good language for massive computation. It only supports one kind of number (double), has no vectorization or multicore capabilities, has no unboxed arrays, and even for basically scalar code is some 40x slower than C, let alone optimized ASM compute kernels. (This is for crypto on Google Chrome. Other browsers are considerably slower on this benchmark. YMMV.)
I hereby place the above post in the public domain.
for those like myself that had no idea what MapReduce was:
http://en.wikipedia.org/wiki/MapReduce
Oh, please, make the MapReduce fanboyism stop.
Yes, it's a neat technique. It's also very old and obvious. Google's implementation is also good, but this stuff is just not rocket surgery. It's just a simple pattern of how to massively parallelize some types of computational tasks.
But somehow, just because some dudes at Google wrote a paper about it, it's become the second coming of Alan Turing or something among some silly folks. Hell, a couple of weeks ago somebody was saying on the comments here that MapReduce was a good alternative to relational databases. Now that is silly.
Are you adequate?
"Javascript...â" every browser can run it..."
There is a huge difference between being able to run javascript apps and run javascript apps well - not to forget that a lot of the javascript I see out there really only works on PC's with IE or Firefox, Opera and Safari, especially on OS X seem to have trouble with some sites that aren't coded for compatibility, but instead pushed out quickly with little regard for anything other than IE on Windows.
Ave Molech Setting
A common mistake in multi-server builds is that bandwidth is free.
Bandwidth Costs Money and Time. Both are reduced by having the network closer to the processing. This is one of the reasons google bought all that "dark fiber" left around after the .com bust.
Another flaw is that computation of data is difficult to provide "good results" in blocks unless they're doing relativity matrices (Think PageRank).
Something to think about:
If I'm sending names to your pc, what can I derive from that list without having the entire list?
If there were a couple-few or more orgs competing to use my extra cycles, outbidding each other with money in my account buying my cycles, I might trust them to control those extra cycles. If they sold time on their distributed supercomputer, they'd have money to pay me.
As a variation, I wouldn't be surprised to see Google distribute its own computing load onto the browsers creating that load.
Though both models raise the question of how to protect that distributed computing from being attacked by someone hostile, poisoning the results to damage the central controller's value from it (or its core business).
--
make install -not war
Is this why my browser keeps telling me scripts on the slashdot main page are taking too long and do I want to stop them for the last few months?
This seems to me a self-defeating idea. The obvious goal is to get more processing power. Yet using a scripted language is inefficient, and a waste of processing power. If you want more processing power, you need to group computers of the same general instruction set, and which can run compiled (or, dare I say it?) assembled machine code.
My CPU time isn't idle. It's keeping my laptop from being too hot to touch and too noisy to work on. And there's no reason to pay more for electricity than I already do.
-B
Ash and Hickory, straight-grained and true, make excellent bludgeons, dandy for the cudgeling of vegetarians.
and you don't think you could get 100 times more users to visit your web app than you could convince to download and install an exe?
Because you can - or because you should?
I think this approach to MapReduce is a pretty creative angle to take on it. However, there are a number of distributed systems-type problems with doing it this way, that would need to be solved to actually make this realistically possible:
1) The dataset size is currently limited by the web server's disk size.
Possible solution: push the data to S3 or some other large store.
2) There is a single bottleneck/point-of-failure in the web server. In theory 10,000 clients could try to emit their map keys all at once to the web server. IIRC, Google's mapreduce elects nodes in the cluster to act as receivers for map keys during the map/sort phase.
Possible solution: Again, if you were using S3, you could assign them temporary tokens to push their data to S3 -- but that would be a large number of S3 PUT requests (one per key).
3) Fault-tolerance -- what happens when a node in the browser compute cluster fails for any of N reasons? How does the web server re-assign that map task? You'd especially want to ensure that computation finishes on a job in an unknown environment such as 1,000,000 random machines on the internet.
Possible solution: If you haven't heard from a node in N seconds, you could reassign their map task to someone else. This is a similar idea to the MapReduce paper's description of sending multiple machines on a single map task, and racing them to the finish.
4) Security -- there is no way to deterministically know whether the data emit()ed from a user's browser session is real or not. How do you trust the output of 1,000,000 users' Javascript browser executions (I think the answer is, you don't).
But, you can just close the browser or type "killall firefox" and the program dies and you have to go to the URL again to get to it. So, though this is bad for other reasons, yours just isn't one of them.
Taxation is legalized theft, no more, no less.
Javascript really isn't suited for this kind of thing, even with worker threads, for two reasons I can think of. First, web clients are transient... they'd have to report back often in case the user clicks away.
I don't see why web clients being transient is a problem. The whole point of the MapReduce algorithm is that each worker (the web clients in this case) don't need to know anything about what the other worker is doing, what the system as a whole is doing, nor what it had done with any past job.
It would need to be 10000x at the very minimum.
If a user downloads, say, folding@home, it's running all day, every day, on all cores of the machine, whenever the computer is on and idle, which is most of the time. The user doesn't have to remember to run it, doesn't have to devote screen real estate, attention and so on, and the program is less annoying because of its low priority and relatively low memory footprint (less boxing).
Additionally, the 40x I cited was in the fastest available browser (Chrome), compared to a relatively slow implementation (OpenSSL), for code that doesn't benefit from vectorization (at least, not on x86-based processors). I expect that the difference between a scientific compute kernel in JS and in assembly would be at least 100x, maybe 200x or more.
Let's suppose that everyone in your rosy world uses FF 3.1 with JIT. That's 3-5x slower than Chrome in my benchmarks; say 4x. Let's suppose that Chrome is 25x slower than unvectorized C, which is 4x slower than optimized assembly. Let's say people run the site 5 hours a day on one core for a week, but have their dual-core computers on for 10 hours a day, 90% idle and would keep folding@home installed for a year.
Then the EXE is 4 * 25 * 4 * 2 * 2 * 50 * 0.9 = 72000x more productive.
Use the right tool for the job.
I hereby place the above post in the public domain.
This could be a possible way to generate revenue from popular websites... instead of selling something of such dubious quality as "advertising impressions", high-volume sites such as /. could support themselves by taxing, say, 10% of a viewer's CPU with an unobtrusive background thread, and selling the aggregated processing power to customers. I'd certainly be happier donating a percentage of my otherwise totally wasted CPU time to a site than having to read crappy ads for products I don't want.
Rampant carbon sequestration destroyed the Dinosaurs' tropical paradise. I'm here to help repair the damage.