World Wide Cluster

gwjc writes: "There is a pretty good Ian Foster article on Web-based computing clusters at the Nature site. The usual SETI@home, condor, and entropia mentions as well as a few that were news to me such as "Compute against Cancer" and "Fight Aids At Home" with links. I wonder how I go about declaring."

Comments

Curse j00 seti

[kastaverious]

seti@home was fun

seti@work got me fired

Re:Curse j00 seti

[andyh1978]

seti@work got me fired

Well, you can't say they (Seti@Home) didn't warn you. It's part of the licence agreement.

Restrictions
You may use this software on a computer system only if you own the system or have the permission of the owner.

Why do someone else's work for free?

[Chuck Flynn]

More and more companies are starting these distributed computation projects that feed off newbies' altruistic intentions, but why? Are the companies' motives altruistic as well? Hardly. Whatever drug molecule you calculate will be instantly patented by the corporation without so much as a by-your-leave. At least with those key-cracking contests, the winner got a prize. Here, you just get the shaft, and Entropia Incorporated gets a cut.

I'm sick of corporations and I'm sick of patents. It's getting to the point where I feel like sabbotaging all patent-seeking enterprises, even if it means we'll never find an AIDS drug. You can't do good by doing evil first, no matter what Macchiaveli tells you.

There are downsides to this..

[Dan Hayes]

After all, whilst the advantages of distributed computing are clear in that they can provide a way of harnessing a lot of computing power for a cheap cost, there are also downsides to this kind of project.

If people are so taken up with this sort of thing, imagine how easily it could be abused. People don't tend to be able to recognise and deal with email viruses, let alone a rogue distributed project that claims to do one thing whilst in actuality do something else. It sounds to me like a perfect opportunity for intelligence agencies to get their software on people's computers without anyone knowing!

How can you tell whether that client you run 24-7 on your home computer is actually helping calculate the next prime number or whether it is scanning all of your net connections and sending the information to a giant government database to be perused at their leisure? Police states like Britain already want to keep records of everything you do, this seems like a damn good way of doing it on the sly.

Personally, I'd be very wary of any piece of software that sits on your PC and has a constant connection to the internet. Unfortunately, most people are too trusting when it comes to their security online...

Re:There are downsides to this..

[epaulson]

Look, It's not hard to build sandboxes to protect yourself against foreign code. The way to do this has been well-documented - it can be as simple as a chroot() call or as full-out as the virtualization tricks that VMWare and plex86 pull. Just because Sun and Microsoft haven't managed to get it right yet doesn't mean it's not possible.

And if you're really worried about abuse, let's take a quick paranoid-look at things.

Evil Groups:

1. Microsoft

2. NSA/CIA

3. Telecoms

I'd say that if the intelligence community wanted it's software on computers it's already got plenty of opportunities.

-Erik

This is quite scary.

[Lover's Arrival, The]

I mean, could this be the next stage of the Internet we are seeing? Could the internet end up as one big supercoumputing cluster, and when we use it we timeshare? It does seem to be the ultimate long term direction that the Web is heading in, doesn't it?

The thing that scares me is the possibility of said cluster being abused, and hackers using it for ill purpose. Also, what are the implications for privacy? Look at .Net and the like, this is the nest step, and all my private files will be spread all over the Web! Ultimately, the superdupercluster could become conscious and ruin us all! ;-) I would like to see these technologies more strictly controlled. Sharing of data is one thing, but sharing of processing power seems a bit on the dangerous side, don't you agree?

Thanks for reading.

Re:This is quite scary.

[Peter Dyck]

the superdupercluster could become conscious and ruin us all!

What if it would become conscious and most of its information data base and the way it communicate with the outside world would come from the usenet and sites like Stileproject and Slashdot?

Scary indeed.

Re:This is quite scary.

[drinkypoo]

I mean, could this be the next stage of the Internet we are seeing? Could the internet end up as one big supercoumputing cluster, and when we use it we timeshare? It does seem to be the ultimate long term direction that the Web is heading in, doesn't it?

I sure hope so.

I was just talking about this very concept with a coworker just the other day, about how to fairly share processor time.

The concept is something like this: Every N instructions executed on your system (probably in some sort of virtual machine) constitutes some sort of computing "unit". CPU-hours and such are meaningless when people have different processors, so I think that this is the best way to measure it. (Any resource can be shared this way, but let's just talk about processing power for now.) You should be able to either sell blocks of time on your systems (which opens up the possibility for companies which make their money by selling compute time) or you should be able to simply trade it.

We would need a central system to manage the block counts. This system must be free. Who should run such a system? I have no idea. In any case, when you send out your jobs, they get picked up (peer to peer) by whoever is idle, and processed. The doling out of jobs would be handled by the block count system.

So now let's say my system has had the cooperative network plugin sitting on it for a couple of months and I've built up, oh, a thousand blocks of processing. I now send out 500 blocks worth. My priority should be weighted positively because I have not been using my credit. Therefore, someone who has 500 credits and sends out the same amount of work should have to wait longer (and/or get less concurrent jobs) if my request alone will make the system busy.

Now, if the system is largely idle, and I have 1000 credits and send out 1500 blocks' worth of work, then my jobs still get processed immediately. However, I will then be at -500 credits. If I then submit another 500 block job and the system is busy, I will have a very low priority.

However, you might decide that you want certain people to have priority on your system even if they have a low number of credits. It's your system, it's your right. You should be able to add (or remove!) priority from a user/project ID, disallow all use by an ID, or allow use only by a certain ID. You should also be able to specify times during which only you have access, no one has access, a certain group is prioritized higher, or what have you. Again, it's your machine.

And finally, the source to the server code should be freely available so that anyone can run a distributed processing network of their own, for public or private use. You should also be able to merge networks and later split them again if a group wants to share their private resources with others. Key management should be peer to peer but brokered through the server (or at least signed by the server) for security purposes. Only people with a current (and verified) key should be able to use your resources, if you so specify.

Re:This is quite scary.

[Hard_Code]

You could have said the same thing for newspapers->radio, or radio->TV, or TV->internet...etc. The issues aren't new. There are benefits and drawbacks. Hopefully we can draw the lines in the correct places so that we get the most benefits with the lowest percentage of misuse/abuse/whatever. This is the whole crime and security question in general. If we wanted to be absolutely safe from each other we'd all be locked up.

Still no-go on the paycheck

[Brento]

None of the companies this article mentions will compensate you for your computing time. Until they do, this whole thing isn't going to take off, because there just isn't enough motivation for Joe Q. User to install this software. Why would they install something they don't understand, at a privacy risk and a stability risk, for no immediate personal benefit?

I used the Distributed client for quite a while, but I switched to Seti just because it had a cooler screen saver. I've got a bunch of computers in my office that are usually idle, and this at least looks cool when the PHBs walk by.
At least Entropia (one of the companies in the article) gets that part of the motivation, and provides you with a color screensaver. It's not nearly as good as the Seti one, but it's something.

Re:Still no-go on the paycheck

[gimpboy]

first off joe q user doesnt understand what a security and privacy risk is. if joe q is using windows the stability risk will not make that big of an impact. all of that aside it could be that joe q cannot do complex math but he understands that his computer can. he might also realize that helping to find cures for diseases may not directely impact him, but it's a good thing to do.

i dont know about the rest, but the _only_ one i went to: cure for cancer gives away money.

i didnt know seti at home paid money. screensaves are pretty worthless in my mind, especially now that power management can turn off the monitor. so if the computers in your office win money do you give it to your company?

use LaTeX? want an online reference manager that

Re:Because it is for the common good

[atrowe]

Let's face it, most people are too self centered to run any kind of distributed project just for the scientific benefit. Joe AOL would rather leave his pretty Flying Windows screensaver up than donate CPU cycles to find ET/cure cancer/crack encryption/whatever. There needs to be some kind of monetary reward to encourage the majority of users to run a distributed client. I'm pretty sure that the "pay based on the numbers you crunch" business model is out of the question, since most of these organizations aren't operating as a commercial entity. And don't have any way to raise the capital to fund such a project, so I would likepropose an alternative method: The owner of the computer that gets the lucky number and ends up completing the goal of the project gets $1,000,000. I'm sure that would be only a small percentage of the revenue generated by marketing the cure for cancer, but I think that an approach where one lucky user gets a huge sum would be more enticing to the average user and help to foster more widespread use of distributed computing.

Just a thought.

Os integration

[kastaverious]

I have been interested in this kind of distributed computing for a while. I got to thinking how cool it would be if some kind of distributed computing system could be added to say linux(as a module or something)

I envisage a system were you have resource_buddies-people you have agreed to share idle cpu time with. These buddies would probably need permenent ip add's, but they are becoming increasingly common with broadband links. Anyway, when you are doing something like rendering something with gimp, ior any other cpu intensive task, the kernel module could kick in at a user defined cpu usage level. When active the module could test a few resource buddies to see if any are active and if any have cpu idle time. If it detects idle time it then shares the processing load with the remote system.

Anyway, just a thought, probably never happen

Re:Because it is for the common good

[Jose]

Let's face it, most people are too self centered...

yes, but people like to think that they aren't self-centered. So a project like this could take advantage of that.
If all a person had to do was double-click[1] on some file, then have it install. (ie a _really_ easy install), then people would do it. They get to cure cancer in their spare time, that makes them feel good. Look at all those chain-letters bouncing around the net saying "forward this to ten people to make some sick and dying girl happy" or some shit. They'll do it...if it is _simple_.

[1]People will be somewhat scared that it is a virus or something, but will only hesitate for about 2.5 seconds, then run it.

Making such a thing possible...

[chipuni]

I've been donating spare computing time to distributed.net for the past two years (though I'm starting to reclaim those clock cycles for my own projects again...) However, I would not donate spare computer power for any other project unless either the source code is available, or it is run out of a sandbox that I trust it cannot get out of. (It would also have to be for a good cause.)

We already have such a sandbox which is multi-platform (including Linux.) Although it's not the fastest possible implementation, I'd be much more willing to donate my spare computing cycles if the program were written as a Java applet.

The same restrictions that make Java applets safe also, to me, sound like the restrictions that would make distributed computation safe. They have no access to your local disk. They cannot make network connections, except to the source of the code.

Aside that people think of applets only for displaying graphics, and maintaining one of them up 24/7 would be difficult, are there any reasons why Java applets shouldn't be used for distributed computing?

Analysis of Distributed Projects

[�nubis]

Being a long time distributed computing advocate, I've used (and crunched many blocks for) GIMPS, distributed.net, Seti@Home, Folding@Home, and United Devices. I'm currently using all my spare cycles for United Devices. Why? Well here's a brief explanation/analysis of the projects I've used:

GIMPS - They have a good, clean client, but the critical problem is that the project has no conceivable benefit.

distributed.net - Probably the best client/site out there, and definitely the largest pre-Seti@home project. However, the cracking of encrypted messages has next to no scientific benefit (it is quite easy to calculate the chance per try of cracking any of their ciphertexts). Recently they've been doing some work with OGRs. Finding new OGRs looks like something that at least has a marginal benefit. On a side note, distributed.net has partnered up with United Devices.

Seti@home - It seems like everyone and their mom is running seti@home. However, reportedly seti@home actually has more clock cycles than they need. (they can only get so much radio info per day to analyze)

Folding@Home - Definitely a lesser known project which is being run by some researchers from Stanford, they analyze proteins. The project definitely has scientific merit, however they're experiencing some growing pains due to their recent popularity. Also their client is definitely beta-esque.

United Devices - This is the project that I'm currently contributing to. (so of course I'm bias) I chose them because they're doing something useful (working on cancer stuff with some researchers from the University of Oxford), have a fairly good client, and have a 'rewards' program for their users. (btw, GIMPS and distributed.net users also have the chance of winning a large cash prize) In addition, UD has partnered up with distributed.net, so it looks like UD just might be the commercial corporation to win the Internet-based distributed computing market.

Re:Analysis of Distributed Projects

[drinkypoo]

Folding@Home - Definitely a lesser known project which is being run by some researchers from Stanford, they analyze proteins. The project definitely has scientific merit, however they're experiencing some growing pains due to their recent popularity. Also their client is definitely beta-esque.

Right now, I'm running Folding@home. In the past, I've crunched blocks for distributed.net. Your objection to key cracking is spot-on the reason why I don't run their clients any more. As for SETI@HOME, the reason I don't do that (and never have) isn't that they have plenty of CPU time already; It's that I don't expect SETI to actually find anything.

Folding@Home can help us right now. I'm not suggesting we kill off SETI, or even SETI@HOME; We do learn useful things from both, though not where the eetees are. But Folding@Home has more immediate applications.

Re:Analysis of Distributed Projects

[WillWare]

I agree that a number of these projects offer little scientific or societal benefit, so it's very gratifying to see this approach starting to be applied to the advancement of medicine.

You've listed several projects, and I know there are others, each of which has developed its own client from scratch. This seems like pointlessly duplicated effort. Much of the functionality must be the same for all the different clients, isn't it? If there were an open-source distributed computing client project, it could be developed and debugged by all these teams and be much more reliable. With a standardized client there could be an economy of MIPS, some given freely and others sold. It would considerably advance all these projects, and any future ones.

I realize there are security issues; if badly implemented, client code could present awesome opportunities for viruses. But suitable measures should prevent this: digitally signed work units, maybe verifying checksums with the server, and there are probably a dozen other possibilities.

If the nature of the problems is so diverse that there are necessarily deep fundamental differences between the various clients, then this would be a bad idea. But I'm guessing that, except for variations in needed bandwidth of peer-to-peer communication, the clients ought to look mostly pretty similar.

Amazing

[Ravagin]

Wow, I read one book, and....

You see, this is the second time in this week that I have been able to relate a /. comment to this book I read a while ago, Wyrm by Mark Fabi. It's set in the days leading up to 01.01.00, and it has an interesting premise. Part of it is a vast, complex, "AI" who is sort of born out of unified processing on the internet. It's basically an accident.

Like some of the other stuff in the book, it's somewhat... far-fetched. But that's OK, because it's still a great book. ;)

-J

Just Like Science Fiction

[Begin2See]

Robert Heinlein saw it all 50 years ago... Just read The Moon is a Harsh Mistress In that book, the Lunar colonies main processing computer (which, after gaining sentience, was called Mike) became sentient after many modules and auxiliary processors were added. It brings up an interesting thought... I really know nothing about neural nets or how they are simulated, but couldn't one of these distributed processing systems be used to simulate a neural net? With enough CPU power behind it could the system develop a somewhat sentient persona?

Grrr...

[lpontiac]

There is a pretty good Ian Foster article on Web-based computing clusters at the Nature site

None of these projects use the HTTP protocol, or hypertext in any form (let alone represented by HTML or one of it's variants.) So what on earth do they have to do with the web?