Can You Spare A Few Trillion Cycles?

rkeene517 writes "11 years ago I did a simulation of 29 billion photons in a room, that got published at SIGGRAPH 94. The 1994 image, called Photon Soup is here . Now computers are 3000 times faster and I am doing it again only much better, with a smaller aperature, in stereo, with 3 cameras, and with some errors fixed, and in Java. The 1994 image took 100 Sparc Station 1's a month to generate. I need volunteers to run the program for about a month in the background and/or nights. The program is pure Java." Read on for how you can participate in the project.

"The plan is to run the program on a zillion machines for a month and combine the results. All you have to do is run it and when the deadline arrives, email me a compressed file of the cache directory. So email me here and I'll send you the zip file. The deadline will be June 1st 2004.

The running program has a More CPU/Less CPU button. Every half hour it saves the current state of the film. The longer and more machines that run this, the cleaner and sharper the image gets. If you have a lot of machines, I can give instructions how to combine the results so you can send in a single cache directory.

Of course, you will get mention in the article if it gets published."

Comments

More distributed processing

[isugimpy]

This is a wonderful thing to see. Distributed processing is a wonderful way to spend those extra clock cycles that most of us have, while at the same time benefitting someone else. I really hope to see more projects like this in the future.

Java?

[Kent Simon]

nothing against Java it has its place, but for something this CPU intensive, it seems like you'd be wasting CPU cycles. This sounds like a job for C.

Re:Java?

[hayds]

Java can actually be quite fast and efficient for number crunching or scientific applications because of the JIT compilers and automatic optimisation. Its only painfully slow when you need a GUI. It also has a great class library so he should be able to do things like the visualisation and the networking for the clients to send the data home relatively easily, whereas it would take a lot longer to write and debug in C.

Also with Java, he can just offer the JAR file on his site or whatever and people can d/l it and run it. I guess this isnt really important if he's aiming at geeks, but if he's trying to get others to participate, it is handy that people wont need to worry about compiling it themselves or picking the right version (like at the seti@home site).

Oh boy...

[MosesJones]

He needs networking connection, a decent threading model and doesn't want to crash your box.

So while he could spend a huge amount of time doing all these basic things in C and still have major risks for the people running it, he has chosen to use the right tool for the job.

Also the Maths libraries are IEEE compliant in Java and not in C on the PC, so I'm assuming that also played in to his reasoning.

Re:Oh boy...

[pla]

He needs networking connection, a decent threading model and doesn't want to crash your box.

False, on all of the above.

For "networking", users will manually send him their cache directory (as the FP explicitly stated). As for threading... To do what? He wants to run a completely straightforward trajectory simulation, iterated a few "zillion" times. I'll admit that I have a bias against most uses of multithreading and consider them inappropriate 99.9% of the time, but if you can even force them onto this project, you need to go back to the drawing board.



So while he could spend a huge amount of time doing all these basic things in C and still have major risks for the people running it, he has chosen to use the right tool for the job.

Umm... Yeah, whatever. He wants to run a CPU-intensive background process, performing a totally straightforward set of calculations, and nothing else. No GUI (beyond a few simple controls to make it play nice), and nothing server-side - sounds like a perfect candidate for anything but Java.

Personally, I'd say this even sounds like a good candidate for hand-tuned assembly. But then, at least from my alma-mater, they don't even require that to graduate in CS anymore. Sad... And people actually wonder why tech jobs keep heading for India. Well, the FP and you just provided a nice answer - Using Java for a tightly CPU-bound problem? Using threads for the same (Ever heard of "cache consistancy"? Yet another reason to avoid multithreading in a program of this nature)? Why not just downgrade to a 486? Same effect, less complex.

Re:Oh boy...

[David Leppik]

He wants to run a CPU-intensive background process, performing a totally straightforward set of calculations, and nothing else. No GUI (beyond a few simple controls to make it play nice), and nothing server-side - sounds like a perfect candidate for anything but Java.

Except that what makes Java slow is its do-it-yourself GUI and run-time object management. Those disadvantages mostly go away if you program Java as if it were C: use and re-use arrays of primative types in preference to short-lived objects, and cluster things together in memory (via arrays) when they are used together. The just-in-time compiler will convert the bytecode into machine instructions on the fly, using information that's not available at compile time.

Personally, I'd say this even sounds like a good candidate for hand-tuned assembly.

Okay, I'll bite. I've got two macs and to Linux boxes here. Each has a different processor architecture (Celeron, Athalon, G3, G4.) I don't know much about hand-tuning for x86, but on the G4 you shove your floats into 128-bit vectors and do your ray tracing in chunks of four floats. The G3 lacks the vector coprocessor, so you'd optimize it in a very different way.

Besides, by writing in Java and not hand-written assembly, he gets to run it on machines like mine, which probably weren't the original target platform. The name of the game in distributed processing is to use as many spare processors as possible, even if some are slow. An hour of work to promote his project is more likely to pick up 10% more (or 1000% more, with \.) CPUs than an hour of hand-tuning assembly is likely to get 10% more speed out of a particular processor.

Re:Java is a slow cruncher

[LarsWestergren]

He sounds like a clever guy, but "pure Java" for number crunching!?! With "pure C", it'd take half the time with half the number of computers.

...but he would have to spend a lot more time porting it between different architectures and OSes. God, how many times do you have to explain this to people? These days, processing cycles cheap, programmer time expensive.

FP ops in Java are incredibly slow and broken.

Er, do you have any more recent numbers than a lecture from 2001, originally published in 1998??

Trust?

[wan-fu]

What's to insure the trust within this project? Call me a cynic, but what's preventing some jerk from swapping some bytes in his set of data before sending it off, thus, rendering your combined result different from what you intended?

which would you rather run?

[theguywhosaid]

wow, its slower than C. i'd rather run a random java app than a random native app because you can easily sandbox it and know its not going to screw your computer. thats one less barrier to people helping the dude out. and theres no recompile for the various linux platforms, win32, solaris, macOS, etc etc. its certainly slower, but more friendly to the community.

power consumption

[lightray]

The fact of the matter is that a machine with 100% CPU utilization uses a lot more electricity than one with low utilization. The extra cycles aren't free.

I measured this in 1997 on some kind of AMD K6 machine. IIRC, running dnetc doubled the power consumption of the machine.

Enter applet.

[Kingpin]

Applets are bad for a LOT of things. But this is one thing they would work really well for. Using an applet:

1. The client PC runs the program in a sandbox
2. Most client PC's don't need additional software installed (if written for JDK 1.1)
3. The user does not need to know how to invoke a Java application
4. There's no administrative overhead in iniating the application, just go to a URL

Anonymous grid computing

[mec]

First there are resource allocation problems. The OS has to provide a sandbox with strict limits on all resources: memory, filesystem, and networking, as well as CPU time. It's fine with me if the "background compute demon" takes 25% of my processor but I don't want to take more than 10% of my memory.

Then there's the security issue.

But I see another problem which is even harder to solve: the tragedy of the commons. Consider a university campus, and suppose that anyone on campus can submit jobs to the Campus Grid. You come in the next morning and see that there are 10000 jobs in your grid queue, and 9800 of them are encoding random people's MP3's.

The problem is that if you give free resources to a large anonymous community, it takes only a few of those people to suck up all the resources. So you need some way of identifying everyone who submits a job, and some way of charging for the jobs.

Re:Java can be faster then C sometimes

[orthogonal]

Either I'm suffering deja vu, or this has been posted nearly verbatim before in a previous discussion of Java vs. C.

Not only that, Face the Facts (770331)'s last three or four posts are word-for-word copies of other people's posts, copied from anti-slash.org's "database tool".

Note only that, but anti-slash.org has posted links to his posts, asking their members to mod him up, with the notation "another karma whore account" -- which implies he's karma whoring in order to get mod points in order to troll.

(Implies but doesn't prove: anti-slash.org at one point asked its members to mod one of my posts up, why I'm not sure.)

But whatever Face the Facts (770331)'s motivations, his posts are plagiarism and he's a plagiarist, apparently not talented enough to write his own posts.

Mod him down.

Re:Java eh?

[AllUsernamesAreGone]

This is true for pure number crunching where you spend a lot of time in loops.

Java is quite good at that.

Now, try writing a real application. One with an interface, one that does real work and spends most of its time interacting with a user rather than banging numbers. Run the test again and you'll find that C++ is significantly faster than Java in this situation because Swing is slower than arthritic snail carrying a small planet and Hotspot isn't good at optimising the sort of stuff a general program has to do.

Re:Java eh?

[BlackHawk-666]

If this is true then why is Java so goddammed slow still? Why is it every medium sized or above Java app I've used performs like crap compared to a similar one compiled in C++ or simlilar languages? It just seems to me there is a major disconnect between what the Java adherants are claiming and the reality I am faced with every time I use a Java app.