Xgrid Agent for Unix

My Experience by artlu · 2004-06-23 11:24 · Score: 5, Interesting

My company has had experience using XGRID on our G4 notebooks. We always leave XGRID running and when we are at the office it is like having 20-30CPUs available at any given time. Now with Linux, we can have about 300 CPUs available, I just wonder how efficient it really is in the non-osx atmosphere.
Time to find the download.

GroupShares Inc. - A Free Online Investment Community

--
-------
artlu.net

Re:My Experience by JamieF · 2004-06-23 17:05 · Score: 4, Informative

Here are three gotchas that can make this sort of thing less appealing than it may seem at first:

Problem type: The problem may not be well suited to running on a bunch of PCs (especially when the agent app isn't allowed to take 100% of the machine's resources to accimplish the task) over typical office networks. Basically if the app needs to communicate frequently with other nodes, or if a huge data set is involved (or both), latency or bandwidth issues might outweigh the possible advantage of putting more CPUs to work.

Security: The data may be highly sensitive, in which case you might not want to put it on ordinary desktop PCs that might have untrustworthy users, spyware, etc.

Configuration: The configuration of your office's PCs may vary enough to make the cost of getting a companywide desktop cluster working unacceptably high. You'd have to pick a few target configurations and settle for that. Hopefully drivers and such wouldn't matter as much as CPU, RAM, disk, and OS version, but there are still companies that are just now getting their desktops updated to Win2K. There's also the headache of installing yet another required application on a large number of heterogeneous machines, which is virtually guaranteed to result in confusing installation problems. Oops, our app crashes if the user has this or that service pack installed. Oops, our app requires strong encryption. You could build your app on top of some sort of moderately portable framework or VM or whatever but that will have system requirements too, and probably will have some surprising gotchas when deployed in a real-world environment.

How many clusters by Anonymous Coward · 2004-06-23 11:24 · Score: 5, Interesting

actually have hetergenous hardware platforms? It would be interesting to see a G5/Xeon/Athlon cluster make the top 10 in speed.

Re:How many clusters by aixou · 2004-06-23 11:29 · Score: 5, Funny

That would bring a tear to my eye. "I have a dream that one day, all different architectures can work together in a single cluster, and processors will be judged not by the flavor of their bits, but by the speed of their results."
Re:How many clusters by foidulus · 2004-06-23 11:30 · Score: 4, Funny

I have a dream that one day, all different architectures can work together in a single cluster, and processors will be judged not by the flavor of their bits, but by the speed of their results.
You forgot, "as long as they don't run Windows!"
Re:How many clusters by foidulus · 2004-06-23 11:39 · Score: 3, Informative

Well, windows runs on Athlons and Xeons....and technically it runs on a G5(the XBox 2 dev kit that MS distributed is a modded NT kernel is run on a dual G5), so it's possible to do it on Windows, but why would you want to?
Re:How many clusters by 0racle · 2004-06-23 12:06 · Score: 4, Informative

Well firstly, since Windows is more then just a kernel, just because the Dev kit has it does not mean Windows as a whole does. Secondly, Athalons, P4, Celerons and Xeons are all the same architecture, ia32. If you had used the P4 and say, Itanium or Opteron/A64 you would have a point, which is why I qualified the statement in the beginning with 'to a large extent.'

--
"I use a Mac because I'm just better than you are."
Re:How many clusters by 0racle · 2004-06-23 14:46 · Score: 2, Funny

You could have a whole cluster on one machine, think of the performance and all the space you'd save.

--
"I use a Mac because I'm just better than you are."

imagine by Anonymous Coward · 2004-06-23 11:26 · Score: 4, Funny

imagine a beo...oh...

Re:imagine by wpmegee · 2004-06-23 11:51 · Score: 4, Funny

Imagine a beowulf cluster of /. trolls!
Re:imagine by Anonymous Coward · 2004-06-23 12:03 · Score: 5, Funny

Don't have to.

Mixed Company by jasno · 2004-06-23 11:26 · Score: 3, Insightful

Somewhat silly, but wouldn't you incur a bit of overhead mixing machines of different endian-ness? I suppose for non-communication intense algorithms this wouldn't be a big deal.

--

http://www.masturbateforpeace.com/

Re:Mixed Company by Carnildo · 2004-06-23 11:35 · Score: 4, Informative

Somewhat silly, but wouldn't you incur a bit of overhead mixing machines of different endian-ness? I suppose for non-communication intense algorithms this wouldn't be a big deal.

Not really. Everyone uses network byte order for communication, so you won't have more overhead in a mixed system than you would in a homogenous system.

--
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
Re:Mixed Company by kma · 2004-06-23 12:40 · Score: 4, Informative

Not quite. "Network byte order" is big endian. So on big endian ppc's, which macs are, all those "ntos" macros, etc., expand to NOPs. Once you introduce little endian machines into the mix, they start doing real work to transform internal representations for the wire.

The real tragedy is when you have homogenously little endian machines; e.g., a network that only has PCs on it. An integer gets byteswapped twice to end up in exactly the same byte order it was all along.
Re:Mixed Company by DonGar · 2004-06-23 12:51 · Score: 4, Interesting

What's worse.... that often ends up happening for loopback connections.

--
plus-good, double-plus-good
Re:Mixed Company by joe_bruin · 2004-06-23 13:04 · Score: 3, Informative

So on big endian ppc's, which macs are, all those "ntos" macros, etc., expand to NOPs. Once you introduce little endian machines into the mix, they start doing real work to transform internal representations for the wire.

not quite.
first, i think you mean "ntohs" (and ntohl and friends).
second, they are not macros. they are, in fact, real functions (in glibc, bsd libc, and windows' winsock library). i'd imagine it's the same on macs.
third, a macro that does nothing is not expanded to a NOP, it is simply removed by the preprocessor.

so, assuming the macs are conforming to bsd networking standards, ntohs is required to be a function, so there is still a function call per conversion (which is much more costly than doing the actual byteswap).

The real tragedy is when you have homogenously little endian machines; e.g., a network that only has PCs on it. An integer gets byteswapped twice to end up in exactly the same byte order it was all along.

a real high performance implementation (ie, the kernel) would not use ntohl, it would implement a similar byteswap macro. a byteswap can be done on x86 in one instruction, so it is fairly trivial to do.
Re:Mixed Company by Tjp($)pjT · 2004-06-23 13:40 · Score: 3, Informative

Unless ntohs is an inline function. Most compilers will optimize out inlines that return their calling argument unchanged. Of course reality differs and they are actually null macros on OS/X.
These routines convert 16 and 32 bit quantities between network byte order and host byte order. On machines which have a byte order which is the same as the network order, routines are defined as null macros.
The above quote brought to you by HMUG.

--
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
Re:Mixed Company by boots@work · 2004-06-24 17:23 · Score: 2, Informative

unless you're linking your libc statically, it can't be inline. it similarly can't be inline if you use a function pointer to it in some fashion.

Don't be silly. (Do the moderators actually think before scoring +1?) gcc is perfectly capable of inlining functions even when glibc is dynamically linked. It can also inline functions whose address is taken, just by generating a separate copy. Any other compiler clever enough to have inlines is very likely to do the same.
Re:Mixed Company by liteyear · 2004-06-24 20:19 · Score: 2, Informative

then osx has a broken bsd socket implementation. ntohs should be a function.
Better let the BSD team know that then, because they'll surely want to make sure their code complies with the "bsd socket implementation" spec you mention. Or... and here's a crazy idea, you could realise you're wrong and that Apple didn't decide to deliberately break the BSD code they used and actually have a very similar implementation to the BSD code.
/* $NetBSD: endian.h,v 1.7 2003/08/07 16:34:03 agc Exp $ */ /* * Copyright (c) 1987, 1991, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)endian.h 8.1 (Berkeley) 6/11/93 */ --SNIP-- #if BYTE_ORDER == BIG_ENDIAN && !defined(lint) #define ntohl(x) (x) #define ntohs(x) (x) #define htonl(x) (x) #define htons(x) (x) --SNIP--
Source: http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/sys/en dian.h

--
* Nothing is foolproof to a sufficiently talented fool *

Plenty of power to be had.... by Grant29 · 2004-06-23 11:26 · Score: 3, Interesting

This is really great news as it's becoming more popular to add CPU clusters to improve performance. Google is probably not the originator of this type of computing, but they have definately pushed it into the mainstream. Anyone living in NC might want to check out this new cluster going into RTP NC. I wonder if this will be the biggest cluser ever

http://www.rtp.org/index.cfm?fuseaction=in_the_new s_item&id=159

--
9 Gmail invitations availiable

Kinda Cool by hypermike · 2004-06-23 11:27 · Score: 5, Insightful

Imagine waking up one day to find your Mac has solved a vexing scientific problem. While the cure to cancer, super-efficient solar power and ending world hunger are a ways off, you can combine your computing resources using Xgrid -- and help usher in a new era of biological breakthroughs, rocket science and advanced models of scientific phenomena.

Everything is better clustered...

--

I've been dying to know.... by numbski · 2004-06-23 11:29 · Score: 4, Interesting

I have my G4 powerbook, 866 and my 800mHz iMac on my LAN at home.

If I use XGrid on the two, what kind of performace could I use it for day to day?

Faster compiles of applications would be the first thought. Any usefulness, say running photoshop? How about Quake? MAME?

--

Karma: Chameleon (mostly due to the fact that you come and go).

Re:I've been dying to know.... by Colol · 2004-06-23 11:44 · Score: 4, Insightful

Of the applications you've mentioned, only compiling things in Xcode would have any benefit. To utilize Xgrid, the application has to be written for it, which most apps simply aren't (and given turnaround issues, it would be suck for things like Quake and MAME).

Xgrid's main benefit is in "grunt work" calculations that aren't necessarily needed immediately. Things like SETI@Home or Folding@Home would be the sort of thing Xgrid excels at: throw some data out, have it processed, get it back when it's done.

While Apple has made clustering drop-dead easy, it's really not targeted at the home or small-business user, and the potential uses are pretty limited in that field.
Re:I've been dying to know.... by guile*fr · 2004-06-23 11:46 · Score: 2, Informative

basically, for your kind of applications: nothing.

I doubt you compile applications that big
photoshop: get an smp instead and plugins that support it
quake,mame: u kidding get a faster gpu instead
Re:I've been dying to know.... by mrchaotica · 2004-06-23 11:53 · Score: 4, Informative

You don't need XGrid for faster compilation - Developer Tools already includes distcc

--
"[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz
Re:I've been dying to know.... by Twirlip+of+the+Mists · 2004-06-23 13:35 · Score: 4, Informative

To utilize Xgrid, the application has to be written for it

Not so, not so.

If your problem is embarrassingly parallel, chances are you can use Xgrid to run it right now.

For example, let's say you're rendering a 3D animation. (I haven't done real 3D work since the PowerAnimator days, so pardon me of some of my jargon is antiquated.) You've got a scene file on which you can run a render command. A command-line argument tells the renderer which frame to render.

No problem. Just use use Xgrid's Xfeed plugin. Xfeed lets you set up a job that runs a single command with a variety of command-line arguments. You tell Xfeed that you want to run the "render" command with "-f" and the numbers 1 through 720.

Xgrid goes to the first available machine on the grid and says, "Run render -f 1." Then it goes to the second machine and says, "Run render -f 2." And so on, until there are no available machines. Then it waits until a machine becomes available and says, "Run render -f n."

As each output file (a frame, in this case) becomes available, Xgrid (the client application itself, I mean) collects them in whatever directory you specified when you submitted the job.

The cool part comes when you realize that this isn't a cluster. It's a grid. That means machines can come and go as they please. If this job is running overnight, when I come in the next morning and sit down at my workstation, the agent on my computer stops the job and de-registers itself. The job goes back in the controller's queue for processing on whatever the next available machine is.

And you don't have to have any special software for this. It can be done right now with the tools that already exist in Preview 2.

--

I write in my journal
Re:I've been dying to know.... by WoodChuckNorris · 2004-06-23 15:37 · Score: 2, Informative

Aye, you don't need to be running an XCode application. My friend and I were running XGrid on our PowerBooks (867 MHz Ti and 1.5 GHz Al), and he was able to write something that would allow us to use our combined power for Blender rendering. It was rather awesome, breaking 2.something GHz on G4 processors.

Mind you, I don't know how he did it, as I am still a code monkey-in-training.
Re:I've been dying to know.... by genericpenguin · 2004-06-23 17:40 · Score: 2, Interesting

Pardon the expression, but "Not so, not so".

While the Xgrid application does indeed allow you to create custom interfaces for command line programs, there is still the issue of data. Xgrid will start processes on remote machines but as to how data is read and distributed is another matter.

i.e. If you have an application that simply generates data(eg. a calendar) then that would work well with the custom plug-in feature. However, if your program needs to be fed data(eg. sort a list read from stdin), your program would have to have a way of splitting the data and giving it to the appropriate process. To achieve this, you would really have to use the Xgrid API to write your own plug-in.

--
"Why, Johnny Ringo. You look like somebody just walked over your grave." Doc Holliday, Tombstone.
Re:I've been dying to know.... by Graymalkin · 2004-06-23 21:17 · Score: 3, Interesting

Feeding a remote process data is not as difficult as you're describing. If the program you're using doesn't support ranges in arguments it isn't too difficult to wrap a script around it that does understand input ranges. The Xgrid client makes it pretty simple to use ranges as arguments for programs. It's possible to use the likes of Blender and Xgrid to do distributed rendering. Ergo input control doesn't seem to be a terribly difficult hurdle to overcome with Xgrid.

--
I'm a loner Dottie, a Rebel.
Re:I've been dying to know.... by Curious__George · 2004-06-24 10:50 · Score: 2, Informative

He probably did it kinda like this:
http://www.atpm.com/10.06/blender.shtml

--
***General Consultant to the Human Race*** My opinions are free. You get what you pay for.
Re:I've been dying to know.... by Graymalkin · 2004-06-24 23:29 · Score: 2, Interesting

You're trying to shoehorn the wrong sort of application into Xgrid and for some reason expecting it to work. Problems that can't be solved in a loosely coupled parallel environment are not well suited to be run on something like Xgrid. In the case of your fax numbers a custom Xgrid plug-in would probably be the least efficient way of doing the work. It would likely take longer to distribute the list and binaries to agents than it would for one fast system to run through the program.

Xgrid isn't meant to solve all computational problems. It is designed to solve the ones involving long independent or at worst loosely coupled problems.

--
I'm a loner Dottie, a Rebel.

How would that work? by Anonymous Coward · 2004-06-23 11:30 · Score: 5, Funny

[iMac] GOOD MORNING

[Me] Good morning, computer. How are you?

[iMac] PRETTY GOOD. I SOLVED A VEXING SCIENTIFIC PROBLEM LAST NIGHT.

[Me] Oh is that so.

[iMac] YEAH. I FOUND A SOLUTION TO THE HEISENBERG-BERTELLSMAN PROTEIN FOLDING DELIMMA.

[Me] Huh.

[iMac] THE ANSWER TURNED OUT TO BE 42.

[Me] .. That's... nice. So how about some Doom 3 then?

[iMac] OK

Re:How would that work? by Anonymous Coward · 2004-06-23 12:08 · Score: 5, Funny

[iMac] ALSO, MY SPELL CHECKER NEEDS AN UPGRADE.

[Me] Ok. Also, could you not use so many caps? It's like YELLING.

So could someone please inform me by Anonymous Coward · 2004-06-23 11:38 · Score: 2, Insightful

How the developers actually benefit from OSS. The way I see it is that these people put the time and effort in to make a great product - which they give away for free.

Large corporations then download and use these products to increase productivity, get better results without paying a cent, but possibly making themselves even richer in the process. This isn't a troll, i'm just after an answer. I'm not saying OSS is bad, but i'm curious as to what motivates developers.

Re:So could someone please inform me by Anonymous Coward · 2004-06-23 11:40 · Score: 3, Insightful

How the developers actually benefit from OSS. The way I see it is that these people put the time and effort in to make a great product - which they give away for free.

In this case, Apple, the developer of XGrid, is benefiting because in order to use XGrid you have to buy hardware. Apple sells hardware.
Re:So could someone please inform me by tupps · 2004-06-23 12:38 · Score: 5, Interesting

Stop thinking of developers as individuals who are trying to sell a product and think of developers as people who work/contract for organisations.

Instead of buying a product that is 95% of what I want I can take a OSS package that is 90% of the way there and pay a developer to customise it to exactly my needs. Now I have a solution that is perfect for my business, maybe given something back to the OSS community. While if I had bought the product I would probably have to change my business to use the product. The company now is also free of licensing and upgrade issues. Also they do not have to worry about the vendor going out of business or introducing a new version with no support for the old version.

If you think of software as tools for business rather than something that a developer trys to sell OSS makes a lot more sense.

--
Go out and get sailing!
Re:So could someone please inform me by Anonymous Coward · 2004-06-23 12:42 · Score: 5, Insightful

Maybe that's not the right way to look at it. The way I see it is that these people put the time and effort into making great tools, not end product. Now they, and everyone else, are free to use those tools to create great products, which they don't give away for free.

The mistake I see in every Microsoft attack on OSS and the great fallacy behind every purchased white-paper that predicts that OSS will destroy the economy is that writing and selling software is only a very small part of the economy! Most of the economy is involved in creating real, tangible things like cars and planes and food, etc, etc. Most of the economy is not involved in endlessly copying and selling the same pattern of bits.

OSS creates tools that promise to improve the creation of many, many things on this planet and improve the prosperity of all. The only ones threatened by this are those that have made a business of monoplizing ideas. Ideas that are so easy to duplicate or recreate that they are deliberately trying to setup and use the force of law to keep people from producing ideas on their own.

OSS is really a "paradigm shift". This phrase has been used so emptily so many times by senseless marketing droids that it has lost impact over the years. But it is here, it is now, and it is unstoppable. How can they stop it? We have the source!
Re:So could someone please inform me by eamacnaghten · 2004-06-23 13:08 · Score: 5, Insightful

I am involved in both proprietary and open software.
In the proprietary model the software is becoming worth less and less. 5 years ago run time licenses accounted for over 80% of the income of commercial software provider companies, now you will be lucky to see it account for 40% and it is going down rapidly. The rest being made up of support, training and other services.
However, the cost of producing software is the same, and what is more, it is an upfront cost. You cannot get money for it until after you have paid a programmer to write it.
Open source takes the above to a logical conclusion. As software is becoming relatively worthless (as far as run-time licenses go) you do not lose by giving the software away for free, and if you Open Source it you have available a 90% solution from free software out there before you begin thus cutting down on the production costs.
It is not about "giving stuff away" or people "not paying a cent" to use your software, it is about facilitating an extremely cost effective way for which software companies can provide services to the customer by using open source predecessors, and passing the benefit on to successors.

--
Web Sig: Eddy Currents
Re:So could someone please inform me by Anonymous Coward · 2004-06-23 13:09 · Score: 5, Insightful

Well, when I develop a piece of software (or hire someone to do it for me), I solve one of my problems. That's the benefit. End of story, really.

For instance, I need a special library for an app. And none of the off-the shelf ones exactly match it. So I write it.

Now, I find out that other people have a similar problem. So I think to myself "well, I already got my ROI, so to speak. I solved my problem. So now I'll put this software out as open source and see what happens".

And people use the software in ways I didn't think of. They give suggestions on how it might be better. A few send in patches. Suddenly my solution is an even better solution, at no cost to me.

On the flip side: I download an open source library. It works okay, but there are some bugs and it needs a little refactoring. It will cost $1000 in labor to fix this library, vs. $5000 to write it from scratch. So I do it and send the patch to the author. The author is happy (free patch), I'm happy ($4000 worth of code for free), and I don't have to re-do my fixes in the next version! I sure wish commercial software worked that way.

A lot of folks make it seem that OSS is a bunch of people working for others, for free, like communists or something. Not true .. I write software *to benefit myself only*. I am a capitalist. I fully believe in free markets. I believe people should make as much as they can and get to keep it all. I also believe there's no justification for charging for something that costs nothing to copy, so I don't. It goes against my thinking: the only way something that costs nothing can be charged for is if you have authoritarian government enforcing it (which we do). Charge for service, sure. Charge for installation, sure. Charge for consulting, yup. Charge for the box, the CD, whatever. All of that takes time or materials and I can't "copy" it for the next guy.

Of course, you don't have to explain *how* OSS works. Just look and see that it exists and is self-sustaining, that's enough to prove that *something* about it works!

It Doesn't Show That At All! by Alphanos · 2004-06-23 11:43 · Score: 3, Insightful

Just goes to show that when companies embrace open standards and code, the world doesn't fall apart.

Don't get me wrong, I support open standards/code, but it doesn't show any such thing if this linux client has only just been released. I bet Apple, and others for that matter, will be watching sales of Mac machines for use in clusters. If they drop because everyone starts using linux PCs, then Apple will probably not try this again.

--
Alphanos

Re:It Doesn't Show That At All! by xchino · 2004-06-23 11:55 · Score: 4, Insightful

If everyone was going ot be using Linux PC's in their cluster, they would just use one of the existing clustering applications. The only reason anyone would use Xgrid is because they plan on using SOME mac nodes, which is better than none. This could increase sales by opening up Macintosh hardware to projects that couldn't use it before. Due to the cost of a Mac hardware, it is often not feasible to build an all Mac cluster, but if I can throw some G5's in here and there Apple gets some of my money as opposed to none.

--
Everyone is entitled to their own opinion. It's just that yours is stupid.
Re:It Doesn't Show That At All! by aristotle-dude · 2004-06-23 12:21 · Score: 5, Informative

Actually, no it is quite feasible if you do it on a large scale and depending on what you use the cluster for. Big Mac and the Army cluster are two examples of where a mac cluster can be cheaper.

--
Jesus was a compassionate social conservative who called individuals to sin no more.

Home cluster by vaguelyamused · 2004-06-23 11:46 · Score: 3, Interesting

I wonder how effective this really is for home use? Will the performance improvement on my Powerbook be worth running XGrid on it and firing up a couple older computers (600Mhz IMac, Pentium III 1.0 Ghz) on Linux/OS X and adding them to the cluster. Would 100Mbs Ethernet cut it, what about WLAN?

--
STOP ROCK VIDEO

but.... by jwcorder · 2004-06-23 11:48 · Score: 5, Funny

"Just goes to show that when companies embrace open standards and code, the world doesn't fall apart."

But the world hasn't fallen apart using Microsoft either...oops, I said that outloud....

--
http://jayceecorder.blogspot.com

Re:but.... by javaxman · 2004-06-23 12:15 · Score: 2, Insightful

what world are *you* living in that hasn't fallen apart over the past few years??
I might like to move there, but I suspect, like some other folks, you've simply stopped following the news...
So, you're saying your PCs are completely problem-free? You don't get tons of spam and haven't heard of major web hosting services DDoSed by zombified Windows users? Huh.
Re:but.... by Tjp($)pjT · 2004-06-23 13:46 · Score: 4, Funny

It is OK. You didn't utter the name three times.

--
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!

In general it makes life easy by SuperKendall · 2004-06-23 11:49 · Score: 5, Informative

In the past, as I have moved between jobs, I've written a number of Object->relational mapping tools.

After a while they cease to become fun to write, and you'd rather just get on with writing code that does something instead of infrastructure. By using and contributing to OSS projects, you can use the same code no matter what company you end up at. Because the code is portable it can become part of the package you can offer to a potential employer - they not only get an employee but potentially one that can producive almost right away because they are familiar with the tools they'll be using, with no cost to the company for said tools.

So it makes life easier for you, less re-work. And it makes life easier for employers, as they get richer products sooner. And if the employee becomes really proficient at a widely used OSS project they can write their own way through consulting or training.

--
"There is more worth loving than we have strength to love." - Brian Jay Stanley

GridEngine by Anonymous Coward · 2004-06-23 11:51 · Score: 2, Interesting

I find Sun Grid Engine better than other similar grid tools...

http://gridengine.sunsource.net

Re:GridEngine by dsouth · 2004-06-23 13:28 · Score: 2, Informative

that's because the other tools are PBS, Condor and LoadLever, all complete crap. Grid Engine is also crap, but at least doesn't have the 80s baggage.
What's funny about this is that the actual lineage of GridEngine is DQS-->Codine-->Codine/GRD-->GRD-->GridEngine . So grid engine is indeed carrying that "80's baggage". And for that matter, LoadLeveler is actually a decendant of Condor (though it branched off so long ago I doubt there is much of the orignal left). You also left out the half-dozen or so decendents of NQS that all carry the same baggage. Actually, if I had to choose, I'd say Condor has the least baggage, mainly because it's been targeted for something rather different than what most batch schedulers do (in particular, the classad stuff is an interesting direction).
Personally, I found PBS to be the best open source solution last time I had to choose, but that was just prior to the Sun buyout of GRD, so things may have changed. [My current employer rolls their own batch scheduler, so I haven't had a need to survey the field for a few years.] There are also some things Condor rocks at (cycle scavanging, userspace checkpoint/restart/migration) which none of the others even attempt, so it's definitely worth a look for some sites.
If your paying $$ for your batch scheduler, LSF pretty much trumps all of them, but the price is too steep for me.

great job by rainman1976 · 2004-06-23 11:58 · Score: 2, Insightful

Good job with the clustering ... as for the pro-Mac users that believe that this should not be, keep in mind that the computer is just a tool to simplify a job. Using a pipe on the base of the wrench to solve a problem easier doesn't mean that Sears Craftsman is now going to start making longer wrenches, it just shows that people will use whatever they have to solve/simplify problems, and if it means clustering in non-Mac computers, then so be it. Job done, cheaper, simplier, and quicker. -Rainman

Probably a silly question but... by Thaidog · 2004-06-23 12:07 · Score: 2, Interesting

Can anybody confirm if the linux and unix ports are smp aware?

--

||| I still can't believe Parkay's not butter.

Re:Probably a silly question but... by Novajo · 2004-06-23 12:58 · Score: 5, Informative

Can anybody confirm if the linux and unix ports are smp aware?

(I wrote the xgridagent).

As the other poster said, XGrid does not care what the binary does (so it can be smp aware, multi-threaded, whatever). However, the xgridagent itself is not explicitly smp aware, but it is multi-threaded. Each task is started in its own thread and depending on the OS(?) I guess they could spread to other CPUs. The other aspect of the question is "Does the Unix XGrid agent support MPI like Apple's GridAgent for OS X?". It does not and I can't say for sure how difficult it would be to support it. However, since all communication is done via the XGrid protocol, I don't see what would prevent it from being implemented. BUt other things need to be done first.

The most pressing issue is to fix the annoying "large message" issue which makes the agent hang (while it waits forever for the controller to accept more frames). I am convinced it is trivial, I just don't know enough about BEEP to fix it. I am hoping somebody who knows BEEP will take a look at xgridagent-profile.c and fix the xgridagent_SengMSG() function and send me the patch.

Daniel Côté

Re:Apple embraces opensource? by Anonymous Coward · 2004-06-23 12:32 · Score: 5, Informative

This troll is getting old. MS does not and never did own 40% of Apple. They bought a large chunk of non voting shares in exchange for making IE Apple's default browser. As soon as the 3 year contractual agreement was up, MS sold the shares, and for a decent profit.

Why bother? by jweage · 2004-06-23 12:37 · Score: 2, Informative

There are many other open source cluster/queuing systems available.

The one I prefer is OpenPBS. It works very well for engineering compute clusters, and there are many different resource schedulers available which use the PBS job and node management system.

Re:Why bother? by oudzeeman · 2004-06-23 13:22 · Score: 2, Interesting

OpenPBS is shit. It needs about 25 3rd party patches(many of which have to be applied in order) to be halfway decent, (Altair Engineering doesn't actively develop it anymore so these patches won't be integrated into OpenPBS). Also it often falls flat on its face. I wouldn't want to use OpenPBS unless I had a trivially small cluster.

If you want something free, TORQUE is OK. It is a OpenPBS derivative (they started with the last OpenPBS version and added all the popular scalability and fault tolerance patches). TORQUE is actively developed under some DOE contracts, and even the company that has the DOE contracts to develop and support TORQUE will give other people some free support (I've had them on the phone helping to debug some of the code). You can get TORQUE from Supercluster.org.

PBS Pro is very good, but costs a lot unless you are a degree granting department (then it is free).

Re:Apple embraces opensource? by The+Ancients · 2004-06-23 12:38 · Score: 4, Informative

The only funny thing about this comment is that MS actually owns 40% of apple.

I wouldn't say that - I find it pretty amusing you've been registered at ./ for so long and are still so wrong.

p.s. I know I should reference - how about 'MS owns fuck all anymore' - will this do?

--
The Mothership

Re:Why another technology by Anonymous Coward · 2004-06-23 12:48 · Score: 3, Funny

Is "mac" some guy who works for Apple? Or do you normally refer to companies by their product names?

When will Windows write similar grid tools?
Why does Unix keep suing people?
When will Mac make a Windows box?

Pretty Interfaces.... by Anonymous Coward · 2004-06-23 12:49 · Score: 3, Insightful

Apple is good at writing pretty, easy to use interfaces over complex to configure things. Many of these complex things are written by people who ASSUME the end user has the same skill set as they do. But as we all know... The average computer users is an IDIOT. The average system administrator is usually no better.

Let's face it, some slash-dotters who don't even work as system administrators know more them.

Good for home use too. by Gordon+Bennett · 2004-06-23 12:51 · Score: 5, Interesting

Some households have a mix of computers and one can begin to see the benefits - for example, to halve the video compression time of iMovie when making a DVD.
Considering Apple's ease-of-use for heavyweight *NIX apps this would empower more people to have more computing resources available rather than the big fish out there - schools with low budgets would be able to stretch their capabilities that bit further. And so on.

Re:Good for home use too. by Twirlip+of+the+Mists · 2004-06-23 13:43 · Score: 2, Informative

for example, to halve the video compression time of iMovie when making a DVD.

Video compression is a difficult task to parallelize. If each frame were compressed individually it'd be easy: just and an uncompressed frame to a node and get the compressed frame back. But that's not how it works.

Now, for something like Pixlet, which is frame-based, there's the possibility of distributing the task. But you will never use Pixlet. It was designed to compress 2K or 1080 material losslessly at a ratio of about 2:1. Very specific tool for a very specific purpose.

So using Xgrid for video compression isn't going to be the wonder that you might wish it could be.

--

I write in my journal
Re:Good for home use too. by zalas · 2004-06-23 14:28 · Score: 4, Interesting

Might video compression work if the first scanning pass is done on one computer and the keyframe locations are extracted and then each computer in the grid/cluster would render the chunks between keyframes in parallel?
Re:Good for home use too. by Twirlip+of+the+Mists · 2004-06-23 14:49 · Score: 2, Informative

Maybe. I don't know enough about the internals of how compression works, but that sounds plausible enough to fool me.

--

I write in my journal
Re:Good for home use too. by JamieF · 2004-06-23 18:37 · Score: 2, Interesting

It might work well for a different usage scenario: ripping a DVD.

The first step would be getting the data off of the DVD. Clearly that wouldn't be parallelizable unless you had multiple copies of the DVD (or some sort of hideous DVD multicast SAN).

The second step (decryption) would probably be parallelizable. I don't know much about how CSS works but I imagine that it would be a fairly easy task to split the ciphertext into blocks for distribution and decryption.

The third step (recompression) could be split up for parallelization by scene.

The fourth step would be serial - reassembling the compressed movie into a single file, or maybe a few files if a maximum size (say, 700MB) for each single file were desired.

Ideally, the second and third steps could be combined. Empirically it's clear that it's possible to jump to a scene without decrypting the entire title up to that point, so it should be possible to split the encrypted scene data out, pass it to a node, and get back that same scene in recompressed form.

Splitting things up this way would probably also overcome the problem the parent post describes (that of not wanting every frame to be a key frame). Starting each scene as a key frame would add trivial overhead at worst; at best this is what a serial encoding process would do anyway since the frame content probably changes drastically with each scene change anyway.
Re:Good for home use too. by TheRaven64 · 2004-06-23 22:05 · Score: 2, Informative

Compressing video is very easy to do in a parallel manner. The first step is to perform a DCT (or DWT or similar) on each frame. This is embarrassingly parallel, especially for DCT where each macro block (usually an 8x8 pixel square) can be done in parallel. Next some form of quantisation is applied to the result. This, again, can be done in parallel on a per-frame basis. Finally, delta frames are computed for inter-frame compression on codecs that support it (MPEG and friends). Since key frames are usually a fixed number of frames apart you can simply have a number of nodes running these in parallel for each key frame block. If they are not, then the problem is a little more tricky but basically doable (I'd imagine that you'd start at fixed points, and insert key frames as required on fixed size blocks, possibly requiring some backtracking).

--
I am TheRaven on Soylent News
Re:Good for home use too. by Twirlip+of+the+Mists · 2004-06-24 04:50 · Score: 2, Funny

That's all well and good... but in discussing potential uses for Xgrid, can we please make a concerted effort to stay away from stuff that illegal?

--

I write in my journal

embracing? by dekeji · 2004-06-23 13:20 · Score: 4, Insightful

Xgrid is proprietary, closed-source software. I think that hardly counts as "embracing" open-source software. Many other parts of the Macintosh platform are proprietary and closed source as well.

I'm not disputing that Apple released Darwin source code. But before you start cheering, keep in mind that Darwin started out as open source: the CMU Mach kernel and bits and pieces of BSD. And it's not like Apple made a big sacrifice in releasing a kernel that looks and feels like half a dozen other open source kernels.

Re:embracing? by DeifieD · 2004-06-23 14:15 · Score: 2, Insightful

Well... At least they are releasing software for Linux... I don't think Photoshop would be OpenSource either. But Adobe releasing it for Linux would be considered embracing.
Re:embracing? by babbage · 2004-06-24 01:46 · Score: 2, Informative

Xgrid is proprietary, closed-source software

Actually, that is completely false:
Xcode uses distcc to manage distributed builds. The distcc client manages the setup and distribution of build tasks. The server process (distccd) manages communication with the remote computers hosting the build tasks. The server process runs on both the local, or client, computer and on the remote computer.

As 30 seconds of Googling will tell you, distcc [is] a fast, free distributed C/C++ compiler.
As they have done with KDE's KHTML engine in Safari, so is Samba's distcc engine being used in XCode.
Care to try again ?
:-)

--
DO NOT LEAVE IT IS NOT REAL
Re:embracing? by CableModemSniper · 2004-06-24 05:06 · Score: 2, Insightful

And as 30 seconds of thinking will tell you, Xgrid is not Xcode.

Care to try again?

--
Why not fork?
Re:embracing? by jurv!s · 2004-06-24 05:30 · Score: 5, Informative

But Xgrid uses the BEEP protocol for all communication, which is open, and allowed this project to interoperate easily. The closed source part of Xgrid is just a Cocoa GUI that was thrown together with Interface Builder. This made it a lot easier to interoperate than say, the nasty Exchange/Outlook communication combo.
If Apple breaks this intentionally (meaning not for adding significant, enhanced functionality) in their next release, I will stand with you as an anti-Apple nay-saying zealot and deride them all up and down /.
-Potentially recovering Mac zealot (it's so hard with WWDC right around the corner :-( )

--
sigs are for fools and trolls. no signature is *always* appropriate. you should turn them off in your preferences.

It's for ad hoc cluster creation... by csoto · 2004-06-23 13:48 · Score: 4, Informative

The other packages require a bit of planning, whereas Xgrid excels at locating nearby resources for pawning off processing tasks. Rendezvous (ZeroConf) is exactly about the need for ad hoc networking. Xgrid extends that to the cluster...

--
There exists no way of exchanging information without making judgments. --Bene Gesserit Axiom

Re:Why another technology by Anonymous Coward · 2004-06-23 13:54 · Score: 2, Interesting

"Mac" "embrased" open standards, and produced an easy-to-use solution with a reasonable GUI that actually has a chance of being adopted by end users. Have you actually tried setting up Globus yourself? It ain't easy, and it doesn't really do the same thing as Xgrid. As for OpenMosix -- on OS X???

wake up! by GISGEOLOGYGEEK · 2004-06-23 15:13 · Score: 2, Insightful

So, this is an example of those open standards, and the world not falling apart over it?

shall I quote from the download page? yes, yes I shall ...

Quote:

Several notes on compilation:

1. If you use this for anything other than testing, you are insane.
2. The configure script isn't great: it does not check for all compatibility issues and might even fail to run properly without telling you. /Quote

I'll assume that Pudge is just another Michael in disguise, endlessly posting over hyped BS articles that are easily refuted.

--
George Bush + Linux = "I will not let information get in the way of the fight against Windows"

Re:Why another technology by kryptKnight · 2004-06-23 16:24 · Score: 3, Informative

"Either that or OpenMosix"
Xgrid treats the cluster as one proccessor, while OpenMosix assigns each to thread to a cpu thats not doing muck work.

--
Facts do not cease to exist because they are ignored. -Aldous Huxley

Re:Why another technology by Anonymous Coward · 2004-06-24 00:50 · Score: 2, Funny

Here is a quick general ( Mac ) knowlege test for Skaunch and anyone else who seems confused the products of 1 Infinite Loop.

Question 1. - Win is to Microsoft as Mac is to...
A. Windows
B. Macintosh
C. Apple
D. MACH
E. Steve Jobs

Question 2. - Mac is...
A. A fruit
B. An operating system
C. An acronym
D. A computer
E. An abbreviation

Question 3. - "RDF" as made famous by Steve Jobs is...
A. An interconnect protocol (RAM Double Frequency)
B. Relational Database Function
C. A QuickTime codec
D. Reality Distotion Field
E. Real Dumb F*cker

Question 4. - During the 1990's Apple was described in the press as...
A. The beleagered computer maker
B. The beleaguered computer maker
C. The beleguered computer maker
D. The biggest computer maker
E. The best computer maker

Question 5. - The official name of the Virginia Tech G5 cluster was...
A. System X
B. Mac
C. Big Mac
D. X
E. Virginia

Question 6. - The X in Mac OS X is...
A. A letter of the alphabet symbolizing the unknown
B. Refering to the uniX core of the OS
C. Used because they couldn't think of a name after Rhapsody
D. Implies the seXy "lickable" Aqua interface
E. A Roman numeral following 9

Question 7. - Which is true?
A. Gates runs Microsoft which makes Windows computers
B. Gates runs Windows which makes Microsoft
C. Jobs runs Mac which makes Apple Computers
D. Gates runs Apple which makes Macintosh computers
E. Jobs runs Apple which makes Macs
F. A and E
G. Gates doesn't make computers

ANSWERS:
1.C 2.B,D & E 3.D 4.B 5.A 6.E 7.E & G
SCORE:
7 = Mac "Zealot"
6 = Mac "Fanboy"
5 = Mac "Enthusiast"
4 = Mac "Weenie"
3 = PC "Weenie"
2 = Windows user
1 = "Luser"
0 = Why are you here on /. ?

Re:Apple by tbone1 · 2004-06-24 05:15 · Score: 3, Informative

Yeah, everyone on /. says Apple is so great. But they just rip stuff off from the open source community. When is Apple going to give something back?

Well, there's Darwin, their (improved, IMnsHO) version of BSD.

Rendezvous is their (improved) version of ZeroConf.

Safari runs on the KHTML engine. Apple made some improvements and gave them back to the KHTML people, who thanked and praised Apple.

They've worked to improve gcc on PPC-based compilers.

They also provide the standard tools like apache, perl, python, etc etc etc, with OS X. I don't know if they have worked on these specifically, but it wouldn't surprise me in the least.

--

The Independent: Reverend Spooner Arrested in Friar Tuck Incident - ISIHAC, Historical Headlines

Receiver swaps. by tlambert · 2004-06-24 09:28 · Score: 2, Informative

Receiver swaps.

In DCE RPC, the receiver does the byte swapping, if necessary. One of the main reasons Windows network services are built on DCE RPC is that between homogenous systems, there's no swapping taking place: all that data goes out in host byte order, and there's no such thing as network bte order.

One of the big arguments about this had to do with Windows machines on Intel not "playing fair" with systems that natively implement network byte order as their host byte order. When talking to Intel boxes, these machine have to gain additional overhead.

This also gives a big disadvantage to servers whose byte order doesn't match that of their predominant clients.

Actually, from a computational overhead point of view, a more correct approach would have been to have "client swaps to seerver byte order", to put the computational overhead on the most efficient side of the link for it (by offloading the most computationally loaded component, the server).

As far as I recollect, this lost out in committee to people who were arguing against it in order to have leverage to enforce vendor lock-in for both clients and servers. 8-(.

-- Terry

77 of 219 comments (clear)