Slashdot Mirror

but that is ok! rather than asking quickie questions and expecting quickie answers, you can start by learning the difference between a system administrator and IT professionals.

"user friendliness" is proportional to sysadmin's abilities or proportional to $$$ for commercial tech support

1. A (stale) link to get you going on hpc clusters: http://www.hpccommunity.org/section/kusu-45/

2. http://www.platform.com/ - Dell's/Redhat official hpc cluster (at least a couple of years ago) which was based on kusu (see previous link). In other words, RH was(is?) using a third party for their RH HPC - correction needed if things have changed. - a great yo-yo system (DellRedHatPlatform) in case you have issues.

3. http://www.caoslinux.org/

Up until now, I'd assumed that most corporate developers were using Fedora, simply because of its similarity to Red Hat Enterprise and for its maturity. However, I'm curious to know, for those fortunate enough to develop for embedded Linux, what distribution do you expect to be supported for a build environment?

When I worked for ARM (quite recently) they could work with most platforms, since all the heavy lifting was submitted to a cluster of machines and if you wanted a specific OS/distro/version you could specify it and the job would go to that machine (e.g. bsub -I -r "solaris" foo would queue command 'foo' to run on a solaris box).

However, the machine they had the most of was RHE3 (Red Hat Enterprise edition 3), so that's probably a good one to target.

Other good ideas for your tools include:
* Batch/command-line mode, so your tool can be called by a script.
* Parallelisability, if your jobs take more than ~10 minutes on a high-end machine. It's easy to get 4 * 3GHz cores, but hard to get 1 * 12GHz core. Xilinx place and route, this means YOU.

There is also a commercialized version of Rocks called Platform Rocks which is based on SDSC's Open Source release which provides enterprise support and provides other goodies. You can find it at http://www.platform.com/products/rocks/ -ShawnX

Since the article doesn't really have to do with grid computing. Here are some real Grid Computing links.

Globus Toolkit
LSF
openPBS
gridengine
OSCAR
ROCK MPP
maui

and last but not least: beowulf cluster
---

GNU Queue offers batch scheduling for clusters of computers; however, a cluster only needs to contain a single computer.

One additional commercial tool we use where I work is Platform Computing's Load Sharing Facility. It works well, but it's expensive (read "over priced") and I suggest you try something else first.

"However, these add-on clustering solutions come from various sources, do not conform to any set standards, and are often implemented on a particular Linux distribution."

What are they talking about?

Hmm, MPI and PVM are standards, more so they are _the_ standards, and are supported in Linux beowulf clusters.

Even if it were true, with Linux clusters in the top50 (not a typo) of supercomputers, whatever they use for clustering is a standard on its own. Beowulf so widely used that it is a de facto standard too.

The MS stuff is not a standard and only implemented on a particular MS distribution...

Actually, LSF runs on RedHat, Suse, OpenLinux, TurboLinux (LSF v4.1), Debian, and Suse. It even says 'tested with', so it doesn't even force you to use one of those. So which particular Linux distribution did they miss (ok, mandrake and gentoo)? And which "potentially financially unstable Linux vendor" does that bind you to if it'd very well possible to run it on the other distributions, just not tested by the supplier?

Maybe MS thinks clustering is mainly failover, but that's much less valid for stable operating systems.

This new page will not survive long either.

Talk to the folks at Platform.

Software such as Platform's LSF take care of this magicly... it even allows for checkpointing, assuming your task allows it. Because my render software didn't really do checkpointing, I had to add that in to my wrapper code.

We do use desktops at night to work with our render farm. Platform has some cool tools to work with for such environments. I have never tried LSF in conjunction with PVM or MPI but they have support for it, so I imagine it does pretty well.

I fail to see what is impressive about this.
It looks like the wheel reinvented several times.
For cluster installs on several machines, use system imager .
For using and controlling a cluster of machines for various taskes, use LSF .

The number of machines is pathetic too ... 225 @ 733 mhz? That makes it to #325?
How sad. I need to bench mark our render farm (200+ boxes, 120 are dual 1ghz) and see what we can come up with. I know it is higher than that... and we have a smaller install for the industry.

I looked for info to spec our machines but I couldn't find any info.... any help?

Batch systems have been around a long time in the HPC world. Gridware was orginally developed by GENIAS Software GmbH. GENIAS produced a batch scheduler called Codine, which was a commercial version of DQS. In fact, Sun's Grid Engine FAQ even states that Sun Grid Engine is a new name for CODINE.

Of course, DQS/Codine/Grid isn't the only batch-scheduling/cycle-scavenging game around. Other players are:

• Condor
• openPBS and it's commercial version PBS Pro
• Load Leveler (which IIRC is IBM's commercial implementation derived from Condor)
• LSF which is the product Sun was previously co-marketing until they purchased Gridware (probably because of the high per CPU cost of LSF).
• and lots of others that I've forgotten, many based on the once-common NQS/NQE batch system.
• There are also systems like Legion that represent a sort of ``next step'' computing enviroment.

Many of these predate newcomers like SETI@home and Mosix by serveral years. Most also provide hooks into parallel computing APIs like MPI, PVM, openMP, or something similar.

Batch scheduling and cycle-scavening are old concepts. Having wasted away my years as a graduate student submitting large quantum chem jobs to Crays, it's nice to see lots of groups continuing to squeeze every useful cycle out of existing hardware. Sun's recent annoucements are just the latest update to an old product---not a new idea, and not a Mosix/SETI rip-off.

Gridware is definately an also-ran in the distributed environment at the moment. Companies who use this type of product generally go for LSF, a commercial product. Many big IC design houses use LSF to use the spare cycles from workstation cpu's + to run their compute farms. Sun is one of these companies. If you want a GPL equivalent go for GNQS.

Having just purchased a linux render farm, I can really appreciate this article. We went through the process of determining what the best solution was for our system and for the software that we use for rendering (A|W's Maya) and for load balancing (Platorm's LSF) was to expand to linux boxes and use the same software.

We explored Beowulf, but after talking to those that are in the know, Maya's tile renderer is not well suited to a Beowulf system.
I looked at other solutions as well, but due to shared memory and the network bottle neck, nothing could take what we saw as a distributed system and turn it in to a parallel system.

By using a load balancing cluster, we are given the opertunity to render multiple frames at the same time, giving us a speed advanteage. This uses more overall memory than a massively parallel beowulf cluster, but it keeps the speed gain of a parallel system the same. The overhead exists for scene file loading becuase that is done on every machine, but it takes minutes when rendering takes hours. A fair trade.
The distributed system needs horsepower and memory more than network speed or file system speed. It is true that an increase in those will speed up the process, but the money is better spent in CPU and mem concerns. Our systems are all dual 600 mhz with a gig of ram per box. It may seem extreme but from our SGI render benchmarking, the scenes that we render can take over 500-600 mb of system memory.

Is it worth the cost?

We are taking our current render system of SGI boxes, which currently are used as desktops durring the day and render boxes at night and adding full time render boxes as well. The cost comparison of a linux render box can be seen in the hardware price alone. We are using these linux boxes to keep par with boxes that cost at least 3x's as much.
The only disadvantage is that the linux boxes can not be rolled out to desktop systems when new hires arive, where as the SGI boxes can. This is due to Maya's modeler being SGI/NT only and our support of Maya on the SGI only.

All in all, in our situation, a linux cluster is a God send, allowing me to have more horsepower and to allow the company to save money.

Clustering far predates Beowulf or Linux; at the University of Toronto, we've been doing clustering work for years, and some of it has been successfully commercialized, e.g. LSF. The techniques are fairly straightforward, and there's no real way to keep any particular country from building a compute cluster. Of course, a cluster is not a supercomputer: as Henry Spencer once put it, you can get a lot more work out of a couple of stout oxen than a hundred chickens.

Some of the more interesting pieces, like LSF, are only licensed by Sun, thus will not be included in this `deal.' (For a free improvement over LSF, check out GNU Queue. If it doesn't do something you want, you can support the community and extend it.) If you read the announcement carefully, you'll see that the only new codes to which it applies are the parallel file system (the Sun CTO thinks distributed file systems are dead, anyways), the Prism debugger, and the parallel run-time environment.

Of those, the only with no available substitute is the debugger. The ROMIO library is a good place to start for the MPI file I/O stuff (a good database would be a better place, imho). I already mentioned queue management software. The Ptools Consortium and the Globus Project have links to other HPC cluster tools.

Many of the pieces for debugging are available (combine ddd and gnuplot), but some notable ones are missing. The ability to control multiple GDBs easily from one processes and the visualization of parallel execution are needed, and quite difficult to implement. There seems to be interest in making GDB easier to use from other processes, which is a good start towards solving the larger problem of general, distributed debugging. And both the mpich and LAM MPI implementation have some profiling information, but few tools to dig through it.

To be fair, Sun has contributed (and supported contributions) to the original packages. Why they are releasing the rest under their Exploit the Community license is beyond me.

Jason, ejr@cs.berkeley.edu