Linux Clusters Explained

tramm writes: "As someone who works on massively parallel Linux clusters everyday, I get tired of explaining why it is not 'Just another Beowulf'. Linux World has a good article on the four major types of Linux clusters. Our work is in supporting scientific codes that have a high degree of communication. This requires a very different system from the standard Beowulf-class machines that excel at the 'embarrasingly parallel' codes that do not require as much communication. The cost of the network interconnect for a high performance cluster is vastly more than that of a generic 100base-T system."

Comments

Yup.

[-brazil-]

Preach on. The need for really fast node interconnection is what still makes people buy honest-to-god supercomputers. Look at this baby that was just installed next door from me.

No Linux clustering project will ever reach the performance of such systems (though some of them might eventually run Linux), but the low-end high performance computing market (yes, I know it sounds oxymoronish) is bound to be taken by Linux.

Re:Yup.

[Zurk]

it depends..fiberoptic interconnects are already dropping in price and if you can get a fast enough bus on your cluster nodes and a fast enough interconnect it might catch up to traditional supercomputers. e.g. if you have 256 SGI octane machines with internal gigabyte per second class databuses connected via improved fiber interconnects (say 16Gbps cards & terabit switch connects) it would equal your supercomputers internal bus. granted, this thing is a bit far off, but not as far off as some people think.

Re:Yup.

[sjames]

No Linux clustering project will ever reach the performance of such systems (though some of them might eventually run Linux),

With standard Wintel hardware, agreed. I predict that the hardware will migrate to comodity componants as the off the shelf cluster becomes more popular. In many supercomputers, the CPUs are nothing special, it's all in the interconnect hardware. The real bottleneck at the moment is the PCI bus. However, 64 bit 66 Mhz busses are becoming a bit more common now. The rollout of 64 bit CPUs will speed that along. AGP shows some promise. Clearly, the industry is realizing that 33 Mhz 32 bit PCI isn't fast enough. Bridge chips are getting smarter. The next step is to replace the bus with a crossbar switch driven by the bridge chips and providing backwards compatability for existing PCI devices. I also wouldn't be surprised to see gigabit serial devices connected to the FSB soon.

At that point, the gap between custom supercomputing hardware and commodity clusters will be much smaller. Every time the interconnect gets faster and lower latency, the problem set that can be handled by commodity clusters expands.

Clustering

[Signal 11]

I suppose you don't consider putting a dozen webservers into a round-robin a cluster, either...

Well, anyway.. "clustering" can mean just about anything to anyone, given the context. A system administrator probably thinks clustering is taking the same network resource and distributing it across several machines (which typically appear on the network as one logical object). A physics major might think clustering is a huge room full of SGIs all doing rendering and particle analysis... and an anonymous coward might think that clustering allows them to download pictures of natalie portman pouring hot grits down her pants. The point is that it means different things to different people.

Boiled down to the basics, a cluster is simply a group of machines working towards a common goal (whether it be filesharing, parallel computations, or whatnot). There may only be 4 types now, but next week, there'll be 6.

Strategy

[Bob(TM)]

The beauty of the Beowulf type solution is that you can tailor the configuration to the problem.

In the past, you had to shell out beaucoup bucks to just get out of the shoot. With a Beowulf strategy, you can make hardware improvements in concert with different software approaches to achieve an optimal price/performance approach.

They didn't talk about...

[rho]

Nerd Clusters, which are more widespread than Beowulf, and more scalable.

For example, a Nerd Cluster, using ChineseTakeOut messaging, are often used in last-minute, panic-striken Intranet roll-outs, yet each node of a Nerd Cluster can answer simple management questions such as, "Hey, my PC at home crashes all the time. How can I fix it?"

Nerd clusters are, however, more dangerous to operate. If, for example, you say "Let's migrate our core applications from Solaris to NT", you run the risk of massive memory leakage as individual Nerd-nodes began to prioritize jobs such as "update_resume" over your request queue.

Nerd clusters need a "master" node as well. These can generally be identified by their bushy beards, or a long string of nodes queueing up to beg for static IPs.

Step 1 of Big Computer Creation....

[Arctic Fox]

Come up with a cool name.

If i was on one of these teams,
the beast would be called...

COCK
Centralized
Organizational
Compuational
Kluster.

A leading female computer scientist was over heard saying, "My, what a big COCK you have!"

"University of ________ researches have built the world's largest COCK."

"Some bugs were found in the COCK today."

It could go on for days.

Be thankful you are not my student. You would not get a high grade for such a design :-)

They forgot Condor...

[epaulson]

Condor, from the University of Wisconsin, should have been listed on page two. Condor is a high-throughput computing system, that runs on UNIX (virtually all flavors) and NT. We support MPI and PVM. We can run regular jobs, or you can relink with our libraries and get transparent checkpointing and remote I/O. You can use sockets in your job.

You don't need to have a dedicated cluster - Condor started life as a scavenger of idle workstations. We run Condor on every workstation here at CS, and routinely recover several thousand CPU-hours a day that otherwise would have been wasted. You can configure Condor to run with any policy you want on a per-workstation level - only run jobs at night, only run jobs from this group, only run jobs if the wind is blowing from the west - whatever makes sense to the workstation's owner.

Best of all, we're free-as-in-beer.

If you have any questions, send us mail at condor-admin@cs.wisc.edu

Nice to see...

[zorgon]

...that SOMEONE out there at least is using Linux and clustering technology(ies!) to do some real work (pause to don asbestos gumby suit), as opposed to merely sucking down Internet bandwidth. Which is what I'm doing right now. Oh no!
SIGHYPOCRISY received: Dumping core
panic: Hypocrisy error in SIMM 0x0B
panic: Hypocrisy error in SIMM 0x0B
panic: Hypocrisy error in SIMM 0x0B
panic: Hypocrisy error in SIMM 0x0B
Syncing filesystems... [11][8][6][3.14159][0][0][0][0][0][0][0][0][-1]
System Halted
Press any key to reboot

Good in theory, good in practice

[tolldog]

Having just purchased a linux render farm, I can really appreciate this article. We went through the process of determining what the best solution was for our system and for the software that we use for rendering (A|W's Maya) and for load balancing (Platorm's LSF) was to expand to linux boxes and use the same software.

We explored Beowulf, but after talking to those that are in the know, Maya's tile renderer is not well suited to a Beowulf system.
I looked at other solutions as well, but due to shared memory and the network bottle neck, nothing could take what we saw as a distributed system and turn it in to a parallel system.

By using a load balancing cluster, we are given the opertunity to render multiple frames at the same time, giving us a speed advanteage. This uses more overall memory than a massively parallel beowulf cluster, but it keeps the speed gain of a parallel system the same. The overhead exists for scene file loading becuase that is done on every machine, but it takes minutes when rendering takes hours. A fair trade.
The distributed system needs horsepower and memory more than network speed or file system speed. It is true that an increase in those will speed up the process, but the money is better spent in CPU and mem concerns. Our systems are all dual 600 mhz with a gig of ram per box. It may seem extreme but from our SGI render benchmarking, the scenes that we render can take over 500-600 mb of system memory.

Is it worth the cost?

We are taking our current render system of SGI boxes, which currently are used as desktops durring the day and render boxes at night and adding full time render boxes as well. The cost comparison of a linux render box can be seen in the hardware price alone. We are using these linux boxes to keep par with boxes that cost at least 3x's as much.
The only disadvantage is that the linux boxes can not be rolled out to desktop systems when new hires arive, where as the SGI boxes can. This is due to Maya's modeler being SGI/NT only and our support of Maya on the SGI only.

All in all, in our situation, a linux cluster is a God send, allowing me to have more horsepower and to allow the company to save money.