Slashdot Mirror
Linux Clusters Explained
tramm writes: "As someone who works on massively parallel Linux clusters everyday,
I get tired of explaining why it is not 'Just another Beowulf'.
Linux World has a good article on the four major types of Linux clusters. Our work is in supporting scientific codes that have a high degree of communication. This requires a very different system from the standard Beowulf-class machines that excel at the 'embarrasingly parallel' codes that do not require as much communication. The cost of the network interconnect for a high performance cluster is vastly more than that of a generic 100base-T system."
No Linux clustering project will ever reach the performance of such systems (though some of them might eventually run Linux), but the low-end high performance computing market (yes, I know it sounds oxymoronish) is bound to be taken by Linux.
The illegal we do immediately. The unconstitutional takes a little longer.
--Henry Kissinger
Well, anyway.. "clustering" can mean just about anything to anyone, given the context. A system administrator probably thinks clustering is taking the same network resource and distributing it across several machines (which typically appear on the network as one logical object). A physics major might think clustering is a huge room full of SGIs all doing rendering and particle analysis... and an anonymous coward might think that clustering allows them to download pictures of natalie portman pouring hot grits down her pants. The point is that it means different things to different people.
Boiled down to the basics, a cluster is simply a group of machines working towards a common goal (whether it be filesharing, parallel computations, or whatnot). There may only be 4 types now, but next week, there'll be 6.
The beauty of the Beowulf type solution is that you can tailor the configuration to the problem.
In the past, you had to shell out beaucoup bucks to just get out of the shoot. With a Beowulf strategy, you can make hardware improvements in concert with different software approaches to achieve an optimal price/performance approach.
The little guy just ain't getting it, is he?
Nerd Clusters, which are more widespread than Beowulf, and more scalable.
For example, a Nerd Cluster, using ChineseTakeOut messaging, are often used in last-minute, panic-striken Intranet roll-outs, yet each node of a Nerd Cluster can answer simple management questions such as, "Hey, my PC at home crashes all the time. How can I fix it?"
Nerd clusters are, however, more dangerous to operate. If, for example, you say "Let's migrate our core applications from Solaris to NT", you run the risk of massive memory leakage as individual Nerd-nodes began to prioritize jobs such as "update_resume" over your request queue.
Nerd clusters need a "master" node as well. These can generally be identified by their bushy beards, or a long string of nodes queueing up to beg for static IPs.
Potato chips are a by-yourself food.
If i was on one of these teams,
the beast would be called...
COCK
Centralized
Organizational
Compuational
Kluster.
A leading female computer scientist was over heard saying, "My, what a big COCK you have!"
"University of ________ researches have built the world's largest COCK."
"Some bugs were found in the COCK today."
It could go on for days.
Be thankful you are not my student. You would not get a high grade for such a design
You don't need to have a dedicated cluster - Condor started life as a scavenger of idle workstations. We run Condor on every workstation here at CS, and routinely recover several thousand CPU-hours a day that otherwise would have been wasted. You can configure Condor to run with any policy you want on a per-workstation level - only run jobs at night, only run jobs from this group, only run jobs if the wind is blowing from the west - whatever makes sense to the workstation's owner.
Best of all, we're free-as-in-beer.
If you have any questions, send us mail at condor-admin@cs.wisc.edu
Yes, the current switch adapter is a PCI card. There is also a microchannel version since there's still a lot of MCA based nodes out there.
However, IBM has said that the PCI bus is too limiting and the next generation switch will have a special interface (probably right on the CPU bus).
...that SOMEONE out there at least is using Linux and clustering technology(ies!) to do some real work (pause to don asbestos gumby suit), as opposed to merely sucking down Internet bandwidth. Which is what I'm doing right now. Oh no!
SIGHYPOCRISY received: Dumping core
panic: Hypocrisy error in SIMM 0x0B
panic: Hypocrisy error in SIMM 0x0B
panic: Hypocrisy error in SIMM 0x0B
panic: Hypocrisy error in SIMM 0x0B
Syncing filesystems... [11][8][6][3.14159][0][0][0][0][0][0][0][0][-1]
System Halted
Press any key to reboot
I am quite civilized, and I should be brought a beer immediately. -- Bruce Sterling
I want a system that allows every CPU on my network to be available to every process on the network as if it were another CPU in the same machine. Sort of Distributed Multiprocessing.
I want a system where I can add a HD to any machine on my LAN and have that added into a single pool of diskspace much like multiple drives are attached to the root filesystem in Unix.
Then in this model every computer consists of a "CPU server" a "Disk server" and a terminal with attached peripherals, like keyboard, mouse, scanner, joystick etc. The whole network is literally a single high availability computer.
If I set fire to a particular box then the computer/network just doesn't use those resources anymore. When I replace that box the whole network/computer is faster/has more space seamlessly.
Every application sees a "simple" multitasking environment. It tries to execute on the local node. (i.e. the one the terminal is connected to) and draws resources as needed from any other nodes.
The entire thing should be asymetric so if I try to run Quake on a 386, it just runs out to the network right away and uses additional CPU's to run possibly even assigning the whole process to another more capable CPU like the athlon in the next room.
Don't post innacurate information
If you do, I swear by my pretty floral bonnet I will end you.
Having just purchased a linux render farm, I can really appreciate this article. We went through the process of determining what the best solution was for our system and for the software that we use for rendering (A|W's Maya) and for load balancing (Platorm's LSF) was to expand to linux boxes and use the same software.
We explored Beowulf, but after talking to those that are in the know, Maya's tile renderer is not well suited to a Beowulf system.
I looked at other solutions as well, but due to shared memory and the network bottle neck, nothing could take what we saw as a distributed system and turn it in to a parallel system.
By using a load balancing cluster, we are given the opertunity to render multiple frames at the same time, giving us a speed advanteage. This uses more overall memory than a massively parallel beowulf cluster, but it keeps the speed gain of a parallel system the same. The overhead exists for scene file loading becuase that is done on every machine, but it takes minutes when rendering takes hours. A fair trade.
The distributed system needs horsepower and memory more than network speed or file system speed. It is true that an increase in those will speed up the process, but the money is better spent in CPU and mem concerns. Our systems are all dual 600 mhz with a gig of ram per box. It may seem extreme but from our SGI render benchmarking, the scenes that we render can take over 500-600 mb of system memory.
Is it worth the cost?
We are taking our current render system of SGI boxes, which currently are used as desktops durring the day and render boxes at night and adding full time render boxes as well. The cost comparison of a linux render box can be seen in the hardware price alone. We are using these linux boxes to keep par with boxes that cost at least 3x's as much.
The only disadvantage is that the linux boxes can not be rolled out to desktop systems when new hires arive, where as the SGI boxes can. This is due to Maya's modeler being SGI/NT only and our support of Maya on the SGI only.
All in all, in our situation, a linux cluster is a God send, allowing me to have more horsepower and to allow the company to save money.
-I just work here... how am I supposed to know?
There was a cool project at Berkeley called Global Layer Unix that was the backbone of the NOW (Network of Workstations) project.
There were two design goals: operating system independance and program transparency. You could install and run the GLU daemons and programs on any system and programs would be able to run on any node in he cluster without knowing about it (of course, for special GLU features there was a library of routine to use).
The GLU project purposely didn't make any kernel modifications to help in portability. A few of the research papers gave ways to modify the kernel to help performance or feature set, but that wasn't the goal.
GLU also supports another Berkeley project called Split-C, which is a parallel extension of the C language. Use this and you do not need to use the GLU library, gcc will generate appropriate code (I think that is how it worked?).
http://now.cs.berkeley.edu/Glunix/glunix.html
http://now.cs.berkeley.edu
> "The limits on today's Internet are no longer determined by raw bandwidth, but rather by how
> well the different network components work together," said Brian Valentine, senior vice
> president of the Windows Division at Microsoft.
It is clear to me that Microsoft and I do not share the same uplink provider...
For instance, Deja uses a large cluster of Linux machines to index their News feed.
Their software is custom written, though (Including the databases)
It's the same situation at most search engines - Google uses multiple machines for indexing and searching.
It's going to be a problem finding a commericial database that will allow you to distribute it over multiple Linux machines, though.
For that kind of work, you probably don't want a cluster, but a Sun E6500 or a similar machine instead. They are built specifically for buisness applications, like huge databases and that sort of thing. Clusters are generally useful for scientific computing, which is another thing.
Argh. It would have been nice if they had mentioned Dolphin Interconnect Solutions as a (the?) vendor of SCI-based interconnects, or that Dolphin supported VIA before and better than GigaNet. Dolphin had all sorts of problems, including production problems, but at a basic technology level they were always way ahead of anyone else. 3.2Gbps, 2.5us latency - that's the real deal. If they'd had the resources that Myricom's hype machine generates, they'd have cascadable mega-switches and much better economies of scale (lower prices) and nobody would ever consider buying GigaNet or MyriNet pieces of crap.
*sigh* But it was not to be. Dolphin ran up against compatibility and performance problems with PCI chipsets before anyone else did, and the drivers weren't stabilized soon enough, and they never really figured out who their market was, and they made the major mistake of being honest with customers while competitors were bullshitting about stuff as though they actually had it ready to ship when in fact it was barely even on the drawing boards. In the end the liars and cheats got the mind and market share, and Dolphin is barely eking out an existence nowadays.
I also take issue with the following from the article:
>HA clusters may perform load-balancing, but systems typically just keep the secondary servers idle while the primary server runs the jobs.
Bull. Idle standby is just _so_ early-90s. I worked on eight-node mutual-standby (i.e. load sharing with potential for full failover) clusters in '94. We were before most people, but not first. Nowadays almost nobody would buy an HA solution without this capability.
Slashdot - News for Herds. Stuff that Splatters.