Slashdot Mirror
Linux Desktop Clustering - Pick Your Pricerange
crashlight writes: "A Linux cluster on the desktop--Rocket Calc just announced their 8-processor "personal" cluster in a mid-tower-sized box. Starting at $4500, you get 8 Celeron 800MHz processors, each with 256MB RAM and a 100Mbps ethernet connection. The box also has an integrated 100Mbps switch. Plus it's sexy." Perhaps less sexy, but for a lot less money, you can also run a cluster of Linux (virtual) machines on your desktop on middle-of-the-road hardware. See this followup on Grant Gross's recent piece on Virtual Machines over at Newsforge.
The primary disadvantage of clustering is the network bottleneck. You lose out because even 100mbps is only a small fraction of what the pci bus of even low end pentium systems are able to handle. At LEAST go with gigabit ethernet so you can push over 100 megs per second between processors. This will greatly increase the usefulness of an integrated cluster by decreasing the one primary disadvantage.
Also a bit pricey, but there would be some cost advantage in reduced footprint for some environments.
-Restil
Play with my webcams and lights here
I just thought of something else. I have never used a Beowulf cluster, so maybe I'm completely wrong, but virtual machines could make a Beowulf more easily upgradeable. The idea is that you'd make a cluster with a whole bunch of virtual machines, say 1024. The cluster is fixed at that size for all the software that runs. But in reality, you've got 32 processors actually running. When you upgrade the cluster to 64, you don't need to reconfigure any of the software that runs on the cluster, because they all assume that you've got 1024 processors. But, you get a performance increase because there's now more physical processors. As I said before, I don't know much about clusters. I imagine that somebody who really does know will quickly either confirm what I said or reduce my idea to a pile of stinking rubble.
If tits were wings it'd be flying around.
You're mostly off the mark, I'm afraid. Most software that uses a cluster runs through MPI or simply through scripts. Both mechanisms allow for easy adjustment in the number of nodes/CPUs you use.
Many large compute problems are embarassingly parallel, i.e. the same calculation needs to be repeated with slightly different input parameters. There's basically no interprocess communication, just a little forethought about filenaming conventions, total disk and memory usage, etc.
Execution of such tasks reduces essentially to a simple loop:
foreach parameter_set
rsh nodeN myprog outfileM
end
For those programs that actually run a single instance of the code on several CPUs, you have to be acutely aware of how many nodes you use. Your code has its own limits on how well it scales to multiple CPUs, and your cluster imposes limits on how well (in terms of latency and bandwidth) nodes can communicate. Very few codes in this world scale well beyond 64 CPUs, especially not on run-of-the-mill clusters with plain ethernet interconnects. Fortunately, it is trivial to readjust the number of nodes used for each invocation of the code.
Lastly, virtual nodes cannot easily simulate the behavior of real nodes. Again, it's the interconnect latency and bandwidth. When it comes to supercomputing, only trust what you have run and measured on a real life setup with your own code and input data.
Although I'm joking, let's just take a look at some numbers, hypothetically speaking.
*borrowed from Tom's Hardware*
Linux Compiling Test
3.35 minutes for a Athlon XP 2000+
14.2 minutes for a Intel Celeron 800mhz
(now, here's where we stretch it)
Figure 1.7 minutes for a dual Athlon XP 2000+, 50% of the other time.
1.7 x 8 = 13.6 minutes
But, who really compiles with a cluster, really?
It'd still be faster....At least on a few benchmarks, and at least in theory