Slashdot Mirror
A New Approach To Linux Clusters
rkischuk writes: "InformationWeek has an article about a group of ex-Cray engineers working on a new architecture for clustering Linux systems. 'It's not easy trying to build scalable systems from commodity hardware designed for assembling desktop computers and small servers.' Per the article, 'As the number of CPUs in a Beowulf-style cluster-a group of PCs linked via Ethernet-increases and memory is distributed instead of shared, the efficiency of each processor drops as more are added,' but 'Unlimited's solution involves tailoring Linux running on each node in a cluster, rather than treating all the nodes as peers.'" Looks like Cray engineers think about clustering even when they're not at Cray.
There was only one PPU on the CDC 6600, equipped with a hardware multiprogramming unit to make it look like ten independent CPUs. The PPU had ten sets of machine state and I/O channels, but only one arithmetic-logic unit (ALU). This was back when the ALU was the expensive part of the CPU.
one of the CPUs should be dedicated to the OS
MacOS 8 on multiprocessors was the last major commercial incarnation of that idea. But that was an ugly hack to put multiprocessing on a uniprocessor OS.
There's a lot to be said for channelized I/O, like mainframes have. All the peripherals look roughly similar to the OS, security is better because peripherals can't write all over memory, and there's less diversity in drivers. Intel tried this, but ran into troubles because they made the channel controllers fully programmable and put a little OS under Windows to run them. Microsoft hates it when you put stuff under their OS.
Just to sumarise my basic point: The Top 500 is a benchmark and is not necesarily a good indication of who makes better computers than who.
Go Badgers! -- #include "std/disclaimer.h"
i have been thinking about for quite a while now. Beauwould clusters are all nice and well, but what about joe sixpack with some it background like me who wants to get some sort of cluster going on. what methods are available (be it a simplified beouwoulf cluster or whatever...) for the guy with 3 or 4 old machines who wants to waste some electricity and try his hand at clustering some machines. is it possible to do it without being a CS major, or is it just a matter of having enough time/resources...
I lost my concept of community when my community lost all concept of me.
These folks are using standard rack mounts, fitting 5 standard ATX motherboards in 8u of rack space (no special motherboard needed). They mount them vertically which makes cooling much more efficient when piling large numbers of CPU's into a small space.
When I die, please cast my ashes upon Bill Gates -- for once, make him clean up after me!
I would imagine you would use this as follows; first, you'd get some data points with which to calculate from a server. Then, you'd also get the name of a shared library which is on a server (NFS mounted probably). This library has a function name 'calc' or some such that does that calculation. You can then call that function, and post the results somewhere.
I would avoid using MPI or PVM, since those are not designed for farming out data the way you are. You should probably use your own job control protocol. Also, you might want to allow for multiple archetectures, naming the library foo-0.0.1.i386.so and foo-0.0.1.alpha.so and so forth,
Yes, I'm still a junky. Are you still a bitch?
There were a few "Because Linux does not scale well with multiple servers" posts about why someone would use a mainframe as opposed to a Beowulf.
Well, it looks like there are people working on the task. But that's not the real point, the right tool for the right job is the point. A whole lot of processes that do not require another process to finish before the next one is where Beowulfs shine, if you want throughput or process with dependancies then a mainframe is your best bet.
But it's still nice to have an alternative for those of us who cannot afford a mainframe.
DanH
Cav Pilot's Reference Page
UNIX - Not just for Vestal Virgins anymore
I don't know. What's the tradeoff? Lots of chicken droppings on your field? The soil can use some nutrients, I'm sure. Can the chickens do it? Have you tried or is it a hunch of yours? Who eats more? The two oxen or the chickens? What about the sleeping place? Sure, those are a lot of chickens, but they don't have a problem if you sit them side to side and they don't have a problem with sleeping in three or four rows. Manageability, yes, that's a problem... now we see a real tradeoff. Being able to replace a chicken if it dies (and chickens are cheap) or one (possibly both) of your oxen, which are not cheap, versus manageability of the two oxen. Sure, feeding the chickens is also a problem, as well as collecting the eggs wow! byproducts! suddenly I can do something with the chickens that I couldn't do with the oxen.
I like this analogy.
If you were plowing a field, which would you rather use?
Two strong oxen or 1024 chickens?
--
E2 IN2 IE?
I think this may be a case where a bit more thinking and literature research about the problem would help a great deal.
We're looking at using MINUIT, a package written by the computing divsion at CERN, as our fitting engine. MINUIT's algorithms are quite advanced, and it's commonly recognized within the physics community as the best general-purpose fitting package out there.
I think you may not realize how complicated the functions we're trying to fit are. Here's the quick and simple version: we study magnetic fields within superconductors and semiconductors on a microscopic level. We do this by using spin-polarized muons or radioactive light ions as a probe, and measuring anisotropy of the emitted decay products. That data then has to be compared against complex models of superconductivity. The computationally expensive part here is calculating the values predicted by the model for a given set of parameters. This has to be done once for each data point to calculate chi-squared, and repeated many times (once in each iteration of the fitting process), each time with different parameters. The models typically contain difficult integrals which must be evaluated numerically thousands of times with very high precision.
Since the function we're trying to fit changes fairly often depending on the sample and measurement techniques used, it's not practical for us to spend huge amounts of time optimizing each individual function to be fitted. The fitting package is already optimized, so the only thing left is to parallelize it.
Carve up a big mainframe with 24 or so processors and bind the OS to a few of them. Get another mainframe and use it as a mezzanine backplane to gang together other mainframes each with their OS images bound to specific processors. Run all networking and IO through their own processors to offload the CECs. Add some more ASICs to handle crypto. Toss all console activity off to another 'special' processor. Run a hypervisor over the whole shebang to control all of the guest images.
Voila you've reinvented parallel sysplex with VM for Linux running on 'cheap' hardware.
Except how cheap do they expect it to be?
I'm trying to design a specialized data-fitting program to be used for accelerator-based condensed matter physics (and maybe ultimately other branches of science as well). I need information on adding clustering support to this program. Here's a brief description of what the program does:
The user writes a small chunk of code that calculates the function they're trying to fit the data to. We require the user to code the function him/herself because speed is important, and some of these functions are too difficult for Mathematica or the like to fit. Once the user writes their function, it's linked (dynamically) with the rest of the code. The user then passes in a parameter file, and away it goes.
Many of these fits can take days, and, since they often have to be repeated many times with slight changes to the fitted function or initial parameters, this is a serious concern.
Can this new approach to Linux clusters be used here? We have tons of Linux boxes lying around that are being used for other things, but have lots and lots of spare cycles. We probably couldn't afford a dedicated processing farm, but we could easily live with something like distributed.net where the program transparently takes all the spare cycles.
I know the problem is parallelizable, since each node can calculate the value of the function at a few of the data points, then send back to the "master" the chi-squared contribution of those points. Each iteration of the fitting process, the master sends out the current parameter values, and then the nodes grind away... There's not too much communication required.
One of my big concerns is how to get the user-written function from the "master" computer to all the "slaves". It's unrealistic to expect the user to manually install it on all the machines each time something in the function gets tweaked and it's recompiled. Are there pre-existing standards on how to send code to nodes in a cluster, then have it executed?
Any advice or pointers to good starting places on distributed computing would be much appreciated.
BTW, as a hint to all the other comp sci geeks out there--physics is a great place to find new and challenging computing problems (I'm not claiming this is one). In particular, the particle physics people often have to deal with spectacular data rates, and do extremely complicated event reconstruction. Check it out some time.
> In that incarnation, the two central CPUs ran only user applications, while the operating system, with all its interrupts, OS code, and device drivers, would reside nearby in the ten Peripheral CPUs (called PPUs) provided for this purpose.
When I heard a CS professor talk about putting multiple CPUs on a single chip, I suggested that one of the CPUs should be dedicated to the OS, which would mean that it wouldn't even need a FP unit. So for (say) 4 or 8 computers on a chip, one would be a "OS server" and the others would be "application servers". Ditching the FP on the "OS server" might allow an extra-high-performance design for it. And the others would only need context switches when the OS demanded it, rather than one for every stinking interrupt that came along.
Sheesh, evil *and* a jerk. -- Jade
Treating each node as a peer! Don't they know that Peer to Peer networks are stealing from our musicians and corrupting our youth! I just hope they can repent before the heavy hand of justice comse down on them.
Steven
-- I have marked myself unwilling to moderate-- I don't have other accounts to artificially inflate the karma of
LINPACK is computationally O(n^3) and O(n^2) wrt. communications, for problem size n.
That's not a completely unfair benchmark - but of course you're right it's a benchmark and therefore it does not cover every possible problem out there. However, it is based on the common linear-algebra routines that are the core of a very large part of the scientific computing problems being run out there.
Well, duh, Cray machines are massively parallel processing machines, so they're not clusters in the sense that they don't use network cards and separate computers as basic computing units, <OVERSIMPLIFICATION>the processors talk to each other on the same bus and share the same memory</OVERSIMPLIFICATION>, but basically in either case it's about parallel processing. I *hope* Cray engineers think about clusters. I'd hate to see them think about single Athlon supercomputers ...
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
(I know--not the best analogy)
Power to the Peaceful
The real limitation of Beowolf style computing is RAM. Beowolf is great if you have programs that paralellize with little intercommunication and low RAM usage. The bigger problem is RAM. Big iron like Crays/SUNs/SGIs all have about a Terabyte of RAM in one place. When you are trying to do large physics calculations you usually have a huge data set you need to store for every time series. Supercomputers aren't cool just because they are fast, but because they can hold HUGE amounts of data in RAM for easy acess. Until PCs get a few gigs of RAM per box cluster computing is still going to be Kludgy no matter what kind of message passing scheme you use.
bash-2.04$
bash-2.04$yes "Don't you hate dialup connections?"| write USERNAME
Furbeowulf
Funny. :)
I like the idea of a furry cluster: furbeowulf.
No one got beat up more often than the mimes of the old west!
SJVN commentary on distributed computing and some interviews with various people in the field.
Best Slashdot Co
What is important to realize is that in order to use these boxes as a cluster, you will have to wrote you own custom software. Yep, it means C and C++, and hours of hacking.
But as mentioned in a previous post, Mosix can do that for you, if and only if your program can use several instances at the same time. Compressing MP3s is a good example.
Nobox: Only simple products.
Of the top 500 supercomputers in the world, 47 are vector processor machines - the kind of processors that Cray became famous for.
Not one single of these are made in the U.S.
Cray today is a name. It's a brand. It's not a manufacturer of high performance computers.
Just for the record, IBM produced the two fastest computers currently, Intel the third, IBM the fourth, Hitachi the fifth, SGI, IBM, NEC, IBM, IBM, and Finally, number *11* is a Cray based on the Alpha processor (the T3E).
So, tell me again, who was playing catch-up with who ?
Guess who the CDC6400 designer was? Seymour Cray.
Cray's engineers seem always willing to consider every possibility, whether it be clusters, p2p, parallel, etc.. showing us that they're considering things well outside of what they're currently offering is also showing us why they're still in the game and even ahead in serious computing power after so many years.. IBM, Sun, etc.. have had their rise and falls, but Cray is always mentioned with reverance...
I am curious about where clustering will go for the connection between nodes. Ethernet, Fibre Channel and various proprietary formats are around but all have issues. InfiniBand is also on the horizon. While I work with InfiniBand development, I am not involved with any kind of clustering work. I see Sockets over InfiniBand as interesting method for inter-node communication. What do those of you who do work in the field think?
-- soldack
what these guys want to do is to build, say, a cluster of 2 CPU system where one of the CPUs only computes while the other manages I/O and communications. Indeed, the I/O part is really a problem on Beowulves, and dedicating a CPU on it and communication can be cheaper than dedicated network cards like Myrinet (at 1000 $/port) or SCI, and hi-perf I/O like HiPPi. I wonder though if they can beat the price/performance ratio of the latter the way Beowulves beat on raw Flops the ones of traditional supercomputers.
Google passes Turing test : see my journal
Mentioned with reverence, but still slowly going bust.
The reason that high speed computing has not taken off is that there are currently no consumer apps that require it. Only a few scientific, research and governmental organizations have a need for it. However, let's say there is a breakthrough in AI technology, it will require googles of CPUs and memory. And when that happens, the market will explode.
People are going to want their mechanical maids, baby sitters, gardeners, chauffeurs, lawyers, companions, stock market experts, and what not. I predict they are going to crave their mechanical servants to the point of pathological obssession.
Don't be so sure this won't happen in your lifetime. In fact, there is every reason to suppose that it might happen anytime. There is an awful lot of minds thinking about intelligence and an awful lot of money being spent on it right now. IMO, the solution to the intelligence problem is probably simple. As Dr. Rodney Brooks of MIT says, "Maybe this is wishful thinking, but maybe there really is something that we're missing." Any day now.
In conclusion, I would recommend that you don't sell your shares in the supercomputing sector just yet.
This article covers three distributed OS options, with some intro explination of the difficulties. I would think the easiest (not necessarily the best) solution could be to use Mosix (listed last in the article) and thread your application to a logical extent. Mosix won't interfere with your current linux boxes, just add it on. The tasks will automatically be load-balanced among the machines.
Developers: We can use your help.
If you're interested in general-purpose clustering (i.e. you don't want to re-write all your apps to use MPI), I really suggest checking out Compaq's Single System Image Clustering (SSIC)project. For Linux, this is basically in a pre-Alpha state, but the older, UnixWare-based version was very strong.
They also have a good comparison of clustering technology features on this slide. For now, you need a shared SCSI disk that can run GFS or something similar, but it may be possible to hook in PVFS eventually for low-end stuff.
Basically, SSIC is like MOSIX, but with killer high availability features. If a node goes down (from hardware, OS, or application failure), its workload is seamlessly migrated to another, functioning node. On MOSIX, unfortunately, each process has a "home node." If the home node goes down, the process is dead. SSIC also does load balancing by process migration, and all of that good, high scalability stuff.
Anyways, just give a look, and check out their slideshow...
--JRZ
Yeah, I will definetly admit that as benchmarks go, LINPAC is a very good one. I was just trying to make the point that it is not the be all/end all of the world of scientific computing :)
Go Badgers! -- #include "std/disclaimer.h"
... project is already doing it. See their Linux Scalability Project.