Slashdot Mirror
What Do You Need To Watch For In A Linux SMP System?
thefin asks: "My research group has finally received funding (~$200K) for a single SMP box. I've looked over the offerings from SUN, IBM and Compaq. I was wondering what others think of the SMP offerings and in particular, opinions of Linux as a SMP OS. What should we know before purchasing such a machine? What should we look for and what should we avoid? We will be using the box for large, individual based ecological modeling efforts. These models are CPU intensive, and make heavy use of inter-node communications. In particular, inter-node communication can be a serious bottle-neck in our models."
Why are you buying a single box? Is it necessary? You can find a lot of information about distributed computing at the aggregate. You could use the money to build a computer as fast, and have computers to use for other things when the computer is obsolete (6 months). :)
Secondly, if internode communication is as high as you say and you can't parellelise your code/algorithms you made a good choice with SMP (as opposed to beowulf). Avoid x86, go for Alpha. Avoid Linux or *BSD, go for the native OS which scales to multiple CPUs much better.
If you can parrellelise your code the obvious thang is beowulf and cheap x86 boxes.
-- Eat your greens or I'll hit you!
-- Eat your greens or I'll hit you!
Above all, use the 2.4 kernel whenever posssible if you do SMP. The 2.4 kernel shows noticable improvement over 2.2 even on a 2-processor machine, and while optimized for 8 or less processors, has been reported to run on 64 processors (though I can't remember just who did this, might have been Linux Care but I really doubt this). It doesn't have NUMA yet, though this will probably be the next major change to Linux the way things are looking now. At any rate, just stick to 2.4 if you do use SMP. It scales far beyond its predecessor, to an extreme. Just review the list of changes and you'll see what I mean. (Gawd, I'd love to see some 'Mindcraft' benchmarks now!)
Also, check your drivers for SMP compatibility, some are better than others and have reputations for it. On the extremely rare occasions that you can get a manufacturer-supplied driver, you should be ok if it's hardware meant for big boxes like this.
Lastly, don't forget clusters. (warning: a long rant on clustering follows, if you aren't interested, this is the time to hit 'Back')
Linux is probably the ideal solution for clusters because of its efficiency and reliability, and the fact it hoses just about everything that is benchmarked against it on lower-end (ie. two processors or less) machines. If you have a lot of clients on the network that perform mundane tasks (ie. word processing, most spreadsheets, etc. that don't fully utilize system resources), utilizing idle cycles ala SETI@Home, combined with a network renovation to 100TX or better (not very expensive these days, but you'd want to eliminate the use of hubs for any machines involved, since bandwidth is at a premium) could yield a lot of horsepower that, at the very least, could supplement your big box (especially in the off hours provided the computers are left on). The money invested in this may give you more bang for the buck than buying new equipment, and if you want to do clustering smaller machines instead of one big box, you can use the remaining money to buy a set of dedicated machines and string them together on gigabit fiber-optic or something, and then bridge them to the rest of the network. x86 machines are great for clustering, PPC is a little overpriced for the power it gives you in many standard configurations(especially compared to Athlon processors if you do floating-point-intensive stuff) unless you use Apple's Altivec, where PPC tends to kick butt, and as for SPARC, ALPHA, MIPS, and anything I forgot to list here, you're on your own since I've never fooled with any of these. If you want to cluster, I'd stick to whatever the rest of the network uses so you only have to code for one processor.
Three things are critical when parallelizing work:
SMP machines usually have better throughput and latency than raw CPU power - whereas clusters have more CPU power and weaker throughput and even worse latency.
The major advantage of cluster is that you can re-use them in multiple configurations (tree, (hyper)cube, pipeline, flat neighbourhood) - whichever meets your needs best.
As for cheap clustering: maybe you have enough clients/workstations that can be (mis)used as cluster during nighttime (and NICEd down during working hours) for a first test run?
All boils down on how (good) you can parallelize your simulation. If you cannot, multiple processors - in either configuration - won't help you.
I have two smp boxes, both are dual PII/III's.
So far I've tried Linux, FreeBSD, BeOS, and win2k on them. I usually use Linux, I like it, and it runs well, for the most part, under smp. There are a few issues though with linux SMP. For starters, before the 2.4 kernel FreeBSD was much faster. I'm not sure if the 2.4 kernel is faster than FreeBSD, I havn't run numbers yet.
From my experience I'd say, if your going to use linux, make sure the drivers for your hardware all work under SMP. I've had some scsi cards, and some ultraDMA 66 ide cards whos drivers are terribly unstable under SMP. (You might have to read the code for the drivers and look at the comments)
Of course EVERYTHING I've done with smp has been on Intel hardware, and while I'm sure you can pick up 200K smb Xenon boxes, there probably not the most bang for the buck. If you really want to run linux SMP on a machine of this size, you'll probably have to go Sun, SGI or Alpha. Linux is pretty good on all three platforms there, but I'd stay away from RedHat on anything but intel hardware (ok me personally, even on intel).
So maybe the steps you should take will be,
1 Decide if you need 32 or 64 bit machines.
2 Pick up a dual proc workstation with similar processors, to develop and test on. Lets say you think you want an e450, so pick up used dual spark station from one of the many used dealers that cater to ISPs.
3 Make SURE all the drivers and code work well on it.
4 Buy the HUGE machine.
You are only young once, but you can stay immature indefinitely.
SGI's systems are really designed for this. The NUMA architecture is a way to mix what you're doing (i.e. lots of cpus, lots of memory) with the ability to make some things "closer" to others. If you're able to think of your app at least slightly NUMA like, it works well.
Otherwise, let me recommend the Alpha boxes. Tru64 UNIX is a phenomenal operating system, and it scales beyond your imagining. It really is that good.
The other thing to think about is what kind of CPU usage are you doing. Assuming that you're using floating point computation, you need to immediately discount the Intel architecture. It's STILL hobbled by the terrible FPU that it's had for years, which is why the Athlon kicks it around on this stuff. Alpha and MIPS have the best FPUs implmented (SPARC is okay, but definitely not as good as the Alpha).
What does your app need? Here's the questions I'd look at first, and a few thoughts.
First, is it fpu or integer intensive? If integer, intel can make sense. If fpu, can you use SIMD (MMX, 3dNow, alta vec, etc)? If you can, intel can still work, but if not, RISC is the only choice.
Second, how much memory bus does it really need? Ok, a lan will not cut it, but could you live with everything (cpu-cpu or cpu-memory) going over 1 100Mhz 64 bit bus (ok, I think the 8 way boxes have two busses, 4 cpu on each)? If so, an 8 way intel box may be the right choice. Test to see if you need 2MB cache, or if everything is good with 1MB or 512k, you can save some money. I don't know how the RISC boxes are with regards to bus, but they are better by a huge margin than intel. If you are going to wait a bit, the AMD 8 way boxes may be availible, you'd get stunning integer performance, ok fpu, and good memory bus performance, it is the same bus the Alpha uses.
Disk I/O, how much? are you loading from disk, and just chewing on it for a few days? If so, disk io doesn't much matter, but if it does, linux may not be the best choice.
I'm guessing that network io does not matter.
For this kind of task, I bet that linux will work just fine (spliting the multi threaded or multiple process app over multiple cpus). I'm guessing that if a big intel box is not suitable, a big alpha box is the best choice.
How much ram does the box need? If memory speed is the limiting factor, RISC usually has better memory designs.
$200k is not much money, really. I'm guessing it will buy you a fully tricked out 8 way Xeon Intel box, or somewhat less RISC. Of course, if you get more $, you can add more CPUs to your box, but not true for intel.
Hmmm, spend $100,000 on an intel box now, and $100,000 when you can get an 8 way AMD Athlon? Might be best of all worlds for your money.
Better questions people!
Plato seems wrong to me today
The most significant and compelling reason I can think of to not go with x86 architecture is that every PC MP setup besides AMD 760 MP (which isn't out yet, and will only support two processors to begin with anyway) has a truly pathetic bus architecture. If you're doing something where bandwidth makes a difference, which you would seem to be if you're getting a single SMP box rather than contemplating clustering, the bus bandwidth is going to figure in, too.
As you probably know, in an intel-based solution, all CPUs sit on the same bus to the so-called "memory masters";
Meanwhile, the AMD CPU bus architecture is more closely based on the Alpha EV6 bus, and each CPU has its own bus to sit upon.
Since you can't go with a AMD 760MP solution, you should get a real live unix box; Something slick from Compaq/DEC, Sun, IBM, or even HP (though I don't recommend it.) If you're looking for raw power and the ability to handle large numbers quickly, Alpha is probably your daddy.
Side note: Apologies to Busta for the title of my comment. I couldn't resist.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
For that kind of money, you can afford to get a big box and an industrial strength unix like HP-UX, Solaris, or AIX. Linux is great if you don't have a lot of money and aren't going to need lots of processors and lots of RAM, but I don't think it's a good fit for what you're talking about.