Slashdot Mirror
Linux Clustering Hardware?
Kanagawa asks: "The last few years have seen a slew of new Linux clustering and blade-server hardware solutions; they're being offered by the likes of HP, IBM, and smaller companies like Penguin Computing. We've been using the HP gear for awhile with mixed results and have decided to re-evaluate other solutions. We can't help but notice that the Google gear in our co-lo appears to be off-the-shelf motherboards screwed to aluminum shelves. So, it's making us curious. What have Slashdot's famed readers found to be reliable and cost effective for clustering? Do you prefer blade server forms, white-box rack mount units, or high-end multi-CPU servers? And, most importantly, what do you look for when making a choice?"
(a) Your cost budget
(b) Your work requirement: A Search engine is different from a weather forecast center.
(c) Cost of ownership which includes maintenance etc
when all his three-CPU XBoxes become Linux clusters!
Bwahahahahaha!!!
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
I always choose the one with go faster stripes on the side.
For the size and performance, they are hard to beat. A dual opteron setup in a 1U rack case is a very powerful setup in and of itself. The bonus of using off the shelf components with no need for proprietary hardware or software also make them very affordable. The added bonus is that you can simply get the parts from regular retailers for replacement.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
MacOS X Server is not UNIX, it's based on a UNIX-like OS.
Ammasso is a startup that makes iWarp-based RDMA hardware that runs over gigabit ethernet. Their technology is like Infiband, but much cheaper and almost as fast. Their drivers and libraries also provide MPI and DAPL support. The only support Linux (all 2.4 and 2.6 kernels) and they're way ahead of their competition in terms of performance, product availability, and support. Once you've decided on the servers, I strongly recommend you use Ammasso's hardware for the interconnects. Your hardware vendor may even bundle it with their systems - be sure to ask about that.
And the men who hold high places must be the ones who start
To mold a new reality... closer to the heart
At Apple we sell the Xserve Cluster node which has been used for clusters as large as the 1,566 node COLSA cluster. We also sell it in small turn-key configurations.
Probably the most interesting news lately for OS X for HPC is the inclusion of Xgrid with Tiger. Xgrid is a low-end job manager that comes built-in to Tiger Client. Tiger Server can then control up to 128 nodes in a folding@home job management style. I've seen a lot of interest from customers in using this instead of tools like Sun Grid Engine for small clusters.
You can find some good technical info on running clustered code on OS X here.
The advantage of the Xserve is that it is cooler and uses less power than either Itanium or Xeon, and it's usually better than Opteron depending on the system. In my experience almost all C or Fortran code runs fine on OS X straight over from Linux with minimal tweaking. The disadvantage is that you only have one choice: a dual-CPU 1U box - no blades, no 8-CPU boxes, just the one server model. So if your clustered app needs lots of CPU power it might not be a good fit. For most sci-tech apps, though, it works fine.
If you're against OSX but still like the Xserve, Yellow Dog makes an HPC-specific Linux distro for the Xserve.
- "When you want something with all your heart, the entire universe conspires to give it to you" -Paulo Coelho
At the current time I would choose blades based on dual core Opterons form many reasons. Some of the main ones are:
- Price
- Software availability
- Power consumption
- Density
Brand depends on what your company is confortable with. Some companies would want to have the backing of IBM, SUN or HP. Others will be quite satisfied with in house built blades. This days it's quite easy to build your own blade, some mother boards builders take care of almost all components and complexity (for example Tyan). But again, maybe the PHBs at your gig will run for the hills if you mention the word motherboard alone.
I would love to see a picture of the google hardware...
Definitely worth checking out. It's one bad-ass Linux server -- and probably the only one to offer instruction execution integrity. That's a fancy way of saying 2+2 will always equal 4 on zSeries -- because everything is executed twice and compared at the hardware level -- or it won't execute.
If you need this, you need it bad.
In the paper, it goes into tedious detail on the architecture and low-level operation of the application. Why do you think it does this? Because it is the application that *totally* depicts the solution, they chose lots of systems because of reliability, they made those systems "desktop class" because they didn't get much extra from using super-MP/MC systems.
It's a great article, I strongely suggest you read properly, and do what they said they did - evaluate need against what's available.
we're hoping that upgrading to OpenSSI 1.9 (which uses a 2.6 kernel instead of the 2.4 kernel in the current stable release) will show better disk performance... but... yeah.
Yeah but this is the first time you get to watch them decay "live"....
I run a 4096 node NES beowulf cluster. Works great!
http://bhs.broo.k12.wv.us/homepage/staff/seti/stac k.htm
Some plywood, a power supply powering multiple motherboards.. crunching seti.
Life is good.
That would be typical of a prima donna company like Google that's floating in cash from their IPO.
Around here, we don't waste money on fancy designer metals like aluminum. Salvaged wooden shipping palettes work just fine for us; they're free. And screws!? No need to waste resources on high-end fasteners when you can pick up surplus baling wire for less than a penny per foot. A couple of loops of wire and a few twists are all you need to assemble a working server.
The dotcom days are over. There's no reason to throw money around like there's no tomorrow.
How about Orion 's Desktop and Deskside clusters? Where else can you get a cluster or 12 or 48-96 CPU's in one machine that you can plug into a normal power socket?
Note that racks full of blades (and even dual-cpu-1U systems) can easily exceed the power and cooling capacity of many colocation center's racks. I'm not convinced blade systems are cost effective anyway; but just wanted to point out that the answer may depend on where you store the systems.
Are you clustering for performance? Are you clustering for high availability? Are you clustering for fault tolerance? There are differnent types of clusters for different applications. We can't tell you what is "best" for clustering without knowing what you are trying to accomplish.
We have a farm of freebsd webservers in small tower cases on shelves. We have 10 of those. All servers were built from very cheap components that we got online. I think that each one is something like $300 - $400. The farm serves dynamic pages via php and apache and also does some image manipulation using netpbm tools. Servers talk to the outside via 2 redundant ballancers running freevrrpd. These have the same hardware as webservers. Our application stores a lot data so I have 2 (for redundancy) 2TB raid arrays served via nfs to the webservers. These are in a rack of course, as well as 2 (for redundancy again) Postgres servers. Rackmount servers are built from off the shelf components again. Works great.
My .02 cents worth ...
:) Suffice it to say that bioclusters are rate limited by file I/O issues and are tuned for compute farm style batch computing rather than full on beowulf style parallel processing.
I build Linux and Apple clusters for biotech, pharma and academic clients. I needed to announce this because clusters designed for lifesci work tend to have different architecture priorities than say clusters used for CFD or weather prediction
I've used *many* different platforms to address different requirements, scale out plans and physical/environmental constraints.
The best whitebox vendor that I have used is Rackable Systems (http://www.rackable.com/ . They truly understand cooling and airflow issues, have great 1U half-depth chassis that let you get near blade density with inexpensive mass market server mainboards and they have great DC power distribution kit for larger deployments.
For general purpose 1U "pizza box" style rackmounts I tend to use the Sun V20z's when Opterons are called for but IBM and HP both have great dual-Xeon and dual-AMD 1U platforms. For me the Sun Opterons have tended to have the best price/performance numbers from a "big name" vendor.
Two years ago I was building tons of clusters out of Dell hardware. Now nobody I know is even considering Dell. For me they are no longer on my radar -- their endless pretend games with "considering" AMD based solutions is getting tired and until they start shipping some Opteron based products they not going to be a player of any significant merit.
The best blade systems I have seen are no longer made -- they were the systems from RLX.
What you need to understand about blade servers is that the biggest real savings you get with the added price comes from the reduction in administrative burden and ease of operation. The physical form factor and environmental savings are nice but often not as important as the operational/admin/IT savings.
Because of this, people evaluating blade systems should place a huge priority on the quality of the management, monitoring and provisioning software provided by the blade vendor. This is why RLX blades were better than any other vendor even big players like HP, IBM and Dell.
That said though, the quality of whitebox blade systems is usually pretty bad -- especially concerning how they handle cooling and airflow. I've seen one bad deployment where the blade rack needed 12 inch ducting brought into the base just to force enough cool air into the rack to keep the mainboards from tripping their emergency temp shutdown probes. If forced to choose a blade solution I'd first grade on the quality of the management software and then on the quality of the vendor. I am very comfortable purchasing 1U rackmounts from whitebox vendors but I'd probably not purchase a blade system from one. Interestingly enough I just got a Penguin blade chasssis installed and will be playing with it next week to see how it does.
If you don't have a datacenter, special air conditioning or a dedicated IT staff then I highly recommend checking out OrionMultisystems. They sell 12-node desktop and 96-node deskside clusters that ship from the factory fully integrated and best of all they run off a single 110v electrical. They may not win on pure performance when going head to head against dedicated 1U servers but Orion by far wins the prize for "most amount of compute power you can squeeze out of a single electrical outlet..."
I've written a lot about clustering for bioinformatics and life science. All of my work can be seen online here: http://bioteam.net/dag/ -- apologies for the plug but I figure this is pretty darn on-topic.
-chris
When doing clkustering and super computer work. Cheap isnt always the best way to go , if you take into consideration that 5 - 10 % of nodes will either not be functioning correctly or will have some sort of hardware failure. The more you cluster the more man power it takes to repair these nodes. if you buy 1000 $499 colomachines , and 50 of them are failing at any given time, it becomes very time consuming and tedious to keep the cluster going. Spending the extra bucks on high quality hardware , will save you money and head ache in the end. I always use the analogy when talking to older folks who want to get started in computers. spend the extra bucks to get a new machine. The extra money you spend on buying new good equipment , will more than pay for itself in comparison , to the amount of frustration you get from buying old used slow computers. My $.2
"When they invent bitch slaps that can go through a monitor you better f'ing duck" --deft (253558)
Damn, how many times have you tried to karma-whore and link that phrusa.org site now? Don't you have a job or something?
To those who say Apple isn't targeting the enterprise, look no further.
Let me know when they stop trying to force their iPod updater (you know, the one that breaks Real's compatability DRM software) onto my servers. No matter how many times you put that update in the "Never update this" category, it shows back up the next time you run Software Update. Until they stop trying to play childish games on my production servers, I'll not consider them ready for the enterprise.
What are people's experiences with OpenSSI vs OpenMosix?
We typically build clusters that are NOT destined for a fancy, high end data centers, so 110 volts and a less dense configuration is essential. We build them into the skinniest mid towers we can, with 2 80mm fans in front and two fans in back. we use the baker's rack method and a hot isle/cold isle arrangement of racks. It's hard to find desktop cases without crappy fans in the sides, which are no good to us. Currently, the ASUS A7V400-MX and SEMPRON 3000+, 1GB DDR400 ram, and 120 gig ide hd is the choice. OnBoard NIC and video are essential to clusters, IMO. We'll be moving to SATA drives and gb ethernet onboard soon, but IDE's are dirt cheap right now. Coincidently, this is the same white-box web server we usually build.
Tiger Server lets you run your own Software Update Server, which would solve this problem for you. You run a central update server, point all your servers and clients at that, and then you can approve or disapprove each update before it goes out.
- "When you want something with all your heart, the entire universe conspires to give it to you" -Paulo Coelho
Isn't it obvious that the best technology is blade servers? I mean, c'mon fucking BLADE servers! It's far and away got the coolest name of any of them. The only way you could beat them would be if some company came out with something cooler like ninja star servers, now that would be awesome.
We run a very large website. We have a 1U dual Athlon MP box for administration and log processing, and a 2U dual Xeon box with a 6 disk RAID 10 solution for apache + mysql. Its a great solution to us because we run things in a data center co-lo where space is a premium. Both servers use off-the-shelf components and are easy to upgrade and maintain.
If we were running some scientific application and had our own location and good network connection (for example, a university), then I suppose a cluster of standard hardware components like how Google does it would be perfect. If you were doing the same CPU-intensive stuff, but at a co-lo facility, then perhaps a blade solution would be best... it really all depends - you have to look at what your main needs and constraints are, and look at what is available and what will best suit your needs. There is no way to give a standard answer.
If I had mod points, this is definitely where i'd put one of them.
Star Wars redux: "Medeiros' team and RDECOM will also use the supercluster to improve the design of intercept missiles that defend against attack."
Currently 65 (1 master, 64 nodes) of AMD Mobos on Ikea shelves. Cheap, easy to swap out, good air flow around the hardware. The shelves are wood, so everything just sits on them. It would be nice to find power supplies with extra connections to power more than one system.
off-the-shelf motherboards
Mostly true, though increasingly Google has commodity vendors producing custom hardware runs (ie - "Google" branded RAM, variant ASUS motherboards with a custom BIOS and no built-in video or sound, etc.).
screwed to aluminum shelves
Nope, no screws. The motherboards are attached to the trays by plastic pop-on/pop-off standoffs and everything else is held together by velcro straps; no screwdrivers or other tools are necessary when swapping parts. The trays can also hang from the front of the racks while parts are being swapped.
Are you running iTunes on your production servers? Can't you just uninstall iTunes and be fine?
Why would a system configured as a fileserver have that software on it to begin with? Is Apple's apt tool so bad that it tries to patch software that hasn't been installed?
Look for the "ULTRAsparc Driven" logo on the front .. and you've found an ideal base for your cluster...
-GenTimJS
It's tempting to just go buy a bunch of motherboards on ebay and some bread racks to build your cluster. It's certainly the cheapest and most flexible approach.
However, it takes a special type of people to manage that kind of hardware. You have to deal with a high amount of failure, you have to be extra careful to avoid static problems, you've got to really think through how your going to wire things.
On the other hand, if you get something like a IBM BladeCenter, you have a very similar solution that may cost a little more but is significantly more reliable. More importantly, blades are just as robust as a normal server. You don't have to worry about your PHB not grounding himself properly when he goes to look at what you've setup.
I expect blades are going to be the standard form factor in the future. It just makes sense to centralize all of the cooling, power, and IO devices.
CAN we just ban the parents IP??
please...
You mean how Windows 2003 updates Windows Media Player?
SunFire v20z or v40z Servers.
http://www.sun.com/servers/entry/v20z/index.jsp
http://www.sun.com/servers/entry/v40z/index.jsp
They're the entry-level servers from Sun, so they have great support. They're on the WHQL List, so Windows XP, 2003 Server and the forthcoming 64-bit versions all run fine.
They also run Linux quite well, and as if that wasn't enough, they all scream along with Solaris installed.
The v20z is a 1 or 2 way Opteron box, in a 1RU case. the v40z is a two or for CPU box that is available with single or dual core Opterons.
Plus, they're one of the cheapest, if not the cheapest, Tier 1 Opteron servers on the market.
Specialist Mac support for creative pros, Melbourne
...it's the firmware Dell shoves on them.
It's only designed to hook up with Dell disc arrays and tape drives and everything else can shove it (from their point of view).
Do yourself a favor and skip 'em and just by the cards straight from LSI.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
He, like so many others these days, is clustering for style. It's fashionable to do clusters right now and he doesn't want to be left out.
He's a slave to fashion and that means having the most kick-ass cluster that ever served a Word document! w00t! w00t! w00t!
two servers is not a cluster. it is not what the poster asked.
We wanted to set up a small 4-8 node cluster mostly for testing and as a compute resource. For various political reasons we were looking at an IBM solution. At my uirging we went for dual Opterons in the 1U format. And the price seemed right. Here's where it gets wierd *after* the OBM sales people step in. Going thourgh it peice by piece I thought I could put a decent system together - with our substantial IBM discount -- for $14k. By the time we got the quote with all of the crap they thought we needed it was 34k! Just to give the flavor, the rack and assorted pieces was 4k. But thats not the funny part. We were like, "well for this much money, we assume you are putting it together for us." "Um no...didn't you see the services quote that went along with this?" We hadn't -- with the services/support quote came in at $60k! So at this point we asked, can't we just buy the individual pieces we need and put it together ourselves. "Well, yes, but then it won't be an IBM e1350 cluster 'solution'..." "Yea, well, we don't really care what its called, it'll be just as fast and 75% cheaper..." At that time they were getting rid of their 325 servers for way cheap and we actually put that system together for as cheap as a whitebox and probably as cheap as if we'd tried to put it together ourselves. The moral I guess is that if you have to deal with the big vendors, have a very sharp pencil handy!
...after posting I realized that the 34 and 60k quotes were before discount (I think). SO the actual price descrepncy was "only" 62%.
I think most of my posts to Slashdot begin with "It Depends..." =)
Answering this... It Depends...
What is your cluster's tolerence to failure? If a node can fail, then you have the option of buying a lot of cheap hardware and replacing as necessary. This is the way that most big web farms work.
What is your cluster machine requirements? Do you have heavy I/O? Does cache memory matter? Do you need a beefy FSB and 64G of RAM per node? You may find that spending $3000/node ends up being cheaper than buying three $1000 nodes because the $3000 node is capable of processing more per unit time than the three $1000 units are.
What is your power/rack cost constraint? Google is an invalid comparison simply because of their size. They boomed when a lot of people were busting and co-lo's were hungry for business. I'd bet they have a song of a deal in terms of space, power, and pipe. You are not Google and I doubt you have a similar deal. Thus, you may find that there is a middle ground where it is better to get a more powerful machine to use less rack space/power.
In the end, you have to optimize between these three variables. You'll probably find that the solution, for you specifically, is going to be unique. For example, you may find that: Node failure is an option since the software will recover, power/rack costs are sufficiently high that you have to limit yourself, and CPU power with a good cache is crucial, but I/O isn't. This means getting a cheaper Athlon based motherboard with so-so I/O and cheesy video is a good choice since it frees your budget for a fast CPU. Combine with the cheapest power suppy the systems can tolerate and PXE boot and you have your ideal system.
Best of luck.
Obviously, there ain't no such thing as a free lunch. It all depends on what you want.
For sheer processor density, if you need complete servers, the IBM BladeCenter servers offer the most "Bang" (Fast), and they are fairly reliable and compact (Good). They are not cheap. They do have better density than the HP Blades. WETA Digital (Peter Jackson's FX company) uses them.
That will get you 2 server processors, two server-class IDE drives + 2 GigE ports + all peripherals (Power, KVM, CD, Mangement, GigE switches, SAN switches if you want, etc.) per one-half of a rack unit. This is well over twice the density of pizza box units when you count external peripherals like the networking switch, KVM, etc.
Google's setup is Fast and Cheap, but their hardware reliability is quite lousy. However, their clustering setup is specifically designed around expected hardware failure.
(As a side note, Google no longer uses bare boards for their basic nodes. They use fairly small and slow nodes with a LOT of RAM from some company I can't remember. They look kind of like over-sized hard drives.)
If you need crap-loads of raw computing power, in a relatively compact power-efficient chassis (1024 processors/rack), IBM's Blue Gene simply cannot be beat. This is Captial-F Fast, and Capital-G Good, but you certainly can't afford one. (While it provides more cycles for the watt and dollar than any other setup, it isn't exactly as simple as a Beowulf cluster.) And you would still need to buy pesky things like large GigE switches and storage. Check out the current issue of the IBM Journal of Research and Development on IBM's website (or your local university library) for all sorts of juicy details.
[Yes, I am an IBM shill]
So realistically, you really need to look at your application. If it can tolerate failure of any individual node on a regular basis, get the cheapest stuff you can find that will fit in your space and CPU requirements. If node reliability is important, but space is not, 1U servers from any of the three major vendors (or Apple, if that is your thing) will do the job just fine. If you need reliability and space, then honestly IBM's BladeCenter boxen are the best, as long as they fit your application. (I am not just speaking as an IBM'er here... they really are the best blades out there.)
SirWired
Like most Apple zealots, you're completely missing the point here. Why should he have to run his own software update server to circumvent Apple's childishness? Quite a waste of resources, don't you think?
Yes, but that also means he has to buy more pricy Apple hardware.
The vast bulk of the computing in the branch of the sciences that I build and manage clusters for are basic 1U Intel/AMD Dual servers combined with medium level networking and a growing number are managed by Rocks, the RHEL based rocks cluster distribution available at rocksclusters.org.
If you are building a cluster and do not try out rocks you are doing yourself a disservice regardless of hardware.
I would recommend against using Xserve's for clustering. We've been using one for the last 5 months, and we've had nothing but trouble with the damn things. The only thing good about them is their comparatively low price :(
Not recommended.
Hum... It's good to know they have upgraded... A little bird once told me that they had a fire at their colo once due to their rack just being stuffed with MB separated with bits of cardboard. Course I'm probably lying, so don't take my word for it ;-)
I posted an "Ask Slashdot" question almost identical to this one about 6-months ago - but it was rejected (go figure). Regardless, I got what I wanted from this one! Thanks folks.
This sig is a test. If this had been an actual sig, you would be reading something quite a bit wittier than this now.
On the blades, I would hold out just a bit for the LS20's (68W Opteron's) to mature. It will/is available as dual core as well.
The best way to build a cluster on a budget is www.cheap1u.com. For what our servers cost no one can touch the price and performance of a basic server without all the bells and whistles that you don't want or need. Built with off the shelf parts you cant even buy most 1U rack mounts for less than $1000, ours start at $399 and then if anything burns out on their server you are stuck with buying high priced replacement parts.
Check out VirtualIron.
These guys make a hypervisor that sits on top of infiniband between multiple blades and turns them into one big shared-memory linux NUMA system, up to around 32 cpus and probably even bigger eventually. Plus they can dynamically move cpus and memory in and out of each virtual machine, split the group into multiple smaller virutal machines, etc.
I have no connection to them, just saw them at one of the east coast linux tradeshows. I think their hypervisor will eventually be superceded by Xen or one of the other virutalization systems, but for now what they've got looks like hot snot on a gold platter.
When information is power, privacy is freedom.
We've been using Shuttle XPC systems - the older VIA chipset kind 333MHz/P4 based with a GigE card in the PCI slot. Very cheap, easy to self-assemble, so far stable. The same boxes are on the office desks with an AGP video card added. Every box is the same hardware. 8 of these plus switches and UPS fit in a 1/4 rack - OK its a bit crammed, but hey so is NY.
Most likely this is some idiotic setting in the default update settings (although it take some digging to find it).
Even more disturbing than obnoxious practices like this are the other "updates" applied.
For example, when you update iTunes, you must accept retroactive licensing changes to all of the music that you have purchased.
Conformity is the jailer of freedom and enemy of growth. -JFK
Well, he's complaining about something quite minor, which is just having to uncheck the updates you don't what when you run an update. By comparison, in the Linux world Novell forces you to purchase Red Carpet update service for the same feature, and I believe Red Hat does the same with RHEL.
So, I'm not sure why all the hate. The software update server is bundled with Tiger Server, which you can get for as little as $500.
- "When you want something with all your heart, the entire universe conspires to give it to you" -Paulo Coelho
Not iTunes, iPod updater. The thing that tries to put new firmware onto [my non-existant] iPod when you plug it in. And no, I do not have iTunes installed on my productions servers.
And to answer Guido's question, it's not a matter of fixing dependencies for iTunes - on desktop systems [which do have iTunes installed] you can upgrade iTunes just fine if you tell it not to install the iPod Updater, but it will keep trying to install it again and again and again.
I wouldn't know, having not used WK3. Trying to update the media player that frequently has security holes is different than trying to stuff your iPod firmware updater onto every system possible though. And I've never seen an instance of XP/W2K (which I use quite a bit) adding an update you told it not to apply back to the list of patches to download and install, except for the notable exception of things like SP2 for XP.
Well, he's complaining about something quite minor
It's not just a minor thing, it's symptomatic of what's wrong with Apple. You will use your computer the way Steve says you'll use your computer. You will use your iPod the way Steve says you will. Alt(Flower)-Tab will work the way Steve says it will (they intentionally broke the ability for third party software to change its functionality to match Windows/[many] Linux style window switching).
When I say I don't want a package installed on my system, that's final. Any company with that attitude is not ready for the enterprise IMO.
By comparison, in the Linux world Novell forces you to purchase Red Carpet update service for the same feature, and I believe Red Hat does the same with RHEL.
Access to the Red Hat Network is included with every RHEL subscription. I've not used any of the Novell branded Linuxes, but in SuSE (owned by Novell) all I do is fire up YAST and point it at a mirror and download away. No separate purchase or Red Carpet involved. All of which is irrelevent to my point.
Imagine a beowolf cluster of ask slashdot articles about beowolf clusters!!!!
By comparison, in the Linux world Novell forces you to purchase Red Carpet update service for the same feature, and I believe Red Hat does the same with RHEL.
And if they then ignore the customers choice, they aren't ready for the enterprise either.
But, for the record, you can uncheck updates you don't want in YOU just fine without Red Carpet.
Under capitalism man exploits man. Under communism it's the other way around.
is what I do. We have a linux network, 50 identical pc's, ldap authentication and nfs-imported homedirs. This means I can ssh to each and every of these 50 pc's, and that all these pc's can communicate with each other by exchanging data through this common homedir.
;), and it goes FAST. Although this is not really parallel computing this method is very simple to implement (some bash-scripting, an expect-script for auto-login...) and I've gotten really good results with it.
When I need to run a program (calculation, simulation, something that takes a LOT of power) I just make a list of parameter-values I would need to run, and feed them to a script that automatically distributes this over these 50 pc's. With renice 19 nobody notices (except for casefans that go 100%
int main(void) {while(1) fork(); return 0;}
Google are doing to I.T. what Arkwright did to textiles during the Industrial Revolution and Ford did to manufacturing at the turn of the century. The vast majority of those in I.T. who don't follow their basic model are going out of business in the near future. There will only be a few niche players and a few Google like companies.
p a=showpage&pid=80&page=4
Anyway, the magic formula for the system to use is MIPS per Watt, or MFLOPS per Watt. The power requirements and heat produced by high density computers is a real problem to deal with, it's the real limiting factor for your data centre. The cost of the individual computers is relatively small in comparison.
e.g.
http://www.acmqueue.org/modules.php?name=Content&
Anyway, I'd be building a Transmeta based cluster even though the absolute per CPU performance is lower than the latest AMD or Intel offerings (I'm assuming you need to keep ix86 compatability). So, go look for Transmeta blades or Transmeta 1/2 U or 1 U based servers.
Deleted
And power requirements significantly increases the cost of your data centre. There's no point buying lots of cheap servers if you have to then spend a fortune on uprating your air conditioning and UPS.
Deleted
Man, how do you get any work done?
Comparison of various CPU in MIPS/Watt or MFLOPS/Watt.
p a=showpage&pid=80&page=4
http://www.acmqueue.org/modules.php?name=Content&
And some suppliers of efficient servers.
http://www.transmeta.com/success/server.html
Having dealt with HP, I would be inclined to look at one of the other vendors.
Deleted
We recently puchased a 13 node dual Opteron cluster from Penguin Computing after evaluating clusters from them, Dell, IBM, HP and single memory image machines from SGI. Penguin provided a solution with the best price/performance as well as ease of use. They let us benchmark our codes on some of their test clusters to determine wheteher the Opteron or Xeon based clusters would be better suited to us. My favorite aspect of their system besides price was the plug-and-play setup. The cluster was shipped fully assembled, configured, and tested to our site. All we had to do was roll it out of the crate, plug in the power and network connections, configure the network settings in the OS, and start running our simulations. All of the solutions from the other vendors would have required significant setup time on our behalf unless we spent a large amount of money for the services. I also really like the Scyld operating system that was included in the cluster. It makes the cluster work almost like a single image memory machine. Scyld on the compute nodes is setup to download the kernel image and necessary libraries from the master node at boot-up, so any changes made to the master node automatically propagate to the compute nodes. After several months of running simulations, the cluster has not given us any problems. It has been very reliable (never needed a reboot). Their technical support has been very responsive about ansering questions we had with the initial startup.
Apple may be targeting the enterprise, but they are off target.
Let us compare, shall we?
Sun & HP & IBM all are perfectly willing to have response times under 4 hours. Not 4 business hours, not 365.24 days per year except.... 365.24 days per year period.
Does Apple do that? No.
Let's take sysadmin certification. Are they like Sun, HP, and IBM's AIX certs where they are good for the version of their OS you take it for? Or is it like Microsoft's, where it expires? Is it one test or set of tests...or is it a la cart? Hmmm, it is like Microsoft's.
Nope, Apple isn't serious about the enterprise, at least not where Unix is concerned.
By the way, for the original poster: Here's a nickle, sonny; get yourself a real computer. http://www.sun.com/servers/highend/>
Agreed. This is the major problem with apple (other than apple's terminal case of NIH syndrome).
OSX dictates to the user how they will perform functions. Users are given no choice in how to do things. OSX is a dictatorship, you're forced to use a computer with training wheels on and you can't take them off.
Apple also keeps changing the API on developers. Whenever they manage to do something with OSX that Apple doesn't like, Apple insists on removing or breaking the API so you can't do that anymore. So some very useful utilities have slowly lost functionality over time and OSX revisions.
Re: the updater -- I too have run into the problem where OSX software updater wants to install software you've removed. I've removed itunes from my Mac Mini and all that other trash, but software updater always wants to keep 'upgrading it', even though I told it not to.
From your description and what we've gone through, the advantages you list for Penguin -- esp. configured setup before shipping and decent support -- are key issues as you can spend a *lot* of time getting everything setup.
I ask because this is one other solution we looked at -- acutally, we dropped Orion almost immediatly -- of you add up the power of all of those Celerons, its still not a very good deal. :) But I liked the Rocketcalc very much for a personal cluster. 2-way, 8 nodes all in one package, single power-on, etc.
Take a look at The Value Cluster Project.
HPC for Primates. Read Cluster Monkey
Last time I checked, FreeBSD was based on the 4.4BSD codebase, which is directly descended from the original ATT UNIX. So, while FreeBSD isn't Open Group certified as UNIX, and neither is Darwin, both are directly descended from the UNIX codebase, so yes, OS X is UNIX, it just isnt UNIX(tm).
Phus. Sysiphus.
Re: the API bit, Apple has said that after Tiger, they won't be changing the APIs, just adding new ones.
Bít, zabít, jen proto, ze su liska!
http://www.atxblade.com/
Of course, if you are really cheap, you can gather any Mac running Tiger, get one that runs Tiger Server, and use Xgrid. (http://www.apple.com/macosx/features/xgrid/).
So for smaller shops...maintaining the hardware and this level of complexity involves quite a bit of time and effort. Time, I might add, that might be better spent doing business development for our ASP offering rather than investing in equipment and doing all the work setting up the boxes, then worrying about hardware failures and misconfigurations and so on. Any good virtual solutions out there yet?
"My .02 cents worth ..."
$0.0002
?
Bart: Don't you mean "ThunderDOME"?
Willie: Quiet! You want to get sued?
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.