How Well Does Windows Cluster?

cascadefx asks: "I work for a mid-sized mid-western university. One of our departments has started up a small Beowulf cluster research project that he hopes to grow over time. At the moment, the thing is incredibly weak... but it is running on old hardware and is basically used for dog and pony shows to get more funding and hopefully donations of higher-end systems. It runs Linux and works, it is just not anything to write home about. Here's the problem: my understanding is that an MS rep asked what it would take to get them to switch to a Microsoft cluster. Is this possible? Are there MS clusters that do what Beowulf clusters are capable of? I thought MS clusters were for load balancing, not computation... which is the hoped-for goal of this project. Can the Slashdot crowd offer some advice? If there are MS clusters, comparisons of the capabilities would be welcome." One has to only go as far as Microsoft's site to see its current attempt at clustering, but what is the real story. Have any of you had a chance to pit a Linux Beowulf cluster against one from Microsoft? How did they compare?

Comments

Re:Licensing

[CodeMonky]

Followup:
From reading the MS Site it looks licensing is based of the EULA of the software being used, so if you are using win2kpro you have to have a copy of win2kpro for each machine etc etc.

Here's the deal:

[Null_Packet]

MCS (Microsoft Cluster Services) are designed for load balancing and fault tolerance, as where Beowulf Clusters (AFAIK) are more for distrubuted processing load for performance increases (massive threading). MCS works quite well, especially well on Fibre Channel and Brand Name Hardware such as Dells and Compaqs.

Simply put, it works well (but the cost is often an issue due to the cost of hardware in an enterprise) but it is not the same clustering you see with the Unices. E-mail me at my account if you have more specific questions.

My intent is not to start or participate in a flame war, but the term clustering simply implies different things on different OS'.

Re:Here's the deal:

[iankerickson]

MCS works quite well, especially well on Fibre Channel and Brand Name Hardware such as Dells and Compaqs.

Except your post is factually incorrect. MSCS is a POS -- to say it works "well" is true if you mean "well... it works.... kinda."

It basically just enables multi-initiator support for SCSI chains (so a chain can be connected to 2+ hosts), allows more memory for large applications (if the application is written correctly to use it) and (this is the main feature) allows services to fail-over from one host to the other.

This is where it MSCS should be good, but it just isn't. Basically imagine you have 2 NT servers. A is running Services, and B isn't running any Services except the basics. Do a NET STOP on all the services on A, wait for it to completely finish, and then, and only then, do a NET START on those same services on B. Visualize how long in your mind that would take, and then double it. If anything goes wrong, like a service won't stop (imagine that) or a service can't start due to a dependancy, it throws a monkey into the whole works.

Also, the clusters disks can only be used by one node at a time, and while it would have been trivial for Microsoft to expose each disk to both hosts always (by automatically mounting the disk on the "other" node over the network) they just didn't bother.

It's also got alot of setup caveats. Read the entire manual very carefully and take notes before you even purchase hardware. Then go on-line and read all the addendums and known issues. A good understanding of NT is not enough -- MSCS is a different build (compile) of NT than the Workstation/Server version. She is a woman who has serious issues, some of which can't be fixed by you.

And then there's the blue screens. And the 7 hour installation procedure. And the way you are strongly cautioned from deleting or changing some MSCS settings after being set, with loving MS-style advice that a reinstall is your best bet.

However, for just plain applications, it's OK. Anything you can run from the command line proper can be put in the cluster and will fail over. So if your one of the majority of Acrobat Distiller user who installs in a manner that violates the EULA, i.e. on NT polling the "In" folder of a network share, MSCS can fail over Distiller VERY FAST (it's not a service, so no delays). However, with a little brains and a little ActiveState Perl (or cygwin I suppose) you could hack together a work-a-like using DFS + rsync and save a lot of money.

Kudos to your post for not trying to engender a flame war. But you kinda imply that MSCS is worth the exorbitant price tag, and it just isn't for what little it does and the problems and extra headache it brings with it. I'm not flaming you, just spreading the word:

DON'T BUY MSCS -- IT SUCKS. IF THEY GIVE IT TO YOU FOR FREE, SEND IT BACK OR GIVE IT TO SOMEONE YOU DISLIKE.

Back on topic, what MS may try and sell you is something based on the Microsoft Message Queue and the Microsoft Transaction Server. Those are more BackOffice-variety PHB-entitled products that really don't do much except provide an API for sending guaranteed IPC and doing transactions, even for VB monkeys who don't really understand what that means but think it sounds just plain awesome. Free with the option pack.

This is part of that Microsoft program to divert "wins" from Linux to Microsoft at all cost, especially from IBM. So the sales rep probably doesn't have a clue what your cluster really does, what you want it for, or what MS products it would actually take to build a knockoff. They may have a anti-beowulf team cooking something up right now, and guess what pal?! They're hoping your administration will take the bait of free hardware and licenses, and you'll end up beta-testing a 0.1a version of some bizarro-beowulf for MS. What a deal!!!

Good luck. I'd stick to you guns and inside on using something already proven to work for your goals, like Beowulf or AppleSeed.

Beowulf

[Usquebaugh]

A beowulf cluster is not limited to Linux, it could run on top of any OS. I believe NASA did the original design work to be OS agnostic.

http://www.windowsclusters.org/projects.htm gives a list of current Windows clusters.

Finally, are you out of your tiny little mind? I wonder why M$ is so keen to help. There is no such thing as a free lunch, espically from M$.

"It runs Linux and works" - 'nuff said?

[Booker]

Ok, you have a solution in place. It works. Some sales guy wants you to change your solution that works.

Make him convince you that the time and cost of the switch is going to gain you something.

Does your current setup not do what you need it to do?

MS Cluster is not the same

[merlin_jim]

Hello,

We run a MS cluster here. VERY big app... so big, I am loathe to name figures, because that would identify to MS just who is talking here...

But, we use MS clustering for our web app. Our setup is that we have a database server with 4 procs, and a growing array of web servers with 1 proc each, all of which use disk space on a SAN. W2K clustering manages the load balancing as well as allocating disk space out of the SAN to virtual partitions as needed. The original poster is correct; MS clustering is for load balancing, not computation. I have seen many times Microsoft sales reps don't have a clue of what they're trying to sell; they're just told from on high to replace Linux with Microsoft wherever they can. I think this is clearly a case of that.

My advice? Ask the sales rep to demonstrate how MS clustering will solve a common comp-sci problem with more MIPS than each box alone has. Point out that you're not running a web server or any such service on these boxes, but that they're for raw computation. Even better, see if he'll let you talk to a technician on how W2K clustering can meet your 'unique' (at least to MS) needs.

Now, for everyone else... Don't get me wrong. W2K clustering is a great technology for building highly performant, highly reliable, highly scalable applications quickly and easily. But it scales in the direction of millions of users, not millions of computations.

Re:MS Cluster is not the same

[crimoid]

Apparently you (and most everyone else) didn't take the time to even look at the link provided. Microsoft DOES have computational clustering, not just "traditional" clustering.

MS Computational Clustering

Re:MS Cluster is not the same

[merlin_jim]

I must now put on the traditional monkey hat of shame, for the naysayers are quite correct. There are TWO microsoft products called clustering. One is used by Windows 2000 Advanced Server to do load balancing, and is, in fact, split into two parts, the first called Clustering, the second Network Load Balancing... see this page, which includes the statement "Both [of the Windows 2000 Advanced Server] Clustering technologies are backwards compatible with their Windows NT Server 4.0 predecessors". The other is High Performance Clustering (HPC), in its current form called Computational Clustering Technical Preview (CCTP), which I am certain has nothing to do with the previous Clustering technology... I doubt it was available for Windows NT 4.0, among other things (thus the Technical Preview status).

Notes for any and all interested in this; it's a technical preview, which any other company would call a pre-Beta or an Alpha release. The only way anyone sane would use this in a production system would be as an Early Adoption Partner...

Re:Licensing

[the_2nd_coming]

oh, but remember,

the TCO!!!!!!!!!!!!! :-p

you know how expencive a CS student is!!!! oh my god, how can they afford the astronomical amount of having 5 or 6 of them on one project.

don't you know that if you move to windows for all your reaseach project clustering needs, you only need a chimp....and since educating a chimp is much cheaper than educating 6 bright young men, your university will save a considerable amount of money....especialy when you lay off all those expencive profs and hire an animal trainer.

Re:Licensing/Reliablity

[MrWinkey]

My managers will only buy windows products as they have a site liscense with MS. They are looking into Linux a little bit because of the Terminal server w/ load balancing does not load balance and the clusterd computers do not talk to each other. The profiles on the 3 clusterd servers do not update each other at all. This was much better than the last attempt my boss did using an IBM pre configured configured box the whole cluster got a BSOD and corrupted a drive losing data for 3 days. People were not happy.

I can only hope MS's poor performance will make them switch.

what??

[Lumpy]

an MS rep asked what it would take to get them to switch to a Microsoft cluster.

first the rep needs to prove that $199.00 per node for software fees has to provide major benifits over the Linux cluster. How many windows clusters can he list for you to call and ask about it? refrences, ones you can call and talk to the guys running/maintaining it. Show where microsoft provided increased profits or savings over an open alternative.

If they cant give you a dollar amount that shows increased profits or major savings then be sure to tell the rep that he shouldn't let the door hit him in the ass on the way out. It isnt MS versus Open anymore in today's economy.. it's what can get it done and save me money or can give me more profits... and this is what makes Open solutions win... microsoft can't give savings and the performance difference isnt enough to give profits that will more than overcome the added expense of Microsoft.

Get real numbers, talk to real people running real clusters on all platforms. if you have real numbers then you can make solid decisions.

Stability issues

[The Panther!]

At my last job, we had a COW (Cluster of Workstations) running all sorts of operating systems. Except Windows. Why? Because they won't run in a production environment for more than a few days without freezing or crashing, and the system administrators refused to babysit them. With Windows 2000, I've had my home machine run for upwards of 28 days without a reboot, but only if all the video drivers are stable and the machine is not doing too much at any given point (say, burning cds while watching movies and keeping my net connection above 200k/s). But so help you if a driver freezes. There's no way to reset them. Your hardware will play into your decision as much as the operating system, I believe, due to stable driver support.

In terms of performance, Windows kernels have pretty good latency compared to 2.2.x linux kernels, so running a full screen dos app might give very good performance, but there's a lot of overhead munching into your RAM, which is likely to be an expensive premium on older hardware.

Lastly, with Windows, I've never heard of doing channel bonding for ethernet (3 100TX cards ~= 1 gigabit), nor diskless booting that I know of. These can be really necessary for large clusters to keep maintenance down and performance up without buying higher end equipment.

Don't know if this answers your question...

[Fizzlewhiff]

but based on personal experience, Windows ME is pretty much a cluster.

The OS doesn't matter - tools do

[Oestergaard]

For a computational cluster, the OS itself shouldn't really matter. What matters is, do you have the tools you need, and does the environment allow you to work with the cluster in a flexible way.

For a typical compuatational cluster, what determines the performance will be the quality of your application. Only if you pick an OS with some extremely poor basic functionality (like, horribly slow networking), will the OS have an impact on performance.

People optimize how their application is parallelized (eg. how well it scales to more nodes). The OS doesn't matter in this regard. They optimize how well the simple computational routines perform (like, optimizing an equation solver for the current CPU architecture) - again, the OS doesn't matter.

So, in this light, you might as well run your cluster on Windows instead of Linux, or MacOS, or even DOS with a TCP/IP stack (if you don't need more thatn 640K ;)

However, there's a lot more to cluster computing than just pressing "start". You need to look at how your software performs. You need to debug software on multiple nodes concurrently. You need to do all kinds of things that requires, that your environment and your tools will allow you to work on any node of the cluster, flexibly, as if that node was the box under your desk.

And this is why people don't run MS clusters. Windows does not have proper tools for software development (*real* software development, like Fortran and C - VBScript hasn't really made it's way into anything resembling high performance (and god forbid it never will)).

Furthermore, you cannot work with 10 windows boxes concurrently, like they were all sitting under your desk. Yes, I know terminal services exist, and they're nice if you're a system administrator, but they are *far* from being usable to run debuggers and tracing tools on a larger number of nodes, interactively and concurrently.

Last but not least, there are no proper debugging and tracing tools for windows. Yes, they have a debugger, and third party vendors have debuggers too. But anyone who's been thru the drill on Linux (using strace, wc -l /proc/[pid]/maps, ...), and needed the same flexibility on windows, knows that there is a world of difference between what vendores can put in a GUI and what you can do when you have a system that was built for developers, by developers.

So sure - for a dog&pony show, windows will perform similar to any other networked OS with regards to computational clusters. But for real-world use ? No, you need tools to work.

Ask for modifiable code and no injurous NDAs

[WeeGadget]

...what it would take to get them to switch to a Microsoft cluster.

Simple... ask for :

1. Modifiable source code... essential for University level research.
2. Blanket permission to publish research methods and results, including code.
3. No NDAs that could limit a student's job oportunities. (i.e. "No Compete" clauses etc.)
4. Free or low cost would be nice :o)

Jonathan Weesner

I just think that its funny...

[acoustix]

that he says he "works for a mid-sized mid-western university" when his handle has a link to a Ball State University email address.

Just come out and say it.

What's your app?

[gcoates]

IAABA. (I am a beowulf admin).

Beowulf clusters get built to support your application, not the other way around. Your choice of hardware and OS will depend on the parallel nature of your code. Do you need myrinet, or can you get away with fast ethernet? Will your code even compile under win32? Do the supporting libraries (PVM/MPI/BLAS whatever) run under win32? What about the queuing system?

How are you going to manage the cluster? You need automation, even for small clusters. How easy is it to add a new user, apply a patch or change a bios setting on your cluster without having to plug a keyboard and monitor into each node? What about central logging? How about automated OS installs when you add another 100 nodes when you get your funding?

Oh. Benchmark, benchmark, benchmark. That means your code, running your datasets, on your hardware and OS. Not vendor supplied numbers. If you have a serious hardware vendor, you should be able to wrangle demo mechines off them. Try before you buy.

Here's what to tell them.

[NerveGas]

Years ago, I worked at an ISP that ran partly on Solaris, mostly on Linux. A few MS reps came in to try and get us to switch to NT. We let them go through their routine, then walked them around the operations room, telling them the capabilities of what we had, and asking if NT would match them. The response was repetetively "no". When we pressed them on a few issues, they gave in rather easily. When we asked them why you couldn't bind another IP to an ethernet card under NT without a reboot, they admitted "lazy programming."

So, take the MS reps through the operation, tell them the capabilities. Ask them if they can meet or exceed them. If they say "Yes", you're either not using the real capabilities of your Linux machines, or they're lying.

steve

so what do you call it?

[hawk]

Would a cluster of Windows be a Grendle???

:)

hawk

use Application Server, not Clustering

[spongman]

Microsoft has a few types of clustering:

1. Failover clustering. This is an OS service that servers like SQL Server and Exchange plug into that allows Active/Passive or Active/Active clustering over a shared SCSI/Fibre bus. In theory you could write your app to use this service but I think it would be overkill.
2. Network Load Balancing. This is just a software version of the standard kinds of NLB found in cisco boxes.
3. Component Load Balancing. This is the most suitable. It's provided by Application Center and it allows you to deploy COM+ objects on a cluster of machines and have the calls distributed according to the load on those machines. You can control the threading and lifetime of the objects and view the status of the machines pretty easily using the Application Center MMC plugin (or SNMP, I believe). You'd have to wrap the computational part of your application into one or more COM objects. Once you've done that then you can create and call those objects in the cluster as if it were one machine - the clustering is transparent to the client application. I played around with AC a bit when it was in beta for a project that I was working on. We didn't go with it in the end because the design of our application ended up not requiring it (we just went with hardware load balancing), but it seemed like pretty cool technology - if you're into the whole COM thing. It has a really cool rolling deployment feature where you can redeploy your components (and/or IIS application if you have one) to your cluster incrementally while it's still running.

Here's some links to docs on MS's site:

Introducing Windows 2000 Clustering Technologies
Application Center home page
Component Load Balancing

Re:Licensing

[TheCarp]

> Let's just leave BSODs out of it. Maybe an issue but not always. Some people can get
> BSODs down to near nil and others can't but it is always the OS's fault. Hmmmm.

I find this intetresting because I have seen it too.

In my experience a well run Windows system by a person with real clue can last a while and be pretty blue screen free. The same is true for a system run by an idiot who got it all installed right and hardly does anything with the box, just plays some specific game or uses Word or something.

However, when you start installing software and doing different things, they gan get real flakey real fast. Not just in reliability either... users shit all over the box!

I saw someone turn on their computer...it came up... and the desktop was just littered with icons... full. They never manage their stuff, they just keep all that crap that every little software package installs.... is it just me or are companies that make Windows software extremely arrogent? id say MAYBE 1% of the software I use is something important enough that I want an icon for it on my desktop made special... but every peice of windows software seems to think its that special.

my little rant... the unmanageability is why I don't use it. I installed debian GNU/linux on this box 2.5 years ago, have installed software and iuninstalled it over and over... and it never gets unstable.

In a cluster, where software isn't being installed and uninstalled, windows will probably be just fine. Tho frankly, id rather a bunch of unix boxen with tools like cfengine to manage such things.

-Steve

Re:BTW: MS Slashdotted

[Cinnibar CP]

This wouldn't have happened if they had thier servers clustered to handle the load-balancing issues.

What it would take

[Arandir]

my understanding is that an MS rep asked what it would take to get them to switch to a Microsoft cluster.

You've got a golden opportunity here! Microsoft does it your way or they don't get the sale.

Let them know the nature of a cluster in a research project. Nodes will be swapped in and out. New ones will be added. Different OSs will be used. So tell them you want a copy of Windows for each potential node, licensed to the University and not to any individual node. Tell them you need full rights to install, reinstall, and uninstall any particular copy on any particular node. Tell them you will not accept any terms restricting the cluster to Windows only.

If you really want to play hardball, tell them you don't even want licenses, but bonafide user-owned copies of Windows subject only to the provisions of copyright. In other words, you don't want to be subject to any EULA. Then you'll discover how much Microsoft wants your cluster to be a Windows cluster.

Simply...

[Glock27]

ask the Microsoft rep to point out how many machines on the Top 500 Supercomputers List are running Microsoft operating systems.

Then, point out the scads of Beowulf clusters and Linux/Unix based systems.

Finally, inform the rep and your management that you've chosen to use the more cost effective, higher performance and standardized choice...Unix.

If management resists further, do a cost analysis. That'll convince them.

299,792,458 m/s...not just a good idea, its the law!

I can always count on Ask Slashdot...

[doorbot.com]

...to ask a question that I wanted to ask as well. Granted, this topic seems a little strange, considering the Linux cluster is in place, and it seems like the kind of question which encourages a Microsoft vs. Linux world domination showdown for grandmaster of the universe. It also shows a limited business sense on the part of the poster (why change something that works well when you can't afford a replacement?).

Right now a coworker and I are looking at pricing and configuring a fault-tolerant cluster for a client who runs Windows 2000 and Exchange 2000. They're a bit paranoid, so they've decided they want a cluster. We've tried to educate them on exactly what a Microsoft cluster can and can't do, so it's difficult to understand exactly what they want (basically an entire network exactly like Microsoft's own, but for $1000).

Pricing on a two system cluster is around $50,000. Buying two copies of Exchange and Windows Advanced Server will total $20,000. Then there's the hardware costs. For our client, they've specifically requested this, so they're ready to pay.

My question to Whamo is are they really taking the Microsoft rep seriously? If they have to pay software costs for their new cluster that's going to mean two things: either buying less CPUs to add to the cluster, or not doing the project at all, because just the software will put them over budget. With Advanced Server running somewhere around $4000 that's a lot per machine when Linux costs at most $5 to burn a CD after downloading it via the university's T1/T3/etc. Whamo says "it is running on old hardware and is basically used for dog and pony shows to get more funding and hopefully donations of higher-end systems" and to me that is your answer. If you can't afford the hardware you can't afford to buy Microsoft's software...

Also, there's MOSIX as well, but I don't have much experience with MOSIX and thus cannot comment on it.

Re:Licensing

[lynx_user_abroad]

You have to pay someone to clean the cage...

What the chimps leave on the floor will be nothing compared to what the Windows-based cluster will leave on the floor.

Re:Licensing

[sigwinch]

The question to the group was comparing WCS to Beowulf from a performance/functionality perspective, not from a licencing perspective.

Licensing is cost, and cost IS performance.

Every man-hour spent reading, auditing, and managing licenses is a man-hour that is not applied to real work (he says, posting to /. from his desk at work ;-). Every hour the compute nodes sit idle while licensing is sorted out is a 4.17% performance hit for that day.

All those licenses cost money, which means fewer CPUs. If a compute node costs $400, and licensing is $100/node, you can afford 25% fewer nodes. This is indistinguishable from a free OS that has a 25% performance flaw.

Then there's risk. The software mafia aren't going to audit a Linux cluster, sapping administrative time, and perhaps cease-and-desisting it offline. Linux cluster admins are never going to go to jail because they threw another machine online for the hell of it. Linus Torvalds will never sue a Linux cluster operator into oblivion to make an example of them. These are all possibilities with a proprietary product, and all-too-likely with a notorious lawyer-pit like Microsoft.

MS parallel tools

[ajv]

Getting past what are the wrong tools first: Beowulf is an architecture to do massively parallel computation, so we can eliminate two of the best known HA tools. Microsoft Cluster Service is two or four node high availability, similar to HA Linux's efforts. NLBS is a software form of a hardware load balancer, similar to Cisco Local Directors and only really good for web farms. So what does MS provide to do similar stuff as Beowulf?

COM+ and Queueing Components. AppCenter.

The way it works is this. You write a COM+ component that is transactionally queuing aware. Each component takes a work unit in, processes it, and then sends the result of the transaction to the queueing components for reassembly or re-issue (if a node fails to submit a result, for example, good for checkpointing).

You can use normal Windows 2000 Professional boxes for the worker bees, and use a few Windows 2000 Server boxes to co-ordinate the issuing of jobs and control, and munging the result sets coming back in.

If you need to submit a wide variety of jobs, obviously the COM+ components will be changing regularly, it'd be a good idea to go to AppCenter so that you can treat a bunch of machines as single whole. This allows you to upgrade or deploy an app in a few mouse clicks to literally thousands of machines in a few seconds. AppCenter also has pretty good resource management, something that might be necessary if multiple jobs are running at the same time.

The cool thing is the development environment is really friendly and you can make COM+ components pretty easily and test them locally (for the n=1 case) before deploying to the farm.

There are also specialist MP libraries for the Win32 platform, such as PVM or MPI (WMPI). These have the benefits of re-using the knowledge and API's that users might already be familiar with - one of the biggest thing when a place converts from one supercomputer to another is rejigging and reoptimizing the code for the new architecture.