Microsoft Competes In Supercomputer Market

HoboMaster writes "Microsoft is releasing a public beta of Windows Compute Cluster Server 2003 in their first attempt to compete in the supercomputer OS market. Gates is planned to speak at the 2005 Supercomputer Conference, which will be Microsoft's first appearance at the conference. Gates, as always, has high hopes for this new version of Windows, even claiming it to be as powerful and easier to use than Linux."

Comments

Marketing

[didit]

I've read many times here that having Linux in the top500 supercomputers list was not worth marketing because it is a niche. Now Microsoft is marketing a beta of what they dream might enter someday this list. Go figure ...

Re:Marketing

[einhverfr]

For whatever Microsoft spends improving Windows so that it can be used on today's supercomputers, the benefits they will reap for their server and workstation lines could easily repay that investment.

Well, I disagree with you here. To be a quality node in a supercomputer, I would think you would want a lightweight kernel with an efficient and simple programing environment. You want the ability to strip down everything you don't need and keep everything as simple as possible.

This sort of design has absolutely nothing to do with any other market. It is sort of like the undersea aircraft carriers that Germany was looking at developing during WWII. You end up with something that is passable at either being a sub or an aircraft carrier and frankly lousy at being the other. Linux can succeed here because it is extremely modular and *is only a kernel.* Windows is a bit more than a kernel and is not so modular. So what improvements can be made?

My point is that the improvements that could be made to Windows would be entirely useless in Microsoft's core markets. It doesn't even help them get licenses on big iron because the requirements are entirely different.

The point is that Microsoft doesn't have to do any of this. As Windows becomes more stable and more scalable it is eating proprietary UNIX's lunch. Of course Linux is eating it faster, but that is another issue. Yet traditionally HPC has been the domain of a very small number of UNIX-like operating systems, such as UNICOS, UNICOS/mk, AIX, and a few others. Linux has been a more recent member of the family.

Re:Wake up, Bill

[ma_luen]

Actually the importance of the raw performance of the machine is on the decline. More emphasis is being placed on the idea that super computers are only useful in the sense that they help researchers solve problems. So there is growing interest in the notion of "time to solution" as a combination of ease of programming for, ease of using, and of course running a data set on the machine.

Mark

NY Times Article (free reg. required)

[__aaclcg7560]

The NY Times has this article. The opening paragraphs were a bit more intriguing:

"In January a group of Microsoft researchers set out to discover how much computing power they could buy for less than $4,000 at a standard online retailer. They found the answer at NewEgg.com, where they were able to purchase - for just $3,632 - 9.5 gigaflops of computing speed. That is the amount of computing power offered by a Cray Y-MP supercomputer in 1991 at a cost of $40 million."

Re:NY Times Article (free reg. required)

[Anonymous Coward]

Yeah, its called an Apple G5 Xserve! (5GF rmax/CPU x 2 ~= 10GF for the $3000 cluster node) I'm sure the 970MP will make it into these at some point and that gets us 20GF/box (4 cores) Last time I checked you needed about 1000 of these boxes to get "noticed" on the Top 500 (see entry number 15 and 20.) If you can afford IBM Power5 boxes maybe you can take number 3.

Somehow I think Windows Compute Cluster(fsck) Server 2003 does not run on the above hardware. Quick, there was just a November Top 500 list article posted yesterday...who was in the number 1 and 2 spot? Oh yeah, IBM BlueGene/L with 131072 custom PowerPC CPUs and a total rmax of like 280TF. Wow, that's only 30,000 *times* the CPU power of the MS $3500 off the shelf supercomputer. True, a 9GF Cray Y-MP may have cost $40mil 15 years ago. Today for $100mil you get 30,000 times the CPU power from BlueGene/L and can claim no 1.

Can you imagine trying to put 15,000 "beige boxes" (I assume dual proc) purchased from NewEgg.com into a single room and have any kind of reliability? You think there might be a reason the "big" commodity CPU clusters top out at around 4000 boxes? (And that the hardware was not purchased from NewEgg.com...I mean WTF?)

Re:NY Times Article (free reg. required)

[fishybell]

Alright, maybe Microsoft wants to be on the Top500, but I don't. If they can offer a solution that, through clustering techniques, is more reliable, faster, and more expandable than their current solution, those of us who want to make the Top500000 (ie. what it takes to run a medium size enterprise database, a website that won't slashdotted to oblivion, etc.), $100,000 would probably get us there with 25 top-of-the-line, off-the-shelf boxes. That's the linux solution.

Now take into account an extra $25,000 (wild-ass-guess) for licenses and you've got the windows solution. If it costs you $25,000 less a year to get a Microsoft trained IT team than a linux or Unix IT team, then this would seem like a decent trade off to the CFO and CTO; especially if they get to keep their same IT team with minimal training.

From the NYT article: "Our focus is not on the very highest-end systems but on divisional and departmental computing systems," said Kyril Faenov, Microsoft's director of high-performance computing.

Re:NY Times Article (free reg. required)

[Anonymous Coward]

So what we are talking about here is web servers and enterprise database systems? (Essentially high end "business" software.) Doesn't Microsoft already have software available for web server and database clusters? These are normally pseudo-realtime in nature. (In other words, when someone does a search on your website they expect a response now. Not a month from now.) If Microsoft doesn't, I'm sure Oracle will be glad to sell you a monster of a database cluster. I do realize database systems are used in other ways such as data-mining and that these types of applications are not realtime and are probably vaguely HPC. (Though I suspect you want IOs per seconds and not Mflops or Gflops or Tflops for this.)

Normally I associate "supercomputing" or "HPC" with science and engineering. Finite element analysis, computational chemistry, that type of stuff. Maybe high end synthesis and verification for integrated circuits. Some people might try to argue that Hollywood style render farms (special effects) are a HPC application. Any computational process that takes a rack or possibly an entire room full of high end processors days, months, or years to complete might qualify as a HPC application of sorts. (Usually time to completion is proportional to the number of CPUs available if the computation can be parallelized else you probably own a fancy Cray or NEC vector machine instead of a parallel processing cluster.)

I certainly got the impression that Microsoft is talking about this second variety of application because yes, this type of application does exist in many companies doing research and engineering. (Be that a biotech or pharmaceutical company, an automotive company, an energy company, a fables semiconductor company, etc.) Obviously you don't need to be on the top 500 if you are running these types of applications. (Though you might be, the COLSA system is being used for CFD / aerodynamics for military aircraft and missiles. Looks like number 12 is being used for biotech research.)

Did I mention you get the OS for free with that Xserve cluster node? It's bring your own high performance interconnect and switches though. ;)

Re:NY Times Article (free reg. required)

[Anonymous Coward]

$3,632 doesn't take into account the energy bill (to keep those systems up 24/7), the square meters of real estate you need to place those systems. Plus air conditioning is needed if you need a lot more than 9 gigaflops. Last but not the least, those systems may not compute 24/7 and they may break over time requiring expensive maintenance (software and hardware).

Software solutions like http://www.cpushare.com/ havn't those hidden costs.

Re:Claiming?

[VJ42]

Really? I've only just started switching from Windows to Linux, I have a dual boot Windows XP Pro\Linux Ubuntu machine, the one and only problem I've had is connecting to the internet via my wireless card.
My brother had exactly the same problem on his Windows PC, i.e we can both see the WAN, but can't connect to the internet via the router. He solved his problem before I started installing Ubuntu, so the two arn't related, but Windows is far from perfect, and Linux is far from the disaster you paint it as.

Re:How much?

[schon]

"ease of use" isn't exactly going to be a big selling point for the guys putting these kinds of computers together

Excuse me while I find my tinfoil hat.... .. OK, that's better:

What if MS is doing this so that it can strongarm universities and research institutions? Something like going to the bean counters and saying "hey, we have this great new OS for supercomputers - we'll give you a reduced rate on it, *AND* a reduced rate on the licenses for the rest of your desktops, if you just agree to kick that smelly, communistic, viral, legally-dubious Linux off your clusters! (Did we mention that if you're using Linux, you might have to give up all your precioussss IP?)"

Re:It's no surprise...

[MightyMartian]

Look, it's one thing to support small clusters. That's a reasonably profitable market place, and I can't imagine a modern OS that is marketed as a server solution not offering that feature. But what we're talking about here is supercomputer clusters, beasties used in nuclear weapons research, weather forecasting and other forms of computational-intensive work. This isn't exactly a huge market. In fact it's a downright small one, dominated by custom applications and by a few companies with a lot of years of expertise in high end computing. This seems more an example of the sort of megalomonia that runs in the bloodstream at Redmond. "Yeah, we gotta have a presence in the supercomputer market! How come no one's modelling black holes or doing long-range climatological forecasting on Windows 2003?" What do they think, that supercomputers are going to be running Exchange 2003? "Oh yeah, baby, look at how fast Excel comes up now!" I'm used to the idea that Microsoft is going to try to dominate huge sectors of the computer industry, but supercomputers? It's as if Gates and his toadies are losing their collective marbles.

If you want easy of use pick *ANYTHING* but Apple

[J. Chrysostom]

We've got a VA Tech-style Apple cluster over the course of the last year here at our university, and let me tell you, that thing had serious problem. It works now, but it took over a year of fighting Apple's stupidity to make the thing function decently. And we're one of the first people to get an Apple cluster up and running for serious computation (unlike VA Tech who have been building it, running the LINPACK benchmark and taking it apart time and time again). Examples of Apple's stupidity:

• Apple's NFS server only supports 64 simultaneous connection.
• Apple attempts to explain that "real" supercomputers don't have networked file systems.
• Disk I/O was slow as molasses. Apple suggests using the AppleTalk protocol to communicate to the file server to speed things up.
• Disk I/O still slow as molasses. Grad student discovers the software defaulted to non-buffered disk output (flush to disk).

Re:oh boy

[Misch]

Windows Zombie Network. I think Microsoft has already scueeded in creating large "supercomputers", of course they're oncontrolled and spit out viruses and spam... oh, where did you want to go today?

Clippy: "It looks like you want a super cluster."

[khasim]

Seriously, how much "easier" will it be at the cluster level? With Linux, you don't even have to install the OS on the hard drive. It can run off of a CD.

At this level, the extra wizards and such just don't matter.

Re:How much?

[Ninjy]

Windows on the alarm clock? Sign me up for one of those.

Finally I'll be able to use the "my alarm clock didn't work" excuse a few times a week. :)

Structure of the stack

[BigFootApe]

There are a few features which Microsoft would have to implement on any OS targetted at cluster computing. Many are not directly within their control.

First, and most important for users, what would be the APIs provided. Would Microsoft package MPI? PVM? Would they use a proprietary technology? XML based technologies are way too heavy for this application.

Second, what interconnect transports would be provided? VIA, Globus, IB, good old stinky rsh encapsulation? What about independent vendors like Myricom and Dolphin? Would these companies be willing to support a substantially different architecture? Would there be enough customer demand for them to support Compute Cluster Server at the outset (MPICH-GM is old old old for Windows, Dolphonics and Scali are pretty well exclusively LINUX)?

Third, what software will Microsoft be providing for remote batch management? You'd need a secure remote shell, good scripting functionality, non-GUI device management, etc.

Lastly, how suitable is the NT kernel to doing this sort of work? VMS was ahead of basically everyone when it came to clustering technology, yet _nobody_ uses or used it for parallel processing. What are the lessons that can be applied to NT?

There are a few clusters built on NT, but most of the ones mentioned on the Beowulf mailing list (and they are few) are networks of workstations with CONDOR installed which do double duty as computer clusters at night.

Better than Moore's law?

[DonGar]

1991 to today is 14 years, or 9.33 cycles or Moore's law. 40 Mill, halved 10 times is $39,062.50. Since the article is talking about hardware for under $4k, the price is about a tenth of what was predicted.

I'm not drawing any conclusions, just pointing this out.

Opportunity window : OS Paradigm shift !

[Anonymous Coward]

Okay, ill admit to a round of buzzword bingo :

There is an OS paradigm shift looming.

With the incorporation of "virtualization features" in the next round of x86 cores from both intel and AMD, the two-layer software world (OS / userland) of the last 40 years is complemented by a third "virtualization layer" at the SW/HW boundary.

This means that with the next OS-Upgrade that supports this virtualization, every corporate desktop is able to run several instances of one or several OSs with a high degree of isolation.

This mean Joe Beancounter at GM Headquarters can pop a Britney Spears CD from sony with a rootkit into a CD-Drive, catch a virus and begin to spread SPAM through GMs fat pipe to the world, while a tiny slice of the crash test simulation for the new Cadillac is running on the spare cycles on his desktop completely undangered from virtual bluescreens and reboots.

Of course, Linux could do this just as easily as Windows, but how many identically configured linux desktops does GM run ?

When GM rolls out Windows Vista, in, say, three to five years time to 100.000 Desktops, they get a 100.000 node cluster thrown in for free.

It is available with 80% capacity 8 hours a day and 100% capacity for 16 hours a day plus weekends.

Okay, it is very loosely coupled, and thus much less effective than a dedicated cluster, but I guess that for a lot of applications it would be more powerful than a 20.000 node dedicated cluster.

So this would be a $40 million machine for basically free.

All GM would have to do is start porting one of its apps to the new architecture now.

And of course the savings would look even better for underfunded universities.

I think that if "Windows HPC OS" is included in the "virtualization support" in each Windows Vista OS, and they do a decent job of implementing the application deployment and data distribution, Microsoft stands a good chance of getting a hold in the market.