What do you want large caches for? Large caches aren't simply a trump card to be played to magically make all your applications faster. Misusing a cache can be detrimental to performance unless your cache is big enough to fit the entire application and all its data. Algorithmic enhancements can be used to make a huge difference... and in some situations, I can get higher performance out of a processor with 1/4 the cache size of another processor where a poorly written version of the application runs on certain types of data.
All that being said, yes, larger caches will improve the execution of the majority of software (and certainly of algorithmically tuned software) but it is not the end-all, be-all solution.
Today, SMP code usually requires the code to be written to take advantage of multiple CPUs. There are compilers out that can do some automated threading (and have been a while) but many threaded applications are threaded by hand. Basically, we'd need better compilers and OSs to go along with those computers than we have now -- compilers that can make runtime decisions on how many threads to fork/etc and OSs that can report system resource reports accurrately to the programs.
That being said, your term "power" is heavily overloaded here... I'm sure you can put 4 G4 processors into a box and the total (electrical)power usage of the 4 G4s would be comparable (or less) than a P4. If you are talking about four processors that are basically 1/4 of the computational power of a P4 (so four of them equal a P4), some applications will still need higher 'power' so that they can finish in times comparable to today. To paraphrase an old saying, a process is only as fast as its slowest thread =)
I'm also developing using.NET. My app is database driven and mostly waits on user input or data returned from the database... I guess your G4 waits twice as fast as mine does on my type of app/shrug.
I haven't really written any truly performance demanding apps, but I did play around and wrote a version of Sieve in both C# and (unmanaged) C++ just for kicks. The C# version was actually slightly faster but that was because of the memory allocation schemes of the two languages from what I could tell. The C# program allocated the array (8M) all at once... bascially in one update of TaskManager the memory size of the C# went from X to X+8M or so. The C++ program seemed to be only allocating pages in the 8M as it needed them... the program started at size Y and grew steadily in relatively small increments until it was Y+8M or so in size. I believe the seemingly allocation-on-demand allocation scheme of the C++ library caused the program to run slower. I guess I should pull up an old VC++6 compiler from somewhere to compare.
I'll go into some ramblings I guess:
As far as speed in developing apps, C# w/.NET is pretty fast. I was able to throw together my sieve in about 10 minutes (full windows with input/output text boxes, labels, go buttons and such) from start to finish. My console app C++ was done in similar time. I wrote a time-sheet style calculator using C# the other day in about 10 minutes from start to finish (windows app with textboxes, labels buttons, etc.) It was faster to write than search/download one I figured. So far, I've been fairly pleased with the development environment except there are a few things that get annoying about it but they are either trivial to work around or come from doing stuff you shouldn't be doing anyway. The ADO stuff has changed a bit from previous versions and looks a bit more like Java libs now (which weren't too far off the previous version of ADO). DataSets and such are very useful as you can just plop em in data grids and such and gets lots of default functionality out of simply that. One thing is for sure... the IDE is orders of magnitude better than JBuilder (which I had to use on a previous project) and the.NET framework is far, far easier to use than the convoluted and incredibly manual labor intensive JBuilder/JBoss/etc. J2EE platform we used earlier this year for that software. I'd rather be sentenced to slam my fingers repeatedly in a car door for several weeks than go back to that stuff. I can only hope and pray that there is a better environment for Java and enterprise solutions out there than that nastiness. Good ideas... extremely poor implementation (I hope it is just the implementation that has the problems).
So yeah, Windows does have some junk about it and makes me want to pull out my hair sometimes but as a development environment, I can crank out good code faster in C# and.NET than most of the other platforms that I've used in the past... sometimes by several factors faster.
Go check on your history... I believe Apple was the first thief in the chain of events (from when Jobs visited Xerox-PARC and saw the GUI they had there). For some reason, people always conveniently forget that part. Of course, then Microsoft stole it after that... but Mr. Pot meet Mr. Kettle.
People get fixated on the big-bad-wolf and must live/breath it. For some, computers aren't just tools, they are religions and these people behave about their religi^H^H^H^H^H^HOS just as the Christians do about their religion... which includes shouting on streetcorners for all to hear and drawing as much attention to Sata^H^H^H^HMicrosoft as they can in bad light.
For what stuff? a) you don't suddenly start using 64-bit ALU ops in your software, it has to be written/compiled to use them, b) what type of calculations will Joe Sixpack be doing that needs 64-bit values as opposed to 32-bit values? The only one I can think of (and even then it is a stretch) is games.
Also don't forget that anyone who writes commercial software will have to Q/A on both 32-bit and 64-bit platforms if they decide to provide both AND that there will be a lengthy transition period from 32-bit to 64-bit where *many* people will still have the 32-bit processors. There will be little advantage to anyone other than server type apps (DBMSs/etc) (read: very expensive apps) to provide a 64-bit version of their software. I predict that even after Hammer is released, other than a few apps that will want to make use of the marketting hype, there won't be this exodus to 64-bit computing that so many here seem to think will happen.
Joe Sixpack will neither know what this is nor care. Does it run his latest version of Deer Hunter(tm)? Does it run his email client? Will it let him surf pr0n? Which of these things need to be 64-bit?
It forces M$ to directly support Hammer or resign to allowing Linux ownership of at least the market share that AMD has
I disagree with this statement. If Microsoft doesn't support Hammer, then those people who want 64-bit apps or who already run Linux will go to AMD if not already there. There are plenty of people who buy AMD processors to run Windows (and only Windows). There would be no reason for them to go to Hammer excepting the case where they are like me and just upgrade for the speed benefits:)
Those who are already sensitive to cost/price are already running AMD processors. My prediction is this:
that the people who run Linux only will upgrade much quicker to Hammer than others.
Those who run Windows and Linux will continue to do so on whatever processor they have, but have potential to upgrade to Hammer.
Those who run Windows only may buy a Hammer for the performance benefits but will continue to run Windows.
Whether Microsoft supports x86-64 or not cause people to flock to Linux and Hammer. There will possibly be a small shift for those who really want to run 64-bit apps but there aren't many of those. When you get down to it, Joe Sixpack probably doesn't care whether his app is 64-bit or 32-bit and probably doesn't know the difference anyway. In fact, there aren't many reasons for Joe Sixpack to even want 64-bit apps.
Although I typically take Tom's technical information with a grain of salt, the article http://www4.tomshardware.com/column/02q1/020215/in dex.html is fairly interesting talking about some of the hurdles that DDR has to overcome as it attempts to reach higher clockspeeds.
Ok, so I am talking to either William, Ewing, or Anthony, or even possibly Don. Neat how these conversations arise.
No, I'm not one of these four guys but I know them too. I might even know you:)
You will note that a small section of that was about Myrinet (and MPI by indirection). What I claimed was the following:
Windows cannot handle many simultaneous users "logged" in. If I am wrong, please give me a pointer so that I may learn. If you prefer to just go ad-hominem, do it do/dev/null. Windows is process oriented, not user oriented. You can have many "servers" running, not many users. Very important distinction.
It depends on what you mean by "many". If you mean hundreds/thousands (or even a few 10s) as you stated above, then I will agree with you. However, I have used a number of machines with 10s of users logged into it. However, if you are using a node to do parallel computation, wouldn't you typically reserve the node for one user so that use can use all of the CPU resources there? In debugging/development, sure... you will have nodes that many people will access simultaneously, but when you get down to the actual simulation you will probably use a batch scheduler or something to reserve the nodes for one user.
If you want a demonstration of multiple users logged into a Windows box, one easy way is to use Terminal Services. I use it quite a bit in everyday work to access servers simultaneously with others here on our servers on the West Coast and Europe. As you say as well, you can have a number of processes running on a single Windows box, each running under a different set of user permissions/credentials.
Windows cannot manage large dataflows with high bandwidth/low latency. This is based upon my (non-MPI) experience, especially moving large chunks of data around on a CiFS network versus an NFS or similar network. This is actually more due to the way Windows uses its disks, and that is the limiting factor in its data motion. The networking speed is fundamentally similar to linux. I should have clarified this better for consumption.
I would also agree with this point in the DiskIO and NFS networking but would say it no matter the commodity clustering platforms. You have to load the data somehow, granted. However, I have not seen this to be a problem when performing large transfers (90% of physical memory -- 1/2GB and larger) between machines when using networks such as Giganet and Myrinet. So we agree on this point.
Scaling costs: Well, if you can show me how Wink is cheaper than Linux, great. Be my guest.
I believe I indirectly stated that most Linux systems are cheaper.
porting issues: Yes, most supercomputer users started out in government/academia. Much of this has filtered into industrial areas. Clusters are displacing other iron when the code makes sense to run on the cluster.
I agree. I do know of a number of Windows clusters in use that aren't advertised to the public. UN*X/Linux clusters get far more press and there are more of them. Also if the software is originally developed on a UN*X cluster, then it is a pain to port it.
The point I made was fundamentally, let MSFT contribute to the school to buy the level of hardware you need to accomplish the mission. Otherwise you are going to wind up with many unfunded costs, and lots of headache.
I would say that it depends on what they are doing.
Now you claimed that I provided disinformation, when fundamentally you agreed with me. Odd.>p> Am I biased? Absolutely. I fully admit that I will follow my customers/clients desires. In the 7 years I have been involved in selling this stuff, I have not seen a single windows cluster get sold. I know about 3 in the world, the CTC, the NCSA, and one in the UK. I am sure there are a few more.
Those are the large clusters, yes.
But my points remain, and are valid, unless you wish to contest that:
the windows system will cost more (due to all the extras you have to buy, the extra cost of management, the extra cost of the larger hardware needed), the extra cost of licenses, and the windows licensing model which requires you to pay more per number of "clients" (e.g. users in this case), the extra cost of doing the code ports (once again, not all clusters run MPI jobs)
It will be harder for you to get your work done as a result of a lack of tools (for the correct reasons you pointed out)
Depends on what you mean by "lack of tools". Development environments on Windows platforms are quite nice.
performance will be slower, as Windows disk I/O system doesnt perform as well as Linux, most of the runs I do today have a large component of disk I/O so I will freely admit bias here.
my experience on using/building/testing networking leads me to conclude that despite the super-fantabulous drivers you may/may not have written, the network performance of the windows box will not be as good as that of the linux box (for non-MPI issues, such as moving files about).
A cluster is not a device to turn MPI code into results. A cluster is a generalized computing platform of a particular set of designs. Some may or may not run MPI (or PVM, or Linda, or TCGMSG, or...). For the generalized cluster, Win2k is, IMO, a far more costly total solution than the Linux route. If you received great value for the extra cost, it might be worth it. You dont get anything additional.
I'm not sure I would agree completely, although I would agree with you generally. I wouldn't be able to guess entirely. All that I see mentioned is that they currently have an older cluster they use for research on computational tasks. I would have to know more before I could suggest one or the other. Initially, I would suggest a Linux cluster without knowing more of the research they are doing. I might would change my mind given certain issues but I cannot simply blanket his problem with a statement such as "Linux is faster and always will be no matter what the problem you are trying to solve is".
An example: Our software runs on our production cluster of 17 machines (with no MPI) about 1,700 miles from my cube and on another of our clusters of 8 machines in Europe. The US cluster has 4 quads that are database machines and 13 dual processor machines that process the data from the databases (42 processors - not huge but not tiny). The European cluster has a quad DB machine and 7 duals. We have ~200 customers that also run in either a single machine or a cluster configuration (our software behaves the same either way really, it's just licensing). Our development environments are quite nice (Visual Studio) for both code development and debugging. We use Terminal Services to access the machines in both clusters along with the other 14 developers in my group. We do have the number of simultaneous users per machine limited though, as the machines won't support a large number of them. Our cluster is about as general purpose as you can get as far as the tasks that we perform from it with our processing. Development time for our applications isn't too bad because the tools we have work satisfactorily. Could we do it on a UN*X or UN*X-alike? sure. Would it cost more/less? To be honest I don't know because our design criteria was to be Windows based. However, my gut feeling (and I have ample gut) would be that it would be more expensive if we had to have done it on UN*X.
As far as the performance of this system, there is a similar system in place in another part of our company that was implemented on a UN*X (Solaris actually) cluster of 450s. They bought us a few years back because of the similarity of products/market/etc. The Windows cluster performs similarly to the Solaris cluster on many things, faster on some things, but slower on others.
Not at all. We actually sold more Linux licenses than Windows licenses. The point is that those sites stick to what they know. There's nothing wrong with that, but it *is* a reason why you won't find Windows clusters in those sites.
Yes, I see your subject was appropriate for your message.
MSFT at one time was very interested in computational science. I talked to quite a few people from Intel and MSFT on the subject. Your information about what Windows and Linux can and cannot do is simply wrong. I have used Linux/Windows clusters (and wrote the MPI libraries for each) supporting TCP/IP over good old Ethernet, Giganet, Myrinet, and a variety of other networks (in addition to a variety of embedded systems and super computers). In truth, in many benchmarks the Windows cluster (same hardware, dual boot) outperformed the Linux cluster on the same MPI codes. There are a number of reasons for this from compiler support (at the time) to Linux kernal problems.
The reason why people use Linux/UN*X clusters for computational sciences is because a huge portion of computational science problems are carried out in Academic/Government organizations. Those type organizations are familiar with UN*X and UN*X-alikes and will stick with what they know. The other main reason why UN*X and UN*X-alike clusters are cheaper out of the box than MSFT clusters. A university can throw together a bunch of hardware and have grad students maintain it for practically free running UN*X/Linux. Government labs can write the cost of the hardware into a contract and not pay for the software and use that cost to either reduce the contract billing, apply the money into more/better hardware, or put it into some other part of the contract. The fact is that there are a number of Windows clusters out there that perform well and are used as serious computational engines, if you care to look for them. In Academia/Gov't they are very rare though.
I, too, have worked for a cluster vendor and have been in clustering for some time now. The main difference between us, I think, is that I am less biased.
A company out there provides commercial MPI libraries for a variety of operating systems, Windows included
http://www.mpi-softtech.com/
A couple years back, I was one of the MPI writers/maintainers for a number of platforms (worked with MPICH). As far as performance went with real applications (as well as synthetic benchmarks), Windows and Linux on the same hardware were pretty much the same. Typically computational problems were faster on Windows though because of the better compiler support at the time. Communications performance benchmarks were interesting.
Also, the at the URL above you can find cluster management software (batch scheduling and stuff).
Yes, I was thinking that thread mobility in those benchmarks was probably the reason why the poor scores were generated. Along with moving the threads, you of course have all the problems associated with it. If you look at the benchmarks on that page, Hyperthreading makes the single CPU faster usually, which we could have predicted (and is surely what Intel wants). On the dual CPU configuration, it actually hinders performance in many cases. There are some other issues with treating one of these CPUs as two virtual CPUs. First, it isn't really two CPUs and threads scheduled on it will not necessarily be executed with fair time slices. If you schedule two threads on one of these CPUs, it is possible that one of the threads uses the majority of the resources for the timeslice and the other does little - until you reschedule it because the timeslice has passed. So basically, you can pay the penalty of two context switches for little/no work done. Also, remember that a CPU, even in highly optimized code, is not running at 100% of its potential - using hyperthreading you aren't getting a 30% faster CPU, you (should be) using the one you have more efficiently in terms of keeping the various execution units busy. I read somewhere that a typical instruction stream in a P3/P4 only keeps 33% to 50% of the execution units (ALU, FPU, etc.) busy. Hyperthreading simply tries to keep more units busy more of the time.
Yes. Most programmers have to sign a non-conflict of interest agreement when they are hired. If you work for IBM and also help develop a competing product on the side, then you are in Conflict of Interest (honestly whether or not you signed the agreement, imo).
Another system that used something like this was Tera: http://www.sdsc.edu/SDSCwire/v3.18/tera.html
However, what they did was have 128 contexts per CPU and it round-robin'd through them all. You could also "daisychain" multiple CPUs together in a system. It was interesting but I don't know what ever happened to the machines they were building.
It is easier to think of this in terms of the OS concepts of a paging system.
When Thread1 gets a Load-Fault (think of Pagefault), it executes Thread2 instuctions that it has cached. Much like an OS will execute Process2 if Process1 hits a pagefault. Basically what the hardware is there to do is to fill up the bubbles in the pipeline that are caused by one thread with instructions from another thread.
Slashdot Mirror
What do you want large caches for? Large caches aren't simply a trump card to be played to magically make all your applications faster. Misusing a cache can be detrimental to performance unless your cache is big enough to fit the entire application and all its data. Algorithmic enhancements can be used to make a huge difference... and in some situations, I can get higher performance out of a processor with 1/4 the cache size of another processor where a poorly written version of the application runs on certain types of data. All that being said, yes, larger caches will improve the execution of the majority of software (and certainly of algorithmically tuned software) but it is not the end-all, be-all solution.
Today, SMP code usually requires the code to be written to take advantage of multiple CPUs. There are compilers out that can do some automated threading (and have been a while) but many threaded applications are threaded by hand. Basically, we'd need better compilers and OSs to go along with those computers than we have now -- compilers that can make runtime decisions on how many threads to fork/etc and OSs that can report system resource reports accurrately to the programs.
That being said, your term "power" is heavily overloaded here... I'm sure you can put 4 G4 processors into a box and the total (electrical)power usage of the 4 G4s would be comparable (or less) than a P4. If you are talking about four processors that are basically 1/4 of the computational power of a P4 (so four of them equal a P4), some applications will still need higher 'power' so that they can finish in times comparable to today. To paraphrase an old saying, a process is only as fast as its slowest thread =)
I'm also developing using .NET. My app is database driven and mostly waits on user input or data returned from the database... I guess your G4 waits twice as fast as mine does on my type of app /shrug.
I haven't really written any truly performance demanding apps, but I did play around and wrote a version of Sieve in both C# and (unmanaged) C++ just for kicks. The C# version was actually slightly faster but that was because of the memory allocation schemes of the two languages from what I could tell. The C# program allocated the array (8M) all at once... bascially in one update of TaskManager the memory size of the C# went from X to X+8M or so. The C++ program seemed to be only allocating pages in the 8M as it needed them... the program started at size Y and grew steadily in relatively small increments until it was Y+8M or so in size. I believe the seemingly allocation-on-demand allocation scheme of the C++ library caused the program to run slower. I guess I should pull up an old VC++6 compiler from somewhere to compare.
I'll go into some ramblings I guess:
As far as speed in developing apps, C# w/ .NET is pretty fast. I was able to throw together my sieve in about 10 minutes (full windows with input/output text boxes, labels, go buttons and such) from start to finish. My console app C++ was done in similar time. I wrote a time-sheet style calculator using C# the other day in about 10 minutes from start to finish (windows app with textboxes, labels buttons, etc.) It was faster to write than search/download one I figured. So far, I've been fairly pleased with the development environment except there are a few things that get annoying about it but they are either trivial to work around or come from doing stuff you shouldn't be doing anyway. The ADO stuff has changed a bit from previous versions and looks a bit more like Java libs now (which weren't too far off the previous version of ADO). DataSets and such are very useful as you can just plop em in data grids and such and gets lots of default functionality out of simply that. One thing is for sure... the IDE is orders of magnitude better than JBuilder (which I had to use on a previous project) and the .NET framework is far, far easier to use than the convoluted and incredibly manual labor intensive JBuilder/JBoss/etc. J2EE platform we used earlier this year for that software. I'd rather be sentenced to slam my fingers repeatedly in a car door for several weeks than go back to that stuff. I can only hope and pray that there is a better environment for Java and enterprise solutions out there than that nastiness. Good ideas... extremely poor implementation (I hope it is just the implementation that has the problems).
So yeah, Windows does have some junk about it and makes me want to pull out my hair sometimes but as a development environment, I can crank out good code faster in C# and .NET than most of the other platforms that I've used in the past... sometimes by several factors faster.
Go check on your history... I believe Apple was the first thief in the chain of events (from when Jobs visited Xerox-PARC and saw the GUI they had there). For some reason, people always conveniently forget that part. Of course, then Microsoft stole it after that... but Mr. Pot meet Mr. Kettle.
People get fixated on the big-bad-wolf and must live/breath it. For some, computers aren't just tools, they are religions and these people behave about their religi^H^H^H^H^H^HOS just as the Christians do about their religion... which includes shouting on streetcorners for all to hear and drawing as much attention to Sata^H^H^H^HMicrosoft as they can in bad light.
Yeah, but that was about "crossing the streams", not about the Staypuft Marshmallow Man.
Depends on which operation you are performing.
For what stuff? a) you don't suddenly start using 64-bit ALU ops in your software, it has to be written/compiled to use them, b) what type of calculations will Joe Sixpack be doing that needs 64-bit values as opposed to 32-bit values? The only one I can think of (and even then it is a stretch) is games.
Also don't forget that anyone who writes commercial software will have to Q/A on both 32-bit and 64-bit platforms if they decide to provide both AND that there will be a lengthy transition period from 32-bit to 64-bit where *many* people will still have the 32-bit processors. There will be little advantage to anyone other than server type apps (DBMSs/etc) (read: very expensive apps) to provide a 64-bit version of their software. I predict that even after Hammer is released, other than a few apps that will want to make use of the marketting hype, there won't be this exodus to 64-bit computing that so many here seem to think will happen.
Joe Sixpack will neither know what this is nor care. Does it run his latest version of Deer Hunter(tm)? Does it run his email client? Will it let him surf pr0n? Which of these things need to be 64-bit?
I disagree with this statement. If Microsoft doesn't support Hammer, then those people who want 64-bit apps or who already run Linux will go to AMD if not already there. There are plenty of people who buy AMD processors to run Windows (and only Windows). There would be no reason for them to go to Hammer excepting the case where they are like me and just upgrade for the speed benefits
Those who are already sensitive to cost/price are already running AMD processors. My prediction is this:
- that the people who run Linux only will upgrade much quicker to Hammer than others.
- Those who run Windows and Linux will continue to do so on whatever processor they have, but have potential to upgrade to Hammer.
- Those who run Windows only may buy a Hammer for the performance benefits but will continue to run Windows.
Whether Microsoft supports x86-64 or not cause people to flock to Linux and Hammer. There will possibly be a small shift for those who really want to run 64-bit apps but there aren't many of those. When you get down to it, Joe Sixpack probably doesn't care whether his app is 64-bit or 32-bit and probably doesn't know the difference anyway. In fact, there aren't many reasons for Joe Sixpack to even want 64-bit apps.Although I typically take Tom's technical information with a grain of salt, the article http://www4.tomshardware.com/column/02q1/020215/in dex.html is fairly interesting talking about some of the hurdles that DDR has to overcome as it attempts to reach higher clockspeeds.
All computers wait at the same speed(rate)...
Not at all. You can implement programs that are SMTP clients just fine and send email all you want. No SMTP server required.
Ok, so I am talking to either William, Ewing, or Anthony, or even possibly Don. Neat how these conversations arise.
:)
/dev/null. Windows is process oriented, not user oriented. You can have many "servers" running, not many users. Very important distinction.
...). For the generalized cluster, Win2k is, IMO, a far more costly total solution than the Linux route. If you received great value for the extra cost, it might be worth it. You dont get anything additional.
No, I'm not one of these four guys but I know them too. I might even know you
You will note that a small section of that was about Myrinet (and MPI by indirection). What I claimed was the following: Windows cannot handle many simultaneous users "logged" in. If I am wrong, please give me a pointer so that I may learn. If you prefer to just go ad-hominem, do it do
It depends on what you mean by "many". If you mean hundreds/thousands (or even a few 10s) as you stated above, then I will agree with you. However, I have used a number of machines with 10s of users logged into it. However, if you are using a node to do parallel computation, wouldn't you typically reserve the node for one user so that use can use all of the CPU resources there? In debugging/development, sure... you will have nodes that many people will access simultaneously, but when you get down to the actual simulation you will probably use a batch scheduler or something to reserve the nodes for one user.
If you want a demonstration of multiple users logged into a Windows box, one easy way is to use Terminal Services. I use it quite a bit in everyday work to access servers simultaneously with others here on our servers on the West Coast and Europe. As you say as well, you can have a number of processes running on a single Windows box, each running under a different set of user permissions/credentials.
Windows cannot manage large dataflows with high bandwidth/low latency. This is based upon my (non-MPI) experience, especially moving large chunks of data around on a CiFS network versus an NFS or similar network. This is actually more due to the way Windows uses its disks, and that is the limiting factor in its data motion. The networking speed is fundamentally similar to linux. I should have clarified this better for consumption.
I would also agree with this point in the DiskIO and NFS networking but would say it no matter the commodity clustering platforms. You have to load the data somehow, granted. However, I have not seen this to be a problem when performing large transfers (90% of physical memory -- 1/2GB and larger) between machines when using networks such as Giganet and Myrinet. So we agree on this point.
Scaling costs: Well, if you can show me how Wink is cheaper than Linux, great. Be my guest.
I believe I indirectly stated that most Linux systems are cheaper.
porting issues: Yes, most supercomputer users started out in government/academia. Much of this has filtered into industrial areas. Clusters are displacing other iron when the code makes sense to run on the cluster.
I agree. I do know of a number of Windows clusters in use that aren't advertised to the public. UN*X/Linux clusters get far more press and there are more of them. Also if the software is originally developed on a UN*X cluster, then it is a pain to port it.
The point I made was fundamentally, let MSFT contribute to the school to buy the level of hardware you need to accomplish the mission. Otherwise you are going to wind up with many unfunded costs, and lots of headache.
I would say that it depends on what they are doing.
Now you claimed that I provided disinformation, when fundamentally you agreed with me. Odd.>p> Am I biased? Absolutely. I fully admit that I will follow my customers/clients desires. In the 7 years I have been involved in selling this stuff, I have not seen a single windows cluster get sold. I know about 3 in the world, the CTC, the NCSA, and one in the UK. I am sure there are a few more.
Those are the large clusters, yes.
But my points remain, and are valid, unless you wish to contest that: the windows system will cost more (due to all the extras you have to buy, the extra cost of management, the extra cost of the larger hardware needed), the extra cost of licenses, and the windows licensing model which requires you to pay more per number of "clients" (e.g. users in this case), the extra cost of doing the code ports (once again, not all clusters run MPI jobs) It will be harder for you to get your work done as a result of a lack of tools (for the correct reasons you pointed out)
Depends on what you mean by "lack of tools". Development environments on Windows platforms are quite nice.
performance will be slower, as Windows disk I/O system doesnt perform as well as Linux, most of the runs I do today have a large component of disk I/O so I will freely admit bias here. my experience on using/building/testing networking leads me to conclude that despite the super-fantabulous drivers you may/may not have written, the network performance of the windows box will not be as good as that of the linux box (for non-MPI issues, such as moving files about). A cluster is not a device to turn MPI code into results. A cluster is a generalized computing platform of a particular set of designs. Some may or may not run MPI (or PVM, or Linda, or TCGMSG, or
I'm not sure I would agree completely, although I would agree with you generally. I wouldn't be able to guess entirely. All that I see mentioned is that they currently have an older cluster they use for research on computational tasks. I would have to know more before I could suggest one or the other. Initially, I would suggest a Linux cluster without knowing more of the research they are doing. I might would change my mind given certain issues but I cannot simply blanket his problem with a statement such as "Linux is faster and always will be no matter what the problem you are trying to solve is".
An example: Our software runs on our production cluster of 17 machines (with no MPI) about 1,700 miles from my cube and on another of our clusters of 8 machines in Europe. The US cluster has 4 quads that are database machines and 13 dual processor machines that process the data from the databases (42 processors - not huge but not tiny). The European cluster has a quad DB machine and 7 duals. We have ~200 customers that also run in either a single machine or a cluster configuration (our software behaves the same either way really, it's just licensing). Our development environments are quite nice (Visual Studio) for both code development and debugging. We use Terminal Services to access the machines in both clusters along with the other 14 developers in my group. We do have the number of simultaneous users per machine limited though, as the machines won't support a large number of them. Our cluster is about as general purpose as you can get as far as the tasks that we perform from it with our processing. Development time for our applications isn't too bad because the tools we have work satisfactorily. Could we do it on a UN*X or UN*X-alike? sure. Would it cost more/less? To be honest I don't know because our design criteria was to be Windows based. However, my gut feeling (and I have ample gut) would be that it would be more expensive if we had to have done it on UN*X.
As far as the performance of this system, there is a similar system in place in another part of our company that was implemented on a UN*X (Solaris actually) cluster of 450s. They bought us a few years back because of the similarity of products/market/etc. The Windows cluster performs similarly to the Solaris cluster on many things, faster on some things, but slower on others.
Not at all. We actually sold more Linux licenses than Windows licenses. The point is that those sites stick to what they know. There's nothing wrong with that, but it *is* a reason why you won't find Windows clusters in those sites.
Except this one:
11/2001 320 Dell Dell PowerEdge Cluster Windows 2000 120.70 Cornell Theory Center USA 2001 252 252 155000 50000
Maybe you should have done the search for yourself before you posted.
This is true. Beowulf is used to describe computational engines. The Microsoft stuff is for failover/availability. Two entirely seperate issues.
...and then point out which of those machines in the list that are Linux Beowulfs are at universities or in Gov't labs.
Yes, I see your subject was appropriate for your message.
MSFT at one time was very interested in computational science. I talked to quite a few people from Intel and MSFT on the subject. Your information about what Windows and Linux can and cannot do is simply wrong. I have used Linux/Windows clusters (and wrote the MPI libraries for each) supporting TCP/IP over good old Ethernet, Giganet, Myrinet, and a variety of other networks (in addition to a variety of embedded systems and super computers). In truth, in many benchmarks the Windows cluster (same hardware, dual boot) outperformed the Linux cluster on the same MPI codes. There are a number of reasons for this from compiler support (at the time) to Linux kernal problems.
The reason why people use Linux/UN*X clusters for computational sciences is because a huge portion of computational science problems are carried out in Academic/Government organizations. Those type organizations are familiar with UN*X and UN*X-alikes and will stick with what they know. The other main reason why UN*X and UN*X-alike clusters are cheaper out of the box than MSFT clusters. A university can throw together a bunch of hardware and have grad students maintain it for practically free running UN*X/Linux. Government labs can write the cost of the hardware into a contract and not pay for the software and use that cost to either reduce the contract billing, apply the money into more/better hardware, or put it into some other part of the contract. The fact is that there are a number of Windows clusters out there that perform well and are used as serious computational engines, if you care to look for them. In Academia/Gov't they are very rare though.
I, too, have worked for a cluster vendor and have been in clustering for some time now. The main difference between us, I think, is that I am less biased.
A company out there provides commercial MPI libraries for a variety of operating systems, Windows included
http://www.mpi-softtech.com/
A couple years back, I was one of the MPI writers/maintainers for a number of platforms (worked with MPICH). As far as performance went with real applications (as well as synthetic benchmarks), Windows and Linux on the same hardware were pretty much the same. Typically computational problems were faster on Windows though because of the better compiler support at the time. Communications performance benchmarks were interesting.
Also, the at the URL above you can find cluster management software (batch scheduling and stuff).
Yes, I was thinking that thread mobility in those benchmarks was probably the reason why the poor scores were generated. Along with moving the threads, you of course have all the problems associated with it. If you look at the benchmarks on that page, Hyperthreading makes the single CPU faster usually, which we could have predicted (and is surely what Intel wants). On the dual CPU configuration, it actually hinders performance in many cases. There are some other issues with treating one of these CPUs as two virtual CPUs. First, it isn't really two CPUs and threads scheduled on it will not necessarily be executed with fair time slices. If you schedule two threads on one of these CPUs, it is possible that one of the threads uses the majority of the resources for the timeslice and the other does little - until you reschedule it because the timeslice has passed. So basically, you can pay the penalty of two context switches for little/no work done. Also, remember that a CPU, even in highly optimized code, is not running at 100% of its potential - using hyperthreading you aren't getting a 30% faster CPU, you (should be) using the one you have more efficiently in terms of keeping the various execution units busy. I read somewhere that a typical instruction stream in a P3/P4 only keeps 33% to 50% of the execution units (ALU, FPU, etc.) busy. Hyperthreading simply tries to keep more units busy more of the time.
Yes. Most programmers have to sign a non-conflict of interest agreement when they are hired. If you work for IBM and also help develop a competing product on the side, then you are in Conflict of Interest (honestly whether or not you signed the agreement, imo).
1 Gigabit of RAM?
Another system that used something like this was Tera: http://www.sdsc.edu/SDSCwire/v3.18/tera.html However, what they did was have 128 contexts per CPU and it round-robin'd through them all. You could also "daisychain" multiple CPUs together in a system. It was interesting but I don't know what ever happened to the machines they were building.
It is easier to think of this in terms of the OS concepts of a paging system.
When Thread1 gets a Load-Fault (think of Pagefault), it executes Thread2 instuctions that it has cached. Much like an OS will execute Process2 if Process1 hits a pagefault. Basically what the hardware is there to do is to fill up the bubbles in the pipeline that are caused by one thread with instructions from another thread.