Typically, you'd keep coherence on the cache line level, not the page level. That way, you can have a multithreaded job that has a lot of stuff in the same page share it nicely if each thread doen't bang on the same cache line as the others. Also, any processor with multi-processor capabilities will have the ability to synchronize cache lines between CPU's.
Actually, NUMA, in general, is for a single box, not multiple boxes. It is also most assuredly *not* software. There are certain things you can/have to do to the software (especially the OS) to get it to run well, however, the base support is in hardware. You don't want the local processor to be involved at all (except for cache line invalidation) when a remote processor wants some memory on this node. The "4-way Xeon box" is probably a special 4-way motherboard with a big ASIC on it to handle the memory exchanges over a special (and probably very fast) bus interconnecting the nodes.
I'm not sure how IBM/Sequent does it, but we (SGI) do it that way - the Origin 2000 and the followon to it have a board with processors, memory, I/O channels, and a "network link" to the intramachine memory network. These are linked together with a big ASIC, which we call the Hub. The Hub is linked to routers and the routers are connected in something approximating a hypercube. The latency to memory gets longer the further you are away and the more router hops you have to go. Our architecture scales to 512 (and perhaps 1024 in the future) processors. A future version will be based on IA64 and will run Linux also. I'm not sure if IBM needs routers as they only go to 64p.
So what do you have to do to make the OS run? The big thing is getting the OS to recognize holes in memory which may exist between nodes. Once you have done that, you can probably boot. To actually run well, you have to modify the memory allocation and scheduler (among other things) to try to keep jobs physically close to their memory. This gets *really* hard, especially when the job takes more than one node's worth of memory.
One last thing I should point out is that there are two flavors of NUMA. Regular NUMA and ccNUMA. In regular NUMA machines (the Cray T3D/T3E is the only one that comes to mind) you can get access to memory that is not on your node, but you will only get a snapshot of what is there at a given moment. In ccNUMA, the caches of all the processors are coherent so not only do you get a snapshot, but if you modify your copy, everyone will perceive that the modification happened at the same time. The T3E runs a seperate copy of the OS on every processor partly because of the lack of the "cc" part. The Origin stuff can run a single kernel on the entire system and my group is doing software which breaks the machine up and runs seperate kernel images in different pieces for reliability reasons. The images can then talk to each other over the memory interconnect at very high speeds.
Finally, 4GB of memory is nothing:) I regularly play with a machine that has 196 gigs of RAM (and 512 processors:) and it's nowhere near maxed out:)
You keep asserting that declaring one machine a firewall won't "create more cycles" and you're right. However, it may decrease the number of cycles being consumed by the OS. Let's say you do a lot of filtering on the back end server. Now, you're context switching between the filtering software (which may be a part of the kernel) and the web server, which is going to be generating a lot of I/O interupts on its own accessing disk/SQL server/whatever. All of that context switching takes time. By declaring one machine "a firewall", it is free to basically sit and run the firewall software all the time and not really context switch ever. This means less cycles consumed by the OS. Which leaves more cycles free, overall, for the application (web server) to run. So, in essence, you *have* "created cycles".
For an extreme example of how having dedicated processors can help get more cycles for applications, check out how Unicos/mk allocates processors on the T3E. It has seperate interactive/ OS / I/O type processors with a much larger number of dedicated compute processors. This is the same idea, for a different application.
It's more like saying "Ferrari has de-emphasized speed and will produce freight trains instead":) Large boxes are in a different market from where we were, but they still fly:) Also, there will be two new SGI graphics cards coming out (one may have already shipped), both of which should blow everyone else away:)
Well, I'll assume your ex-SGI friends were in Graphics. I can see why they might have been a bit irritated as we seem to have de-emphasized graphics, but overall, SGI is a very nice place to work (at least in Eagan at the old Cray place). We all have our own offices in a very nice building and we'll be building 512 processor Linux boxes soon:) In the mean time, we're building 512 processor Irix boxes. It's a very exciting place to work. Yes, I do work there:)
There may be, but I think the point of this is that they were actually talking *to* the satelite rather than using it as a passthrough for other data. I think I read someplace in an article on the LAN they are planning for the ISS (if it ever gets built) that the ISS will have a TCP/IP downlink to NASA.
There is basically only one disadvantage that I can think of, but it's kinda big - if you open the source to the driver of the video card, it would definetly help anyone who knows video cards (say, someone at ATI) who was trying to reverse engineer nVidia's chip. You'd be able to see what registers they have on the chip and how software can take advantage of them. Sure, you could just disassemble a closed source driver but it would sure be nice to have all the code, with the comments, right?:) I doubt that nVidia cares if Joe Random Linux user sees the source - what they would be afraid of is people at 3dfx or ATI or whoever looking at the source and then incorporating it into the next hardware revision.
Are their fears justified? Who knows, but nVidia has decided not to take that risk and unless you own stock in them, you don't have the right to argue about that descision.
All of this, of course, is seperate from what they did here - they can't just pick up random code off the net and include it in their drivers without permission!
I'll pick a few nits here. First, on what statistics did you beat SGI's machines on (ok, yeah, I know that that's probably at least partly confidential info but I figured I'd ask anyway:) I guess it would not surprise me if it were price/performance, but it would surprise me if you beat us on things like MPI latency. The last I heard, the O2k was an order of magnitude better than Myrinet on IA32 Linux. I'd expect Alpha Linux to fare better, though. The biggest nit I'm going to pick is your assertion of running a single system image. That is true only if you can migrate processes between nodes in the cluster or transparently change your interconnect fabric to keep nodes running the same job physically close. (Assume a 512p machine or cluster) What if I allocate 128 processors for job A, then start 256p job B followed by 128p job C. Jobs A and C finish, but their processors are on opposite sides of the cluster. Now I start Job D which needs 256 CPU's. Unless you can reorg your interconnect the way a job can move on a single machine, you will end up with fragmented processors for Job D. That'll hurt your latency. Also, not everyone runs only MPI. Shared memory jobs are not going to be too easy on a cluster, unless you have distributed shared memory. Further, it's not exactly an SSI if the sysadmin has to install the oS on every node or has a seperate console connection to every node. The point is, if you provide a "single system image", you provide it *only* to the users of MPI, not to the sysadmin.
I should also point out that SGI is not the only company that makes traditional supercomputers. I'd like to see how your cluster stacks up against a T3E or an SP2. Probably not quite so well.
Finally, the Origin is about 5 years old. The next machine will be an order of magnitude better.
Now, I'm not trying to say that clusters suck for all applications. They just aren't the solution to *every* problem, as a lot of people claim they are.
I'll use this as a good excuse to answer the "why don't they just use 10-base-T or stripped down PC mobo's" questions people have been asking.
Once you start trying to solve problems that require large amounts of inter-node communication, low latency and high speed interconnects become your limiting factor. Traditional supercomputers have massive interconnects. I have heard 45 gigabits/sec per Alpha (or per router board, can't remember which) in a Cray T3E. The new machine that the project I'm on at SGI is working on has 1.6GigaBytes/sec off each node (4 MIPS or Itanium CPU's/node) for inter-machine memory access plus I/O. 10-base-T clearly doesn't cut it. Things like Myrinet have gone a long way towards closing the gap with traditional supercomputers, but they have also dramatically bumped up the cost of your network. GSN is a prime example of this.
Now, for why you don't want a cheap mobo. Just as important (perhaps moreso) is the bandwidth from the processor to memory. The thing about working with huge datasets is that cache doesn't really help you. If you are about to look at 4 gigs of data sequentially, you aren't going to cache a whole lot. Therefore, your memory has to be capable of streaming a very large number of reads and writes to the processor. Remember that it doesn't really matter how fast your processor is if it spends half its time stalled waiting for memory accesses to complete. One of the goals in all the old Cray products was to never have your processor sitting idle. The memory could handle a constant stream of data going both in and out (ie, one memory read and write per cycle). Your standard cheap PC mobo just can't do that.
I'll end with my sales pitch for traditional supercomputers - by the time you buy supercomputer class nodes and a supercomputer class interconnect, even if it's built from comodity components, your costs will approach a traditional large system. You also don't get the advantages of having a single system image (I use a single image 512p Origin 2000 on a regular basis) or even direct memory access from one node to another (the project I'm on at SGI is to break large systems into multiple images, but allow them to share user memory. That way if one panics, the worst that happens on the other side is the loss of the user ap that was sharing the memory). This is why people are still buying big iron. On the other hand, we're starting to sell Linux clusters to the people who don't require the massive bandwidth.
> No one has seen a machine until they've seen a Starfire
I look at Crays and SGI Origin 2000's (up to 512 processors) all day long and I can assure you that they look way more impressive than a Starfire. Remember that Starfire was designed by Cray engineers for the low end of Cray's market....
Where would a 1 TF computer that they describe rank? That depends. The big deal in speed isn't just how many flops your processor can crank out. Note that they say that that is 1 TF peak, not 1 TF achieved. I suspect that you wouldn't even come close to that with a mobo that was as cheap as they are talking about - who cares how fast the processor goes if it's waiting for RAM half the time? Then, once you have gotten a fast enough bus to provide the processor, you are only going to get peak speed when the processors aren't talking to each other. If you are running, say, a render farm where each machine is basically off doing its own thing, you might hit your peak performance. However, if you are doing a river flow analysis, there's no way you're going to hit peak unless your interconnect is *really* fast. This is where large single system images like a Cray T3E or an SGI Origin 2000 are big wins. You can do direct user to user copies of memory without involving the I/O path on either node. Things like GSN are trying to change that but we've found that they are actually as expensive or more expensive than the same number of processors hooked up in a single system image.
Prepare to be surprised:) An 18processor Origin 2000 is well under $1M. Though I suppose that depends on what you mean by "node". The "official" SGI definition of a node board is 2 processors + some RAM. Up to 4 nodes go in a module. 2 modules in a rack, up to 4 racks before you start adding metarouters, which gets messy. The largest O2k system is 40 racks which is 512 processors.
Anyway, my point was that unless you are calling an entire module (8 processors) a node, that machine was probably less than $0.5 million. If you call a node a processor, then an 18p Origin is probably less than $200,000.
The numbers above are quite approximate and aren't official SGI price quotes. I speak for myself, not SGI. (Yeah, I *do* fear the lawyers:)
Well, SGI is committed to producing new MIPS machines through at least 2004 (? that may not be the exact number but I know it's about 4 years) at a minimum. If Merced flops, we will still have MIPS/Irix to sell:) In fact the machine under development releasing soon (SN1) will be able to use exactly the same hardware (I think you may need bigger power supplies with Intel) for everything except the processor cards which will be either MIPS or Intel. And it will scale to 512 processors on both! The above was not an official product announcement from SGI. Actual announcement may vary. Void where prohibited. Batteries not included.
There are a lot of advantages of large systems over a group of PC's. Chief among them is I/O bandwidth. A web server, unless it's running some really complex perl scripts or is also running the database back end (not the best idea due to security, BTW...), is not usually CPU bound. Basically, all the CPU is doing is saying "send this file to this IP address". Not a massive amoung of processing power needed there. So then, your limiting factor becomes your I/O bus. The I/O on a PC is pathetic compared to what you can get on a mainframe class system. Granted you can tie a bunch of systems together and get that much I/O, but then you have a bunch of systems which leads me to my next point - sysadmining one system is a lot easier than sysadmining 8 systems working together. Then, on top of that, you have the load balancing software and firewall you use to make them all look like one machine. Now, instead of your single point of failure being the machine, it's the firewall. By the time you buy all that hardware to get up to the reliability of a mainframe, you may end up spending almost as much as buying a mainframe.
One last point in favor of mainframes is one very few people think about until it's too late - ease of service. Hardware, especially hardware that's being run at full capacity 24/7, breaks. When it breaks, it's nice if you can replace it easily. This is where PC's really suck and mainframes are quite nice - they are designed to be serviced, often while running. Your ethernet board died? No problem - swap in a new one without rebooting (I assume IBM can do that? I'm sure someone will flame me with a correction if necessary:) Redundant power supplies are also quite nice.
This is not saying that a mainframe is right for *every* webserver, but when you lose millions a day for going down, even a small increase in reliability or ease of service may well be worth the extra cash up front!
#53 is irony. Happy?:) Perhaps it's the lack of sleep and the general frustration with seeing my company and indirectly my coworkers flamed every time we release something as Open Source, but I sure read that as flamebait. If it wasn't intended that way, I appologize for the tone of my reply.
This is kinda unrelated but I would like to say this - perhaps this post was meant as a joke, but others of this nature often aren't. I've noticed that any time a company releases a product open source, the first thing that happens is they get flamed on Slashdot. This does *not* encourage companies to release more software. I've been in meetings where first and second line managers (I'm a peon, not a manager, though) have questioned SGI's strategy of going open source because they have seen comments here and wonder if we will ever be accepted. Remember that when you post, people! Your posts *do* get read, sometimes even by important people! EOR (End Of Rant)
*sigh* I don't even know why I bother to respond to crap like this. But I will anyway...
a) SGI can do nothing right: so I guess switching to Linux is wrong? Making a very high percentage of the machines on the Top500 list is wrong? Um, ok...
b) They have crappy unscalable hardware: So I guess Onyx II Infinite Reality Graphics are crappy? Hate to break it to you, but, while they may be a bit pricey, there ain't nothin' much faster. As for unscalable, 512 processors isn't scalable? Please. I run Irix on a machine with 512 processors and 196 gigs of RAM. Can Linux do that? Other than Cray, and Intel's one-off for ASCI, does anyone make anything bigger? Granted we (I work for SGI, in case you couldn't tell) are selling Cray, but the T3E has been sold in configurations of 1800 processors and the architecture scales even further. I think that qualifies as "scalable".
c) Inferior OS with no features Linux doesn't have: Pass me whatever *you're* smoking please. How about a journaling file system that is production ready? Scalable to 512 processors? ccNUMA support? Runs Alias|Wavefront applications that produce probably at least half the special effects you see on TV/movies? I'm sure there are more, but I don't feel like coming up with them. Now don't get me wrong - Linux is a great OS, but that doesn't mean that Linux lacks no feature found in Irix.
d) Public commercial company: RedHat. VA Reasearch. Need I go on?
e) Secret motives to steal the genious from Linux: And just how would we do that even if we wanted to??? All we would succeed in doing is getting everyone upset with us and ending up with a propietary version of Linux. Where, exactly, would that get us besides bankruptcy court? If it were up to us, we'd probably insert massive scalability features into Linux like, say, support for 512 processor SSI's. But, the Linux community would never accept those changes so we simply won't make them until the community will. Trust me, SGI is far more interested in playing by the rules than I'll bet most "Linux companies" out there.
If you want a company that keeps mumbling about contributions to Linux/Open Source and doesn't deliver, think of Sun, not SGI.
That was the "other" Cray. I believe it was called "Cray Supercomputer". The Cray most people know is Cray Research which started in Chippewa Falls, Wisconsin and expanded to Eagan, MN (where I work for its remnants). The "other" Cray was started when Seymore Cray left Cray Research. Rumor has it that Cray Supercomputer (I don't think that's the name anymore) has recently shipped a prototype. Cray Research is (assuming SGI gets it done) going to become a seperate company again and has a *very* cool product coming down, which will blow this cluster off the map:) (Since I'm on the SGI side of the split I am obligated to say that we also have a product coming down that is very cool and will take on this cluster very nicely. And not only that, a later version of it will run Linux - you could really make a nice Beow....oh nevermind:)
> Move over SGI graphics workstations, Linux is here.
There will be future SGI graphics workstations that run* Linux:) This would be (at least from what I've heard) one of the reasons we open-sourced OpenGL.
* - This is in addition to the ones that run Irix, which, of course, will continue to be produced.
>I want to see what the big players (IBM, Sun, NEC, Fujitsu) have to comment [if you note the absence of SGI in the above list, it was intentional.]
So I guess the company with 6 of the top 10 systems isn't big enough for ya, huh? From the Top 500, the top 10 systems are made by 4 manufacturers: Intel, IBM, SGI/Cray, and Hitachi. Those are what I would call the "Big players". (Well, not Intel). NEC and Fujitsu, I guess are also pretty big. But Sun?? Sorry - not yet in the supercomputing market. Take another look at the list and then tell me SGI isn't a "big player".
Despite all perceptions to the contrary, SGI and Cray aren't dead yet!! Both companies (though at the moment we're one) have phenomenal new products coming out.
I speak for myself, not my employer (SGI/Cray Research).
I suggest you take another look at Irix. We regularly have 128 processor systems internally that are running everything from vi to MPI simulations to software compiles all at the same time. I doubt that we are using the system at 100% of its potential, but that's because a lot of what we need (interactive jobs and compiles) are I/O bound. So yes, there are operating systems that "just scale to 100 processors." Not that I am saying it's easy. My group (long prior to my joining it) put a *ton* of work into making Irix scale well on large single system images. We now have a 512 processor system running a single copy of the OS.
Well, I won't claim to be the world's expert on this, but I do work with real supercomputers and (to a lesser extent) clusters on a daily basis (disclaimer: I work for SGI at the old Cray Research, though I don't speak for them).
I too am very skeptical of this. From what I can tell of their web page, they don't scale very far since they only have 8 compute nodes. Even assuming 4 processors per node (and I think they only have 1) that would only be 32 processors. Granted, for 99.9% of the users out there, that is a whole boatload of processors. However, our large systems go to 512 processors (1024? Maybe in not too long). The Cray T3E goes up to 2048p. Even our clustering product generally has more than that. The cluster we recently installed at Ohio Supercomputing Center was 256 compute nodes, I believe. I just don't see how 8 to 32 processors is going to compete with that. Now their reliability looks pretty solid, though.
Another problem, in general, with x86 "supercomputing" is that a lot of scientific code out there likes 64 bit math. Merced^H^H^H^H^H^HItanium, MIPS, Alpha, and the Cray Vector processors have a nice lead there.
Lastly, someone previous in this thread said something to the effect "wouldn't it be great to make the top 500 for In short, I don't consider this to be a supercomputer. An HA cluster, maybe. But it's hard to tell since their site is pretty sparse on technical details. I am *very* suspicious of a "supercomputer" company that doesn't post benchmarks. One of Seymore Cray's rules of supercomputing is that your machine should be the fastest in the world at *something*. Lastly, they need to learn how to put in "'s instead of ?'s. Their HTML is inept at best...
Re:Here goes Katz again
on
AOL Nation
·
· Score: 1
I wouldn't exactly call the popular vote a "sham". True you don't get to vote on the electoral college electors directly, but (unless I have majorly confused my facts), the candidate who won the popular vote has always been elected president. While the margin of victory is usually much higher in the electoral college, the result is the same. However, this then makes the electoral college uterly useless, which is *my* reason for wanting to see it go away:)
Well, here are several reasons I didn't stay (graduated from CMU in May) - one is that commuting is aweful. The parkway *sucks* at rush hour and if you have to go through one of the tunnels, you lose big time. Roads in general suck (I hate playing "dodge the pothole" - you can feel the border between the Ohio Tpk and the PA Tpk) in Pittsburgh and are twisty and easy to get lost on. The weather is also quite messy - way too much rain and ice. Give me snow any day:) Last, but certainly not least, most of the high tech jobs that I found in Pittsburgh were either in robotics (which is mostly electrical engineering type stuff) or applications software of various kinds. I like operating systems.
I am now in Minneapolis and the reasons I like it here are climate (hey, I'm from Wisconsin - I like winter:), easy commuting, low cost of living, availability of lots of theater that I can get involved in, and the room full of Crays in the basement:)
Well, I certainly don't think big insurance companies need any more laws to protect their bottom lines. I was mearly trying to point out that your descision to prescribe yourself a drug when you aren't qualified to do so doesn't necessarily only affect your own body. If it results in higher health care costs overall, we all pay for that.
The question you ask (how dangerous is too dangerous) is definetly both valid and extremely hard to answer. I certainly don't pretend to have the answer. Note that medicine is hardly the only area where you need a license to practice - we consider it too dangerous to have unlicensed drivers on the road and unlicensed pilots in the air. Why? Because they can be a danger to others if they screw up. Self prescribed medicine can be a danger too if you prescribe something like morphine and get addicted (for instance).
Slashdot Mirror
Typically, you'd keep coherence on the cache line level, not the page level. That way, you can have a multithreaded job that has a lot of stuff in the same page share it nicely if each thread doen't bang on the same cache line as the others. Also, any processor with multi-processor capabilities will have the ability to synchronize cache lines between CPU's.
I'm not sure how IBM/Sequent does it, but we (SGI) do it that way - the Origin 2000 and the followon to it have a board with processors, memory, I/O channels, and a "network link" to the intramachine memory network. These are linked together with a big ASIC, which we call the Hub. The Hub is linked to routers and the routers are connected in something approximating a hypercube. The latency to memory gets longer the further you are away and the more router hops you have to go. Our architecture scales to 512 (and perhaps 1024 in the future) processors. A future version will be based on IA64 and will run Linux also. I'm not sure if IBM needs routers as they only go to 64p.
So what do you have to do to make the OS run? The big thing is getting the OS to recognize holes in memory which may exist between nodes. Once you have done that, you can probably boot. To actually run well, you have to modify the memory allocation and scheduler (among other things) to try to keep jobs physically close to their memory. This gets *really* hard, especially when the job takes more than one node's worth of memory.
One last thing I should point out is that there are two flavors of NUMA. Regular NUMA and ccNUMA. In regular NUMA machines (the Cray T3D/T3E is the only one that comes to mind) you can get access to memory that is not on your node, but you will only get a snapshot of what is there at a given moment. In ccNUMA, the caches of all the processors are coherent so not only do you get a snapshot, but if you modify your copy, everyone will perceive that the modification happened at the same time. The T3E runs a seperate copy of the OS on every processor partly because of the lack of the "cc" part. The Origin stuff can run a single kernel on the entire system and my group is doing software which breaks the machine up and runs seperate kernel images in different pieces for reliability reasons. The images can then talk to each other over the memory interconnect at very high speeds.
Finally, 4GB of memory is nothing :) I regularly play with a machine that has 196 gigs of RAM (and 512 processors :) and it's nowhere near maxed out :)
For an extreme example of how having dedicated processors can help get more cycles for applications, check out how Unicos/mk allocates processors on the T3E. It has seperate interactive/ OS / I/O type processors with a much larger number of dedicated compute processors. This is the same idea, for a different application.
It's more like saying "Ferrari has de-emphasized speed and will produce freight trains instead" :) Large boxes are in a different market from where we were, but they still fly :) Also, there will be two new SGI graphics cards coming out (one may have already shipped), both of which should blow everyone else away :)
Well, I'll assume your ex-SGI friends were in Graphics. I can see why they might have been a bit irritated as we seem to have de-emphasized graphics, but overall, SGI is a very nice place to work (at least in Eagan at the old Cray place). We all have our own offices in a very nice building and we'll be building 512 processor Linux boxes soon :) In the mean time, we're building 512 processor Irix boxes. It's a very exciting place to work. Yes, I do work there :)
There may be, but I think the point of this is that they were actually talking *to* the satelite rather than using it as a passthrough for other data. I think I read someplace in an article on the LAN they are planning for the ISS (if it ever gets built) that the ISS will have a TCP/IP downlink to NASA.
Are their fears justified? Who knows, but nVidia has decided not to take that risk and unless you own stock in them, you don't have the right to argue about that descision.
All of this, of course, is seperate from what they did here - they can't just pick up random code off the net and include it in their drivers without permission!
I should also point out that SGI is not the only company that makes traditional supercomputers. I'd like to see how your cluster stacks up against a T3E or an SP2. Probably not quite so well.
Finally, the Origin is about 5 years old. The next machine will be an order of magnitude better.
Now, I'm not trying to say that clusters suck for all applications. They just aren't the solution to *every* problem, as a lot of people claim they are.
Once you start trying to solve problems that require large amounts of inter-node communication, low latency and high speed interconnects become your limiting factor. Traditional supercomputers have massive interconnects. I have heard 45 gigabits/sec per Alpha (or per router board, can't remember which) in a Cray T3E. The new machine that the project I'm on at SGI is working on has 1.6GigaBytes/sec off each node (4 MIPS or Itanium CPU's/node) for inter-machine memory access plus I/O. 10-base-T clearly doesn't cut it. Things like Myrinet have gone a long way towards closing the gap with traditional supercomputers, but they have also dramatically bumped up the cost of your network. GSN is a prime example of this.
Now, for why you don't want a cheap mobo. Just as important (perhaps moreso) is the bandwidth from the processor to memory. The thing about working with huge datasets is that cache doesn't really help you. If you are about to look at 4 gigs of data sequentially, you aren't going to cache a whole lot. Therefore, your memory has to be capable of streaming a very large number of reads and writes to the processor. Remember that it doesn't really matter how fast your processor is if it spends half its time stalled waiting for memory accesses to complete. One of the goals in all the old Cray products was to never have your processor sitting idle. The memory could handle a constant stream of data going both in and out (ie, one memory read and write per cycle). Your standard cheap PC mobo just can't do that.
I'll end with my sales pitch for traditional supercomputers - by the time you buy supercomputer class nodes and a supercomputer class interconnect, even if it's built from comodity components, your costs will approach a traditional large system. You also don't get the advantages of having a single system image (I use a single image 512p Origin 2000 on a regular basis) or even direct memory access from one node to another (the project I'm on at SGI is to break large systems into multiple images, but allow them to share user memory. That way if one panics, the worst that happens on the other side is the loss of the user ap that was sharing the memory). This is why people are still buying big iron. On the other hand, we're starting to sell Linux clusters to the people who don't require the massive bandwidth.
Or Cray - they put the dot in .mil :)
I look at Crays and SGI Origin 2000's (up to 512 processors) all day long and I can assure you that they look way more impressive than a Starfire. Remember that Starfire was designed by Cray engineers for the low end of Cray's market....
Where would a 1 TF computer that they describe rank? That depends. The big deal in speed isn't just how many flops your processor can crank out. Note that they say that that is 1 TF peak, not 1 TF achieved. I suspect that you wouldn't even come close to that with a mobo that was as cheap as they are talking about - who cares how fast the processor goes if it's waiting for RAM half the time? Then, once you have gotten a fast enough bus to provide the processor, you are only going to get peak speed when the processors aren't talking to each other. If you are running, say, a render farm where each machine is basically off doing its own thing, you might hit your peak performance. However, if you are doing a river flow analysis, there's no way you're going to hit peak unless your interconnect is *really* fast. This is where large single system images like a Cray T3E or an SGI Origin 2000 are big wins. You can do direct user to user copies of memory without involving the I/O path on either node. Things like GSN are trying to change that but we've found that they are actually as expensive or more expensive than the same number of processors hooked up in a single system image.
Anyway, my point was that unless you are calling an entire module (8 processors) a node, that machine was probably less than $0.5 million. If you call a node a processor, then an 18p Origin is probably less than $200,000.
The numbers above are quite approximate and aren't official SGI price quotes. I speak for myself, not SGI. (Yeah, I *do* fear the lawyers :)
Well, SGI is committed to producing new MIPS machines through at least 2004 (? that may not be the exact number but I know it's about 4 years) at a minimum. If Merced flops, we will still have MIPS/Irix to sell :) In fact the machine under development releasing soon (SN1) will be able to use exactly the same hardware (I think you may need bigger power supplies with Intel) for everything except the processor cards which will be either MIPS or Intel. And it will scale to 512 processors on both! The above was not an official product announcement from SGI. Actual announcement may vary. Void where prohibited. Batteries not included.
One last point in favor of mainframes is one very few people think about until it's too late - ease of service. Hardware, especially hardware that's being run at full capacity 24/7, breaks. When it breaks, it's nice if you can replace it easily. This is where PC's really suck and mainframes are quite nice - they are designed to be serviced, often while running. Your ethernet board died? No problem - swap in a new one without rebooting (I assume IBM can do that? I'm sure someone will flame me with a correction if necessary :) Redundant power supplies are also quite nice.
This is not saying that a mainframe is right for *every* webserver, but when you lose millions a day for going down, even a small increase in reliability or ease of service may well be worth the extra cash up front!
This is kinda unrelated but I would like to say this - perhaps this post was meant as a joke, but others of this nature often aren't. I've noticed that any time a company releases a product open source, the first thing that happens is they get flamed on Slashdot. This does *not* encourage companies to release more software. I've been in meetings where first and second line managers (I'm a peon, not a manager, though) have questioned SGI's strategy of going open source because they have seen comments here and wonder if we will ever be accepted. Remember that when you post, people! Your posts *do* get read, sometimes even by important people! EOR (End Of Rant)
a) SGI can do nothing right: so I guess switching to Linux is wrong? Making a very high percentage of the machines on the Top500 list is wrong? Um, ok...
b) They have crappy unscalable hardware: So I guess Onyx II Infinite Reality Graphics are crappy? Hate to break it to you, but, while they may be a bit pricey, there ain't nothin' much faster. As for unscalable, 512 processors isn't scalable? Please. I run Irix on a machine with 512 processors and 196 gigs of RAM. Can Linux do that? Other than Cray, and Intel's one-off for ASCI, does anyone make anything bigger? Granted we (I work for SGI, in case you couldn't tell) are selling Cray, but the T3E has been sold in configurations of 1800 processors and the architecture scales even further. I think that qualifies as "scalable".
c) Inferior OS with no features Linux doesn't have: Pass me whatever *you're* smoking please. How about a journaling file system that is production ready? Scalable to 512 processors? ccNUMA support? Runs Alias|Wavefront applications that produce probably at least half the special effects you see on TV/movies? I'm sure there are more, but I don't feel like coming up with them. Now don't get me wrong - Linux is a great OS, but that doesn't mean that Linux lacks no feature found in Irix.
d) Public commercial company: RedHat. VA Reasearch. Need I go on?
e) Secret motives to steal the genious from Linux: And just how would we do that even if we wanted to??? All we would succeed in doing is getting everyone upset with us and ending up with a propietary version of Linux. Where, exactly, would that get us besides bankruptcy court? If it were up to us, we'd probably insert massive scalability features into Linux like, say, support for 512 processor SSI's. But, the Linux community would never accept those changes so we simply won't make them until the community will. Trust me, SGI is far more interested in playing by the rules than I'll bet most "Linux companies" out there.
If you want a company that keeps mumbling about contributions to Linux/Open Source and doesn't deliver, think of Sun, not SGI.
That was the "other" Cray. I believe it was called "Cray Supercomputer". The Cray most people know is Cray Research which started in Chippewa Falls, Wisconsin and expanded to Eagan, MN (where I work for its remnants). The "other" Cray was started when Seymore Cray left Cray Research. Rumor has it that Cray Supercomputer (I don't think that's the name anymore) has recently shipped a prototype. Cray Research is (assuming SGI gets it done) going to become a seperate company again and has a *very* cool product coming down, which will blow this cluster off the map :) (Since I'm on the SGI side of the split I am obligated to say that we also have a product coming down that is very cool and will take on this cluster very nicely. And not only that, a later version of it will run Linux - you could really make a nice Beow....oh nevermind :)
There will be future SGI graphics workstations that run* Linux :) This would be (at least from what I've heard) one of the reasons we open-sourced OpenGL.
* - This is in addition to the ones that run Irix, which, of course, will continue to be produced.
I speak for myself, not for SGI.
So I guess the company with 6 of the top 10 systems isn't big enough for ya, huh? From the Top 500, the top 10 systems are made by 4 manufacturers: Intel, IBM, SGI/Cray, and Hitachi. Those are what I would call the "Big players". (Well, not Intel). NEC and Fujitsu, I guess are also pretty big. But Sun?? Sorry - not yet in the supercomputing market. Take another look at the list and then tell me SGI isn't a "big player".
Despite all perceptions to the contrary, SGI and Cray aren't dead yet!! Both companies (though at the moment we're one) have phenomenal new products coming out.
I speak for myself, not my employer (SGI/Cray Research).
I speak for myself, not for SGI.
I too am very skeptical of this. From what I can tell of their web page, they don't scale very far since they only have 8 compute nodes. Even assuming 4 processors per node (and I think they only have 1) that would only be 32 processors. Granted, for 99.9% of the users out there, that is a whole boatload of processors. However, our large systems go to 512 processors (1024? Maybe in not too long). The Cray T3E goes up to 2048p. Even our clustering product generally has more than that. The cluster we recently installed at Ohio Supercomputing Center was 256 compute nodes, I believe. I just don't see how 8 to 32 processors is going to compete with that. Now their reliability looks pretty solid, though.
Another problem, in general, with x86 "supercomputing" is that a lot of scientific code out there likes 64 bit math. Merced^H^H^H^H^H^HItanium, MIPS, Alpha, and the Cray Vector processors have a nice lead there.
Lastly, someone previous in this thread said something to the effect "wouldn't it be great to make the top 500 for In short, I don't consider this to be a supercomputer. An HA cluster, maybe. But it's hard to tell since their site is pretty sparse on technical details. I am *very* suspicious of a "supercomputer" company that doesn't post benchmarks. One of Seymore Cray's rules of supercomputing is that your machine should be the fastest in the world at *something*. Lastly, they need to learn how to put in "'s instead of ?'s. Their HTML is inept at best...
I wouldn't exactly call the popular vote a "sham". True you don't get to vote on the electoral college electors directly, but (unless I have majorly confused my facts), the candidate who won the popular vote has always been elected president. While the margin of victory is usually much higher in the electoral college, the result is the same. However, this then makes the electoral college uterly useless, which is *my* reason for wanting to see it go away :)
I am now in Minneapolis and the reasons I like it here are climate (hey, I'm from Wisconsin - I like winter :), easy commuting, low cost of living, availability of lots of theater that I can get involved in, and the room full of Crays in the basement :)
The question you ask (how dangerous is too dangerous) is definetly both valid and extremely hard to answer. I certainly don't pretend to have the answer. Note that medicine is hardly the only area where you need a license to practice - we consider it too dangerous to have unlicensed drivers on the road and unlicensed pilots in the air. Why? Because they can be a danger to others if they screw up. Self prescribed medicine can be a danger too if you prescribe something like morphine and get addicted (for instance).
Anyway, I've rambled on long enough.