Sun Releases Starcat

SilentChris writes: "Sun has released the Starcat server, a beast with up to 106 processors running Unix. Anyone have an extra couple [million] bucks lying around?" They're not cheap.

Comments

For those beowolf comments

[segfaultcoredump]

Lets remember, that this system is not intended to replace a beowolf cluster of cheap pc's. It is intended to do something that most beowolf clusters can never do: present a single OS image with half a terabyte of memory that any cpu can access at very high speed.

This is a system that is very good at things like fluid dynamics and massive database operations. It is not a good idea if all you want to do is get to the top of the list for the SETI@Home project

Re:For those beowolf comments

[Webmonger]

In terms of memory bandwidth and latency, they are very different.

The fastest networking technologies do not approach the speed and responsiveness of a memory bus. Yet a cluster design uses networking in place of a memory bus some of the time.

If there's not a lot of data, it doesn't matter much. If there's tons and tons of data, a cluster design is inefficient.

Re:For those beowolf comments

[tolldog]

Agreed.
This would make a killer render system, assuming the renderer can handle that many threads.

This is why beowulf rendering is bad. Network performance for shared memory sucks.

With renders hitting the 2GB + mark for memory useage, do you really want a network passing that data arround.

What could happen with systems like this is that the render time vs. load time would get extremely lopsided. 30 minute loads and under a minute a frame. It would force a rethink of how the render jobs get distributed and ran.Best case would be a few of these, for each different departments render needs. But then we are talking 20+ million for rendering. That buys a lot of intel boxes.
If I was given one, I would try to use it. But I don't think I could ever seriously suggest buying one. But that is me and my particular application.

Re:For those beowolf comments

[Valgar]

On the bigger boxen, you have the happy ability of dynamic reconfiguration. Plus the kernel_cage_enable=1 tag in /etc/system (There is more but I'm going to keep this simple).
I'm pulled entire memory/cpu boards out of running E4500's without a problem when cpu failure occurs.
I even have had to script dynamic online-offline of processors for various load testing purposes to determine how the system scales as CPU's are added/removed
If there is one thing about the big iron I like, it is the ability to manipulate the system on a level such as this.
Now Stratus systems are even more amazing...but as they said in Conan. "That is another story"

Re:For those beowolf comments

[Ryan Amos]

Unless they're using superconductive alloys (which are not at all conductive until they hit their temperature) then that shouldn't be a problem. There probably is some sort of superconductive activity if they're using liquid hydrogen, in which case they WANT it to be superconductive.

*yawn*

[eXtro]

This isn't even a really impressive box. I'd rather have an sgi O3K system if I'm going for the ultimate in servers you can actually purchase. The SGI Origin 3800 has anywhere from 16 to 512 processors, 716 GB/sec system bandwidth and up to a terabyte of memory. It's also a single system image machine. Oh yeah, and you can cluster them to scale way beyond 512 processors.

Re:106?

[segfaultcoredump]

The system grows to 106 in the following way:

There are 18 "cpu/memory" boards that hold 4 cpu's each. This brings the system up to a total of 72 cpu's and 576GB of ram.

Now, if you want an server that just does number crunching and dont care about I/O, you can then add 'MaxCPU" modules. Each module holds two additional cpu's (no memory) and occupies the hPCI module slot (a hot swap PCI case that can hold what looks like two to four pci cards). You can use up to 17 of the hPCI module slots to hold MaxCPU modules. (there are 18 pci channels on the system, and at least one must be used for accessing the boot disk).

So there ya have it, 106 cpu's and half a terabyte of ram. I think that in most cases, folks will opt to not use the MaxCPU modules and just stick to the 72 cpu limit.

Re:Clarification please

[Cheetahfeathers]

These can be either. It depends on how you configure it. And the fun this is, you can reconfigure it on the fly. You want a cluster in a box? You got it. You want 2 seperate instances of Solaris running, each using 1/3rd the resources of this box, while you pull out the hardware on the rest of the box for maintenance? You got it. This thing is _configurable_. You can hot swap everything except the backplane, pretty much. It's _sweet_.

Re:partitions

[Doctor_D]

According to the specs each processor board holds 4 processors and 32 gigs of memory.

Now, if the starcat treats domains (partitions) the same as the E10k (I haven't been to training yet on it), then each domain at minimum will consist of 4 processors and 32 gigs of ram, ie 1 processor board. Basicaly these doamins are treated as seperate boxes as far as Solaris is concerned. You configure a domain to say contain 2 system boards, and then when you load Solaris, it then sees 8 processors and 64 gigs of memory. This way you can allocate resources as the need fits. But this means it doesn't look like the virtual processor that mainframes present.

The starcat may deal with processors above 72 in a different way, but I honestly don't know at this time how it deals with them.

Hope this helps answer your question.

Re:Scheduling airplanes?

[heliocentric]

But a lot of the cost of doing business things, which can be done using a graph, don't have to be.

Look into operations research. An early on thing you will learn in OR is the simplex method, where in you boil down a lot of your information (cost to move part A from here to there and buying B if needed, etc...) into a matrix and then use something like the Simplex Method you can maximizie say your profit, or minimize the delays, or minimize the total number of planes in the air, etc..

Now, does the world's flight system have a lot of inputs - you bet, planes, fuel, flight crew, union regulations, holidays, tons of things that you and I could probably never think of...

Now... I just wanted to post that there are ways of solving these problems without a graph - yes a graph is a lovely way for highly dynamic systems, but if you want to answer the question "What is the best schedule given these 1,000 limitations?" then look into operations research. Yes, I mentioned the simplex method, and I fully exepct following posts arguing Parametric Linear Programming or maybe some Markov Chains and their impact, but the core is still the same - if you have a set of limitations and are looking to maximumize (or minimize) something, then operations research is a fun thing!

Re:106 CPUs?

[Anonymous Coward]

In addition to the 72 procs on system board, you can replace most of the hot-swappable PCI daughter boards with procs. So 72 proc on board + 34 PCI board swapped for procs and you get 106 procs.

106 procs, so what

[jshawley]

I'm not sure what is so exciting about a system with 106 processors. When the SGI Origin 3000 can scale to 1024 with a single image of the OS running on it, now that is impressive. Maybe everyone should check out http://www.sgi.com/origin/3000/3800.html Though the site states that it only goes to 512, there is now an official installed system running 1024 that you can see at www.sgi.com/streaming/products.html#sara Now that is IMPRESSIVE!

Re:106 procs, so what

[segfaultcoredump]

The SGI origin has a ccNUMA architecture, which makes it great for some tasks, ok for others, and awful for yet others. (the trick is to make sure that your particular app falls under the 'great' category)

The sun system is an smp based system, everything connects to a common backplane and each board has equal access to all of the other boards. With the sgi, the speed of accessing memory on the local board or boards in the same cabinet is much faster than hits to memory in remote cabinets.

From what I can tell, Sun is planing on producing a special system board that goes into one of those 18 slots. Thus, with 19 StarCats you can create one big system with 1836 cpu's and 9.7TB of ram. (think of a system in the middle that acts as the center of a star) it will most likely be based on a COMA architecture rather than a ccNUMA. Like the SGI, memory access will depend on the distance between the requesting cpu and the storage location. The difference is that under COMA, if a cpu requests a particular bit of memory a lot, that page is either migrated or copied to a memory bank on that cpu's memory board (so if 5 cpu's all need read only access to the same bit of memory, then they can each have their own copy in a local memory bank. write updates are what make the system a pain in the ass to manage ).

Re:partitions

[cornflux]

Not really a follow-up, per se... but...

The Register has an article about the launch of the StarCat including a quote where McNealy said, with tears in his eyes, "God I hate my job."

Apparently, McNealy had a hard time speaking during the event, which was held in New York City, due to the death of a long-time Sun employee in the terrorist attacks on the WTC.

Re:SGI Origin 3000

[fgodfrey]

What are *you* smoking? The Origin 3800 is certainly *not* using the "same back plane a the sun was 2 gens back". In fact, if you were anything other than a hopeless troll, you'd realize that the Origin 3800 doesn't *use* a backplane at all. You get, on a 512p system, 128 I/O channels, each of them supporting up to 12 PCI slots or 4 XIO slots. I can't remember off the top of my head what the bandwidth per channel is but it's on the order of a gigabyte/second (I wanna say it's 1.6 GB/sec, but I might be wrong).

Re:partitions

[theyman]

Not only can they be partitioned by system board (so long as a system board has a CPU, memory and access to disk it can be made a domain) but, assuming it follows e10k functionality, you can 'blacklist' any component if it starts mis-behaving and it won't be used again until you say it's OK. Unlike 'redlisting' (ask your friendly sun bod about that...)

Where 106 probably comes from

[MadDog Bob-2]

I mean to say what's the difference between 106 (what an odd number) ...

Given the way other Sun boxen like the E3500 work, I expect that's the 15K has 18 boards, each of which takes 3 modules, either 2xCPU of 8GB RAM.

That means that 72 CPU / 288 GB memory is 18 boards, each with 2 2xCPU modules and one 18 GB memory module, and the box is full.

Since you always need some memory, the most CPUs you can get is 17 boards w/ 6 each and one with 4. Of course, that leaves you with 8GB of memory for your 106 CPUs.

The other end is (17 x 3 + 1 x 2) 8 GB memory modules for 424 GB on a pair of CPUs

But that's just a guess...

SGI can do 1024 processors :-P

[halfelven]

You think 100 processors are a lot? Take a look at SGI3000 which can come with 1024 processors at any time. Now that's a lot! ;-)

Slow but reliable

[linuxbert]

most ATC systems run on really slow equipment.

most AFTN Mesassages (fliight plan, met data etc) fly between airports as 2400baud. some of the faster ones use 28,8 modems, or X25

the computers are often pentium class machines (running slack 4.something) or stratus servers (hpux ones)

the point is speed is not nessasary, ensureing the message gets there reliably before the aircraft does is nessasary.

Re:Still small CPUs

[Anonymous Coward]

Yes, IBM mainframe CMOS CPUs are roughly twice as powerful as UltraSPARC III CPUs given similar clock rates.

But the reality is almost every commercial application has been ported to multithreaded C and C++ code.

Mainframes are good at processing single-threaded COBOL code, but not running Oracle or DB2. For that, get a good RISC/UNIX machine.

Hell, even Java now scales to 72 CPUs.