Slashdot Mirror

That's 64 threads, not cores. T5120s have only 8 physical cores, but allow for up to 8 cores per thread, hence why you see 64.

There is a closer counter-example to the article's claims, though: SiCortex's 72+ core machines:
    http://sicortex.com/products

Drool-worthy...

SMP will only bring you so far - i'll bet 8 VCPU VMs on Atoms will be beat by a 2 VCPU VM on a Core 2 Duo.

Perhaps not, depending on the other load the system is working on. Because of the way VCPUs are scheduled (at least in VMWare) that 8-vCPU VM won't get a time-slice until such time as there are 8 real cores available for the duration of that slice. If your task is CPU intensive and can be easily separated into distinct tasks not overly chatty (i.e. cross VM latency is not going to be a major issue) and the host has gobs of RAM available, you are often better off having several VMs with one cVPU each than one VM with several vCPUs. This may be much less of a problem on a many-CPU monster like the 512 core unit being discussed than it is on 2/4/8-core boxes, but I expect the balance to still be in favour of multiple single-vCPU VMs in cases where the task can be efficiently split between them.

It appears many of you have a horrible misunderstanding of how the VMware hypervisor works. One instance of VMware ESX can be installed on a single system image machine. SSI is defined by shared memory. There are two classes of systems with shared memory: SMP and NUMA. This 512 Atom machine is neither. It is a system of 512 individual machines cobbled together with very neat thermally and electrically efficient packaging. If one were to run VMware ESX on this machine, it would require 512 instances of the VMware ESX kernel. You could then assign guests to each single Atom machine, but you would not be able to schedule guests across/among the 512 machines. VMware, or any virtualization would be useless on this platform. That's not what it's designed for. To get efficient compute bound applications to run effectively on this machine would require something like MOSIX. For other applications such as web or webmail, you'd use a standard load balancer and a a cluster of 512 individual servers, just as is done now.

Again, this machine is not SMP and not NUMA. It is not a "server" in the traditional sense. In reality it is a _cluster_ in a box. The concept is nothing new, this company is rather later to the game, and is already beat in the efficiency department. One example beating it is the Sicortex SC5832 which houses 5832 64 bit MIPS cores running at 700MHz. The CPU die contains 6 MIPS64 cores, dual memory controllers, a PCIe interface, GigE interface, and a connection to the Kautz digraph network running at 2.5GB/s bidirectional to from each node. The system consumes less than 20KW for almost 6000 cores and over 8Tflops of floating point performance. This highly integrated cluster machine is designed for supercomputing, not web applications, though it could be used for such if desired.

http://www.sicortex.com/
http://www.sicortex.com/registration/download/265/1886/file/TechSummary_FINAL.pdf

SMP will only bring you so far - i'll bet 8 VCPU VMs on Atoms will be beat by a 2 VCPU VM on a Core 2 Duo.

Perhaps not, depending on the other load the system is working on. Because of the way VCPUs are scheduled (at least in VMWare) that 8-vCPU VM won't get a time-slice until such time as there are 8 real cores available for the duration of that slice. If your task is CPU intensive and can be easily separated into distinct tasks not overly chatty (i.e. cross VM latency is not going to be a major issue) and the host has gobs of RAM available, you are often better off having several VMs with one cVPU each than one VM with several vCPUs. This may be much less of a problem on a many-CPU monster like the 512 core unit being discussed than it is on 2/4/8-core boxes, but I expect the balance to still be in favour of multiple single-vCPU VMs in cases where the task can be efficiently split between them.

It appears many of you have a horrible misunderstanding of how the VMware hypervisor works. One instance of VMware ESX can be installed on a single system image machine. SSI is defined by shared memory. There are two classes of systems with shared memory: SMP and NUMA. This 512 Atom machine is neither. It is a system of 512 individual machines cobbled together with very neat thermally and electrically efficient packaging. If one were to run VMware ESX on this machine, it would require 512 instances of the VMware ESX kernel. You could then assign guests to each single Atom machine, but you would not be able to schedule guests across/among the 512 machines. VMware, or any virtualization would be useless on this platform. That's not what it's designed for. To get efficient compute bound applications to run effectively on this machine would require something like MOSIX. For other applications such as web or webmail, you'd use a standard load balancer and a a cluster of 512 individual servers, just as is done now.

Again, this machine is not SMP and not NUMA. It is not a "server" in the traditional sense. In reality it is a _cluster_ in a box. The concept is nothing new, this company is rather later to the game, and is already beat in the efficiency department. One example beating it is the Sicortex SC5832 which houses 5832 64 bit MIPS cores running at 700MHz. The CPU die contains 6 MIPS64 cores, dual memory controllers, a PCIe interface, GigE interface, and a connection to the Kautz digraph network running at 2.5GB/s bidirectional to from each node. The system consumes less than 20KW for almost 6000 cores and over 8Tflops of floating point performance. This highly integrated cluster machine is designed for supercomputing, not web applications, though it could be used for such if desired.

http://www.sicortex.com/
http://www.sicortex.com/registration/download/265/1886/file/TechSummary_FINAL.pdf

$2000? Try $10k.

But it will require a barrage of tests to get certified as medical equipment.

Interestingly, we see the same thing in other fields - like supercomputers.

72 core system - Price $23,695.00

I wouldn't hazard a guess on what their 5832 core system costs - but a few PS3's would be far far cheaper. (though perhaps not an option anymore)

You can get a MIPS-based desktop system with 72 processors that consumes 300 Watts, from SyCortex. They call it their Deskside Development System for their bigger parallel computers, and they say it does have a fast backbone bus.

It does run Linux, but at $23,695.00 (48 GB RAM) it's not, I suspect, what you were asking for. I would also like some cheap barebones I could just go on populating with CPUs as I wanted.

The GP might like SGI's Molecule better though, it being Atom-based: 5000 chips, that's 10000 cores, in 3U size. But this one is only a concept computer.

Nah, these guys have the highest power per watt (excluding initial setup cost)

But they're taking the supercomputer angle rather than server farm angle. Unless you cache everything in RAM, your webserver won't have enough I/O. Not feasible for a company like Google, but potentially feasible for MMOs or even sites like /. (Where you have more processing than disk IO. Heck, with terabytes of memory, just cache everything in RAM until the discussion is locked.)

But honestly, I wouldn't want to deal with another architecture. There's too much good free software on x86.

Sounds like you're referring to SiCortex, who uses MIPS chips. They also have a desktop workstation with 72 processors and 48 or 96GB of RAM that only consumes 300W. http://www.sicortex.com/

When you mentioned this, I started Googling around but couldn't find any sources for them. Got any favorites?

http://sicortex.com/products/sc072_pds

12 processor chips, 72 (6 cores per chip) 700MHz MIPS64 cores 1.4GLOPS each peak, 48GB RAM, 4GB/s bidirectional peak (yes bytes not bits) Kautz topology RDMA cluster interconnect, less than 300 watts power consumption, the Cray draws up to 1600 watts. Runs MIPS Linux and standard MPI apps. Much much less expensive than the Cray, and this one comes fully loaded. Peak GFLOPS performance is much lower than the Cray "deskside", however the SC072 is a desktop unit, interconnect bandwidth is 4GB/s vs. the Cray's 2.5G/s Infiniband interconnect, meaning a higher percentage of peak parallel performance can be obtained.

Sicortex's really cool machine is this:
http://sicortex.com/products/sc5832

Too bad none of their machines are true shared memory designs. One can perform OpenMP and forking across the 6 cores of a node chip as if it were a 6-way SMP. This opens interesting opportunities for hybrid SMP/MPI code, and some applications have been proven to greatly benefit from such a scheme, one being Parallel Ocean Program, IIRC: http://climate.lanl.gov/Models/POP/

When you mentioned this, I started Googling around but couldn't find any sources for them. Got any favorites?

http://sicortex.com/products/sc072_pds

12 processor chips, 72 (6 cores per chip) 700MHz MIPS64 cores 1.4GLOPS each peak, 48GB RAM, 4GB/s bidirectional peak (yes bytes not bits) Kautz topology RDMA cluster interconnect, less than 300 watts power consumption, the Cray draws up to 1600 watts. Runs MIPS Linux and standard MPI apps. Much much less expensive than the Cray, and this one comes fully loaded. Peak GFLOPS performance is much lower than the Cray "deskside", however the SC072 is a desktop unit, interconnect bandwidth is 4GB/s vs. the Cray's 2.5G/s Infiniband interconnect, meaning a higher percentage of peak parallel performance can be obtained.

Sicortex's really cool machine is this:
http://sicortex.com/products/sc5832

Too bad none of their machines are true shared memory designs. One can perform OpenMP and forking across the 6 cores of a node chip as if it were a 6-way SMP. This opens interesting opportunities for hybrid SMP/MPI code, and some applications have been proven to greatly benefit from such a scheme, one being Parallel Ocean Program, IIRC: http://climate.lanl.gov/Models/POP/

Yawn. Lame. This isn't even close to the high density, high performance machines from my friends at SiCortex. If you're a serious player in HPC-land, the SiCortex machines have you salivating. I'm not trying to sound like an advertisement but they're simply awesome machines. Up to 5832 64-bit MIPS processors running Linux in a relatively small footprint. I don't have a direct connection to the company but I've worked closely with them in the past and it's the real deal. Check it out - SiCortex.com

Not to burst any ones funding bubble, but this has been done. Take a look at SiCortex. They did it, they are shipping product, and it works quite well. And it runs Linux.

I took the editors' title of this story too literally.

Now this is a new concept in supercomputers.

SiCortex was at Supercomputing this last year and they also have a desktop supercomputer.

In my opinion, it also looks a heck of a lot better than the cylone: http://sicortex.com/

SiCortex

They're more focused on computation than giant racks of storage, but their 2 systems are rated at max 3W/core total power consumption including drives, power supplies, interconnect, etc. I suspect the actual power draw will be much less.

How much storage does a "typical" datacenter have? (I know any answer would have huge variances.) For probably over two million USD or so, you should be able to get their larger system with 8TB of RAM and run their RAM-based Lustre filesystem along with the HD storage. Even with more places getting into petabyte ranges, I suspect 8TB of RAM and 5832 CPU cores would make it fly (16TB on 11,664 cores at $4-5M or so for redundancy). To me, that cost looks like a whole lot less than the facilities construction, cooling, and administration of the typical rows of racks... if it would meet the need.

(not affiliated with them; just a customer looking forward to their products)

SiCortex still seems to be the best bet to me. About $1.5 mil for the 5832-core 18KW system, $200k for the 648-core 2KW system, in base configurations according to internet hearsay. 1GFLOPS per core, and an interconnect that's incredible. They've apparently demoed the 648-core system at SC'07 now and are slated to ship "this summer".

Blue Gene is a specialized design that is based on using large amounts of low power CPUs. This approach is also the one taken by SiCortex. One of the big problems with heroic computers (computers that are pushing the envelop in terms of performance) is heat and power. Just stacking Intel and AMD servers gets expensive at the high end.