Slashdot Mirror
Linux Clusters for sale
Fred M writes "For
use in high-computing area's Siemens has build their
hpcLine systems and showed it on a meeting for customers
in februari. Base of the systems are cpu modules, each
consisting of 2 Dual-CPU-Boards. Each module has 2
Pentium-CPUs - then PIIs 450 MHz - and memory of 2048 MB
max. 8 modules (= 32 cpu's) can be mounted in one rack.
Connection of the modules is done with Scalable Coherent I
nterface (SCI), that has a bandwidth of 500 MByte/s and
uses a Ring-Topology. A story (in German) can be found
here "
Those specs make me creamy. Any word on how much something like
that would cost? I'm pretty sure I'll never have the coin to
pick up a box like that until it's lapsed well into obsolesence.
read an article in wired this morning about and IBM linux cluster that benchmarked the same as a cray supercomputer
actually at idg.netg i?allowFeedback=false&referer=&outside_sou rce=newsletter&url=http%3a%2f%2fwww%2einfoworld%2e com%2fcgi-bin%2fdisplayStory%2epl%3f9903 9%2eecsuperlinux%2ehtm&doc_id=103610
http://www.idg.net/idg_frames/english/content.c
Clusters of storage would have a much greater market than clusters of CPU's.
For about $15,000 you could put together a terabyte of low-cost IDE drives. Another $10,000 for the CPU's/cases and you would have a TB of online storage for less than the cost of the 1 TB tape library you would need to back everything up.
Unfortunately the glue for the whole system is missing from the Linux world as the Beowulf Bulk Data Server appears to be dead....
http://beowulf.gsfc.nasa.gov/bds/disks.html
Why does CESDIS have a web page for a dead project? Seems like a waste of tax payer's money....
Siemens SCI box comes with a binary only SCI
driver. So you can't even upgrade it to a new
kernel without risking problems, or expect anyone
to support the resulting mess.
(Note its a driver thats using published API's only it seems so its probably not a license issue)
Just they are selling you closed junk
http://www.siemens.de/sap/loesungen/linux.html
The "start paket" (8 nodes, 16xPII/450, 512MB, 8x4.3GB HD) costs 130000 DM (approx. 80000$, I think)
Hendrik
The SCI adapter cards don't cost more than other high speed adapter cards.. html for Linux Cluster Based on SCI
See http://www.dolphinics.no for more information. The University of Paderborn has a 192 node cluster interconnected with SCI.
See http://www.uni-paderborn.de/pc2/s ystems/index.htm and
http://www.uni -paderborn.de/fachbereich/AG/heiss/arminius/index
This system sounds like an IBM mainframe. Start off with one 4 processor board and memory and add up to 15 more when the need arises. Newer mainframes will use the new integrated processor and memory (512 MB initially) on a single chip for improved performance. Just goes to show that you should still look to mainframe technology to see where high performance computing is headed.
One question is how many IO channels does this system have? You've got to send all that data somewhere. It's typically not the lack of processing power that affects mainframe performance, but how the IO sub-system is tuned to take advantage of the IO channels.
SCALI have had Linux/SCI based clusters for some time. Check this url.
Looks like you should HAVE paid attention in
the English class as well.
You wanna hack the driver for the Dolphin card? BWAHAHAHA! Good f***ing luck! I was on the team that wrote the drivers for this card at Dolphin, and it was a _bitch_ even with access to the hardware engineers and all sorts of analyzers and so on. It turns out that none of the PCI chipsets we worked with were really very comfortable with a memory bridge (that's what the card is) because the chipset designers had never seen one. I'm not sure whether Siemens is using the Dolphin driver or a different one produced somewhat illicitly by a former Dolphin employee as partial fulfilment of doctoral thesis requirements at the University of Oslo, though. Either way I don't think you can expect to be seeing the source code any time soon.
;-)
It'd be nice if the source were released, in a purely philosophical sense, but in a practical sense it wouldn't make a damn bit of difference.
jdarcy@mediaone.net (once @dolphinics.com)
P.S. It's still nice to see something I worked on get out into the real world and be considered "cool".
Dolphin's SCI technology actually supports many different technologies. One of the most common deployments I've seen is actually a bunch of "ringlets" connected to a switch.
the price quoted in the article was DM130,000/node
which brings it to a cool 75-80,000 USD. better
hum for that. up to 32 nodes? whew, big bucks for
the home cruncher; not too bad for cray-like
performance, prob give a discount for full, balls
to the wall system
cschroeb@umaryland.edu
SCI is Scalable Coherent Interface.
It is an IEEE standard to support shared memory
multiprocessing (typically in a NUMA environment)
when will one of these SCI boxes support a single
system image across all nodes?
With a binary only SCI driver, the answer is NEVER! Pfffht.
I presume that you are referring to the demonstration that IBM gave on the LinuxWorld "Trade Show" floor. I believe it was 48 CPUs in 17 boxes - IBM Netfinity's. They ran a ray-tracing (POVRay) package across the nodes.
The current benchmark record was set on a Cray in 1997; they "tied" the record. I say "tied" because the benchmark rounds to the nearest second, and their time equalled that recorded
for the Cray. Cute, but not a *real* problem, imho. The only relevant fact of the demo was the decrease in the cost of hardware necessary to perform the test. That is, anyone with access to 17 boxes/48 CPUs could have done the same thing, and probably even cheaper.
Data General have some SCI NUMA systems, I don't know if they support Linux or work as single system image or if each node has its own OS
http://www.dg.com/aviion/html/av_25000. html
NUMA does not preclude single system image. All NUMA means is that access _times_ are non-uniform, not the _semantics_.
That said, the Dolphin card in question actually uses a non-coherent subset of SCI. There was a project to provide SSI despite these limitations, involving a half-dozen engineers for two years, but the project was eventually scrapped because management didn't understand the need.
The Siemens cluster is not using 100Mbit Ethernet, it's using SCI (IEEE 1596-1992) at 500Mbyte/s.
With SCI you can use alse use cables (up to 5 meters I beleive) between the different nodes, so you can cluster Linux boxes.
Actually SCI has been used to connect VME craters, check out the RD24 project at CERN (European center for nuclear research).
I'm aware of the meaning of NUMA, actually one could claim that any system with a cache is NUMA since the access time is non-uniform :-)
I should have said ccNUMA, since it's probably simpler to design a SSI using a Cache Coherent system.
The Dolphin adapter cards use the Dolphin LC-2 (the physical layer interface) and the Dolphin PSB (PCI to SCI Bridge). DG are using the LC-2 and the SCC (SCI Cache Controller) which implements the SCI cache coherency protocol.
I don't remember much about the VME bus (is it still used?), but would that be any better than using a 100MBit ethernet backbone? It's an industrial 32/64bit wide bus as far as I recall, right?
Of course, the pricetag attached to such a system wouldn't compare very well to a normal cluster, but what about the speed increase?
Anybody knows more about VME and if linux supports VME architecture?
---
"Hasta la victoria siempre!" El Comandante
Siemens developed the hpcLine specifically for use in High Performance Computing. The machines were presented at a customer presentation late in February. The system is based on a modular architecture featuring two dual-CPU boards. Each board can carry two Pentium-CPUs - currently PIIs with 450 MHz - and a maximum of 2048 MB RAM. Eight of these modules can be put into a rack, which is then a system with 32 CPUs.
For communication between the nodes Siemens uses the Scalable Coherent Interface (SCI), which delivers a bandwidth of 500 MB/sec within a ring-topology. SCI cards are made by Dolphin. The communication software, the Message Passing Interface (MPI) has been developed by Scali.
For Fortran, High-Performance-Fortran, C and C++ Portland Groups Compilers are available from Pallas.
Systems of these type have already been deployed by the Paderborder Center for Parallel Computing and at RWTH Aachen.
Siemens offers Windows-NT, Solaris x86 and Linux as operating system for these machines. The entry package, an 8 node system with 16 PII, 450 MHz and 512 MB RAM and 4.3 GB disk space per node comes at 130 000 Deutschmark (~75-80 Kilo-$).
This continues the trend towards the professional use of Linux in Clusters.
At 22watts each the PIIs would consume 704w. HDs and memory would add another 1200w. Thats about ~2000w. Or 20 light bulbs.
Now we're nothing.
It's far easier to forgive your enemy after you get even with him.
We've seen that the g77 from egcs 1.0.3 generates code that's up to 50% slower than that generated by the Portland Group compilers's pgf77 (v3.0) on Linux/x86. My guess in that the difference between g77 on Linux/Alpha and the DEC Fortran compiler is at least as great and probably larger. The problem is, you can't legally use the DEC Fortran compiler under Linux/Alpha, and I don't think the Portland Group or Absoft have plans to support Linux/Alpha either. That leaves us with only g77, which generates fairly lousy code.
(Responses to the effect of "Use C!" will be gleefully ignored. Believe it or not, most scientific programming is still done in Fortran.)
--Troy
"My life's work has been to prompt others... and be forgotten." --Cyrano de Bergerac
With IBM embracing Linux, I wonder if they will consider setting up an RS/6000 SP cluster running Linux.
That wouldn't be too bad. The Power2 and Power3 processors are pretty fast (although their L2 cache sizes are kinda small compared with the Alpha 21264 and MIPS R10k/R12k), and the SP switch's performance is respectable. The memory bandwidth on the SMP SP nodes is kinda sucky, though; the memory's on a shared bus, like on an SMP Intel box. It's be nice if they'd put in a memory crossbar switch like Sun uses. That'd drive the price way up, though. Compilers would be a big problem, too (just as they are now on Alpha); gcc and g77 don't cut it for high performance code.
We have an older SP-2 where I work, and we're trying get rid of it in favor of a comparably sized Beowulf. Almost everybody hates AIX and LoadLeveler (IBM's eeeevil batch system) with a passion, and the maintenance costs are murder.
"My life's work has been to prompt others... and be forgotten." --Cyrano de Bergerac
With IBM embracing Linux, I wonder if they will consider setting up an RS/6000 SP cluster running Linux.
"In true sound..." -Agents of Good Root
In that case one would have to look into Portland Group's high performace development tools.
"In true sound..." -Agents of Good Root
http://babelfish.altavista.digital.com/cgi-bin/tra nslate?lp=de_en&urltext=http%3a%2f%2fwww %2eheise%2ede%2fnewsticker%2fdata%2frh-09.03.99-00 0>Bablefish
:-)
Munges it a bit, but you can get the idea
Everybody, please check www.microway.com before speaking of Yummy, etc. There is no such thing as a Yummy x86 based system.
On the subject, Microway have been offering Beowulf Alpha clusters for quite a while. Actually they were even supposed to present one at the Linux expo, but it looks like nobody have noticed it. This is understandable because as you can see from the pictures their stuff does not look that shiny. But it works...
And I bet that it can blast any x86 based box out of the sky...
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Yep, cost is the issue -- WAS: Yummy!
If they start charging >$5K for these things, the price just moves closer to Sun/SGI solutions.
I hope the Corel rummor on a 2+8 cluster that fits in a 9U (or similar) rack form-factor is true.
Why hasn't anyone come up with a low-profile form-factor cluster of of Socket370 or Slot1 CPU mainboards yet? The market is there!
-- Bryan "TheBS" Smith
Independent Author, Consultant and Trainer
I should of paid attention in german class.. But it sound interesting.. What happened to the first posts?!?!
I ate my tag line.
-=Ellis (D)25=-
I have ported the Scali SCI driver to Linux as part of my work at the University of Paderborn, our research group would like to see an open source driver, but it contains too much stuff from Scali and they do not allow to release the source.
Currently the PC^2 (Paderborn Center for Parallel Computing, part of the university) operates two large SCI clusters, a small one with 64 PII 300 (32 dualprocessor nodes) and a large one with 192 Xeon 450 (96 dualprocessor nodes).
For more info have a look at:
http://www.uni-paderborn.de/cs/heiss/arminius/