Slashdot Mirror
Student and Professor Build Budget Supercomputer
Luke writes "This past winter Calvin College professor Joel Adams and then Calvin senior Tim Brom built Microwulf, a portable supercomputer with 26.25 gigaflops peak performance, that cost less than $2,500 to construct, becoming the most cost-efficient supercomputer anywhere that Adams knows of. "It's small enough to check on an airplane or fit next to a desk," said Brom. Instead of a bunch of researchers having to share a single Beowulf cluster supercomputer, now each researcher can have their own."
A beowulf cluster full of these!
(Okay, now back to responsible mature posting)
Hopefully I didn't put any [] around my words.
It's just four motherboards sitting in a single frame. connected by an ethernet switch.
True supercomputing machines (sun, ibm) have a little bit better interconnectivity between the components than a mere 1Gb/s line. This can serve its purpose though, VASP will run wonderfully on it. GAMESS probably as well.
B.
Every experiment which ends in a big bang is a good experiment.
It's small enough to check on an airplane
With security concerns nowadays, it's the amount of cables coming out of it that worries an airline, not the size or weight of this machine.
Virtual Betting on Facebook for non-geeks.
It looks rather fragile, quite like the iRack (http://www.youtube.com/watch?v=xcjLEwZqcQI), and I dont think it would survive checking in on an airplane given how some suitcases looks like at baggage claim.
Cool achievement nevertheless.
Probable impossibilities are to be preferred to improbable possibilities.
Aristotele
They just linked 4 motherboards together. My cat could do that.
And it looks like they'll be running into heat buildup issues. An enclosure ventilated by one or two desktop fans would have provided sufficient cooling. Mere convection (outside of the tiny on-board fans) is often not enough. The Sun E450's were well ventilated machines, with a clear air path going from the front to the back. The temperature monitors (ambient, cpu (x4), PSU (x3)) were useful as well. One was used for a long time at Stack (www.stack.nl) as a room temperature monitor.
B.
Every experiment which ends in a big bang is a good experiment.
Now Microsoft have their next development target for Office.
Deleted
is it powerful enough to run Windows Vista?
I am impressed with how amazingly lame this story is. It should have been entitled, "College Senior and Professor discover Ethernet, MicroATX, and PXE boot. Funding dried up before paying for cases. News at 3 am because we can't find anything else to report."
Honestly, our whole research lab is filled with PXE booting MicroATX computers connected via ethernet. And I guarantee that four "nodes", aka Linux PCs, are cheaper than $2500. Whoop-de-freaking-do.
This seems pretty similar to the way google builds their racks, with just mb's and no cabinets. What would have been really cool was if someone made som e kind of network driver for a pci express slot, with them being able to use external cables, is it possible to use a dedicated pci express slot as a interface to another computer, skipping the network bottleneck ?
Doolittle :
Bomb no.20 : To explode of course.
...this is *hardly* a supercomputer. This is 152.57 times slower than entry number 500 on the Top 500 List. There isn't a nice neat definition of what a supercomputer is anymore, but "capable of running Beowulf" isn't it. Leaving aside the more custom machines that the company I work for (and a few others) build, there are plenty of Linux clusters that *do* qualify. The fastest one seems to be number 8 on the current Top 500 list (a Dell Infiniband cluster at NCSA).
Go Badgers! -- #include "std/disclaimer.h"
One of the problems with supercomputers is that there aren't really very many of them, because of the size and cost. It means that the tools you use to run your supercomputing applications are similarly unusual. The skills to use and develop on parallel systems are then equally scarce. Access to a supercomputer isn't exactly common.
Microwulf could make all of the above common. For the price of a high spec PC. The commodity nature of it could bring super computing and super computing applications to the masses.
Then you can scale your application from microwulf to miniwulf to superwulf with little more effort than installing it on the bigger machine.
Course, they'd have to produce a commodity pre-built system.
Deleted
I see a few people making the expected "It's just four motherboards wired together with Gig E"-comments. While I won't object to that, I'd say this is not about a groundbreaking evolution in hardware, more a case of demonstrating what's possible today with COTS parts. Adding to that the compact packaging, and the ability to run off of a single power cord, it's a nice setup IMHO.
While it does not have the interconnect of "true HPC" hardware (a bit of a fleeting distinction, but bear with me) it'll surely be suitable for a lot of the simpler, yet still compute-intensive tasks out there ("simple" here meaning not needing a lot of intra-node communication).
On the flip side, it might fuel the "hell, I'll just build my own cluster"-mentality going around these days. I work in the HPC group at a university, running linux clusters, IBM "big iron" and a couple of small, old SGI installation, and we certainly see a bit of that going around. Problem is, sure, the hardware is cheap and affordable, but getting it to run in a stable and sensible manner without spending large amounts of time just keeping the thing together is a challenge, mainly due to the immature state of clustering software. As many researchers are not exactly keen on spending time solving problems outside their specific field, they're usually better off letting somebody else administer things, so they can just log on and run their stuff.
But for individuals and small groups of people who are computer savvy enough to handle it, things like these are definately a "good thing" (TM).
The more computing power is available in the world, the less it will be used to its potential. If everyone had an Earth Simulator in their basement, how much of that power would be wasted?
Not saying that proliferation of computers is bad, just food for thought.
-:sigma.SB
P.S. SETI@home, Folding@home, etc. are cheating. :P
WARN
THERE IS ANOTHER SYSTEM
Look at this design: http://www.mini-itx.com/projects/cluster/. It uses DC-DC converters on each motherboards (mini-itx, so low power), a single 12V PSU and a UPS for regulation:
bundaegi is good for you
Is 26 GigaFlops significant anymore? I hear that the PS3 can do 20-25 from Folding@Home people. And it is only about a 5th the price. But I hear so many different numbers that I can no longer make sense of them. Why do they bother comparing with DeepBlue, an over 10 yr old super computer? Can anyone with a PS3 can report what their PS3 with Yellow Dog Linux is doing? And what are the numbers for the latest desktop processors? Any recommendations on software to benchmark in flops for my own computers?
This is kind of like the old joke about a dog chasing a car...what's it gonna do with the thing if it catches it.
I've thought several times about building a small cluster, just for the experience and the nerd factor. But I never do because I also get in to the issue of just what am I going to do with it once its finished, other than heat my workshop.
I want a new quote. One that won't spill. One that don't cost too much. Or come in a pill.
Minesweeper under Vista - No
-- Intelligence is soluble in alcohol
CPU: AMD Athlon 64 X2 3800+ AM2 CPU $67.50 * 4 = $270
Main Memory: Kingston DDR2-667 1GByte RAM $48.49 * 8 + $4.99sh = $392.91
Power Supply: (can't beat price): $76.00
Network adapter (node to switch): (cant beat their price) $164.00
Network adapter (switch to node): (cant beat their price) $15
Switch: Trendware TEG-S80TXE 8-port Gigabit Ethernet Switch $46.99+$7.04sh = $54.03
Hard drive: Seagate 7200 250GB SATA hard drive $69.99
DVD/CD drive: (can't beat their price): $19
Cooling: (can't beat their price): $32
Fan protective grills: (can't beat their price): $10
KVM: (can't beat their price): $50 Grand total (incl. 15 in hardware): 1416.89 $1000 saved by using Newegg!
I'm to lazy to run the numbers tonight to compare actual speeds but our dual CPU four-core Xeon (8 cores total) servers cost around $2500 each to build. Looking at their specs I doubt they could be doing much better and they require special clusterish programming.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
gigaflops, schmigaglops.
/.
this is
i thought performance was measured in fps?
Revolutionary? Everything old is new again...
http://www.mini-itx.com/projects/cluster/
http://news.taborcommunications.com/msgget.jsp?mi
http://www-03.ibm.com/systems/bladecenter/ -- a 7U chassis that holds 14 blades, and is a bit spendy, but not completely unreasonable for some situations
http://www.linuxjournal.com/article/8177 -- My personal favorite, this page talks about several small portable miniclusters that have been made over the last six or seven years...
Yes, 8 cores of Athlon64 is faster than 8 cores of low power VIA CPU's from several years ago, but the concept isn't revolutionary, and there isn't a lot of headline worthy engineering that goes into a project like this... I'm sure it's a very handy tool, and I'm not suggested it shouldn't have been built, or that it was entirely trivial to build, but in the end, it's just four ordinary motherboards and ethernet.
Sorry for replying to myself, but I found an interesting paper about the subject. Seems that a PS3 should have Rpeak of 14 Gflop/s with double precision floating point operations. Sounds to me that with a proper clustering solution a four-node PS3 cluster would be significantly faster than Microwulf. And it would probably be a smaller, too :)
Discovering that you can build an even more cost effective supercomputer than these guys: priceless
which is totally what she said
A striking resemblance for a box of bits. I wonder if it's got the same surly attitude.
Am I missing something here? The Sisoft Sandra MFLOPS measurement for a top end Intel Core 2 is 47 GFlops http://www.tomshardware.co.uk/overclocking-intel,r eview-2395-28.html/. OK admitedly this is a sythetic measurement, but it's a ballpark figure right?
Yes. A single processor will perform several steps in one cycle. Typically, the steps are something like:
1. fetch (an instruction from memory)
2. decode the instruction
3. execute the instruction
4. access (some memory location)
5. writeback (some values calculated during this cycle)
In reality, this cycle is usually more complex and processors are designed to predict certain events in order to pack more into a single processor cycle. On top of this, note that the processors used in this machine are all dual-core processors. This means that instead of the 4 processors listed on the hardware manifest, it's really more like 8 processors (well, not quite).
No.
Hey, those computer engineering classes I was forced to take as a part of my CS major have actually proven useful! Oh wait, this is Slashdot.
The University of Kentucky (where he is coincidently going to grad school) beat his price point years ago on a "real" supercomputer. This super computer was built for about $84 per GFLOP in 2003 and it made the Top500 list when it was built. The Aggregate team at UK is one of the tops in the field when it comes to supercomputers on the cheap.
Seems like 2GB per (dual core) node is a little on the low side for practical usage. Not surprisingly though, RAM is the biggest cost of the system (992$ total) and switching to 2GB or 4GB modules will raise the system price considerably. Would still be cheap though.
I expect the design is very well suited to clustering. The PPUs handle all the data dispatching & balancing with the SPUs left to do the leg work.
Good paper - it also says that by using mixed precision (iterated 32-bit math for rough matrix factorization then fine-tuning the precision in 64-bit) the double-precision matrix performance is up to 155 Gflops.
"Is life so dear, or peace so sweet, as to be purchased at the price of chains and slavery?" - Patrick Henry
RTFA They bound the onboard NIC to one core of each CPU and bound the add-on NIC to the other core. That way, each core had its own dedicated communications channel.
The cluster depends on gigE for the interconnect, which means data transfers are going to be slow, and have a high latency. He'd be better off spending a little more and using Infiniband equipment.
Clear, Dark Skies
It might just about be powerful enough to run Vista though....
A processor normally takes 2-3 clock pulses to perform any instruction, as it cannot perform the operation in the same clock cycle that it receives the operation in. If the operation requires a call to a memory location it will take 3 cycles (one to get the info from the memory location) which is why pre-fetch is so important in modern processors.
A cycle is triggered by the rising edge of the clock pulse. Whatever the computer does must be completed before the start of the next cycle
The instruction execution cycle is triggered by the clock cycle, but has several stages
- Each stage is triggered by successive clock pulses
- The exact timing depends on the details of a particular machine
- A complete instruction cycle usually takes several clock cycles to execute
The instruction cycle is divided into several stages
- In some machines, some of these stages are performed simultaneously, which speeds things up
- The stages are common to most architectures
Sometimes called the Fetch-Decode-Execute Cycle.
Taken from this pdf.
A modern processor may in fact take a dozen or more clock cycles to finish a single instruction. However, by utilizing pipelining, reordering and multiple execution units, a single core may be working on upwards of 50 instructions at once. The resulting throughput can be several instructions per clock on each core.
In http://clustercompute.com/ you could find better design in term of compactness. Another thing is that the cluster does not need KVM (in process only at test mode) and as noted in several research papers dual 100M can beat gigabit (source http://en.wikipedia.org/wiki/Kentucky_Linux_Athlon _Testbed )
Beowulf is a good idea for a very limited number of number crunching applications, or as a student learning tool for comp sci or related studies. Yeah, we built one of those a couple years ago, the professors ended picking up intel quad core machines that were faster (no effing network latency). Beowulf is gathering dust.
Oh, and try writing your own lam-mpi code sometime...
Aren't these things like chick magnets?
Although not as cheap as the Microwulf, Nvidia has a desktop super-computer for sale http://www.nvidia.com/object/tesla_deskside.html at 500 GigaGlops, to start.
The basic definition of a supercomputer is a system which has top performance compared to other computer systems (within the top 500 or 100).
In the past, this could only be achieved by having custom CPU's to perform pipelining or parallel processing. Processors in the Cray supercomputers had extremely deep vector pipelines, which was good for three-dimensional simulations like CFD or computer animation. But other systems followed the parallel processing method. The Connection machine had 2^16 one bit processors which was good for encryption/decryption. Other systems used standard CPU's (Intel 80x86's, DEC Alpha's and M680x0's) connected together through a high-speed bus network.
The different types of systems could be defined according to how these processed instructions/data.
SISD - Single Instruction, Single Data - Early home computer
SIMD - Single Instruction, Multiple Data - Vector processors
MISD - Multiple Instruction, Single Data - Fault tolerant systems
MIMD - Multiple Instruction, Multiple Data - Parallel processing CPU's
Some systems had hardwared interconnect configurations - either a 2D square grid, a 3D square grid or torus network, or even star networks, while others had dynamic routing capability. Transputers only knew about the adjacent processors in the four compass directions (NESW).
But all of these techniques have been incorporated into mainstream CPU's now - you now have dual-core and quad-core CPU's that can be used by laptops.
Modern day methods are to make the systems super-scalar. Multi-core CPU's can be arranged side by side onto multi-CPU boards which in turn can be rack mounted into chassis which communicate through high-speed interconnect systems. There is no limit on the number of racks that can be used except space and money.
Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
List of #500 on the TOP500 by year
Year .
06/1998 - 15.0 GPLOPS | Southwestern Bell, USA. . . | HPC 6000, Sun
11/1998 - 20.5 GFLOPS | Koeln Universitaet, Germany | HPC 10000 Sun
06/1999 - 34.2 GFLOPS | CIEMAT, Spain . . . . . . . | T3E900 Cray
11/1999 - 38.4 GFLOPS | Bank, United States . . . . | HPC 10000 400 MHz, Sun
06/2000 - 51.2 GFLOPS | EDS, United States. . . . . | HPC 10000 400 MHz, Sun
11/2000 - 78.0 GFLOPS | Zurich American, USA. . . . | SP Power3 375MHz, IBM
Really, calling this a supercomputer is lame. It has only one 250GB disk; it will have utter crap IO performance. Most compute heavy jobs are also disk heavy because you want to checkpoint your intermediate results in case of a crash. Since there is only one disk, one machine must be serving it up to the others (NFS, ISCSI, whatever). It is clustered through gigabit ethernet, which will act as a limit on performance. They even skimped on the connection to the outside world and got a 100MBit card. "Real" clusters use Infiniband or Myrinet, both of which are optimized for high throughput with low latency and low contention. Gigabit is not. Linpack is rather kind to clusters; more finely grained parallel tasks will pay more for the poor linkup.
Also, with only 4 processors one could also build a 4-way SMP machine which would then not have to deal with any sort of message passing at all. You instead get one shared memory interface. It may be slightly NUMA, but the extra latency cost of hypertransport is amazingly low. Instead, by putting only one dual core die per motherboard, you have to jump through hoops to move work from one die to another, and pay really bad latency costs. You could also do better with a 2 quad-core processors on the same mobo (although you'd have to go Intel for now...). It's easier to program, supports finer grained parallelism, and allows potential savings on other parts.
I can get 2 quad-core Xeons at 2.4 GHz each and a 2 socket motherboard for $820 at newegg. They spent $980 on 4 dual-core 2GHz processors and 4 single socket motherboards. They also spent $240 on gigabit cards and the switch. So, for $400 less, I can have an SMP machine; one which probably has higher floating point performance as well. Rather than 4 cheap power supplies I can get one nice one (which is probably more efficient too). Further, I don't have to run 3 of my nodes diskless. Really, at this small scale a cluster is not the way to go.
256 MB XDR @3.2 GHz for system memory and 256 MB GDDR3 @700 MHz for video memory. Not to bring on a console war extravaganza, but the M$ xbox 360 has the same amount of total memory, but instead theirs is 512 MB UMA (Shared with CPU)...
Anyway, memory could very well be a limitation with a cluster of ps3s vs. the microwulf cluster, but it all would depend on the operations your performing. I'd venture to guess the 256mb of system memory would be enough for anyone. :)
That wasn't true even in 1991, except maybe for Intel processors which are notoriously wasteful of clock cycles (which is why they have always advertised clock speed rather than instructions per second). A 1 MHz MOS 6502 was just as fast as a 4.7 MHz Intel 8080 (and needed fewer support chips).
If you throw more transistors at the problem, and/or different architectures, you can complete instructions in a single cycle. (Especially e.g. register-to-register instructions where the answer comes out of the inputs at the speed of propagation delay through the gates.) If you do it right, you can even design clockless CPUs where the completion of the previous instruction triggers the start of the next one without waiting on an external clock.
This assumes your CPU is not microprogrammed; the instruction words contain the relevant bitmasks for source and destination registers as well as the control code for the ALU. See for example the PDP-11 instruction set (IIRC).
Of course as others have pointed out, modern processors pipeline multiple instructions at different stages of execution at the same time, for a net throughput of multiple instructions per cycle.
-- Alastair
I thought the hip thing was GPU based supercomputing. NVidia even has a dedicated GPU based, desktop sized, scalable supercomputer line called Tesla.
The basic Tesla unit c870 = 518 Giga flops for ~$1300.
Tesla s870 = 2 Terra flop for ~$12000 (still desktop size)
NVidia Tesla
I code for systems with 800 4 Optron nodes, with 10GB/s interconnect, and a couple hundred terabytes of SAN attached storage. That is a supercomputer :) Well sort of, a lot of people in the HPC community consider it just a cluster, as some programs need 64+ CPU's in SMP mode, so any loosely coupled memory model would be considered a serial farm :)
Also, note that high end platforms, would have redundant power, redundant high end interconnects, redundant hot swap drives etc. There also would be enough of them to need, high end switches, blowers, power conditioners, air circularators, and various other room coolers. Of course a custom built workstation without a graphics card, monitor, or even case is going to beat the pants off of HPC architecture price per flop, good work to the group, but hardly newsworthy in the HPC community.
But you aren't really a supercomputer at that point, you're a cluster. These days the line is more blurred than in the past but more or less the difference is interconnect speed. In a real supercomputer, there are very high speed interconnects, so you can run things that heavily rely on one part communicating with another, like particle simulations. That's why the US Department of Energy buys so many, rather than clusters. They do things like weather simulation and simulation of nuclear weapons, where every node as to be able to talk to every other node with essentially no penalty.
Now if you have a job that doesn't use a lot of inter-node communication, like say 3D rendering, then a cluster is a better answer. Normal hardware with Ethernet interconnects. Works great and is cheap since you can use commodity parts. But don't confuse that cluster with a real super computer, you throw one of those intense inter node problem at it, it'll fall over because the interconnects are too slow.
Unfortunately these days people really blur the distinction. You'll see systems on the top 500 list that are really questionable. It'll be commodity hardware connected with something like infiniband. Ok, great, that is faster (both more bandwidth and less latency) than Ethernet, but it still isn't necessairily up to what you'd get from a real supercomputer.
However in the case of this deal, no, not a super computer. It's a small cluster and they are just calling it a super computer as marketing, effectively.
The PS3 comes out of the box with a Cell uP that gets something like 20 GFLOPS on each $500 PS3. It's already networked into clustered supercomputing like this MicroWulf.
A $500 PS3 has 20 of the 26.5 GFLOPS the $2800 MicroWolf has. MicroWulf runs Ubuntu, which can also run on PS3. If people can port Linux libraries like Mesa/OpenGL/X to the PS3 SPEs, where most of the power lies, then we'd be looking at $25:GFLOPS, not the $94:GFLOPS on the MicroWulf.
And while taking a break, you can play Gran Turismo 5, and 40 more games you can afford with the money you save on HW.
--
make install -not war
Sounds like somebody is bitter about getting an English degree instead of an Engineering degree...
Don't worry, they're Supermicro, they wouldn't be stable if you cooled them in a swimming pool of liquid nitrogen.
/.er has in their closet - just imagine a Beowulf cluster of those!
Still, meh indeed, scrape together the piles of computers your average
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"