Slashdot Mirror
Where to Spend $1M on a Cluster?
Natchswing asks: "My university has been given a $757,825 NSF grant to build, 'A 256 node (128 pair) Beowulf parallel computing cluster ... to improve the realism of gravity-wave modeling by permitting treatment of the three dimensional problem and multiple wave interactions.' They want to pay a company to just show up and drop off a functional cluster rather than build it themselves. Since word has leaked out regarding the purchase intent, every computer manufacturer under the sun (including Apollo himself) has called up trying to sell their cluster. Since I'm no cluster expert, I'm writing Slashdot. If you had $0.7 mil to buy a pre-built cluster who would you go with and why?"
Have the companies submit bids... then compare them and make a decision.
This isn't rocket science.
Conformity is the jailer of freedom and enemy of growth. -JFK
Try an Xserve Raid cluster here.
"Fortunately, I'm adhering to a very strict drug regimen to keep my mind limber..."
I'll do it ... send via paypal to my /. username @ yahoo.com ...
It is more productive to voice thoughtful opinions (reply) than to judge (moderate) others.
I didn't think it would take long to get a few willing parties...
Penguin Computing does this kind of stuff for a living. I think they are an all open source shop, too... There may be others, too.
get 7 free Japanese lessons.
...you seriously need to put this out for RFP (Request for proposal).
Once you've done that, look through the proposals and pick which one sounds the best.
Definitely not Slashdot, partially because of responses like this one.
Does anyone else see the VERY obvious discrepancy between the submission title and the submission? Where are the editors? Last time I checked, 0.7 mil != 1 mil.
could you say, "Imagine a beowulf cluster of those!" and actually be asking a legitimate question!
Condemnant quod non intellegunt.
Intergalactic NatchswingCo Research.
Shit, my department needs to take lessons from you guys. We need to specify the budget down to the last rubber foot and network cable just to have them review the application, and you got a grant without even having an idea what you were going to spend it on?
"gravity-wave modeling"?
I thought that E-R was an aviation school.
Are they developing an anti-gravity levitation vehicle?
I think that you should look at your intended application.
- How much disk space are you going to need in total?
- How much disk space are you going to need per node?
- How much RAM is each node going to need?
- Is your application going to benefit from a low-latency or a high-bandwith connection between nodes?
- What about cpu? which cpu family will provide the best bang/$ for your calculations? PPC or X86? x86-64 maybe?
Once you know what you need, put it together in an RFP and send it out to every company that shows up under a google search for "beowulf cluster"
Review the responses and pick the best.
Since you are asking this question here, I'm going to refrain from suggesting the better option which is to build your own.
Hector
3. Pocket the leftover $499.5K
I run a 48 Node Microway beowulf and I must say that it is the most stable system available. Everything came assembled and ready to go (of course, I built the enclosure and did the networking, but they will do that for you if you'd like). If you're not very knowledgeable about beowulfs, how do you know you'll need so much power? Do you know how well the software you will be using will scale? Is it close to embarassingly parallel or does it lose efficiency over X nuber of nodes? What type of resources and consumption does the program use? Is it extremely processor hungry, or does it deal with dense matrices and require low-memory latency and high bandwidth or both? Do you know if you will need the power of Myranet or will you be able to get by on GigE?
These are important questions you must ask your researchers and yourself before you purchase this cluster. But, to answer your question, I believe Microway is the best choice and I plan on having them build our next cluster in the next fiscal year.
-brian
Whoever you chose to go with (I'm partial to Apple, but that's just me - and just because they have sexy hardware), see if you can get them to give you either more for your money, or free implementation/consulting help, or something like that in exchange for using your implementation as a success story. I think Virginia Tech got a bunch of free stuff from Apple when they decided to build their supercomputer.
All these vendors want to be able to talk about their work. Letting them use you for marketing may help you get more for your money.
I currently maintain some Opteron based Angstrom Microsystems Linux clusters. We've had them for less than a year, and already 30% of our nodes have had to be replaced. Support has been a nightmare.
Sadly, I was not around when the proposal was made, otherwise I would have rejected this cluster outright. There is no way to hook external storage up to this beast. There is no USB, Firewire, SCSI, external SATA, or fibre channel options. You can't even run an ATA cable out of the thing without drilling holes into the blade walls.
Personally? I'm looking at an XServe or an IBM Bladecenter.. but maybe it's just because I'd like some real support.
First of all, you really should put out an RFP for your cluster.
.75M, like maybe doubling your size and getting AMD64 nodes. Look at your primary problemset first, see if it's IO-intensive or CPU-intensive to figure out what you want in the way of disk/networking.
We've got a 128 node (1 cpu per node) cluster from Atipa http://www.atipa.com/ that cost CDN$ 0.25M.
128 P4 Xeon, 1GB RAM, 120Gb IDE, Gigabit Ethernet.
I'd expect you to get a lot more for your USD$
The only thing I don't like about it is Atipa's configuration of Redhat8 (they didn't offer anything newer at the time). Look for something newer there.
Atipa is one of the suppliers for SGI-branded clusters as well.
I'd really like a cluster from http://adelielinux.com/en/, but I wasn't aware of them at the time we did our RFP and cluster purchase.
ICQ# : 30269588
"I used to be an idealist, but I got mugged by reality."
Maybe I'm dumb but I always thought apollo was bought by HP not sun ;)
Peter.
...f'ked. Like, seriously arse backwards.
You got then grant *then* went shopping? Does all US academia work like this? Aren't you supposed to work out what you want to do, how to do it, how much and only then apply for the grant?
Dave
I write a blog now, you should be afraid.
I see you mentioned the problem set, which is good. to me and my only somewhat novice mind (I work with scientists all day, hear all kinds of stuff), this sounds suspiciously like a fine grained problem. that is to say, there will be a lot of interprocess communication, so don't skimp on the network. I'm not talking "get gigE". I'm talking "look at myrinet, or quadrix, or infiniband".
Most people can do you up a 256 node cluster for under half a million, but doing up one with high speed and low latency network is another story. that net costs bucks, around $1500 per machine (for a card, a cable, and a port on the switch).
Make sure you know your problem. if you understand how it works, then you can buy a cluster that meets the need much better. Make sure the nodes are not being starved for ram iif the problem is a ram hungry one (your researchers should be able to tell you, even from data off a single machine). Find out if it's heavily integer based or floating point based (my guess is that it's a floating point problem). Find out if it's a lot of vector and matrix manipulations.
every machine type is a little better at something than the others. for instance, on integer based problems, x86 will generally scorch everything else. on floating point, there is lots of good competition (apple, intel's itanium, opterons). Don't be afraid to say to the vendor "we want to run on it for a week before we buy".
as others have said, do a RFP, but get their specs. get your tech guys to look it over hard. ask them "what sucks".
All that having been said, i'm a fan of apples and of verari systems. Dell is also quite good.
-- Who is the bigger fool? The fool or the fool who follows him? --
You mean Myrinet?
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
But there's more here than figuring out who can plunk down the best system for the specified price. There's the maintenance/support costs. And picking a particular hardware platform kind of defines your choices for software -- so whose compiler do you like best? And any serious school needs to ask: can we maybe do a better job, more cheaply, cobbling together a cluster from cheap (abandoned, commodity, or donated) hardware? Which has the additional advantage of giving ones students some solid practical experience. Slashdot has run any number of stories on projects that did just that; one or two have achieved a small measure of success.
$1M = $1,000
$1mm = $1,000,000
I'd probably just buy 10 blade enclosures with 14
2-way Xeon blades each from ibm off the shelf.
They have blades with dual gigabit nics. A Pair of
3-Com 16-ways nics give you 2 parallel networks,
which makes it flexible. Run OpenMOSIX.
I'm pegging the whole shooting match at roughly
$420k. Spend the rest on NAS, pack out the RAM,
get a nice visualization wall, etc.
-I like my women like I like my tea: green-
Second: you're almost certainly going to have to put it out to bid. For example, at UIUC, the bid limit is $28,100. Anything over that *must* go to bid unless you can provide a really good reason why you have to "sole source" it.
Now, you need to start thinking about stuff. First off, forget the number of nodes. You need to start by thinking about how they'll be used. Like, how much communication will there be? A few large packets, or many small ones? Myrinet is nice and fast, but will increase your costs by 50% over gigE. Similarly, you need to figure out how much ram to put in each node. How many processors, keeping in mind they'll be competing for the memory bandwidth. 32 bit vs 64 bit. The list goes on and on....
Of course, you didn't give any details, which means you probably don't have a clue. So maybe start by purchasing a couple of test systems and benchmarking your code on them, to see where your bottlenecks are.
Good luck. Sounds like you'll need it.
...but I need it to run Doom3 in "Ultra". Sorry.
One of these days I'm moving to Theory - everything works there
If you haven't already, google for beowulf clusters at other universities and contact those departments.
Infiniband or gigabit ethernet are your main options. IB is low latency, and probably even more cost effective then gigE, but you may need the gigE anyway for a maintainance network (netbooting, NFS etc.). gigE usually comes with the motherboard, but you still need to budget for a fat tree of switches to connect it all. Myrinet's too pricy and (I think) slower then IB, but might be simpler to connect and has more mature MPI implementations for it.
Watch out for big vendor cluster software people - they may not actually know what they're doing.... not naming any names. What big vendor actually did (for the cluster next door to ours) was make it all slower!
IMHO you don't need that serial maintainance network crap they try to sell you, or even IPMI or similar. these Xeon/P4/Athlon64/Opteron clusters should be reliable enough that it's a waste of money. Our 264 node (528 Xeon) machine is fine without it.
If you want real bang for your buck then avoid the large expensive gigabit ethernet switches - they usually have limited backplane bandwith anyway. We use 2D mesh networking made from a stack of 24port gigE switches and had the fastest machine in Canada for a while... our networking is now way simpler than the hypercube-like topology on that page, but every node is still a router, and it works really well.
OSCAR is a great install system for a cluster. Do it yourself - it's the only way you'll ever be able to maintain the machine in the long term anyway... Just buy the hardware from anyone who gives you the best deal and looks like they'll be around for 5 years to replace nodes as they die.
Drop us a line if you want more dodgy advice
You'll be back, believe me. You'll be back in no time.
..Dell
...IBM
and *talk* to a sales rep. I know how hard this is (not!) but asking Slashdot is kinda silly. Sure you might want some impartial advice but
ciao
When my research group decided to build one, I was incharge, opted for OpenMosix and after a tweaking period worked really well. Now with the various bootable CDs with OpenMosix (PlumpOS, BCCD, Quantian, ClusterKNOPPIX...), tests and upgrades are done by just pressing reset !
Of course with clusters your mileage may vary.
Non-Linux Penguins ?
Building or buying a cluster is serious business. Talk to supercomputing experts. Issues involved are numerous. Just a short list:
This list is nowhere near complete... there are so many issues involved in buying a cluster, you really need expert advice.
Talk to other sites who do the same thing as you want to do, who run the same kind of applications as you ant to run etc.
I found mine on eBay... but make sure you check for bad RAM first (use Knoppix)
I would very much recommend this research site from one of my professors at the University of Kentucky. He has been doing work with cluster super computing for quite some time now and has managed to build some very impressive systems at low costs. Much lower costs than what your current grant is for. With a grant of that size using this professor's techniques you could build a whole bunch of clusters. I would suggest you taking a look at his group's research site aggregate.org.
You can also see one of the specific examples of a very low cost efficient cluster computer. KASY0
those people who think they know everything are a great annoyance to those of us who do. -isaac asimov
Chances are, your school has a hefty existing contract with one of the vendors bidding on your cluster. If you like that vendor, and they haven't fucked you over in the past, why not go with them?
The are less likely to take advantage, since they want to continue doing business with you. Your existing relationship will give you a little leverage.
That is a horrible definition of capitalisem, maybe a good one for the markets though.
Capitalism would be invest in the opportunities offering the best return.
It's obvious you know nothing about government accounting. The CASB has the right to re-open the contract for an error or $ .01 (yes, that's right, one cent.)
And they do it.
700,000 US dollars would probably be about a million in Canadian funds. Obviously the slash editor is canadian and just showing his true colours (notice how I spelled "colours")
We bought cluster and less than half of it is functioning and its still "under warantee".
Posting AC for obvious reasons.
Have you bothered to CALL them?
When we purchased ours, we had two nodes that had bad power supplies within the first two months... Replacements arrived within 2 days. The cluster withstood a sever AC outage, where the Ambient Temperature rocketed to 105 degrees and failsafes had not yet been implemented. We've had no further problems with the system since the initial hicup, with a consistent load (added up) of 105.0 and an uptime of 104 days for every node.
How you could allow your cluster to run at half capacity is beyond me. You would have had to let that go for quite some time (unless you've a two node cluster)
Posting AC for obvious reasons.
Probably trolling.
Perhaps consider using a team member from a free software clustering project as your consultant (check credential though)? That way you hopefully get someone who is an expert and will be up front with you.
I would have the research group that I work with at the University of Kentucky build it. Maybe you should contact my professor, Dr. Hank Dietz.
KAYS0University Of Kentucky Supercomputer Breaks The $100 Per GFLOPS Barrier
They built the supercomputer for under $40,000 with 128 nodes + 4 spare nodes, just think how many nodes and how powerful it could be with $700,000!
Let me start off with a disclaimer: I do work for IBM however the following represents my opinions, not that of IBM.
There's a reason that they say you never get fired for going with IBM. IBM has more super-computing experience than anyone. We've got an amazing turn-around capability when it comes to building clusters. But perhaps the best thing with going with IBM is the fact that it builds the relationship.
IBM is very involved with universities especially in the areas of high performance computing. We offer a number of grant programs to help out. I've seen how we handle universities where we make hardware investments. The people handling it really care about making sure things work out well for the students and professors involved.
It's definitely worth calling an IBM sales person about it. If you need a number, feel free to email me and I'll do my best to find you one.
int func(int a);
func((b += 3, b));
http://www.linuxhpc.org/ has a form that will submit to 20 diff companies, most specializing in HPC.
Be a good idea to get in contact with the right people.
I do NOT work for Apple but am enthusiastically using MacOS X (I'm an old UNIX hand who is tired of struggling with Windows and Linux/*BSD when I have 'real' work to do...); on the other hand, I hate the idea of computing power being focused on such esoteric goals (no matter how laudable, a nice generic cluster is not like a telescope looking up at the stars...).
I would suggest 96-112 Xserves and 32-16 PowerMac G5s with supporting hardware (graphic displays for the G5s and XRAIDs for the rackmounted Xserves...)... you could then buy a nice 20 foot truck or an RV to install the cluster and most of the Powermacs, allowing you to take it paying customers who would like to buy time when it is not being used for the research you describe (I am presuming you will own the cluster after the project is over and that you could make an arrangement with NSF that was above the board...).
Heck, you could even buy a few copies of X-Plane (www.x-plane.com) and challenge high school students to develop projects that you could drive up and run with them as part of a competition that could lead to scholarships at Embry-Riddle...
"People asked me why we chose Compaq," says Marshall Peterson, Celera's vice president of infrastructure technology. "The answer is simple. We took a benchmark and gave it to all the vendors. Only two could run it. One ran it in 87 hours.
Compaq ran it in seven." Peterson didn't disclose the name of the other vendor.
From Forbes.com
Whoa, I've been thinking about going to Embry-Ribble, except the one in Daytona Beach, Florida. Maybe I should think about the Arizona one now.