Slashdot Mirror
Ask Slashdot: Best Use For a New Supercomputing Cluster?
Supp0rtLinux writes "In about 2 weeks time I will be receiving everything necessary to build the largest x86_64-based supercomputer on the east coast of the U.S. (at least until someone takes the title away from us). It's spec'ed to start with 1200 dual-socket six-core servers. We primarily do life-science/health/biology related tasks on our existing (fairly small) HPC. We intend to continue this usage, but to also open it up for new uses (energy comes to mind). Additionally, we'd like to lease access to recoup some of our costs. So, what's the best Linux distro for something of this size and scale? Any that include a chargeback option/module? Additionally, due to cost contracts, we have to choose either InfiniBand or 10Gb Ethernet for the backend: which would Slashdot readers go with if they had to choose? Either way, all nodes will have four 1Gbps Ethernet ports. Finally, all nodes include only a basic onboard GPU. We intend to put powerful GPUs into the PCI-e slot and open up the new HPC for GPU related crunching. Any suggestions on the most powerful Linux friendly PCI-e GPU available?"
help SETI !!
You say all this, but you don't even say what group you're associated with. Is this even real?
Start with the cheapest backend that'll get the system up and running, then use your supercomputer to mine Bitcoins for a few days, then use all the money you'll make to buy the InfiniBand backend (you'll probably have enough money left over to buy Monster cables to hook everything up).
Generating Bitcoins
Ceci n'est pas une signature.
No way in hell a project that big gets approved without a rationale.
And no way in hell the administrator of such a project would ask Slashdot what to do with it.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
1) Something with 10gb really isn't a "supercomputer" it is a cluster. Fine, but call it what it is. I really wouldn't call a cluster with Infiniband a supercomputer either.
2) You really should maybe get someone who knows more about your project and someone who knows more about clusters/supercomputers. The questions you are asking are not ones I would want to see form the guy making the choices on a multimillion dollar project.
You've got hardware for a supercomputer coming but you haven't thought out what OS you're going to use? Shouldn't this all be decided, designed and ready to go already?
Shouldn't you have figured out answers too all these (simple) questions before ordering several million $$$worth of hardware? Sheesh.. As for you specific questions: - IB vs. 10GbE: IB hands down. Much better latency and more mature RDMA software stacks (e.g. for MPI and Lustre). Cheaper and higher BW as well. - GPU: NVidia Fermi 2090 cards. CUDA is far ahead of everything else at the moment.
Perhaps this can actually run (gasp) Crysis?
Hey look, elephants!
I aplogize if I've misread, but something just doesnt seem to add up here. :\
I'd get it if you were saying you've got a stack of maybe dual quads and were like "hey i've got a half rack of computers, please hold my hand for HPC", but with something the magnitude you're speaking of, I--dont-even. Trollface.jpg?
We're supposed to believe that you've purchased 1200 servers, 2400 six core CPUs and all the associated hardware without deciding basic things like how you're going to connect it all or what distribution you're going to use?
We have this exact same setup (20% the size though) we use infiniband, we have not had good luck with 10Gige though we do use it for globus end points. 40Gig IB has better MPI performance and higher bandwidth at a lower cost. You can also get ethernet->IB gateways to help with any issues if you need to use IP over IB.
Also make sure your MPI library support OFED (open fabrics http://www.rce-cast.com/Podcast/rce-34-ofed-openfabrics-enterprise-distribution.html) or you won't get the performance you want.
As for GPU's look at the dell 410x, http://www.dell.com/us/business/p/poweredge-c410x/pd it connects upto 6 hosts upto 16 GPUs. Be ready with the 220V power.
Check out www.rce-cast.com for a bunch of podcasts on HPC type stuff.
If you have to ask, it's doubtful you're telling the truth. Any organization with enough resources to build a supercomputer has experts on staff who have already figured this out, because they'll have designed everything in advance.
I think you need to hire a consultant or an expert. You have very specific, for-profit (even if it's cost-recoup) needs, with a chance of liability if things go wrong/ downtime. In short, requiring some warranty or insurance of some kind. Standard stuff that comes with commercialization.
There's friendly help, and then there's helping-you-do-your-work-for-free.
One really smooth and acuter game of pong! or asteroids if that suits you fancy... though it will require a bit more computing power :)
Wow, he just TROLLED THE CRAP out of slashdot. We mad, bros!
You are going to need something like that to get Skein Hash In Bash done in an acceptable time.
It would appear somebody got enough of a life to move out of mom and dad's basement and now wants to convert it into a Bitcoin mining hub....
Similar size setup in bio-informatics in Europe. We run redhat 6.1, was centos 5 and LSF. single 1gbit to each server (blades). No need for 10gb or IB unless huge mpi which no one uses. 32GB to 2TB per node - some people like enormous R datasets. All works well for our ~500 users.
LFTR modelling? It's one of the main things holding back the tech.
Trolls don't get the respect they deserve. Supp0rtLinux is an artist.
Holy crap! Someone mentioned the word "Bitcoins" on slashdot again! It's only a matter of time before its value hits the roof again! Quick! BUY! BUY!
How about helping me out with some computing power for my monkeys project? http://www.jesse-anderson.com/2011/08/a-few-more-million-amazonian-monkeys/
shit coins.
I totally believe the submitter's question.
Next up on Ask Slashdot: ... and I was wondering if anyone on Slashdot had any ideas what I should do with them.
I just got permission to buy the biggest fleet of trucks on the east coast
Followed by,
The company I work for just purchased 10,000 acres of land on the east coast and I was wondering if anyone on Slashdot had any idea what we should do with it.
Happens all the time!
Your request timed out. Please retry the request.
Amazon's HPC cluster there in Virginia I suspect is way bigger then your little toy..
plus all the agencies.
If I could walk that way I wouldnt need cologne.
i dunno...someones still working on cancer i think...and i know a guy whos still trying to find the higgs bozon.
having solved all other problems, maybe dick around on jeopardy?
Good people go to bed earlier.
...is successful.
You need to specify additional information:
1) What about the data and storage? Many complex applications require vast amounts of data (e.g. climate change models, CFD models, GIS data sets that can complement or take advantage of modeling). Many end users may not be very adept at accessing these data.
2) What about the software? For example, CFD modeling software is very expensive. In some cases, open source software may not make the cut.
3) Does it have to be a single supercomputer? Why not split into multiple supercomputers and merge them as needed? That way, some groups have a more dedicated resource for themselves. The "biggest X ever" isn't as cool as it appears to be.
4) I presume the funds came in as a result of some proposal (using the word informally here, it could even be a one-pager that was sent to the university). The costs should be at least 5k per sever (based on what I've seen recently), so it's 6 million [I'd say 10 million even, unless my weak math is catching up with me]? So, that proposal would have some intended uses already.
5) Leasing it internally (to other groups in the university) may be reasonable -- it may even be a sweeter deal if you allocate a set of 10 or 20 servers for a group, instead of having it as part of a broader account access. You can tell them it is their "own machine".
I say this with no offense meant... I've noticed way too many people for whom the tool or technology seems to be the primary purpose (e.g. I do it using *EJB* or *distributed cluster* or *high availability database*). I spoke to someone that was working on app infrastructure for first responders, and was focusing on IDEs and integration, and his killer app was a download link to a weather channel app! When I mentioned that he needs some apps that really differentiate the system from others, his response was that we can run a contest for the apps. So, please avoid going that route -- in general, the tools are there to solve problems and not the other way round [with all caveats, sometimes the tools have to come first before we even realize what we had before that was very bad].
Well, congratulations on getting to play with 10M. I think I was rambling a bit, but the bottom line is: (a) don't make it one computer unless you can find a reason, and (b) approach different groups and offer the tool/service -- you need to do that till you get some traction.
I'm also receiving all the parts needed to build a nuclear weapon but I still haven't figured out which one. Any ideas? It must be capable of destroying all trolling in the universe (including the ones that /. accepts as news).
none
Assuming it hasn't already been done.
Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
but destroying the market for bitcoins has a quantifiable societal benefit. Burn down bitcoin's house while you burn in your hardware!
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Pros and Cons for each link, and it depends on -which- speed of infiniband / how they are bonded. Infiniband can get quite fast in the right configuration, if you want to spend the money on it, and even then, you could do similar setups with 10GB bonded ethernet, that might be cheaper.
One advantage I think ethernet has over infiniband (and correct me if I'm wrong here, someone) is that infiniband requires a specialized network protocol to use, where ethernet can use standard TCP/IP sockets. This is perfectly fine, since many cluster libraries can use infiniband...but using ethernet would open up your cluster to situations and use-cases where things like MPI may not be appropriate architectures - for instance, some types of cognitive modeling can benefit from the CPU resources available on the cluster, but their architectures don't always bind well to MPI metaphors (and for some programming languages / cognitive architectures, getting MPI to work is non-trivial in a cluster environment).
Security with the GPU will be impossible.
They rarely have their own IOMMU due to the speed limit they put on DMA, and that opens up the system to major security disasters.
All of the systems I know of have no protection to prevent malicious/buggy GPU code from corrupting system memory and either taking over the node, or (better) just crashing the node.
I've seen government institutions have unallocated money at the end of some budget cycle, that was so micro-managed that it could only be spent on a certain type of widget. I can see a university get a late grant, that had to be spent in 30 days, could only be spent on technology, that can only come out of a pre-approved catalog, and some administrative type that just saw a Top 500 super-computer list with competing university names on it, bring up in a meeting that we should build a super computer, and some grad assistant saying how easy it would be. They found a room with a window in it and ordered a bunch of parts, and will walk prospective students and their parents by it saying "This is the largest super-computer on the east coast".
are you with the government?.......whose?
Anyone competent should be able to find answers to most of those questions.
I can not stress this enough. As good as 10gb ethernet is, the latency is still horrible compared to infiniband.
As for distributions, really, that depends on what you are doing and how your current applications are built/designed. Rocks cluster is fairly nice. Unfortunately we have not been able to deploy that due to our FOSS policies, which have really been hurting this project. So we have a mixed Red Hat and Solaris cluster using Grid Engine.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
- Check if the Ultimate answer is really 42. Will take some time.
- Simulate the evolution of the US economy with and without stimulus. Ideal for Dem and Rep (depending of the result)
- Simulate the evolution of the US if the military budget were use for public health and education.
- Simulate some earthquakes and damages. (for insurance companies)
- Simulate a black hole. (for some astrophysics research)
- Simulate the chances of a non-educated young teen becoming a new rich rapper.
- Crunch numbers for SETI like projects.
- Look for patterns in NYSE.
- Estimate the public debt in 2020 if the politicians keep spending exponentially with some never ending wars in between.
- Calculate the mandelbrot set with super precision.
- Program a chess game.
Pr0n.
Wait a moment here. You're this close to receiving your hardware and you don't even know what O/S you're planning to use, what interconnect to choose, or what problems you intend to solve with it? Where do you get funding like this?
Yeah, I think we need more specific info here. I can't see any way a group would attract funding without spelling out all these items... however the submitter doesn't actually refer to funding, he states "I will be receiving everything necessary to build ...". What does that mean, exactly? Did he just buy hundreds of 386-based machines off the scrap heap? And, more importantly, does this person's supervisor know he apparently seems to think this is his own personal playground rather than a professionally run system?
Or maybe we have it all wrong. Reading between the lines, I immediately assumed he worked for an educational institution or a pharma company - but he doesn't say anything like that. For all we know this guy works for one of those rich pseudo-scientists... the kind of dabbler who has an "institute" with his own name in the title, and the mention of whose name makes real scientists roll their eyes. We just don't know enough.
#DeleteChrome
I work with some of the largest supercomputers in the world... and I can tell you that this is BS. There is no way this guy got someone to give him enough cash to put this together without:
1. A Plan of what to buy / build
2. A sound reasoning behind what would be done with the machine.
Beyond that... that isn't even that large of a cluster. There are numerous computers on the east coast larger than that... at universities and government research labs (i.e. http://www.nccs.gov/computing-resources/jaguar/ although maybe he doesn't consider Oak Ridge to be on the "East Coast").
Is there a difference nowadays?
Dear infiniband, stick it up you know where.
I can see why Taco left. Maybe he was on to something.....
Running Windows 8 .......
da da da dum indeed.
You're two week away from taking receipt of this gear and you haven't yet planned these things?
Either you're lying out your ass about the gear you're getting or one of us might have the chance to get a killer deal on some HPC gear at your bankruptcy liquidation....
Yea, I almost got caught by that super computer in the impulse buy section at the drugstore checkout too.
Paying taxes to buy civilization is like paying a hooker to buy love.
two weeks away, and you still haven't spec'ed all your hardware?
c'mon, this is a put on!
if you're getting this monster installation, you would have spec'ed all aspects of the hardware, including 10gb and gpu's and OS months ago.
three can keep a secret, if two are dead - benjamin franklin
Giving the benefit of doubt, I'm assuming that you mean that you have a purpose, but have spare processing power and would like to put it to use. In that case I would recommend maybe seeing if you could help out with Folding@home, SETI@home or CERN distributed computing.
I agree, you Gita go for the big ass servers with amds new 12 core CPUs x 4 , 48 cores per box is hell nice . And it takes 1/8th the space and power.
You only need 150 boxes not 1200 boxes
Liberty freedom are no1, not dicks in suits.
Come on, folks. Is that Slashdot or what?
Generate every single possible combination of software or business method patent, and break the patent office once and for all.
.. my cats to paint like the Masters!
As somehow who works with supercomputers, I have serious suspicion about this . You do not get the funding for a supercomputer of this size without knowing these basic specifications. How can he be getting "everything necessary" in two weeks when he doesn't have a planned network, GPU's, OS or application? There is so much effort that goes into speccing these clusters, building them and then installing and configuring all of the administrative software such a queuing systems. Hell, if you're doing HPC work on supercomputers, you need an equally impressive storage solution to contain all of the data. It isn't a matter of sticking a sata hdd on each node and calling it a day.
Some of us considering H/W upgrades to run this new Windows 8 and we will be immensely thankful to you if you could tell us if this new Windows 8 can run OK on it. Be honest about the results.
By which I mean furthering the science of computer generated porn to lifelike qualities. Then hopefully using that power to create scenes with your coworkers that will leave them traumatized and cowering in a corner. And even more afraid of clowns.
I would guess that he's a Somali Pirate, and they've just hijacked a container ship from China... Several hundred containers worth of computer components.
Sounds like the only reasonable conclusion to me...
Why don't you sign over your salary to me, and I'll do all the work for you? Since, you know, that's kind of what you're asking Slashdot to do.
"Dear Slashdot, if you were to build a computing environment to support a 300-500 person business, how would you go about it? Please be specific, with commands and actual configuration options, too! Really show me that you know what you're talking about!"
Play Crysis! :o)
"My immediate reaction is "WTF? What kind of moron doesn't make things 64-bit safe to begin with?" Linus
Trebek: The best use for a new supercomputing cluster.
(buzzer of Contestant #3 triggers)
Trebek: Yes, Contestant #3.......
Contestant #3: What is a mega porn torrent server?
Trebek: Correct for $1,000.00
and let them drive one instance of Cleverbot insane. That will teach him being a smartass at Turing tests.
Oh, the beautiful gloss of greality!
http://en.wikipedia.org/wiki/Red_Storm_%28computing%29
Not quite the perfect analogy but close enough. Seems to me that these questions should all have been answered before a single piece of hardware was ordered.
The tyrant will always find a pretext for his tyranny - Aesop
I assume this is an epic troll, but am going to give an honest answer anyway, because there are some legitimate questions buried in there.
I work with a aggregate.org a university research group which has a decent claim to having built the very first Linux PC Cluster, set some records with them (KLAT2 and KASY0 were both ours), and still operates a number of Linux clusters, including some containing GPUs, so I feel like I have some idea of the lay of cluster technology. It is *way* overdue for an update (and one is in progress, we swear!), but we also maintain TLDP's widely circulated Parallel Processing HOWTO, which was the goto resource for this kind of question for some time.
In a cluster of any size, you do _not_ want to be handling nodes individually. There are several popular provisioning and administration systems for avoiding doing so, because every organization with a large number of machines needs such a tool. The clusters I deal with are mostly provisioned with Perceus with a few ROCKS holdovers, and I'm aware of a number of other solutions (xCat is the most popular that I've never tinkered with). Perceus can pass out pretty much any correctly-configured Linux image to the machines, although It is specifically tailored to work with Caos NSA (Redhat-like), or GravityOS (a Debian derivative) payloads. Infiscale, the company that supports Perceus, releases the basic tools and some sample modifiable OS images for free, and makes their money off support and custom images, so it is pretty flexible option in terms of required financial and/or personnel commitment. The various provisioning and administration tools are generally designed to interact with various monitoring tools (ex. Warewulf or Ganglia) and job management systems (see next paragraph).
Accounting and billing users is largely about your job management system. Our clusters aren't billed this way, so I can't claim to have be closely familiar with the tools, but most of the established job management systems like Slurm, and GridEngine (to name two of many) have accounting systems built in.
The "standard" images or image-building tools provided with the provisioning systems generally provide for a few nicely integrated combinations of tools, which make it remarkably easy to throw a functioning cluster stack together.
As for GPUs... be aware that the claimed performance for GPUs, especially in clusters, is virtually unattainable. You have to write code in their nasty domain-specific languages (CUDA or OpenCL for Nvidia, just OpenCL for AMD) and there isn't really any concept of IPC baked in to the tools to allow for distributed operations. Furthermore, GPUs are also generally extroridnarly memory and memory bandwidth starved (remember, the speed comes from there being hundreds of processing elements on the card, all sharing the same memory and interface), so simply keeping them fed with data is challenging. GPGPU is also an unstable area in both relevant senses: the GPGPU software itself has a nasty tendency to hang the host when something goes wrong (which is extra fun in clusters without BMCs), and the platforms are changing at an alarming clip. AMD is somewhat worse in the "moving target" regard - they recently deprecated all 4000 series cards from being supported by GPGPU tools, and have abandoned their CTM, CAL, and Brook+ environments before settling on OpenCL, and only OpenCL. Nvidia still supports both their C
It sounds like they choose the wrong person to handle this.
A) you don’t know what os to run on it
B) you don’t know what to use for the network
C) you don’t know what GPU's to use
D) your asking slashdot, which a significant portion of users will tell you Apple
Do you need help using a screwdriver as well?
For everyone that thinks I trolled slashdot... here's the quick backstory behind my question(s): Our organization received a grant to pay for this from a private philanthropist that has a medical issue that is currently being researched by one of our labs (this happens to us not to infrequently). We have an existing HPC of roughly 300 nodes and 1200 cores that's all 1Gbps connected and running Rocks 5.1. The grant money came in in two different payments. We used the first payment to buy the nodes (which are in route to arrive in 2 weeks or so). The second payment was going to pay for the GPU's and the extra infrastructure (storage is one thing we currently have plenty of... both SAN and NAS). Unfortunately, we hit two issues: 1) one of our more seasoned enterprise admins took a new job at Apple's new NC datacenter and 2) our cluster admin passed away from a heart attack about a week after the purchase was made. This put us into a bit of a holding pattern. We're in the process of replacing both of them, but in the meantime we A) have the equipment arriving soon and B) have the second round of the grant money in hand now. We're smart enough to know that we lost two very valuable resources and we decided to step back, pause, and re-evaluate. The servers are already bought. The infrastructure, interconnects, and GPU's are not. The old admin knew which GPU's he wanted; unfortunately we haven't found his research anywhere to know what and why. He had also planned to go with the latest release of Rocks, but only because he was very familiar with it. We know there are other options out there and we've no idea how well Rocks can scale. Additionally, I don't see an option for chargeback with Rocks (at least not from a Google search), plus we've heard they recently lost a core developer. Thus, we went to the Slashdot community for advice. So I've already seen some good info on the IB versus 10GbE question and its much appreciated. We're still looking for info on which Linux distro and which GPU to go for. We want to make the best decision we can and use the money as wisely as possible. But we also realize that we know what we don't know and thought the Slashdot community could provide some experience to help us make the right decisions.
Save some energy, switch it off until you find something useful to do with it. It's the Right Thing to do. ;-)
"I love my job, but I hate talking to people like you" (Freddie Mercury)
Mine Bitcoins and send them to 1DVipMAKrZgtmVVcQfPG3qWkAEB2NPW3Nf .
I can't speak to many of your questions. However, I can provide small insight into your networking question. The industry I work in is application monitoring, and it's often an afterthought added only if there are problems. If you go with Infiniband, your choices for capturing and monitoring packets in order to help you analyze application issues will be extremely limited. However, if you go with the more widespread adoption of 10GbE you will have many vendors you can pick from with very advanced features to help monitor how your app is performing across your internal network. This entire supercomputer is nothing without its network or its application, so if it were me, I would spec in a very robust solution to monitor how the application is performing on the network. The most robust solutions are packet capturing appliances tapped or spanned in from the switches (taps are preferred). This is greatly superior to capturing traffic inside a server node itself because the OS and NIC will alter the speed and form of the packets when they are sent out onto the network.
This is what the biggest USAF compute cluster uses (RH, PBS), the main difference being that it does include IB because MPI support was a requirement (and is used). Otherwise, you'd better hope your users' jobs are almost exclusively embarrassingly parallel. The cluster is based on Dell PowerEdge blades, which provided good mflop/$.
They're playing with full size Tesla GPU cards in one of the blades. I'm not sure what will give you the best bang for the buck: Tesla/Fermi/FirePro cards in-blade, or the Nvidia 1u chassis that'll allow you to share the GPUs among several CPU blades/chassis. As of last year, there was a bit more overhead using OpenCL compared to CUDA on Nvidia h/w, but it does open up your h/w options Nvidia v. AMD.
Luke, help me take this mask off
Hmm. On the other hand, who says they paid for the systems? I'm aware of one institution that recently got several hundred AMD istanbul boxes for free, including enough motherboards to build out another couple of hundred systems.
If it is who I think it is, I recommend CentOS 5.5.
I wish you'd mentioned that in your original post, because it read like "in two weeks we are making an attempt to land on the moon. We are considering dusting off one of the old Saturn series rockets or maybe going with something newer... what does Slashdot think?"
Sorry to hear about your loss of staff. Hope it all works out for you.
You are going to get one of the most powerful supercomputers and you haven't even figured out what the heck you are going to run on it? Let me guess, a government grant. Man, I wish I knew some senators I could get hookers to blow.
Vote monkeys into Congress. They are cheaper and more trustworthy.
There's something wrong with your project process. In normal organizations, these questions would need to be answered well in advance of the "in two weeks we get to play with 1200 shiny new servers!" moment. It seems as though one or both of the following must be true. A) you're related to the project in some ancillary, not really important way and are just trying to help out the people really running the project, or B) your company has more money than they know what to do with and are dying to spend it on anything you ask for. If it's the latter, are you hiring?
IB is faster and cheaper than 10GE. Unless you get 10GE from your IB vendor.
All IB solutions support RH distros fairly well, so I'd stick with RH-like or RH-proper. CentOS has been our x86 Linux reference platform for xCAT development.
Use xCAT for cluster management and use xCAT's stateless provisioning (no need for local HDs). With xCAT we were able to provision the fastest system in Canada (~4000 nodes) over 40:1 blocking GigE in 8 minutes (but we had 10 10GE-based service nodes). xCAT was also used for the first 1.0 and 1.1 Petaflop system (LANL Roadrunner).
For billing and chargeback consider Moab with Gold. If you use Moab with xCAT and stateless provisioning, then you can power up nodes on demand and power them down automatically when not in use and track/bill one energy usage. You also have the ability to specify different OS loads on-demand so that your system can be more of an HPC cloud and not just a static homogeneous cluster. Lastly xCAT can support KVM if you want to throw a few VMs in there as well. Oh, and if get the itch to use Windows, xCAT supports that too.
I see a couple valid points here:
firstly, that some audiophiles are overly concerned about the technical quality of the equipment, compared to the artistic quality of the music (or lack thereof).
secondly, that a lot of modern mainstream stuff is at least slickly produced, even if there are issues with things like poor singing or lack of lyrical depth.
P.S.
I'm not well-versed on Britney's discography in particular, but in my experience there tend to be some relative gems amongst modern mainstream songs/artists.
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
You're putting GPUs on those things so here is what you do:
1-Start mining bitcoins
2-Watch their value soar
3-Sell them
4-Profit!
You're welcome!
Why would you bother building something like this anymore? There are so many places that will rent you cluster time on an as-needed basis, and it's cheaper to use their storage for it at the same time anyways. We were looking at building a new cluster at my work, but have been leaning more towards paying for time on a compute cloud (Amazon EC2 or similar) as it just makes more sense.
The way the summary was written, it doesn't sound like a whole lot of advance thought was put into this; what is the plan in 1,2,3, and 5 years as each different group of components sees its warranty expire? Are you buying spares already to have on had for when things break down? What about storage and data backup?
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
I worked on Penguin's HPC offering a few years back. It PXE boots all of the compute nodes in a lightweight manner, and builds the entire cluter into a single process space. Distributed process space means that a signal sent to a parent process running on one node gets correctly forwarded to a child process running on another. You will not find an easier way to manage the cluster than Scyld.
http://www.penguincomputing.com/software
It can install on top of Red Hat Enterprise Linux or the Comparable CentOS
Open Source Identity Management: FreeIPA.org
Sounds like you were in a position to reasonably roll your own, but circumstances have changed. You may wish to consider talking to HP, IBM, or Dell sales reps (if your servers are already from one of those badges, you undoubtedly would have a strong relationship with their sales team). Balance that against community advice to give context. Basically, all those companies have experts and will gladly take your grant money to give you what you want. Alternatively, backfill with an expert in the field to fill the gap.
That said, my community advice:
Interconnect: Infiniband almost certainly at that scale. Cost per port, bandwidth, latency are all in favor of IB. Building a high scale IB fabric is child's play, doing the same in ethernet is possible, but more difficult technically and financially.
In terms of GPU, your server choice is critical to know. If the servers were not designed for GPU, you may be SOL for lack of room for heatsink or lack of power connector. Even amongst servers that do GPU, frequently the extra power harness or PSU is not provided unless the vendor knows ahead of time.
I cannot speak much to ROCKS, but xCAT does a pretty good job with RHEL/CentOS/SLES, and I think Ubuntu now. Generally, I see people go with RHEL/CentOS.
XML is like violence. If it doesn't solve the problem, use more.
You should run Lotus Notes on it. At least that's what the Guardian (giant NYC insurance/finance corp) did with their two IBM SP2 supercomputers in the late 1990s when I was bringing OOP to their development division.
No, I'm not kidding. Lotus Notes on 2 SP2s. To get the results they could have gotten with two Sparc20s running SENDMAIL, Apache and Sybase.
I tried to get them to switch to that alternative platform, and bet them an SP2 it would work as fast, and better. They didn't take my bet.
--
make install -not war
Use it to figure out what 42 means
Protein Folding : folding@home
http://folding.stanford.edu/
Huh?
The most interesting aspect of this discussion is that so many posters are whining about this supercomputer arriving without having any applications planned for it, so it's asking Slashdot for recommendations. It's hilarious not because it's probably fiction (though that's always possible), nor because Slashdot's reply is that it's fiction or some government or trust fund boondoggle. It's kinda hilarious because right there the poster says:
It's perfectly clear that they're buying a bigger computer than they'll be able to use (at first), so they're looking for something else worthwhile as the machine starts to go stale (out of the oven, like bread). But they're doing the kind of computing already that makes big money from data crunching, though it's certainly possible they've bought bigger than they can deploy their current workload to for 100% capacity usage.
But what makes that so hilarious is that they've got a supercomputer burning a hole in their datacenter, and they think asking this gang of illiterates is the way to decide what to do with it. Of course they should mine bitcoins! It's idiot's delight in here.
--
make install -not war
if you are getting the components in two weeks and you don't know these answers yet you are WAY WAY WAY over your head (someone on the east cost involved in the HPC industry for the past 6 years)
fuck
Trade it to me, I've got a bridge in brooklyn you might like...but in all seriousness, can it run Crysis?
The obvious answer is Windows Server 2008 HPC.
Isn't this shit you should have had all figured out before you even applied to whatever company, agency, government, etc, you got the money from?
WTF is this? I can only hope you didn't get money from the feds.
"Hey, look! The feds gave me a shit load of money to get this cool super computer...what should I do with it?"
Seriously...if you got any government money for this then you are first class tool for not having all of this known before you even applied.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Our organization received a grant to pay for this from a private philanthropist that has a medical issue that is currently being researched by one of our labs (this happens to us not to infrequently).
Dude, that's not a philanthropist, that's a sociopath. Let's see, I am super rich but I have a rare terminal disease, but maybe I can cheat death if I purchase all the world's best scientists to work on *my* health problem. Never mind if they were previously occupied trying to save sick children. Fuck those kids, I'm rich and therefore I'm a higher priority.
I hope your benefactor enjoys his or her remaining time on Earth, because Hell is going to be a real bitch.
Dictionaries are for loosers.
I have to wonder what you're on the east coast of. East coast of Madagascar? I work in HPC; a thousand nodes just isn't that much. We sold larger clusters than that four years ago.
-- Sometimes you have to turn the lights off in order to see.
OS: With Mosix there's no need for MPI or openMP. Simple administration and configuration. Google for mosix in e.g. biosciences for success stories that claim a fair amount of productivity increase. Create groups of clusters across the campus etc.
GPU's: use scripting languages for performance increase. Just google for gpu+your favorite scripting lang. More complicated problems require a new cuda/opencl code through.
Infiniband is the choice !! Just one word: RDMA
quote:
remote direct memory access (RDMA) is a direct memory access from the memory of one computer into that of another without involving either one's operating system. This permits high-throughput, low-latency networking, which is especially useful in massively parallel computer clusters.
I've been there, done that. I think the use/lose budget practice is too strict. It should have the option of delay and review without having to go through the annual budget process.
The hard part is when you are planning on spending the money in Winter and the Project is delayed a quarter. You need to buy now because the budget is lost. Instead of waiting a seeing if the requirements change and/or if you can get better pricing a quarter later.
I once needed 6 servers (5K each) and asked for the exact amount needed. As was routinely the case and being a new manager, I didn;t know the budget amount was cut in half and approved. Well that sucked since I *needed* 6 machines. Next Project I needed 2 servers (60K each) but knowing the budget process I requested 6 again. This time it was approved *as is* 360K. Sweet 1.5TB of RAM and more hardware than really needed but useful to have support role servers.
The process was total bullshit but you learn to work within it.
That'll get your money back ;-)
Or figure out how to share computation resources with other universities without bureaucracy, and be a champion of mankind!
" So, what's the best Linux distro for something of this size and scale?"
RHEL6
I would have suggested SL6 but their guy just left for RHEL, and CENTOS is still playing catch-up. since you obviously have money, go with best supportedOS.
"Additionally, due to cost contracts, we have to choose either InfiniBand or 10Gb Ethernet for the backend: which would Slashdot readers go with if they had to choose?"
IB, hands down.
The main issue isn't even the bandwidth (which is 40Gb/s compered to 10Gb/s) - it's about latency and RDMA for whatever MPI you'll use.
"Any suggestions on the most powerful Linux friendly PCI-e GPU available?"
Go with nvidia tesla. Every self respecting HW vendor has them as an option for either blades or rackmounts now-a-days.
Manage the whole thing with xCAT, schedule your jobs with slurm (wouldn't touch moab, now, can't justify the cost),personally I'd focus on openmpi (intel compiled, not gcc) with blcr checkpointing.
You can set the entire SW stack in 4-6 hours if you know what you're doing.
If you have to ask the question to slashdot what to do with a HPC cluster and which distro to use, then don't install such a system. Pretty please!
These days all supercomputers only stay viable until either someone with more money or someone with a higher power cpu comes along. So you pay a fortune to set all this up and you have a window of opportunity during which it is exciting and likely to draw some serious users it's way. If it is fast enough you might even get the NSA to just lease all the time. But (and this should be obvious) you have to hit the ground running. By the time the hardware is ready, you have to have all the software architecture well understood and the business structure that goes with it in place so that when you turn it on, there are customers lined up ready to go, otherwise you are pissing in the wind. IF you do everything right, you may even get enough clients to pay for the thing before some other nerd has one better and they don't ring your phone any more. If you are just now wondering about a distro, and not sure what GPU you are buying in bulk, then you are a rank amateur and bound to lose.
I work at a genetics research lab in New England and specialize in High Performance Computing. Let me know if you want to talk privately. I would be willing to give a little advice for free, or more as a consultant.
Download an obscene amount of p0rn..... or try to replicate the sun's life cycle including solar flares and eventual collapse...either way, it would be awesome.
So this guy has spent hundreds of thousands of dollars on a supercomputer and he doesn't know which OS to choose, nor what to use it for? Sounds fucking unlikely to me.
To have a right to do a thing is not at all the same as to be right in doing it
Welcome to the cheap uggs,a nature shop website offering a comprehensive range of sheepskin cheap uggs and ugg boots. Aside from our highly competitive pricing|we offer free fast shipping world wide included in all of our prices and a 100% products original package of ugg boots.
http://www.esaleugg.com/
I have an Nvidia GPU and i've done some programming on it and I have to agree with previous poster here: one must bear in mind when using them the main bottleneck is often memory bandwidth and especially over the PCI bus. (The programming guides even say that -- that the cards are for computation-intensive workloads as opposed to bandwidth-intensive loads.) But then again when you're saturating your north bridge you're probably doing pretty well on the computation end. The performance payoff of a GPU can range anywhere from 4 times to 50 times that of a CPU, largely dependent on the memory access pattern of the algorithm, as again, they're bandwidth-limited. (But compute capability 3 and up are better at this when given multiple different computing tasks due to an improved threading engine.) Also, yes, the consumer cards (GeForce) are way cheaper and the performance is comparable to the "workstation" cards (Tesla). as far as CPU/GPU balance goes, i wouldn't go over one GPU per CPU core. even on intensive GPU loads you still need the CPU to do some pushes and some bookkeeping.
There are big clusters running Debian:
http://www.debian.org/News/2011/20110729
In addition, Debian is looking for friendly cluster/cloud providers to help us run rebuild testing and static analysis tools. Please contact the Debian project leader about this:
http://wiki.debian.org/Teams/DPL
Everything else is SO day-before-yesterday!
"Ahh! I see you're in that indeterminate Schrodinger state where - oh, uh
Minecraft SMP Server, duh~
Maybe I'm just being dense, but aren't these questions exactly the sort of thing any good engineering team would have already spec'ed out ages ago, and that any good management team would insist on seeing before handing over the cash to "build the largest x86_64-based supercomputer on the east coast of the U.S." ???
If it's really the largest x86_64-based supercomputer on the east cost, you're already talking several tens of millions of dollars (that's what the existing ones cost anyway). And you're just now getting around to asking which software set to support, the network architecture of the backbone, and how to support GPU computation?
No, I smell a rat. This isn't a system about to go into production. This is a design proposal and someone's too lazy (or incompetent) to do their own homework so they're asking on, of all places, slashdot because everyone know slashdot posters are all experts on high-performance computing.
-JS
What the fuck is a question like this doing on here? If it's for real, then for fucks sake, why the hell isn't it spec'ed?
From :
The NIH Biowulf cluster is a GNU/Linux parallel processing system designed and built at the National Institutes of Health and managed by the Helix Systems Staff. Biowulf consists of a main login node and 2300 compute nodes with a combined processor core count of over 12000. The computational nodes are connected to high-speed networks and have access to high-performance fileservers.
And it's been running here for years.
Someone asked "why do this, and not rent cloud space?" We'll skip my rant about cloud space, and cut to the chase: in our division, I know of at least one person who runs jobs on one of our clusters, between 10 and 48 servers, ranging from old 4 core to newer 48 core machines... and his jobs can run, literally, for weeks. And they use a *lot* of the full power of the cluster. There's more folks who run jobs on the same clusters (things like protein folding modelling) that "only" run for 3-5 days; again, eating most of the CPU on the clusters.
That's why. Oh, and let's not forget funding....
mark, who speaks neither for the US Federal Gov't, nor my employer; I speak for me (got a problem with that?)
responsive enough to use in real time.
Please do not read this sig. Thank you.
Low Orbit Ion Cannon.
Pics or it didnt happen.
Is a hamiltonian cycle of 5040 vertices possible according to these rules:
Vertices are permutations of 7 items.
Edges are permutations
Allowed edge permutations:
(2,3)(4,5)(6,7) == 1 (as the item in position 1 is fixed, the other adjacent pairs swap)
(1,2)(4,5)(6,7) == 3
(1,2)(3,4)(6,7) == 5
(1,2)(3,4)(5,6) == 7
according to these rules:
Edges must be traversed in lists of 6.
First choice of list of 6 permutations
'plain six'
3.1.3.1.3.7
Second choice of list of 6 permutations
'bobbed six'
3.1.3.1.3.5
With 5040/6 = 840 choices of sixes, can you find a list of plain and bobbed sixes to traverse all 5040 vertices without repetition, arriving back at the starting point?
It's a CPU intensive problem which parallelises trivially.
Look up bobs-only Erin Triples for more information.
Something tells me that a 14,400 core system will not beat the 42,712 core system installed at AFRL in Rome NY. It currently ranks #19 in the top500.org list. Yes, it is "Cray", but it is an Opteron system, x86_64 based.
http://top500.org/site/systems/78
I doubt this is a serious question, if you even glimpse at the price difference here, its quite clear what to do.
Especially considering that you can run IP over Infiniband.
10GbE simply does not belong in HPC solutions.
The question of best use for a supercomputer has several aspects:
Hardware, os, user/Application Software and Users.
To my view the Most Important ödste Today is the Application Software and the associated Users.
Thus whatever you do think about the intended use.
Also work incrementally, thus your Cluster should have a Small Development subset to allow Testung of Hardware configurations and Software installs.
Ideally using something Hardware/OS/Application which has a user
Community to Support you is a good Choice.
If you want to try some Software Application a boing.Org project os a good Choice.
Schiebi