Slashdot Mirror

← Back to Stories (view on slashdot.org)

Making Best Use of Data Center Space: Density Vs. Isolation

Posted by samzenpus on from the best-of-both-worlds dept.

jfruh writes The ability to cram multiple virtual servers on a single physical computer is tempting — so tempting that many shops overlook the downsides of having so many important systems subject to a single point of physical failure. But how can you isolate your servers physically but still take up less room? Matthew Mobrea takes a look at the options, including new server platforms that offer what he calls "dense isolation."

37 of 56 comments (clear)

  1. TL;DR by Anonymous Coward · · Score: 1

    Man buys 1/3 rack and fills it. Looks for faster servers.

    1. Re: TL;DR by saloomy · · Score: 3, Interesting

      He should consider using virtualization to increase his uptime since he is worried about multiple important systems on a single server. Virtualization gives you such good yields in consolidation, you can come out ahead while still using redundancy features like VMware FaultTolerance. Your vm runs "in-step" on two hosts, and will survive even if either host fails. Just requires 2X the used memory. That's still only the most extreme case though like for databases, as most servers should be able to survive a reboot (which is what happens when your host dies and there is capacity left in your cluster. The VM powers back up on another host.

    2. Re: TL;DR by mlts · · Score: 1

      Of course, there is the fact that the VM running with VMWare's fault tolerance can only have one vCPU... so this means that you can't really use it for high-availability database apps. Even a Splunk instance will set off high CPU alarms.

      There are other restrictions as well. VMWare's high availability is somewhat useful (lose a running VM, it will restart the instance)... but there is the downtime waiting for the VM to come up, load its stuff, and start taking requests.

      All and all, it is better than nothing, but it isn't a silver bullet.

  2. Blade Servers aren't "new server platforms" by Walking+The+Walk · · Score: 1

    I read the blog post, and he's just comparing having a beefy server with multiple VMs to instead having a bunch of blade servers. How is this new? Heck, 13 years ago at a Canadian federal government job we swapped our web servers for blades.

    --
    A recursive sig
    Can impart wisdom and truth
    Call proc signature()
    1. Re:Blade Servers aren't "new server platforms" by Anonymous Coward · · Score: 2, Interesting

      Heck, 13 years ago at a Canadian federal government job we swapped our web servers for blades.

      Which was pretty bleeding-edge at the time, since the first blade server was 2001. So not sure what your point about the government is - they weren't late to the party, far from it.

    2. Re:Blade Servers aren't "new server platforms" by Walking+The+Walk · · Score: 1

      Heck, 13 years ago at a Canadian federal government job we swapped our web servers for blades.

      Which was pretty bleeding-edge at the time, since the first blade server was 2001. So not sure what your point about the government is - they weren't late to the party, far from it.

      If I hadn't posted on this story, I would mod the above interesting. I just assumed we were at least a couple of years behind the curve. We were buying off the shelf hardware, nothing custom.

      --
      A recursive sig
      Can impart wisdom and truth
      Call proc signature()
    3. Re: Blade Servers aren't "new server platforms" by Anonymous Coward · · Score: 1

      It's turtles all the way down I tell ya

    4. Re:Blade Servers aren't "new server platforms" by mlts · · Score: 1

      It really depends on the blades and 1U machines. Without exact machines, it can be a tossup, as a blade chassis takes up a ton of rack units. If comparing HP G8 blades to HP G8 1Us, the blades will edge out if they are just being use as compute nodes with the onboard storage used to load the hypervisor, then they hit the SAN for everything else. However, stacking a bunch of 1U machines can be just as good, and the advantage of 1U boxes is that you don't have to worry about the server maker discontinuing the enclosure the blades are in.

      If HP can get the Moonshot environment with 45 blades in a fairly skinny enclosure going, then things will change big time, but for now, I personally lead towards a rack/blades, but there isn't anything wrong with stacking the 1Us, provides there is a decent storage and network fabric [1] that is available.

      [1]: One can use the same fabric for both. Toss in some Isilon heads and a subnet for NFS or iSCSI access, call it done.

  3. Twins by darkain · · Score: 1

    Personally I keep eyeballing the SuperMicro TWIN line. Extremely dense configurations of multiple servers per unit. Spread the workload across multiple physical boxes. Use something like vCenter Server to manage the networking and other resource configurations to simplify making them all the same and adding easy of migrating VMs from one physical host to another.

    1. Re:Twins by K.+S.+Kyosuke · · Score: 2

      Judging from their figures, "extremely dense" ("375 GFLOPS/kW") is actually fairly mediocre, considering that the Blue Gene/Q figures with POWER chips is several times better. That's obviously not exactly relevant for generic server workloads, but I wonder what the figures on those are.

      --
      Ezekiel 23:20
    2. Re:Twins by enjar · · Score: 1

      We use Twins extensively in our data center and have several racks full of them. We've been using them for several generations and are pretty pleased with how they have evolved over time. We now use the Twin2 units pretty much exclusively. We like the shared, hot-swappable power supplies and 4 systems in 2U layout -- which is certainly dense enough for our needs. We also have a great local VAR (greater Boston area) who is awesome in terms of RMAs, warranty service, and no-nonsense quoting when we need new systems -- they set us up with a login on their web page that will get the price dead on so we can get approval for that amount ... no "we are running a special 50% discount just for you" that requires a phone call and/or meeting. They will also send out guys to do a rack and stack who are really good at it -- you get systems shipped, they will put them in the rack in serial number order (easier RMA when you can just count up!) The prices are also very reasonable, and the extra bit of space for a card allows us to add an expansion card as we need it. We'd been through some other server vendors and we have stuck with these guys the longest because they work hard and are great to do business with.

      We looked long and hard at blades, too -- but in many cases they were simply TOO dense for our needs, as we do sometimes need an expansion board, USB slot or some other thing on one of the machines, where we don't have to go up to the full 1U or 2U server to accommodate that need.

      We also get into cases where we need traditional 1U/2U systems for something or other, and we generally just use the same guts that are in the Twins, which means we don't have to deal with weird driver issues for a different system board, so we can deploy or base operating systems + packages onto it without issue.

      My day job is in HPC land, so I know there are more dense things out there, but for x86_64 computing, the Twins are really good for what they are. When you start stepping off the "mundane mainstream server" path and get into the "ultra specialized, boutique" stuff, the cost starts rapidly outstripping the benefit. Of course, for some applications you need to go there, but for what we do, we have more flexibility with space than we do with budget so the Twins strike a nice balance.

    3. Re:Twins by enjar · · Score: 1

      Thinkmate

  4. Blades by Stripe7 · · Score: 1

    You should have your VM images on some storage system like a NetApp, this lets you transfer the entire VM to another blade if one fails. So you have two blade racks both connected to the NetApp with software set up to fail over all the VM's from a failed blade to a blade on the second blade rack. You would probably run all the blades active on 1/2 load where on failure you transfer to the alternate blade on the 2nd rack and go to a full load on that blade. This protects you from a rack failure as well as an individual blade failure. The router/hub on a blade server is a single point of failure which is why you need 2 racks.

    1. Re:Blades by drinkypoo · · Score: 1

      You should have your VM images on some storage system like a NetApp

      Nope. That's a single point of failure. You need two of those, too.

      Basically you need to have two racks in different DCs with replication between their filers

      Or you need to accept that you can't guarantee uptime

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    2. Re:Blades by mikey1134 · · Score: 2

      The SAN is usually less of a single point of failure because they usually have a lot of redundancy built-in, redundant storage processors, multiple backplanes, etc. You're right that off-site replication is still important, but usually more for whole site loss than storage loss.

      --
      <gir voice> I love this sig... </gir voice>
    3. Re:Blades by drinkypoo · · Score: 1

      NetApp became famous by stuffing a PC in a box, adding their software, and calling it a filer. They've advanced since, but lots of people are still cheaping out on filers so the point still stands

      --
      "You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
    4. Re:Blades by hawguy · · Score: 1

      You should have your VM images on some storage system like a NetApp

      Nope. That's a single point of failure. You need two of those, too.

      Basically you need to have two racks in different DCs with replication between their filers

      Or you need to accept that you can't guarantee uptime

      You don't really need two separate filers, you just need a two headed filer to prevent any single point of failure. You can lose anything (even a controller on a disk shelf) and not even notice until the replacement is fedexed to you tomorrow.

    5. Re:Blades by Zondar · · Score: 1

      Til you lose the site...

    6. Re:Blades by hawguy · · Score: 1

      You don't really need two separate filers, you just need a two headed filer to prevent any single point of failure

      Til you lose the site...

      I'm pretty sure that it goes without saying that single-site redundancy means that if you lose the site, you lose everything. Though if you have a segmented datacenter, Netapp will let you separate the heads by up to 500 meters. Likewise, you can separate the disk trays so you can lose an entire datacenter segment without losing data.

      If you want replication across sites, Netapp will be more than happy to help you out with a variety of synchronous and asynchronous replication options. For a price, of course.

    7. Re:Blades by dnavid · · Score: 1

      The SAN is usually less of a single point of failure because they usually have a lot of redundancy built-in, redundant storage processors, multiple backplanes, etc. You're right that off-site replication is still important, but usually more for whole site loss than storage loss.

      People assume the biggest source of SAN failures is a hardware failure, and believe hardware redundancy makes SANs less likely to fail. In my experience, that's false. The biggest source of SAN failures are (usually human-) induced problems from the outside. Plug the wrong FC card with the wrong firmware, knock out the switching layer. Upgrade controller incorrectly, bring down SAN. Perform maintenance incorrectly, wipe the array. SANs go down all the time, and often for very difficult to predict reasons. I saw a SAN that no one had made any hardware or software changes to in months just suddenly crap out when the network that connected it to its replication partner began to experience flapping which noticeably affected no one except the SAN, which decided a world without reliable replication was not worth living in and committed suicide by blowing away half its LUNs.

      Keep in mind the last time I saw a hard drive die and take out a RAID array was so long ago I can't remember. However, the last time I saw a RAID *controller* take out a RAID array - and blow the data away on the array - was only a couple of years ago. Its important to understand where the failure points in a system are, particularly when it comes to storage. These days, they often are not where most people are trained to look. Unless you are experienced with larger scale storage, you're not trained to look where the problems tend to be.

      Storage fails. Any vendor that tells you they sell one of something that doesn't fail is lying through his teeth. Anything you have one of you will one day have to deal with having zero of. It doesn't matter how "reliable" the parts in the one thing are. You should plan accordingly.

  5. this article by Idimmu+Xul · · Score: 2

    is not good.

    --
    The problem with slashdot is that most of its users were bullied and stuffed into lockers as kids!
  6. Simple by ledow · · Score: 4, Insightful

    Put all your eggs in one basket.
    Then make sure you have copies of that basket.

    If you're really worried, put half the eggs in one basket and half in another.

    We need an article for this?

    Hyper-V High Availability Cluster. It's right there in Windows Server. Other OS's have similar capabilities.

    Virtualise everything (there are a lot more advantages than mere consolidation - you have to LOVE the boot-time on a VM server as it doesn't have to mess about in the BIOS or spin up the disks from their BIOS-compatible modes, etc.), then make sure you replicate that to your failover sites / hardware.

    1. Re:Simple by ledow · · Score: 3, Interesting

      I have just put in a Blade / VM configuration at a school (don't ask what they were running before, you don't want to know).

      Our DR plan is that we have an identical rack at another location with blades / storage / VM's / etc. on hot-standby

      Our DDR (double-disaster recovery!) plan is to restore the VM's we have to somewhere else, e.g. cloud provider, if something prevents us operating on that plan.

      The worries I have are that storage is integrated into the blade server (a SPOF on its own, but at least we have multiple blade servers mirroring that data), and that we are relying on a single network to join them.

      The DDR plan is literally there for "we can't get on site" scenarios, and involves spinning up copies of instances on an entirely separate network, including external numbering. It's not a big deal for us, we are merely a tiny school, but if even we're thinking of that and seeing those SPOF's, you'd think someone writing their article into Slashdot would see that too.

      All the hardware in the world is useless if that fibre going into the IT office breaks, or a "single" RAID card falls over (or the RAID even degrades, affecting performance). It seems pretty obvious. Two of everything, minimum. And thus two ways to get to everything, minimum.

      If you can't count two or more of everything, then you can't (in theory) safely smash one of anything and continue. Whether that's a blade server, power cord, network switch, wall socket, building generator, or whatever, it's the same. And it's blindingly obvious why that is.

  7. Physicalisation by Pegasus · · Score: 2

    Look at the likes of HP Moonshot and AMD Seamicro. Those are some nice toys to play with ...

  8. Mod parent up. by khasim · · Score: 2

    Nor are they "isolated". All of the blades connect to the same backplane.

    And moving VM's between individual blades is a hassle unless you use some form of shared storage. Which makes them even less "isolated" but more redundant.

    This reads more like he just wanted to show off that he calls blade servers "dense isolation".

    So is it better to have a bunch of isolated servers which reduces the VM domino effect in exchange for increased hardware maintenance? Or just a few massive servers and be ready for the 4 am call to replace a CPU at any given moment?

    VM is not magic. Also look into "fail over".

    If you have to be called in to replace a CPU at 4 am then you have not planned correctly.

  9. It's all about cost, not physical space. by rodrigoandrade · · Score: 1

    At most places I've worked, the situation described was requested / imposed by the PHBs and bean counters to save costs, reliability and isolation be damned.

    Lack of physical room was NEVER, I repeat, NEVER an issue. This ain't Tokyo we&#194;&#180;re talking about. If your office building doesn't have a spare room, you'll have other major issues down the road when ytour company expands its business.

  10. Blade servers blow by swb · · Score: 3, Interesting

    I'll accept the idea that somewhere somebody has so many servers and so little space that a blade center was the only way they could achieve the density they needed.

    Except I've never seen it -- all the blade centers I've ever seen have been partially full and the equivilent 1U and 2U servers probably would have fit in the same or less space than the blade chasis was occupying.

    And almost always there's a mongolian clusterfuck when they decide to add blades to the chasis -- which they inevitably do, because they have so much money sunk into the blades that there's no way out from under it.

    The mongolian clusterfuck is the result of the byzantine cofiguration rules each vendor has for determining a blade's NIC or FC mapping with the blade center's (overpriced) internal switch bays. Half or full height? LoM or mezzanine slot? Which mez slot? Which blade slot? Oh, you want an extra NIC on that blade? Sorry, the mapping requires an additional switching module which will cost you more than any decent L3 48 port gig switch.

    Whatever the savings from the blade center (and maybe in some metered situation there is power savings of couple hundred watts) is easily lost in hours of troubleshooting when trying to do something different.

    Blade centers always look like some kind of pre-virtualization version of server consolidation that became obsolete once 24U of servers could easily be run on 8U or less of VM host and SAN. They would be a lot more interesting if their mapping regeimes weren't hard wired -- blade advocates give me blahblah "point of failure" about a switchable/configurable backplane.

    1. Re:Blade servers blow by Connie_Lingus · · Score: 1

      that was awesome...thanks.

      --
      never bring a twinkie to a food fight.
    2. Re:Blade servers blow by Jaime2 · · Score: 2

      HP Blades put a 2U server in 1.6U or a 1U server in 0.8U. The only downside is that there is very little room for local storage. If you are virtualizing, SAN storage is inevitable anyways. The power backplane is just a hunk of copper, and all the intelligent stuff is duplicated, so there isn't really a single point of failure - but I wouldn't go blades unless I was at the scale of needing at least three blade chassis so it would be possible to shut one down and not interrupt production. The most legitimate complaint about blades is that they're too dense. A rack that consumes 56 kilowatts is a challenge to cool.

      I've found that two fiber modules - at $6K each - are cheaper than two fiber cards in each of 16 individual servers. Don't buy the embedded switches, use the ones that act as a rear mounted ethernet port, along with your switches of choice. If you do want switching on the chassis, get the ones with 10G uplinks so you can drastically reduce the amount of wiring you need.

      As for configuration complexity - don't use blades if you are a shop where every server is different. Their ideal use case is one where you need 100 or more of the same configuration - like a VM farm.

    3. Re:Blade servers blow by neurovish · · Score: 1

      I'll accept the idea that somewhere somebody has so many servers and so little space that a blade center was the only way they could achieve the density they needed.

      Except I've never seen it -- all the blade centers I've ever seen have been partially full and the equivilent 1U and 2U servers probably would have fit in the same or less space than the blade chasis was occupying.

      And almost always there's a mongolian clusterfuck when they decide to add blades to the chasis -- which they inevitably do, because they have so much money sunk into the blades that there's no way out from under it.

      The mongolian clusterfuck is the result of the byzantine cofiguration rules each vendor has for determining a blade's NIC or FC mapping with the blade center's (overpriced) internal switch bays. Half or full height? LoM or mezzanine slot? Which mez slot? Which blade slot? Oh, you want an extra NIC on that blade? Sorry, the mapping requires an additional switching module which will cost you more than any decent L3 48 port gig switch.

      Whatever the savings from the blade center (and maybe in some metered situation there is power savings of couple hundred watts) is easily lost in hours of troubleshooting when trying to do something different.

      Blade centers always look like some kind of pre-virtualization version of server consolidation that became obsolete once 24U of servers could easily be run on 8U or less of VM host and SAN. They would be a lot more interesting if their mapping regeimes weren't hard wired -- blade advocates give me blahblah "point of failure" about a switchable/configurable backplane.

      The HP c-class isn't that bad. It's been pretty set it and forget it. The ESX runs off of an SD card (or maybe it's just a boot image, there's a VM team that deals with that stuff), then all the datastores are hosted on a SAN. The blades themselves are just compute and memory.

      Of course your original argument still stands, I've never seen a case where real estate is at such a premium that blades are the only way to go. Usually I see racks and racks of storage taking up room instead of servers, but for me they make adding/configuring physical servers easy. No storage and networking teams to send tickets for configuring zoning and vlans. I just go into the bladecenter and connect one to the other.

    4. Re:Blade servers blow by swb · · Score: 1

      I don't doubt there are density scenerios where they make sense, especially in some kind of purpose built farm where you can get major benefits from reduced cabling (although 10G ethernet works to the advantage of standalone servers, too).

      But in so many installs I've seen there hasn't been a full-density buildout, ever. It's always piecemeal and always leads to head scratching and downtime to sort out port mapping and other issues.

      I had a client with older HP blade center not getting the port mapping he wanted (10G mez cards not mapping to 10G switch modules) and his HP vendor was just like "just reset all the switches and you can probably map them anyway you want" because they didn't know, either.

      I've worked for a Dell partner and we literally got lucky and found a Dell document that detailed port mapping scenerios, but it was a hard document to come by. I hate to put on my tinfoil hat, but part of me believes they want it complex so people buy a lot more switching modules than they might need and get sunk into an investment they can't get out of.

    5. Re:Blade servers blow by bobaferret · · Score: 1

      We're actually moving away from blade servers to standalone servers. You can get so many cores on a chip now that each set of 8 blades can be replaced by 1 2U server. Each server has a number of raided 2.5" drives that we use with glusterfs for redundancy between servers which we then re-export over Fiber channel so that we can get rid of our dedicated FC arrays, yet continue to use the FC infrastructure we have in place.

    6. Re:Blade servers blow by nabsltd · · Score: 1

      The mongolian clusterfuck is the result of the byzantine cofiguration rules each vendor has for determining a blade's NIC or FC mapping with the blade center's (overpriced) internal switch bays.

      Cisco's blades do all of this through software...you can add and delete NICs and fiber channel cards with a couple of mouse clicks on the Java applet that runs in the browser.

    7. Re:Blade servers blow by Monoman · · Score: 1

      Blade severs should only be considered when you have space, power, or cooling constraints. They are more expensive and create their own specific issues like you said.

      As far as scaling up vs scaling out in a VM cluster. I prefer more smaller nodes over fewer larger nodes. In a two node VM cluster you can't exceed 50% of any resource unless you are willing to deal with oversubscribed resources during a node failure. Well that isn't a very efficient use of equipment. As you add nodes there you can increase the usable percentage of a resources.

      --
      Keep the Classic Slashdot.
  11. Reads like an ad... by Drewdad · · Score: 1

    He mentions just one product, while ignoring a host of other offerings.

  12. Jails Are a Steal by Blaskowicz · · Score: 1

    In the current US situation at least, you have room to move servers from figurative jails to real ones.

  13. Cost Calculation- 3U, 8 nodes microcloud: $10,072 by Anonymous Coward · · Score: 1

    SYS SM MicroCloud 3U SYS-5038ML-H8TRF (8 Nodes Intel Xeon E3 v3 Nodes)
    CPU Intel Xeon E3-1270v3 Quad-Core 3.50GHz (3.90GHz Turbo) 8MB 80W
    MEM DDR3 1600 8GB ECC Unbuffered (32GB Per Node)
    SSD Intel 530 Series SSDSC2BW240A401 2.5 inch 240GB SATA3 Solid State Drive
    KIT SM Black Hotswap Gen 6 3.5" to 2.5" Hard Disk Drive Tray (MCP-220-93801-0B)

    $1259 per node. $10,072 for 8 nodes

    Please call for more info.
    rackmount server specialist 408-736-8590
    www.kingstarusa.com