An Inside Look At Warhammer Online's Server Setup

An article at Gamasutra provides some details on the hardware Mythic uses to power Warhammer Online, courtesy of Chief Technical Officer Matt Shaw and Online Technical Director Andrew Mann. Quoting: "At any given time, approximately 2,000 servers are in operation, supporting the gameplay in WAR. Matt Shaw commented, 'What we call a server to the user, that main server is actually a cluster of a number of machines. Our Server Farm in Virginia, for example,' Mann said, 'has about 60 Dell Blade chassis running Warhammer Online — each hosting up to 16 servers. All in all, we have about 700 servers in operation at this location.' ... 'We use blade architecture heavily for Warhammer Online,' Mann noted. 'Almost every server that we deploy is a blade system. We don't use virtualization; our software is somewhat virtualized itself. We've always had the technology to run our game world across several pieces of hardware. It's application-layer clustering at a process level. Virtualization wouldn't gain us much because we already run very close to peak CPU usage on these systems.' ... The normalized server configuration — in use across all of the Mythic-managed facilities — features dual Quad-Core Intel Xeon processors running at 3 GHz with 8 GB of RAM."

Comments

70 * 16 ? Are they kidding me?

[Anonymous Coward]

They have like eight servers (worlds) up, tops. They closed all the other ones down. There's no way they have 1120 servers running.

Re:70 * 16 ? Are they kidding me?

[SCPRedMage]

That's PHYSICAL servers. A single game "server" is made of multiple PHYSICAL servers.

At least gamasutra labels their paid placements

[sleeper0]

Sponsored Feature: Restless Entities Never Sleep -- The Back End of Warhammer Online: Age of Reckoning

New features enhance the overall gaming experience, such as Intel Turbo Boost Technology (to maximize speed for demanding applications), Intel Hyper-Threading Technology (for advanced multi-tasking and support for up to eight threads), and Intel Smart Cache (to provide a higher performing, more efficient cache subsystem). Experience Warhammer Online in its best light with the processor that has become the gold standard in the gaming world, the Intel Core i7 processor Extreme Edition.

Re:At least gamasutra labels their paid placements

[_Sprocket_]

From the beginning of the article:

[In this Intel-sponsored feature, part of the Visual Computing section, the technical experts behind Mythic and EA's Warhammer Online discuss the mechanics of keeping the MMO running across multiple servers and data centers.]

So yeah - if you start picking up on the glossy brochure language... it's because it is, in fact, marketingspeek. But burried in all the "gosh golly isn't this product swell" blather is some interesting little tidbits about the environment; wheat from chaff.

Re:At least gamasutra labels their paid placements

[sleeper0]

Ahh, I was trying to figure out how I missed that - turns out the link to the article starts you on the second page where the disclaimer is nowhere to be found. Not that it's a dig on gamasutra as it's clearly stated in the title. Just wish /. had mentioned it before I read most of the way through and was wondering why I felt like I was reading and intel ad.

Article summary

[chanrobi]

Blah blah blah, intel cpus are the best, blah blah blah. Reads like one giant intel advertisement

Re:Article summary

[iamapizza]

Be thankful it isn't another iphone advert.

Re:WTF is Warhammer Online?

[Anonymous Coward]

Heh. They have a mac client. Nerd rage much?

Virtualization

[Fractal Dice]

I've always felt virtualization was pushed too heavily as a concept, making up for failings of the operating systems and how they are used. Many admins have become so used to consolidating existing legacy servers together that we start thinking every app should be wrapped in its own private copy of an OS - as if the whole point of the concepts of users and groups and limits have been forgotten.

Re:Virtualization

[PhrstBrn]

Sometimes virtualization makes sense for an app where you don't really know exactly what the usage requirements are going to be. You know at first, your app not going to need a full machine to run. So you wrap it in a VM, and throw it onto a shared server. But you think, in the future, you're going to need to scale up to bigger and better hardware. But you're not sure.

If the app is already contained in a VM, it's trivial to just move it from Server A to a Bigger Server B if you need more power. The process takes no longer than it would take you to reboot the machine - sync up disk image (as much as possible), power off machine, finish syncing disk image, power on machine on new hardware. Doing this without virtualization would not be so trivial, and would force you to reinstall the app from scratch.

If the time comes that an app outgrows it's virtual server environment and needs dedicated hardware, again, it's fairly trivial to just copy the disk image onto a disk partition, boot off a rescue disk to repair any driver/hardware incompatibilities between virtual machine and the real hardware, and then simply start up the new machine. Doing this again, would require reinstalling from scratch.

Of course, some things are trivial to move, so it might not make sense to virtualize those things. But other could take a good number of man-hours to install, configure, import data, etc. You probably only want to do this once.

Also, when keeping everything separate, you don't need to worry about things where updating App A on this server, which has new version of dependency X, if app B if going to be affected by this. If everything is separate, you don't need to worry about some future new dependency of your app will break another dependency on another unrelated app.

It's not the end-all-be-all of solutions, but it definitely makes sense in a lot of situations. Throwing more hardware at the problem to make it go away is fairly cheap, and usually much cheaper than paying people people to try and fix interoperability issues between unrelated pieces of software.

Re:Virtualization

[Anonymous Coward]

It is also great when you have an old server that has been running for a long time on piece of crap hardware that is about to die. You can just do P2V and not worry about trying to transfer data over, especially those ones that have been in production for years and reinstalling software is not feasible.

Re:Virtualization

[dkf]

Basically, enabling legacy applications to survive by giving them a slice of a real machine and running them that way is a great crutch. But not more. It would be more efficient to revamp the system and bring it up to contemporary code, but often that's not possible. I blame closed source and companies that wrote it going out of business, but that's me... I could ramble about shortsighted management decisions and putting the life of a company on the line and dependent on the existance of another company, but ... I won't.

The open/closed status of the program code has got nothing to do with it. We use virtualization with plenty of code where we have the source (either open source or written in-house) because that lets us greatly improve the utilization of hardware. Why take up a whole rack of servers to do what one modern blade can cope with? (Most servers aren't CPU-bound.) What's more, it can do this without you having to figure out how to get all those silly deployments to work together nicely.

The other good thing about virtualization is that it lets many people have control over their own machines without needing lots of "servers" under desks. That means you can do things like ensuring that everything that the business really depends on has UPS power and sane networking. (I know. Critical stuff shouldn't be put under someone's desk. Virtualization makes it easier to bring reality closer to that ideal.)

Virtualization isn't perfect at all, but it does cure a bunch of problems that crop up in reality and at far less cost than "doing it properly". (For one thing, it's not cheap to build a new datacenter. Even fitting out a new server room isn't something that you want to have to spend on every day.)

Re:Virtualization

[mlts]

Virtualization gives some advantages:

1: You can move the VM between physical hardware with little trouble. Power off VM, robocopy the files, power it on. For older Windows operating systems that required a reinstall if the underlying HAL changed, this is a large lifesaver.

2: Fast backups with the snapshot functionality.

3: Cloning -- need more instances, grab more hardware, fire up Hyper-V or ESXi, slap the VM on and go to town.

4: Clustering -- several physical machines can host one VM through a SAN and if one box fails, the failover can pick up where the main machine left off on the machine (not the app) level. This means you don't need to worry about how apps will deal with jumping MACs or hardware changes unexpectedly.

5: Security. If a VM got infected, it can be powered off and rolled back to a safe snapshot, and also a snapshot taken of its dirty state for forensics.

6: Ability to run on future hardware. Say everyone ditches x86 and amd64 and decides to go to IBM's POWER architecture and emulate legacy stuff. The stuff in the VM won't care that is is actually isn't running on a different CPU.

Of course, virtualization's disadvantage is performance losses due to the added overhead of more context switching.

For a MMO, virtualization isn't really needed except at the database core. If a zone server [1] goes down, there will be people nerd raging on the forums, but in reality if someone gets to it in 24 hours or so, people won't be pulling their subscriptions. The only real thing that would cause people to bail is a large player database rollback, so days to weeks of playing are lost. However if you have a good database cluster, this isn't going to happen.

Virtualization is just one of many IT tools. Sometimes it is an excellent thing to have. Other times, there isn't any real need to have it, especially for CPU intensive stuff on a server that can be cloned or easily reimaged with the apps on it.

[1]: I'm assuming zone servers handle the combat mechanics, only sending updates to the core player database when a player loots an item, dies, logs out, disconnects, or at a periodic interval if nothing else changes.

Re:Virtualization

[TheRealFixer]

6: Ability to run on future hardware. Say everyone ditches x86 and amd64 and decides to go to IBM's POWER architecture and emulate legacy stuff. The stuff in the VM won't care that is is actually isn't running on a different CPU.

This is not true. Hardware virtualization is not emulation, which is what you're talking about here. Processes in the VM are run directly on the host processor, they're just managed by the hypervisor. There's no emulation layer, since that would make performance pretty atrocious. So, the stuff in the VM absolutely will care about what processor you've moved to, especially if you've suddenly changed instruction sets. Binaries compiled for x86 won't magically run on PowerPC just because it's running on a VM.

Dell ad?

[Degro]

This game failed in big part because of their extremely poor server performance. Who cares how they did it?

Re:Dell ad?

[Vrallis]

This game failed in big part because of their extremely poor server performance. Who cares how they did it?

Well, both poor server performance (Fortress battles were completely unplayable when I quit) plus they opened WAY too many servers at the start. If they'd started with 1/3 as many servers the game would probably be in far better shape today. The server transfers that they opened up as a last-ditch effort prior to merging servers was the straw that broke the camel's back for me, destroying my server's population.

WTB a fantasy (okay, I'll say it...WoW-like) MMO modeled after EVE's economy, industry, PVP, territory control, etc.

Re:Dell ad?

[Opportunist]

You enjoy squaring circles?

You want an MMO that is approachable and very easy to play, where learning a pattern of repeated keystrokes ensures success that runs along the creed of "insert time, get reward", where items and their stats are the be-all, end-all defining element when it comes to the strength of your character without too much dependency on your playing skill (i.e. a WoW-like game) with a complex player driven economy, where almost all equipment is entirely player created, with a complex resource/research/production chain, where groups of players hold territory that determines who has access to which resources (i.e. EvE's economy)?

I think such a game is impossible. Or, at least, it would not be fun to play for the majority of its population...

Re:Holiday slashvertising or simply a slow news.

[Darinbob]

Actually, it is kind of interesting to see what's in the back room. I know in some MMOs if a "server" is down there's inevitably some wiseguy who says "they should buy a better machine", or "I'm an IT dude and I could run the place better than those bozos". Other times people are confused why some regions of the game are working but others are not. Or why it takes so long to reboot the "server" to apply the next game patch. Or why there's scheduled maintenance. I think there are a lot of players who really believe that the companies just go out and buy a single computer (probably a tower) for each game server. The scale of the operations are pretty large.

Re:WTF is Warhammer Online?

[jhoegl]

It is impossible for Mac users to nerd rage.
Mac users would have to be nerds for that statement to qualify.

Re:WTF is Warhammer Online?

[glwtta]

It is impossible for Mac users to nerd rage. Mac users would have to be nerds for that statement to qualify.

So... douche-rage?