An Inside Look At Warhammer Online's Server Setup

An article at Gamasutra provides some details on the hardware Mythic uses to power Warhammer Online, courtesy of Chief Technical Officer Matt Shaw and Online Technical Director Andrew Mann. Quoting: "At any given time, approximately 2,000 servers are in operation, supporting the gameplay in WAR. Matt Shaw commented, 'What we call a server to the user, that main server is actually a cluster of a number of machines. Our Server Farm in Virginia, for example,' Mann said, 'has about 60 Dell Blade chassis running Warhammer Online — each hosting up to 16 servers. All in all, we have about 700 servers in operation at this location.' ... 'We use blade architecture heavily for Warhammer Online,' Mann noted. 'Almost every server that we deploy is a blade system. We don't use virtualization; our software is somewhat virtualized itself. We've always had the technology to run our game world across several pieces of hardware. It's application-layer clustering at a process level. Virtualization wouldn't gain us much because we already run very close to peak CPU usage on these systems.' ... The normalized server configuration — in use across all of the Mythic-managed facilities — features dual Quad-Core Intel Xeon processors running at 3 GHz with 8 GB of RAM."

Comments

At least gamasutra labels their paid placements

[sleeper0]

Sponsored Feature: Restless Entities Never Sleep -- The Back End of Warhammer Online: Age of Reckoning

New features enhance the overall gaming experience, such as Intel Turbo Boost Technology (to maximize speed for demanding applications), Intel Hyper-Threading Technology (for advanced multi-tasking and support for up to eight threads), and Intel Smart Cache (to provide a higher performing, more efficient cache subsystem). Experience Warhammer Online in its best light with the processor that has become the gold standard in the gaming world, the Intel Core i7 processor Extreme Edition.

Virtualization

[Fractal Dice]

I've always felt virtualization was pushed too heavily as a concept, making up for failings of the operating systems and how they are used. Many admins have become so used to consolidating existing legacy servers together that we start thinking every app should be wrapped in its own private copy of an OS - as if the whole point of the concepts of users and groups and limits have been forgotten.

Re:Virtualization

[dkf]

Basically, enabling legacy applications to survive by giving them a slice of a real machine and running them that way is a great crutch. But not more. It would be more efficient to revamp the system and bring it up to contemporary code, but often that's not possible. I blame closed source and companies that wrote it going out of business, but that's me... I could ramble about shortsighted management decisions and putting the life of a company on the line and dependent on the existance of another company, but ... I won't.

The open/closed status of the program code has got nothing to do with it. We use virtualization with plenty of code where we have the source (either open source or written in-house) because that lets us greatly improve the utilization of hardware. Why take up a whole rack of servers to do what one modern blade can cope with? (Most servers aren't CPU-bound.) What's more, it can do this without you having to figure out how to get all those silly deployments to work together nicely.

The other good thing about virtualization is that it lets many people have control over their own machines without needing lots of "servers" under desks. That means you can do things like ensuring that everything that the business really depends on has UPS power and sane networking. (I know. Critical stuff shouldn't be put under someone's desk. Virtualization makes it easier to bring reality closer to that ideal.)

Virtualization isn't perfect at all, but it does cure a bunch of problems that crop up in reality and at far less cost than "doing it properly". (For one thing, it's not cheap to build a new datacenter. Even fitting out a new server room isn't something that you want to have to spend on every day.)

Re:Virtualization

[mlts]

Virtualization gives some advantages:

1: You can move the VM between physical hardware with little trouble. Power off VM, robocopy the files, power it on. For older Windows operating systems that required a reinstall if the underlying HAL changed, this is a large lifesaver.

2: Fast backups with the snapshot functionality.

3: Cloning -- need more instances, grab more hardware, fire up Hyper-V or ESXi, slap the VM on and go to town.

4: Clustering -- several physical machines can host one VM through a SAN and if one box fails, the failover can pick up where the main machine left off on the machine (not the app) level. This means you don't need to worry about how apps will deal with jumping MACs or hardware changes unexpectedly.

5: Security. If a VM got infected, it can be powered off and rolled back to a safe snapshot, and also a snapshot taken of its dirty state for forensics.

6: Ability to run on future hardware. Say everyone ditches x86 and amd64 and decides to go to IBM's POWER architecture and emulate legacy stuff. The stuff in the VM won't care that is is actually isn't running on a different CPU.

Of course, virtualization's disadvantage is performance losses due to the added overhead of more context switching.

For a MMO, virtualization isn't really needed except at the database core. If a zone server [1] goes down, there will be people nerd raging on the forums, but in reality if someone gets to it in 24 hours or so, people won't be pulling their subscriptions. The only real thing that would cause people to bail is a large player database rollback, so days to weeks of playing are lost. However if you have a good database cluster, this isn't going to happen.

Virtualization is just one of many IT tools. Sometimes it is an excellent thing to have. Other times, there isn't any real need to have it, especially for CPU intensive stuff on a server that can be cloned or easily reimaged with the apps on it.

[1]: I'm assuming zone servers handle the combat mechanics, only sending updates to the core player database when a player loots an item, dies, logs out, disconnects, or at a periodic interval if nothing else changes.

Re:Virtualization

[TheRealFixer]

6: Ability to run on future hardware. Say everyone ditches x86 and amd64 and decides to go to IBM's POWER architecture and emulate legacy stuff. The stuff in the VM won't care that is is actually isn't running on a different CPU.

This is not true. Hardware virtualization is not emulation, which is what you're talking about here. Processes in the VM are run directly on the host processor, they're just managed by the hypervisor. There's no emulation layer, since that would make performance pretty atrocious. So, the stuff in the VM absolutely will care about what processor you've moved to, especially if you've suddenly changed instruction sets. Binaries compiled for x86 won't magically run on PowerPC just because it's running on a VM.

Dell ad?

[Degro]

This game failed in big part because of their extremely poor server performance. Who cares how they did it?

Re:Dell ad?

[Vrallis]

This game failed in big part because of their extremely poor server performance. Who cares how they did it?

Well, both poor server performance (Fortress battles were completely unplayable when I quit) plus they opened WAY too many servers at the start. If they'd started with 1/3 as many servers the game would probably be in far better shape today. The server transfers that they opened up as a last-ditch effort prior to merging servers was the straw that broke the camel's back for me, destroying my server's population.

WTB a fantasy (okay, I'll say it...WoW-like) MMO modeled after EVE's economy, industry, PVP, territory control, etc.

Re:70 * 16 ? Are they kidding me?

[SCPRedMage]

That's PHYSICAL servers. A single game "server" is made of multiple PHYSICAL servers.

Re:WTF is Warhammer Online?

[jhoegl]

It is impossible for Mac users to nerd rage.
Mac users would have to be nerds for that statement to qualify.

Re:WTF is Warhammer Online?

[glwtta]

It is impossible for Mac users to nerd rage. Mac users would have to be nerds for that statement to qualify.

So... douche-rage?