Slashdot Mirror
The Problem of Shards, Servers, and Queues In MMOs
An editorial at GamesIndustry takes a look at a couple of problems many MMOs have failed to solve as the genre has evolved over the last decade: log-in queues and a split player base. The most recent example is Aion, which launched in Europe and North America a few weeks ago. Players on some of the game's servers had to deal with lengthy queues until enough people left the starting areas and spread throughout the game. To NCSoft's credit, the queues are mostly gone already, and it wasn't simply launching with too few servers that was the problem (nor was simply launching more servers a perfect solution, as Warhammer proved). In fact, several servers had no queues at all, but many players had set their sights on the more popular ones — a problem facing other MMOs as well. At this point, it becomes a matter of programming — how can the developers for these MMOs build the networking aspect of the game such that more hardware can easily be allocated when it's needed, and also make it easier for people to play together without the restriction of different shards or servers? EVE Online has done well with a single game universe, but it's not clear how far that model can scale upwards.
http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000002990 Just FYI
The question of scale for an MMO applies to more than just the ability of the servers to host an increasing number of simultaneous players in a single virtual world. It's also about gameplay, and the MMO paradox: the more massive the world, the less important each player. I would argue that one of the factors in WoW's enduring success is that Blizzard knew when to add new servers not purely for performance reasons, but also to keep the number of players in any particular server at a particular sweet spot.
Too few players and there's no sense of a living, persistent world; too many players and that world is stifling and uninviting.
Actually, it will be interesting to see how things play out with Sony's MAG -- an action game that sits somewhere between classic multiplayer and MMO scale.
Your mind is clear / The things that you fear / Will fade with how much you / Believe what you hear
I fail to see how multiple servers is better at all. With an architecture like Champions', if the people with whom you want to play aren't in your zone, you go to them. With traditional server separation, such as WoW's or Aion's, your only hope is that server transfers or the like are allowed. I'm also not sure that it complicates things in any significant way. What's complicated about clicking "Change instance," and looking at the top of the list--which is where the instances in which you have friends, teammates, and guildmates will appear? The confusion of "I'm standing right by [landmark here] and I don't see you!" doesn't really last long.
As the above poster describes, it's the social aspect. I appreciate the camaraderie being part of a well defined realm brings. And adding hundreds of personalities to a friends list isn't really the same thing. The most prescient example I can think of was the gate opening event for Ahn'Qiraj in classic WoW. Realms competed to be the first to achieve a collective effort of quests in order to open their gate first. My realm was neck and neck with another to be the world's first, resulting in one of the most exciting atmospheres I've ever been a part of in a game.
This fellow was inadvertently correct. Representing space (volume) by putting sections of it onto single computers is a bad idea. Inevitably, no matter how good your design or how well-ordered your content is, some areas are going to become more popular than others. Hence, you're going to get congestion.
A much better model is representing player (and non-player) actions as work units then distributing them evenly across a network of linked computers then getting an integrated result for each "region" (zone, map, city, whatever) each server frame. Make the server frames something like 50 frames per second and have player actions lag about 2-3 frames behind server-side action and you'll see little delay on the client machines but help mitigate potential race conditions between player actions (both players simultaneously attacking and reporting that they attacked on server frame 2,348,342 and both score a fatal blow on the other).
To mitigate player lag you can distribute update packets based on the density of the update vs the distance of the events from the player vs the player's average data rate.
Of course, that's just my two cents.
The land shall stone them with the bread of his son.
Ditto. It's really annoying, from my experience with Tabula Rasa (which uses the same schema.)
"Hey where are you?"
"City A."
"I'm in City A too, meet near the bank?"
"I'm standing in the bank."
"Oh, I am too but I don't see you..."
"Which City A are you in?"
"City A3"
"Oh damn, I'm in A6, let me find a portal so I can move from City A back into the other City A."
There are few ways to break immersion quicker than that.
Comment of the year
Whether it was meant that way or not, that's a good point; I would wonder if it wasn't servers that represented space in the game (in terms of your computational point of view) but rather database connections - because when you think about it, you're really just moving data structures around during game play.
All the clever graphics runs in the memory of the gamers' PCs; details about in-game items and their status (i.e. item stats, equipped, in the backpack, where in the backpack, their state of repair etc.) just live in little pages that move up and down the link between gamer and "gamespace".
True, there's a lot of item contention involved (who killed what mob, rolling on loot etc.) but I wonder if the true question of managing parallel game spaces and making them work isn't embedded in the replication of tables from one database instance to another. This is well established technique for most databases. That could imply the need for very fast interprocessor communications, presumably because warlocks consume rowlocks.
The most complex software I've been into deeply enough to notice was VMS, some years ago - and that appeared to be a case where clever data structures did nearly all the work (that and REI of course ;) and good data structures meant a lot less algorithm. It's an approach. To recap, I think it's easier to think of "game space" as a database issue, rather than a processor one. Your Beowulf cluster of hot grits just provides CPU cycles, really, and that's not quite as difficult to share.
Do not mock my vision of impractical footwear
NO is also the only game I know where the biggest flamewars (and even whole PvP events) revolve around whether and what game you get transfered to after permadeath. Curious. What other game do you know where there are players that spend a sizable portion their time pondering what to do when they stop playing?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
We designed a "peer-to-peer" MMO many years ago, although I have to say we didn't implement it and the devil is definitely in the implementation. Anyway, you can read the design docs here. After it was clear we weren't going to write it, I published the docs just to give a priority date (1998) to invalidate any stupid patents ...
Rich.
libguestfs - tools for accessing and modifying virtual machine disk images
You're close to right. There's been a lot of work on completely non-spatial distributed server architectures (mostly using distributed hash tables and occasionally multicast between servers) but they don't scale as well as expected. There's a reason for spatial partitioning - things close to each other tend to interact more.
Fixed partitioning is the easy way to do spatial partioning - you drop boundaries and migrate objects that cross them. Often you use design to allow interaction between partitions to be ignored, so you don't need any communication between servers besides migration. Dynamic partitioning would largely solve this problem (as, generally, you're never actually in a region where you can interact with more than a several dozen other players) but is HARD if you want consistency.
Guaranteeing instantaneous consistency is impossible - you've got hundreds to thousands of servers and keeping them all in lockstep at 30Hz with a potential for same-frame interaction between objects just isn't going to happen. Instead, what if you thought of the game as a simulation of abstract moore machines. Every frame each object looks at the state of of the wold and then sets it's state for the next frame and maybe creates other objects. Inter-object events can be modeled as objects that exist for a single frame and the receiver looks for. This means no instantaneous inter-object communication, but that's generally acceptable and likely unavoidable.
Now, network latency between servers could still be a problem: as the system grows it's impossible to keep it in lock-step even with the slightly relaxed communication requirements. This can be solved by employing a technique used in distributed databases: eventual consistency. Lets explicitly allow inconsistent state to exist between servers instantaneously, but guaranty that it will be eventually be resolved. Lets have objects use a subscription model for their observations and send those subscriptions to all servers that might contain matching objects. When a server sees an object matching a remote subscription it sends over the object with enough state to run a dumb predictor (that can't look at the dynamic state of any other objects). The server with the subscribing object can then use that state and predictor to keep the object in sync and then service the actual subscription. Back on the server with the object that mached the subscription, it makes a local copy of the proxy it sent. It updates the copy along the the rest of the work, and compares it with the real object to detect when the remote server made an incorrect prediction. In this event, it sends an update to the subscribing server with the new state.
Now you should say, "Wait! That update won't arrive until the server had already run the simulation for that frame!". Yep, you're probably right. However, because there's this subscription data available, the server can very efficiently re-run the simulation, touching only objects that might have detected the change (and any consequential changes). This might cause further updates to other servers, but because of generally sparse nature of interactions in games, the state on all servers for a given frame will quickly be consistent.
This was almost my thesis before I got pulled into graphics and wii remotes. If this interests you and you'd like to see an early paper about it, drop me a line: [my_username]@[my_username].org.