Slashdot Mirror

← Back to Stories (view on slashdot.org)

The Problem of Shards, Servers, and Queues In MMOs

Posted by Soulskill on from the you-are-two-thousandth-in-line-to-read-this-story dept.

An editorial at GamesIndustry takes a look at a couple of problems many MMOs have failed to solve as the genre has evolved over the last decade: log-in queues and a split player base. The most recent example is Aion, which launched in Europe and North America a few weeks ago. Players on some of the game's servers had to deal with lengthy queues until enough people left the starting areas and spread throughout the game. To NCSoft's credit, the queues are mostly gone already, and it wasn't simply launching with too few servers that was the problem (nor was simply launching more servers a perfect solution, as Warhammer proved). In fact, several servers had no queues at all, but many players had set their sights on the more popular ones — a problem facing other MMOs as well. At this point, it becomes a matter of programming — how can the developers for these MMOs build the networking aspect of the game such that more hardware can easily be allocated when it's needed, and also make it easier for people to play together without the restriction of different shards or servers? EVE Online has done well with a single game universe, but it's not clear how far that model can scale upwards.

29 of 253 comments (clear)

  1. Champions did it too by Locklear93 · · Score: 5, Informative

    Champions Online has also launched with a single world architecture. Each zone has multiple instances, dividing the population in dozens of copies of each region, across which players can freely move if they wish to do so. Zones with friends, supergroup (guild members), and party members have priority, of course. These instances CAN fill, but if they do--just get your friends and all go to a new one.

    1. Re:Champions did it too by Locklear93 · · Score: 3, Interesting

      I fail to see how multiple servers is better at all. With an architecture like Champions', if the people with whom you want to play aren't in your zone, you go to them. With traditional server separation, such as WoW's or Aion's, your only hope is that server transfers or the like are allowed. I'm also not sure that it complicates things in any significant way. What's complicated about clicking "Change instance," and looking at the top of the list--which is where the instances in which you have friends, teammates, and guildmates will appear? The confusion of "I'm standing right by [landmark here] and I don't see you!" doesn't really last long.

    2. Re:Champions did it too by Nylathotep · · Score: 4, Insightful

      the complication is its impossible to meet people because they are scattered amongst a hundred shards. Multiply it by the different zones and you never see the same people. Add to it the drop in drop out nature of grouping and the social aspect suffers. Having seperate servers like WoW have their own issues with having friends who play but are not on your server, but at least the people who are on the server you see day in and out.

      I'm kinda stalled at 29 because of the group instances, but I don't know people. I don't even know where to go for lfg grouping (ala IF). The built in lfg tool seems to be ignored, to the point that when I tried it, trying to send a message to the people in the list showed noone was online. Maybe they were on an alt, I don't know. You would think the interface would send the message to the currently online character.

      I suppose I could join a random supergroup recruiting but I like to actually meet people before joining a group.

    3. Re:Champions did it too by theantipop · · Score: 3, Interesting

      As the above poster describes, it's the social aspect. I appreciate the camaraderie being part of a well defined realm brings. And adding hundreds of personalities to a friends list isn't really the same thing. The most prescient example I can think of was the gate opening event for Ahn'Qiraj in classic WoW. Realms competed to be the first to achieve a collective effort of quests in order to open their gate first. My realm was neck and neck with another to be the world's first, resulting in one of the most exciting atmospheres I've ever been a part of in a game.

    4. Re:Champions did it too by Blakey+Rat · · Score: 3, Interesting

      Ditto. It's really annoying, from my experience with Tabula Rasa (which uses the same schema.)

      "Hey where are you?"
      "City A."
      "I'm in City A too, meet near the bank?"
      "I'm standing in the bank."
      "Oh, I am too but I don't see you..."
      "Which City A are you in?"
      "City A3"
      "Oh damn, I'm in A6, let me find a portal so I can move from City A back into the other City A."

      There are few ways to break immersion quicker than that.

      --
      Comment of the year
    5. Re:Champions did it too by Grail · · Score: 3, Informative

      Strictly speaking, the server maximum for EVE Online is about 1000-2000. Each star system is a different server, with travel between systems happening through a "loading screen" cunningly disguised as a star gate.

      Effectively, each star system in EVE Online is a "continent" from WoW, with each "zone" in WoW scaling to a "grid" in EVE. Except that "grid" is a local construct, not a fixed geographic location: you can join the same "zone" as another player by simply walking over a line in the sand, to join the same "grid" as a player you actually have to warp to their ship.

    6. Re:Champions did it too by cloricus · · Score: 4, Insightful

      Close, though a little inaccurate.

      The solar maximum is still unknown for EVE as Jita has yet to push it to its limits. Systems on reinforced 'supernodes' can handle 1,500+ players with little to no lag at all. Those same servers have been able to sustain hours of fleet combat involving over two thousand players with acceptable lag (10 seconds to 2-10 minutes on module activation). Normal nodes that haven't been reinforced can handle limited fleet combat of about 700 pilots before lag sets in and 1,000-1,200 with no combat. A normal node will also host a handful of solar systems.

      In regards to each solar system being like a continent that is a conceptual divide. WoWs 4 continents are still considered to be a contiguous 'world' as users are all acting towards the common goals of the world and interacting with the same player set. So EVE has 5,300 odd continents in its world compared to 4. Also note that you can 'walk over' the line between two grids by simply pointing your ship in the direction of another grid and flying to it. Like in WoW you also have the option to warp between grids, this would be comparable to flying a gryphon there. Grids are better thought of as a dynamic draw distance of the world around you.

      Excluding technical differences like these the defining factor of EVE is that your actions effect over 300,000 players in the only (heh) instance of this 'world'.

      --
      I ate your fish.
  2. Eve online runs Windows Server by DAldredge · · Score: 4, Interesting

    http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000002990 Just FYI

    1. Re:Eve online runs Windows Server by sopssa · · Score: 5, Funny

      Dear Sir or Madam,

      The responsible Anti-Microsoft Troll that should have replied to this post by now is on sick leave and was unable to prepare a custom flaming reply to this particular post. In lieu of that, attached is our generic template which we use to write all our flaming responses.

      1. Make a general anti-Microsoft jab
      2. Blame Microsoft for it's stance against Free Software (and also for lack of network neutrality, the current state of patent laws, the Iraq war, and the extinction of the dinosaurs)
      3. Accuse the poster who wrote something positive about Microsoft of being either a fanboy or a Microsoft employee. If the poster in question made a comment about Microsoft's actual support of Free Software in a particular instance, accuse the poster of being an oblivious idiot unable to see through their Embrace-Extend-Extinguish approach
      4. State that the Linux revolution is inevitable
      5. Finish off with another outpour of flames

      We hope you will be able to infer the potential content of the post that should have been done by the respective Troll. Please accept our apologies.

      Sincerely,

      Assistant Secretary,
      Anti-Microsoft Trolling Association, Ltd.

    2. Re:Eve online runs Windows Server by DAldredge · · Score: 5, Funny

      Sorry, I'm new.

    3. Re:Eve online runs Windows Server by Anonymous Coward · · Score: 4, Funny

      So THAT is why they need one hour of downtime every day...

  3. Computational Problem by phantomcircuit · · Score: 4, Insightful

    This is a computational problem that the major game studies are hopelessly under prepared to solve.

    Mostly they hire people who get degrees in game design that include very little in the way of computer science. This is actually a fairly difficult problem to solve.

    The fundamental design flaw they all have is that servers represent space in the game, it's a flawed assumption about the best model to use.

    1. Re:Computational Problem by Matthew+Weigel · · Score: 4, Informative

      Heh, no. I'm an MMOG server programmer, and I know a fair number of others, and a lot of us have backgrounds in distributed computing, with plenty spending time in academia before being lured into games. That game companies mostly hire people with game design degrees is a falsehood propagated by the institutions that offer those degrees. For one thing, there simply aren't enough people going through those schools to feed the industry's need for fresh meat; for another, the quality of programmers fresh out of any school is generally insufficient.

      As for fundamental design flaws... eh. I've heard plenty of that kind of talk from (for example) Project Darkstar; it's easy to say that, it's a lot harder to actually do the research to understand all the options and the inherent flaws in each. Interestingly, even EVE Online - lauded for its one-world approach - uses geographic decomposition too, and it works just fine most of the time (and now they've got a system for dedicating special hardware to the corner cases).

      --
      --Matthew
    2. Re:Computational Problem by Bat+Country · · Score: 4, Interesting

      This fellow was inadvertently correct. Representing space (volume) by putting sections of it onto single computers is a bad idea. Inevitably, no matter how good your design or how well-ordered your content is, some areas are going to become more popular than others. Hence, you're going to get congestion.

      A much better model is representing player (and non-player) actions as work units then distributing them evenly across a network of linked computers then getting an integrated result for each "region" (zone, map, city, whatever) each server frame. Make the server frames something like 50 frames per second and have player actions lag about 2-3 frames behind server-side action and you'll see little delay on the client machines but help mitigate potential race conditions between player actions (both players simultaneously attacking and reporting that they attacked on server frame 2,348,342 and both score a fatal blow on the other).

      To mitigate player lag you can distribute update packets based on the density of the update vs the distance of the events from the player vs the player's average data rate.

      Of course, that's just my two cents.

      --
      The land shall stone them with the bread of his son.
    3. Re:Computational Problem by Nefarious+Wheel · · Score: 3, Interesting

      The fundamental design flaw they all have is that servers represent space in the game, it's a flawed assumption about the best model to use.

      I'll bite, what's the best model to use?

      Whether it was meant that way or not, that's a good point; I would wonder if it wasn't servers that represented space in the game (in terms of your computational point of view) but rather database connections - because when you think about it, you're really just moving data structures around during game play.

      All the clever graphics runs in the memory of the gamers' PCs; details about in-game items and their status (i.e. item stats, equipped, in the backpack, where in the backpack, their state of repair etc.) just live in little pages that move up and down the link between gamer and "gamespace".

      True, there's a lot of item contention involved (who killed what mob, rolling on loot etc.) but I wonder if the true question of managing parallel game spaces and making them work isn't embedded in the replication of tables from one database instance to another. This is well established technique for most databases. That could imply the need for very fast interprocessor communications, presumably because warlocks consume rowlocks.

      The most complex software I've been into deeply enough to notice was VMS, some years ago - and that appeared to be a case where clever data structures did nearly all the work (that and REI of course ;) and good data structures meant a lot less algorithm. It's an approach. To recap, I think it's easier to think of "game space" as a database issue, rather than a processor one. Your Beowulf cluster of hot grits just provides CPU cycles, really, and that's not quite as difficult to share.

      --
      Do not mock my vision of impractical footwear
    4. Re:Computational Problem by inio · · Score: 3, Interesting

      You're close to right. There's been a lot of work on completely non-spatial distributed server architectures (mostly using distributed hash tables and occasionally multicast between servers) but they don't scale as well as expected. There's a reason for spatial partitioning - things close to each other tend to interact more.

      Fixed partitioning is the easy way to do spatial partioning - you drop boundaries and migrate objects that cross them. Often you use design to allow interaction between partitions to be ignored, so you don't need any communication between servers besides migration. Dynamic partitioning would largely solve this problem (as, generally, you're never actually in a region where you can interact with more than a several dozen other players) but is HARD if you want consistency.

      Guaranteeing instantaneous consistency is impossible - you've got hundreds to thousands of servers and keeping them all in lockstep at 30Hz with a potential for same-frame interaction between objects just isn't going to happen. Instead, what if you thought of the game as a simulation of abstract moore machines. Every frame each object looks at the state of of the wold and then sets it's state for the next frame and maybe creates other objects. Inter-object events can be modeled as objects that exist for a single frame and the receiver looks for. This means no instantaneous inter-object communication, but that's generally acceptable and likely unavoidable.

      Now, network latency between servers could still be a problem: as the system grows it's impossible to keep it in lock-step even with the slightly relaxed communication requirements. This can be solved by employing a technique used in distributed databases: eventual consistency. Lets explicitly allow inconsistent state to exist between servers instantaneously, but guaranty that it will be eventually be resolved. Lets have objects use a subscription model for their observations and send those subscriptions to all servers that might contain matching objects. When a server sees an object matching a remote subscription it sends over the object with enough state to run a dumb predictor (that can't look at the dynamic state of any other objects). The server with the subscribing object can then use that state and predictor to keep the object in sync and then service the actual subscription. Back on the server with the object that mached the subscription, it makes a local copy of the proxy it sent. It updates the copy along the the rest of the work, and compares it with the real object to detect when the remote server made an incorrect prediction. In this event, it sends an update to the subscribing server with the new state.

      Now you should say, "Wait! That update won't arrive until the server had already run the simulation for that frame!". Yep, you're probably right. However, because there's this subscription data available, the server can very efficiently re-run the simulation, touching only objects that might have detected the change (and any consequential changes). This might cause further updates to other servers, but because of generally sparse nature of interactions in games, the state on all servers for a given frame will quickly be consistent.

      This was almost my thesis before I got pulled into graphics and wii remotes. If this interests you and you'd like to see an early paper about it, drop me a line: [my_username]@[my_username].org.

  4. It's not just technical scale by StreetStealth · · Score: 3, Interesting

    The question of scale for an MMO applies to more than just the ability of the servers to host an increasing number of simultaneous players in a single virtual world. It's also about gameplay, and the MMO paradox: the more massive the world, the less important each player. I would argue that one of the factors in WoW's enduring success is that Blizzard knew when to add new servers not purely for performance reasons, but also to keep the number of players in any particular server at a particular sweet spot.

    Too few players and there's no sense of a living, persistent world; too many players and that world is stifling and uninviting.

    Actually, it will be interesting to see how things play out with Sony's MAG -- an action game that sits somewhere between classic multiplayer and MMO scale.

    --
    Your mind is clear / The things that you fear / Will fade with how much you / Believe what you hear
    1. Re:It's not just technical scale by Rakshasa+Taisab · · Score: 3, Insightful

      Too few players and there's no sense of a living, persistent world; too many players and that world is stifling and uninviting.

      It will only be stifling and uninviting if you force/encourage everyone to go to the same places and dungeons. In EVE there's regularly 50k players online at peak hours, and the central market system has up to 1300 players. The game play is such that you do not feel yourself 'crowded in', while still making it clear that you're one amongst many.

      You might think it's a bad thing, yet that's looking at it from the WoW perspective were 'everyone is a hero'. In EVE there are only a few real hero's, yet those that are well known have done something of true significance in the game. It's not just a fake hero feeling from playing a game that was designed to make everyone achieve the same thing.

      --
      - These characters were randomly selected.
  5. Nature Online by Bazman · · Score: 5, Funny

    ...manages to run a single world instance - it does raytraced graphics in real time, the fees aren't too bad but I'm not sure how to respawn.

    1. Re:Nature Online by Locklear93 · · Score: 5, Funny

      Respawning is easy. It's not coming back as a level 1 dung beetle that's hard.

    2. Re:Nature Online by Cloud+K · · Score: 4, Funny

      Ah, no, you combine two characters roughly around level 30 (although some elite players manage it at 13, against the rules in most zones) and spawn a new avatar from that. Curiously, Slashdot players often neglect to work on that particular skill set and often fizzle the Matefinding spell.

      The new avatar does start at level 1 though and you are normally expected to mentor them until at least level 18. And you don't get to play the new avatar yourself - character death is permanent and linked to your account. However there are rumours that your account may be ported to a secret project when your character dies, either on the good or evil side depending on your actions during the original game.

      I also find it quite difficult to get vendors to buy spider legs and pieces of rotting zombie skin from me in this world - it's a lot more focused on tradeskills as a way of earning money.

    3. Re:Nature Online by _Sprocket_ · · Score: 4, Insightful

      Yeah - but that system is riddled with bugs. People are always finding exploits. And the development team is unreachable. I mean, sure - there are those who claim to be community managers but I think most of them are con-artists and trolls. I don't think I've ever managed to get in touch with a real GM. I'm not even sure how their ticketing system really works.

      So sure - you have one example of a single world instance that's pretty popular. But it has so many flaws. And that is really driving a market for a different implementation. Otherwise, you wouldn't see all this competition trying to come up with alternatives.

    4. Re:Nature Online by BorgCopyeditor · · Score: 4, Funny

      Depends on your race, class, and faction.

      --
      Shop as usual. And avoid panic buying.
    5. Re:Nature Online by Phrogman · · Score: 5, Funny

      The Crafting system is definitely the best in any MMO I have seen. Literally anything can be made and sold if you can find other users who want to buy it, and if the item in question doesn't exist, the system is flexible enough to allow you to create it yourself. Some items are banned though, and there can be severe penalties for attempting to produce or sell them - although in some cases if you have secured a position with an in game guild (called a "Corporation" in Nature Online terminology, or a "Government" in some cases (although usually Corporation own Governments, its complex)), you can get away with it due to guilds have superior rights in game over individual player characters.
      Actually Nature Online is really quite cutting edge with regards to the player created content aspect of the game. Pretty much everything in game is player-created.
      The PvP system is quite evolved, can be highly stressful, and quite well supported. Perma-death does tend to decrease player willingness to participate in PvP however. The fact that PvP can theoretically break out anywhere at any time does lend a certain frisson to the game for some people, although often the likelihood of such impromptu PvP, or even dueling (which has gone out of fashion amongst players) can be determined by the zone you are in.
      Oh, yeah, zoning. The Zoning system is perfect, there is literally no noticeable difference when you zone from one area to another.
      Communications can be a problem. There are great communications tools available (although strangely no universal guild chat of any kind), but while spatial/local chat is free, virtually every other type of chat costs the player "money" (the term for credits/gold in the game).
      Its quite common for players to complain about the maintenance periods in an MMO. Virtually every MMO requires some downtime to do maintenance on the system, but players often find it extremely annoying that they can't play during this period. NO has solved this with a unique approach - each player character regularly undergoes a downtime for maintenance - often as much as 8 hours per day - but the whole system remains running at all times. This asymmetrical approach to server maintenance means that each individual can choose the time for their downtime maintenance, and its even flexible enough to allow you to break the roughly 8 hours required up into smaller periods stretched out over the day (although this is not recommended) or even to skip nightly maintenance entirely - although this can end up in resulting in a longer maintenance period the following day, and if repeated too many times in a row can result in distorted graphics, lag and lowered response times in many areas of the game.
      Overall Nature Online is a superior title and one that many MMO players should investigate more (although virtually all of them have one of the free accounts you recieve if you are lucky enough to be invited to the game, many do not participate beyond the minimums required to get started). It does have a long learning curve. It does have permadeath (although as noted above some people believe you do get a new account at a later date, some people believe you end up in new secret projects that are perhaps in beta at the moment. Its all based on wild player speculations, so take it with a grain of salt). It does have the most unique and engaging new player creation system ever devised, which allows 2 players to create a new account that will be handed out to a random recipient. In fact this system is so much fun many players engage in it just for the enjoyment of the process - sometimes even to the degree that they ignore the actual requirement that both players cannot be of the same "gender" (an obscure classification of character sub-type that is one of the things you can't choose when you are spawned). If for no other reason I highly suggest you explore this subsystem of the game, it really is unrivaled by any other system in any other MMO, period. Its even better than PvP for most players!
      My personal ratings for Nature On

      --
      "The first time I got drunk, I got married. The second time I bought a chimpanzee, after that I stayed sober" Arian Seid
    6. Re:Nature Online by Opportunist · · Score: 5, Funny

      Respawn is tricky. Only one managed to do it so far, but it took him three days and he did it in godmode.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    7. Re:Nature Online by Opportunist · · Score: 3, Interesting

      NO is also the only game I know where the biggest flamewars (and even whole PvP events) revolve around whether and what game you get transfered to after permadeath. Curious. What other game do you know where there are players that spend a sizable portion their time pondering what to do when they stop playing?

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    8. Re:Nature Online by Pumpkin+Tuna · · Score: 4, Funny

      Graphics: 10/10 (The graphics are perfect, its hard to imagine any improvements that could be made)

      Speak for yourself. I ran into a glitch where the graphics were great for a while, but started fading around level 20. At first I thought it was my video card, because I could still see things nearby, but everything far away was blurry. By level 32 or I had to rate the graphics a 20/60. I finally dropped 2000 gold pieces on a permanent healing buff that fixed the problem. I'd give it a 20/20 now.

    9. Re:Nature Online by gmhowell · · Score: 3, Insightful

      Screens or it's fake.

      --
      Jesus was all right but his disciples were thick and ordinary. -John Lennon
  6. Why not go fully peer-to-peer? by Richard+W.M.+Jones · · Score: 4, Interesting

    We designed a "peer-to-peer" MMO many years ago, although I have to say we didn't implement it and the devil is definitely in the implementation. Anyway, you can read the design docs here. After it was clear we weren't going to write it, I published the docs just to give a priority date (1998) to invalidate any stupid patents ...

    Rich.

    --
    libguestfs - tools for accessing and modifying virtual machine disk images