Slashdot Mirror
The Problem of Shards, Servers, and Queues In MMOs
An editorial at GamesIndustry takes a look at a couple of problems many MMOs have failed to solve as the genre has evolved over the last decade: log-in queues and a split player base. The most recent example is Aion, which launched in Europe and North America a few weeks ago. Players on some of the game's servers had to deal with lengthy queues until enough people left the starting areas and spread throughout the game. To NCSoft's credit, the queues are mostly gone already, and it wasn't simply launching with too few servers that was the problem (nor was simply launching more servers a perfect solution, as Warhammer proved). In fact, several servers had no queues at all, but many players had set their sights on the more popular ones — a problem facing other MMOs as well. At this point, it becomes a matter of programming — how can the developers for these MMOs build the networking aspect of the game such that more hardware can easily be allocated when it's needed, and also make it easier for people to play together without the restriction of different shards or servers? EVE Online has done well with a single game universe, but it's not clear how far that model can scale upwards.
Had one "shared world". It was initially restricted to US/EU/Korea or some such, but you could change your territory, at first X number of times, and later on as often as you wanted.
Champions Online has also launched with a single world architecture. Each zone has multiple instances, dividing the population in dozens of copies of each region, across which players can freely move if they wish to do so. Zones with friends, supergroup (guild members), and party members have priority, of course. These instances CAN fill, but if they do--just get your friends and all go to a new one.
http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000002990 Just FYI
Make it $ based... After a certain number, start charging extra to get on more popular servers....
This is a computational problem that the major game studies are hopelessly under prepared to solve.
Mostly they hire people who get degrees in game design that include very little in the way of computer science. This is actually a fairly difficult problem to solve.
The fundamental design flaw they all have is that servers represent space in the game, it's a flawed assumption about the best model to use.
The question of scale for an MMO applies to more than just the ability of the servers to host an increasing number of simultaneous players in a single virtual world. It's also about gameplay, and the MMO paradox: the more massive the world, the less important each player. I would argue that one of the factors in WoW's enduring success is that Blizzard knew when to add new servers not purely for performance reasons, but also to keep the number of players in any particular server at a particular sweet spot.
Too few players and there's no sense of a living, persistent world; too many players and that world is stifling and uninviting.
Actually, it will be interesting to see how things play out with Sony's MAG -- an action game that sits somewhere between classic multiplayer and MMO scale.
Your mind is clear / The things that you fear / Will fade with how much you / Believe what you hear
typo: EVE online has done well with a single universe game, not single game universe.
DAoC is now, for most intents and purposes, one server. The cluster is called Ywain, and I think it goes from Ywain-1 to Ywain-9. Each server shares its RvR (realm versus realm) areas with the other servers, so all Ywain players go to the same RvR instance. The main cities do as well. However, outside and in dungeons, the servers are independent. This is to keep 1,000 people from showing up in the same area.
You can change the server in main cities and other important areas by means of an NPC.
It's a nice way to do things. If Ywain-1 is too populated, go to Ywain-2. It lets the player decide how full or empty they want their experience to be.
Job? I don't have time to get a job! Who will sit around and bitch about being broke and unemployed then?
...manages to run a single world instance - it does raytraced graphics in real time, the fees aren't too bad but I'm not sure how to respawn.
The reason queues are such an issue is because the game "world" is layed out linearly. I.e. everyone does the level 1-10 zone, then moves to the 11-20 zone, etc. If each zone had a range of content for all levels, when the game launches you would be able to split the player base evenly across each zone (assuming each zone was on it's own box) without the need to throttle so heavily, rather than the current system of "limit the number of people online until X% amount have dinged level 10 and gone to the next zone, then let the next lot in".
Regarding EVE Online, the single-shard thing works because each solar system is it's own instance (you can't traverse the space between the solar systems and arrive at the next one) - when you use a star gate (or if in a big enough ship, your own build in jump drives) you are moved to a different instance which could be located on any number of the hundreds of blades CCP has for EVE. Usually the only time large amounts of players congregate together are in the big trading hubs (because they are lazy and won't sell the items in the system 60 seconds away) or during huuuuuuuuge space battles where usually everyone is killed by mass Titan AoE before the servers have a chance to crash.
Getting a single-shard working for a "traditional" - i.e. on one planet - MMOG would either require an enormous amount of spread-out content to allow hundreds of thousands of people to play in the same world at once, or heavy instancing like Guild Wars (please no "Guild Wars isn't a proper MMO arguments"). And if you didn't go for the instancing route, then I hope to god you don't have "kill xxx amount of mobs" in the starting areas. Having 20-50 people after the same mob was bad enough. 2000-5000? *shudders*
One of the major problems facing worlds such as Everquest is the fact that it's been wonderfully expanded to accommodate the huge player base - and then they've all left. What you're left with is a huge sprawling world with a few players rattling around in one corner of it. Login queues certainly aren't an issue at that point.
You know how sometimes when a popular game is released, you can't get it at the store because they're sold out? Do that for MMOs. Basically, limit the number of people who can start playing at launch, and let them eagerly await when their copy will arrive in the store, rather than letting everyone start playing immediately and be disappointed by having to wait 3 hours to log in.
Makes me think of Diablo, there's heavy instancing for you.
If you want a world massive enough to scatter players without instancing, you're probably looking at procedurally generated content. Though people tend to flock together due to social factors anyway. Even if you had an infinite amount of "level 1-10" places, I'm betting a good half of your players would end up in the same spot and then complain about lag. And as they level up they'll become even more integrated and homogenous, unless perhaps you try to segregate them based on choices like class, guild or other game choices you can be reasonably sure will vary among players.
I have been interested in Project Darkstar ( http://www.projectdarkstar.com ) for a while now. It's a middleware solution written in Java. It seems to be quite usable but the one "big point" is still not implemented afaik: Network Load Balancing. But might be worthwhile to remember it if you are trying to take on such an enormous project.
As I'm not a CS Major or even a competent programmer (I failed Basic), my idea might not hold water but here goes: Why not go with a Load Balancing setup, just like any high availablilty website/service? You'd need a Master Controller (Root DNS anyone?) and it parcels out clients to each server based upon how much load that sever is under. To me it makes more sense to do away with the Server/Shard/Region system in favor of the Load Balanced setup as it should allow for a seamless addition of resources instead of the problems currently being seen. Of course, as GW (NCSoft) offers, you can ensure that guild-members (special groups) are placed on the same server but other then special membership groups like that, why?
Mod me up/Mod me down: I wont frown as I've no crown
Since the game world is essentially a 2-D surface, why not arrange the servers in a grid topology? Each server can talk directly to the servers north, south, west, and east of it. Long-range traffic can either be routed through the grid, or, depending on its urgency, sent through a central router that is one hop from every node (i.e. the nodes are connected both in grid and star formation). A front-end load balancer takes care of transferring players' TCP connections from one node to the next as they move in game-space. That's a total of six physical network links: four for the cardinal directions, one to the star hub, and one to the load balancer.
Depending on how busy any one tile of game-space might be, the node serving that tile could be anything from a dual-core to a four-way quad core. For sudden, extremely heavy use of a particular tile, a more powerful node in hot standby could be switched in to handle the spike -- you could have an array of such "super nodes" always in hot standby.
*insert Wailing Caverns joke*
Shop as usual. And avoid panic buying.
The real problem is the way MMOs bind your character to a server. If you could pick the server at login, you eliminate the immersion problems instancing seems to cause people, and you give people a way to self-select their way out of overloaded server situations. It's still instancing/sharding, but without the restrictions MMOs feel they need to impose these days. Then again, you wouldn't get to rake in $25 every time someone wants to play with their friends on another server.
Because everyone has to start from the starting area. And it's kind of hard for a server to handle 100K+ players. Even if the server technically could, the game wouldn't be playable because it would be too crowded.
I can suggest two solutions, both with their flaws. They all allow single worlds to happen though.
* Make the world big. Really big. Big enough to have dozens of starting areas (and as much end-game areas, as the problem will move to those areas after a while). Problem is it costs more to develop and it's fragmenting the community in various areas.
* Let the servers dynamically add instances to crowded areas with an easy way for people to switch to join their friends. For example a party invite to someone on a different instance will "magically" move the players on a single instance (one that isn't already full). Here there is also a fragmentation happening but everyone will experience more or less the same game.
It seems like the better way would be to develop and deploy the MMO in an environment much like super computers and their applications are. Connecting servers with ultrahighspeed interconnects like InfiniBand (or now maybe even 10Gb ethernet would suffice) to create one large fabric of processing - not individual hotspot servers. This would allow the environment to dedicate more processors and memory to hotspots and not have idle servers laying around waiting for a group of players to explore that part of the map.
It seems a little ridiculous in this age of virtualization and distributed computing that they couldn't have come up with something a little better.
Perhaps it's time to start looking at using an RDBMS for such a contraption. They've been bypassed in the past for such because their response time is not predictable and/or consistent enough for most real-time activities. (That's not the same as being "slow" I would note.) However, if the complexities of the other ways are causing timing problems anyhow, then maybe the RDBMS's downsides are no longer as large in comparison.
Table-ized A.I.
Most of the Problems are sort of being handled in SecondLife by working with a "Grid"
1Each sim/region has its "stuff" running on a single computer (for full level regions Homesteads and OpenSpace regions are "stacked")
2 Player inventory is on a separate set of servers
3 communications is also on a separate set of servers (in fact i think text and voice are also separate)
Of course this is not all working perfectly but it does scale since SL has a running average of about 50K-70K logged in accounts and most of the time
server updates are the only times that a sim will be down for more than 5 minutes.
Any person using FTFY or editing my postings agrees to a US$50.00 charge
Has none of you seen 'The Dark Crystal' for details on how to solve that ?
Aren't there any 'real-time' databases out there ? I find it hard to believe, but there ya go...
For the launch of a game, why do there need to be starting areas at all? Why not just let the entire world be the "starting" area?
Rather than sticking with fixed stats for enemies, most areas could start out at a base level and have the stats for enemies ramp up based on players' average levels for the server.
To make it work long-term, different areas could scale faster (or slower) than others so that there would still be challenges for newer players.
Have the game "roll" the server you're gonna play on when you create your character... you don't control what server you'll get on so there will be no server with too many players on it compared to deserted servers... the only thing that sucks is to try to reach your friends, you have to get some server switching items to do that...
What you talk about is already done, to a certain extent, in that a given MMO "server" is often not a single computer but rather a whole cluster of them all working on different parts.
The real problem you run in to is too many people in one area. This creates a few problems that have nothing to do with server architecture, and no solution in it:
1) Client graphics. Every character in an area is another object the client's system has to render. Even if you keep the models fairly simple, you start overloading systems. The GPU is asked to push more triangles than it can handle and slows down. Also, people want shiny games so just having low detail models isn't a good answer. As such you can't just have areas with tens of thousands of people in them, because it'll cause even the most powerful systems to choke, and many people play games on something far less than the most powerful system.
2) Network bandwidth. This is again largely a client problem, but also a server problem. Your server must inform the clients of what all the other clients are doing. Tell them where they are moving to, what relevant actions they are taking, etc. Now the good news is that this can generally be pretty minimal information. Ok fine however, even a minimal amount of information is a problem when multiplied thousands of times over. You run in to the problem that you start exceeding lower bandwidth connections. Also it is a problem on the server end. Since the server has to send all updates to all people, the load isn't a linear thing. As an example say each additional client in an area needs 500 bits/sec more data sent. Means if you have 100 people you are sending each of them 48kb/sec so 480kb/sec total. 1000 people? 480kb/sec each, 468mb/sec total. 10,000? 4.7mbits/sec each, 46gbits/sec total.
3) Space and stuff to do. If everyone is crowded in the same area, it is hard for anyone to do anything. Suppose the first intro quest is something simple like go kill 10 bugs. Ok, that gets harder if there are 10,000 other people all trying to kill the bugs too. Just trying to expand the space the bugs are in or number of bugs or respawn rate doesn't help, it just makes things more confusing.
What it comes down to is you hit all kinds of limits in terms of productively having lots of people in one area. You can have lots of people on one server, that is no problem as the logical server can be made up of many physical machines to scale with the load. The problem is that the people won't evenly distribute. They'll cluster in areas and that introduces other gameplay issues that more hardware doesn't solve.
We designed a "peer-to-peer" MMO many years ago, although I have to say we didn't implement it and the devil is definitely in the implementation. Anyway, you can read the design docs here. After it was clear we weren't going to write it, I published the docs just to give a priority date (1998) to invalidate any stupid patents ...
Rich.
libguestfs - tools for accessing and modifying virtual machine disk images
This article is primarily about load, but the idea of single game-world has to fundamentally change the game system. In many MMOs, such as WoW, RuneScape and so on, there would be a gaming problem. Even with small world populations it's a minor issue.
Fundamentally it's the suspension of disbelief that would be required.
For example. Across WoW there might be one guild trying to complete some über quest at a time. However, if you compressed all US worlds, there would be hundreds of guilds doing it simultaneously. Now I'm just me, but it would strike me as just slightly odd if 300 people all handed in Onyxia's dead body to Stormwind one after another.
In EVE Online, nobody is a hero. Everyone's one cog that can be a 'Butterfly Effect'. The problem with most modern MMOs is that 99.9% of the time it's lots of people playing the same RPG. Not everyone contributing in some way to an overall plot. Ahn'Qiraj is the only time WoW got near that and as http://www.youtube.com/watch?v=71sVv__DryA perfectly demonstrates, that was frankly rubbish.
Just do one server with threads running on a SSI on top of your cluster of nodes interconnected with low-latency links.
That way you have one server that makes the best usage of your hardware.
It's very misleading to talk about an "EVE 45k single server population". It's obvious that EVE uses multiple machines to manage the single universe, transitioning you between machines during gate jumps. This only works out as a single universe as long as everyone doesn't pile into a single system (which is hosted on a single machine).
Popular systems in EVE, such as the infamous JITA, can be just as laggy as an overpopulated AQ opening. Also, if EVE had 1/2 of WoW's total online playerbase, the chance of busy systems causing lag would be much higher.
These games all operate in a world with the same laws of physics.
Your computer is only so fast. It can only draw so many other players on-screen at once.
You're network is only so fast. It can only handle network updates for so many nearby players at once.
A single server machine is only so fast. It can only handle simulation and network for a limited number of players at once.
All these games have designs to try to limit situations so the above factors are not a problem. Their designs may be different, but the issues are the same.
Graphics rendering has the same exact problems It's an n^2 problem to do raytracing each light has to cast a ray to each object you can apply the techiquies to make ray tracing faster, like space partition, limits on how many rays, and making computaiton parallel these concepts also work when mmo players interact
I'm not technically inclined enough to talk about how this would work, but why not combine phasing with the way the real world works?
That is you don't really notice people untill you interact with them or have to notice them.
The biggest problem is as stated in a previous comment, you either have this insanely massive world like a football stadium full of people, which leads to all it's related problems: or you have a nice crowd of people like a bar you can interact and mingle with...
So while you're playing you see a set amount of people to make you feel like you are living in a game world, but the game only shows you people and those people's actions that it detects you are interacting with in some way... be it chat trading items, killing kobolds.
Why do I need to see 20 people killing kobolds?
And if I can't loot their kobolds why do I need to see them?
If I can't see their kobolds or see them then their should be enough kobolds for me. Which makes me feel in my game brain that I personally was sent out to save elwynn forest from the kobolds (and take a lot of candles :)
The balance of such a thing and the way to do it. I have no idea. How do you mix it so you can still run into a random warlock taking some candles and be best friends for the next 79 levels? I'm not really sure.
Just an idea
"In fact, several servers had no queues at all, but many players had set their sights on the more popular ones"
ALL EU servers had long queues. Most of them was so long, you couldn't even get IN the queue.
The NA forums reported some server queues, but not on all servers.
Quite the opposite on EU forums, where people were going mental on the boards, complaining about massive queues and no response from NCSoft what so ever.
- Don't do what I do, it's probably not healthy nor safe. -
I'm gonna say no on that. Eve only works because the combat is relatively slow and automatic compared to other games. Imagine playing with 10 second to 10 minute lag on skill activation in WoW.
Why not... Sounds feasible.
Or you could stick with the hi/lo-lvl Mob-Zones, but make them available to all player-lvls. Granted, that would only work with some hi-lvl char pulling a newbie through rougher waters, but that could be turned into a kind of mentoring system. Instead of punishing "kill-stealing", reward "assisting". Plus, this could be used as a new distribution method. Why would you want to let walmart and others have their cut of your game? Remember the hype around the gmail-invitations? To play the game, you have to go and find someone who is willing to give you a copy of the Install-CD (=invite; after all, piracy is a non-issue for subscription based games, so take advantage from people willing to distribute your software instead of paying for it.) When you sign up, that person will become your 'liege'. Your starting area will be your lieges hometown. He will be your mentor and main quest-giver.* It's up to him if he does so by pulling you quickly through hi-lvl areas, or sending you out alone on some erradns in a lo-lvl area. To cap the number of invitations, each vasall costs a certain amount of gold per day. In exchange, you're oblieged to support your lieges raiding partys. Add a system for mutany, and you have an intresting new system.
* Either by making up his own quests as "Hunt me 10 boars for tonights feast and I'll give you an item I've outgrown" or by assigning or hinting to availaibe stock quests "Why not go to Fooshire and see my old friend Generic NPC. I owe him a favour and he asked me to send someone out to hunt down the rats in his cellar"
bickerdyke
Afterthought:
With "averaging" you'll run into the problem that the average mob is too weak for 50% of the players and way too dangerous for the other 50%. You'd have to find a way to either even that out on player-side (as in finding a mechanism for hi- and lo-lvl chars to play together) or on the mob-side. (Areas with below and areas with above average enemies. Thus assigning "new" starting areas, but at least dynamicaly)
bickerdyke
Eve Online works because they have some serious travel time still built in. This is on top of of being deep in 0.0 (i.e. no guards) space that must be traveled in groups or with great care.
Jita, one star system, is often very crowded because it's a hub for the auction house -- you can auction house from anywhere in that region, but the stuff you must pick up from its particular space station. So people conglomerate in Jita.
And, while Eve has a way to issue courier contracts to shlep stuff for you, then you have to wait from hours to days to get it shipped to local. The more cash you wanna hemorrhage, the faster, of course!
Champions Online has "one" server, but they go way out on the instances of common areas. They're typically 50, or 100 for Millenium City, before they're locked out and you must join a different instance. They're still public, just not over-populated.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
wow, I never thought i would see what i think is one of the most innovative mmo ideas on /. . I'm really quite impressed. forgive me if i ever make/design an mmo and you see these features in it. mea culpa.
Strengths of multiple instances with one giant server:
1) If there is a jerk griefing you by killing the boss you want over and over, you just move to another instance and the problem is solved. Grief killing is one of the oldest things in MMORPGS dating back to muds, and this instancing helps solve your most basic kind of griefing.
2) You meet more friends. If you find someone you like, you can tend to be in their zone more because you flagged them as friends. A bigger cycle of people = More chances you'll meet a new friend
3) Since it isn't small instances like Guild Wars, you get to meet a lot of people instead of being sheltered from them, so it overcomes the typical gripe with instances.
There were some other reasons it is so great too, but I forget them right now. If I was to do a MMORPG, I'd do it this way. It has many strengths and no real weaknesses. I have a max level on the game, and max level is sort of boring, but I had a lot of fun playing it mostly because I found a fun friend to group with.
God spoke to me.
Best way to solve this issue is to acutally focus on server tech while your building your freeken game.
that or crush up leprocans and use em as liquid cooling for your server.
Im betting thats why we cant find em becuse blizzard uses em all.