Building Scaleable Middleware for MMORPGs

CowboyRobot writes "ACM Queue has an article exploring the challenges of developing a reliable platform for an MMORPG, specifically looking at Wish by Mutable Realms. From the article: 'A common scalability problem for distributed multiplayer games relates to managing distributed sets of objects... A player may not be a member of more than one guild, or a guild may have at most one level-5 mage (magician). In computing terms, implementing such behavior boils down to performing membership tests on sets of distributed objects.'"

Comments

Scalability and joining guilds

[Space cowboy]

Surely this is a classic example of the Manager pattern. You have a bunch of objects [Avatar] (all alike, at least programmatically :-) who want to perform operations on other objects. If the system has a [GuildManager] class, then access to this distributed network of avatars can be forced through the choke-point of 'can this avatar join this guild'.

The trade-off in terms of scalability is in frequency versus computation. If the operation is commonplace (such as moving around), then a distributed system has a problem. If the operation is not commonplace (such as joining a guild!) then it's painless to use the 'choke' of a manager class to resolve any issues.

Even in the commonplace situation, I would have thought it useful to use overseer-objects whose job it is to remove the extra (unnecessary) information from the problem before trying to solve it... There's no need to care about the avatar in sector (-1000,-1000) if we're currently in sector (0,0)...

It's a cliche, but the rule is 'divide and conquer'. Screaming and leaping is a satisfactory, but usually fatal approach to problem solving, unless you're Kzin.

Simon

Re:Scalability and joining guilds

[t0qer]

Even in the commonplace situation, I would have thought it useful to use overseer-objects whose job it is to remove the extra (unnecessary) information from the problem before trying to solve it... There's no need to care about the avatar in sector (-1000,-1000) if we're currently in sector (0,0)...

Been there, seen that. Alternate Reality had a creature in place for doing just the thing you were describing above. Taken from the AR Faq.

The problem with this was that in the AR system each object was unique (except commodities like food, money, gems) and had a structure with attributes (like "spells" which were small intrepreted programs embedded in objects). Those data structures took up memory (16 bytes to about 64 bytes) and on an 8-bit 64K (or 48K) computer we had to limit the amount of items somehow. The way we came up with that was least artificial was to introduce a creature that would eat up objects when the object queue was getting full.

Re:Scalability and joining guilds

[humankind]

Even in the commonplace situation, I would have thought it useful to use overseer-objects

I know this is done to some degree in Everquest. There are NPCs in each zone that exist to augment existing zone-related, PC and NPC situations.

For example, in each zone in EQ, there's an invisible NPC called, "pain and suffering" which appears to inflict damage on a player in certain situations (falling or bleeding to death). I would imagine that similar objects exist to control the weather, which in many cases might signal the client to narrow a player's depth of view and receive less information on objects in the vicinity.

Re:Scalability and joining guilds

[MMaestro]

Indeed, digital 'trust' is something that cannot be easily handed to, let alone established. Not counting real life relationships, anybody who meets another person online immediately (should) distrust that person. In a sense, this is destroying the foundation of the internet (the sharing of information openly and freely) but it not only works, it is also necessary (sadly). Why? Simple.

When you play a game online, would you rather trust "HarryGoatDeezNtz" who has an absolutely offensive name but you've seenen play online and play well for over 3 months, "UnnamedNewbie(6)" who you've never seenen before and is asking how to play in the chat, or "KingofSpades" who is an absolute asshole with no skills but has been playing the game longer than anyone other than the developers?

The same thing is true with businesses trying to estabish themselves online.

Would you trust Microsoft's Windows which is virus/bug/hack/etc prone, Linux which would require you to hire a full time IT staff just to keep your servers and computers working, or Macs and have your staff constantly ask 'wheres the right mouse button?'

So the question remains? Do you trust the 'veteran' of the three? The 'pro' or take a chance and give the 'newbie' a try?

"Authors thought of everything".

[SharpFang]

Lots and lots of modern cRPG, online or not, have way less features than games like Nethack. Just think what you can do about a towel - wrap it around your head, dip it in a fountain, wipe your face. Lots and lots of options. Incredible richness. Seems it can be done after all...

Are there any online rogue/nethack clones out there?

Who needs 10,000 people in a zone?

[bartash]

They say:

Wish is the first Ultra Massive Multiplayer Online Role Playing Game (UMMORPG(TM)). "Ultra" means that Wish supports more than 10,000 simultanous players in a single, seamless world, without any zones or "shards".

In EQ you can have an effect on other characters in your zone (say a hundred people) but you can talk with all the other people on your server (thousands, maybe tens of thousands of people). This is a limitation, but in practical terms it works OK. I don't actually need to interact with more than a few people at a time.

Wish sacrifices

[Rexz]

You make major one major, major sacrifice for so many simultaenously players in Wish. Movement is point and click. It feels like you're playing a strategy game rather than living in a real world. Those of you complaining that you can't joust and dodge in today's MMOs will hate the stilted movement mechanism of Wish, where the path you take is left to a pathing routine.

Re:Wish sacrifices

[Anonymous Coward]

Everquest, where you can't even see anything around you? Fight a monster melee and you interact with it by clicking on its floating name, because it's out of your field of view? And totally impossible to get the tactical overview that comes with an overhead perspective. Everquest style is the equivalent of wearing blinders and having no neck, which isn't immersive at all, since you spend your time wondering where the hell that thing that's attacking you is.

If you want a decent interface look at Neverwinter Nights. The overhead point and click system is infinitely superior for tactics and strategy, moving through clautrophobic areas, and dealing with objects around you. But you can seamlessly switch to keyboard based third person control for immersive perspective and dodging in combat.

Lacking confidence here...

[Fiz Ocelot]

I don't really get a warm fuzzy feeling of confidence after seeing things like this:

"At ZeroC we used Java because some of our development staff had little prior C++ experience..."

"...however, a few of us had previously built a commercial object request broker..."

"...designing and implementing middleware is difficult and costly, even with many years of experience. If you are looking for middleware, chances are that you will be better off buying it than building it."

Frankly, I'd feel rather uncomfortable using ICE 1.0 as middleware for my new MMORPG. Yes they could succeed and do a nice job, but that rarely happens especially in the world of MMOs where nearly all games are released way too early in beta form.

Re:MMORPGs need better real-time characteristics

[AuMatar]

I've played damn near every MMO out there. if one came out that made me aim and run around dodging, I wouldn't play it. Its not what I want from the genre. I've done twitch combat. I've done it for 20 years. It bores me. So you can hit the left arrow button faster than me. I don't give a damn. Currently MMOs test if you can use your skills and tactics to out think me. That is fun. And luckily, it seem the genre will stay like this for some time to come.

Terazona

[meehawl]

one's control over one's character is not real-time.

If you've attended GDC then you may have played ZonaBattle, a real-time mechanized battlecar demo game for the Terazona MMOG system. Disclaimer: I work for Shanda Zona, the developer of this MMOG architecture, and my views may not represent those of the company, etcetera.

The purpose of the ZonaBattle technology demo is to illustrate that MMOGs do not have rely on sluggish, pseudo-turn-based gameplay. Using the right architecture produces excellent results.

ZonaBattle is not as fluid as some FPS games, but it is peppy and, unlike peer-based FPS games with~64-138 players, Terazona's client-server design enables you to scale the playfield to several tens of thousands of players and those players will experience no increased lag or message bottleneck.

Of course, you can also use Terazona to build "classic" seamless MMOGs. Terazona games do not have to have zones or "shards" and feature a heuristic, autoconfiguring grid system for game servers with dynamic region ownership, environmental simulation, and load balancing. You want more performance to support more players or more complex environment? Just slap in a few more commodity servers, or racks. The game will integrate them automatically and immediately begin dispersing Players and Entities among them.

Players can also exchange state with other local or non-local Entities using various bandwidth- and set-based configurable channels. This is not as easy as it might first appear.

Finally, the entire Server-side system is Java-based, for maximum flexibility and cross-platform support.

Re:Terazona

[Bryan Ischo]

I probably wasn't very clear when I asked my question. Or maybe my question didn't make any sense.

Maybe I could illustrate the problem I am wondering how Terazona solves, and you can tell me how you solve it (or at least give me a general idea) ...

The world includes models of objects which the player interacts with. Let's say a player is in a certain location and wants to walk north. But they can't because a tree is in the way. Whatever server is keeping track of the player's movements and doing collision detection or whatever, has to know that there was a tree there. So the server has to have a map of at least the part of the world that the player is currently in, to know what obstacles there are for the player to walk into.

What I'm wondering is, does every server have this entire map, so that it can, for any set of players distributed anywhere around the game world, keep track of what obstacles they are walking into? Or does each server only deal with a part of the map, and only keep track of the players that are in the part of the game world represented by the server's map?

The way that Terazona was described, I got the feeling that the servers divide the players up pretty much randomly, in which case each server would have to have a map of the whole world to be able to represent the area that any arbitrary player that they are managing is in.

Is that what it does? Or does each server keep track of a part of the map, and only those players that are in that part? If so, what does it do when there are many many people in its segment of the game world? And, how does the map get broken up by the servers? Automatically or does the game designer have to break the world up on behalf of the servers and assign servers to different quadrants of the world map?

Am I making any sense at al?!?

Truly a Dilbert moment.

[psoriac]

I came across this quote:
At ZeroC we used Java because some of our development staff had little prior C++ experience.

and immediately though of that Dilbert strip (sorry, no link) mocking the "if all you have is a hammer, every problem looks like a nail" saying. That strip was particularly memmorable to me because the last panel featured a porcupine saying "we must stick them with quills! it's the only way!"

Re:Do some of the work client-side...

[Peredur]

The problem with this is you cannot trust the end user. They will find a way to catch this information and use it to cheat. Everquest was plagued by this. Encryption is the only way to go if you are going to do this. However, you may find that you are spending more time finding a _easy_ encryption that is _hard_ to break.

Java Server-Side, Clients... whatever

[meehawl]

You lost me at the "java-based" though

And oh yeah, only the Servers run Java only. The Client-side API is language-agnostic and platform-agnostic. So you can write Clients in C++ or Java and compile them to Win32, XBox, PS2, GameCube. The Servers don't care which Client belongs to which platform.

The analogy I like to use is NTSC. In the early days of TV without NTSC you had no guarantee that your GE TVs would be able to pick up Motorola format broadcasts. TVs competed within closed markets and featured lock-in. Creating a common broadcast standard enabled all TVs to pick up all broadcasts. TVs could compete on quality anf fucntionality, and broadcasters could compete using content. Using a platform-agnostic MMOG Middleware lets you enjoy economies of scale because your Servers communicate with all kinds of Clients. Client experiences vary, of course, according to display resolution and frame rate ability.

Re:Scalability problem?

[Anonymous Coward]

Um, you most definitely do NOT want that data to be too local; that is, stored on client. That's asking for trouble. In no time hacky players will find a way to modify it locally, and get access to things they are not supposed to.

As to having it in multiple places (local caching); sure, but problem then is cache coherency. It's nothing too specific to such games, of course... it's rather common problem on any distributed system.

Re:MMORPGs need better real-time characteristics

[EvilTwinSkippy]

My answer is write a game that plays itself. Rather than be a contest between who can keep the most number of balls in the air (ala RTS), we should instead write a system where you code your own AI's, and let the computers to the dirty work.

It would be like a stock-market simulator. You send your moves in, and you can check your porfolio, but the game goes on no matter how much (or little) attention you pay.

My thought is that you set the game in a universe where it's only possible to send robot fleets everywhere. As the Lord High Stick In the Mud, you control the fleet for your planet. Inhabited worlds are off limits. (So if you space off and your fleet and colony are destroyed, you can always start over.) It will really be about who can devise the best automated strategy, because everyone's got to sleep sometime.

Anyone interested, my address is out there. TTFN.

Re:MMORPGs need better real-time characteristics

[dwpro]

I too wish that there was more strategy in the fighting for most mmorpgs...I play final fantasy XI, and there are elements of strategy(you have specific abilities that can be used in succession with other characters that do more damage, heal life, ect.) but for the most part it lacks what you are looking for. Also, such a heavy reliance on other party memebers is particularly frustrating sometimes, but that is another story altogether.

Region Ownership

[meehawl]

I can't give you specifics because you have not signed an NDA. I can talk in generalities, but I can tell you that TZ's load balancing is not random. Much of the information about how to load-balance such systems can be obtained from reading "Distributing Object State" which ran in GamaSutra a while back. The key is dynamic ownership, not static ownership. As you rightly fear, static ownership leads to slowdown and player hangs.

The best way to maintain ownership would be dynamically, using some sticky heuristics to predictively anticipate where a player will "be" following a move, and alert Servers within some defined "neighborhood" or "ZOC" to update their state. This is non-trivial, because you may be dealing with non-Euclidean geometries, distance metrics, or set/guild membership. Therefore, each distributed Server can update its affected Clients on-demand, without those annoying lags you get with some systems when you can "feel" the Client loading the data from a new Server.

Alerting Servers that currently "own" those possible Regions to prepare to update relevent Entities with info is also required. If no Server owns that Region, then you should have a whole other set of heuristics to determine which game server should own that Region. It may, or it may not, be the Server that "owns" the Entity that is moving into that Region. You probably need to do cost-benefit calculations for assigning/re-assigning Region ownership. You can run Monte Carlo simulations to see how best to describe possible Entity "walks" within the topology.

Similarly, because of the expense of instantiation, you need some pretty tricky finagling to figure out when to relinquish ownership and purge any "ghost" copies of the Entity State that have been following the main Entity "around" within the topology. Of course, the nice thing is that Server-Server entity state exchanges will take place along a fat pipe backbone.

Interestingly, such systems end up looking a little like a Kohonen n-tier feedforward neural network.

Has anyone used JavaSpaces for this?

[tcopeland]

JavaSpaces is more or less Sun's Jini take on distributed processing.

From a programmer point of view, you start up a "space", and then you can write objects in, take them out, and read them. And that's all. So there are a very few simple operations, and you structure your app around those.

Anyhow, it seems like a couple of JavaSpaces on a rack of servers might serve as a good way to distribute processing/notification/etc. Of course, you're limited to Java and to moving around Serializable objects....

Re:This project is going to fail

[Michi]

However the article is not about rolling your own ORB, it's about designing a MMORPG middleware, which can have little to nothing to do with an object broker.

Hmmm... To build a game on that scale, we needed a distribution platform of some kind. Having looked at what we needed, and having considered CORBA (which some of us knew a lot about, having previously built a commercial ORB), we knew that CORBA wasn't going to meet our requirements. So, we ended up building a new middleware.

It's quite surprising how much the middleware and its object model end up influencing game design and performance. For example, I don't think Wish could have been built without object migration, without some of the protocol features we came up with for efficient event distribution, or any number of other things. The game requirements influenced the design of the middleware, and vice-versa, so I would say that the two actually have a lot to do with each other.

In my experience this usually happens when the people involved don't really understand the problem they are trying to solve

You might want to have a look at my home page -- I think I can honestly state that I know quite a bit about middleware and the problems we are trying to solve :-)

Cheers,

Michi.