Building Scaleable Middleware for MMORPGs

CowboyRobot writes "ACM Queue has an article exploring the challenges of developing a reliable platform for an MMORPG, specifically looking at Wish by Mutable Realms. From the article: 'A common scalability problem for distributed multiplayer games relates to managing distributed sets of objects... A player may not be a member of more than one guild, or a guild may have at most one level-5 mage (magician). In computing terms, implementing such behavior boils down to performing membership tests on sets of distributed objects.'"

Comments

Scalability and joining guilds

[Space cowboy]

Surely this is a classic example of the Manager pattern. You have a bunch of objects [Avatar] (all alike, at least programmatically :-) who want to perform operations on other objects. If the system has a [GuildManager] class, then access to this distributed network of avatars can be forced through the choke-point of 'can this avatar join this guild'.

The trade-off in terms of scalability is in frequency versus computation. If the operation is commonplace (such as moving around), then a distributed system has a problem. If the operation is not commonplace (such as joining a guild!) then it's painless to use the 'choke' of a manager class to resolve any issues.

Even in the commonplace situation, I would have thought it useful to use overseer-objects whose job it is to remove the extra (unnecessary) information from the problem before trying to solve it... There's no need to care about the avatar in sector (-1000,-1000) if we're currently in sector (0,0)...

It's a cliche, but the rule is 'divide and conquer'. Screaming and leaping is a satisfactory, but usually fatal approach to problem solving, unless you're Kzin.

Simon

Re:Scalability and joining guilds

[t0qer]

Even in the commonplace situation, I would have thought it useful to use overseer-objects whose job it is to remove the extra (unnecessary) information from the problem before trying to solve it... There's no need to care about the avatar in sector (-1000,-1000) if we're currently in sector (0,0)...

Been there, seen that. Alternate Reality had a creature in place for doing just the thing you were describing above. Taken from the AR Faq.

The problem with this was that in the AR system each object was unique (except commodities like food, money, gems) and had a structure with attributes (like "spells" which were small intrepreted programs embedded in objects). Those data structures took up memory (16 bytes to about 64 bytes) and on an 8-bit 64K (or 48K) computer we had to limit the amount of items somehow. The way we came up with that was least artificial was to introduce a creature that would eat up objects when the object queue was getting full.

Re:Scalability and joining guilds

[humankind]

Even in the commonplace situation, I would have thought it useful to use overseer-objects

I know this is done to some degree in Everquest. There are NPCs in each zone that exist to augment existing zone-related, PC and NPC situations.

For example, in each zone in EQ, there's an invisible NPC called, "pain and suffering" which appears to inflict damage on a player in certain situations (falling or bleeding to death). I would imagine that similar objects exist to control the weather, which in many cases might signal the client to narrow a player's depth of view and receive less information on objects in the vicinity.

Re:Scalability and joining guilds

[MMaestro]

Indeed, digital 'trust' is something that cannot be easily handed to, let alone established. Not counting real life relationships, anybody who meets another person online immediately (should) distrust that person. In a sense, this is destroying the foundation of the internet (the sharing of information openly and freely) but it not only works, it is also necessary (sadly). Why? Simple.

When you play a game online, would you rather trust "HarryGoatDeezNtz" who has an absolutely offensive name but you've seenen play online and play well for over 3 months, "UnnamedNewbie(6)" who you've never seenen before and is asking how to play in the chat, or "KingofSpades" who is an absolute asshole with no skills but has been playing the game longer than anyone other than the developers?

The same thing is true with businesses trying to estabish themselves online.

Would you trust Microsoft's Windows which is virus/bug/hack/etc prone, Linux which would require you to hire a full time IT staff just to keep your servers and computers working, or Macs and have your staff constantly ask 'wheres the right mouse button?'

So the question remains? Do you trust the 'veteran' of the three? The 'pro' or take a chance and give the 'newbie' a try?

"Authors thought of everything".

[SharpFang]

Lots and lots of modern cRPG, online or not, have way less features than games like Nethack. Just think what you can do about a towel - wrap it around your head, dip it in a fountain, wipe your face. Lots and lots of options. Incredible richness. Seems it can be done after all...

Are there any online rogue/nethack clones out there?

Who needs 10,000 people in a zone?

[bartash]

They say:

Wish is the first Ultra Massive Multiplayer Online Role Playing Game (UMMORPG(TM)). "Ultra" means that Wish supports more than 10,000 simultanous players in a single, seamless world, without any zones or "shards".

In EQ you can have an effect on other characters in your zone (say a hundred people) but you can talk with all the other people on your server (thousands, maybe tens of thousands of people). This is a limitation, but in practical terms it works OK. I don't actually need to interact with more than a few people at a time.

Wish sacrifices

[Rexz]

You make major one major, major sacrifice for so many simultaenously players in Wish. Movement is point and click. It feels like you're playing a strategy game rather than living in a real world. Those of you complaining that you can't joust and dodge in today's MMOs will hate the stilted movement mechanism of Wish, where the path you take is left to a pathing routine.

Lacking confidence here...

[Fiz Ocelot]

I don't really get a warm fuzzy feeling of confidence after seeing things like this:

"At ZeroC we used Java because some of our development staff had little prior C++ experience..."

"...however, a few of us had previously built a commercial object request broker..."

"...designing and implementing middleware is difficult and costly, even with many years of experience. If you are looking for middleware, chances are that you will be better off buying it than building it."

Frankly, I'd feel rather uncomfortable using ICE 1.0 as middleware for my new MMORPG. Yes they could succeed and do a nice job, but that rarely happens especially in the world of MMOs where nearly all games are released way too early in beta form.

Re:MMORPGs need better real-time characteristics

[AuMatar]

I've played damn near every MMO out there. if one came out that made me aim and run around dodging, I wouldn't play it. Its not what I want from the genre. I've done twitch combat. I've done it for 20 years. It bores me. So you can hit the left arrow button faster than me. I don't give a damn. Currently MMOs test if you can use your skills and tactics to out think me. That is fun. And luckily, it seem the genre will stay like this for some time to come.

Terazona

[meehawl]

one's control over one's character is not real-time.

If you've attended GDC then you may have played ZonaBattle, a real-time mechanized battlecar demo game for the Terazona MMOG system. Disclaimer: I work for Shanda Zona, the developer of this MMOG architecture, and my views may not represent those of the company, etcetera.

The purpose of the ZonaBattle technology demo is to illustrate that MMOGs do not have rely on sluggish, pseudo-turn-based gameplay. Using the right architecture produces excellent results.

ZonaBattle is not as fluid as some FPS games, but it is peppy and, unlike peer-based FPS games with~64-138 players, Terazona's client-server design enables you to scale the playfield to several tens of thousands of players and those players will experience no increased lag or message bottleneck.

Of course, you can also use Terazona to build "classic" seamless MMOGs. Terazona games do not have to have zones or "shards" and feature a heuristic, autoconfiguring grid system for game servers with dynamic region ownership, environmental simulation, and load balancing. You want more performance to support more players or more complex environment? Just slap in a few more commodity servers, or racks. The game will integrate them automatically and immediately begin dispersing Players and Entities among them.

Players can also exchange state with other local or non-local Entities using various bandwidth- and set-based configurable channels. This is not as easy as it might first appear.

Finally, the entire Server-side system is Java-based, for maximum flexibility and cross-platform support.

Truly a Dilbert moment.

[psoriac]

I came across this quote:
At ZeroC we used Java because some of our development staff had little prior C++ experience.

and immediately though of that Dilbert strip (sorry, no link) mocking the "if all you have is a hammer, every problem looks like a nail" saying. That strip was particularly memmorable to me because the last panel featured a porcupine saying "we must stick them with quills! it's the only way!"

Java Server-Side, Clients... whatever

[meehawl]

You lost me at the "java-based" though

And oh yeah, only the Servers run Java only. The Client-side API is language-agnostic and platform-agnostic. So you can write Clients in C++ or Java and compile them to Win32, XBox, PS2, GameCube. The Servers don't care which Client belongs to which platform.

The analogy I like to use is NTSC. In the early days of TV without NTSC you had no guarantee that your GE TVs would be able to pick up Motorola format broadcasts. TVs competed within closed markets and featured lock-in. Creating a common broadcast standard enabled all TVs to pick up all broadcasts. TVs could compete on quality anf fucntionality, and broadcasters could compete using content. Using a platform-agnostic MMOG Middleware lets you enjoy economies of scale because your Servers communicate with all kinds of Clients. Client experiences vary, of course, according to display resolution and frame rate ability.

Region Ownership

[meehawl]

I can't give you specifics because you have not signed an NDA. I can talk in generalities, but I can tell you that TZ's load balancing is not random. Much of the information about how to load-balance such systems can be obtained from reading "Distributing Object State" which ran in GamaSutra a while back. The key is dynamic ownership, not static ownership. As you rightly fear, static ownership leads to slowdown and player hangs.

The best way to maintain ownership would be dynamically, using some sticky heuristics to predictively anticipate where a player will "be" following a move, and alert Servers within some defined "neighborhood" or "ZOC" to update their state. This is non-trivial, because you may be dealing with non-Euclidean geometries, distance metrics, or set/guild membership. Therefore, each distributed Server can update its affected Clients on-demand, without those annoying lags you get with some systems when you can "feel" the Client loading the data from a new Server.

Alerting Servers that currently "own" those possible Regions to prepare to update relevent Entities with info is also required. If no Server owns that Region, then you should have a whole other set of heuristics to determine which game server should own that Region. It may, or it may not, be the Server that "owns" the Entity that is moving into that Region. You probably need to do cost-benefit calculations for assigning/re-assigning Region ownership. You can run Monte Carlo simulations to see how best to describe possible Entity "walks" within the topology.

Similarly, because of the expense of instantiation, you need some pretty tricky finagling to figure out when to relinquish ownership and purge any "ghost" copies of the Entity State that have been following the main Entity "around" within the topology. Of course, the nice thing is that Server-Server entity state exchanges will take place along a fat pipe backbone.

Interestingly, such systems end up looking a little like a Kohonen n-tier feedforward neural network.