Slashdot Mirror
Building Scaleable Middleware for MMORPGs
CowboyRobot writes "ACM Queue has an article exploring the challenges of developing a reliable platform for an MMORPG, specifically looking at Wish by Mutable Realms. From the article: 'A common scalability problem for distributed multiplayer games relates to managing distributed sets of objects... A player may not be a member of more than one guild, or a guild may have at most one level-5 mage (magician). In computing terms, implementing such behavior boils down to performing membership tests on sets of distributed objects.'"
That page is annoying. For anyone on low-bandwidth or Lynx connections, here is the printer/human-friendly page.
Background: 28/M/Bi-Sexual; Owner of a Linux company; MBA Harvard 2003; B.S. Comp Sci MIT 2000
The issue here is you are looking at MMORPGS. RPGs are traditionally turn based combat. I haven't seen many successful RPGs that required one to actually swing or fire their weapon manually.
There are two massively multiplayer games that feature realtime combat
* Planetside, which is an MMOFPS with RPG characteristics (levelling, improving your character by gaining extra implant slots and additional simultaneous skill sets)
* Neocron, which is an MMORPG/FPS (I may be wrong on this one, it was a while ago and I only played the offline trainer, which was supposed to simulate online play)
One of the biggest issues is lag; to reduce lag, which would get horrendous when there are many people in close promixity doing things, the client-side visual representation and simulation, and the server side simulation are never in sync with each other. The server is the final arbiter, but the client tries to the best of it's ability and available information to provide a visual representation of what is going on.
Planetside (and I assume Neocron) solves the lag issue by moving combat resolution for attacks to the attacker's client, and trusting the client's integrity. As a result, you can easily die 3-4 seconds after running behind cover; likewise you can run through an intersection, be in the clear on your end, yet be shot and killed 5 seconds later as someone sees you go through the gap 3 seconds earlier and shoots you.
- MaineCoon
Hunt your preferred prey at Aliens vs Predator MUD. Join the war at avpmud.com port 4000
Dunno, but I've been playing this Eve Online stuff and it seems pretty realtime to me. And it supports 7500 people at the same time with no lag... And I must say it's pretty fun to play too, if you like space rpg's ;)
Veni, Vidi, Velcro!
Avatar doesn't mean any particular "agent of god" powers in this message. He's using Avatar as the generic word for "the persona that a player plays as".
Looks like some people out there need to review what design patterns are.
from first result:
wtf?
The seti-at-home-type approach doesn't work here for several reasons:
First of all, they aren't implying that they're having scalability problems.
Second, on a MMORPG server, many of these calculations have to 1) take place instantaneously (latency of communication over the 'net precludes this) and securely 2) and operate on enormous datasets (i.e. databases containing world/player/object/etc information) that can't exactly be sent down some DSL line due to size and security reasons.
Third, you can never ever trust the client to make decisions that can adversely affect other clients or provide undue advantage.
-fren
"Where are we going, and why am I in this handbasket?"
Yep!
how does the game world map get distributed amongst the grid servers
/. angle - we support MySQL for persistence, among others. In fact, any RDBMS with JDBC should work. We also support Entity creation, modification, bandwidth optimization using an XML-based schema editor (written in Java, of course). So game programmers don't have to fiddle forever making lots of structs and trying to optimize dirty bitmasks for message delta optimization. This lets you get past the tedious stuff quickly and get to the game logic.
Dynamic ownership, distributed object-view model. Very similar to the system described in Queue. You would never maintain a complete unitary in-memory representation of a world - that sucks up too server juice.
I come from a CORBA background as well - what you see with all this kind of MMOG Middleware (Butterfly, Quazal, There.com, BigWorld) is a classic example of evolutionary convergent adaptation.
I forgot to add a standard
Da Blog
I really think computing power is less significant than the overall game development design when it comes to MMORPGs. After design, bandwidth becomes a factor, and only then is computing power a factor. The only exception I can think of would be requiring power for encryption/decryption.
The notion of parsing datasets for something like guild membership is really trivial. If you want to design a solid MMORPG, it's going to come down to how the world, objects and players are represented.
I continue to be in awe of the capabilities of games like Everquest and SWG. SOE has really created a very robust MMORPG technology -- it's hard for any other game developer to really say they have anything comparable when they can't demonstrate superior performance under the same conditions due to no other MMORPG having anywhere near the quantity of simultaneous players (as Everquest).
IMO, the client side of EQ is pretty straightforward. What makes the game special is the server side and how they manage to manipulate so many players and objects in real time. People complain that too many objects/players per "zone" can lag things down, and that is true, but I have yet to see a better implementation than Everquest. SWG has done away with the concept of "zones" to some degree, but basically, they seem to have implemented some client-side intelligence to indicate at which point additional graphics and information on objects in the distance should be loaded or reported. There are still "zones" in all these games. Some of them implement noticeable loading lags, and others don't.
My outside impression of the technical layout of Everquest is something like this... and I'd love anyone with more info/insight to correct me or elaborate further. I ASSume their system is made up of racks of servers, running Solaris I think. The have some low-level, propietary engine that manages the objects in the world, probably to a back-end database like Oracle. The reason for zones in EQ is that when you enter a new zone, you may actually be switching from one physical server to another. Not only do they have different servers for different shards/worlds, but different servers for different zones. When I see a system message such as, "North Karana, Velketors and Plane of Mischief going down for a brief update", I think that perhaps that's one server they're rebooting, which runs those particular zones. I suspect they stagger high-traffic zones with low-traffic zones on servers, and occasionally when the X number of zones managed by a single server have an unusually high amount of traffic/visitors, you get lag.
What's interesting about MMORPG game design is the balance between handling as much client-side as possible without creating security issues. If the server keeps track of players, NPCs and objects, it's much more difficult for someone to hack, or at least, logs are available to identify issues. The more client-side processing done, the more likely the game can be inappropriately manipulated.
When you take into account the amount of real-time data that goes back and forth, EQ (and SWG) are quite impressive. I don't think database/dataset issues are really the problem as being able to efficiently encapsulate, protect and send/receive the large amounts of data in the real-time world.
This is a typo in the article. It should say that some of the Mutable Realms staff don't have a lot of experience with C++. The authors of Ice have many years of experience with both C++ and Java.
Also, the philosophy of Ice is the exact opposite of the "golden hammer" anti-pattern. We fully recognize that there is no one-size-fits-all with respect to programming languages. That's why Ice works with C++, Java, PHP, and soon also with C#.
For Wish, C++, Java, and PHP are used. The client is pure C++, for performance reasons. The performance-critical parts of the server are also written in C++, the non performance-critical parts (or those there the performance bottleneck is the database, not the code) are written in Java. In additition to that, there are PHP scripts for account management and other administrative tools.
Flat files? Wow! I admire those people's courage :-)
For what it's worth, Ice and Wish use Berkeley DB (http://www.sleepycat.com). It's a DB with a small footprint, high performance, and good transaction support. Nice licensing conditions too -- you pay only if you want to keep your source code secret.
Cheers,
Michi.
It sounds to me like what the guy is saying is this: The map is divided up (distributed object-view) among servers and no one server ever has the entire map in-memory. If everybody in the game moves to one part of the world, you've got servers sitting there doing nothing...so they switch (dynamic ownership) to the part where everybody is to balance the load. As players move out of that area to other areas, some of the servers would "move" with them so to speak.
Of course, I could be totally wrong on my understanding of what Terazona does because I'm basing it solely on what I've read about it here on
World War 2 Online has sort-of beat that. It's a MMOG FPS, in essence.
Basically, there is a 64 unit "visibility limit". You're only ever told about 64 units max (and sometimes, due to oddities, less) player entities around you, prioritized among several criteria (distance, threat, minimum friend/foe allocations), etc.
It works fairly well, and the structure of the game is such that you have dozens of 30-60 player battles going on at all times, and can move anywhere around the map as you choose, realtime, either by slogging it on foot, driving, flying, or steering your ship. You can also jump from place to place but leave your equipment behind.
Best estimates put the peak server load at about 3-4000 players, with 500-1000 during the low tide, but the game runs 24/7 on a single arena.
The developers aren't swimming in money, but they're in the black and have recently turned up the data update rates to make it more smooth, so there's evidently some room in the budget for bits.
Disclaimer: I'm a day one player, from June 6 2001 on, aka Krenn, of the 1/16 Panzerdivision "Windhund".
I find their choice of RPC as the middleware layer surprising. I would imagine that a vast majority of events in a MMP game need to be passed on to a fairly large subset of players, for example, anytime someone moves, attacks, casts a spell, etc. the people visual or audio range of that player need to be informed. RPC is better suited to one-to-one interactions, not the one-to-many interactions we get in MMP games. It seems to me that a distributed publish/subscribe system like Elvin, SIENA or even Mercury (a pub/sub system specifically designed for MMP games!), or a LINDA-style shared memory system would be much more appropriate.
why does the author constantly mention CORBA? Perhaps I'm not familiar enough with the Java perspective, but I assumed CORBA/COM (etc) were basically dead (or dying) when it came to modern internet programming.
That's a fair question. Looking at the requirements, we needed middleware that is language- and platform-independent. That immediately rules out things such as COM, DCOM, RMI, and .NET. CORBA really was the only candidate. Also, all of us (the people who wrote Ice) had a lot of CORBA experience, and Ice borrows a lot of ideas from CORBA. (No, not everything in CORBA is bad ;-) So, I ended up contrasting things to CORBA because Ice is both similar and different to CORBA.
CORBA isn't exactly dead--people are still building industrial-strength systems with it. And, to be fair, CORBA is the most widely-used distribution technology to date. It's just that CORBA is ageing and that, after more than ten years of experience, we now know how to do better and avoid CORBA's mistakes.
The author and company were only familiar with CORBA.
No, not really. I have experience with Sun RPC, DCE, COM, DCOM, Web Services, and CORBA.
Distributed-objects are an older, more established technology. [ie proven]
There are a lot of existing systems, algorithms, ideas, programs, etc. to build a distributed-object system.
Yes, to are large extent. There is a lot of experience we have gained over the past ten years or so on how to build distributed systems that work.
It masks the network from the API (yes, this should be a CON).
That's an old topic that comes up periodically. The most-quoted paper on that topic is probably A Note on Distributed Computing. Personally, I disagree with many of the conclusions in that paper. True, I can't afford to forget that there is a network underneath my API because the semantics are truly different from the non-distributed case. But that does not strike me as a compelling reason to make a remote invocation substantially different from a local one.
Engineering effort is required to make sure clients can access the model with any langauge, and also to allow designers flexibility in the object model. Most of this should be cheaper with a web-services model.
I don't see where web services would be cheaper than any other language-agnostic platform. In fact, everything I have seen so far about web services is more complex than Ice (or CORBA). And web services are horribly inefficient. If you are interested, look for my postings on the topic in comp.object.corba with Deja.
The reason we have web services (or, more accurately, might have them some day) is a political one rather than a technical one: back in the hey-days of the distributed computing war between CORBA and DCOM, Microsoft were about to lose the battle. They weren't going to simply cede the battlefield to the OMG so, rather than fight a war they couldn't win, they simply created a new battlefield. Sadly, web services is about re-inventing the distributed object wheel all over again and, even more sadly, with even more mistakes and corners on the wheel than CORBA.
You don't get to throw around as many buzzwords.
Unfortunately, in some circles, that seems to be an important criterion in the decision making. Personally, I prefer to use things that work and have substance, instead of chasing after silver bullets :-)
There is duplicated code on both the client and server.
Huh? You haven't been listening to my friend Don Box by any chance? ;-) If that is what you are referring to, it's simply wrong (or certainly, it is wrong as Don presented it in his interview). Neither CORBA nor Ice require duplicated code on client and server--they are pure client-s
http://games.slashdot.org/games/04/02/29/1642223.s html
i d=8424235 about Planetside. The one I liked the most was this one http://games.slashdot.org/comments.pl?sid=98732&ci d=8426037 about an early UO player's play-by-play of conquering some ambushers.
There are several people who added comments about how they had to do some very complicated interactions to overcome others - http://games.slashdot.org/comments.pl?sid=98732&c
So at some point the dexterity/skill-players can find a decent game or two... IIRC, a recent ad for MS's Mythica says it will have real-time combat.
No affiliation with any of the above, I just enjoyed the post and comments.
8-PP