Server Structure in EVE Online

Massively takes an interesting look at the server model used by EVE Online. It's unusual for a MMOG because it doesn't divide the player load among different servers or "shards." Instead, the same cluster handles the entire EVE universe and all 300,000 subscribers (total; record concurrent load is around 40,000). The EVE Dev Blog recently announced some upgrades to keep things running smoothly and allow for battles involving over 1,000 ships. They call the technology StacklessIO.

Comments

Shards

[dintech]

In some respects shards can be a good thing. It's nice to have a 'clean sheet of paper' when a new server is brought on-line. With long running shards, the guys at the top of the food chain are very hard to catch up with.

Re:Shards

[nschubach]

Why do you need to catch up? Isn't the game fun without being the player with the most points?

That's my problem with MMOs. Too much concentration on the top players leaving a rather boring middle. It's kind of like society. The rich and famous are looked after, cared for, etc while the middle class has to foot the bill and pay for the servers while not getting half the game.

Re:Shards

[Greyfox]

X points being modified by your stats, so a guy with +5 implants and maxed stats will gain points faster than someone who doesn't.

EvE is also interesting in that the playing field is also relatively level. A fairly small gang of newbies can drop a 3 year player with a multi-billion ISK ship if they use good tactics. Compare with WoW where no number of low-level characters would have a chance against a 70 -- pretty much any character under 60 will be a 1-hit kill for the 70.

Also, the skills in EvE that "really matter" -- the ones pertaining to your ship's capabilities in combat, those skills are fairly limited. After around 15 million skill points or so you'll have maxed out anything. The 30 million skillpoint character may be able to manufacture stuff too, but the rest of his skill points won't help him in combat.

I really like their skill system and would like to see more MMOs adopt it. It does require patience, though, so I suppose it's not likely to be popular with most MMOs. The customers want their instant gratification.

Re:Shards

[RESPAWN]

Actaully, EVE is the first MMO that I've ever gotten into, and I think a lot of it is the skill system for me. I maybe get in 2 - 3 hours a day of good playtime, 4 days a week. So, about 8 - 12 hours a week. It's nice to know that, as long as I keep track of when my skills finish training, I can log in from work, start the new skill, log back out, and not be at all behind in skills as somebody else who started the same day as me.

IMO, EVE really gives a lot to the gamer who has to balance his gaming life with his work and social lives.

Re:Shards

[Inominate]

Chances are someone new will get squashed one-on-one in any ship you can fly. This applies from a frigate up to the most expensive faction battleship. This isn't a skill points thing, it applies equally if you ebay a character from the day eve was released.

One on one pvp in eve is pretty rare, and being able to survive long enough to even find someone can be a challenge. You have to know what ships you can take on. (being in a bigger ship doesn't mean you can win) Even then there's always the chance of a surprise.

This 6-9 month crap is uh, crap. Solo pvp can be done in a T1 frigate or cruiser. It's not easy but neither is doing it in ANY ship.

The real trick to solo pvp is a good ship fitting(whatever ship that may be), and the balls to go out and get blown up many times learning how to do it.

Re:Great!

[Itchyeyes]

While I personally don't find Eve the game to be very entertaining, I love to here any new information about it. That's partially because stories like this can only come out of a game like Eve, but also because it's pretty much the only MMO right now that's really doing things their own way and not just following in the footsteps of WoW and Everquest before it.

Performance upgrades are a must.

[Drakin020]

This was one of the largest reasons why I had left EVE. I was in a fairly large alliance. (SMASH) and I had partaken in my fair share of large battles, but we had one with around 400 people total in one system. The game was just in agony trying to run all that.

I actually saved an image of the fight.

http://s219.photobucket.com/albums/cc252/Drakin030/?action=view&current=MassiveFight.jpg

You will notice to the right, the list of players was cut at the top, the client started to bug out. Also during the battle you really had no control over what was going on. The speed was about a frame every 15-30 seconds. After each from you could be dead....Or another part of your HUD/Overview was missing.

It was battles like that in which I look forward to, but it was to the point where whoever hit the fire button first won, because if you got caught in the stream of lag before you enabled your guns, you would not fire a shot.

I'm glad to see that they are working hard on the performance, I just hope it's good enough to sustain at least a 400 man fight.

Re:Performance upgrades are a must.

[azemute]

In the last two weeks, they've addressed two major issues of EVE. First they implemented a newly written in-house technology that tore down many boundaries they were previously encountering as far as latency was concerned [They call it stacklessIO and it allowed them to run 1400 people in jita, with very little latency, where previously the maximum was approx 800] and secondly, a fully 64bit server process, allowing them to expand to 32GB of RAM for major nodes.
Effectively, they're knocking down as many lag-pins as they can, and I agree, hopefully with some luck.

The elephant's tail

[Moraelin]

That's my problem with MMOs. Too much concentration on the top players leaving a rather boring middle. It's kind of like society. The rich and famous are looked after, cared for, etc while the middle class has to foot the bill and pay for the servers while not getting half the game.

Heh. On WoW about 99% of the content is for levels 1 to 69, and all there is to do at level 70 is a repetitive grind to give you _something_ to do until the next expansion pack comes. It used to be the same at 60, before Burning Crusade.

The same applies to virtually any game out there.

E.g., on COH you still don't even _have_ a proper endgame grind, and it used to have none at all. (Unless you count the bad joke that the Hamidon "raid" used to be. Think: get 50 controllers to spam holds, and the rest of the gang does nothing at all to not get aggro.) Nor any perspective to get further than that, because the game had 50 levels since Issue 1 and just isn't supposed to ever have more than 50 levels.

E.g., on EQ2 much the same applies as on WoW. You hit level 80, you get stuck doing a dungeon 100 times, and then move on to the tier 2 grind and do it another 100 times, and somewhere past the point where you're bored out of your freaking mind, you finally get some of your epic gear.

E.g., on SWG, there didn't even use to be an endgame at all. The grind to Jedi was mostly repeatedly nuking your old "class" (ok, skill combination) and playing another "class" from zero to max. That was it: repeating the normal game again, with whichever class you got told to use this time. And then you got a Jedi to play the game with, from zero to max again.

Etc.

How the _fuck_ does that count as catering more to the top players? Those have 1% of the content, the low and middle guys get 99%.

On virtually any game out there, it's pretty much the illustration of this absurdist joke: "Q: Why does an elephant have a tail? A: So it doesn't end abruptly." That's it. What you get at top level is the elephant's tail, and the rest of the folks get the rest of the elephant.

But of course it doesn't prevent people from imagining that there's some grand and fabulous Shangri-La at the end. That they'll suddenly have 10x more fun, angels will give them blowjobs around the clock, heralds with trumpets will announce their every move, and that a thousand Blood Elf virgins will beg to have their baby. Basically, especially if they don't find the mid-game that much fun, that there must be _some_ grand reward at the end, or people wouldn't do it.

Boy, are they going to be disappointed. What people actually get at the end is... well, look at all the disgruntled WoW bashers and all the people swearing that <NEXT GAME> will bury WoW alive, and their whole raiding guild is swearing off WoW as soon as <NEXT GAME> comes out. That's how great they're catered for at the top.

It's still the same basic game, only more repetitive. If you find the middle boring, you're going to find the top even more so. Oh, it'll be something new to be in a raid for the first time. The second time too. But by the time you've went through it a dozen times, all using the same 1-2 buttons, it starts being a heck of a lot less fun. And if you still didn't get the hint that you've finished the game and might as well move on or start an alt, you're going to keep doing it more and more and enjoying it less and less. Well past the point where that enjoyment dropped below the "watching pait dry" mark. Until one more straw breaks the camels back. And then you join the great mass of burned-out, disgruntled ex-WoW-ers. Or ex-EQ2-ers. Or whatever.

And depending on the game, you might also get asked to respec your character into something you don't like. (See all the "but I wanted to be a Shadow spec priest" whines.) Or you might discover that your class isn't even needed by anyone at the top, at all. (See, again, everyone who wasn't a controller, in the Hamidon "raid" on COH.)

Did you actually play it to the top on those MMOs you have a problem with, or are you just judging them based on what you imagine about the top?

Re:The elephant's tail

[Jack9]

[I]n WoW about 99% of the content is for levels 1 to 69, and all there is to do at level 70 is a repetitive grind to give you _something_ to do until the next expansion pack comes. It used to be the same at 60, before Burning Crusade.

This is a common misconception. The content is for all levels. A common personality type wont use any content that doesn't measureably increase their usefulness at the highest tier. I submit you feel there is nothing gained from trying to solo Uldaman at level 70 or speed level or any of the non-traditional achievements that any person can imagine other than getting a sword that's +12 instead of +11. This personality type helped codify "a repetitive grind". In the asian markets, this does not exist (which is interesting from a sociological standpoint). If repetition was the problem with enjoyment, there would be a much higher suicide rate.

I have stated, many times, that the type of personality that does well with MMORPGs is not the exact same market that is assumed. Teenaged boys are tempermental, have shifting expectations, and are bored easily. You'll find the people that do well over years, are middle-aged and employed or otherwise fulfilled outside of game accomplishments.

What's worse, the content that BC came out with was LARGELY level 70 content. 1%? How do you come to that figure? The multiple subsections of zones devoted to 70-only (Skywing quests, comes immediately to mind) and specific holiday events. The entire zones geared for level 70s? No less than a dozen instances. Shattered Halls, Steam Vaults, Shadow Labryinth, Caverns of Time, The Arcatraz, Mount Hyjal, Serpentshrine Caverns, Tempest Keep, Black Temple, and EVERY Heroic version of the 5-mans? People see what they want to see.

You need to examine how you play the game, not make false criticisms based on an obvious personality quirk that prevents you from enjoying them more than one way. Yes, other people complain as well for the SAME EXACT REASONS (while there are millions who do not), but they complain because they are unable to be introspective, not because there's anything wrong with 'the original wheel'.

Used to be based on Stackless Python

[hargettp]

http://www.stackless.com/wiki/Applications

Don't know if that is still a core part of their technology; certainly makes sense, as lots of high-performance applications avoid creating lots of threads in order to scale--especially when there's IO involved. Thus, Stackless Python also takes the same approach to logical concurrency as Erlang.

"doesn't divide the player load"

[TopSpin]

Actually, players are divided among servers. Specifically, all players in a single solar system (just "system" in EVE lingo) are isolated to a single thread within a single stackless python process.

This design does not permit real multicore concurrency for a single system. Every player action in space is handled by a single thread. Stackless python provides "microthreads" which gives the illusion of concurrency, but it's really just cooperative multitasking. Lacking real concurrency, EVE can not scale beyond what can be processed in a single thread in a reasonable amount of time. Presently that amounts to about 1000 ships/players.

Microthreads provide none of the concurrency mechanisms (locks, CAS, etc.) that permit threads to safely share data. Thus, the stackless design of EVE can not be scaled using SMP. In my opinion this is a major design flaw of EVE.

The stackless microthread design was chosen for ease of development. Today, that choice plagues the game; it can't be fixed (to permit real concurrency) without a major refactoring of the server side game engine. If EVE will ever scale to what the players actually expect (by virtue of the fact that they don't hesitate to try 1000+ ship battles) then CCP needs to begin thinking about this refactoring; systems should be capable of leveraging multiple cores on demand. That means abandoning microthreads, which is the right decision, because the design imperative behind the choice of stackless (ease of development) is obsolete; half a decade later the new imperative is scalability.

The StacklessIO thing is an improvement to asynchronous IO that CCP deployed earlier this week. It's nice in that it enabled about 1000 players/ships to fight in a system with less "lag" then had been the case with only 500 or so ships. The 32 bit nodes will still crash when they run out of RAM, buy hey, it's an improvement.

Re:"doesn't divide the player load"

[niw]

Right right right right right... and wrong.

Great post, but I have one qualm. Microthreading/tasklet models are not explicitly incompatible with SMP systems. Theres nothing inherent to microthreads that would prevent, say, eight separate hardware threads all working in tandem to run a single solar system & its batch of microthreads.

Classical concurrency coordination mechanisms like mutexes and locks are not ideal for microthreading environments, so perhaps scaling out with these concurrency primitives might make abandoning stackless a logical step. Something like an actor model or a message passing scheme may however work extremely well in a stackless environment. Particularly if all you are doing is processing messages, you still want that ability to context switch extremely quickly.

My understanding is the main hold-back for concurrent Eve servers is things like guns firing at a ship thats already blown up, but the local thread doesnt know the ships blown up (not possible in their present non-concurrent environment). You can deal with this either by holding mutexes on the target, or you can use message passing and simply have the blown up ship send a message back saying "sorry, you cant shoot me, i already asploded," and then deal with that failure message in an async fashion.

If I were CCP I know what path I'd be taking. But thus far, CCP has focused largely on platform wins like asynch IO (although given the CPU usage their IO USED to be taking up perhaps that was in order) and proxy servers. They've systematically been ignoring the question of real concurrency.

Absolutely, the larger problem is the Python interpreter itself (CPython) and something called the Global Interpreter Lock (GIL). This lock prevents the real threading from having an actual impact and is the reason that CCP has not persued a OS threading.

CCP Lingorm

Actually this [one thread per system] is a limit of Python. Currently Python is bound to a Single Core (this is called the GIL, Global Interpreter Lock) and it is the centre of a large amount of heated discussion in the Python Community about removing it or keeping it.

IronPython has removed it but has no support for stackless at this point so we would still have to completely rewrite the threading system.

We are continuing to look into options to enable us to span multiple CPU cores and thus 'grow' our architecture. I am hoping to get some more details on this for posting Soon(TM).

Re:I was wondering about that...

[corsec67]

But who sees it? Just you?

The portrait is used:
In the eve-online.com forums, in the in-game chat rooms, and whenever you "get info" about a players ship.

Re:Great!

[Inominate]

That review is dead on accurate for solo type players. Still, it's intended to be funny which it succeeds at, even the most hardcore eve fanboy can admit it's mostly true. It just completely misses the good parts of eve.

Everyone should try eve once. There is about a 75% chance you'll loathe it with every fiber of your being and go out of your way to talk about how awful it is whenever it gets mentioned. However, for the people looking for something different in MMOs, eve can be fun on levels untouched by any other.

Can't distribute 1 sol over N blades = !scalable

[Morgaine]

The reason why EVE is lagging is because the architecture of a sol is not distributed at all. Each sol is a monolithic process.

Any given IBM blade can run any number of sols, but the fewest it can run is 1, not a fraction of 1. Second Life has this same problem, a non-scalable architecture in which several sims can be run on one CPU, but the fewest you can ever run is 1, and it's not possible for N different CPUs to carry the load of 1 sim. That places a limit on the activity within any 1 given sim, or sol here.

When any given sol can be run across N blades in EVE, then we'll have a decent scalable architecture, and not before.

Improving the I/O speed with Infiniband will help, sure, but it's not the answer to non-scalability of sol activity. That can only be done by total redesign of the sol process to make it distributable over an arbitrary number of blades.

Re:Great!

[Sobrique]

As a rabid fanboy, I agree entirely. He's spot on with all the points he's making - the PvE element of EVE gets dull in a hurry, and if you don't interact with other players ... well, yeah.

It's like elite, but worse.

What I find disappointing though, is that he fiddled around with the tip of the ice berg, proclaimed it too small and proceeded to sail on past. EVE is a game which _requires_ you be a self starter - that you go out and do stuff at your own behest. It's for that very simple reason that some people just don't get along with it - they are used to being told what to do, for questing, for getting loot, or ... well, whatever.

They fly missions for a bit - and whilst they _are_ getting better, they're not exactly the most enthralling thing in the world. Conclusion: EVE is dull, and they move on.

I don't actually think that's such a bad thing - EVE is not a game that appeals to every gamer. At a pretty fundamental level, it does involve being horrible to other players. What in other games would be 'griefing' in EVE is 'business as usual'. The kind of player who's not really thinking 'wow, a whole universe, what can I do?' won't get along anyway.

*shrug*. I play EVE a lot, and I like the freeform nature. Others won't.

The only thing that has me miffed by Zero Punctuation, is he took a massively multiplayer PvP game - didn't interact with anyone, and didn't PvP, and proclaimed it crap.

Re:Great!

[darkvizier]

I was a beta tester in Eve, and played the game when it first came out. The game had an amazing community before it was ever released, and there was a lot of anticipation around it. When it opened up, it was this huge world that was mostly empty, and a lot of players went solo, or in small corporations. It had much more of a wild west feeling, especially out in 0 sec. The economy was a grand experiment, and there were many creative ways to turn a profit.

I ended up quitting Eve because I felt that a lot of the things I enjoyed about it were being lost as the game matured. As large corporations and alliances moved into space, we lost the freedom and uncertainty of a lawless land. CCP considered many of the more "creative" ways of making money exploits, and banned players for using them, instead of fixing loopholes in the mechanics which were the root cause. In short, politics and heavy-handed policy enforcement killed the game for me.

Now I'm not saying they went the wrong direction with it. I think it was a natural maturing of the game which led to where it is now. But I do think they lost their most loyal player base by doing that, and made it more of a mainstream endeavor. While I can't fault them for wanting to expand their audience, and pull in some big revenue, personally, I wish they'd taken a different direction. In any case, I'm always interested to hear how "the experiment" is progressing, and I do think they've made a fresh contribution to the gaming industry. It will be interesting to see where their ideas evolve from here.