Slashdot Mirror
Server Structure in EVE Online
Massively takes an interesting look at the server model used by EVE Online. It's unusual for a MMOG because it doesn't divide the player load among different servers or "shards." Instead, the same cluster handles the entire EVE universe and all 300,000 subscribers (total; record concurrent load is around 40,000). The EVE Dev Blog recently announced some upgrades to keep things running smoothly and allow for battles involving over 1,000 ships. They call the technology StacklessIO.
1000 ships of boringness
When you shoot a mime, do you use a silencer?
In some respects shards can be a good thing. It's nice to have a 'clean sheet of paper' when a new server is brought on-line. With long running shards, the guys at the top of the food chain are very hard to catch up with.
This was one of the largest reasons why I had left EVE. I was in a fairly large alliance. (SMASH) and I had partaken in my fair share of large battles, but we had one with around 400 people total in one system. The game was just in agony trying to run all that.
I actually saved an image of the fight.
http://s219.photobucket.com/albums/cc252/Drakin030/?action=view¤t=MassiveFight.jpg
You will notice to the right, the list of players was cut at the top, the client started to bug out. Also during the battle you really had no control over what was going on. The speed was about a frame every 15-30 seconds. After each from you could be dead....Or another part of your HUD/Overview was missing.
It was battles like that in which I look forward to, but it was to the point where whoever hit the fire button first won, because if you got caught in the stream of lag before you enabled your guns, you would not fire a shot.
I'm glad to see that they are working hard on the performance, I just hope it's good enough to sustain at least a 400 man fight.
The greatest revenge in life is massive success.
Fortunate for the newbie, skill points don't count for too much. The bar to fly some quite useful ships such as interceptors, assault ships and the like is pretty low, and once you have those skills, you are just as good as that 4 year vetran who's got the same skills. The extra tens of millions of points that they have will let them excel in other ships and at other activities, but not all at the same time.
It is very easy to start fresh and carve yourself a niche and play with the 'big boys' in just a short while.
I was wondering what the point of setting up a character was if you never got out of your ship, so I figured at some point you got to level up far enough to meet people on their ships or something... but I should have expected that wouldn't happen. I got involved briefly in a space MUD, many years ago, and discovered that it was divided into people who flew around in their ship and did space stuff so they never got into the same room as anyone else and only talked with anyone in IM, and people who didn't have anything to do with the ship stuff and were getting run out of the game because they were wasting server time. Seeing as I'm a pretty dedicated interact with people type, as opposed to an arms length IM type, I didn't hang around to get run out...
So I guess they skipped the whole interacting bit... yes no?
quote from article: "However, this model affects a lot more than its running costs and complexity and may be practically required for any successful next-generation PvP based MMO."
"Could the single server approach become commonplace in the next generation of PvP-based MMO? I, for one, hope it does."
I played Eve for awhile, it is very cool how so many players in one place just change the approach of tasks which are common place in every MMO it seems. I would love for hardware to get a jump on software right now so we can have more games like this, but I think the only reason why it works well with Eve is the near unending world it has created. In a game like WoW, the ocean is the barrier, in Eve there is so much empty dead space that you can just chill in for hours and see nothing. I think if this was adapted for a land based game, the land would have to be even larger causing busy quests (travel from point a, to b, kill 'tree', travel from point b to c, turn in) would take forever to complete. I could see ways of this working but it is unlikely we will get to see it anytime soon.
To above comments:
There was a group that transferred from one MMO to EVE, and took down a high level faction as complete newbs. They did this by rushing one of there big ships, with more starting ships then they could target. They would then literally just keep doing 1 damage till they won.
Also one thing that has attracted so many to Eve, is that casual players have just as much chance as hardcore players to keep up. Since skills are RT, they are both gaining skills at the same rate in a way, allowing casual players to log on, have fun and then just train while they are at work (which a lot of WoW players do, using 3rd party software those cheaters!!! I don't hate you, I just hate the system we chose to live in...). At the same time it allows hardcore players to do what they want. A lot of people in Eve have taken roles such as; trying to take over sectors, assassin guilds, mercs, tradesmen, pirates, police, politics and even spies. Eve has created a wonderful system and we should build off there success in the near future.
~~Noname
Shoot Yourself In the Foot
Heh. On WoW about 99% of the content is for levels 1 to 69, and all there is to do at level 70 is a repetitive grind to give you _something_ to do until the next expansion pack comes. It used to be the same at 60, before Burning Crusade.
The same applies to virtually any game out there.
E.g., on COH you still don't even _have_ a proper endgame grind, and it used to have none at all. (Unless you count the bad joke that the Hamidon "raid" used to be. Think: get 50 controllers to spam holds, and the rest of the gang does nothing at all to not get aggro.) Nor any perspective to get further than that, because the game had 50 levels since Issue 1 and just isn't supposed to ever have more than 50 levels.
E.g., on EQ2 much the same applies as on WoW. You hit level 80, you get stuck doing a dungeon 100 times, and then move on to the tier 2 grind and do it another 100 times, and somewhere past the point where you're bored out of your freaking mind, you finally get some of your epic gear.
E.g., on SWG, there didn't even use to be an endgame at all. The grind to Jedi was mostly repeatedly nuking your old "class" (ok, skill combination) and playing another "class" from zero to max. That was it: repeating the normal game again, with whichever class you got told to use this time. And then you got a Jedi to play the game with, from zero to max again.
Etc.
How the _fuck_ does that count as catering more to the top players? Those have 1% of the content, the low and middle guys get 99%.
On virtually any game out there, it's pretty much the illustration of this absurdist joke: "Q: Why does an elephant have a tail? A: So it doesn't end abruptly." That's it. What you get at top level is the elephant's tail, and the rest of the folks get the rest of the elephant.
But of course it doesn't prevent people from imagining that there's some grand and fabulous Shangri-La at the end. That they'll suddenly have 10x more fun, angels will give them blowjobs around the clock, heralds with trumpets will announce their every move, and that a thousand Blood Elf virgins will beg to have their baby. Basically, especially if they don't find the mid-game that much fun, that there must be _some_ grand reward at the end, or people wouldn't do it.
Boy, are they going to be disappointed. What people actually get at the end is... well, look at all the disgruntled WoW bashers and all the people swearing that <NEXT GAME> will bury WoW alive, and their whole raiding guild is swearing off WoW as soon as <NEXT GAME> comes out. That's how great they're catered for at the top.
It's still the same basic game, only more repetitive. If you find the middle boring, you're going to find the top even more so. Oh, it'll be something new to be in a raid for the first time. The second time too. But by the time you've went through it a dozen times, all using the same 1-2 buttons, it starts being a heck of a lot less fun. And if you still didn't get the hint that you've finished the game and might as well move on or start an alt, you're going to keep doing it more and more and enjoying it less and less. Well past the point where that enjoyment dropped below the "watching pait dry" mark. Until one more straw breaks the camels back. And then you join the great mass of burned-out, disgruntled ex-WoW-ers. Or ex-EQ2-ers. Or whatever.
And depending on the game, you might also get asked to respec your character into something you don't like. (See all the "but I wanted to be a Shadow spec priest" whines.) Or you might discover that your class isn't even needed by anyone at the top, at all. (See, again, everyone who wasn't a controller, in the Hamidon "raid" on COH.)
Did you actually play it to the top on those MMOs you have a problem with, or are you just judging them based on what you imagine about the top?
A polar bear is a cartesian bear after a coordinate transform.
http://www.stackless.com/wiki/Applications
Don't know if that is still a core part of their technology; certainly makes sense, as lots of high-performance applications avoid creating lots of threads in order to scale--especially when there's IO involved. Thus, Stackless Python also takes the same approach to logical concurrency as Erlang.
Actually, players are divided among servers. Specifically, all players in a single solar system (just "system" in EVE lingo) are isolated to a single thread within a single stackless python process.
This design does not permit real multicore concurrency for a single system. Every player action in space is handled by a single thread. Stackless python provides "microthreads" which gives the illusion of concurrency, but it's really just cooperative multitasking. Lacking real concurrency, EVE can not scale beyond what can be processed in a single thread in a reasonable amount of time. Presently that amounts to about 1000 ships/players.
Microthreads provide none of the concurrency mechanisms (locks, CAS, etc.) that permit threads to safely share data. Thus, the stackless design of EVE can not be scaled using SMP. In my opinion this is a major design flaw of EVE.
The stackless microthread design was chosen for ease of development. Today, that choice plagues the game; it can't be fixed (to permit real concurrency) without a major refactoring of the server side game engine. If EVE will ever scale to what the players actually expect (by virtue of the fact that they don't hesitate to try 1000+ ship battles) then CCP needs to begin thinking about this refactoring; systems should be capable of leveraging multiple cores on demand. That means abandoning microthreads, which is the right decision, because the design imperative behind the choice of stackless (ease of development) is obsolete; half a decade later the new imperative is scalability.
The StacklessIO thing is an improvement to asynchronous IO that CCP deployed earlier this week. It's nice in that it enabled about 1000 players/ships to fight in a system with less "lag" then had been the case with only 500 or so ships. The 32 bit nodes will still crash when they run out of RAM, buy hey, it's an improvement.
Lurking at the bottom of the gravity well, getting old
hasn't ever played the game.
it's really fun until you start getting into fleet battles and get about 15 seconds per frame.. you actually find out who has gotten blown up over vent about 10 minutes before your client lets you know... if it doesn't crash
this is on a machine that smokes crysis..
eve is fun as hell single player/solo but no one at that company has a clue how to scale their tech.
Is the technology applicable in the business/database market? If so, without going into the proprietary tech, what bottleneck is it improving on?
As the article is extremely light on actual tech, as a business software dev I'm trying to figure out if these improvements would be applicable in an environment such as ours.
Would love to know more.
The reason why EVE is lagging is because the architecture of a sol is not distributed at all. Each sol is a monolithic process.
Any given IBM blade can run any number of sols, but the fewest it can run is 1, not a fraction of 1. Second Life has this same problem, a non-scalable architecture in which several sims can be run on one CPU, but the fewest you can ever run is 1, and it's not possible for N different CPUs to carry the load of 1 sim. That places a limit on the activity within any 1 given sim, or sol here.
When any given sol can be run across N blades in EVE, then we'll have a decent scalable architecture, and not before.
Improving the I/O speed with Infiniband will help, sure, but it's not the answer to non-scalability of sol activity. That can only be done by total redesign of the sol process to make it distributable over an arbitrary number of blades.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
Eve has two different "massive" fleet battles:
1.
Jump into system.
Black screen for 15 minutes.
Reconnect.
Oops! There went your ship and you clone!
2.
Jump into system
Black screen for 3 minutes.
System loads.
Hear over teamspeak how everyone is killing everyone and barking stuff but you can't see any of it.
System finally loads after 10 minutes and you see wrecks and other ships. You finally get to press F1-F8 to do something.
You wait a good bunch of minutes and hope the keystrokes go through.
Repeat until someone tells you over teamspeak you were dead 1h ago; you never found out because the node desynced as usual.
(note losses are permanent in eve and, frequently, sting a bit)
The main developers and designers, the ones with a clue, moved on to other projects. The ones that are left are, unfortunately, clueless and have no idea of what the implications of anything they do or propose are.
For example the last speed nerf, which whether you like it or not, it is clear that it would had to come along with both drone and missile changes at least. Which in effect affect most of all the ships in the game.
Whatever. The game is trying to appeal more and more to the carebear newbies so I don't really give that much of a damn that it has gone to the gutter.
In games like wow you can't really play with all of your guildmates, only the ones that are about the same as your level.
In eve it doesn't matter if you have 1million or 50 million skillpoints. Sure you won't be able to fly all the ships but you'll still be able to take part and be important in a gang.
Every time I try a new MMO instead of eve I find them all lacking in dept, complexity and most of all player controlled gamechanging events.
Once you get past the complexity( http://farm4.static.flickr.com/3088/2335016192_6003c39c4c.jpg/ ) it is a game unlike you've ever played!
While technically that's true:
1. It was there at a lower level too, hence the claim that only top-levels get catered for, is false.
2. So basically a variant of the common fanboy defense, "well, you could go and do <insert pointless thing> or do it again and again, and that ought to count as more hours the game has." It's been floating around about various games since at least the 90's.
I'm talking in hours. Take the time needed to do _all_ the quests and dungeons that don't need level 70 and/or epic gear. All the quests starting with hunting boars in the Orc newbie area _and_ wolves in Northshire, all the gathering cactus apples _and_ Milly's grapes, and all the instances starting with the Deadmines _and_ RFC. That total is your first number of hours. Then take that dozen instances and measure how much time you spend inside them from start to finish. Once each, because on the second run you don't see that much new content. There are a couple of exceptions, but not that much. I haven't done the exact maths, but as gut feelings go, I'll stick with that 1%.
Oh, spare me the bullshit armchair-shrink gig. You don't know anything about my age, personality, or how I play the game. Yes, I _am_ a middle aged guy _and_ employed _and_ have other accomplishments. And that is exactly _why_ I couldn't care less about a grinding to willy-wave about another piece of gear at the end. I have better things to willy-wave about than "yay, I finally got some virtual boots." Yes, I have RL accomplishments instead. And given that my whole point is that it's the road not the destination, and that I _don't_ want to grind for my top tier gear, you couldn't be more wrong anyway. But that won't stop you from doing... an ego-masturbation gig in which you pretend to know what's wrong with anyone whose tastes differ in the slightest from yours. Geeze, dude, do you even listen to yourself?
You presume to submit how _I_ feel... Heh... How about: I actually feel that I've already done Uldaman at the right level, and feel no need to solo it again at 70? It's still the same instance. I've already _seen_ those halls. But nah, that wouldn't be as good as a strawman that supports your preconceived conclusion, would it? Sorry about that.
Get over yourself. Some people can just have different styles without having some "personality problem" or being impatient kids. You're not the platinum stan
A polar bear is a cartesian bear after a coordinate transform.
But Infiniband isn't so much disk I/O, as an enabled for NUMA architecture in the cluster - it makes multiprocessing much easier, if you can use distributed shared memory.
They will most likely start breaking down the various 'non space' stuff, into processes that can be distributed. They can 'cheat' and do a node handover whenever someone warps to another grid too, but the place where the 'problem' hits, is when you have lots of ships interacting with each other - 500 people all maneuvering and shooting is much much harder to distribute effectively, because you do need to communicate a lot between the different clients. It stops being an implicitly parallel problem, and starts requiring interprocess and possibly inter-node communication, and it needs it to be done 'real time', which is the killer.
But ... well, it's something they're working on, and it's one of those 'big challenges' that I'm watching with interest as to how they'll solve it.
Trinity is pretty stable. Well, aside from the occasional SQL Server crashes and/or database corruption requiring sometimes hours of downtime to rebuild.
Yup. Quite stable.
I really have to think this whole thing was a slashvertisement, since the system structure has been pretty well known for a while. It's not as if they keep it a dark secret - they post plenty of updates at their website. I'm particularly suspicious of it being a /vert since it didn't point to Wikipedia which, last I checked (back in March or April) had all of this technical info already.
..that has a real underground economy, and drug-dealing cartel.
http://www.massively.com/2008/09/24/outlaws-of-eve-online-masudi/
Well I hope they don't end up trying to achieve distribution by volumetric subdivision, as we thoroughly analysed that approach elsewhere and it's a complete disaster.
Some other parameter of the computation is going to have to be partitioned, maybe simply the participants themselves without regard to location within the solar system. It'll undoubtedly require a large amount of inter-CPU communication, but Infiniband will help with that.
My initial stab at it would probably be to split participants randomly into subsets per CPU, then dynamically migrate them on the basis of interactions so that each CPU tends to handle mainly one or more sets of directly interacting parties. Separately from those, there's probably still going to have to be a single master sol process to act as master orchestrator, as distributed multi-body solutions are still a research area.
Like you, I'm interested to see how they do it. :-)
(...) It's not that the server lags out and stuff takes longer to happen, it's that it lags out in totally unpredictable ways. One person might find the server completely unresponsive for 30minutes and be able to do nothing but sit there and die. While another might have only a few seconds of extra delay. Who gets which is completely random.
The unpredictability of lag is precisely one of the things stacklessIO fixes, according to their tests. It seems the problem was somewhere in their network layer, not in the server itself. Just compare these 2 figures, which both measure lag over a couple hours on sunday in Jita:
Before:
http://staff.ccpgames.com/explorer/devblogs/images/delta_jetbyte.png
and after:
http://staff.ccpgames.com/explorer/devblogs/images/delta_stacklessIO.png
Their tests also say that the unpredictable lag only occurs in "extremely stressful lab conditions". Anyone believing CCP at their word at this point is flat out retarded. If it fixed it it does, but CCP saying they fixed it is as meaningful as if the CCCP said they did.
Dude, quit bringing up strawmen.
Dude, I've said _nothing_ about effectiveness of an individual or anything. I keep saying that what bores me is doing the same content again. But that somehow reinforces your belief to the contrary. _Can_ you read, or wtf?
Again, _can_ you read? I've said _nowhere_ that the game is badly designed.
A polar bear is a cartesian bear after a coordinate transform.
What they want to implement is a system where they can use multithreading to program a whole cluster, with threads being able to migrate from a node to another.
And that's exactly what a single system image provides, i.e. it's an operating system that gives the illusion that a cluster is a single computer with many processors and a lot of memory.
They'd better use an SSI that already exists and be one with it, rather than doing a complete rewrite.
Dude, let's take a break and put it in context.
The original claim, that I was answering to, was that somehow the top level guys are the only ones catered for, and the mid-game is left to be boring. That's what I was originally answering to.
And I'm saying that that's flat out not true. That
1. there is less content for you to see there. Look at the surface of those endgame dungeons, or hours to see each once, or whatever metric you wish to use. There is less of it. Yes, you can do it more than once, but then you could also do the Deadmines more than once if you want to. Just doing it more than once doesn't make it more content.
2. _if_ you didn't like whatever there was to do at mid level, I still don't think many people will get extatic about the same things at the end. You still do the same basic things, you know?
3. yes, most people _do_ eventually get bored of that repetition. It's not even as much a personal opinion. Last time I've seen a statistic, it said that the average player stays in a game for about 6 months. (Some more, some less, but that's how averages work.) That's a lot of bang per buck, no doubt, but chances are you _will_ eventually get bored of doing the same things again.
I am _not_ telling you whether you should like it or not, and I'm not telling you how to play the game.
Do you have any problem with those 3 points? Well, then say so. But being subjected to some lame armchair shrink gig, based on someone's inventing outright falsehoods about me, is... lame.
That said:
I never said that WoW wasn't fun. In fact, I'm currently going through the low levels again with a group of 2 other friends who've made matching alts just for that. I don't find the end grind fun, but then I'm not doing it any more anyway. Do you see any basic problem with that?
It's outright silly to be fed the "stop crying about games you don't find fun (after you have played them for a long time)" line, when, you know, usually that's _my_ line. Trust me, I _know_ that any game who kept me busy for more than 10 hours (the average length nowadays of an offline game costing the same), must have had _some_ good points. Usually I come across as some kind of Blizzard fanboy, so it's kinda funny to simultaneously come across as that which I constantly criticize.
I just don't see how can anyone claim that only the level 70's are catered for, and that everyone else is some kind of second class citizen. You know, given hat there is less content for the former than for the latter. That's the claim I was answering to. Again, I'm certainly not telling you that you shouldn't like it. And I'm not claiming that the whole game is bad.
Basically: Address what I've actually said, not what you imagine, please. God knows I probably say enough actual dumb stuff to pick on as it is. No need to build a strawman about how you imagine me to be :P
Well, that's funny, 'cause our armchair-shrink friend was claiming it's _not_ for the loot, and that somehow I have some personality deffect that makes me fixate on loot. Maybe you can have a vote and decide if my mental problem is being for or against phat loot? It's kinda impossible to be both ;)
A polar bear is a cartesian bear after a coordinate transform.
Their tests also say that the unpredictable lag only occurs in "extremely stressful lab conditions". Anyone believing CCP at their word at this point is flat out retarded. If it fixed it it does, but CCP saying they fixed it is as meaningful as if the CCCP said they did.
It's not just talk. These are real response-time measurements on the real server (tranquillity) in Jita, on sunday.