How EVE Online Dealt With a 3,000-Player Battle
Space MMORPG EVE Online is best known for its amazing stories, and on Sunday it added a new epic tale. The leader of a huge coalition, preparing for a moderately sized assault, mis-clicked and accidentally warped himself into enemy territory without his support fleet, endangering his massive ship worth an estimated $3,500. Realizing the danger, he called upon every ally he could, and the enemy fleet rallied in turn, leading to an incredible 3,000-player battle. What's also impressive is that the EVE servers stayed up for the whole fight, when most MMOs struggle with even a few hundred players at the same time. The Penny Arcade report spoke with CCP Games for some information on how they managed that:
"It’s hard to wrap your head around, but they sometimes move the in-game space itself. 'We move other solar systems on the node away from the fight. This disconnects anyone in those systems temporarily, but spares them from the ongoing symptoms of being on an overloaded server,' Veritas explained. 'It helps the fight system a little bit as well, especially if a reinforcement fleet is traveling through those other systems. This was done for the fight over the weekend, but is rare.' ... They do have a built-in mechanism for dealing with massive battles, however: They slow down time itself. ... Once server load reaches a certain point, the game automatically slows down time by certain increments to deal with the strain. Time was running at 10% speed during this 3,000-person battle, which is the maximum amount of time dilation possible."
Space MMORPG EVE Online is best known for its amazing stories [...]
Since when? I thought EVE Online was best known for its elitist asshole userbase, constantly insisting that it's the mostest hardcorest economic sim EVAR, meaning all other forms of electronic entertainment are inferior, and anyone who enjoys any other form of electronic entertainment should feel inferior, especially in the shadow of people willing to dedicate disturbing amounts of their free time paying a real-world company to manage a fake company with as much complexity and sheer spreadsheet-grinding eye-scarring boredom (if not more) as a real one.
How many were divorced the next day?
Hahaha this is nothing new VOTF Xirtam did this a Few times and even a Red Alliance Leader did this.
Although we did not have 3000 players on back in those days.
So relativity is just the universe's way of saying the local server is currently way too crowded with rest mass?
According to the Eve message boards, it was a Leviathan-class Titan. $3600 may be a bit on the high side, but it was worth thousands, definitely.
Incidentlally, estimated losses for the entire battle (which included *three* titans lost before it was all over, all on the side the guy who misjumped) is over 700 billion ISK. That's about *$25,000*, kiddies.
...when my favored game makes the news, AND THEN MAKES IT MORE AWESOME.
All other MMO's are shit before almighty EVE Online.
You'll find that the best memories people have from playing MMOs is precisely this sort of encounter. The fact that EVE is designed to handle this type of event is a testament to the developer's understanding of what makes a good gaming experience.
"...hey remember that time when three of the top server guilds from [X] went to [Y] and it resulted in [C]...." The formula is not unique, but they happen rarely enough that each experience is.
The leader of a huge coalition, preparing for a moderately sized assault, mis-clicked and accidentally warped himself into enemy territory without his support fleet,
UI issue leads to massive server load.
They disconnected people who were in nearby universes, which they say is better than then suffering from overloaded servers. Isn't being disconnected from the game you're paying for because of a large battle going on somewhere else -- suffering from overloaded servers?
And time running at 10% of normal. That would seem like a pretty serious slowdown of their servers as a means of keeping people from suffering from a slowdown of the servers.
Maybe this MMORPG really is a different universe. It seems to have a different version of English than the rest of us have.
So there's a MUD which has PK. Someone accidentally separated from the party he was leading, and found himself in a room with a different party, which consisted of enemies. Instead of running away, he decided to fight, and he shouted for help. The whole MUD heard the shout. Then instead of just his party coming, nearly every player on the MUD ran to that one room, picked a side, and attacked someone. Does that about sum it up?
What a story!
"Believe me!" -- Donald Trump
so who won and what did they get?
world was created 5 seconds before this post as it is.
Computer-wise they need some virtualization-clustering fu. Not having coded so a logical node can run on several physical servers I can understand, but having some crazy-powerful server/nodes but no way to seamlessly move users to them seems a pity.
Military-wise, those who made the first mistake decided not to cut their losses, tried to recoup by throwing the good after the bad, throwing in reserves to save suddenly severely exposed friendlies, and they got severely burned for that. I'm sure there are second-years studying military strategy who are shaking their heads at newbie errors.
Human-wise, as alen said, how many divorces...
"Handled" is an interesting way to put it. While the node didn't crash, there were many in that fight that were having connection and desync issues. While I understand that CCP has a system in place to dynamically allocate resources depending on node loading, being in that fight with that much ti-di was unbearable. It literally took 30 minutes to get my dread out (A process that normally takes about 60 seconds.)
It sounds impressive but when you actually play eve, what you see is:
- little pluses and squares, because you have to zoom way, way out to prevent your computer from melting
- said pluses and squares moving at 1 frame every 5 minutes
Again it's a cool concept but when you factor in zooming way out and the tidi (time dilation, aka throttling, which is how the servers handle this load), it's really not that fun; when you hit F1 to fire and can walk away for literally 20 minutes before the hit lands, the excitement goes away.
Given the so-called epicness for a pay-as-you-pay-ass-game: "cost in damages are still being calculated, but early totals reach beyond 700 billion for both sides combined" I think the article is bloody sparse in details.
It is as if nothing really happened.
No videos to document the event. Nothing re-created.
Until that is publicly available, I'm not willing to believe in it but defer it as minor blip from a sinking company's black box shot to near death in the intergalactic struggle for omnipresence in the ominously luguber gaming world.
We move other solar systems on the node away from the fight. This disconnects anyone in those systems temporarily, but spares them from the ongoing symptoms of being on an overloaded server,
Hmm, not so impressive now. It stayed up the entire fight for those involved. Others were disconnected. So I guess for them the server didn't stay up the entire fight.
Bit of a correction here, the pilot in question is not the leader of the coalition.
EVE always dilates time around her.
So, the answer to how the game stayed up is that it's not a twitch game, and is actually pretty fucking slow with regard to "real time" actions of other games. In other words: It's basically a turn based game where latency isn't an issue so big fucking deal folks.
Okay, in this day in age of scalability and Cloud Services, why the hell can't they host this in an EC2 Availability Zone on Amazon? Use Rightscale or Scalar and like that massive Scale on-demand.. Slow down time. Pfft.. this is like thinking in the 90s.
Now, when my Civilization 5 battle comes into Eve you guys are toast!
Harrison's Postulate - "For every action there is an equal and opposite criticism"
I wish I had mod points....
Pretty good analogy, but the difference is that the guy who went to the "wrong room" was then "trapped there" by the people in the room. Forcing him to fight. So he called in backup so that he could get out.
:D
There were many of us who came in and didn't pick any side but simply started using AOE attacks against everyone else for the lols
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Goddammit Leroy...
Cool: bullet time!
DRM 'manages access' in the same way that a prison 'manages freedom'
Honeybadgers?
ClusterFucks?
The something awful alliance?
Facepalm!
Is this where the meme factory finds its consumers, or is it just a bunch of... I don't even know what.
I couldn't game with these folks based on the naming conventions alone.
At the bottom of EVE on the node CPU that simulates a given solar system there is a single thread. EVE uses what is effectively cooperative multitasking as implemented in a version of Stackless Python that CCP maintains. This micro-threading model precludes any real concurrency. That means a solar system (room, generally) cannot leverage SMP; the simulation must run in a single, coherent thread.
No matter how many cores CCP might have at its disposal in their big London cluster, they can't use more than a single thread on a single core to simulate a room, regardless of how many players there might be in that room. They have factored out some parts of the simulation to be distributed to other cores (networking, mostly,) but the essential simulation model is limited to one thread.
While unintended, these limits are probably a good thing; if they could use real concurrency on SMP hardware to handle the load created by many times 3000 players, some other fundamental constraint would assert itself and cause failures.
This is apparently what (part of?) the battle looked like... talk about a clusterfuck...
http://puu.sh/1TcVz
So all this time when games I played dropped to 5-10 fps, it wan't due to unoptimized code, crappy drivers, or old hardware. It was actually cutting-edge technology called Time Dilation!
The summary reads hilariously reminiscent of this:
http://www.youtube.com/watch?v=LkCNJRfSZBU
So... can you cash out that supposed "worth?"
Cuz if not, you're just another e-peen fanatic. Your several thousand dollar ship ain't worth jack.
What EVE does is just decides its going to slow everything down 10x so its only getting 1/10th the things happening over the same period of time. With this 3000 players are really effectivly only sending the traffic of 300 players operating at normal time. When you think of it like that, them being able to handle 3000 players without problems is not much of a boast, only that they found a way to simulate/force lag on everything. Its not a bad idea, but they are no more impressive than any of the other MMO's out there.
http://interserver.net/
Time was running at 10% speed during this 3,000-person battle, which is the maximum amount of time dilation possible."
I've seen this happen in other mmos as well, only there it's called lag.
pretending that they weren't. News at 10:00
Some of the city raids I went on for my server would have been more fun if things didn't lag like they did. While we never had over 3000 people, the numbers were high enough we had the server buckling a few times just by bringing in 3 raid groups to a whole Alliance city.
~~ Behold the flying cow with a rail gun! ~~
No other MMO has fights or losses like this. Makes it far more worthwhile when you stand to lose something (even a virtual something), as opposed to just reappearing at a spawn location otherwise unaffected by your loss. It's also nice to have a lot of the standard idiots chased from the game by the learning curve, and the need to actually socialize.
"You can't win. You can't break even. You can't quit." -A. Ginsberg
Here's a better article than the PCGamer one from the summary: http://themittani.com/news/asakai-aftermath-all-over-cobalt-moon
It assumes more game knowledge, but you should be able to get the gist of it.
Incidentally, GoonSwarm (one of the major corporations involved) is about to begin its annual newbee drive. EVEMail 'Lazyhound' for recruitment information!
The real-money losses are very sensationalised. A single player in null-sec space can easily farm 50-100million ISK per hour.
Hell, even a rookie like me (in my first battleship) solo'ing level 4 missions can generate 100m in half a day if I get on with it, and I only have 4 or 5 months of casual play under my belt, which isn't much by EVE standards.
With a PLEX (the method of converting real money into ISK) being worth around, say, 600ish million, the average ship on EVE doesn't need to be paid for in real money.
Likewise, titans and big stuff like that is often corp funded, they're not necessarily owned by the player flying the ship.
So I very much doubt that $25k actually flew out the door, or into CCP's coffers.
Thank god I'm too old for this stuff.
3600$ for playing shoot em up games? No, sir.
Also, bad architecture - a quadrant should spread across multiple machines according to load.
Why dose migrating to another server disconnect people?
Surely in a case like this they can just pause the action (if they can slow down time surly pause is easy) and tell everyone it will resume in 10min as the do the migration.
Worst case you could disconnect them but tell them if they don't connect to the new server within 10min then they are treated the say way as if they had yanked their broadband cable?
Anyone that'd spend even a tenth of that much money on that crap oughta be pimp-slapped.
Post the link!
I love this game. I do. But it seems once I get enough SPs that I deem comfortable to leave high sec and venture into nul sec, I get scrammed, webbed, and podded before I have a moment to react. At this point I got bored of mission running and frustrated by getting the absolute shit kicked out of me, I quit. Then, two months later I start a new account and start the process all over again.
I have no idea how CCP makes such a boring yet intense game that is fascinatingly addictive. =P
Please say someone recorded this!
It was one hell of a battle, I only got to see it towards the end, but still was cool. What made it cooler than anything was the fact the Goons got their ass handed to them.
We have systems that process over 10,000 simultaneous tranctions per second with distributed and redundent systems, in case of failures. We have internal teams and vendors working night and day when issues arise to ensure we don't have those recurring issues. Our systems are what make our company.
.05% of all Skill Point Characters). Losing a ship that took me one month of real time in a few minutes due to a laggy system is what led me to exit EVE, permanently.
CCP either lacks the experience or funds. However, since these types of large scale battles are either rare, or players keep paying for the sub-rate system, means that CCP has no great incentive to fix. I played a few years when game originally came out and sold my three characters for $1800 (one of my characters was top
Are you telling me the hardware doesn't exist to help shit like this?
It does exist, CCP just has lazy netcode programmers and doesnt have money to invest in new hardware because they are making a failure of an eve linked FPS called Dust 514
I want to see a link to the political maps for this area.
The last time I looked at those maps was the whole Band Of Brothers issues from years ago.
Apparently, there were two major battles going on this weekend, plus the 3100+ bigger battle from ... what, a year prior?
I want to see how the political control landscape changes as a result of these big fights. Link please?
EVE is completely sweet unlike this paper here by one Partyman Santaman http://prl.aps.org/abstract/PRL/v108/i9/e097403
Fact.
In the old days, in Asheron's Call, we called it lag.
Don't you hate it when you mis-jump to Ironforge and a bunch of dwarves takes out your titan?
Sure enough, the cow costume was hanging up next to the superhero outfit and sailors uniform. (S,Spud)
online games is evil from them only problem ..