How EVE Online Dealt With a 3,000-Player Battle

Space MMORPG EVE Online is best known for its amazing stories, and on Sunday it added a new epic tale. The leader of a huge coalition, preparing for a moderately sized assault, mis-clicked and accidentally warped himself into enemy territory without his support fleet, endangering his massive ship worth an estimated $3,500. Realizing the danger, he called upon every ally he could, and the enemy fleet rallied in turn, leading to an incredible 3,000-player battle. What's also impressive is that the EVE servers stayed up for the whole fight, when most MMOs struggle with even a few hundred players at the same time. The Penny Arcade report spoke with CCP Games for some information on how they managed that: "It’s hard to wrap your head around, but they sometimes move the in-game space itself. 'We move other solar systems on the node away from the fight. This disconnects anyone in those systems temporarily, but spares them from the ongoing symptoms of being on an overloaded server,' Veritas explained. 'It helps the fight system a little bit as well, especially if a reinforcement fleet is traveling through those other systems. This was done for the fight over the weekend, but is rare.' ... They do have a built-in mechanism for dealing with massive battles, however: They slow down time itself. ... Once server load reaches a certain point, the game automatically slows down time by certain increments to deal with the strain. Time was running at 10% speed during this 3,000-person battle, which is the maximum amount of time dilation possible."

Comments

Re:Since when?

[broggyr]

So, how many ships did YOU lose?

Relativity

[Anonymous Coward]

Time was running at 10% speed during this 3,000-person battle, which is the maximum amount of time dilation possible.

So relativity is just the universe's way of saying the local server is currently way too crowded with rest mass?

Re:Relativity

[girlintraining]

So relativity is just the universe's way of saying the local server is currently way too crowded with rest mass?

Not exactly. The servers might have stayed up but the health of the cluster was poor. I was logged in at the time, and was getting live reports from people on grid for the battle. There were a lot of disconnects across the entire eve universe; And this amplified the losses to the individual players. Many petitions were filed for damages due to getting "DC'd" and being unable to reconnect.

The cluster architecture for Eve is actually quite amazing, and the underlying logic exceptionally sophisticated. But the main failure point, which has been mitigated but not eliminated with the time dilation feature, has always been the database. Every action in the game generates dozens of database updates. When you have 3,000 people frobbing the gun buttons and the heal me buttons, things get ugly fast. Time dilation is a way of creating a queuing system so that the actions are accepted to the server, and then serially updated into the master database. The server tries to compress and reduce the amount of updates to this, doing a lot of calculations and updates, but ultimately, this link is of finite size.

The other bottleneck is that because of the caching and buffering mentioned above between each server and the central database, is that a server can't swap its resources to another server. If that server is managing, say, 40 (in game) systems, and one of them goes all nuclear, the other 39 also suffer from lag and such because those other 39 can't be offloaded to another server -- that state information stays on the server because of the buffering and caching issues mentioned earlier. It's a syncronization nightmare -- there's no way to cleanly break the flow of data and redirect it, and if any of those database updates get lost, it can mean real money lost to the players.

And real money was lost in Eve, not just because of player actions, but also cluster architecture. Those big ships don't just disappear when their pilot disconnects: They stay on the field, taking hits. And without a pilot, a lot of defensive actions (like warping away) aren't available anymore. I know at least 1 of those titans was lost because of a disconnected pilot. You can blame the ISP for that, but it was happening across the board, to all Eve players.

This behavior of the eve servers is well-known to regular players. Some alliances (large groups of players) even intentionally try to provoke such server failures, knowing it'll lead to losses like what's described in the article. Far from this being a success story... it's an example only of avoiding a worst-case scenario. The servers saying up means exactly dick if the servers aren't processing the requests in a timely fashion. Ask anyone on Wall St., why there's so many data centers ringed around it; Latency. It costs a fortune to host servers there, but those extra milliseconds matter.

As it turns out, MMOs have similar architectural features to our largest financial institutions. This one, more than most.

Re:Relativity

[EinarTh]

I'm a EVE server dev and this analysis is not quite right. The DB is indeed a central point of failure, but it's rarely a performance bottleneck nowadays. The part about migrating resources is half-wrong, as yes, we can't (yet) move solarsystems around machines without disconnecting the players in it, but unless there's a fight going on in a to be moved system, we still do it to free cpu for the system where a fight is indeed going on. See more here http://community.eveonline.com/devblog.asp?a=blog&nbid=74227 .

Re:3000 players you say?

[icebike]

Yeah, that was my thought too.

How many came out of the computer room sweating on their run to the fridge, uncommunicative, distracted, and wild eyed. Then crawled into bed late to a cold shoulder and a turned back.

Then having to go to work/school the next day and not be able to explain it to anyone because, nobody would understand, and all the raised eyebrows, and looking askance, and rolling of eyes between workmates.

Private little daydreams must be problematic when shared with 3000 other basement dwellers.

$3600 ship

[Chris Mattern]

According to the Eve message boards, it was a Leviathan-class Titan. $3600 may be a bit on the high side, but it was worth thousands, definitely.

Incidentlally, estimated losses for the entire battle (which included *three* titans lost before it was all over, all on the side the guy who misjumped) is over 700 billion ISK. That's about *$25,000*, kiddies.

Re:$3600 ship

[Darby]

which included *three* titans lost before it was all over, all on the side the guy who misjumped

Let me guess, he jumped into the battle screaming "LEEROY JENKINS!!!"

Re:$3600 ship

[Anonymous Coward]

I'm a bit new to this kind of thing so humor me please. You're saying a guy had $25,000 locked up in virtual stuff and lost it in this battle?

Let me clear up a few misconceptions here:

1) It wasn't any single player who lost that much in the battle; those are the losses attributed to the losing group of players, which in this case is huge, so those assets were originally generated by the collective work of probably thousands of players.

2) Even those thousands of players did not collectively pay $25,000 in real money to acquire those assets, they just played the game like anybody else.

3) The conversion of 700 billion ISK (the virtual in-game currency) into $25,000 is based on the ability to buy 30-day play-time cards for $15 and then sell them in-game for (as of writing) ~600 million ISK each.

4) If you do the math on the above, it's clearly wrong; 700 billion ISK would only buy ~1167 play-time cards, which would have cost only $17,500.

5) To top it all off, that real-dollars-to-game-ISK conversion only actually goes one-way; you can use game money to buy the play-time cards, but you cannot (legally) exchange those cards for real money. So the 700 billion ISK isn't *really* worth $17,500 since it's impossible to (legally) exchange the ISK for the dollars;, the conversion ratio is an academic metric.

Re:$3600 ship

[jxander]

Sounds like total losses across the board, not just for one guy.

One of the big selling points for EVE Online is that they fully allow real currency (yes, actual dollars) to purchase in-game goods and services. The general thought process being : in normal video games (specifically MMOs like WoW) people without jobs are at a distinct advantage because they can spend all day killing boars, leveling up, mining ore, etc. EVE balances that by letting employed individuals use the fruits of their daily activities in game. You spent all day farming in-game, I spent all day farming in the real world.

That being said, I'm not intimately familiar with the economy of EVE... but from the article, a single ship is worth upwards of $3,500. A lot of the smaller ships are worth a few hundred bucks at least. Multiply that across 3,000 people involved and, well ... that's a lot of real money blown on virtual space ships.

Re:$3600 ship

[Chris Mattern]

Why he simply didn't jump back to escape from this territory?

Because the first thing any opponent does in this situation is have tacklers web and warp scramble you. And they'll start bumping you to push you away from directions they don't want you go. And you materialize from an incoming jump a few kilometers away from the gate/cyno field. You ain't goin' nowhere.

If he couldn't jump back, why he simply didn't use his escape pod to escape, sacrificing only one titan, instead of three?

*That* is an excellent question. It's probably what he should have done. But he didn't want to eat the loss, so he upped the stakes, hoping he could win.

Re:$3600 ship

[cockroach2]

Technically he didn't jump to enemy controlled territory but to an area that isn't strictly controlled by players. From what I read the idea was to drop some big guns on top of a handful of enemies in a "neutral" system, a couple of enemies that were pleasantly surprised when instead of a sizable fleet they got a juicy target.

Then everybody called in reinforcements plus the locals also wanted to join the party.

Re:$3600 ship

[L4t3r4lu5]

Because the first thing any opponent does in this situation is have tacklers web and warp scramble you. And they'll start bumping you to push you away from directions they don't want you go. And you materialize from an incoming jump a few kilometers away from the gate/cyno field. You ain't goin' nowhere.

Clarification for those who haven't played:
- Tackler: A very fast and highly maneuverable ship fitted no offensive weaponry, just modules designed to prevent your escape. Very fragile; Relies on its speed to survive.
- Web: Propulsion suppression; Severely restricts your speed an maneuverability.
- Warp Scramble: Prevents you warping. In Eve you can warp to planets, stations, asteroid belts etc in the same system from anywhere at any time. Almost all ships need to be at a Jump Gate to leave a system; Capital ships are the exception, of which Titans are an example.
- Bumping: Eve ships have non-catastrophic collision detection. They "bump" off each other, with the imaginable results. You need to align your ship towards a destination to be able to warp there; Bumping prevents that.
- Cyno(saural) field: A point created in space onto which Capital ships can lock and jump to without traversing the systems between your current location and the Cyno. Jump distance has a maximum range, depending on the ship.

One tactic not mentioned is "bubble": A ship called an Interdictor may be fitted with an Interdiction Sphere Launcher. Interdiction Spheres ("bubbles") prevent warping while within their area of effect.

My Takeaway

[Guppy06]

The leader of a huge coalition, preparing for a moderately sized assault, mis-clicked and accidentally warped himself into enemy territory without his support fleet,

UI issue leads to massive server load.

Re:Fascinating article.

[michael021689]

No, you're reading comprehension is just poor. Each solar system shares a node with several others. When the solar system in question overloaded the node due to the battle, the other solar systems on that node were disconnected, likely for less a minute, so that that node could be dedicated to the battle. The other systems in question were placed on other nodes. Being disconnected for a number of seconds so that the system that you are in runs at full speed is much better than staying continuously connected and running at 10% speed. On the topic of reduced speed, there is a significant difference between intentionally running a game at slow speed and it breaking into a slow speed. By intentionally slowing the speed, they are employing a controlled and tested process. That is much more sensible than trying to run at 100% and just letting what happens happen.

Re:Since when?

[Keen Anthony]

The elitists definitely earn it. They have real money in the game, if they didn't buy their characters, their skill levels came with longevity, and they survived the jump from carebearing around in high sec with destroyers and cruisers modded for salvage and mining to doing PVP in null sec with total assholes. I would have loved to be amongst them except I just found the game frustrating for the constant "Join my clan!" invites. I like soloing, and it's not easy advancing fast without help and protection. I remember slipping into near low-sec territory because I wanted to sell some merchandise at a higher price. I decided to make a quick raid on an NPC pirate hideaway and do some good mining when a player jumped in, destroyed me, then held my pod for ransom. He pod-killed me when I refused to pay. Have to say I respect the guy's style. That you can play EVE that way or you can play EVE my way and try to earn a modest living selling components speaks much about this game.

Re:What the ???

[citizenr]

Okay, in this day in age of scalability and Cloud Services, why the hell can't they host this in an EC2 Availability Zone on Amazon?

Because that's a completely asinine idea, not even warranting a technical response?

I'd urge random smarmy Slashdotters to dig through the EVE dev blog and get a glimpse of the boundaries they've been pushing. Their infrastructure team knows their fucking business.

Boundaries of stupidity. They are using Stackless Python (NO multicore support) on the server.

>'We move other solar systems on the node away from the fight

No they dont, They disconnected people left and right, basically kicked them out of that node and made them reconnect. Its been what, 7 years? and they still didnt figure out how to do live migration.

>Once server load reaches a certain point, the game automatically slows down time by certain increments to deal with the strain. Time was running at 10% speed

He failed to mention your FPS counter also goes to 10% :) It takes HOURS to kill one ship in that SHIT hack of a lag fix mode.

Instead of implementing proper multicore support, or even dividing load among many racks they run everything in ONE python thread ....

Re:3000 players you say?

[Jarik C-Bol]

I still contend that EVE is simply a massive social experiment to see if you can get people to pay for the privilege of working a second full time job managing spreadsheets.

Re:Help an old guy understand this

[Cajun Hell]

> l
Gimli the Dwarf (Necromancer)
Strider the Human (Greater Necromancer)
Legolas the Elf (Necromancer)
>'hey guys, are you all ready?
You say, "hey guys, are you all ready?"
Gimli says, "y"
Legolas says, "wait, I think Frodo is coming"
> s
Pirate Cave
You are in a pirate cave. There are various pirate props here. The only exit is to the north.
Sauron the Maiar (Greater Necromancer).
Saruman the Istari (Necromancer) (blocking the north exit).
Grishnakh the Orc (Lesser Sorcerer).
Magic helmet.
Balrog the Balrog (Greater Necromancer).
Ugluk the Orc (Lesser Sorcerer).
Balrog says, "I see them on who, so there's a chance they might attack today"
>ooc oh shit, meant to type 'lead s'
You say (ooc), "oh shit, meant to type 'lead s'"
> n
The exit is blocked.
Strider shouts, "aren't we coming with you?"
Ugluk takes Magic helmet.
Legolas shouts, "wait"
Sauron grapples you!
Sauron says "what do we have here?"
> n
The exit is blocked.
You can't move while grappled.
> kill sauron
You attack Sauron!
Sauron attacks you!
> shout help!
You shout, "help!"
Saruman laughs.
Balrog blocks the north exit.
Gimli shouts "Are you coming back or should we wait?"
Ugluk wears Magic helmet.
Grishnahk attacks you!
Legolas arrives.
Legolas says "come back north, we're not ready"
You hit Sauron hard!
Ugluk attacks you!
Legolas tries to move north but is blocked by Saruman.
Azog arrives.
Strider arrives.
Bilbo arrives.
Strider says, "did you mean to lead us?"
Shagrat arrives.
Frodo arrives.
Sauron shouts "lag!"

Re:Computers, Soldiers, Men

[Anonymous Coward]

Military-wise, those who made the first mistake decided not to cut their losses, tried to recoup by throwing the good after the bad, throwing in reserves to save suddenly severely exposed friendlies, and they got severely burned for that. I'm sure there are second-years studying military strategy who are shaking their heads at newbie errors.

You are wrong. The narrative you are basing your comment on is the story both sides want you to believe (for different reasons) but it is probably not what actually happened.

Here is my current understanding of events:

There are three major coalitions involved in this battle: CFC, HBC and N3. CFC and HBC used to be very close friends but grew apart over the past ~6 months with tensions escalating to a cold war-like state as of late. N3 is nominally hostile to both but has good diplomatic relations with HBC.

There are also two small alliances involved - Drunk ‘n’ Disorderly (DnD) and Liandri Covenant (AZULA).

On January 17th DnD engaged a CFC fleet in a very ballsy maneuver - it didn't pay off for them as the CFC fleet commander DaBigRedBoat (dbrb) was very fast to call in massive reinforcements (many EVE players are connected to their coalition's jabber or IRC server even when not playing and will log into the game if a "ping" on jabber/IRC goes out. In this case dbrb called for everyone to log in which was a completely disproportionate and unnecessary response.). DnD took more losses than they could deal damage but the CFC now knew them for being a bit reckless and DnD had seen dbrb escalate a small engagement beyond reason.

On January 21st DnD showed - while fighting an unrelated opponent - that they could call in support from Pandemic Legion (an alliance that is part of the HBC) and were quite willing to do so if it would get them the advantage they need. Everyone following current events in EVE took notice of this fact.

On January 25th DnD attacked a Liandri Covenant POS (player-owned starbase) but due to game mechanics couldn't finish it off quite yet (a starbase goes into an invulnerable mode for 42 hours after being dealt significant damage - this defenders time to organize a defense and prevents a starbase from being sniped in some off timezone where no defenders are online).

Knowing that DnD would return the next day to finish off the POSm knowing that alone they would have a hard time defending it and being aware of the events of January 17th Liandri Covenant contacted the CFC (some CFC pilots probably have alts in AZULA) to organize a trap: Liandri Covenant would engage DnD when they returned to kill off the starbase, then dbrb would jump with a superior CFC fleet right on top of them.

The CFC was well aware that DnD could call on support from the HBC within minutes and factored that into their plan. Once DnD was engaged they would jump a Nyx supercarrier into the fray while having a supercapital fleet on standby - hoping to bait the HBC reaction. Nobody knows why dbrb thought this would be a good idea - maybe he just thought he could take on the HBC in late US TZ (as some of the HBC's supercapital-heavy alliances are EU TZ).

Today is the day... dbrb has set up his trap and DnD arrives to kill the AZULA starbase. The HBC is probably aware of the CFC trap (spies are everywhere). When DnD is on the field dbrb intends to jump in his bait Nyx first. Multiboxing several accounts (he is known to praise himself for his great multitasking capabilities) he makes a grave mistake and jumps his Leviathan-class titan instead of the Nyx supercarrier (he needs the Leviathan to stay back and "bridge" other ships to the battle, also the Leviathan is about 5x more expensive than the Nyx [which already is pretty expensive]).

DnD tries to prevent the Leviathan from escaping and calls on Pandemic Legion for support (which takes at least 10 - 15 minutes to form up). dbrb knows that this escalation will come and is pinging madly to get more CFC members to reinforce his planned t