How EVE Online Dealt With a 3,000-Player Battle

Space MMORPG EVE Online is best known for its amazing stories, and on Sunday it added a new epic tale. The leader of a huge coalition, preparing for a moderately sized assault, mis-clicked and accidentally warped himself into enemy territory without his support fleet, endangering his massive ship worth an estimated $3,500. Realizing the danger, he called upon every ally he could, and the enemy fleet rallied in turn, leading to an incredible 3,000-player battle. What's also impressive is that the EVE servers stayed up for the whole fight, when most MMOs struggle with even a few hundred players at the same time. The Penny Arcade report spoke with CCP Games for some information on how they managed that: "It’s hard to wrap your head around, but they sometimes move the in-game space itself. 'We move other solar systems on the node away from the fight. This disconnects anyone in those systems temporarily, but spares them from the ongoing symptoms of being on an overloaded server,' Veritas explained. 'It helps the fight system a little bit as well, especially if a reinforcement fleet is traveling through those other systems. This was done for the fight over the weekend, but is rare.' ... They do have a built-in mechanism for dealing with massive battles, however: They slow down time itself. ... Once server load reaches a certain point, the game automatically slows down time by certain increments to deal with the strain. Time was running at 10% speed during this 3,000-person battle, which is the maximum amount of time dilation possible."

Comments

$3600 ship

[Chris Mattern]

According to the Eve message boards, it was a Leviathan-class Titan. $3600 may be a bit on the high side, but it was worth thousands, definitely.

Incidentlally, estimated losses for the entire battle (which included *three* titans lost before it was all over, all on the side the guy who misjumped) is over 700 billion ISK. That's about *$25,000*, kiddies.

Re:$3600 ship

[Anonymous Coward]

I'm a bit new to this kind of thing so humor me please. You're saying a guy had $25,000 locked up in virtual stuff and lost it in this battle?

Let me clear up a few misconceptions here:

1) It wasn't any single player who lost that much in the battle; those are the losses attributed to the losing group of players, which in this case is huge, so those assets were originally generated by the collective work of probably thousands of players.

2) Even those thousands of players did not collectively pay $25,000 in real money to acquire those assets, they just played the game like anybody else.

3) The conversion of 700 billion ISK (the virtual in-game currency) into $25,000 is based on the ability to buy 30-day play-time cards for $15 and then sell them in-game for (as of writing) ~600 million ISK each.

4) If you do the math on the above, it's clearly wrong; 700 billion ISK would only buy ~1167 play-time cards, which would have cost only $17,500.

5) To top it all off, that real-dollars-to-game-ISK conversion only actually goes one-way; you can use game money to buy the play-time cards, but you cannot (legally) exchange those cards for real money. So the 700 billion ISK isn't *really* worth $17,500 since it's impossible to (legally) exchange the ISK for the dollars;, the conversion ratio is an academic metric.

Re:$3600 ship

[jxander]

Sounds like total losses across the board, not just for one guy.

One of the big selling points for EVE Online is that they fully allow real currency (yes, actual dollars) to purchase in-game goods and services. The general thought process being : in normal video games (specifically MMOs like WoW) people without jobs are at a distinct advantage because they can spend all day killing boars, leveling up, mining ore, etc. EVE balances that by letting employed individuals use the fruits of their daily activities in game. You spent all day farming in-game, I spent all day farming in the real world.

That being said, I'm not intimately familiar with the economy of EVE... but from the article, a single ship is worth upwards of $3,500. A lot of the smaller ships are worth a few hundred bucks at least. Multiply that across 3,000 people involved and, well ... that's a lot of real money blown on virtual space ships.

Re:$3600 ship

[Chris Mattern]

Why he simply didn't jump back to escape from this territory?

Because the first thing any opponent does in this situation is have tacklers web and warp scramble you. And they'll start bumping you to push you away from directions they don't want you go. And you materialize from an incoming jump a few kilometers away from the gate/cyno field. You ain't goin' nowhere.

If he couldn't jump back, why he simply didn't use his escape pod to escape, sacrificing only one titan, instead of three?

*That* is an excellent question. It's probably what he should have done. But he didn't want to eat the loss, so he upped the stakes, hoping he could win.

Re:$3600 ship

[cockroach2]

Technically he didn't jump to enemy controlled territory but to an area that isn't strictly controlled by players. From what I read the idea was to drop some big guns on top of a handful of enemies in a "neutral" system, a couple of enemies that were pleasantly surprised when instead of a sizable fleet they got a juicy target.

Then everybody called in reinforcements plus the locals also wanted to join the party.

Re:$3600 ship

[Jim Haskell]

He didn't jump back because he was physically incapable of doing so. In Eve: Online, other ships can use ship modules on you that prevents you from leaving the area. It's called "tackling." The pilot in question, upon erroneously jumping into the system, was tackled by enemy forces before he could escape. Instead of eating the loss, he called up on his allies to jump in to attempt to destroy the ships that were tackling him. (A titan-class vessel is largely unable to destroy the much smaller Heavy Interdictor-class vessels that are capable of tackling it due to the ship's poor tracking and large guns, and requires help if it is tackled. A good analogy here is trying to kill a fly with a cannonball at 30 kilometers -- guns in Eve work similarly.)

When the pilot's allies arrived, the enemies called THEIR allies and joined the fight. With the amount of tackling ships on the field, neither party could easily escape, and things snowballed considerably. The enemy forces in this case had the upper hand of available pilots and were able to inflict heavy losses.

Re:$3600 ship

[L4t3r4lu5]

Because the first thing any opponent does in this situation is have tacklers web and warp scramble you. And they'll start bumping you to push you away from directions they don't want you go. And you materialize from an incoming jump a few kilometers away from the gate/cyno field. You ain't goin' nowhere.

Clarification for those who haven't played:
- Tackler: A very fast and highly maneuverable ship fitted no offensive weaponry, just modules designed to prevent your escape. Very fragile; Relies on its speed to survive.
- Web: Propulsion suppression; Severely restricts your speed an maneuverability.
- Warp Scramble: Prevents you warping. In Eve you can warp to planets, stations, asteroid belts etc in the same system from anywhere at any time. Almost all ships need to be at a Jump Gate to leave a system; Capital ships are the exception, of which Titans are an example.
- Bumping: Eve ships have non-catastrophic collision detection. They "bump" off each other, with the imaginable results. You need to align your ship towards a destination to be able to warp there; Bumping prevents that.
- Cyno(saural) field: A point created in space onto which Capital ships can lock and jump to without traversing the systems between your current location and the Cyno. Jump distance has a maximum range, depending on the ship.

One tactic not mentioned is "bubble": A ship called an Interdictor may be fitted with an Interdiction Sphere Launcher. Interdiction Spheres ("bubbles") prevent warping while within their area of effect.

Re:Since when?

[Cinder6]

I read that as "stories about EVE" (things that happen because of player actions), not "stories in EVE" (things that happen within the scope of the game's narrative). Whether you like the game or not (I couldn't get into it), there certainly have been a lot of interesting/cool stories about things that have gone on inside the game. This event is one of them.

Re:It is as if nothing really happened

[Dragon_Eater]

http://themittani.com/media/pretty-lights-video-battle-asakai

Tada!

Also..
eve-kill.net/?a=kill_related&adjacent&kll_id=16069454
and...
http://dog-net.org/brdoc/?brid=16053

There, it did happen!

Re:Since when?

[Keen Anthony]

The elitists definitely earn it. They have real money in the game, if they didn't buy their characters, their skill levels came with longevity, and they survived the jump from carebearing around in high sec with destroyers and cruisers modded for salvage and mining to doing PVP in null sec with total assholes. I would have loved to be amongst them except I just found the game frustrating for the constant "Join my clan!" invites. I like soloing, and it's not easy advancing fast without help and protection. I remember slipping into near low-sec territory because I wanted to sell some merchandise at a higher price. I decided to make a quick raid on an NPC pirate hideaway and do some good mining when a player jumped in, destroyed me, then held my pod for ransom. He pod-killed me when I refused to pay. Have to say I respect the guy's style. That you can play EVE that way or you can play EVE my way and try to earn a modest living selling components speaks much about this game.

Re:Since when?

[tonywong]

Not an EVE player but I thought the coverage and screen cap was impressive here:
http://themittani.com/news/breaking-massive-super-fight-asakai-lowsec

Re:Relativity

[girlintraining]

So relativity is just the universe's way of saying the local server is currently way too crowded with rest mass?

Not exactly. The servers might have stayed up but the health of the cluster was poor. I was logged in at the time, and was getting live reports from people on grid for the battle. There were a lot of disconnects across the entire eve universe; And this amplified the losses to the individual players. Many petitions were filed for damages due to getting "DC'd" and being unable to reconnect.

The cluster architecture for Eve is actually quite amazing, and the underlying logic exceptionally sophisticated. But the main failure point, which has been mitigated but not eliminated with the time dilation feature, has always been the database. Every action in the game generates dozens of database updates. When you have 3,000 people frobbing the gun buttons and the heal me buttons, things get ugly fast. Time dilation is a way of creating a queuing system so that the actions are accepted to the server, and then serially updated into the master database. The server tries to compress and reduce the amount of updates to this, doing a lot of calculations and updates, but ultimately, this link is of finite size.

The other bottleneck is that because of the caching and buffering mentioned above between each server and the central database, is that a server can't swap its resources to another server. If that server is managing, say, 40 (in game) systems, and one of them goes all nuclear, the other 39 also suffer from lag and such because those other 39 can't be offloaded to another server -- that state information stays on the server because of the buffering and caching issues mentioned earlier. It's a syncronization nightmare -- there's no way to cleanly break the flow of data and redirect it, and if any of those database updates get lost, it can mean real money lost to the players.

And real money was lost in Eve, not just because of player actions, but also cluster architecture. Those big ships don't just disappear when their pilot disconnects: They stay on the field, taking hits. And without a pilot, a lot of defensive actions (like warping away) aren't available anymore. I know at least 1 of those titans was lost because of a disconnected pilot. You can blame the ISP for that, but it was happening across the board, to all Eve players.

This behavior of the eve servers is well-known to regular players. Some alliances (large groups of players) even intentionally try to provoke such server failures, knowing it'll lead to losses like what's described in the article. Far from this being a success story... it's an example only of avoiding a worst-case scenario. The servers saying up means exactly dick if the servers aren't processing the requests in a timely fashion. Ask anyone on Wall St., why there's so many data centers ringed around it; Latency. It costs a fortune to host servers there, but those extra milliseconds matter.

As it turns out, MMOs have similar architectural features to our largest financial institutions. This one, more than most.

Screenshot from the battle

[Aereus]

This is apparently what (part of?) the battle looked like... talk about a clusterfuck...

http://puu.sh/1TcVz

Re:Screenshot from the battle

[Aereus]

And another one:

http://i.imgur.com/xXhcWOy.png

Re:What the ???

[Mr307]

Boundaries of stupidity. They are using Stackless Python (NO multicore support) on the server.
No they dont, They disconnected people left and right, basically kicked them out of that node and made them reconnect. Its been what, 7 years? and they still didnt figure out how to do live migration.
He failed to mention your FPS counter also goes to 10% :) It takes HOURS to kill one ship in that SHIT hack of a lag fix mode.
Instead of implementing proper multicore support, or even dividing load among many racks they run everything in ONE python thread ....

This 100%

Time dilation is a kludge at best, it serves only to make large fights just bearable instead of impossible.

The single threaded nature of EVE is grossly apparent in large fleet fights where all input is processed in order, so if you commanded your ship to go in 10 different directions, your ship goes all those directions in order, even if the last command was an all stop, and during the larger fleet fights this may take minutes to complete so most likely you are way out of position, add to that weapon cycles, reload cycles, warps, jumps and lots more to the queue that are stacked and processed in order with no real time consideration.

IMHO what they really need to do is groom the queue for redundant ship commands, Yes an upgrade to multithreaded for all kinds of good reasons, but grooming the command queue will net them huge improvements in playability when the fights get big.

Re:3000 players you say?

[Anonymous Coward]

They replaced lag with TiDi (Time Dilation) which just makes things go slow motion but everything is executed in order.

Re:Computers, Soldiers, Men

[Anonymous Coward]

Military-wise, those who made the first mistake decided not to cut their losses, tried to recoup by throwing the good after the bad, throwing in reserves to save suddenly severely exposed friendlies, and they got severely burned for that. I'm sure there are second-years studying military strategy who are shaking their heads at newbie errors.

You are wrong. The narrative you are basing your comment on is the story both sides want you to believe (for different reasons) but it is probably not what actually happened.

Here is my current understanding of events:

There are three major coalitions involved in this battle: CFC, HBC and N3. CFC and HBC used to be very close friends but grew apart over the past ~6 months with tensions escalating to a cold war-like state as of late. N3 is nominally hostile to both but has good diplomatic relations with HBC.

There are also two small alliances involved - Drunk ‘n’ Disorderly (DnD) and Liandri Covenant (AZULA).

On January 17th DnD engaged a CFC fleet in a very ballsy maneuver - it didn't pay off for them as the CFC fleet commander DaBigRedBoat (dbrb) was very fast to call in massive reinforcements (many EVE players are connected to their coalition's jabber or IRC server even when not playing and will log into the game if a "ping" on jabber/IRC goes out. In this case dbrb called for everyone to log in which was a completely disproportionate and unnecessary response.). DnD took more losses than they could deal damage but the CFC now knew them for being a bit reckless and DnD had seen dbrb escalate a small engagement beyond reason.

On January 21st DnD showed - while fighting an unrelated opponent - that they could call in support from Pandemic Legion (an alliance that is part of the HBC) and were quite willing to do so if it would get them the advantage they need. Everyone following current events in EVE took notice of this fact.

On January 25th DnD attacked a Liandri Covenant POS (player-owned starbase) but due to game mechanics couldn't finish it off quite yet (a starbase goes into an invulnerable mode for 42 hours after being dealt significant damage - this defenders time to organize a defense and prevents a starbase from being sniped in some off timezone where no defenders are online).

Knowing that DnD would return the next day to finish off the POSm knowing that alone they would have a hard time defending it and being aware of the events of January 17th Liandri Covenant contacted the CFC (some CFC pilots probably have alts in AZULA) to organize a trap: Liandri Covenant would engage DnD when they returned to kill off the starbase, then dbrb would jump with a superior CFC fleet right on top of them.

The CFC was well aware that DnD could call on support from the HBC within minutes and factored that into their plan. Once DnD was engaged they would jump a Nyx supercarrier into the fray while having a supercapital fleet on standby - hoping to bait the HBC reaction. Nobody knows why dbrb thought this would be a good idea - maybe he just thought he could take on the HBC in late US TZ (as some of the HBC's supercapital-heavy alliances are EU TZ).

Today is the day... dbrb has set up his trap and DnD arrives to kill the AZULA starbase. The HBC is probably aware of the CFC trap (spies are everywhere). When DnD is on the field dbrb intends to jump in his bait Nyx first. Multiboxing several accounts (he is known to praise himself for his great multitasking capabilities) he makes a grave mistake and jumps his Leviathan-class titan instead of the Nyx supercarrier (he needs the Leviathan to stay back and "bridge" other ships to the battle, also the Leviathan is about 5x more expensive than the Nyx [which already is pretty expensive]).

DnD tries to prevent the Leviathan from escaping and calls on Pandemic Legion for support (which takes at least 10 - 15 minutes to form up). dbrb knows that this escalation will come and is pinging madly to get more CFC members to reinforce his planned t

Re:Relativity

[EinarTh]

I'm a EVE server dev and this analysis is not quite right. The DB is indeed a central point of failure, but it's rarely a performance bottleneck nowadays. The part about migrating resources is half-wrong, as yes, we can't (yet) move solarsystems around machines without disconnecting the players in it, but unless there's a fight going on in a to be moved system, we still do it to free cpu for the system where a fight is indeed going on. See more here http://community.eveonline.com/devblog.asp?a=blog&nbid=74227 .