Slashdot Mirror

← Back to Stories (view on slashdot.org)

Engineering Everquest

Posted by Zonk on from the iron-behind-the-giant dept.

The IEEE Spectrum site has an article up discussing the engineering required to keep Norrath running. From the article: "The Death Star is a huge, warm, windowless room containing the rows and rows of servers that run Sony's online games. The whooshing of a massive air-conditioning system is so loud that conversation is almost impossible. A large steel cage surrounds more than 500 servers stacked 32 high in towering racks--and this is just one battalion, albeit the largest, in Sony's 1500-machine army of servers."

29 comments

  1. um , had to say it by FidelCatsro · · Score: 1, Funny
    The Death Star is a huge, warm, windowless room containing the rows and rows of servers that run Sony's online games. The whooshing of a massive air-conditioning system is so loud that conversation is almost impossible.
    That's no moon , It's a space station
    --
    The only things certain in war are Propaganda and Death. You can never be sure which is which though
    1. Re:um , had to say it by Anonymous Coward · · Score: 0

      Oh come on
      Moderating that over-rated is surely entrapment.
      "ITS A TRAP"
      no Geek can resist adding a star wars quote

    2. Re:um , had to say it by Anonymous Coward · · Score: 2, Funny

      That's not a redundant comment , its a space station!

  2. What For? by Anonymous Coward · · Score: 0

    EQ doesn't have players anymore, what are the servers for?

    1. Re:What For? by koi88 · · Score: 3, Funny


      EQ doesn't have players anymore, what are the servers for?

      To keep the NPCs alive?

      --

      I don't need a signature.
  3. DRAMA! by Xionen · · Score: 0, Flamebait

    I absolutly love the dramatization of a ..server room. it makes me *sarchastically speaking* SO happy that sony can boast about the server room we cream.. I mean dream about. The only problem is that sony has all of these pretty server rooms and money and still can't hire a team to listen to their customers or atleast do their job in the helpdesk area. I have been boycotting sony for a while, and no I do not have a ps2 in my house or ps1, and by the way it seems the downgrading of ps3 is only gonna make people get away from it too. stupid sony, being complacent ugh, if they would only listen to the little people down here underneith the money cloud they would be on top, and I am bitter. Could ya tell? right ranting post whore I am. bah.

    1. Re:DRAMA! by snuf23 · · Score: 3, Insightful

      From the article:

      "Three people per shift work in the NOC, and there are three shifts per day. During each shift, NOC staff monitor game activity, responding to players in remote locations and working with a custom suite of software tools to fix problems along the way."

      Maybe this is why the GM didn't answer your call. There were only 3 of them.
      I seriously hope and imagine that there are more (maybe 10) but the article totally implies that 3 people handle problems with players in remote locations.
      Which would totally explain SOE's tech support.

      --
      Sometimes my arms bend back.
    2. Re:DRAMA! by Xionen · · Score: 0

      well, u don't have to defend them, and I understand there isn't many Gm's, its cool, just sometimes people need to vent, seemed like the right time. however, I am not sure how many of you out there played EQ back in the day when it first started and all of you'r information was on an email account, after five years this email has slipped into the matrix neverland, now it has ended and you need to get rid of you're account however they won't allow it because you don't have the email from five years ago or the credit card you got the account with becasue you actually moved out of you'r parent's house, and they don't even know which one of their 37 credit cards is the one...so you are stuck in the world of hurt because they will not cancel you'r account or change you're name, or anything possible. I am sure there was an easy fix, but at the time I was quite pissed. its fun to start up the debate though.

    3. Re:DRAMA! by UnknownSoldier · · Score: 3, Insightful

      > Which would totally explain SOE's tech support.

      You mean lack of it.

      *ducks*

  4. Not just Everquest by MMaestro · · Score: 2, Interesting
    At any given time, Sony is hosting more than 150 000 sword-wielding, laser-shooting, dragon-slaying gamers from all over the world.

    Everquest is merely the highlight of the report, but theres also EQ2, Planetside and SWG all running off these servers. With that in mind, 150,000 users 'at any given time' isn't too impressive when spread out over 1,500 servers. Assuming the number of users is equally divided (37,500 users and 375 servers per game 'at any given time') then theres not much workload really being put on the servers. That comes out to roughly 1 server per 100 players.

    1. Re:Not just Everquest by Dachannien · · Score: 2, Informative

      Remember, though, that Raph Koster mentioned in an article linked in another Slashdot post that servers devote roughly 40% of their processing time to pathfinding for NPCs. (The stat was for SWG in particular, though it likely applies to most games with the same NPC-to-PC ratio).

    2. Re:Not just Everquest by snuf23 · · Score: 1

      That's hilarious. At least from a City of Heroes and World of Warcraft (the 2 mmos I've played) pathfinding on npcs can be described as:

      1. Walk mob between predetermined points A and B when not fighting.
      2. Chase after player who is running away in a straight line path. Maybe move around small obstacle to continue chasing player. On the other hand, maybe get stuck and decide following player is too much trouble, so return to path outlined in number 1 above.

      --
      Sometimes my arms bend back.
    3. Re:Not just Everquest by Walkiry · · Score: 1

      City of Heroes is worse than that. In City of heroes, if the NPC finds any difficulty when going from point A to point B it'll merely jump to the next point in the path and merrily continue its way. This mighty leap is undeterred by players (there's collision detection in CoH), crates, scenery, walls, skyscrappers, player-created obstructions (such as "rain" powers or AoE location debuffs)... You name it, the NPC can jump to/through/over it.

      If that "pathfinding" requires 40% of their server resources, I say their code sucks.

      --
      ---- Take the Space Quiz!
    4. Re:Not just Everquest by I8TheWorm · · Score: 1

      Yet they still lag :)

      As SWG is one of the (if not the) lowest populated games, I'd say there's something awry with the numbers. I'm on a faily low populated server, and at any given time there are (just guessing) 1-2k players on. The more heavily popilated servers have around 3k players. SOE stopped reporting number of active accounts some time ago when they dropped from the initial 350k, so I can only guess what the actual numbers are. What I don't know is how many servers run my galaxy. But I can tell you this... the lag can be excruciating at times.

      --
      Saying Android is a family of phones is akin to saying Linux is a family of PCs.
    5. Re:Not just Everquest by Wyatt+Earp · · Score: 1

      Yea, but think of how many hundreds of thousands of NPCs there are to move, keep track of stats, hit the loot tables for when killed every second.

      On that note, I've wondered for a while what the WoW servers are like for a world, say, Uther where I am.

    6. Re:Not just Everquest by achacha · · Score: 1

      And yet NPC pathing in EQ was one of the worst, despite the fact it used the rail system. MOBs moved along defined rails to get to you and often would get lost easily and eventually appear right on top of you. This was also very obvious to pet classes where their pet was told to attack a monster in front of you and it decided to go sideway, through 6 interconnecting rooms, up and down some stairs and finally get to its destination with the whole damn dungeon in tow... which was instant death for you and then an hour or two corpse run if you are lucky and then few days of trying to get your lost xp back. Good riddance to sony and their masochistic game. WoW and CoH are so much better in every way.

    7. Re:Not just Everquest by arkanes · · Score: 1

      Pathfinding requires heavy computation even for crappy versions (WoWs is a little better than you describe, but not much). Even the best pathfinding algorithms have poor worst case performance - which is why developers put early bail out with "cheats", like the walking through walls or jumping to the player you see so often, in.

    8. Re:Not just Everquest by Anonymous Coward · · Score: 0

      > Maybe move around small obstacle

      You say this as though it is a simple algorithm. It is quite the opposite - that is precisely what makes pathfinding so difficult and the algorithms so CPU intensive.

    9. Re:Not just Everquest by Dachannien · · Score: 1

      Actually, the only time that a mob would appear right on top of you like that is if the "cheat detection" went off. Back in the day (1999), when not everybody was completely uberized, people actually used to grind XPs in Lavastorm. The less scrupulous would tag a mob, run to the entrance to Najena, and duck to the left just before the zoneline. If they were fast enough, they could do it before the mob got inside the cave entrance, and it would get stuck on the outside of the cave on the hill, where they could nuke it freely (they were outdoors, so no line-of-sight check, and the mob was hung up on the hillside just inside of spell range).

      The fix was that if the game detected a mob pathing toward a player, but the mob stopped actually moving for a while, it would just port the mob directly on top of the player. This would sometimes happen in dungeons, but generally the planned pathing was a lot tighter, meaning that while the mob might take the long way to get to you, it would eventually be able to find you without actually going off the "rails". Later content was significantly better in this respect, and things got better again when they revamped the pathing system some time back. There are still spots to avoid, though, if you don't want things to go all screwy on you.

      Also, nowadays, pets don't aggro anything unless they are actually attacking something. I'd just note that this is *not* true of WoW - if you drop off a ledge in a dungeon somewhere, your pet will take the safe way down and aggro everything it runs into along the way.

    10. Re:Not just Everquest by achacha · · Score: 1

      WoW has a long way to go until they get the pet issues right, too bad they didn't learn any lessons from EQ. I played a mage and necro to 65, so I was all about pet classes and endured the horrid growing pains early in 1998 and through 2002 when I finally gave up, at that point the pets were mediocre in the lower planes and you really needed a focus to get them to be workable, luckily the LDON one was relatively easy to get, but rest of the focii were a royal pain. I was 2 boxing a mage and cleric just so I could get some xp somewhere away from everyone that could train me... Eventually I gave up (when cleric hit 65) at that point seeing that LDON focus would not get me too far and all I could do was clear Droga/Nurga.

      I really hope Blizzard adds useful things like neutral pets, ability to suspect/recall pets (best way would be to create a gateway to the stables, so you could store pet in stables, navigate over the chasms and such, then get pet back from the stables portal), ability to cure pets' disease and poison, more pet stat boosts, etc.

  5. Second place by Anonymous Coward · · Score: 4, Insightful

    I'd rather have an article describing, you know, the awesome engineering obstacles (failures and successes, thanks) a similar but larger project a few people may've heard of. I seriously felt like this was a copy of an article from Time magazine, what with the information that was just @#$@ old and common knowledge (that is, to the audience of people who would be interested in the first place), and really just an advertisement in disguise.

    I admit I'm new to reading this periodical, and skipped straight to this article, but... I sortof held IEEE to higher standards. Maybe I should go to journalism school, or something, but ... why do I care? I didn't learn anything. They were small, got bigger. They make revenue from customers. Now, if this was a wonderful introduction to When Projects Go Off Expectations, with a "case study", then that would have been something.

    "We budgeted the game to hit 200,000 subscribers, 20% churn, eleventy billion in cash, and so we set up servers (because YUO LIKES TEH OVERFLOWZ) to support 300,000. When 500,000 people showed up on opening day, Leeroy Jenkins shat his pants. He dropped a drumstick and suggested we throttle connectivity, so that players who do get on have the experience we designed, and players that don't have a tangible explanation as to our undercapacity. Thus was born the much hated queue system, and you can bet Leeroy a chicken that we burnt the midnight oil adding capacity. A unique issue games like EverWowSide face is that we can't just take the servers down for teh upgradez. So we had to do a cost benefits analysis and for the first two weeks we did rolling reboots [SEE SIDEPANEL]... our original upgrade-lifecycle plan [SEE SIDEPANEL] expected us to have this sort of user growth over the course of a year, but we had to have had it doned yesterdays liek JeffKs. So [INTERESTINGLY INFORMATIVE ARTICLE ABOUT A MASSIVE ROLLOUT IN A MONTH THAT WAS PLANNED FOR A YEAR, HOPEFULLY AVOIDING TOO MUCH BACK PATTING AND MAY POSSIBLY BE RELATED TO ENGINEERING, AS OPPOSED TO COME PLAY EVERSIDEGALAXIES2]"

    No, wait. I'm wrong. An article about all those empty servers is much more interesting.

    1. Re:Second place by jasonmicron · · Score: 1

      No, wait. I'm wrong. An article about all those empty servers is much more interesting.

      EverQuest 2 servers are hardly empty.

    2. Re:Second place by achacha · · Score: 1

      EQ2 servers were empty by comparisson to the EQ servers maybe 2 years ago. Outside of the newbie zone it's pretty barren.

      I checked on EQ about 3 months ago at a friend's house (I refuse to go back) and almost every zone he went on was either completely empty or had 1 person running through it en route to a raid zone where almost everyone plays in (since many guilds imploded and combined into 1 big guild just so they can handle the high end content).

      We then played some EQ2 (for sh*ts and giggles) and zones for people 20 were semi populated, but zones for mid 30s were pretty empty. I suppose there are more crowded zones for the high end but from 20 on it's a lonely grind.

      By contract most of WoW servers are heavily populated with 20-30 people in many zones. This is how EQ was in 2000-2002 roughly and I suspect WoW will be in a few years when the next best thing is out.

      CoH is hemmoraging people as well mostly due to lack of high end content and the horribly painful grind once you hit level 35, every level is a sheer excersize in repetative killing.

      Well, that's the MMORPG world view from my rather small looking glass.

  6. Wonder about UO server by BloodAngel_Au · · Score: 1

    After reading this I was wondering if anyone had this sort of story about the Ultima Online servers, and how they have changed/ grown onver the last 7 years.

    I'm curious now as to how many servers run the oceania shard (supposedly in Sydney)

  7. NO PICTURES? by mjpaci · · Score: 2, Funny

    This is geek porn! How can you talk of a room full of computers running a popular MMORPG and NOT HAVE PICTURES? /. ppl get excited about pictures of cruddy wiring in LAN closets! This article might as well be a harlequin romance novel.

    --Mike

    1. Re:NO PICTURES? by Anonymous Coward · · Score: 1, Informative

      The print version of the article has a picture of the server cluster and the Death Star.

      Your local library may receive a subscription, but if you live near a university that offers electical or computer engineering degrees you might have better luck contacting the campus library.

  8. SOE glue sniffers by Fiz+Ocelot · · Score: 3, Funny
    From the article: "The NOC, a crowded room in Sony Online Entertainment's main office building, smells faintly of elementary-school glue.

    That really goes a loooong way in explaining a lot of things that go on over at SOE. I wouldn't be at all surprised if they eat it too.