Slashdot Mirror

← Back to Stories (view on slashdot.org)

Behind the Scenes At Sony's NOC

Posted by kdawson on from the 140-servers-each dept.

VonGuard writes "Earlier this year, I spoke to Mark Rizzo, the man who manages the people who run Sony's online game servers. Rizzo learned the ropes of MMO hosting back on Ultima Online, and we chatted about where the tough problems were then versus now. Rizzo compares the operation to a 24/7 scientific simulation, albeit with some sassier and more involved end-users. His favorite innovation since those early days? Rapidly provisioning and deploying Linux installations tailor-made to their purposes. Here's my article on Rizzo and his band of 50-some-odd sysadmin-cum-dungeon-masters, written for the new newspaper The Systems Management News."

13 of 49 comments (clear)

  1. Summary: We have scripts by BadAnalogyGuy · · Score: 5, Funny

    So to sum up, they have lots of programs that are constantly watched by scripts. They get to heave server machines around to expand certain areas and replace old servers. Their lives are mostly taken up with making sure that the backups are properly done on time each day and that no one accidentally steps on the power cord.

    Fascinating!

    1. Re:Summary: We have scripts by thermian · · Score: 2, Funny

      So he worked on Ultima Online eh? I guess one of those scripts will have the line "if (lord_british) keepalive;" then.

      --
      A learning experience is one of those things that say, 'You know that thing you just did? Don't do that.' - D. Adams
    2. Re:Summary: We have scripts by LandDolphin · · Score: 2, Funny

      Only after August 8, 1997

      --
      Spelling and Grammar errors have been added to this post for your enjoyment
  2. sysadmin-cum-dungeon-masters by Anonymous Coward · · Score: 5, Funny

    sysadmin-cum-dungeon-masters


    Anyone else have images of S&M runnin through their minds?
    1. Re:sysadmin-cum-dungeon-masters by somersault · · Score: 4, Funny

      Beowulf's been a naught boy.. PRINT IT!!!

      kill (1000+('od -An -N2 -i /dev/random')%2001)

      Oh, you like that don't you!? Want me to do it again? First, I'm going to show you what a real glob is..

      --
      which is totally what she said
    2. Re:sysadmin-cum-dungeon-masters by RuBLed · · Score: 2, Funny

      And I thought Ballmer Peak was bad enough...

      So anyone working on Sony's MMO division is a cum-dungeon bitch.. err.. minion now?

    3. Re:sysadmin-cum-dungeon-masters by Anonymous Coward · · Score: 1, Funny

      sounds like a badly titled porno.

  3. Arr Eye Zee Zee Oh by Anonymous Coward · · Score: 1, Funny

    Rizzo. Oh hang on a minute, that's Frank.

    1. Re:Arr Eye Zee Zee Oh by Anonymous Coward · · Score: 1, Funny

      Yeah... open your fuckin' ears, jackass!

  4. Promotion Strategy. by Angostura · · Score: 4, Funny

    If you spend an awful lot of time grinding in the NOC you eventually become a level-something 'Network Architect' with no direct reports but with the ability to tell everyone what to do.

    Taking a few management tips from in-game, perhaps?

  5. cum-dungeon-masters TAG! by Anonymous Coward · · Score: 0, Funny

    Pretty pretty please tag 'cum-dungeon-masters' just for this!

  6. Funny mis-read ! by Anonymous Coward · · Score: 1, Funny

    I mis-read part of the last sentence in the summary as "50-some-odd sadism-cum-dungeon-masters" which, oddly enough, makes some sense.

    Yeah, I have a scary mind. Boo !

  7. Re:Would love to hear more from these teams by Minwee · · Score: 4, Funny

    The secret to achieving five nines uptime is not to improve the reliability of the systems, but instead to be very careful about how you define "uptime".

    "Hey, about those two hours of downtime last night..."

    "There wasn't any downtime."

    "No, really, the phones were lit up with people complaining that the applications weren't answering properly..."

    "So the applications were answering queries? Then they were up. It's not downtime."

    "But they were answering queries with error messages."

    "Then that's an application problem. The system was still up."

    "But the error messages said 'No response from database'. The database servers were down."

    "No they weren't. They were still running. They still had power. The servers were up. It's not as if they fell down out of the racks. You can't call it downtime just because a few programs aren't behaving exactly the way you want."

    "So about this SLA..."

    "Five nines, baby. We've still got five nines."