Slashdot Mirror

← Back to Stories (view on slashdot.org)

How Demigod's Networking Problems Were Fixed

Posted by Soulskill on from the crunch-time dept.

The launch of Demigod was troubled by piracy and networking difficulties, which publisher Stardock worked quickly to correct. They've now released a documentary that gives a detailed look behind the scenes of diagnosing and fixing those problems. It includes meetings, interviews with the devs, and part of the bug-tracking process during a frenzied 108-hour work week.

22 of 65 comments (clear)

  1. Nice summary by Jurily · · Score: 4, Insightful

    Am I supposed to watch half an hour of videos before they tell me what they actually did?

    Couldn't you just, you know, summarize it for us?

    1. Re:Nice summary by Anonymous Coward · · Score: 2, Informative

      Don't bother, video 1 was some incredibly boring office meetings. video 2 I skipped to cut to the chase, and video 3 was home movie of the boss walking the halls annoying the ever living crap out of the developers actually trying to fix the problem. Again, boring!

  2. Try testing by Anonymous Coward · · Score: 2, Funny

    Maybe they should have made a documentary about their testing procedures instead, or would that be too short to fill even 10 minutes?

    1. Re:Try testing by ta+bu+shi+da+yu · · Score: 2, Funny

      Yeah, cause if you use the right testing methodology you will never ever have any issues with your code. Netcraft confirms it!

      --
      XML is like violence. If it doesn't solve the problem, use more.
  3. Awesome... by Anonymous Coward · · Score: 3, Funny

    ...yet another Demigod Slashvertisement.

    1. Re:Awesome... by XorNand · · Score: 2, Informative

      Stardock is based in Plymouth, MI which is halfway between Detroit and Ann Arbor. Slashdot was originally founded in the same area (Dexter, MI IIRC). I live about 15 minutes north of Dexter and about 45 minutes from Plymouth. In case you haven't been following the news, Michigan doesn't have a whole lot to be proud of, economically-speaking. It's possible that the editors have a special fondness for Stardock. While independent game publishers are common in California, it's very rare to find a successful one in the Midwest. And honestly, Stardock produces some pretty top-notch stuff. I am consistently impressed about how close they get to EA in terms of production value. They also have a very progressive (customer friendly) stance on DRM and piracy.

      --
      Entrepreneur : (noun), French for "unemployed"
  4. Professional Game Studio? by Manip · · Score: 3, Interesting

    I've spoken with, and listened to, several game developers over the years and it just strikes me how different they have been to the guys on this. I'm watching the videos and it reminds me of my college days geeking it out with other geeky types and screwing around with code.

    Very different from the professional environment that you find in a lot of studios these days. They often seem to be run more like a Hollywood Movie or similar to commercial software companies and less like a college startup.

    But then again I've not played (or heard of) Demigod before this, and if they can produce fun games that's all that is really important. :)

    1. Re:Professional Game Studio? by PhoenixAtlantios · · Score: 2, Interesting

      But then again I've not played (or heard of) Demigod before this

      That seems somewhat odd, as they've tried to get as much free advertising as they can by posting various development or piracy related stories to social news sites; some stories even got caught by the traditional media. It's not a bad marketing effort; people seem to be falling over themselves to get the game's name out there.

      I wonder how effective this type of advertising actually is though; does pointing out your mistakes and how you fixed them to a technical crowd win additional sales? Maybe that can be the next article they submit.

    2. Re:Professional Game Studio? by TheTurtlesMoves · · Score: 4, Insightful

      When a game has problems and they don't publish details everyone bemoans that fact that they are so closed about it. "Just be up front and honest about it, we understand there can be problems..." everyone says. Then when some does that everyone starts crying out " Cheap bastards are just trying to get free advertising... ".

      --
      The Grey Goo disaster happened 3 billion years ago. This rock is covered in self replicating machines!
    3. Re:Professional Game Studio? by dunezone · · Score: 2, Interesting

      Here is the thing about the advertising and no piracy protection.

      Demigod is a DOTA clone which was originally a Warcraft 3 map that is still widely played today and will more then likely be ported to Starcraft 2 if possible. So Demigod is competing against a game that already has a wide install base and is already cheap and will compete against a new game that will sell like bananers.
      Secondly, there is another game called League of Legends which is in the same style of Demigod that is coming out and is currently in BETA. From what I have been told about those in the BETA. Everything that is broken in Demigod is fixed in League of Legends. Remember, first impressions go a long way, many games start like crap and gain momentum but it takes a lot to pull people back into it. Unreal Tournament 3 is a very good example.

      My opinion is that Demigod was released early to get a head start on League of Legends, and it has no piracy protection to help compete against the already installed and well tested Warcraft 3 DOTA map.

    4. Re:Professional Game Studio? by PhoenixAtlantios · · Score: 2, Insightful

      When a game has problems and they don't publish details everyone bemoans that fact that they are so closed about it. "Just be up front and honest about it, we understand there can be problems..." everyone says. Then when some does that everyone starts crying out " Cheap bastards are just trying to get free advertising... ".

      Game developers often run into that problem and most (almost all?) of them have decided it's not worth being candid about their problems. It's interesting that Stardock have been able to twist their problems into publicity though, as over the five articles you'll find on Slashdot over the last couple of months about Demigod there are a lot of comments from people that've never heard of either before; you couldn't deny that these articles have given them a serious brand recognition boost, the only thing that's left to decide is whether it's been positive or negative.

      I was serious when I questioned whether admitting your mistakes wins you additional sales, though. I'd be incredibly surprised if there weren't a measurable fraction of sympathy sales from the piracy problem, or sales generated by the networking fix articles. It would actually be interesting to see whether their candid discussion of Demigod has turned out to be brilliant marketing, whether intentional or not, and whether that publicity has resulted in higher sales than they initially predicted for the game or not.

      They've covered the other stuff, it'd be interesting to see them detail what effect the publicity has had on their sales. Whether intentional or not it has been a massive free advertising campaign, with positive comments about the game popping up on the articles covering it. If they truly want to be open, why not discuss how that publicity has influenced their ability to turn Demigod around from what initially appeared to be a disaster?

    5. Re:Professional Game Studio? by Anonymous Coward · · Score: 2, Interesting

      Your comment doesn't seem to have anything to do with advertising or piracy protection. What it does seem like is an attempt to use Demigod's limelight to get some publicity for LoL.

      Demigod's *retail* version ships without DRM because *all* Stardock games ship that way. It's notable that once you update the game through Impulse, though, the game is 'infected' with online activation DRM (but there are no limits on the number of activations from what I can tell).

      Furthermore, Demigod is not a DotA clone. It is only like DotA in the sense that they currently form a two-game genre. An analogy I've been using is that Demigod plays about as much like DotA as F.E.A.R. plays like Doom 3. The general mechanics are the same but their execution is completely different.

      LoL on the other hand looks like a complete DotA clone. From what I've read the lead designer on LoL is even from the original DotA team.

      I disagree with your final conclusion as well (based on stuff Brad Wardell has written and from playing a lot of Demigod), but since it's purely speculative and I don't think either of us can present any real evidence to back up either claim (i.e. you claim Demigod was rushed, I claim GPG and Stardock thought Demigod was retail-complete) I guess the only rational thing to do is to agree to disagree.

    6. Re:Professional Game Studio? by Kaboom13 · · Score: 2, Insightful

      Maybe if their problems were actually interesting, and not the side effect of them being idiots without a clue how networking works, people would be more interested. Everything I've seen seems to revolve around them digging up a method of connecting players from early rts games, used primarily on LAN environments, not actually testing anything with real world equipment. The problems they had are to be expected.

  5. poor management by timmarhy · · Score: 4, Insightful
    the ineptitude of their management can be seen in the 108 hour week. anything over a 12 hour day is wasted, and you NEED 1 day off a week minimum to recharge the batteries, otherwise you just find ways to waste time on the job.

    i've been there, i'm working 50 - 60 hour weeks and i achieve more now than i did in 90 hour weeks.

    --
    If you mod me down, I will become more powerful than you can imagine....
    1. Re:poor management by julesh · · Score: 2, Interesting

      the ineptitude of their management can be seen in the 108 hour week. anything over a 12 hour day is wasted, and you NEED 1 day off a week minimum to recharge the batteries, otherwise you just find ways to waste time on the job.

      i've been there, i'm working 50 - 60 hour weeks and i achieve more now than i did in 90 hour weeks.

      While I agree with what you're saying, it's only true long term. As long as they are _unusual_, long hours really can pull more work out of developers, without a necessarily huge drop in productivity. 108 hours is perhaps taking it too far (approaching 16hrs/day), but I can and have worked 90 hour weeks without significant issues. It's when you're doing them week after week that you lose productivity.

    2. Re:poor management by SeeManRun · · Score: 3, Insightful

      I just watched the entire video, and have no problem with what they did. Working in a game company, and any software company really, you learn that there is no substitute for real users. Once you get thousands of users online doing no deterministic things, your architecture crumbles. These guys did whatever they could to fix it, and for that I commend them. Hell, I was tempted to email the CEO and ask for a job until I learned they are in MI. I am very impressed with their efforts and the fact they documented it as evidence for their fans.

  6. What we need is ... by The_Quinn · · Score: 3, Funny

    A Government Solution:
    1. Pass regulations that don't allow buggy releases.
    2. Require dev studios to make games of each genre.
    3. Give special legal powers to developers to unionize, preventing firings, and forcing employers to deal with them.
    4. Give companies billions after they fail, and claim it was a failure of the free market.
    5. Appoint a Game Czar.
    6. Takeover the game companies, firing the company heads and appointing government stooges.
    7. Transfer ownership of the company to the government, give a minority stake to the developers.
    8. Make great games!

    --
    Mine is Good
  7. Multiplayer is not fixed by dunezone · · Score: 4, Interesting

    No to be a troll or anything but they haven't fixed multiplayer. They've released several patches over the past few weeks that have fixed issues but they tend to introduce more problems then fix.

    The game is good and I enjoy it a lot, but my god every time my friends and I decide to play we debate on rather we want to deal with all the connection issues. You spend more time waiting to get into a game lobby then playing the actual game.

    Like I said the game is good, and it has its share of issues even outside of multiplayer but to say its fixed is long from the truth.

    1. Re:Multiplayer is not fixed by pantherace · · Score: 4, Interesting

      What's funny is when this article got posted on /., and last night, it was pretty much impossible to play online, due to something with their servers. For

      The game itself is good. Stardock's ImpulseReactor is bad. Impulse isn't so great. The engine isn't really to blame (some fundamental bugs though)
      I've played about 120 games online, and win about 50% of those.

      Here are some of the problems, having played it since pretty much the first day:
      In the initial release, they always tried to use NAT punchthrough. This includes when it wasn't needed. This put more load on their servers and made it slow to connect. Maybe piracy played some role, but given how it worked after it was 'fixed' I rather doubt piracy was the main problem, but more of an excuse.
      There are some ISPs (and routers) that mess with port numbering for UDP receive. This screws up the P2P network connection.
      Demigod/Supreme Commander speak a sandboxed lua in communication. Impulse sends and receives XML, and they wrote some translation layer in there, which was losing messages.
      Stardock implemented some proxies for fixing some problems, unfortunately the proxies are in my opinion, buggy. I've been sitting next to two people, and the same person connects to two of us fine, then proxied to the other. Uhhh, wtf? Additionally, they seem to have a tendency to crash, fucking up the game.
      According to Stardock, Stardock didn't write the NAT punchthrough, but licensed it from Raknet. Also, now Raknet of that denies it's used in Demigod. (And technically, it's not used in the engine, only the connection making, which is the real place where the BIG FUCKING PROBLEM is.)
      Impulse needs to be smacked with even the GNOME HIG guidelines, Apple's, even CDE's. Seriously, wtf? (I know it's trying to look like the latest office. There are so many people that have problems with the interface because of that. Unless you've seen Office a lot you won't recognize that the orb in the upper left is a menu, and even if you have seen office, you will likely miss it because it looks like an oversize decoration. Disclaimer: I don't like Office's new interface style in the first place.)
      Impluse's chat is a webpage, even though it's IRC, which requires IE's security settings to be set to default, it doesn't work if that's set higher.

      Demigod itself:
      Suffers from the problem of being designed with Microsoft's tools that have been focused on the Xbox, and using similar ideas. For example, running at the speed of the slowest isn't a problem when you have a homogeneous environment. PCs aren't homogeneous.
      Has some issues with the UI and communications due to all computers running the sim, and having to wait for every other computer's packets.
      Occasionally a desync (sim wasn't the same across all computers), which I have only seen about 3 times.
      Crashes occasionally, usually on connecting to a port, due to a UPNP call. (This might really belong under Impulse.)

      Plus sides:
      Less so now, but Stardock people are commonly on their IRC.
      If they are on, they generally try to be helpful, or such.
      The game is *very good*, especially on a LAN (or the internet when it works). Remarkably balanced in my opinion, for a newly released game. There are a couple of things which I think might need to be hit with a nerf bat, but they aren't very many.

      Oh, and while there is single player, it's not designed for it. For anyone wanting to play it, play a few games against the AI to understand the basic game, then DON'T PLAY AGAINST THE AI, if you ever intend to play humans. It teaches you bad habits. For example I played with friends against AIs a few games, getting something like a 33-1 K:D ratio against the 'hard' AI. The next games I played against humans, I got waxed.

    2. Re:Multiplayer is not fixed by atamido · · Score: 2, Informative

      Here are some of the problems, having played it since pretty much the first day: In the initial release, they always tried to use NAT punchthrough. This includes when it wasn't needed. This put more load on their servers and made it slow to connect. Maybe piracy played some role, but given how it worked after it was 'fixed' I rather doubt piracy was the main problem, but more of an excuse. There are some ISPs (and routers) that mess with port numbering for UDP receive. This screws up the P2P network connection.

      A big part of the problem was routers that performed Symmetric NAT. Basically, the router will randomize the outgoing port from the router, which is actually a sensible security feature. The problem was that the port would get randomized as something to their connection server, and then randomized to another port when connecting to another player. So that other player would attempt to connect back over a port that wasn't valid.

      What was funny about it is that you could forward specific ports to your computer from your router, specify those ports in Impulse, but it still wouldn't work. The outgoing ports on the router would still be randomized to the connection server, and the clients would attempt to connect back over the randomized ports instead of the ports specified in Impulse.

      They've improved things but, as you say, they still have a bugs to work out. On the plus side, once they've fixed all of networking issues with Demigod, Impulse should be ready to handle whatever other games come along.

      The quote at the bottom of the page, "If things don't improve soon, you'd better ask them to stop helping you."

    3. Re:Multiplayer is not fixed by Rakthar · · Score: 2, Informative

      Your comment of:

      >>Demigod itself suffers from the problem of being designed with Microsoft's tools that have been focused on the Xbox, and using similar ideas. For example, running at the speed of the slowest isn't a problem when you have a homogeneous environment. PCs aren't homogeneous.

      is wrong. They are using the Supreme Commander engine, which was designed for PCs in mind, and had the same issue. The problem is that SupCom and now Demigod are Peer 2 Peer, which means if one node is lagging and can't keep up, everything has to slow down to that pace or things get out of synch. RTS games in particular seem to suffer from this, with Starcraft having slow players and Warcraft 3 having "laggy" connections as well.

      It has nothing to do with Xbox tools.

  8. Summary of Technical Reasons Stated In Videos by The+Real+Nem · · Score: 5, Informative

    I've summarized the technical reasons given by the three videos.

    Executive Summary:

    Seems like their peer-to-peer architecture exasperated otherwise common matchmaking and NAT transversal problems that should be expected and thoroughly tested when developing networked games.

    Video 1:

    • Summary: The game is peer-to-peer.

    Video 2:

    • Quote: "We're trying to figure out why users are being removed from the database, which keeps them from being connected, but their not really disconnected, but the server thinks they're disconnected."
    • Quote: "...and if you put a return to just ignore the disconnect message it works fine".

    Video 3:

    • Quote: We're working on a problem in which something, for some reason, keeps telling the database that manages people's records that they're disconnected and we don't know why.
    • Quote: So you're putting in the the debug messages?
    • Quote: Yep. Just putting in some debug messages to figure out where these ... what is, like ... wh what ... first of who is sending what, um these disconnects.
    • Quote: Right, okay, we'll be back.
    • Quote: "One of the fields in the database are backwards, or something, on IP addresses because there are so many IP addresses and ports and sockets and stuff that I don't know anything about."
    • Observation: They test a fix and it seems to work on several office PCs.
    • Summary: The description of the fix includes faster lobby connection time and visual changes to each players ping in the game lobby.
    • Paraphrase: How about in terms of the robustness of it, like, how much will people be able to connect?
    • Paraphrase: Their games are significantly more intelligent now. Um, there were are lot of crazy things that were going on before and we've been able to simplify a lot of them.
    • Paraphrase: What about proxy servers, do you think we need proxy servers now?
    • Paraphrase: No I don't think we need proxy servers. It might be good for backup; we were tossing around the thought, I think you brought this up, with routing some traffic if we can thorough the host. That may or may not be possible.
    • Paraphrase: Though we had to take care of the case if the host leaves the game we don't want the whole thing to fall apart.
    • Paraphrase: Right, right. But it might be a solution for players with low ping times who, you know, can't seem to connect to anybody, but, you know, want to get in the game.
    • Paraphrase: Right, that's true. Of course as a backup plan; so if the host left it would take out those people who have really bad connections. But based on what we're seeing, we think, this should handle symmetrical NATs now?
    • Paraphrase: What's that?
    • Paraphrase: Asymmetrical NATs; this should handle that now?
    • Paraphrase: Uh, it's going to handle it better, yeah.
    • Paraphrase: Well before it didn't handle it at all, so...
    • Paraphrase: Right before it didn't handle it at all, before...
    • Paraphrase: I mean if I have two IP addresses, will this work with it now?
    • Paraphrase: I don't know that if you have two IP addresses this will work. Uh, this is going to help alot with firewalls and nats, however, um, it does a better job of piercing through the firewall.
    • Paraphrase: Of course if you port forward to the right IP, then, from your router, then you're just set.
    • Paraphrase: Yeah, that's not an issue.
    • Paraphrase: So anyone who's technical will be fine, pretty much for sure, no matter on what kind of crazy connection they've got.
    • Paraphrase: And since we've spent more time improving the base systems in the program instead of doing one of the thing