How Demigod's Networking Problems Were Fixed

The launch of Demigod was troubled by piracy and networking difficulties, which publisher Stardock worked quickly to correct. They've now released a documentary that gives a detailed look behind the scenes of diagnosing and fixing those problems. It includes meetings, interviews with the devs, and part of the bug-tracking process during a frenzied 108-hour work week.

Comments

Nice summary

[Jurily]

Am I supposed to watch half an hour of videos before they tell me what they actually did?

Couldn't you just, you know, summarize it for us?

Awesome...

[Anonymous Coward]

...yet another Demigod Slashvertisement.

Professional Game Studio?

[Manip]

I've spoken with, and listened to, several game developers over the years and it just strikes me how different they have been to the guys on this. I'm watching the videos and it reminds me of my college days geeking it out with other geeky types and screwing around with code.

Very different from the professional environment that you find in a lot of studios these days. They often seem to be run more like a Hollywood Movie or similar to commercial software companies and less like a college startup.

But then again I've not played (or heard of) Demigod before this, and if they can produce fun games that's all that is really important. :)

Re:Professional Game Studio?

[TheTurtlesMoves]

When a game has problems and they don't publish details everyone bemoans that fact that they are so closed about it. "Just be up front and honest about it, we understand there can be problems..." everyone says. Then when some does that everyone starts crying out " Cheap bastards are just trying to get free advertising... ".

poor management

[timmarhy]

the ineptitude of their management can be seen in the 108 hour week. anything over a 12 hour day is wasted, and you NEED 1 day off a week minimum to recharge the batteries, otherwise you just find ways to waste time on the job.

i've been there, i'm working 50 - 60 hour weeks and i achieve more now than i did in 90 hour weeks.

Re:poor management

[SeeManRun]

I just watched the entire video, and have no problem with what they did. Working in a game company, and any software company really, you learn that there is no substitute for real users. Once you get thousands of users online doing no deterministic things, your architecture crumbles. These guys did whatever they could to fix it, and for that I commend them. Hell, I was tempted to email the CEO and ask for a job until I learned they are in MI. I am very impressed with their efforts and the fact they documented it as evidence for their fans.

What we need is ...

[The_Quinn]

A Government Solution:
1. Pass regulations that don't allow buggy releases.
2. Require dev studios to make games of each genre.
3. Give special legal powers to developers to unionize, preventing firings, and forcing employers to deal with them.
4. Give companies billions after they fail, and claim it was a failure of the free market.
5. Appoint a Game Czar.
6. Takeover the game companies, firing the company heads and appointing government stooges.
7. Transfer ownership of the company to the government, give a minority stake to the developers.
8. Make great games!

Multiplayer is not fixed

[dunezone]

No to be a troll or anything but they haven't fixed multiplayer. They've released several patches over the past few weeks that have fixed issues but they tend to introduce more problems then fix.

The game is good and I enjoy it a lot, but my god every time my friends and I decide to play we debate on rather we want to deal with all the connection issues. You spend more time waiting to get into a game lobby then playing the actual game.

Like I said the game is good, and it has its share of issues even outside of multiplayer but to say its fixed is long from the truth.

Re:Multiplayer is not fixed

[pantherace]

What's funny is when this article got posted on /., and last night, it was pretty much impossible to play online, due to something with their servers. For

The game itself is good. Stardock's ImpulseReactor is bad. Impulse isn't so great. The engine isn't really to blame (some fundamental bugs though)
I've played about 120 games online, and win about 50% of those.

Here are some of the problems, having played it since pretty much the first day:
In the initial release, they always tried to use NAT punchthrough. This includes when it wasn't needed. This put more load on their servers and made it slow to connect. Maybe piracy played some role, but given how it worked after it was 'fixed' I rather doubt piracy was the main problem, but more of an excuse.
There are some ISPs (and routers) that mess with port numbering for UDP receive. This screws up the P2P network connection.
Demigod/Supreme Commander speak a sandboxed lua in communication. Impulse sends and receives XML, and they wrote some translation layer in there, which was losing messages.
Stardock implemented some proxies for fixing some problems, unfortunately the proxies are in my opinion, buggy. I've been sitting next to two people, and the same person connects to two of us fine, then proxied to the other. Uhhh, wtf? Additionally, they seem to have a tendency to crash, fucking up the game.
According to Stardock, Stardock didn't write the NAT punchthrough, but licensed it from Raknet. Also, now Raknet of that denies it's used in Demigod. (And technically, it's not used in the engine, only the connection making, which is the real place where the BIG FUCKING PROBLEM is.)
Impulse needs to be smacked with even the GNOME HIG guidelines, Apple's, even CDE's. Seriously, wtf? (I know it's trying to look like the latest office. There are so many people that have problems with the interface because of that. Unless you've seen Office a lot you won't recognize that the orb in the upper left is a menu, and even if you have seen office, you will likely miss it because it looks like an oversize decoration. Disclaimer: I don't like Office's new interface style in the first place.)
Impluse's chat is a webpage, even though it's IRC, which requires IE's security settings to be set to default, it doesn't work if that's set higher.

Demigod itself:
Suffers from the problem of being designed with Microsoft's tools that have been focused on the Xbox, and using similar ideas. For example, running at the speed of the slowest isn't a problem when you have a homogeneous environment. PCs aren't homogeneous.
Has some issues with the UI and communications due to all computers running the sim, and having to wait for every other computer's packets.
Occasionally a desync (sim wasn't the same across all computers), which I have only seen about 3 times.
Crashes occasionally, usually on connecting to a port, due to a UPNP call. (This might really belong under Impulse.)

Plus sides:
Less so now, but Stardock people are commonly on their IRC.
If they are on, they generally try to be helpful, or such.
The game is *very good*, especially on a LAN (or the internet when it works). Remarkably balanced in my opinion, for a newly released game. There are a couple of things which I think might need to be hit with a nerf bat, but they aren't very many.

Oh, and while there is single player, it's not designed for it. For anyone wanting to play it, play a few games against the AI to understand the basic game, then DON'T PLAY AGAINST THE AI, if you ever intend to play humans. It teaches you bad habits. For example I played with friends against AIs a few games, getting something like a 33-1 K:D ratio against the 'hard' AI. The next games I played against humans, I got waxed.

Summary of Technical Reasons Stated In Videos

[The Real Nem]

I've summarized the technical reasons given by the three videos.

Executive Summary:

Seems like their peer-to-peer architecture exasperated otherwise common matchmaking and NAT transversal problems that should be expected and thoroughly tested when developing networked games.

Video 1:

• Summary: The game is peer-to-peer.

Video 2:

• Quote: "We're trying to figure out why users are being removed from the database, which keeps them from being connected, but their not really disconnected, but the server thinks they're disconnected."
• Quote: "...and if you put a return to just ignore the disconnect message it works fine".

Video 3:

• Quote: We're working on a problem in which something, for some reason, keeps telling the database that manages people's records that they're disconnected and we don't know why.
• Quote: So you're putting in the the debug messages?
• Quote: Yep. Just putting in some debug messages to figure out where these ... what is, like ... wh what ... first of who is sending what, um these disconnects.
• Quote: Right, okay, we'll be back.
• Quote: "One of the fields in the database are backwards, or something, on IP addresses because there are so many IP addresses and ports and sockets and stuff that I don't know anything about."
• Observation: They test a fix and it seems to work on several office PCs.
• Summary: The description of the fix includes faster lobby connection time and visual changes to each players ping in the game lobby.
• Paraphrase: How about in terms of the robustness of it, like, how much will people be able to connect?
• Paraphrase: Their games are significantly more intelligent now. Um, there were are lot of crazy things that were going on before and we've been able to simplify a lot of them.
• Paraphrase: What about proxy servers, do you think we need proxy servers now?
• Paraphrase: No I don't think we need proxy servers. It might be good for backup; we were tossing around the thought, I think you brought this up, with routing some traffic if we can thorough the host. That may or may not be possible.
• Paraphrase: Though we had to take care of the case if the host leaves the game we don't want the whole thing to fall apart.
• Paraphrase: Right, right. But it might be a solution for players with low ping times who, you know, can't seem to connect to anybody, but, you know, want to get in the game.
• Paraphrase: Right, that's true. Of course as a backup plan; so if the host left it would take out those people who have really bad connections. But based on what we're seeing, we think, this should handle symmetrical NATs now?
• Paraphrase: What's that?
• Paraphrase: Asymmetrical NATs; this should handle that now?
• Paraphrase: Uh, it's going to handle it better, yeah.
• Paraphrase: Well before it didn't handle it at all, so...
• Paraphrase: Right before it didn't handle it at all, before...
• Paraphrase: I mean if I have two IP addresses, will this work with it now?
• Paraphrase: I don't know that if you have two IP addresses this will work. Uh, this is going to help alot with firewalls and nats, however, um, it does a better job of piercing through the firewall.
• Paraphrase: Of course if you port forward to the right IP, then, from your router, then you're just set.
• Paraphrase: Yeah, that's not an issue.
• Paraphrase: So anyone who's technical will be fine, pretty much for sure, no matter on what kind of crazy connection they've got.
• Paraphrase: And since we've spent more time improving the base systems in the program instead of doing one of the thing