Halo 3 Causing Network Issues

Recently at my university where I'm a student and a sys admin, we have been experiencing some odd outages, in particular since the 25th of September. The outages seemed to occur between 8 PM and 12:00 AM — peak gaming hours for our dorms. It just happens that Halo 3 came out on the 25th of September. Upon further investigation we found that our network routers were shaping TCP packets, but not UDP. Once we applied UDP shaping as well, all network outages ceased. Gamers complained, but university students attempting to access network resources such as our UNIX clusters were satisfied.

Comments

Doubts

[nielsslein]

I'd like to see more proof before I go and blame Halo 3 for this.

Re:Doubts

[CrankyFool]

Dude, you didn't even have to read the FA, it was in the summary! The issue was that the routers weren't shaping UDP.

Re:Doubts

[Anonymous Coward]

Do not try to read the fucking article. That is impossible. Instead, only try to realize the truth: there is no fucking article.

Re:Doubts

[MindStalker]

No in large university networks you have to set bandwidth caps for all users, (pst your ISP does this now and sets it at whatever Mbps you buy). His router wasn't even looking at UDP traffic so a few users were getting more than they were assigned.

Re:Doubts

[KiloByte]

Yeah, I can see three explanations:

• Halo 3's network protocol is so abysmal that it needs a big chunk of bandwidth. I'm not talking that everything must be done The Right Way (ie, sending just what other players pressed and a checksum of the game's state like Doom1/2 did), but even sending the coords for objects you can see won't take more than a few KBs per second.
• the routers were buggy and crapped out after seeing more than X streams, counting every UDP packet as a separate stream (a moderately popular bug). As shaping fixed the issue, I doubt this could be the culprit.
• the whole univ having nothing but a slow DSL uplink or so. I don't know where the article's poster is from, but if that's a 3rd world country it's possible.

Somehow, not knowing anything about Halo, I suspect a combination of the first and third reason.

Re:Doubts

[arivanov]

Missed one - the routers crapped out on packets per second, not on bandwidth. This is probably the most likely one.

As far as the network protocols being abissmall you are about right. They have devolved over time.

Once past the stage of serial connections, the early gaming protocols efforts tried to use multicast+unicast or broadcast+unicast technologies to run peer-to-peers like networks where people truly played against each other.

These times are gone. It is all client-server now.

This explains the admin problem - I bet that most of the students were fragging each other silly together and played within the same map and the same game. That all ended up as a lot of client-server connections. This does not consume a lot of bandwidth on average, but it is capable of flatlining the network for short periods of time every time something interesting happened in the game because the data is tromboned back and forth across the same bottleneck many times. 1 student moves and the server sends the data to 16 others, and so on. Essentially this is a form of amplification/positive feedback loop. If the same students were playing games with other people located elsewhere the effect would not have occurred.

This is a classic example of devolving and microsoftization of the gaming protocols. If the game was running locally using broadcast+ unicast or multicast_unicast to inform all local participants and only one dedicated hypernode checked what is going on outside the small "local world" there would have been no bandwidth/pps problem.

Re:Doubts

[garbletext]

Yet, with TCP, the OS forces as least some congestion control, something that game programmers typically completely disregard. TCP has a lot more than congestion control and packet size that makes it a poor choice for games. It's overhead includes acknowledgements, error correction, and things like guaranteed packet order and retransmission of lost packets. For a game, especially a FPS, you want to fully utilize available bandwidth. If a packet has to get retransmitted, or held because it's not in order, it's going to be stale by the time it arrives, so you might as well ignore it. UDP's connectionless nature means that clients can just spew packets as fast as their link allows.

Re:Doubts

[Anonymous Coward]

If either of you thought about this instead of just acting superior, you'd realize that they tried the "p2p" method until cheating became widespread, and then had to shift to a more complicated (but less efficient) server model that could run checks on the data and make sure nobody was cheating.

Highly paid professionals know more than you, don't worry.

Re:Doubts

[maraist]

It wouldn't be the downloading of the game, as that's TCP not UDP, and it very likely isn't connectivity to the internet, because that would have nothing to do with access to the central UNIX servers - Local networks are always faster than internet connections, thus, unless the summary of the article was lying, it was purely a HUB/Switch UDP network hog, and any good sys-admin of a massive local network would have done the same thing.

Consider a percentage of 10,000 hosts playing the game.. All sending multi-cast UDP packets. You have say 1,000 hosts sending several packets per second to every network that multicast is configured to broadcast to. So you have several thousand packets per second going to at least every network card in the dorms, and probably to/from all the student-accessible computer labs to boot.

While the central servers most likely were on isolated (non-broadcastable) networks, all the client-host accessible points were choked.

Re:Doubts

[Anonymous Coward]

For a game, especially a FPS, you want to fully utilize available bandwidth.

I don't know of any FPS programers who would want to fully utilize available bandwidth. If they did, it wouldn't scale and their game would suck.

What they do want to do, is minimize latency to keep character movement smooth and accurate to give the perception of real-time immersion and do that with a minimum of bandwidth usage, so that their game will scale to a practical number of players. Minimizing bandwidth usage helps improve latency and scaling.

Re:Doubts

[Blakey Rat]

Ok,

1) Saying that you know more about game networking than the crew at Bungie, the crew who have been making games with LAN and Internet play since freakin' Minotaur in 1992, that's just plain stupid. Let's see your credentials if Bungie's coders are so stupid.

Client-server is used because it's the only way to provide fair "hit negotiation" (the server always decides who hits who-- play Mechwarrior III for an example of a game without this) and it prevents cheating, since each client sees only what it absolutely needs to see to function.

2) Never use the word "Microsoftization" again.

Re:Doubts

[Mr. Freeman]

"Ultimately, there is typically only one connection to the net that is provided by the school, and if it is all being used up fine, but it should at least be something that is academically related or general net use."

Yeah, how dare college students try to use the network for gaming.

Seriously though, this is something that A LOT of people miss. College students are living at the college 24/7 for at least 4, more if they go to grad school, years.
It's fine to say something like "No porn, games, file sharing, etc." at work or at a high school where business and education are the only things that should be happening. Why? Because the employees/students should go home and do those things on their own Internet connection.

But that doesn't work in a college. The students can't just go home to their own Internet connection, they're stuck at the university with the network available in the dorms. The universities need to buy more bandwidth instead of throttling the speed down to levels non-suitable for gaming. There will be a time in the near future where Internet bandwidth available to each user will be a large portion of the decision process for students applying to college. Right now I'm applying to about 5 different colleges that are 99% identical, if I'm accepted into all of them then I'm going to have to decide between them based on which one has the greenest trees, is closest to a bar, has the best cell phone coverage, and which one has the least amount of throttling on the Internet connections in the dorms.

Re:Doubts

[JNighthawk]

Thank you *very much*. I'm tired of reading people's bullshit, thinking networking an FPS is easy and shouldn't require much bandwidth, while making sure that no one can cheat easily.

To those that think it is: try it.

Re:Doubts

[cpritchett]

If your university did buy more bandwidth, I can assure you the quality of the students gaming experience didn't cross anyone's mind. It's not quite as simple as the network admin calling up the local ISP and going to the next tier.

Aside from the argument that the bandwidth is for educational use, trying to support gaming in our dorms would be a full-time job based on the number of calls and emails we get on a daily basis about it.

People use to be all "well that sucks, i'll get over it." Now I've got parents calling wondering why Johnny can't play his favorite MMO that he "really, really likes to play." I seriously thought that a guy that came to my office a few weeks ago was going to hit me because he couldn't play FFXI.

The worse part though, is when a group of students try to fake network problems in order for us to fix Johnny's issues, because, apparently, we don't monitor the network at all until 20 people on the same floor complain.

If they really need their fix, they are more than welcome to get a cable modem from the local cable company. I can't vouch for all Universities, but nobody here is forced to use our LAN.

Re:Doubts

[mcrh]

Never use the word "Microsoftization" again.

Why? It's a perfectly cromulent word!

I didn't RTFA...

[A beautiful mind]

...but at least now I have the excuse that there is no FA.

I must be new here

[OAB_X]

What a remarkably useless story.

Re:I must be new here

[Romancer]

Probably the facts:
1. There is no link to an actual article or any other related or corroborating information source.
2. The summary admits that certain router features were not enabled, and simply enabling them fixed the "problem".
3. The title of "Halo 3 Causing Network Issues" has so far been only superficially associated with the "problems" they were having by giant leaps of non technical assumption.
4. This is another KDawson post/nonstory.

Re:I must be new here

[Jarjarthejedi]

Well it's remarkability has been proven by the number of comments.

It's uselessness lies in the fact that it was a mis configured router causing the problems, and configuring it right fixed it. 99% of people on /. don't need to know this (because they're not running a university network, have configured their routers right, or were expecting this sort of thing from the largest game release in history) and the other 1% didn't read the article.

I'm not sure about the story part, though I guess any yarn told by an individual about an event qualifies.

Therefore the tale is a story, is remarkable, and is useless, ergo 'What a remarkably useless story'

Honestly this one irritates even me, and that's saying something. I'm usually fine with non-stories, as I'm bored enough to not care. Talking about how you didn't properly set up your university network and didn't prepare for the largest game release in history (I'm sorry, but if you're running a network for college kids, why in the world weren't you ready for Halo 3? You should know for a fact that it's going to be huge, if you don't then you're living in a cave) doesn't make me want to like you, and then mentioning that your response was basically to shut down the kids ability to play proudly, like 'Look at me, I'm so smart, I killed a bunch of kids fun because I couldn't think of any of the other hundred ways to solve my problem and would rather just shut them down', well that makes me want to punch you in the face for being dumb, and proud of that dumbness.

And?

[VeteranNoob]

So, poor network design caused the network to become saturated. QoS rules were applied to UDP, as they should have been, and the problem has gone away.

Where's the story?

Re:And?

[Blakey Rat]

It's just posted here so we can get reams of replies talking about how over-rated Halo is, how much the Xbox (made by Microsoft!) sucks, how great Nintendo is in comparison, and how games used to have more "fun" back in the olden days. The network problem is entirely secondary.

So it's pretty much like every other Slashdot Games post.

Re:And?

[MightyMartian]

Indeed, the story should be "More dipshit net admins with fancy-ass certificates but no brains fuck up QoS once more."

So what?

[JustShootMe]

Guy had a network problem. Network admins found the source of the network problem. People who caused the network problem complained, everyone else was happy. This wasn't even a technology problem, it was an oversight in the configuration of the routers/switches.

How exactly is this worthy of a front page article on slashdot?

Hey, guess what. The other day I had a process that stopped working. Thinking quickly, I figured out what was wrong and fixed it. Everyone was happy. Do I get a front page article too?

Sheesh. Congrats for doing your job, subby.

(I know this was a journal entry and subby had nothing to do with it getting greenlighted, but seriously, wtf?)

Re:This is a story?

[blowdart]

It has a "blame Microsoft" angle, what more do you want?

Wait a second.

[Entropius]

You're degrading time-critical but relatively low-bandwidth traffic intentionally in order to improve responsiveness for some ssh connections?

Granted, Halo 3 is less important than Prof. Smith's Monte Carlo, but the fact that you have to do this at all means that you need more capacity. Plus it's damn rude to the students: "Oh, they're doing something new that we don't degrade! Ah, well, just degrade student UDP traffic too, that'll fix it!"

I'm not saying that transfer limits are a bad idea -- someone downloading 100GB/month and saturating a line needs to be told off, certainly -- but if a bunch of low-bandwidth gaming traffic from the dorms kills the network...

Don't forget that those guys in the dorms playing Halo pay lots of money to the university, which pays for the network.

If I knew what uni you were at I'd seriously consider adding my (meager) 256kbps upstream to the load by writing a script to refresh your homepage over and over.

Re:Wait a second.

[Bert64]

But this is a university network... Accessing SSH on university systems so that students can do their work is far more important than playing some games.
The network is there for research purposes, so thats students can do the research they need to pass their educational courses. Any traffic that facilitates the educational courses of the university should be prioritised, and anything else should get whatever bandwidth remains. And those games should be grateful they can play online games at all, the university is not obligated to provide them a connection nor allow them to play games on it (they could easily filter gaming traffic completely).

Re:No sympathy...

[damn_registrars]

Sadly enough, I know someone who chose his 4 year college based on ping times to his favorite Quake servers...

You'll probably be shocked to hear that he graduated by some sort of formality...

Re:I JUST POOPED A LARGE TURD

[JustShootMe]

Congratulations. Put it in a journal and you might get on the front page too.

Re:No sympathy...

[timeOday]

You know, I don't think I have any sympathy for the upset gamers on campus networks.

I don't know about sympathy, but it does sound like the university network is inadequate. I'd hate to see these admins in charge at FedEx... "there's just too darn many packages this Christmas, and most of them aren't important business documents anyways. I know, let's just throw out anything in wrapping paper and ribbons!"

Crazy?

[spykemail]

Wait a minute, you limited network usage for gamers in favor of academic users? Sounds like a pretty shitty school if you ask me. Everyone knows that school networks are for three things:

1) Downloading music and movies illegally.
2) Downloading pr0n.
3) Playing games, even crappy ones like Halo 3.

As you can clearly see homework and research are not on the list...

Ah, Doom

[Dachannien]

Way back in the day, Doom's first implementation of multiplayer used broadcast packets to communicate amongst client machines. The university I attended was, at the time, home to the world's largest unswitched Ethernet. Doom's popularity led to the swift collapse of the entire network on a regular basis, since a broadcast packet would result in a response from every other machine on the network.

id shortly thereafter patched the game not to use broadcast packets anymore. Once the cause of the network failures became apparent, playing the unpatched version of Doom became grounds for having your Intargopher turned off (we didn't call it the Intarweb back in those days, ya whippersnapper).

My school

[Beowulf_Boy]

My school has a game design major, that I'm a part of.

The internet in the dorms was shittacular. Horrible horrible service, and we had to pay 30$ a month for it.
And, the IT department, when called out on this bullshit, couldn't even give us a break down on how our money was being spent.

So, 3 years ago me and several friends sent an email out to everyone of importance around campus calling them out, basically saying it was bullcrap they advertise themselves as being all advanced at this university and having this gaming major, but the gaming major students can't even get online half the time in their dorms to play....games.

Within several hours, most faculty was writing back and agreeing with us. We showed up at a meeting, and the head of IT didn't have anything together at all.

Basically what happened was for a few months we could opt to be on a seperate network through the engineering department that wasn't managed by the IT department, but rather a professor in his spare time. And gasp, this network was far far superior and less buggy. It had 50% of the computers on campus on it, and 0% of the budget, yet still managed to be far more reliable.

Then, after the next quarter passed, we were allowed to get outside ISP service in the dorms. Alot of my friends get adelphia internet access. I just chose to move off campus, I was tired of dealing with it. You still had to pay the IT department for their crummy connection, on top of paying another ISP.

Re:My school's network sucks :/

[Xemu]

Do remember that this is a SCHOOL... It's purpose is to educate the kids, not to facilitate them playing games

Humans have educated others through games since forever. Even chess is a strategy game meant to teach others about warfare. The military uses games even today to train soldiers how to behave in combat.

Schools would be better if they used more games to educate their students.

A smart professor could use Halo3 to teach about gender issues or the biology of human perception.

Its a Journal Entry

[mobilesteve]

Journal written by fender177 (1125877) and posted by kdawson on Sunday September 30, @03:17PM

I don't think this article was submitted as a story by the author. It looks like fender117 just posted a little story in his slashdot story, and kdawson stumbled upon it and decided to post it to the front page for some stupid reason.

My Halo 3 story

[suv4x4]

Recently I purchased one of the limited Halo 3 packages. It looked great. But the game wouldn't start! Upon further investigation I remember I microwaved the disk for 3 minutes for no particular reason whatsoever.

I'm still pissed off though. Nowhere on the package it didn't say specifically about microwaving Halo 3.

Re:TAGGED AS KDAWSONSUCKS

[Nicholas Evans]

I must agree with this Coward. Some random university's network didn't have traffic shaping set up correctly. So? This is somehow newsworthy?

UDP Packets

[LordMyren]

Just for the record, dropped (shaped) udp packets are not recovered. TCP/IP notices dropped packets, has them resent, and automatically lowers the connection's transmission rate, whereas with UDP you're just tossing EMP's into people's datastreams. UDP/IP is much more primitive, and relies on application level consistency checks, which, for the record, almost never ever ever monitor packet drops & throttle themselves down when packets start dropping. Thats why most packet filtering systems simply de-prioritize UDP and will not drop UDP.