Finding the Truth Behind Cable Modem Traffic Bursts?

Techi asks: "I help to support a small cable modem network in Kansas, and we keep having recurring problems with all the modems in a node bursting extreme amounts of traffic for a period of about 30 seconds. At the end of this 30 second period, the upstream port that the node in question is connected to dies under the pressure. We have recently implemented a fix to keep people from uncapping modems in the form of a config file update from our DHCP system. I know we could have done it differently, but it wasn't my decision. Does anyone have any idea what could be causing 70 or so modems at a time to suddenly erupt with outgoing traffic nonstop until the upstream dies?"

Comments

ah...

[President Chimp Toe]

one word: DOS

fp?

Re:ah...

[littlerubberfeet]

I am assuming by 'one word DOS' that he ment instead the acronym, Denial of Service. That would cause the uplink to be overloaded. Has someone infected your network with a worm??

Re:ah...

[Anonymous Coward]

Yeah, there's almost certainly a trojan installed that's sending denial-of-service attacks. I'd monitor inbound connections, or even reject known open proxies.

Try monitoring

[MarkusQ]

Why don't you try monitoring? Leave something tcpdump-ish on the line, saving the last however many packets until something dies. Then dust for finger prints.

I've seen something like this happen and suspect either dumb/misconfigured DHCP clients, an election process run amok, or some sort of ICMP flurry. No proof either way, since in this case I'm just a user & I just wait it out.

-- MarkusQ

P.S. I have noticed an interesting patterm to the timing though. You might try looking at the times / dates of past events to see if that suggests anything (and it can often suggest a lot).

Re:Try monitoring

[linzeal]

Program can cycle through variuous level 1 and level 2 network access configurations in attempt to subvert system design.

Possible spyware and/or application updates?

[ShaunC]

I don't know much about DOCSIS and DHCP, so if that's where the issue lies, my comments are probably off-base. However, I have to wonder if perhaps the spike you're noticing is the result of some popular program's "auto-update" feature, or spyware phoning home en masse. Many programs set themselves to do updates at certain times of day, and assuming most of your customers' computers have their system clock set within a reasonable amount of variance, it might be something benign.

For example, all the Macs on my network are set to query Apple's network time server at midnight daily. And on my Windows machines, Windows Media Player is set to check for updates weekly. The amount of traffic involved in either example should be minimal, but you never know what's borked. There was a story here recently about some versions of Windows and MacOS causing too much DNS traffic, so it could even be something at the OS level.

Is this a recent phenomenon? Brilliant Digital said they were going to activate their leechware in May, and May is now more than half-over. Maybe they've flipped the switch and all your users with KaZaAaAaAaA are now sending uberpackets to BD at predetermined times.

Are there any specifics as to where the traffic is destined? Is the traffic burst from all of the nodes going to the same host, or to the same port on multiple hosts? Are ports 25 or 119 involved? There's been a fairly nasty Hipcrime attack (usenet sporgeries) over the past few days, and spam is always a problem; both of these abuse broadband relays as much as possible. Lots of possibilities, I guess - would help to get some more details, if they can be provided.

Shaun

Re:Possible spyware and/or application updates?

[Techi]

The thing is, though, that this only occurs in one geographical area at a time, and our monitoring systems pick up constant traffic from every single modem in the node in question, so it almost seems more like a denial of service attack, or something on the upstream side of things...

Re:Possible spyware and/or application updates?

[T-Punkt]

Is it IP traffic?
If yes: Can you find out the destination of the traffic? If it all points to a single or a handfull IP#s DDOS is very likely. (a tool like ntop is handy to find that out).

Is it TCP or UDP? Are the sender adresses OK or spoofed? What service are the packets targeted at (port numbers)?

Some FPS

[FreeMath]

UT or Quake? Dunno...

Re:Some FPS

[larien]

Most online games don't use that much traffic these days; they have to be able to cope with 56k (or less) modems, so the amount of traffic is limited. It certainly shouldn't affect a cable modem network.

Re:Some FPS

[Antity-H]

Yep but in most recent games you also have a rate setting that allows you to use more bw ifyou have a better connection.
But having 70 users all knowing the rate setting and all going to play at the same time seems a bit weird.

Re:Some FPS

[Techi]

either way, we are watching this through RF, so it isn't just a bunch of traffic, it's all the modems in a network segment sending constant traffic somewhere near 512 Kbps.

Re:Some FPS

[larien]

Sounds like some kind of broadcast storm; e.g. a mass ARP request or something.

As others have suggested, you really want to run some kind of a network trace (e.g. tcpdump) and have a look. My guess is that it should show up what the hell is going on.

Re:Some FPS

[telstar]

Actually, I think your "sig" solves the mystery. The sudden burst of traffic must be happening when Google's spiders hit their own site.

You need to capture the data

[Yarn]

To me it sounds like what I've heard called an 'ACK Storm'.

It seems to occur when a switch somewhere gets it's MAC table corrupted somehow and starts squirting rubbish onto the network.

I accidentally caused one of these at my uni, by changing the MAC address of my netcard, it brought down the whole network for hours, the switch was continuously broadcasting the last packet it saw.

They never found it was me though ;)

Re:You need to capture the data

[JordanH]

• They never found it was me though ;)

Well, that is, until now.

Expect to be called in first thing next week to explain.

Re:You need to capture the data

[Techi]

would something like you describe use primarily ARP, or UDP packets?

Re:You need to capture the data

[Yarn]

This was 3 years ago; I'm long gone ;)

Re:You need to capture the data

[Yarn]

(I am assuming that cable is basically similar to ethernet, I have DSL)

The spammed packets are probably udp, although it isn't impossible for some other broadcast-type packets to cause this (I'm thinking netbios/netbeui)

Best thing to do (if it's possible) is install some kind of packet sniffer (tcpdump/ethereal for unixoids, dunno about other OSes) on a laptop and plug it in at various locations. Sometimes just unplugging the patch from the offending port for 5sec clears this kind of thing up, sometimes you need to reboot the switch.

also cap locally

[Stinson]

cap the traffic locally so users uncapping their modems doesnt make a difference. Most likely you have some equipment thats connected that supports bandwidth-throttling, if not (and i dont know why) go on ebay and grab your self a cisco router or something.

Competency...

[Jonny 290]

It's kind of interesting how somebody who professes to support a cable modem network had to have an Ask Slashdot to realize that a traffic monitor and/or iptables (or another firewall) program could solve his problems in a matter of minutes.

You *do* have a firewall, right, Network Admin?

Re:Competency...

[Anonymous Coward]

He never claimed to be the admin. Maybe admin just isn't doing anything about it...

Re:Competency...

[Anonymous Coward]

Clearly, not everybody who reads slashdot knows everything about networking. Unless his CMTS is running Linux, IPTables would be of little use. The problem may in fact not even be IP traffic. Many CMTS's cannot even understand layer 3 traffic and act as nothing more than a bridge between the customer's pc and the rest of the provider's network.

So mister critic, where in this scheme would you put your IPTables based firewall so as to protect the CMTS from bursts of traffic that disable upstream ports?

Sounds like a broadcast storm

[schon]

Sounds like a broadcast storm to me..

One (misconfigured) machine broadcasts data (say, NMB update) with a source address of the broadcast address - everybody on the segment replies, (which causes everybody on the segment, including the misconfigured one to reply again, ad infinitum) - result: segment meltdown.

As someone else pointed out, a traffic monitor would be your best bet - you don't need to capture all of it, just the first part, to see what's starting it up - then you can decide what to do.

Win2K worms?

[mnmn]

Beside Broadcast Storms, with my experience I can tell you there are IIS worms out there that ask for /winnt/command/something?something etc and these requests arrive in dozens at a time. I dont expect all cable modem users to use WinNT or Win2K but some of them have apparently reached the critical mass for this bandwidth eater to remain on the network and cause dips of latency at least for Rogers Cable here in Canada. Hope this helps

Forget trusting the modem

[fferreres]

Here's how to do it, without ditching the already installed modems.

|---- Linux box (1 )--- [Node]----(internet/whatever)
|
|
|
--|-------|--------|
[modem] [modem] [modem]


Make the Linux box limit the bandwidth for each modem (TCP shaping/QoS) and then, if a user uncaps, you can even automatically cut them off and alert the node which modems need to be recaped (DHCP)

Meanwhile everything still works as usual, only that the linux box drops a lot of "extra" packets from uncapped modems.

I left WinMX running for like 3 days straight

[hackwrench]

My web browser started having difficulties connecting early on and I had to reset the cable modem at the end.

Questions ?

[Martin Spamer]

Have you conducted a traffic analysis, connected packet sniffer ? What type of traffic ?

Are you certain the modems have not been compromised ? Are they all the same type?

I'd suspect a DOS attack by one of you customers pissed at being capped.

check DHCP lease duration

[cronack]

If DHCP lease duration is set very low on the DHCP server, it could be causing excessive IP lease renewal requests to be generated. I believe the client will try to renew an address when the halfway point of the remainder of the lease duration is met. So, the first renewal would occur at 1/2d (where d is lease duration), the second at 3/4d, and the third at 7/8d, and so on. At each renewal, there will be a burst of traffic (including broadcast traffic).

For example, a 4 day lease duration would cause renewals at 2, 3, and 3.5 days into the lease.

You may want to search your traffic logs for one particular client and see if its traffic follows this pattern. Otherwise, I agree with everyone who said "sniff the traffic".

Bursts

[Anonymous Coward]

Sounds like a DDS bot attack