Georgia Tech's ShaperProbe Detects ISP Traffic Manipulation

An anonymous reader writes with this excerpt from Ars Technica: "Two researchers at Georgia Tech can tell you exactly how American ISPs shape Internet traffic, and which ones do so. Bottom line: of the five largest Internet providers in the country, the three cable companies (Comcast, Time Warner, Cox) employ shaping while the telephone companies (AT&T, Verizon) do not — though that fact is less significant for the user experience than it might first sound."

Comments

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

When I pay for the service and it's not in the EULA, 3..2...1...STFU

Re:Cue the cable company bashing in 3...2...1....

[Squiddie]

Their network runs on public land. They are also granted exclusivity by local governments. I think that regulation is in order.

Question

[bloobamator]

Is shaping the same as throttling?

Re:Question

[Anonymous Coward]

no, shaping is done per-protocol, and throttling is done per pipe

Re:Question

[ozmanjusri]

Is shaping the same as throttling?

Sort of.

Online, shaping and throttling are something network companies do to customers. In meatspace, throttling is what customers want to to to network company executives.

HTH.

Re:Question

[Ungrounded Lightning]

Is shaping the same as throttling?

Shaping is when they give you a rate and enforce it. (The faster burst at startup is because you had accumulated some credit by not using your bandwidth in the immediately preceeding time.) There may be separate shaping mechanisms for different protocol families and there may also be shaping on aggregates - like total bandwidth across multiple users of a common DSLAM.

Throttling is when, after they notice that you've used a lot of bandwidth lately, they turn down the rate on the shaper ("traffic manager").

Shaping is mainly about things like keeping protocols from interfering with each other (by giving different classes of them separate allowances) and avoiding congestion and queue-too-full latency (by limiting the traffic sent to a following box to the amount it can handle.)

Throttling is about keeping a user's resource consumption down by slowing him down after he's run fast for a while.

Re:Question

[shutdown -p now]

Surely we could think of some creative shaping of said executives as well - preferably applying shaping before throttling.

Re:Question

[Anonymous Coward]

Would that not lead to queens or transexuals?

Re:Question

[Anonymous Coward]

Is shaping the same as throttling?

is "asking stupid questions instead of googling two simple definitions" the same as trolling?

Re:Question

[ProfessionalCookie]

Pretty sure the internet is made of tubes, not pipes.

Re:Question

[AmiMoJo]

Not quite right.

Throttling is where they simply slow down your entire internet connection. Typically when you download more than a certain amount of data the ISP punishes you with throttling. Kind of like going to McDonald's and ordering a Supersize Big Mac meal, then a member of staff comes over and punches you in the gut as punishment for overeating.

Shaping is where the ISP tries to slow down certain traffic to give priority to others. The typical use is to slow down P2P and large downloads so that web browsing and Skype get priority. There is a bit of an arms race between software developers who try to encrypt or obfuscate traffic and ISPs who look for ways to defeat them. Going back to the McDonald's analogy it is a bit like them giving you a 1mm diameter straw to drink your coke through, while allowing you to eat the Big Mac and fries as you like.

Of course in practice ISPs both punch you in the gut and give you the 1mm diameter straw, and then spit in your burger for daring to ask for the meal you paid for.

Re:Question

[L4t3r4lu5]

It's car analogy time!

Shaping is like putting a bus lane / car pool lane on the motorway / freeway; Buses and car pool drivers can move through quicker, at the expense of car traffic having one less lane on the motorway, much like VOIP would be given priority over BitTorrent, or somesuch, but the cars and buses are all capable of going at maximum speed (should traffic allow).

Throttling is like variable speed limits. In the interest of keeping traffic moving freely across the whole motorway, the speed of heavily trafficked areas is slowed down so it doesn't cause congestion. 70MPH becomes 50MPH in the same way that 10Mbit becomes 2Mbit.

Data caps are like a bastard child of toll roads; You've driven a certain distance on this road which is covered by vehicle excise or fuel tax, now you have pay a toll. To travel further on this road, you pay more tolls. You can drive only so far each month on the toll roads for free.

HTH.

Re:Question

[ArisGT]

A better analogy is during rush hour they have those traffic lights on the on ramps controlling the rate of cars entering the interstate.

Re:Question

[DJRumpy]

According to TFA, the shaping being done isn't protocol specific (although it can be). They equate it to a bucket of tokens, where you start your transfer with a full bucket, and at specific intervals, you must use a token to transfer X amount of data. What that means is that when you start your transfer, you would have full bandwidth until you empty your bucket, after which you have to wait for the bucket to be filled again to continue your transfer. This has the effect of giving you a full pipe during your initial transfer (burst), but that will quickly turn into a more metered throughput according to whatever policy they have applied to your account.

From TFA:

Traffic shaping hardware generally relies on the concept of a “token bucket.” Traffic management hardware will generate a digital token for each Internet user at a predetermined rate. These tokens fill a virtual token bucket; transmitting packets over the Internet removes tokens from the bucket. If the bucket empties, no more data can be transmitted until a new token is deposited.

The practical result is that the user sits down to her computer with a full token bucket and can immediately blast data through her connection as fast as the connection can go. But after some interval of time, usually measured in seconds, this sort of full-throttle data transmission empties the token bucket and the user is now limited to transmitting at the token generation rate.

Re:Question

[Anonymous Coward]

Is shaping the same as throttling?

is "asking stupid questions instead of googling two simple definitions" the same as trolling?

'NO' to both.

Re:Question

[Ungrounded Lightning]

Throttling is where they simply slow down your entire internet connection. Typically when you download more than a certain amount of data the ISP punishes you with throttling.

I note that this can be accomplished in a hierarchical traffic manager / policer: The first level of the hierarchy might be each of several classes of traffic to (or from) your machine, while the second level might be the aggregate composed of all of this traffic. Throttling would consist of turning down the second level's programmed rate from its normal setting (which presumably would be your line rate, where it prevents buffer bloat in the last hop machine, or from your contracted max rate under normal circumstances).

Of course ISPs which don't have fancy hierarchical traffic management boxes can also apply throttling by telling the DSLAM (or other last-hop box) to downsize your link.

not all shaping / policing is bad.

[Anonymous Coward]

I work for a college, and we shape / police traffic to / from the Internet.

This was a necessity on our 3Mb link of many years ago, but has still been useful on our 1Gb link of today.

This policy has greatly improved the user experience. Interactive protocols have low latency, bulk transfer protocols get sent to the end of the line. Where we do slow down things, it isn't really noticed by most folks. After first implementing this many years ago, we immediately got positive feedback. Now it is just "how things are."

Hell, I shape / police traffic at home to my cable modem. VOIP and interactive ssh are still usable even with huge downloads going on now, and users hammering the public wifi I provide to my neighborhood.

Re:not all shaping / policing is bad.

[Anonymous Coward]

Exactly This. As a sysadmin for a small ISP we do the same. It's not about limiting traffic. It is about prioritizing to provide the best experience and utility.

Re:not all shaping / policing is bad.

[Urza9814]

It's great that it improves the quality of service...but here's the problem: My neighbor and I both pay for the same service. So why should he get better service because he's using a more interactive protocol? Why should my file download slow down because my neighbors are all on VoIP calls? Depending on how the traffic shaping is set up, isn't it possible that someone who's paying the exact same amount I am for internet service could be getting several times more real data throughput simply because of what protocols they're using? Granted, it's a bit unlikely, but it certainly seems possible.

Re:not all shaping / policing is bad.

[Idbar]

Hum... that's the issue. You're providing for a bunch of students that don't have a contractual agreement with you. Now imagine that the company you contracted the 1Gb link to, started shaping your traffic. That eventually comes to you, not receiving the 1Gb you paid for.

I understand you sometimes need to shape the traffic to prioritize services you want to perform better. But these companies are getting their money offering services they don't completely provide: First they charge you for access they can shape. And second, they put a cap just in case you get away with it.

Again, I understand your work case, because if you don't pay for it, normally you tend to abuse it. But if you pay for it, why would they need to mess with your traffic?

Re:not all shaping / policing is bad.

[Anonymous Coward]

Why should his voip call sound like Mr. Roboto because you're downloading the latest stuff off BitTorrent? Don't like sharing bandwidth with your neighbor? Plunk down $$$ for dedicated pipe.

Re:not all shaping / policing is bad.

[betterunixthanunix]

why should he get better service because he's using a more interactive protocol?

"Better service" has multiple definitions here. I would say that interactive sessions get "worse service" when the latency increases, perhaps because you have some huge download that is eating away at bandwidth that an interactive session could be using. Do you want your VoIP calls to get choppy, or do you want your downloads to be a few minutes faster?

Re:not all shaping / policing is bad.

[Anonymous Coward]

The only traffic shaping an ISP should be doing is a fair queuing of packets, so each customer gets same backbone bandwidth. Then how the customer chooses to prioritize that traffic should be up the them.

Of course I haven't found a customer-level ISP like that.

Re:not all shaping / policing is bad.

[Anonymous Coward]

It's not always about throughput.

If I'm in WoW, I need my packets NOW. Not later, NOW. I also don't need as many as you do, by like, two orders of magnitude.

Generally, the "interactive" packets are pretty damned small compared to the bulk ones.

My biggest gripe is when "traffic shaping" means "bittorrent mysteriously coincides with your internet becoming ludicrously sluggish". That's a peeve for sure.

Re:not all shaping / policing is bad.

[Anonymous Coward]

Does your college run the last mile through public land that was taken or "borrowed" from the home owners? Does your college charge a monthly fee to the same owners of that land that was taken? Does your college negotiate a monopoly with the local government to ensure they are the only ones that can use that last mile? Effectively making the residents pay for that last mile over and over again and still only have one choice of internet provider over those lines?

I'd rather pay the local government for the last mile and let the providers bid for my individual service over that last mile. The last mile would cost me the same exact amount and I would have a choice of providers.

Re:not all shaping / policing is bad.

[Anonymous Coward]

I think it equates to the freeway, you have small cars that are tiny and capable of getting to their destinations quickly, then you have huge semi trucks that trundle along. Should it be fair to only have 1 lane and have the small cars get stuck behind the huge slow trucks, no. So you create lanes and ask that slow traffic keep right so the rest of the internet can move along.

Re:not all shaping / policing is bad.

[Anonymous Coward]

I want my line to be 3mps always not up to 25 or 200mbps that my ISP promises. I have ADSL not cable for that reason. It is about honesty and a minimal level of service irregardless of the type of data I am sending over it. If cable companies were honest about providing service I might use them. They can't provide the level of service they suggest. Therefore I won't use cable for Internet. I avoid it for TV too.

Re:not all shaping / policing is bad.

[Local ID10T]

Exactly This. As a sysadmin for a small ISP we do the same. It's not about limiting traffic. It is about prioritizing to provide the best experience and utility.

And that is fine -to a point. As long as shaping does not become an excuse to limit traffic. In the case of the larger ISPs at least, it has become a choice -limit traffic in order to maintain higher profit margins vs invest in increasing capacity.

Re:not all shaping / policing is bad.

[Anonymous Coward]

Yeah thats great. You are a private school who did NOT sell all its customers an 'unlimited' connection which was a flat out lie.

100% completely different than the case here.

Where they DID claim it was unlimited.. And it's not. They're lying to their customers. And stealing what they paid for.

Re:not all shaping / policing is bad.

[Anonymous Coward]

Same AC here. No we don't limit traffic by destination but we do prioritize interactive traffic. It's a way to keep the gamers and browsers happy while not throttling the BT junkies.

Re:not all shaping / policing is bad.

[Anonymous Coward]

Same AC here again. It's not a matter of throughput. It is a matter of latency and priority. On our system, over the course of 3 seconds (eons in the domain we are talking about) your neighbor will have packet priority for his VOIP session. However, your donkey porn will still max out your pipe throughput. It's just that his packets are delivered in a more timely fashion.

Re:not all shaping / policing is bad.

[Anonymous Coward]

You understand very little of shaping. Prioritization and fair distribution are two mutually exclusive domains. Under proper prioritization (QOS), you can fill your pipe (FTP BT mail download et.al.), but still be at a lower priority than a truly interactive critical data stream(gaming VOIP SSH). Both can max their pipe but the high priority packets need to go when they need to go or problems will surface.

Get a clue first and you will be qualified to find a good ISP.

Re:not all shaping / policing is bad.

[Corse32]

You have a dedicated 3Mb/s ADSL connection? That kind of thing costs hundreds a month here in Australia, usually the consumer grade plans are fine, cheap and very fast, but the guarantee to always supply you a certain sized pipe costs way way more. Having that bandwidth sitting there doing nothing when you're asleep, just in case you wake up and need your 3Mb/s out of nowhere costs the ISP constantly. Most business plans I see now just quote the "up to" speeds, I guess because in reality most people are fine if their connection drops down a smidgen once in a blue moon, and is blazing fast the rest of the time.

Re:not all shaping / policing is bad.

[Falconhell]

I have never had less than 8Mbits/sec on my consumer grade plan with Internode, makes at 15.

Re:not all shaping / policing is bad.

[Dhalka226]

But these companies are getting their money offering services they don't completely provide

I hesitate to use words like "every," but I can think of no ISPs who advertise their plans as anything other than "up to X Mbps." Why you believe that not getting that is somehow a service they don't provide eludes me, particularly when the reason you are not receiving it is legitimate shaping activity*.

And second, they put a cap just in case you get away with it.

I tend to agree with you on this one, for a lot of reasons. In particular I object to the limits being buried in the terms and conditions somewhere rather than being open about it. Stil, your conclusion is that you're paying for it and they're not delivering it which is false. You're paying for what your contract says you're paying for, and somewhere in that fine print--at least nowadays--is a limit.

There are a lot of things to object about the state of Internet service in the US. The limits are going to quickly become a burden on growing Internet-based industries. The monopoly status of most providers means they are uninterested in serving their customers or competing for their business. They oversell their services far more than they should and pocket dollars intended to improve their networks. That's off the top of my head. "I'm not getting what I pay for," meh. Not so much.

* Admitting that not all shaping activity is legitimate, especially based on source/destination.

Re:not all shaping / policing is bad.

[Lehk228]

your neighbor isn't getting better service, your neighbors VOIP packets are getting more effort to delivering them without excess latency while your torrent packets show up where they can be fit in, but get a lot of bandwidth

Re:not all shaping / policing is bad.

[cerberusss]

because if you don't pay for it, normally you tend to abuse it. But if you pay for it, why would they need to mess with your traffic?

Usually when I'm on public wifi (even if paid for), I tend to put downloads on a slow trickle, overnight if possible. Ye olde wget is awesome for that:
wget --limit-rate=20k http://www.example.com/bigfile
will limit the download speed to 20KB/s. All platforms have wget available.

(By the way, why does Slashdot make text URLs into hyperlinks even in code sections?)

Re:not all shaping / policing is bad.

[Anonymous Coward]

That's sort of like "why should that guy get to use the copier before me just because he's only got one page?"

Interactive protocols tend to suffer by latency, but have low actual bandwidth demands compared to bulk transfers like BitTorrent. Logic dictates you assign higher priority to the data that will take less time to handle.

The problem occurs when this priority system is abused by giving high priorities to non-interactive premium content like streaming TV, or actively throttling BitTorrent beyond what is needed because the content providers want it.

Re:not all shaping / policing is bad.

[arkhan_jg]

There's nothing wrong with QoS and putting some traffic at the front of the router queue. It just means light, latency important packets don't get drowned in a torrent of bulk traffic that doesn't really care what order the packets arrive in. But you still have the same amount of capacity for traffic at the start as the end, you're just determining which app gets to use it first, and which app you're prepared to throw away packets for first if necessary at heavy load.

The article is not about that, but bucket throttling. I.e. each user is throttled to a certain throughput, but have a small 'margin' where they can go faster. This makes the service feel snappier when browsing the 'net or other irregular small bursty traffic, but makes it much slower when doing bulk file transfers or streaming. And of course, the size of the bucket, and the rate at which it gets filled are adjustable based upon how congested the network is - and even based upon your usage pattern.

This isn't traffic shaping in the good sense. This is traffic shaping as in throttling individual users when they're using more bandwidth than you can afford them to shaping. Even the latter is acceptable on a small network with limited bandwidth so it's at least minimally usable - but needing it is a clear demonstration that you don't have enough throughput for all your users. When you're supposed to be in the business of selling bandwidth, that's bad.

Re:not all shaping / policing is bad.

[maxwell demon]

It doesn't hurt your download if there's half a second nothing, then half a second double speed. The average data rate is still the same. However it massively hurts the VoIP connection.

Re:not all shaping / policing is bad.

[Joce640k]

Um, having a low latency doesn't necessarily affect download speeds for large files.

Internet Protocol has this thing with sliding windows, etc., to deal with that.

Re:not all shaping / policing is bad.

[wye43]

What you describe is indeed good, but it should not be done with shaping. It's a functionality implemented at another layer by the TOS flag (http://en.wikipedia.org/wiki/Type_of_Service). While there are some bad apples out there, any decent TCP/IP stack(Linux has it since the stone age) has implemented this at least in its most primitive form (interactive/bulk cases).

Re:not all shaping / policing is bad.

[Anonymous Coward]

I want my line to be 3mps always not up to 25 or 200mbps that my ISP promises.

So purchase a dedicated circuit, that's what they are for. You purchased a variable rate plan and then bitched about not getting a fixed rate... doesn't make much sense now, does it?

I have ADSL not cable for that reason

Then you're confused. ADSL and Cable is the technology that carries your traffic from the local datacenter to your house. Most bandwidth/throttling/congestion issues happen on the provider's transit and core network, not on the "last mile".

Sounds to me like the cable company in your area is a piece of shit. Guess what- the DSL company in my area can't support the speeds they advertise. My point is that instead of blaming the technology, or making large-scale sweeping assumptions, you should realize that a properly maintained network is not going to have problems supporting advertised speeds, and that ANY company can run a piece of shit network which will not.

Re:not all shaping / policing is bad.

[Anonymous Coward]

If I'm in WoW

If there's anything that is more of a waste of bandwidth, it's this.

I need my packets NOW

No you don't. I'd rather VOIP and similar communication protocols be highest priority. Not your amusing misuse of resources. Nevermind your entitlement on your entertainment that you so demand.

Re:not all shaping / policing is bad.

[Cimexus]

I don't think he's saying he actually has a guaranteed 3 Mbit DSL connection (i.e. a plan with an SLA). What he's saying is that ADSL is generally subject to a lot less contention than cable.

Which is true - with ADSL you have the bandwidth between you and the exchange/DSLAM, at whatever sync rate you have, all to yourself, whereas with cable you are competing with a dozen neighbours on the local loop. Once you hit the DSLAM and enter the ISP's network you are subject to the same contention as everyone else, of course, but generally the ISP-level contention (assuming a vaguely decent quality ISP) is a lot less significant than the immediate contention due to the last-mile cable being a shared medium when using DOCSIS (where you could have the best ISP in the world, but it would mean little if you are sharing the cable with a bunch of other leechers).

End result is that if you buy a plan from an ISP with 3 Mbps sync speed, you generally do get that speed. Similarly here in Australia - unless you are unfortunate enough to be on a RIM with poor backhaul (i.e. provided you actually do have a direct line to the exchange), you aren't sharing your line speed with anyone else. Cable OTOH is different in that there isn't really a 'sync speed' as such. The speed you get at any given time depends on various factors and will fluctuate quite a bit.

ADSL of course is still advertised as "up to X" in Australia, because you aren't usually sold plans based on their sync speed. You buy a plan with a certain download allowance and on most plans you'll get 'whatever sync speed your line can manage'. This could be as low as a megabit or two, or as high as 24 Mbps (for ADSL2+) or 8 Mbps (for ADSL1). The ISP does not guarantee you any particular speed. But in the US, DSL plans are sold based on the speed tier, rather than the download allowance. 1.5 Mbit, 3 Mbit and 6 Mbit are common speeds. If your line can support 20 Mbit but you only buy a 1.5 Mbit plan, that's all you'll get. Similarly, if your line can't manage a 6 Mbit speed, the ISP won't sell you that plan. So in all cases, you get the exact sync speed you bought - hence the perception that DSL is a 'guaranteed' speed. It's not that the actual throughput speed is 'guaranteed' - it just isn't contended at a local level like cable is.

Re:not all shaping / policing is bad.

[AmiMoJo]

ADSL ISPs in the UK are pretty much forced to shape and set ridiculously low caps because of the amount the owner of the telephone lines and other equipment, BT, charges them for access.

BT initially charged on a per-user basis, but then they switched to charging by the amount of bandwidth they supply to the ISP. So where as before an ISP paid £X per user regardless of how much data they consumed, under the new system they would pay for say a 144Mb "pipe" and choose how many users to put on it. Of course BT hiked prices too so ISPs had to fit more and more users onto each pipe and contention ratios got ridiculous.

Re:not all shaping / policing is bad.

[PNutts]

If I'm in WoW

If there's anything that is more of a waste of bandwidth, it's this.

Streaming Sophie's Choice on Netflix?

Re:not all shaping / policing is bad.

[Charliemopps]

It's "Up to 1Gb!"(tm)

Re:not all shaping / policing is bad.

[GooberToo]

Exactly. You can always tell when you've discovered a moron, and idiot, and a downright knuckle-dragger, because they immediately equate all traffic shaping as bad. In fact, traffic shaping is in wide use and is generally a good thing.

Traffic shaping frequently means your SSH session stays fast and responsive despite massive FTP downloads elsewhere. It means your games remain responsive and fun despite your neighbor stealing movies over the next several weeks. It means your HTML is delivered quickly, allowing you to start reading the article while the images, and worse, ads, are still downloaded. It means VOIP with your family in another country remains a high quality experience despite your douche bag neighbor stealing ten movies via bit torrent and complaining about you having the nerve to communicate with family while he's stealing shit.

Re:not all shaping / policing is bad.

[GooberToo]

See, that's the problem. You're a selfish, self entitled idiot. That's the only problem here in this discussion.

You're argument boils down to, everyone else's experience must suffer and frequently become unusable so you can download something a second or two faster. That's stupid. That's selfish. That's idiotic. And the simple fact is, its non-discriminatory. When you require interactive services, you get the same quality of service.

Re:not all shaping / policing is bad.

[misexistentialist]

Students pay, maybe even a "technology" fee. The difference is schools operate on a (mostly) non-profit basis and can be trusted to maintain the network for the benefit of the users, unlike ISPs that are more concerned with the shareholders. Contracts or regulation can't really keep them honest, since some shaping is probably always necessary. The answer is more competition, but that is hard to maintain in any market let alone a utility market.

Re:not all shaping / policing is bad.

[Anonymous Coward]

I hesitate to use words like "every," but I can think of no ISPs who advertise their plans as anything other than "up to X Mbps." Why you believe that not getting that is somehow a service they don't provide eludes me, particularly when the reason you are not receiving it is legitimate shaping activity*.

Okay. What if an ISP advertises their service as "up to X Mbps" and they only can consistently deliver 0.01*X Mbps? The statement is true, but it's still misleading the consumer. If I buy a service advertised as "up to X Mbps" I expect that I will be able to consistently and without limit get that rate.

Example: At home I bought a 16Mbps connection. The actual rate works out to something like 12 Mbps. And I'm okay with that, because I can use that 12 Mbps to my hearts content without all this antisocial behavior most ISPs dish out.

Re:not all shaping / policing is bad.

[StikyPad]

You're providing for a bunch of students that don't have a contractual agreement with you.

You mean besides the enrollment contract, but otherwise you're spot on. Just like I have no contractual agreement with my ISP other than the one I signed when they installed my service.

Re:not all shaping / policing is bad.

[AvitarX]

For now anyway, interactive services are low bandwidth, for example, a typical skype call is 400kbps / 400kbps (high-quality), or 1.2mbps/1.2mbps (HD).

Torrenting a popular 24hour+ old file uses significantly more (I get at least 2.4mbps on comcast, peaking at 3.6, I throttle the upstream to between 400 and 800 mbps, I peak at 1.2mbps otherwise). I download news articles at 16mbps, putting me about 6 times the maximum a skype call will use (highest bandwidth interactive connection I can think of).

An FPS is 10mb/hour (http://answers.yahoo.com/question/index?qid=20081020175309AA7IHCa), again very low usage, but can be worthless if an extra 50ms of ping.

You're download on the other-hand is only lessened (on a saturated pipe) by however much bandwidth drops due to the prioritization (theoretically, not at all it will just come a quarter second later with the increased latency). If the pipe is saturated either way, it's the same total bandwidth, the skype call would still be sending out the same amount of packets (it's UDP I think), but they would show up late and degrade the video quality (still slowing down your download), but you would get that extra 1/8th to 1/4 second due to your reduced latency on a bulk transfer protocol.

This is needed because TCP/IP is "fair", but not all traffic is equal, and the ability to smooth out some traffic over a quarter second while keeping other traffic up front is good for everyone, especially since the bulk of traffic is not latency sensitive (Netflix).

If the saturation is so bad you cannot download bulk at a decent rate, or stream your movie/TV, then more bandwidth is needed, but as long as you can keep you4 3.4mbps average (highest netflix rate) over a couple seconds (the buffer), then there is no reason not to keep the neighbor with 1.2mbps (or 2.4 for up and down) average speed over 50ms.

Web browsing is another case where prioritizing makes sense in bursts (like the tokens mentioned in the article). When browsing much of the time is spent with no traffic, taking 5mpbs out of your download for 10 seconds is functionally the same as taking 2.5mbps over 20 seconds, but it makes the other users experience much better. Shaping such as this can allow for better use of a shared resource (and keep in mind if you skype and torrent, or browse and torrent, you see the benefits even without neighbors) making everyone happier (again, the same amount of traffic goes through either way).

Re:not all shaping / policing is bad.

[markxz]

ISP are able to install their own equipment/connections in BT exchanges and use the unbundled local loop for the last stage of the connection.

Users connected to smaller exchanges don't benefit from this as there is not enough customers to make it economic for the ISP. These users are forced to use the BT backhaul network at excessive cost.

Re:Lol

[CTU]

what was so funny?

shaping is better than policing

[Anonymous Coward]

I encourage you to look up goodput. TCP is great and eventually reaches steady state. Shaping can help if applied correctly.

Of course, the telcos are probably doing application or flow level shaping. too lazy to RTFA.

PS, beware the bufferbloat beast!

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

Or maybe it was done for legitimate reasons and you need to get off your high horse?

Downloading a torrent of a movie is certainly not as critical as me getting my pr0n served via HTTP. Or SSHing in to my MUD.

Re: Is shaping the same as throttling?

[obiwan2u]

Looking at http://en.wikipedia.org/wiki/Traffic_shaping it sounds like "shaping" is throttling based on packet type that kicks in when bit rates get to high.

So were you public about HOW?

[Jahava]

I work for a college, and we shape / police traffic to / from the Internet.

This was a necessity on our 3Mb link of many years ago, but has still been useful on our 1Gb link of today.

This policy has greatly improved the user experience. Interactive protocols have low latency, bulk transfer protocols get sent to the end of the line. Where we do slow down things, it isn't really noticed by most folks. After first implementing this many years ago, we immediately got positive feedback. Now it is just "how things are."

Hell, I shape / police traffic at home to my cable modem. VOIP and interactive ssh are still usable even with huge downloads going on now, and users hammering the public wifi I provide to my neighborhood.

You make a good case, and I agree. I'd like to know whether or not you told your customers how you were shaping their traffic.

I have no issue with enforcing (your idea of) quality of service on a network. What bothers me about Comcast is the general lack of transparency behind it all. Their policies should be public and open to scrutiny, minimally so I know what's going on with the service I'm paying for and ideally so they can be held directly accountable if they implement an absurd form of shaping.

Yup

[betterunixthanunix]

Shaping traffic based on the protocol is generally a good thing, assuming you get it right (i.e. give the protocols that need priority priority). Shaping is compatible with network neutrality, as long as you are not using the address packets originate from or the payload as part of the shaping rule.

Unfortunately, I simply cannot trust that an ISP like Comcast will stick to shaping rules that only use the protocol. This is one of those cases where regulation is needed (particular given how many hand-outs large ISPs have gotten).

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

Regulation is rarely the answer. Are you some type of socialist? We need more free market-based solutions, not trying to regulate everything.

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

ISPs that indulge in such practices (some even go in DPI, Bell Canada I'm talking about you) want to have their cake and eat it too.

They want to retain common carrier although engaging in practices that go against common carrier status.

Even worse, those same ISPs have really unrealistic transfer quotas. In Canada, they even want to force resellers to enforce those quotas (see UBB) in order to discourage competition (That would be Bell). As far as quotas and speeds go, a good example would be Videotron (Cable). 8Mbit/1Mb and only 50GB. Meaning Netflix is useless (hampering competition). We're in 2011, perhaps we could get speeds and transfer quotas not dating from the '00s

Re:Cue the cable company bashing in 3...2...1....

[Squiddie]

I guess we should also deregulate food and drugs. You can take care of yourself, right?

Traffic shaping in of itself is not a problem

[voss]

As long as its done in a neutral manner based on whats being sent as opposed to whos sending it.

 

Carrier Status

[Anonymous Coward]

AT&T and Verizon can haz FCC wireless allocation. What about the others?

Re: Is shaping the same as throttling?

[AK Marc]

I find that Wikipedia is good at giving a few people's opinions of terms, but not actually backing up the vernacular definition.

Shaping: controlling bandwidth among various protocols (whether DPI or QoS, port number, etc.). This can be enforced by throttling some traffic or by prioritization.

Throttling: capping or reducing the bandwidth available to some identifiable clump of traffic (I use clump because all the other appropriate terms I can think of have some technical definition more strict than what I want to say). It can be done solely in response to congestion, or in the absence of congestion. It can be done on some subset of a subscriber's traffic, or to the entirety of it. Throttling is a slowing or capping of traffic. Most shaping is a subset of throttling. Oversubscription could be considered a form of throttling. Throttling is much more general of a term than shaping.

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

I guess we should also deregulate food and drugs. You can take care of yourself, right?

Of course we can't take care of ourselves. We need the government to look after us, just like an older male sibling.

Re:Cue the cable company bashing in 3...2...1....

[bondsbw]

Since when is exclusivity a free market solution?

Re:Cue the cable company bashing in 3...2...1....

[CTU]

no, we just need to allow them to have competition and consumers more options the ISp's will have less room for BS if they were not 1 of maybe only 2 options for high speed internet.

ADVERTISED FEATURE of Time Warner and Comcast

[drtsystems]

This is said (although almost in passing) in the article. But I will repeat it because i know how few of us RTFA. Time Warner advertises its PowerBoost feature (and Comcast has something similar) where you get like double your usual bandwidth limit for "burst" downloads and then you get throttled back to your limit after the burst is complete. This is a FEATURE they advertise, not something bad. It allows you to (for example) get 15mbit when download a web page or small file on your 7mbit plan. Notice its a 7 mbit plan, they are not throttling you below your plan's rated speed. They are giving you faster downloads for a quick burst. There is plenty wrong with Time Warner, but this isn't one of the the problems.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[Anonymous Coward]

My neighbors add a couple more tiers to PowerBoost. Weekends the cablemodem borders on useless for any low latency or sustained streaming service (for example hulu, netflix). Weeknights interactive games are tolerable with hit & miss reliability for sustained streaming. Weekday mornings flawless service.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[Anonymous Coward]

Trying to give messing with your bandwidth a positive spin, sounds like newspeak to me.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[Anonymous Coward]

Not true, when I had Time Warner they throttled back my Netflix. I could tell because at first it came through at DVD quality and then dropped down to circa 1990 DIVX blocky crap quite quickly. But the speed was claimed to be 7 Mbit, which should be plenty fast enough for streaming DVD quality, let alone Netflix H.264 high compressed DVD stream. Dropped them for Grande, who was also 7 Mbit and the Netflix was crystal clear every time, all the time.
They may be fixed now, but I initially dropped them as they wanted to add $5 to my account because I didn't subscribe to their TV package. I had them for over 3 years and they claimed that all their advertising was wrong and I was paying too little, and haven't looked back. They are scum and I was a lucky one who has another cable provider, I pity everyone else who doesn't have options.

Back to your point though, with packet shaping it is hard to prove they are shaping your traffic and how much because they probably won't slow down your other traffic that you speed test with which this tool doesn't explore.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[Anonymous Coward]

That's not how Comcast does it in Houston. The "as-advertised" 15mbit plan has a theoretical "boost" of "up to" 15mbit, but there are no guarantees.
I don't know where you are, but I want my Time Warner back.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[EmagGeek]

It wasn't anywhere near "mentioned in passing" in the article. It was stated quite plainly and directly.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[Geoff-with-a-G]

This is true, and furthermore "shaping" is one of the nicer/friendlier methods of managing traffic contention.

The "my plan says 10 Meg I demand 10 Meg!" argument is simply not technically valid. If the ISP has 10G of upstream to a carrier and more than 1,000 customers, it's not physically possible for everyone to run unlimited 10 Mbps all the time. So now that there's a possibility of contention, the good/bad lies in how the provider limits you, not if.

Possibly the worst is usage caps. After transferring X GB of data they either up-charge you or knock your speed way down. You don't want this.
Next is free-for-all. This just means your traffic and everyone else's goes into a buffer, waiting to transmit across the congested link. As the buffer gets full, new packets tail-drop. Better than caps, but not much. Google "bufferbloat" to see some of the problems this creates, but the quick answer is that it's bad for VoIP/streaming and it actually worsens the congestion itself. AQM/WRED can mitigate this a bit, but it still loses valuable information by keeping all traffic undifferentiated.
Then there's policing. This just means that if your traffic exceeds a certain rate (let's say they limit your 10M to 8M) it's dropped immediately. Not as bad as it sounds, but TCP still would rather a packet be delayed for a few msec than dropped and retransmitted.
Lastly is shaping. This attempts to take your input and limit it to a slower output, just like policing, but instead of dropping your packets, now we delay them. If you transmit at 10M, we send your first 8M then wait a second before sending more. TCP windowing adjusts more gracefully and your connection just becomes a working 8M without so much delay and retransmission.

People get irritated at the idea that someone somewhere is doing something to their Internet traffic, but as contention/congestion management goes, this sort of shaping is actually one of the best options.

Re:ADVERTISED FEATURE of Time Warner and Comcast

[VortexCortex]

Weeknights interactive games are tolerable with hit & miss reliability for sustained streaming. Weekday mornings flawless service.

Ok, so I've designed this game network protocol which gives users with good bandwidth smooth gameplay and high precision, but gives lower bandwidth users smooth gameplay and lower precision -- client side prediction and synch rate are determined by a test -- a test at the beginning of your connection. This test says you have a monster awesome responsive network, and I'll give you the best experience you expect -- Until your BS ISP plan decides that you only get that for "bursts", and now you're in a fire-fight, lagging all to hell and wondering who's fucking host connection is retarded?!

Yeah, so now I can do three things -- 1. Tell you about the BS ISP plan you have (actually, that latency meter, it's more than just latency, it shows others just how shitty and unpredictable the connection is too). 2. Try to compensate by writing code that more quickly adapts to your shitty dynamic bandwidth ISP plan, (which sucks to do, because sometimes the algo guesses wrong and you end up with shitty connection for no reason, but it helps stop many other gamer complaints) or 3. Try a few "hacks" to enable bursting all the time.

I found out that most times if you're downloading a stream, and you drop that connection, then start a new one, you get "bursting" again! It's lame, but hey, this actually does work sometimes -- download something big with FF's download manager, watch the connection speed drop... now, hit the "pause" and "restart" button quickly to drop and re-connect (server needs resume support for this to work). The bursting will pick back up again with high speed. Neat eh? This doesn't work everywhere, in fact the server could be throttling too (which this start/stop actually mitigates sometimes), but it doesn't really hurt most connections, so why not do that?

So basically, as your connection gets shittier, I fire up another connection to the server, and once that's connected, and tests as higher than the current connection, we switch to that. (I say connection, but most times it's UDP, just via a different port). Sometimes, now this is strange, opening multiple ports/connections between client/hosts and distributing packets between the different port/connections actually gives better performance.

Overall shaping is in effect, but there is apparently per connection "bursting" going on as well. A slower solid connection sans bursting will give you better online play than a wildly jumpy bursting connection.

P.S. F'ing Buffer-bloat accounts for some of the erratic speeds mentioned above too, (and might also have affected TFA's speeds).

Re: Is shaping the same as throttling?

[Gadget_Guy]

I find that Wikipedia is good at giving a few people's opinions of terms, but not actually backing up the vernacular definition.

As opposed to a single poster on slashdot? At least Wikipedia insist on citations.

So in what way are your definitions superior to those on the linked Wikipedia page? What did the wiki get wrong?

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

Are you some kind of retard? Deregulation doesn't lead to any sort of fair balance, it leads to excessive and unnecessary price increases across the board as monopoly situations are reached. Check out the New Zealand power market if you don't believe this happens.

Re:Cue the cable company bashing in 3...2...1....

[WindBourne]

Well, of course, there will be cable bashing. I use comcast at home, and I KNOW that they hit netflix HARD. I run a router at home and have studied the data coming through. When it comes from a work site, I can download 4 GB no problem. However, when I grab netflix movies that are about 2 gb streams, then I get issues. Interestingly, Crackle and Hulu do not have the same issues, though crackle uses more bandwidth.

Personally, I agree that they own the network and should be allowed to do what they want. HOWEVER, I also think that we should pass laws FORBIDDING a monopoly into the home. At the least, we should change the monopoly to be from the home to the greenbox and any company can then sign up for a deal with providing service to the greenboxes, AT THE SAME RATES. IOW, if comcast wants to own the greenbox-home monopoly, not a problem. However, they charge other providers the same price that they charge the rest of comcast.

Basically, it is time to limit the monopoly's size.

Re:Lol

[Anonymous Coward]

what was so funny?

Well let's see. B. Hussein Obama is the first black president of the USA and he really seems determined to be the last. Best Republican campaign slogan ever will be "at least he's not Obama". That's pretty funny. He's not Hillary too because there actually is mercy in the universe.

Telcos may appear innocent

[SpaceLifeForm]

But it could be because it is already too late to tell.

Re:Lol

[Anonymous Coward]

not sure if this is a non sequitur or ignoratio elenchi

Re:Gitmo: better than living in the Middle East

[Anonymous Coward]

Go right ahead then, asshole.

Re: Just because "I'm not Bush" worked does...

[Anonymous Coward]

...not mean "I'm not Obama" will!

Re: Unless you want millions of signals...

[EzInKy]

...and wires interfering with your private life yes, regulation is the answer.

Re: Is shaping the same as throttling?

[AK Marc]

There wasn't a single cite to the definition. There was a definition asserted without citation, followed by the implications of that or any number of other definitions.

What was "wrong"? Nothing worth dealing with. They give a mostly useless definition that doesn't help differentiate it from anything else, then talk about the technical implications and implementations without regard to the vernaclar, which for the average person is much more important than the overly-cited long and dry article about a term that just needed a couple sentence definition then links to the manners in which it is implemented.

Why, are you going to assert that if I don't agree with it that I should fix it? I've tried (not on that one, but others) and someone reverts changes and so it's not worth my time to correct wrong things when they border on opinion and someone else's opinion conflicts.

So in what way are your definitions superior to those on the linked Wikipedia page? What did the wiki get wrong?

My definition is superior because it's a few thousand words shorter. And properly attributed to the source (me). Did I get anything wrong? If not, why do you accept some uncited definition on Wikipedia and not the one I give?

Re:Gitmo: better than living in the Middle East

[Anonymous Coward]

You may find that some parts of the world are much nicer than the good old USofA. Why not get yourself a passport and go see for yourself eh?
Thought not. I guess you prefer the world accourding to Fox News.

Disclaimer. I've worked or travelled in 63 countries over the past 40+ years. There are many places in the world I'd rather live than most of the USA. I was raised in Tuscon but I'm currently dividing my time between Bath, Somerset, UK and a small village about 10km from Sienna in Italy. Sienna is visible from my Kitchen Window as I write this. It looks beautiful. Oh, and I don't have to carry a gun or lock my doors at night.

Re:Cue the cable company bashing in 3...2...1....

[maxwell demon]

There already is regulation (granted exclusivity) favouring the provider. Either that regulation is to be removed, or there has to be another regulation favouring the customer to counter it.

You need regulation whenever you have monopolies, or the danger of monopolies. That regulation has to either prevent forming of monolies, or if that is not possible, regulate those monopolies so that they cannot do too much harm. Remember, as soon as there's a monopoly (either granted or enforced through market power), there's no free market any more to regulate things. And the free market cannot prevent monopolies when network effects are in play.

Re: Is shaping the same as throttling?

[Anonymous Coward]

As opposed to a single poster on slashdot? At least Wikipedia insist on citations.

Funny you should mention that, since the page linked to is tagged "Needs Citations" in multiple places.

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

Time to nationalize the infrastructure we already payed for through surcharges and fees added to our utility bills, but never got the benefits of. It is unrealistic to have 20 runs of fiber down every street, we need one VERY robust fiber network not owned by any provider, and available to all competition on equal terms. Remove the rent-seeking monopoly bottleneck and the free market can work its magic.

Re:Cue the cable company bashing in 3...2...1....

[cheeseandham]

HOWEVER, I also think that we should pass laws FORBIDDING a monopoly into the home. At the least, we should change the monopoly to be from the home to the greenbox and any company can then sign up for a deal with providing service to the greenboxes, AT THE SAME RATES. IOW, if comcast wants to own the greenbox-home monopoly, not a problem. However, they charge other providers the same price that they charge the rest of comcast.

That is kind of how it works in the UK (See how British Telecom has been split up).

BT Openreach was created to "Ensure that all rival operators have equality of access to BT's own local network" and it works pretty well, I have a BT line and BT Wholesale broadband, but provided by a different company with their own service levels, prices etc. And there are a lot of ISP's like this.

If an ISP doesn't want to use BT's infrastructure in the exchange, they can even install their own whilst still taking advantage of that piece of cable going from the exchange to the home, laid down by public money.

How is it not obvious to the US politicians that this is a sensible move? More to the point, how the hell did something sensible happen in a UK Parliament?

Re:Cue the cable company bashing in 3...2...1....

[OeLeWaPpErKe]

No they don't. The difference between AT&T/Verizon and just about any other ISP is that AT&T/Verizon are "tier 0" isps.

They have a worldwide backbone, with enormous capacity, and peer with major isp's everywhere in the world (even China). This not only has huge bandwidth, but these companies are capable of upgrading said backbone without needing anyone's permission. Comcast, Time Warner and Cox, basically hook up to AT&T/Verizon/Cogent/Level3/... Which is not the same thing. Every network engineer worth his salt knows that there's just no comparing an AT&T uplink to a Cogent one. Every manager worth his salt knows the price difference means you'll go with Cogent anyway, as it's probably cheaper to build your own backbone (esp. these days).

Note that ALL isps did shape in the test, it's just that AT&T/Verizon have a lot less need to slow down connections than the cable companies. Without shaping, TCP networks such as the internet are vulnerable to trivial exploits allowing one client to hog bandwidth.

This probably means it's a good bet that all isp's throttle bittorrent (as they should) and other p2p application, and the theory is right : allowing unfettered bittorrent/p2p means unacceptable network performance.

Re:Cue the cable company bashing in 3...2...1....

[Runaway1956]

The thing is, the American government and the American people "own" a significant part of all the networks. Being granted a monopoly, and accepting subsidies, means that you don't get to make all the rules.

Re:Cue the cable company bashing in 3...2...1....

[Runaway1956]

Want free market? Stop taking subsidies. Stop filing suits against all the small local governments that try to install any system in areas that don't have service. Free market? Just stop being giant fucking douches, determined to milk every dime possible out of the system. Stop abusing the legal system, stop abusing the legislative system, stop ripping the people off. We don't HAVE free trade, you big dummy - not as long as a telco of cable company is permitted to file suit against a community that is trying to provide a service where none exists.

Re:Cue the cable company bashing in 3...2...1....

[Jaysyn]

And how exactly do you think that is going to happen? Magic?

Re:Cue the cable company bashing in 3...2...1....

[Anonymous Coward]

Regulation is rarely the answer. Are you some type of socialist? We need more free market-based solutions, not trying to regulate everything.

I love the way lawless-market advocates try to confuse people by using the word FREE. If by free-market you mean market without regulation and laws, why the hell do we have laws in the first place.
If you don't want laws on market,, why should there be laws on anything else. I know, the market will regulate itself becouse people won't use/buy services with no quality.
How about going a step little further and make a lawless society overall. By your way of thinking this is a free-socity. You probably think that a socitey like that will regulate itself. Example: a person is murdered, and on this "Free-society" the mob restates justice.
Seem like the same to me.

Re:Cue the cable company bashing in 3...2...1....

[muridae]

If the cable company wants to operate without oversight, then they can buy back the copper that the government paid them to install, and pay me and other land owners for the right to run their service across my property line. Until such time that they do so, and as long as they continue taking advantage of government assistance, they can and should answer to government regulation.

Re:Cue the cable company bashing in 3...2...1....

[bipedalhominid]

You forgot, larger. Then it would be,... Wait for it,... Big Brother.

Re:Cue the cable company bashing in 3...2...1....

[WindBourne]

wrong. Doing 20 runs of fiber down the street is exactly what you want. It gives loads of excess fiber and more importantly, redundancy. In addition, with competition, we will see upgrades galore. Each network provider will break their backs to lower their costs and improve delivery. Where you run into trouble is if the monopoly is from home to a CO. When that happens, then the company has little incentive to upgrade from CO to home (in fact, they have a strong disincentive). Look at the RBOCs with their twisted pair. It took competition to get them to move forward on doing anything. And even then it was not much. So the answer here is to do the greenbox (basically, block level) to the home. By doing a monopoly, ONLY in your block, you just solved the most expensive part of any network to the providers. It costs loads of money for that portion. The rest costs a great deal less. In fact, I would guess that the monopoly would charge 15-30/month just for that piece. THe good news is that with competition, you can expect to get 100 MB drops for say 20/month, HD cable for 20/month, with phone for 10-20. We would likely see video phone usage jump.

However, while nationalization of the whole system makes ZERO sense, it might be possible for localities (cities, counties, perhaps states) to consider the idea of doing a locality-owned monopoly of fiber to the residency/greenbox and then doing 10 year contracts for the maintenance. THere are places that I think that the national system can work (for example, I do believe that the feds should allow a private medicare system to compete against insurance), but normally, when real assets are involved, the larger gov is too expensive, and to complicated.

Re:Cue the cable company bashing in 3...2...1....

[WindBourne]

Actually, UK is not similar. UK's approach is the same one that ATT and later, US's RBOCs had. Roughly, what we call a central office, is the 'Main Distribution Frame (MDF) in the exchange' from that wiki. That portion has a length of maybe 30 meters all the way to say 30 km. The problem with that, is that the RBOCs had little incentives to upgrade that network, since it benefits their competitors as much as them. OTH, By doing the block-level 'greenbox' then the competitors have to install various lines of their own. The advantage is that some will put in a simple star configuration (low costs, but prone to outages), while others will put in a ring configuration (higher costs, but requires multiple cuts to lose the connection). Note that the overall system costs is actually higher with what I suggest, but I count on REAL competition to bring it down. Right now, there is a reason why comcast is able to buy NBC and other companies, and it has NOTHING to with competition.

Oh, it is VERY obvious to the pols here. That is why they get LOADS of money from Comcast, Time-warner, etc to NOT do this. Real Competition would bring REAL FAST services here, but it would mean that profits would be minimal.

Re:Cue the cable company bashing in 3...2...1....

[The Dawn Of Time]

Which portion of the network does the government own, exactly? Do you think those tax breaks conferred some sort of property rights? That's an... interesting interpretation.

Learn to write dickweed

[Anonymous Coward]

You write like a retarded moron. Sort of is not a sentence that demands its own paragraph.

Re:Cue the cable company bashing in 3...2...1....

[Runaway1956]

Not tax breaks, exactly. Subsidies.

http://www.reuters.com/article/2011/02/08/us-fcc-internet-rural-idUSTRE71759V20110208
http://www.nber.org/papers/w9090.pdf

The telcos and other ISP's have taken money from the government, many times, mostly to aid in that "last mile" crap. And, the "last mile" just never materializes.

Although - the idea of tax breaks conferring partial ownership upon the government isn't a bad idea either. Tax breaks are, after all, just another form of subsidy.