Slashdot Mirror
Why BitTorrent Causes Latency and How To Fix It
Sivar recommends an article by George Ou examining why BitTorrent affects performance so much more than other types of file transfer and a recommendation on how to fix it. The suggestion is to modify P2P clients so that, at least on upload, they space their traffic evenly in time so that other applications have a chance to fit into the interstices. "[Any] VoIP [user] or online gamer who has a roommate or a family member who uses BitTorrent (or any P2P application) knows what a nightmare it is when BitTorrent is in use. The ping (round trip latency) goes through the roof and it stays there making VoIP packets drop out and game play impossible."
Hey, I have a really spiffy idea. How about creating a router that can determine which packets take precedence? I'll make millions off that idea...
What? Oh, damn Linux! What? Oh, Windows can do it too now? Why do I always have the good ideas about 10 years too late?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Don't download porn while playing WoW.
Do you know how many times I've died in WoW because of his porn downloading?
He's paying up, I need my epic flying mount...
My ZooLoo
Why BitTorrent causes network bandwidth to be used. And network packets to be sent & received. Really sometimes I wonder.
While I prefer Tomato on a WRT-54GL, that would do absolutely nothing at all to solve this issue. A router behind a modem can really only regulate the upload, and can't easily prevent a flood of data on the downstream side.
This issue is with the queue on the Telco's DSLAM, or on the other side of the cable from the modem. This is more like an invited DDOS, which no amount of filtering at or behind the modem can resolve, because the modem is getting the traffic from the DSLAM after it goes through the queue.
The only way to have QOS solve this issue would be to ask the telco to do the QOS for you, and the amount of processing power to do that nicely isn't trivial.
If I have nothing to hide, don't search me
Use the bandwidth capping abilities in all modern P2P clients. If you're trying to torrent, max it's upload and download capabilities below your total network bandwidth. I have a 1Mbit up and 10Mbit connection. Capping my total upload in KTorrent to 100KByte/s and my down to 900KByte/s allows me to do anything else on the internet without issue. Very few online games or other uses of the internet require more than a 100KB down and 30KB or so up. Learn to properly manage your P2P programs and you won't have a problem.
Frozen Insanity
http://frozen-solid.net
So, if the ISPs do traffic shaping "to improve the service" it's bad, but we admit that on the small scale (when it affects ourselfs) there is a real need for traffic shaping! Thats interesting....
Homebrew traffic shaping. *facepalm*
And we admit that on a small scale, we need to control our eating, but we don't want the grocery store telling us how much of things we can buy.
Traffic shaping and QOS will help a little, but the real problem is simply that you can't afford to delay priority traffic by more then one or two full-sized packets on any connection less then a few megabits (meaning: just about all home interconnects). If you wait any longer then that, it becomes noticeable.
Traffic shaping and QOS are not usually able to make that guarantee. A straight priority queue with bandwidth guarantees can, as long as you are able to actually classify the torrent traffic differently from your other traffic.
Part of the problem is that it is often not possible to distinguish between the batch and the interactive traffic with Shaping/QOS. Not only is QOS almost universally set wrong, but the simple fact is that one can mix interactive and batch traffic over the SAME ports (http, ssh, dynamically allocated ports)and that can make it virtually impossible to use traffic shaping or QOS to keep the mess away from your interactive traffic.
The best general solution is to use a straight priority mechanic with minimum bandwidth settings to separate as much of the bulk traffic out as you can, and then run fair-queueing at each priority level to take care of any that leaks through. This will do a very good job cleaning up the traffic. DragonFly has a fair-queue implementation for PF that does this. There is also at least one fair-queue implementation for PF in the wild.
Fair-queueing essentially classifies connections (the one in DFly uses PF's keep-state to classify connections), generates a hash and indexes a large array of mini-queues. One packet is then pulled off the head of each mini-queue. One enhancement I would like to make to the DFly implementation which I haven't done yet is to use the keep-state to actually determine which connections are batch and which are interactive, and have a parameter that allows the queue to give additional priority to the interactive connections by occasionally skipping the hoppers related to the batch connections. A quick and dirty way to do that is to simply check the queue length for each mini-queue.
In anycase, its a problem for which solutions are available. Regardless of what you use it has become apparent in the last few years that the only way one can classify the traffic well enough to properly queue it is by building keep-state knowledge on a connection by connection basis.
-Matt
We long ago learned that when inserting time between protocol events that it is far better to use a time randomized between an upper and lower bound than to use a repeating interval.
When fixed repeating intervals are used, separate instances of a protocol (and other protocols that use repeating intervals) slowly tend to fall into lock-step patterns with pulsating waves of traffic in accord with those patterns.
In other words, fixed protocol timers can create the traffic equivalent of the Tacoma Narrows bridge.
By-the-way, ping (ICMP Echo request/reply) is a terrible way to measure network latency. ICMP is often a disfavored form of traffic as it crosses routers, sometimes even rate limited.
There are better tools for measuring link properties, for example there is "pchar" - http://www.kitchenlab.org/www/bmah/Software/pchar/
I worked on a method to do even better measurements, but I put it aside several years ago: Fast Path Characterization Protocol at http://www.cavebear.com/archive/fpcp/fpcp-sept-19-2000.html
http://lartc.org/wondershaper/
Works in Linux since 2002.
*yawn*
Except, wait for it, almost all p2p clients allow you to throttle your bandwidth anyway.
I want to delete my account but Slashdot doesn't allow it.
It is always easier to manage uplink bandwidth from downlink bandwidth, simply by virtue of the fact that you control the actual packet queues.
Downlink bandwidth can be controlled in numerous ways. The easiest way is to actually run the incoming packets through a bandwidth limiter with a very large packet queuing capability. This will cause a ton of packets to build up in front of the limiter and eventually fill the TCP windows of the senders. The packets that get through the limiter will cause a stream of ACKs back from your machines at the desired data rate. The combination of the two will cause the remote senders to band-limit the packets they send to the bandwidth you desire.
when running incoming packets through a limiter you still need to traffic-shape/QOS, priority-queue, or priority-queue + fair-queue the packets going through the limiter. If you don't then your interactive traffic can wind up getting stuck in a packet queue with hundreds of packets in it. In addition to that you may have to control the advertised TCP window or even implement RED on your limiter to prevent the hundreds of packets built up in front of the limiter from turning into thousands of packets.
If you can classify the bulk traffic then you can use virtually any queueing mechanic. If you can't classify all of the bulk traffic then the only mechanic that will work reasonably well is, again, going to be a fair-queue.
Fair-queueing is not the holy grail but it is typically the most effective mechanism when combined with another queueing mechanic, such as a priority queue.
-Matt
You forgot protocol inspection
NBAR on any current cisco IOS feature set will detect pretty much anything you need to prioritise without seriously impacting performance.
Juniper has something similar on their gear as well.
Easy QoS: Low latency queueing = fair queue with a priority queue as you described.
tag real time traffic as priority queue and allocate enough bandwidth depending on your capacity engineering. tag your important apps and put them in the second queue. Rest in default class.
This is really all you need, I have seen VOIP for over 500 extensions hold up as that sites link is over 90% for an hour And this is Cisco callmanager i.e. the remote phones and gateways bork and go into fallback mode if the keepalives are lost.
Just need to remember it needs to be end to end and in both directions
IMHO, Cisco has the best packet queueing mechanisms that I know of. I've been using their fair-queue stuff for years, and it has only gotten better with each iteration of IOS.
When I went from a T1 to a DSL line to save some money I immediately noticed the missing cisco. That little 2620 was so nice. PF couldn't hold a candle to what the 2620's fair-queue could do so I sat down and wrote a fair-queue implementation for PF (for DragonFly). It still isn't as good as what Cisco has, but it gets a lot closer then the other PF queuing mechanisms get.
I think the bit I'm missing is the batch classification. My fair-queue can still get overwhelmed by dozens of batch TCP connections if I happen to not be able to classify their traffic (and they wind up on the standard queue instead of the bulk queue). The set-up is a priority queue with minimum bandwidth guarantees plus a fair-queue at each priority level.
I keep hoping someone will take up the flag and finish it.
-Matt
Any whatsoever? His part in the Maynor/Ellch debacle was a serious low point for tech journalism; he makes Rob Enderle look good, fer chrissakes. Even if the article were in fact insightful and informative, the simple fact that his name is attached to it guarantees that I'm not going to read it. Someone please tell me what it says.
Yeah, but the action that the ISPs take to correct the negative effects caused by millions of people actually using their allotted bandwidth is unfair (and possibly illegal, IANAL and I have no issues w/throttling so haven't been following closely.)
There is a huge difference between a corporation not giving customers what they have paid for, and the customers using that bandwidth how they see fit.
Just my 0,02
Orbis terrarum est non altus satis
It works like this: if the upstream bandwidth is saturated, TCP ACK packets get delayed and the sender slows transmission so the downstream bandwidth does not get fully utilised.
There is no solution other than throttling the upstream senders (AFAIK good P2P software has settings). Note larger send buffers in broadband modems actually exacerbate the problem by taking longer to flush. Best to keep them empty, and th only way is throttling.
When are ethical issues not directly derived from self interest? The issue with throttling at an ISP level is receiving the service one pays for. Bandwidth shaping for a personal network, deciding what one would like to do with the service they purchased, is an entirely separate issue.
I would say that a network is, by design, a shared interdependancy. Selfish network behavior, regardless of that activity's legality, is a detriment to the entire system, while simultaneously making it harder to maintain, support, and manage.
Look, I'm not for legislation, but a little common sense will tell you that it simply isn't right for a small minority of the customers to use a massive percentage of available bandwidth, using applications that they themselves say wreak havok on their local network.
You speak of not providing people with what they've paid for. How about all those next gen services we want rolled out, how will they ensure they can manage network traffic fairly when all users need a much bigger chunk of bandwidth for standard services? If P2P users can't keep in mind the rights of those not using the same torrent, or their responsibility to be good network neighbors when they KNOW their activity disrupts others, they have no reason to expect the same courtesy. A free Internet only works if there's respect.
If there's no respect, that's when you wind up with silly things like legislation.
I like the way linux bandwidth arbitrator (http://www.bandwidtharbitrator.com/) approaches the problem.
-
Set your total bandwidth minus the guaranteed bandwidth you want to allocate to priority traffic masked/identified either by port/protocol/src/dest or by a deep packet (perl based) inspection.
-
If any app OR host OR connection OR port starts encroaching on the latency of other others, it gets chucked into memory jail for a fixed number of escalating milliseconds.
-
This has a two fold benefit for latency and bandwidth contention issues. One, by chucking the hogs in memory jail, queue space is cleared up to allow priority traffic through on a more consistent basis.
-
Imagine you've got a city bus that goes to sports stadium of your chosing. When it's not a
game day, there are no problems boarding the bus. You get 3 or 4 people on their way home from or to work. On a game day though, you get those same 3 or 4 people plus an additional 20 drunk people who don't know how to board a bus. What usually happens is the 3 or 4 regulars either don't get a seat or have to wait until the next bus. Not good or fair.
-
The way linux bandwidth arbitrator solves the problem is to tell the bus driver to watch for drunk idiots. If she sees more than three cubs fans in a row, she shouts out to them to step back, close their eyes and count to 10. While they're doing this (they're cubs fans, you know they would), she waves the regulars to the front of the line.
-
So, it's good for latency and jitter. It's also good for bandwidth. All those bad packets you threw into memory jail? Well, a good portion of them are probably TCP which means there's a remote end of the connection waiting on an ACK before it sends more data. Even in the case of connectionless protocols like UDP, chances are there is some sort of app or session layer check that will defacto hold back sending more data until it receives a response. You've managed to stem the firehose that's half of the problem without resorting to cheap ass tactics like false RSTs.
-
The beauty is, done right you don't need to know anything about the problem causing traffic to mitigate its effects. Assume you tag your priority packets with a deep inspection based on port/src/dst/prot (This is what almost never gets done). Whether it's eMule is running on port 443 or through a proxy or if it's a worm or the file sharing program of tomorrow, it gets throttled back.
-
The drawback is it's a bitch and a half to get installed and three more bitches worth of pain to get configured and tuned. Once you do though, it rocks.
-
[I kid about cubs fans. I lived off of Addison Ave and dealt with my fair share of that scenario, but the bulk of the people in line were ok.]
There's nothing wrong with reasonable traffic shaping. ISPs, however, DON'T want to do that. They want to damn near cut-off Bittorrent traffic entirely, even though reducing it by, say, 1/4th would have the desired effect.
What's more, with network non-neutrality, what they really want, and what their QoS policies are set to enforce, is to drastically throttle all applications that COMPETE with their own... You can see this most dramatically with VoIP services, but also with P2P you can see that the ISP's own applications and services that use up bandwidth just a badly do NOT get throttled.
Those issues are why there is "moral outrage". People aren't angrily upset that their torrents were just slightly slowed down...
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
I think you may be using different units (kbps vs KBps) than he is using.
850 KBps = 6.6 mbps
70 KBps = 560 kbps
I use RR also, and those are both reasonable numbers.
http://www.cypherpunks.to/~peter/zdnet.html Schneier is a moron if he thinks telling Hollywood no will force them to use non-DRM content. All you need to do is look at the CableCard fiasco. You give Hollywood the finger and they give you the finger right back because they'd
rather NOT have any content on the PC to begin with. Like Apple, Microsoft
will humor Hollywood so they come join the party. Once they're in, they'll
get screwed out of their DRM protections because Microsoft won't patch the DRM
holes and let their customers bypass DRM. The latest DRM stripper for Windows
Media has worked for almost 2 months now and Microsoft hasn't patched it yet. Ok, so it's nasty to call someone a moron. And it's not really true either. It's ideology that causes Schneier and all the Web 2.0 'experts' to say this. He's no fool but he can't differentiate between it would be good if something being true and something being true. It would be good if Hollywood would give up on flakey DRM schemes. But if Microsoft and Apple had somehow agreed to boycott them, then Windows and Mac users would just have been left with no way to play HD content, because Hollywood is mortally afraid of people ripping HD content and uploading it to Pirate Bay. But George Ou is right that once stuff gets on open platforms like the PC it will get cracked anyway, so the OS vendors were just humouring them. And they probably knew it. FOR THE LAST TIME, I want the DRM on my system so I can play my DVDs, HD DVDs,and Blu-ray like MOST people.
You don't want it, more power to you. I've given you the links to the
software you need get avoid enabling MFPMP at all. I've shown you the lower
CPU utilizations using cheaper hardware. I don't know what else you want. ...
You know, you are a f***ing moron. End of discussion. Well, he's certainly tactless and outright rude. But he's also right about the following -
* Hollywood forced OS vendors like Microsoft and Apple to add DRM to allow playback of HD content.
* Both did, because it would be hard to sell an OS which can't play next generation content.
But this doesn't really matter because
* DRM will be cracked anyway.
* It doesn't have any effect on the OS if you don't use HD content.
He's only get flamed because he's defending Vista which is the subject of the current geek 3 minute hate. Now I don't really like Vista compared to XP, you don't need to believe that it 'causes global warming' as he puts it to dislike it.
BluRay is a product. If you don't like, don't buy and don't use the content distributed over it. I know I won't. And if you don't want Vista as a bundled OS, buy a computer it doesn't come on (like a Dell) or build your own.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
More seriously: Me shaping my own traffic is very different from someone else shaping my traffic against my will.
To borrow another poster's analogy:
I have no problem with choosing what kind of food I eat. If I had kids, I'd have no problem choosing what kind of food they eat.
I would very much not like the grocery store to choose what kind of food is best for everyone.
Fortunately, it's in the grocery store's best interest to give customers what they want. For some reason, ISPs think it's not in their best interest to do the same.
Don't thank God, thank a doctor!
Install a bandwidth management tool like cFosSpeed and you will see that latency drops down to essentially the same levels as you have without BitTorrent running without reducing the torrent speed whatsoever. This doesn't even require any of the fancy prioritization features of the bandwidth manager tool - just avoiding overloading the transmit queue.
In other words, your DSL line is perfectly capable of handling an uplink that is actually used for more than an occasional HTTP request without bogging down. The reason it doesn't do it is poor engineering of the DSLAM. With better tuning and queue management algorithms like RED (Random Early Drop) they will cooperate with TCP congestion control to avoid overloading the uplink buffers. Your DSL line will work just fine without a third-party bandwidth management tool.
Why is the DSLAM poorly engineered? The simple explanation is incompetence. Conspiracy theorist would probably claim that it's intentional because ISPs don't want you to use bandwidth-intensive applications. The truth is probably somewhere in the middle: the original flaw was a combination of lazy engineers and the fact that most users don't really use their uplink so much. It's not being fixed beacuse it serves the interests of the ISPs.
Stop worrying about the risks of nuclear power and start worrying about the risks of not using nuclear power.
Let's take the whole thing from the top.
1) Microsoft's marketing department decided that Vista needs to support BluRay.
2) The BluRay Disk association said that if they want to do this they need to support protected media paths and all the other nonsense.
3) Microsoft did that.
4) The net result is that you can Windows Vista and a software player to play BluRay DVDs. You don't need to crack anything to do this, or break any laws.
If they hadn't implemented PMP et al, you would need to crack to watch the disks because no software players would have been licensed by the BluRay consortium. I read somewhere that with DVD they originally planned not to allow software players because they were scared the keys would leak. And they were right, the Xing Mpeg player was hacked and the key was discovered.
http://en.wikipedia.org/wiki/Xing_Technology
So they sort of had a good case for only allowing hardware players. But Microsoft convinced them that PMP and so on would avoid cracks. Inevitably one of the software players was cracked.
http://en.wikipedia.org/wiki/AACS_encryption_key_controversy
Note that Windows DRM is 100% ineffective against this sort of thing, which is why PMP is a bit of a con. You can always use WinDbg to kernel mode debug a Windows machine and read every single byte of memory. But from what I can tell, the AACS key was extracted from the user mode software player, so even this wasn't necessary.
But you don't need to know the crack anything to play BluRay discs on Vista. Just use the BluRay player software that came with the machine. But that player would not have been licensed if Microsoft hadn't implemented DRM in the OS.
Now Linux can't implement DRM that will satisfy the BluRay consortium that a user won't get the keys. So to play BluRay discs on Linux you must rely on the crack. But cracked software isn't exactly user friendly. It's illegal to link to it in the US and the studio will keep tweaking the disks so it breaks and you need to download a new version.
If Microsoft hadn't implemented DRM the Windows users would be in the same boat.
Now if Blu Ray is like DVD then writable disks will only allow unencrypted content. So to copy a Blu Ray disk you'd need to crack. But just to watch a disk you don't.
Personally I pretty much rent or buy the odd DVD and watch cable. I'm in Asia and BluRay isn't too common here. I think the technology is overpriced and the requirment that the whole playback path be protected makes the whole process too fiddly. I can't see much difference in quality between HD and normal content. So I'm not going to buy it. But let's not get carried away. Windows users will watch BluRay disks in a userfriendly way. Pirates and Linux users will be able to copy/watch it too, it will just take a bit more work.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
Let us take this occasion to have a moment of silence for the above AC's sense of humor. It died in a tragic accident and all attempts at resuscitation failed. It will be missed.
Now imagine sending 10 private emails to someone (Karel Donk) and the guy continues saying annoying and idiotic things. Then imagine you lose your temper and use some profanity in a private email. Now most people can get away with that, but someone like me who is a high-profile blogger at ZDNet should have known better to write that in an email. So Donk forwards my emails to Gutmann and Gutmann posted it on that link of his pretending like I was sending Gutmann harassment email. Initially, Gutmann posted it on his University web page but he took it down because it didn't belong there. So that was Guttmann's only defense that I referred to him as a moron in some email that wasn't even sent to him.
So I used profanity in a private email and it got posted without the full context. I should have known better and I won't make that mistake again. Guttmann on the other hand never conducted a single test, never even used Vista, and he presented a bunch of web forum postings as a scientific study from a respected university. That is by definition academic misconduct.
I explain how Karel Donk is one of Gutmann's primary sources here. http://blogs.zdnet.com/Ou/?p=723)
Anyhow, thanks for being logical and email me any time.