Slashdot Mirror
Enhancement To P2P Cuts Network Costs
psycho12345 sends in an article in News.com on a study, sponsored by Verizon and Yale, finding that if P2P software is written more 'intelligently' (by localizing requests), the effect of bandwidth hogging is vastly reduced. According to the study, redoing the P2P into what they call P4P can reduce the number of 'hops' by an average of 400%. With localized P4P, less of the sharing occurs over large distances, instead making requests of nearby clients (geographically). The NYTimes covers the development from the practical standpoint of Verizon's agreement with P2P company Pando Networks, which will be involved in distributing NBC television shows next month. So the network efficiencies will accrue to legal P2P content, not to downloads from The Pirate Bay.
Yeah! Lets increment a number that isn't actually carrying a numeric value! We no longer transfer peer 2 peer. Information is now provided by peers, 4 peers.
How do you reduce the number of 'hops' by an average of 400%? Negative number of hops? Also, FP.
God, root, what is difference ?
Network topology isn't & can't be a secret...
[Fuck Beta]
o0t!
So suddenly the BitTorrent protocol is illegal now?
... or is it encouraging to see network providers taking a stance other than p2p is bad? This looks good - kind of like "p2p isn't going away, so as long as we have to live with it, let's try to make the best of it"
I'm a student. I write iPhone apps.
It is by my will alone my thoughts acquire motion; it is by the juice of the coffee bean that the thoughts acquire speed
Reducing hops by 400%, eh? That's a nice trick. Can we reduce bandwidth usage by the same amount? I wouldn't mind some free bandwidth.
I honestly can't figure out where "reduce by 400%" came from. They say the average hops were reduced from 5.5 hops to 0.89 hops, which is either 84% if you're not an idiot or 616% if you are. So I'm really quite confused here. Go figure.
Breaking Into the Industry - A development log about starting a game studio.
While I understand what they're saying here, and I understand the surface intent of the message, I get this feeling that there is some sort of devious underlying motive here. Or it could just be that I have my Slashd^H^H^H^Htinfoil hat on a bit too tight.
That works out to an average 84% reduction.
But basically, if you're a pirate, this might make you nervous.
I record my sleeptalking
The NYTimes covers the development from the practical standpoint of Verizon's agreement with P2P company Pando Networks, which will be involved in distributing NBC television shows next month. So the network efficiencies will accrue to NBC's content, not to non-sanctioned P2P such as distributing open source software, free software, music, videos, and art in the public domain and licensed under creative commons, or to help distribute software updates for packages such as Azureus.
There, fixed that for you.
Twinstiq, game news
http://hosted.ap.org/dynamic/stories/P/P2P_VERIZON?SITE=MITRA&SECTION=HOME&TEMPLATE=DEFAULT
Honestly I think its kind of a cool idea, but the sad part is I don't really see how this could be done on a software level... I think thats why they're citing legal content only... it will take some modifications for routing equipment, won't it?
For this reason, Verizon doesn't suck for broadband uses. In my area, I have Verizon DSL (they haven't given us Fios yet, but they ran the fiber cables a few years back) and I don't have any port blocking (that's right folks, I can send email to ANY server), and they don't limit P2P or Bittorrent (My downloads are fast and fresh). And they haven't turned records over to the government (or at least not reportedly, yet). So far, in the category of BIG ISPs Comcast vs Verizon, Verizon is being the underdog. Which is funny, because start arguing cell phone policies and prices, and watch the argument change completely.
Belief? Hope? Preference?The Existential Vortex
Just because I appear to the network as pop-123.ny.isp.com doesn't mean I'm in New York. I could be halfway around the world.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
They keep touting this P2P protocol, but never actually say what it is. I'll assume it's bittorrent, unless they need to replace protocol with network. I'm guessing it's just the buzzwords that they like.
Absolute power corrupts absolutely. indymedia
And let's face it, people, the next protocol will have to have a few features to be accepted, and having "local peers" isn't on the top of the list.
What the list includes? Easy:
1. Encryption
2. Onion routing
For very obvious reasons. And neither of them decreases bandwidth used. Quite the opposite.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
less of the sharing occurs over large distances, instead making requests of nearby clients (geographically).
How about a BitTorrent client that gives preference to peers on the *same ISP*?
Yeah, less hops and all is great, but if an ISP can keep from having to hand off packets to a backbone, they'll save money and perhaps all the hue and cry over P2P will die down some. I'm sure Comcast would rather contract with UUnet to handle half of the current traffic destined for other ISPs than they do now.
Sort of a 'be nice to the ISPs and they'll be nicer to the users' scenario.
It would have made Internet broadcasting much more efficient, but it never took off. Why? Because providers never wanted to turn it on, fearing their tubes would get filled with video. So what happened? People broadcast videos anyhow, they just don't use the more efficient Mbone multicasting method.
Furthermore, when I download a video via Bittorrent, there are usually only a few people, whether they have a complete seed or not, who are sending out data. So how local they are doesn't matter. If there are more people connected, usually most people are sending data out at less than 10K, while there is one (or maybe 2) people sending data out at anywhere from 10K to 200K. So usually I wanted to be hooked to them, no matter where they are - I am getting data from them at many multiples of the average person.
I care about speed, not locality. The whole point of the Internet and World Wide Web is locality doesn't matter. Speed is what matters to me. For Verizon however, they would prefer most traffic goes over their own network - that way they don't have to worry about exchanging traffic with other providers and so forth. Another thing is - there is tons of fiber crisscrossing the country and world, we have plenty of inter-LATA bandwidth, the whole problem is with bandwidth from the home to the local Central Office. In a lot of countries, natural monopolies are controlled by the government - I always hear about how inefficient that would be and how backwards it would be, but here we have the "last mile" controlled by monopolies and they have been giving us decades-old technology for decades. In fact, the little attacks by the government have been rolled back, in a reversal of the Bell breakup, AT&T now owns a lot of last mile in this country. Hey, it's a safe monopoly that the capitalists, I mean, shareholders, I mean, investors can get nice fat dividends from in stead of re-investing in bleeding edge capital equipment, so why give people a fast connection to their homes? Better to spend money on lawyers fighting public wifi and the like, or commissars and think tanks to brag about how efficient capitalism is in the US of A in 2008.
What about if a torrent has no seeds or leeches in any remotely local area?
This is why any "massive improvement" on this aspect makes me skeptical. We all know the reason they want to tie it to local is to save bandwidth costs using only their own uploaders basically which would slow speeds down astronomically. Overseas hosts that can do 300KB/s or more on an upload vs a local that can do a cap of 40KB/s. You decide.
I conjectured a couple years ago that this could be done simply by matching up IP addresses to autonomous system numbers and picking peers that are in the same AS number in preference to other peers.
So let's see here. We have a serverless infrastructure (peer to peer), but it requires a server (tracker). The powers that be don't like it, because they can't take control of the servers.
So along comes an ISP with a new way to control P2P by turning the ISP's gateways into the server.
Does this not give the ISP absolute control over the traffic? When they have this absolute control, don't you think they might use it to their benefit? Don't you think it just might give them a bit of "intimate" knowledge of the client activities?
I see no need for this "technology". It is not an advance. It's a trap.
Modding Trolls +1 inciteful since 1999
Some of us working in the bleeding edge of p2p have been playing with these ideas for years to improve performance (I'm building open VR/MMO over P2P), here's the basics...
Most true p2p systems use something called a Distributed Hash Table (DHT) to store and search for metadata such as file location and file metadata. Examples are Pastry, Chord, and (my favorite) Kademlia. These systems index data by ids which are generally a hash (MD5 or SHA1) of the data.
Without going into the details of the algorithms, the search process exploits the topology of the DHT, which becomes something called an "overlay network". This lets you efficiently search millions of nodes for the IDs you're interested in in seconds, but it doesn't guarantee the nodes you find will be anywhere near you in physical or network topology space.
The trick some of us are playing with is including topology data in our DHT structure and/or search, to weigh the search to nodes which happen to be close in network topology space.
What they are likely doing is something along these lines, since they have the real topology instead of what we can map using tools like tracert.
If they really want to help p2p, then they would expose this topology information to us p2p developers, and let us use it to make all our applications better. What they're likely planning is pushing their own p2p, which will be faster and less stressful on their internal network (by avoiding peering point traversal at all costs, which is when bandwidth actually costs THEM). The problem is their p2p will likely include other less desired features, like RIAA/MPAA friendly logging and DRM, and then they'll have a plausible reason to start degrading other p2p systems which aren't as friendly by their metrics, such as distributing content they don't control or can't monetize... Then again, maybe I'm just a cynic...
Freenet seems to be designed primarily for anonymity, and I have read that it does not have the best performance. However, it does try to become efficient over time by moving frequently requested data around automatically on the various nodes in order to reduce overall bandwidth use and improve performance. That is, the network adapts itself to optimize for something.
I wonder if, in principle, using something like freenet would accidentally be beneficial for providers like Verizon, at least with respect to the issue at hand.
Is for ISPs to seed the most popular torrents. As torrenting uses the fastest peer to download its packets as a priority.
Bit Tyrant already does?
This has been my main criticism of "p2p" user-level networking for years. The selection of "peers" has no clue about network structure. The routing performance is just awful. Finally, someone is doing something about it.
One problem is that, from an endpoint perspective, it's tough to extract network topology and bandwidth. Hop count is only moderately useful. But there are a few tricks one can use.
There are several basic numbers of interest - bandwidth, delay ("lag"), hops,"bottleneck points" and commercial boundary crossings. Each of these can be measured.
Delay, or lag, is the easiest to measure. A few pings and you've got it.
With bittorrent, you're not committed to staying with a peer for an entire download. So you can observe the bandwidth of the peers you're talking to and preferentially use the higher bandwidth ones. You really have to transmit for a while to get a solid bandwidth number, especially since Comcast introduced "Boost" quality of service, which increases bandwidth allocation for a few seconds on demand, then reduces it.
If you do a traceroute, you'll usually observe that many hops show low lag (those are usually hops within a single data center) while others show higher lag. The number of high-lag hops is the number of "bottleneck points" in the path.
Commercial boundary crossings occur then packets cross from one ISP to another at a peering point. Users don't notice this much, but carriers are very interested in minimizing that traffic. Converting IP addresses to autonomous system numbers, as someone mentioned, can tell you when you're crossing a boundary.
So it's possible to collect enough data to do intelligent routing without much help from the network provider. What to do with that data is a separate question, but a solveable one.
2P or not 2P that is the question.
THis may turn out to be a classic economics case of the tragedy of the commons.
let's say that if I use P4P that the total path length (i.e. measured in router hops, or cable value, not distance) traversed by my data is less thereby not utilizing as much network resources. However suppose that by voluntarily restricting myself to nearby peers that my download time increases by a factor of 50% (just to make something up). Personally it costs me no more if I use P2P while everyone else is volunteering to use p4p. I'll get my downloads 50% faster and the heck with everyone else.
Of course if everyone did that then, more network resources are consumed than neccessary, the costs of my ISP rise and my downloads are slower.
But as an individual, if everyone else is obeying p4p I have an incentive to selfishly use p2p.
Thus the corporate pay-networks that can actively manage peering altruism and force users to obey to resptrict themsleves to local peers over remote peers can pull this off. They can prevent defections and get better netowrk utilization and possibly even better average performance to boot.
Whereas on voluntary networks p2p may be harder to enforce locality. It would have to be a new bit-torrent protocol in which peers would shun or share less often any nodes with long ping times.
Some drink at the fountain of knowledge. Others just gargle.
How do you possibly get an average of LESS than one hop, unless you're getting the file from yourself?
Usually when people are talking about hops, they are referring to routers. The only way you would not go through a router is if you and the source were in the same LAN. If you get an IP address from your ISP and it is one of the private ones (ie. 192.168.0.0/16) then you will likely have to go through a NAT machine before you will be able to see anyone. If your IP address is a publicly route-able address then it will most likely be in a LAN with your neighbors. There are ways to explain why 0.89 would be possible.
Claiming 0.89 hops is more interesting because they are claiming that others in your network are already downloading or have downloaded the file. It seems unlikely that someone in my own network would be downloading the same file, or would be seeding the file that I wanted. It seems to me that the most likely way that they could get 0.89 hops is by limiting the number of actual files distributed by their P4P software. Maybe they just had 10+ test files that just happened to be all over the network already.
This is really freaking obvious. I wrote a p2p application that cached based on search requests and then fetched based on router hops years ago, and presumed it was nothing new then. I strongly doubt this will be an unencumbered technology if it ever sees the light of day.
Existing P2P programs would support this sort of thing today if the programmers hadn't been busy trying to work around blocking and traffic shaping.
Seriously... if the ISPs would spend their time *making their network better* instead of *trying to break user applications*, then we could compare P2P programs based on efficient network usage (= download speed) rather than ability to avoid traffic shaping (which = download speed today).
-- The act of censorship is always worse than whatever is being censored. Always.
What you say is true, but there was an interesting paper which came out in 2007, which deals with inferring the low level network topology (detecing the presence of switches/routers and how they connect up the end nodes) using RTT time in a rather slick manner. They mainly tested it on an ethernet LAN, so it might not work as well over a wider network, especially in the face of redundant links, etc. in the network.
;-)
~digs out paper~ ah, here we go: "A Fast Topology Inference -- A building block for network-aware parallel processing" T. Shirai, H Saito, K. Taura
Pretty cool stuff, though it would be nicer if everybody could just publish their network structures in a handy parsable format
The snow doesn't give a soft white damn whom it touches. -- ee cummings
Congratulations to the multi-billion-dollar telecom megacorp and the Ivy League university whose reputation is more for building social clubs of rich kids than producing people of competence: the Econ101 concept of "transaction costs" has finally been revealed.
So fantastically-new is this concept that in the early 1600s, English businessmen created a map of that island to try to optimize their shipping routes.
*yawn*
Is Capitalism Good for the Poor?
No, it's pay for play, the way they want the internet to look. This is, of course, the opposite of net neutrality. Insultinly enough, it makes you use your equipment as part of their service. Only "secure" platforms will be allowed the privilege because it will require DRM. Expect P4P to look just like pay per view TV and P2P to see further interference. Without better regulatory oversight or a liberated spectrum, network freedom and software freedom are doomed.
Friends don't help friends install M$ junk.
The problem isn't 'university scientific' P2P traffic. The problem is everything else that is flowing on the network. Therefore by thumbing their noses at the 'rest' of what is downloaded, Network Companies have assured that they won't solve any problem of network traffic. Their business is not what flows across their network, their business is to keep *everything* running through those networks as smoothly as possible. Were I a shareholder in these firms I'd be quite annoyed that management seems to be eager to take on the liabilities of being a policeman and a priest, rather than on maximimising profits by actually focussing on the company's business!
No matter, the public will research and produce it's own protocol no doubt, and CISCO will make billions selling equipment to counter it. And so the silly, costly game will continue to be played without any real solution being found.
Clearly solving the actual issue of bandwidth use isn't in anyone's corporate interest.
-Gel214th
I think this kind of optimisation cleverly done with Ono plugin for Azureus bittorrent client. I also think it does even more.
Check details and remember it sends statistics (without private data, not torrent names etc) to the project domain. So, there is some FUD around.
http://azureus.sourceforge.net/plugin_details.php?plugin=ono
Azureus and Ono, both are open source (including Azureus 3) and massively multiplatform thanks to Java.
Mbone has been obsolete for years. Native multicast has been done with sparse mode PIM (for joins and prunes), BGP multicast NLRI (for source information), and a (1,*) approach for several years now. The (*,*) (S,G) (source,group) approach simply did not work out in practice.
Several ISPs offer multicasting as part of leased line services. There is little magic in it, and initial setup is pretty much fully automatable after assigning group addresses. Consequently, these ISPs usually don't charge extra for native multicast.
There are three big issues with respect to native multicast for content distribution.
(1). PIM-SM+BGP-mNLRI means few (one) transmitters per multicast group. This is generally called single-source-multicast (SSM), and scales to large numbers of G in (S,G) == (1,*) state. Multiple transmitters can feed into the single source through other channels (unicast or anycast, for example), and Mbone was ultimately shut down operationally by gatewaying Mbone groups through a single source controlled by David Meyer, then of the University of Oregon, several years ago.
P2P applications are not geared for SSM, and (1,*) native multicast is probably not useful for distributing anything other than tracker or other directory information.
Although one could align a P2P system such that each chunk gets its own S,G state, the processes for assigning group addresses is not well automated globally, and worse, the process for *announcing* active sources is essentially restricted to a combination of BGP and the equivalent of bulletin board pastings (i.e., out of band pre-announcements). This is not fundamentally different from how most P2P systems work now, but introduces more complicated machinery for little (or no) obvious gain.
(2). Most point to point connections (e.g. DSL) use routing equipment which does not transmit native multicast, does not talk PIM-SM, and/or does not gateway IGMP. Sourceward joins and prunes therefore require some anycast magic, and actual traffic requires some form of unicast tunneling. There are a variety of approaches to this, but they all introduce network overhead. This is unattractive for P2P.
(3). The nature of multicast leads to an efficiency gain when large portions of a S->G tree are shared among multiple joiners. Generally this favours live broadcasts; P2P is almost always recorded content, and while multicast can repeat the content in a loop with a schedule that helps clients determine when to join or prune, actual P2P that searches swarms for missing appears to be more efficient. A hybrid approach is possible, but the potential gain does not appear large enough to justify the additional complexity and overhead of (1) and (2).
(3) (a) Missing chunks / side channels. When multicasting to a large group, a source does not want to be buried in control messages (ACKs, NAKs), so flow control is almost essentially non existent, and missing chunks are usually replayed to a set schedule as per (3). Listeners need side channels to acquire that schedule. A P2P search is more straightforward and probably more responsive, and rare chunks may distribute faster in a reasonable swarm than waiting on a loop from a relatively slow SSM originator. Asks propagated to the SSM originator are known to cause swarm slowdowns for P2P; they can become unintentional DDOSes (or exploited to create intentional DDOSes) on multicast sources in any (S,G) model.
(4). Multicast does not have a flow control mechanism. A listener joining to a group takes all that group's traffic, or none. If the traffic exceeds bandwidth availability, the listener will observe many missing packets, and there may be collateral damage due to congestion. The only reasonable response a listener has is to unjoin the group.
There are two approaches to this which scale with (1,*) state, namely many parallel groups, where each group adds extra detail, and listenerward caching. The former runs into administrative issues (group add
Hey, Sam.
You want to get this server out of your DNS? It's hosting viruses linked from slashdot through a rds.yahoo.com forward.
That would be nice. Thanks.
Help stamp out iliturcy.