Slashdot Mirror
P2P Through Firewalls
An anonymous submitter writes "A few stream-through-firewall applications have been announced recently. p2pnet has an interview with Ian Clarke about his new 'Dijjer' program, which promises to reduce bandwidth requirements from HTTP servers by transparently distributing the load. Slyck.com has an article about LimeWire's new version that offers firewall-to-firewall transfers (code here). [Both Dijjer and LimeWire are GPL'd.] There's also been a lot of discussion on the p2p hackers list about reliable UDP transfers."
p2pnet.net News:- Freenet author Ian Clarke is developing Dijjer, a new open source p2p content distribution tool, and he's looking for people to test drive it before it goes online in beta.
"Dijjer is a peer-to-peer HTTP cache, designed to allow the distribution of large files from Web servers while virtually eliminating the bandwidth cost to the file's publisher," he told p2pnet.
"Dijjer is designed to be simple, elegant, and to cleanly integrate with existing applications where possible. Dijjer uses "UDP hole punching" to allow it to operate from behind firewalls without any need for manual reconfiguration.
"Dijjer's distributed and scalable content distribution algorithm is inspired by Freenet."
Below is a brief Q&A.
p2pnet: When did you start working on this?
Clarke: Several months ago. It's hard to pinpoint a specific time because it's a combination of a variety of ideas that have been at the back of my mind for quite some time.
p2pnet: What prompted you?
Clarke: Dissatisfaction with apps like BitTorrent, and a desire to demonstrate that the ideas behind Freenet could be applied to solve other problems.
p2pnet: When do you expect (hope) it'll be completed?
Clarke: Well, I'm sure that development will continue for quite some time, but I hope to release a beta version in four to eight weeks that will be suitable for large-scale adoption.
p2pnet: Who do you see as the principle users?
Clarke: Anyone who needs to distribute large files to large numbers of people but who can't afford to pay for the bandwidth that this would normally require.
The download site says features include:
"No Firewall configuration
With many P2P applications you must reconfigure your firewall to get the most out of them. Not so with Dijjer, we use state-of-the-art "NAT2NAT" techniques to get the most out of your internet connection without any reconfiguration.
"Sequential downloads
If you tried to download a video through Dijjer you may have noticed that you could start watching the video before the download completed. This is because Dijjer behaves like a web server, pieces of a file are download in-order and fed to your web browser when they arrive, allowing your browser to start displaying content before it has completely downloaded.
"No "Tracker" necessary, works with virtually any URL
This is a big one, Dijjer will work with almost any direct URL, the content publisher doesn't need to lift a finger - they may not even realise that people are using Dijjer to save their bandwidth costs!
"Cross platform and native compilable
Dijjer is implemented in Java, meaning that it will run on Windows, Linux, and Macs. Those who don't wish to install the Java Runtime Environment (JRE) will be pleased to note that Dijjer can be compiled with the GNU Compiler for Java (JCJ) to native code thus eliminating the need for a JRE. Native compiled versions of Dijjer will be available from this site in due course.
"Free as in Speech
Dijjer will be released under the GNU Public License.
"No cumbersome clients
Dijjer downloads through your web browser or preffered HTTP download application. You don't need to learn to use yet another P2P client user interface.
"Advanced scalable distributed caching algorithm
Dijjer uses a highly scalable distributed caching algorithm inspired by Freenet. This will allow it to deliver faster download speeds while placing less burden on the web server, and will be better able to handle sudden increases in demand for content."
"Now all I need are some people to help me test it,"says Clarke.
TCP has slow start / back-off retransmit problems that for long transfers over links with some packet loss can cause it to not fully use the pipe.
Most modern UDP transfer systems use NACKing, where the receiver just tells the sender if it didn't get a packet (the packets are numbered sequentially) and that it should put it in the retransmit queue. The sender just goes about it's business spewing out packets until it's informed the receiver didn't get one.
I have bittorrent behind my firewall. Rather than statically allowing ports, I set up a "torrent" user, and told the firewall to let it listen for connections. This also has two beneficial side effects. First, if there's an arbitrary code vulnerability, an attacker can be somewhat contained. Second, bittorrent doesn't always use the common range of ports, so prioritizing by port is problematic. Having a seperate user lets me throttle the bittorrent connections so that interactive traffic has priority.
While I imagine this is possible with Linux, I have no specific knowledge of how to do it. I did it with PF on OpenBSD.
I rarely criticize things I don't care about.
LimeWire doesn't have any spyware, at all. In fact, it has absolutely zero bundled software. Even with the free version.
Having written most of the udpconnect code in LimeWire, the basic idea works fairly simply. The downloader starts pinging the desired uploader with UDP SYN messages. At the same time it uses what we call a PushProxy in Gnutella to tell the uploader to start doing the same thing. So then both computers are sending UDP SYNs. This makes the NAT/firewall open up to this traffic and the LimeWire hosts on both end respond to the UDP SYNs with UDP ACKs in order to identify their connection ID.
After receiving the ACKs, the connections can send UDP data messages in both direction safely. The only trick is you need to ensure that a message is sent every so often so the NAT/firewall doesn't close. If nothing else is sent, a special KEEPALIVE message is sent. Beyond this, the communication is somewhat similar to TCP with a FIN message shutting things down at the end.
"UDP hole punching" is a simple technique, already used by many games, to allow two computers behind NAT firewalls to talk directly to one another.
Basically it works because UDP doesn't work very well with NATs, and so the NAT has to have a very general policy on what it forwards. UDP is a packet (datagram) based protocol. Each UDP packet is actually just an IP packet with two extra headers added - the source port and the destination port, and then just the data. So how can a NAT know which host on the local network it should send a UDP packet to? It can't really, so it is forced to guess, and the classical way to do this simply to forward incoming UDP packets with a given source port to a host that recently sent an outgoing UDP packet from that source port.
This allows hosts behind the NAT to open something like a server port, by simply sending packets from a certain source port out to the Internet regularly, thus making sure that packets sent to that destination port from the Internet will be sent to them. Note though that this also reveals the scalability problem with UDP and NATs: if you have many machines sending UDP packets from the same ports you get a problem.
On modern, stateful, firewalls, the NATs are slightly smarter, and will only forward the UDP packet to a node in the internal network if that recently sent a packet from the destination port of the incoming packet, and to the host that the incoming packet was sent from. This makes it impossible to act as a general "server", but UDP hole punching is still possible if you have an intermediary who can tell two NATed hosts to start sending UDP packets to each other with certain port values. This means that a non-NATed host is still needed, but it doesn't need to forward all the traffic between the two others, like it would with a proxy solution.
Blah, I meant this to be short, but instead I wasted my time writing a long slashdot post, and now there is probably already a +5 with a shorter description. Everybody mock me...
Just tried it out, so this is speaking from actual experience. Digger doesn't limit itself to sharing files you have already downloaded - it will *actively* download files other people are requesting, so that it can share them.
This is simmilar to freenet, and indeed will maximize everyone's bandwidth. But it has grave issues when not combined with Freenet's huge anonymimity factors like encryption and hiding IPs , and will open you up to all sorts of legal problems.
I don't want the FBI knocking down my door because my Dijjer client has been downloading kiddie porn for someone else without my knowledge. Sure, I *may* be able to argue in court that it was not me, and hey, I may even be able to prove it. But is that potential trouble worth my saving on some bandwidth? I think not.