Finally Real P2P With Brains

dfelznic writes: "The mp3 archives of CodeCon are now availble, which is news in itself. But what makes this real interesting is that they are being distributed by BitTorrent. BitTorrent allows users to download a file from multiple different people. Instead of everyone nailing one server, users get the file from other users. Furthurnet uses a similar technology to distribute legal bootlegs of concerts. The archive is available at the BitTorrent demo downloads page. As soon as I started downloading (cable modem) at around 300k I got a request for the file and began uploading at 40k. This could be the answer to the slashdot effect;) Now, who is going to be the first to complain about the use of mp3s instead of oggs?"

Comments

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Nice.

[shankark]

That's true. But I think a workaround for this would be to have md5 signatures computed for each of these parts and verify them before they are downloaded. I'm not sure if this isn't being done by others already.

Been there done that...

[mo]

I hate to rain on the parade but Morpheus et al. as well as the latest version of BearShare both do this, and have for some time.
When you say p2p with brains, to me it means somebody has come up with a elegant balance between centralization and search speeds.

Re:Been there done that...

[Wintersmute]

When you say p2p with brains, to me it means somebody has come up with a elegant balance between centralization and search speeds.

Ditto, Holmes. The real question is the scalability issue, and I'm not convinced that the traffic cop features implemented by Gnutella front-ends have really sorted this out.

When that's the case, that will be some p2p with brains. Right now, the networks only seem to be hanging on because the critical mass of crash-inducing traffic hasn't hit the super-peers yet; at least not on the permanent basis.

What would really make my evening interesting is if someone would be kind enough to contradict me.

And this is new?

[Edgewize]

Could someone explain how this is different from FastTrack (Kazaa), eDonkey, or the more reputable Swarmcast?

Peer broadcasting is hardly something to write /. about, I'd say.

Re:And this is new?

[mo]

They're marketing BitTorrent as a solution to web providers with bandwidth limitations. The client registers a mime-type so when you click on a BitTorrent download link it hands it to the p2p client which then downloads it from the network.
The technology is nothing spectacular, but it's nice to see a simple install method that integrates nicely into the browser.

One interesting side-effect of this implementation is that there is no searching. You only download stuff from BitTorrent if you find a link on a web page for it. However, without the requirement for searching, Freenet would be a great replacement for this role of browser-download accellerator. All you really need to do to implement this would be to provide a nice installation .exe of freenet that could parse a meta-file pointing to the freenet key of the object you wanted to download.

Re:And this is new?

[PureFiction]

Sure. Fast Track is decentralized file sharing network where there is no guarantee that what you ask for is what you get. They may be codecon mp3's, they may be nasty midget porn incognito.

eDonkey likewise is more of a filesharing (aka, keyword search, then dowload hits) method.

Swarmcast is the closest relative to BitTorrent, but BitTorrent avoids the FEC encoding and cryptographically secure block verification in favor of a more centrally controlled broker that uses multi source downloading at various offsets to accomplish the same task.

In short, BitTorrent is a distribution system where a central server provides content, and peers requesting that content join a mulitsource downloading group where they also share offsets of data with each other (preferably) and download from the central server when necessary.

This isnt file sharing (really), this is content distribution in a fast and effective manner using peer networking concepts.

Comment removed

[account_deleted]

Comment removed based on user account deletion

It is NOT a p2p Network people!

[Com2Kid]

It is a browser plugin (IE) that creates mini distributed networks based around a website.

So say you start downloading the latest Counterstrike patch from some server. Well you know how servers giving out the CS patch get filled up quickly.

Well if the users were running this program (plugins to IE, no restart neccisary, look if there is a {browser here} version yourself!) then when they started downloading somebody ELSE could start downloading FROM them.

No file synch issues (same file, same source) the server just re-directs future downloaders to current downloads and has the original downloaders forward the files along.

BitTorrent

[Eloquence]

I'm sure Bram will notice his server being slashdotted soon enough, but let me say a few words in defense of BT anyway. What makes BT different from Morpheus and BearShare is that files are sent by users to each other, while they are still downloading. This way, the downloaders themselves act as backup. It's not simple multi-source downloading, but targeted towards content-providers who want to reduce load on a central server. In its advantages and disadvantages it's similar to Multicast. Good for high load for specific files at specific times. Kernel.org should use it.

eDonkey has the same feature (with some differences in the publishing process), but is really an application of its own, very file sharing oriented, closed-source and banner-supported. Not exactly what a content provider would want users to download before they can access his files. Still, ed2k has the advantage of a large user base, and also supports ed2k:// URIs that can be used on webpages.

SwarmCast is interesting, but the company behind it mostly died, and now it is somewhat in limbo. Its Java base has made it problematic as a desktop application. The only real alternative to BT is Mojo Nation, which is currently being reworked as "MNet".

If you want to know what CodeCon is all about, check the Feature box on infoAnarchy, we had some detailed coverage.

What about Gnunet?

[Anonymous Coward]

I'm very surprised at the little ammount of attention that GNUnet has gotten in the P2P arena. GNUnet is anonymous, distributed, encrypted, reputation based, has accounting, allows for distributed queries, and uses dynamic routing. While GNUnet is still beta software, I think it's a great anti-censorship tool. What all this means in non-buzzword speak, is that you have a tool that combines a lot of the great qualities from other similar networks (FreeNet, mojo nation, etc) and doesn't have all of the short comings. Give it a shot.

Kazaa/Morpheous/eDonkey compairisons

[goofy183]

There seems to be a lot of people who really haven't read the site or understand how the technology works. Yes all those P2P filesharing utilities allow you to download the same file from multiple people at once, it's not all that impressive and many of the problems such as validating matching files and such have been worked out.

This solution is different in a few very large aspects. It allows a company to keep track of who is currently downloading a file from their webserver. This information is then sent to the clients who can start the P2P poriton of the process and download segments of the file from other users, releaving the load on the companies server. In contrast to those other P2P FILE SHARING programs which share all your files not just ones you are currently downloading. A system like this makes the file server not only the original source for that file but the P2P server to find other people to download that ONE file from.

I can see where people may not want their upload bandwidth being used by others. For this reason any site implementing this feature would probably end up having to provide the file for normal download. The selling point would be a possibly faster download for users of the technology.

I would personally love to see huge sites like FilePlanet put this to use. Granted it would only be truely usefull for sites that have a constant stream of concurrent downloads for a file at any point in time but it would be much better than having to wait 2 hours in line to download a file :-P

Re:Will not help the slashdot effect

[Com2Kid]

Uh.. . .

The idea was to help sites that GET linked to BY Slashdot. /. itself is not, AFAIK, having bandwidth problems.

You know, those small user pages with some cool casemod on it?

This network would allow viewers of the site to download the images from EACH OTHER instead of from the main server.

multiple identical different same distinct

[Dr. Awktagon]

BitTorrent allows users to download a file from multiple different people.

Or if you're downloading the latest boy band single: multiple identical people.

I'd like to hear peoples's experiences

[Anonymous Coward]

I'm Bram Cohen, the author of BitTorrent. This little slashdotting seems to be going well so far from my end (over 40 downloads currently, and still going smoothly) but I'd like to hear about peoples's experiences doing the download. Here are some questions -

• Are you getting pauses where no download is happening? If you are, please be patient, it should kick up again (or start in the first place) after a while.
  
• Are you behind NAT? People behind NAT may be getting worse performance, it's a complicated issue.
  
• How's your upload/download ratio? There are enough people now that you may see the phenomenon of getting about the same download rate as your upload rate - Cutting off your uploads wouldn't help with this, your peers would just get pissed off at you and stop uploading (I'm not kidding, it has tit-for-tat leech resistance.)
  
• Did you run into any technical glitches? It's still fairly young software, so there may be a few little things to iron out.
  

So far, this looks like it's going pretty well. Any and all feedback is much appreciated, and will hopefully help make BitTorrent an even better product. Please mail me about your experiences.

Re:I'd like to hear peoples's experiences

[XRayX]

1. Sometimes I get massive speed-drops (around 5 k/s), but no freezes until now.
2. I'm behind a NAT. May it be possible to configure an incoming-connections port?
3. Very Various. At The Moment it's 30 K/s down (max 90 k/s) and 7 k/s up (max 14 k/s).
4. No problems! Plugged in pefectly into IE.
Very good work so far. I'll try to set up set up some files later.
X

bootlegs : recordings :: crackers : hackers

[Wakko Warner]

Please, call them "legal live concert recordings", not "bootlegs". That's like saying "legal pirated MP3s".

- A.P.

Re:bootlegs : recordings :: crackers : hackers

[mark_lybarger]

just to clairify.

bootleg:

1. To make, sell, or transport (alcoholic liquor) for sale illegally.

2. To produce, distribute, or sell without permission or illegally: a clandestine outfit that bootlegs compact discs and tapes.

it was very hard to find someone using the term bootleg to not mean anything more than a live recording though. lots of people call even a bands released live album a bootleg. here's another definition i found, sorry no link, i could only get it on google cache:

"When someone tapes a show, that is called a live recording. When a company releases an unauthorized copy of that show, that is called a "bootleg". Bootlegs are usually found in compact disc form. However, a CD can only hold approximately 78 m inutes of recording time, forcing the bootlegger to cut songs out of long shows. In essence, a live recording will maintain the original, unadulterated full show while a bootleg version will have songs missing. In fact, they may even be out of order.

When individuals trade live recordings, no money is transferred or involved. However, when someone buys a bootleg, someone is making money--and usually a lot of money--off music that someone else wrote and performed."

Actually reasonable requirement, useful feature

[billstewart]

Some environments, like the Gnutella/Napster/Freenet things, have communities that hang around connected for a long time even if they're not downloading anything. But others, like distributing a new release of RedHat/Debian/Mandrake CDs, or even just Mozilla, have a lot of users who want to show up, download stuff, and leave. This feature makes it possible for them to be a temporary community providing services to each other without requiring longterm committment. If you download a CD using BitTorrent, you're useful for 95-99% of the time you're on line, rather than being consumer-only for the first 100% of the download time and having to hang around for another 100% of the time to be any use to anybody, so the community scales much more cleanly even if the first thing you want to do after downloading the latest Linux release is install it. (Software's a much different usage pattern than music here.)

Additionally, it makes it very efficient for the first set of people who are downloading the file. Instead of having to download the whole thing from one source, which is probably overloaded, you're able to download pieces from lots of different people. The server takes advantage of this - instead of giving Alice chunks 1, 2, 3, ..N in order, and giving Bob the same things, it spreads around the load, so Alice is downloading chunk 1 while Bob downloads chunk 2, and when they're done, Alice starts downloading Chunk 3 from the server and Chunk 2 from Bob, and other chunks from Dave, Eve, and Freddie if they've gotten them.

This also reduces the latency required for later people in the process to get their material - instead of waiting for the entire 600MB CD to be copied N times in a row, the downloading gets pipelined.

"chaining" is DIFFERENT than "swarming"

[rubybroom]

I'm not completely versed in morpheus/kazaa/bearshare/whatever, but I understand they allow you to download a file from more than one other person simultaneously, known as "swarming" the download (btw, this is called "anteloping" on furthurnet). It is my further understanding that you can only do this from people who have the *complete* file.

What bitTorrent (I think) and furthurnet (I know) are doing is different than this. If 5 people are downloading a file from the one person who is sharing it, those 5 people can be the beginning of 5 chains of people, relaying each packet down the chain as they get it, regardless of whether or not anyone has the complete file.

Furthurnet uses a protocol called PCP (Packet Chain Protocol) to do this, and it automatically arranges the chains so that those with faster upload speeds are toward the top, with the dialup users toward the bottom.

If the main host goes offline, even if no one on the chain has the entire file, everyone on the chain can still continue downloading everything that the topmost person on the chain has already saved.

A good example: say a dialup user has large file that is in high demand. A T1 user comes along and spends a long time downloading it off of the dialup users horrible upload speed, and gets about 80% of it before anyone else comes to download. Then you show up with your cable connection and instead of being at the mercy of the upload speed of the dialup guy, you have access to 80% of the file from the plentiful upload speed from the T1 guy. And of course Furthur knows to hook you up to the fastest open slot available when you come along.

The result of this is that the underlying host and network shape becomes transparent, and you just see a list of shows to download, you start downloading one, and all this stuff happens in the background. The longer everyone stays connected to the network, the more efficient it comes because it has more time to structure it with the faster folks in the "middle", and the slower ones on the "outside".

Over at furthurnet, the current record is having 71 people on a downloading chain. Combine PCP with the Anteloping and you can have some serious improvement over "dumb" p2p.

I wont even go into the benefits of the md5 checking furthur does...

Lossless - big files - Bittorrent helps a lot

[billstewart]

If you're shipping around small files, like MP3s, there are lots of transfer systems that can do the job. But the Lossless Compression movement for music means that a concert tape is typically a few hundred megabytes large, maybe 1/3 the size of the uncompressed original, so it takes much longer to download, just as ISOs for Linux distributions are large. In that environment, you can't always depend on connections being up for a long enough time, so you need to be able to download parts of files, and swarming systems like BitTorrent help a lot.

Re:My NAT configuration.

[bramcohen]

How the hell is it that my upload is working at all? I'm on a network so private that it's scary.

BitTorrent connections are bilateral, so you're able to both upload and download on all connections you have to your peers, regardless of which side initiated them. Your uber-NATing keeps anyone else from connecting in, but once you establish a connection out it can send in either direction.

Upload/Download ratios and ADSL

[osolemirnix]

How will tit-for-tat leech resistance work if someone has an Asynchronous DSL connection? If my download bandwith is 768 kbps but my upload bandwidth is technically limited to 128 kbps (as is common with many DSL offers for private home users), will the leech resistance feature think I'm guilty?

Re:Upload/Download ratios and ADSL

[bramcohen]

Yes, there is a general tendency for people to get about the same download rate as the upload rate they provide, due to the tit-for-tat algorithms. That's just a general tendency though, practical download and upload rates are dependent on many factors.

As to whether being on ADSL makes you 'guilty' I don't know, it's very non-judgemental software :-)

Re:Nice BUT....

[treat]

MD5 is *not* suitable for ensuring that two files are identical when a malicious user is involved. It *is* suitable for ensuring that a malicious user may not hand you anything that passes but pure garbage (given what we know about MD5 today).

I challenge you to find me any two sets of data with the same md5.