Slashdot Mirror
Decentralize BitTorrent with Kenosis
UnderScan writes "Eric Ries, writer/programmer/CTO, authored an article 'Kenosis and the World Free Web' at Freshmeat [Owned by Slashdot's Parent OSTG]. Kenosis is described as a 'fully-distributed peer-to-peer RPC system built on top of XMLRPC.' He has combined his Kenosis with BitTorrent & removed the need for a centralized tracker. He states: 'To demonstrate Kenosis's suitability for these new applications, we have used it to improve upon another peer-to-peer filesharing application that Just Works: BitTorrent. BitTorrent does one thing incredibly well. Using a centralized "tracker," BitTorrent manages efficient distribution of data that is in high demand. We have extended BitTorrent, using Kenosis, to eliminate this dependence on a centralized tracker.'
See also the Kenosis README for details on using Kenosis-enabled BitTorrent."
n. Christianity
The relinquishment of the form of God by Jesus in becoming man and suffering death.
Thinkin' Lincoln - a web comic of presidential proportions
How is the RIAA and MPAA supposed to stamp out bittorrent if you guys keep improving it? Where's your compassion?
Then this falls a bit short of the "killer p2p app" moniker that it *almost* deserves.
The old Lie: Dulce et decorum est Pro patria mori
QUite useful, of course! We could distribute spatial-data, and Wi-Fi locations to PDAs and laptops in this way. There are metric tons of useful applications for BT and K.
Zhrodague.net - I do projects and stuff too.
Decentralization is generally useful for any application where failure of some critical node results in failure of the entire transaction. Distribution of any data via bittorrent will benefit - regardless of content - if there is a possibility that a tracker host could fail.
From the feature list...
Kenosis works in almost any networking environnment, including restrictive corporate firewalls, because it uses XMLRPC for its network communications. It can also work with an HTTP proxy.
This alone makes a worthwhile project, for those stuck behind firewalls/proxies.
This will probably considered a troll also, but I think the obvious answer is porn.
And welcome to KBTR (formerly K/.), all Bit-Torrent stories, all the time.
Enough, already!
Athletic Scholarships to universities make as much sense as academic scholarships to sports teams.
One of the problems with BitTorrent is that the trackers themselves can get overloaded with too many clients. If this system can eliminate something like that happening then that'd definately be a good thing.
;)
That being said, the busiest torrents I've seen are for copyright-infringing material, so I guess it's still a boon for piracy.
"People that quote themselves in their signatures bother me" - athakur999
We all knew this was coming, but would this app get this kind of exposure had the MPAA not cracked down on those BT tracker sites?
It is just like Scour net (web based/centralized), then napster (p2p/centralized), then kazaa (p2p/decentralized). Every time they go after a technology, they force it to evolve into the next phase. They will never win IMHO.
I'm thinking that your comment is a combo of both insightful and redundant :)
i use linux and windows oh god how can i have an opinion
Allow oppressed people to anonymously distribute large incriminating videos of their corrupt government?
The problem with this approach is dealing with untrustworthy peer. Without substantial protections, one peer can ruin everybody's downloads.
Piracy isn't the only activity one might want to keep private and decentralized. Political dissent in totalitarian states, for example, is another.
Help prevent the slashdot effect; stop reading the articles.
Well, since there is a central DNS server at bt.kenosisp2p.org, how can they sincerely declare this to have no central point of failure? Yeah, of course dns propagates, but turn off this central DNS server and in a few days everything is gone, right?
3 is "Get sued by MPAA anyway". Step 4 refers to the studios' profits.
Look, so long as the MPAA goons can trace at least one packet of a movie download to your IP address, you have liability. And with BT, as long as you're downloading, you're also uploading. Use torrent, and your ass is exposed, regardless of whether the index is centralized or decentralized. Call me paranoid, but that's how I look at it.
When will the Empire^H^H^H^H^H^H *AA ever learn?
You have two hands and one brain, so always code twice as much as you think!
This is an important step, but it still does not hide the user's IPs from the *AA.
From the Article:
It does not address problems of anonymity, privacy, or distributed data retention, although we hope to address these issues in future versions.
If I understand this correctly, this doesn't affect communication between the bittorrent peers, just the client and the tracker. It still won't work through an HTTP proxy.
It's fun to see how book-writing hackers act.
;)
What, you mean, by using the right tool for the job instead of language snobbery?
I think I found a defect.
This thing doesn't make any fucking sense.
I was really excited by this slashdot story, because I think something like this could be very very useful. I have to say that I was disappointed a bit by the download.
No docs or pointers at the top of the tarball.
One of the READMEs on the site says try "test.py" for an example, which seems to just hang.
Elsewhere it says to fire up bittorrent
trackers and clients.
There clearly is a lot of work that has gone into this, and the idea sounds really promising, but it looks like it needs a better end-user documentation before it's ready for primetime.
Remember, with enough lawyers _everything_ you do on your computer or download from the Internet may be considered illegal.
OTOH - if - for example - you crack a proprietary video codec so that it is suited for full-length movies distribution, add a cracked proprietary sound codec, name it all after a proprietary technology, then take some others' source, tweak it a bit, urge other peoples to contribute promising that it will be "free for ever", then demand money for it - it is still ok if you form a company! You can even put your certification on hardware players and stuff. Voila. (Yes, I do troll, "mod me down". But better yet - reply).
Because you have a fixation on money like some I've been acquainted with.
Seriously... I don't see how to make money off this...
Seriously...why is that important? Did you even read the article? The author of this BitTorrent enhancement does not even use the word "money"--it is WAY down the list of motivations for its creation, not does it seem to be about getting pr0n and warez. This guy sounds like an idealist in a very true sense--it's about decentralisation of control--making content available without being reliant on central servers.
I think this would be immensely useful. The reliance on central BT trackers has been shown to be BTs primary weak point--once a torrent is located and transfer is initiated it is incredibly robust.
Besides the fact that the admins of BT trackers are being harassed into submission by MPAA and RIAA, the more popular trackers seemed to be quite unreliable. If this innovation (open sourced to boot) addresses the reliablity issues in LOCATING the content that BT is so good at DISTRIBUTING then it could be start a dramatic shift in how we use the Internet, much like the WWW was.
It doesn't even have to be about piracy. Used within a VPN or on a corporate WAN it would make distribution of a large number of big applications much easier to distribute. I make VMWare and ghost images of machines that are many gigabytes and this solution would be a great way of distrubuting them to a large customer with global sites (keep in mind that these clients are legally permitted to use these images--my employer is a stickler for that).
A small operator could distribute software this way and save on the costs and time associated with maintaining a critical server with big pipe to the 'net. Security patches could be distributed this way very effectively without reliance on a single entity for distribution. The possibilities are endless. It might not be a money making machine, but it is the kind of thing that (if it works well) could change the face of computing.
I just read about Kenosis from its homepage. And, I'm forced to ask:
Do we really need yet another bloated python p2p app? I can feel the flamebait and troll mods comming.. but seriously: Python sucks at gui work. It has to use generic wrappeers, like wxPython, that are extremely inefficient. Sure, like Pearl or Java, you can write gui apps using Python... but they always come out slow and over-weight.
Consider the BitTorrent client. Just running the application, without an actual torrent being transfered, consumes 23 MB of memory (on Windows) -- for that cheesy, very simplistic little GUI. When you actually start running a torrent through it, it'll easily chew 40 MB's and gobble considerably more CPU time than a comparable program written in C/C++.
I'm not saying Python isn't a useful language... But it was not designed to run P2P apps.
Just because a programming language can be extended to creating GUI applications does not mean it's a good idea. Python's strengths are elsewhere, and I for one am tired of the BitTorrent community using it to write p2p clients in.
Now go ahead and mod me down for having a modicum of common sense.
/dev/random
If you read the article carefully (or not so carefully), you'll note that this product does NOT include a fully distributed / decentralized tracker... an web server tracker is still necessary for the initial torrent retrieval. If that tracker becomes overloaded / unavailable this system will have real value, but there's still an originating central tracker for the MPAA to go after.
However, it's only a very short matter of time. The author explains that such a thing could be easily created with this framework. Clearly he could have done it if he wanted, so I'm guessing this is a purposeful strategy on his part to avoid any potential direct or indirect personal liability or legal issues down the road...
-R
Now we are just waiting for a platform-dependent implementation in C++ and MFC that is supposed to be faster because it's "native code", which all the clueless kids with 8mbit internet connections are going to download...
Even network-limited programs like a bt client still need to worry about GUI responsiveness and memory usage. It would be insane to write the first implementation of such a program in C/C++ -- Perl and Python were given to us by gods to prototype these sorts of projects -- but once the basic protocol and UI behavior has been figured out, I would try to rewrite the client in a statically compiled (or at least a JIT-capable) language.
RTFA and read it properly. The server is merely a pretty interface for older BT clients that will search the decentralised version and return the tracker address of the last known tracker for that item.
That's got nothing to do with the decentralised network itself.
The problem with Kenosis is, of course, it's reliance upon a central DNS server to point to a list of distributed trackers. Many will undoubtely point out, that this DNS server could be taken off, and that's it.
Now how can we really circumvent this problem? One solution would be to advertize a list of DNS resolvers on USENET. A preconfigured list of newsgroups could be used to bootstrap this, and new usegroups (should the original newsgroups get closed) could be regularly advertized as well. A client would just go to those newsgroups, and fetch the updated list of DNS servers, newsgroups etc...
This system would be much more resilient to attacks by RIAA or MPAA because they won't have a single point to attack. Closing newsgroups is much more difficult than taking one DNS server from the upper zone.
Another way to advertize the DNS servers would be via spam! Yes, you didn't misread this. One can easily encode the location of DNS servers in spams and have clients read those spams, effectively extracting an updated list every now and then!
This is very important, because spam is already used as a covert channel to prevent traffic analysis. Specialy crafted spam checkers can extract useful information from spams. One such information would be the distributed location of trackers (or DNS servers that point to them).
Just because it's unethical (to piggy back useful data on top of spam), doesn't mean that it's not already used on a quite wide scale. There's no reason why it shouldn't work on a new generation of distributed BitTorrent trackers!
cpghost at Cordula's Web.
I see several weaknesses in this, however. If I wanted to attack this, the route is pretty obvious to me. Several lines of attack stand out:
1: Start serving falsely-labelled file data that is correctly represented internally. There appears to be no moderation system built in, so bogus file data will pollute the system.
2: Start serving any file data that is inaccurately represented internally. For example, make all of your hash entries but one accurate, but make that one hash entry inaccurate. Users end up downloading most of the bad file before it errors out. Depending on the setup of the server and client, they may continue trying to get the data from elsewhere, in which case you could serve larger amounts of corrupted data, possibly by using bad clients working in conjunction.
3: Hash cracking: Brute force hash cracking could allow fake data to pass as real, hash-matching data; only a single cracked piece per file is needed. This would probably be economically inefficient, however, compared to #1 and #2 in terms of the ability to disrupt network usage.
4: Mass peer suits. If BT is the download manager here, getting the people who have the file being shared is laughably simple.
There's probably also some risks for their proposed change to allow multiple seeders, but I'd need to think about it for a while.
1, 2, and 3 require an "intelligent" client. In real life, we inherently weed out those who give bad data simply by our experiences and the experiences of those we know. The more we trust a person, the more we trust what they tell us about others. This sort of system tends to inherently isolate out the bad apples, even if they work together. Even if, working together, they manage to convince a good client that they're right and others are wrong, that good client too will simply be viewed as a liar and its data shunned. Overall, the system will remain intact. It's no easy programming task, however; yet, it is doable, as evidenced by the fact that we, as humans, do it every day.
#4 has a simple solution: Involuntary mirroring. If this system would automatically force the mirroring of data into a cache on the destination machine, and serve it from there, there would be no way to know whether a person was actually uploading copyrighted material or simply acting as a "router". Since our law has finally started to catch on to the fact that it is unreasonable to sue those whose computers pass through illegal data that they had no realistic way of knowing about, it would effectively anonymize *all* data on the network.
Hey, guys, I'm just pleased as punch to report that it's a fleet of a hundred Vogon Battle Destroyers!
Lets simplify this. You are a program that doesn't know anything about the world, because you are a de-centralized program. You are started by your master ("user," in human speak). What do you then do? Who do you connect to? Surely if you had an address hardcoded somewhere you would no longer qualify as being decentralized. Do you start walking the IP space, trying to connect to 1.1.1.1, 1.1.1.2, and so on? Oh, so the IPs you have coded in your config are "only hints," huh? Okay, then you should be able to cope with all those "hints" having gone bad. When those hints are all bad, what do you do, Mr. D. Centralized Program?
Decentralized, my ass.
Must-not-watch TV!
There is a ton of good legal content that will be created once the bandwidth issue is solved. It's sad that the default comment is "well this sucks because the **AA will still be able to track me down when I use it to break the law." Most of use see the cultural usefullness of these things but the handfull of anarchists among us are hurting the movement.
The fact that this can get through firewalls and that it won't fail under heavy load (as happens with bittorrent trackers) are the important things.
What if Digg added local news and a Slashdot inspired comment karma system? ---
http://houndwire.com
I was thinking that it was more funny than insightful, but then I remembered that porn is a very serious thing not to be taken lightly here at /.
(I am one of the authors of Kenosis.)
We are planning improvements to Kenosis in a number of areas such as better integration with BitTorrent, a more distributed BT tracker, simulation of larger Kenosis networks and making Kenosis work over NAT.
We'd love help with any of these or other areas.
Please join the mailing list to get involved.
Azureus is an open-source Java-based BitTorrent client with a built-in tracker.
While this looks like a good start, this isn't likely to catch on until it can be installed from a single .exe file for windows users. Then it would have to have one GUI that provides a seemless interface for finding and downloading .torrent files distributed among Kenosis nodes, and then automatically starts downloading the files using the Kenosis distributed trackers.
what sig?
1: Start serving falsely-labelled file data that is correctly represented internally. There appears to be no moderation system built in, so bogus file data will pollute the system.
Can't be done. There is a moderation system built-in to BT. The SHA1 hashes which identify the file chunks simply cannot be 'correctly represented internally'. If you know a way of doing this without changing the chunk size by several gigabytes, I think some crypto researchers would like to talk to you.
Any reasonably coded client will start ignoring a peer who delivers enough bad chunks.
For example, make all of your hash entries but one accurate, but make that one hash entry inaccurate. Users end up downloading most of the bad file before it errors out.
Which will leave you with no 'seeds' but the original one. People would wise to this quickly.
Hash cracking: Brute force hash cracking could allow fake data to pass as real, hash-matching data; only a single cracked piece per file is needed.
See above. This is nowhere near practically possible. You have to add gigabytes of data. A chunk is far smaller than that. It won't fly.
#4 has a simple solution: Involuntary mirroring. If this system would automatically force the mirroring of data into a cache on the destination machine, and serve it from there, there would be no way to know whether a person was actually uploading copyrighted material or simply acting as a "router".
This would not work. First, most net users have lower upload speeds than download speeds. Second, and more important is that this requires a lot of bandwith for helping someone else. Expect a hacked client to be produced which does not proxy anyone else, saving their bandwidth, but leeching others'. You could create a tit-for-tat scheme where someone will only proxy for you if you proxy for them. On the other hand, bandwidth is still the main constraint here.
This may be off topic...
I was wondering what the slashdot community thinks of the estimates that BT uses 1/3rd of available internet traffic.
Considering that it is a bit more private and "exclusive" than things like Kazaa and Edonkey does that number seem possible. You know. It takes the install of the basic BT binary as well as the GUI client of your desire. Then you need to find a decent BT website/ community.
If this figure IS true, wouldn't making BT even more viable eventually choke the internet?
Thoughts?
It would be ironic that something that was designed to "manage" bandwidth may end up hogging all that is available.
DHTs work like this: Every node on the network has a 160 bit identifier. Given a key, through the DHT we can find the node whose identifier is 'closest' to the key. In Kademlia, the closeness of a node is quantified by treating its identifier XORed with the key as an unsigned integer. The node with the smallest such integer is the closest, and is therefore responsible for the key.
.torrent as the key (extracted from the tracker URL, in the .torrent). So say you have a .torrent whose tracker you would like to eliminate. Just choose your node identifier, when you join the network, as either equal to the hash in the URL or close to it (such as by simply flipping one of the lower-order bits). That way, you will with near-certainty be the closest node to that hash, and thus be designated the tracker for that torrent. Now just ignore all requests from clients.
If you look at their readme file, they're just using the hash of the file kept in the
It can easily be done.
- sm
There is a ton of good legal content that will be created once the bandwidth issue is solved.
Oh really? Watch the big copyright owners sue the independent producers of so-called "good legal content", alleging that the so-called "good legal content" is in fact subconsciously copied from an existing copyrighted work. It could happen, especially with music.
And because some people won't get it, this Informative post will explain.
;-) 3.11 was a hacked version of the Microsoft MPEG 4 layer 2 codec used for .asf files. A french hacker, Jerome Rota, extracted this codec and made it work for .avi files.
DivX
Wikipedia seems to skip a bit of history here, not mentioning the cracking of the sound codec etc, but does conclude with the information on "DivXNetworks, Inc" creating a Clean room design version of DivX (similar, in my opinion, to editing someone elses paper enough to make it not technically plagarism) and releasing it as DivX 4.
For context, click Parent.
Basically a bunch of speculation, that COULD happen. I COULD get hit by a bus ... Call us when it ACTUALLY happens.*
Had you taken the time to click the link, you'd see a bunch of cases leading up to one where it has happened. In Bright Tunes Music v. Harrisongs Music , George Harrison got sued and lost for inadvertently copying a song on his solo debut album.
I can only speak hypothetically, but considering that the BitTorrent client, GUI and TheShad0w's experimental client (BitTornado) are all written in Python, writing Kenosis in Python makes sense, since if it becomes popular, plonking it straight in to the main client could be achieved far easier than if you'd written it in any other language. (Of course, I haven't really researched this - it's an unproved hypothesis...)
catch (HumourFailureException e) { e.user.send("You, sir, are a humourless idiot."); }