Open Content Network (P2P meets Open Source)

Orasis writes "The creators of Swarmcast have announced a new peer-to-peer content delivery network called the Open Content Network. The OCN will allow users to download open source and public domain content from multiple peers and mirrors in parallel. The system is designed to augment the existing mirrors with bandwidth from the p2p network and should eliminate the "Slashdot Effect" for popular open source content."

Comments

OK, but...

[scrm]

The Open Content site just announces a list of intentions. Anyone can put this kind of info up. It looks to me like nothing has been achieved yet, making this not really news.

Need for Checksumming

[skroz]

A secure system for validation and verification of downloads will obviously need to be implemented. Imagine all of the fun things someone could do if they, say, inserted a rogue module into the linux kernel code. Or the latest release of samba, gtk, glibc, Mozilla, ssh, openssl... the list goes on and on.

Re:Need for Checksumming

[popeyethesailor]

It is part of the specs

1.2 Untrusted Caches

It is currently unsafe to download web objects from an untrusted cache or mirror because they can modify/corrupt the content at will. This becomes particularly problematic when trying to create public cooperative caching systems. This isn't a problem for private CDNs, like Akamai, where all of their servers are under Akamai's control and are assumed to be secure. But for a public CDN, the goal is to allow user-agents to retrieve content from completely untrusted hosts but be assured that they are receiving the content intact. The CAW solves this problem by using content addressing that includes integrity checking information.

Re:Need for Checksumming

[Salamander]

It's probably worth pointing out that the solution to this problem is really orthogonal to the use of content-based addressing. Also, while signatures etc. can be used to verify the integrity and provenance of the delivered data, there's a whole separate problem of ensuring that it's current or consistent.

Re:Need for Checksumming

[56ker]

Most data on the web isn't current anyway.

Security question

[YanceyAI]

"indviduals will be able to contribute to the open source movement by donating their spare bandwidth and disk space to the network. "

Perhaps this is a silly question, but I worried about it with Napster and subsequent file sharing software, too. Is is possible to contribute and be secure?

Open Content, Open Source, whatever

[new death barbie]

Maybe it's just me, but this makes me a little nervous that when the "Open Content Network" gets too popular and dragged down in litigation, the "Open Source" folks are going to find themselves tarred with the same brush; guilty by association. Not what's needed at this juncture.

Re:Open Content, Open Source, whatever

[peddrenth]

Presumably everything on the network will need to be electronically signed by whoever uploaded it anyway (for many many reasons) so it won't be hard to track down whoever put restricted material on the system.

Openft

[z-man]

What about the openft protocol, they've been working on that for a while gift.sourceforge.net. They originally used the fasttrack protocol (KaZaa), but after kaZaa changed there specs, they decided to create their own protocol.

The Spirit of the OLD Internet Lives

[ausoleil]

...longbeards can remember the "good ole days" where the free flow of ideas and not making money were what made the pre-commodity internet a very worthwhile place to be. Everyone was expected to contribute their resources for the benefit of all, and none of it was (apparently) designed to make help smartass b-school dropout come up with enough cash to buy a 4,000 square foot "bungalo" in Palo Alto.

Count me in.

Guesses as to how long this is going to last?

[inkfox]

While this is a great concept, it scares me a bit.

I'm fully expecting that if we ever reach a point where a substantial percentage of users' traffic becomes outbound traffic, the cablemodem and DSL providers are going to start to rethink the current pricing and service packages.

How long before we find ourselves NATted away, able to originate connections only? A few cablemodem providers have already done this to reduce the traffic from file sharing and to knock out code red and other such silliness. And each time a major ISP does this, it leaves a slightly smaller number of other ISPs providing the outbound service, causing the traffic on the holdout systems to rise.

At some point it's going to snowball, and most of us are going to find ourselves NATted away, with only those paying premium prices for real IP addresses getting the priviledge of having their uplink monopolized by strangers.

Re:Guesses as to how long this is going to last?

[inkfox]

Why shouldn't this be the case? I have DSL and I have the AudioGalaxy Satellite setup to run at logon. Whenever I take a look at the satellite, I see that I have only received usally less than 1/10th of the data that I have sent. Why shouldn't people (individuals included) be required to pay for the bandwidth they use?

At current, I'd say say we are paying. We've signed a usage agreement that says we get this speed down and that speed up.

From a business standpoint, the upstream hasn't been the interesting part of the pricing equation to date, however. But when they usage increasing, and their costs going up for something they can turn off or collect an extra fee for, you can bet we won't see so many symmetrical connections or directly-addressable and unfiltered IPs.

Re:Guesses as to how long this is going to last?

[shepd]

>At some point it's going to snowball, and most of us are going to find ourselves NATted away, with only those paying premium prices for real IP addresses getting the priviledge of having their uplink monopolized by strangers.

Close, but no cigar.

There's no reason why a NATed box can't "upload" a file. The client simply needs to send a push-type request to the NATed server, rather than trying to pull the file from it.

This, of course, requires the client to have a real, non-NATed IP address. And this means the client will have access to more software in exchange for a higher fee per month.

Sure, you could be nice and use some of that bandwidth for sharing with the people who chose not to pay for the higher service level, but you would still benefit largely from it.

Re:Guesses as to how long this is going to last?

[BCoates]

NAT alone is not an effective method of preventing people from using p2p programs. All it does is prevent incoming TCP connections, so as long as someone in the network (well, some reasonable minority of peers) can get incoming connections to bootstrap people into the network, everyone can still comminicate despite the inability to get new incoming connections.

Good NAT bypassing is annoying to program (in the extreme case, it requires implementing something like TCP over UDP) but it's not a huge techncal hurdle, the main reason it's not commonly done is because too few people have hostile NATs for it to be worth the effort.

--
Benjamin Coates

Re:Guesses as to how long this is going to last?

[jilles]

True, however sharing from a NAT connection to another NAT is problematic since there is no way for one of the parties to address the other (as is required for a peer to peer connection). If only one of the parties is NATed, it can still initiate connections with a regular connection (this is how Kazaa and gnutella clients allow uploading for clients behind a firewall). However nobody can initiate a connection with the NATed box because it doesn't have an address. Thus if the majority of users is behind NAT, that would effectively kill p2p networks because it would be hard to establish a connection between the majority of nodes in the network.

However, broadband providers have an interest in p2p since it is a major reason for their clients to have broadband in the first place. A cheap modem connection will handle mail and instant messaging pretty effectively. Only when you start downloading mp3/movies/... you need the bandwidth they offer. Healthy p2p networks create a demand for broadband.

My hope is that as ipv4 addresses get scarcer, adoption of ipv6 will finally happen. This would largely remove the need for NAT.

Great idea, but I can see a problem

[Rogerborg]

The problem being that people are bastards.

• "The Open Content Network will work with the Creative Commons to use their machine-readable licenses to automatically identify open source and public domain content to be distributed through the OCN"

Why is this a problem? Well, what's to stop an ignorant or malicious individual wrapping up some content with an CC complaint license and injecting it into OCR?

I'm thinking of:

• Advertising porn with embedded html links that pops up adverts (gnutella is rotten with this stuff).
• Virii.
• Other people's copyrighted content.

Why would anyone do this last one? Pure malice, to open OCN up to DMCA attack, simply because people (as I said) are bastards, and can't be trusted to behave in a rational civilised fashion. OCN will be a trusted network, and that leaves it open to abuse. I really hope that an actual trustable human will vet everything injected into it.

Re:Great idea, but I can see a problem

[Disevidence]

One major problem i can see is that you'll have a few "unauthorised" mp3's, and the decss.exe file, and it'll be sued to hell and back by the MPAA and the RIAA.

People cannot be trusted. Maybe some sort of signup, registered usage is needed. Though given the caution of most OS people, that won't happen.

Swarmcast will MAME you

[BlueboyX]

Recently some Mame devs (www.mame.net) have been working on some Cojag drivers(cojag is an atari arcade system that uses harddrives - Area51 is a cojag game).

Someone made compressed harddrive images that mame will eventually require. Dispite compression, two of the images were half a gig and one was a gigabyte. The guy who was distributing these files used swarmcast to prevent getting swamped.

It worked pretty well in that tons of people were able to download those huge files without killing servers. However, swarmcast is new enough that swarmcast itself had some server problems. The server had to use an older version of swarmcast to be stable. That pretty much fixed the problem. It used to be that just hoasting ~40meg neogeo roms was nearly impossible, now it is possable to host half-gig files.

Eliminate the Slashdot effect?

[mtnharo]

Fools, little do they realize the powers they are dealing with!

such a good move?

[tps12]

Okay, I'm sure I'm with the rest of the slashdot communisty when I say that my first reaction was "wow, awesome, score another 3 points for Open Source and freedom."

But I've reconsidered. Before you mod me down, please read what I have to say.

Basically, we are talking about P2P filesharing here. Now remember, other P2P services, like Napster, Gnutella, and IRC, were all originally based on good, sound, legal, moral ideals. But in the course of time, they each became corrupt with those who would use the infrastructure for illegal filesharing and copyright infringement.

Now, I don't want to throw the baby out with the proverbial bathwater. And I don't want to get rid of a useful tool because of a potential for abuse, since by that logic we would not have silverware, cars, or handguns. But we in the Open Source community need to ask ourselves, is now the time when we want to risk associating Linux, *BSD, and Open Source with illegal activities? Don't we have enough anti-hacker rhetoric to fight against?

We need to pick our battles. This isn't one of them.

Re:such a good move?

[EvilAlien]

Excellent point.

The threat to filesharing as a technology comes from the rights-holders and from the legal system. Precident is being set all the time which threatens ISPs with liability for illegal activities on their network that they are "made aware of", DMCA notice and takedown letter or not.

Its much more scary in the UK and in Canada. Canada just passed new legislation which will make ISPs vulnerable for distribution of child pornography on their network. SOCAN Copyright Tariff 22 was just it through the Court of Appeals and makes ISPs liable for infringing material stored on their "cache servers".

P2P technology may by-pass these, but it is only a matter of time before some powerful organization convinces some judges that ISPs should be held liable for allowing P2P on their network. Blocking of ports, account terminations, and worse are all coming if the rights-holders have their way... and technology such as swarmcast which acts to distribution free software will likely get lumped in with the Napter/Kazaa/Foo P2P technologies.

Re:such a good move?

[BCoates]

Now remember, other P2P services, like Napster, Gnutella, and IRC, were all originally based on good, sound, legal, moral ideals. But in the course of time, they each became corrupt with those who would use the infrastructure for illegal filesharing and copyright infringement.

Nonsense. The original purpose of both Napster and Gnutella was to enable the sharing of copyrighted music. That's about all they've ever been used for (well, Gnutella and Gnutella-like networks have since branched out into other forms of mostly copyrighted content)

This, OTOH, appears to be primarily designed to let people pool bandwidth, which is both legal and useful, and since bandwith costs are a big problem for the distribution of free content, it's entirely a good thing.

--
Benjamin Coates

Re:such a good move?

[asv108]

Of the three P2P networks you mention, well really 2 since IRC is not P2P, Gnutella and especially Napster were basically created with the transfer of illegal content in mind. There are certain steps you can take to control what type of content is published to a P2P network. The easiest way to do this is have an activity log & authentication, but OSS folks would never go for that. A good example of a 100% legal public P2P network is the Furthur Network which has managed to stay legal by limiting what people can share by specific bands that allow taping. This system could possibly do the same thing by only allowing certain files to be shared and implementing cheksums so when you downloading a deb package your not getting goatse.mpg.

Re:such a good move?

[EvilAlien]

Sure, further litigation. What a nice waste of money.. thats the problem with dumb laws, it costs a lot to counter them. Common sense is free, but law != common sense.

Uninformed

[BlueboyX]

Maybe you didn't notice that these guys are the makers of Swarmcast. Or maybe you posted before figuring out what that meant.

Swarmcast is a (working!) program for parallel p2p file downloading. In other words, the technology IS implimented. They basically are just making a modified program to work with a somewhat different set of files. No biggie.

Those were the good old days, all right...

[gaudior]

When the Internet was available only to career college students and others feeding at the public trough. The internet was paid for by public tax money and corporate subsidies, but unavailable to most people.

The good old days really weren't so good. It's kind of funny, though, listening to some so-called old-timers constantly whinging about the commercialization of the 'net. Do they really think the huge advance in capabilities would have come about without the economic incentive? Sure, the early days were inventive. They invented the bricks and mortar of the Internet. But the commerce guys have driven the construction of cathedrals, roads, libraries and schools with those bricks.

Kazaa doesnt download in parallel

[BlueboyX]

You know how Download Accelerator lets you get files faster by downloading different parts of the same file via multiple simultaneous connections. That is what this is about, except the Swarmcast guys have each connection going to a different person rather than tons of connections going to the same server.

Eliminate the slashdot effect? bah!

[ipmcc]

If the goal here is really to eliminate the "Slashdot Effect" a much more effective solution would be to set up a network of load-balanced caching proxies on geographically distributed fat pipes.

Some will argue that this is in essence what a P2P network is, but why not do it right, using technology we already have that everyone can use(squid.)

Other users' comments regarding the cumulative effects of NAT on P2P networks are incredibly apropos.

But realistically, theres nothing I love more than when the story submitter posts a link to a Google cached version of the content he's posting. We're an agressive bunch and that calls for aggressive measures :)

Re:Eliminate the slashdot effect? bah!

[Salamander]

If the goal here is really to eliminate the "Slashdot Effect" a much more effective solution would be to set up a network of load-balanced caching proxies on geographically distributed fat pipes.

Who pays for all that equipment and bandwidth? The idea here is not to solve problems by throwing resources at a problem, but rather to solve them by using existing resources as effectively as possible. The technology involved can be applied to any resource base. The technology-intensive approach using almost-zero-cost resources might well make significant headway against the Slashdot Effect, even if you still think your capital-intensive approach based on older technology is even better.

Another factor you seem to've overlooked is that software like CAW or BitTorrent are distributed for reasons beyond scalability. For example, consider the inherent attack-resistance characteristics of a highly distributed P2P network, vs. your centrally-administered servers. There are other goals as well, such as avoiding legal culpability or financial dependence on corporate benefactors to provide the systems and bandwidth. Whether you agree or disagree with those goals, the fact remains that many people believe in them. Networks like you describe are old hat, dozens have been deployed already, and yet a lot of people still want something different. You've proposed a solution to a different problem than the one Onion Networks et al seek to solve. There's a term for that; we call it missing the point.

Re:Eliminate the slashdot effect? bah!

[Thing 1]

If the goal here is really to eliminate the "Slashdot Effect" a much more effective solution would be to set up a network of load-balanced caching proxies on geographically distributed fat pipes.

I started a project to help distribute the load. So far I've written code to pull out all the links from each Slashdot story. What's left is to cache those links, then transform the Slashdot main page HTML so that it points to the cached versions.

If anyone's interested in taking this further, you can find the Perl code here .

Then just set up some machines (say, 3 to start just to test it), and cache the main page to all three machines, generating different HTML pages for each machine. When a new request comes in, round-robin it to the 3 machines. (Better algorithms can come later.)

I agree with what you say about geographically distributing them, though, which would need to be handled by an entity with money (i.e., VA?).

Wheres the code

[nervlord1]

In those now famous words, wheres the code?

Im sorry i wish i could say im excited, its certianly a VERY good idea, and noe in desprate need of realising, but untill i see the code, its just more hype.

Certianly be a great way for non coders to contribute though, so many times my linux friends say "oh id love to contribute to open source but i can't code", this would definatly be one way, and one which requires very little effort too.

Re: Open Content

[peddrenth]

"I would love it if this were completely true, but this merely means this band got 1 sale -- from 1 person"

I like your logic. So obviously the record store is failing when I go in and buy my one CD. From one person. My one CD from one person. To clarify, only one CD from only one person. The record shop is obviously failing.

Yeah right.

No, it means that MP3.com is succeeding, that Aura are succeeding, and that Faithless are shafted (who made the CDs I wanted, but which I'm not gonna buy with the current state of the record industry, and their political representatives)

Here's a hint: when you have lots of people buying one thing each, you make lots of money. Find a maths book. Revise the chapter on multiplication.

MD5 sum

[chill]

Well, for one example, the new RedHat 7.3 .iso files have MD5 sums embedded in them. From the boot prompt, type "linux mediacheck" and it will prompt for a disk to be validated.

A feature to take detatched/attached MD5 sums, GPG signatures or the like could be pretty easily added in.

You're right, it is needed.

Re:If you NAT'ed i hope it's IPV6

[autocracy]

Sounds nice, but to site another point: What happens when they get somebody who knows how to do packet filtering? Drop all inbound SYN packets and...

split, cat, gnutella

[chill]

Okay, how about:

split -b 65m filename.iso filename.iso.

breaking the 650+ Mb iso in about ten 65 Mb chunks with the suffixes .aa-.aj.

Share them on Gnutella, KaZaA and any other P2P services.

Once downloaded, cat all the files together into one and check the MD5 sum (also downloaded, or embedded like RedHat 7.3 does).

The perfect task for Freenet

[autocracy]

Read the line above. Now, somebody be ambitious and make a package for everything freenet needs in one RPM / deb package and put up a system to make it known. Now we play...

Re:The perfect task for Freenet

[delta407]

Throughput - Browsers will be able to download content from multiple sources in parallel
Bandwidth Savings - Browsers will automatically discover and select the closest mirror for a piece of content.
Fault Tolerance - Even if a site goes down in the middle of a download, browsers will automatically locate another mirror and continue downloading.
Scalability - Any number of machines may be added to the network, creating a CDN ad hoc, with very little administration.
Security - Browsers will be able to safely download content from untrusted mirrors without risk of corruption or viruses.

Right on target. Freenet accomplishes these goals, and actually works right now. Freenet is essentially an anonymous, distributed caching system into which anyone can insert data and retrieve it later. It supports both locating information by content hashes or by a human-readable redirect, as well as lots of really cool features like anonymous websites ("freesites"). So... what are you waiting for? Install Freenet today!

</plug>

1 of many alternatives.

[leuk_he]

There are pleny of other open p2p products.

Freenet scaleable, not vaporware, very much beta.

Alpine.
based on trust
Gnunet. Sounds very open. based on electonic money. also seach for gnet.

chord Very efficient to find files.

distrinet At this stage: vaporware.(there is code....) But if you look at the description it beats any p2p software!

But in the end the network with the most data (gnutella/kazaa) will be used. Note that users will switch networks very quickly. Look what happened to napster.

Re:If you NAT'ed i hope it's IPV6

[Ian Peon]

I keep on hearing this, but the truth is the net was rolling back when you had to obtain a 3rd party TCP/IP stack to make Win 3.1 work.

If the net went IP6, Microsoft would either release a patch for WinME,2k,XP, and I bet some 3rd party would offer a d/l to support 95/98 (maybe even 3.1!)

so, if this covers all open content, mabye

[the_2nd_coming]

this is what we need to make the open music movment happen.....people will make music, license it as being free to trade, and then folkes will do more and more of it.....who knows, mabye this can become the "good example" needed to show the courts that P2P file sharing can be done with out infringing the rights of others, and even lead to some mainstream artist releasing some music on the system to advertise.

Is "Open Content Network" just a PR thing?

[WEFUNK]

Using P2P networks for this kind of caching is something that is long overdue and if these guys can pull it off in a major way then I'm all for it.

I'm just not sure that I buy the description of the "Open Content Network":

"the OCN will allow users to download open source and public domain content from multiple peers and mirrors in parallel."

I presume that it could just as easily be used for copyrighted material and is in no sense different from Napster etc. in its restrictions and potential (read: probable) use.

Sounds a bit like a PR thing: our network is for Open Source material, if people use it for other things - well that's none of our business.

Personally, I agree that they should have the right to focus and brand themselves however they want. I also agree that they shouldn't be held liable for the type of files users actually submit (unless they're either actively screening them or branding themselves as the "Illegal Warez Network" or something). I'm just not sure that this approach will help to limit their liability (although I sure hope it does). Or did I miss something and they are proposing some method of ensuring the content meets some guidelines, thus avoiding any of the Napsteresque controversy?

A Law-Abiding Distribution Network

[HappyCamper]

I'd like to draw your attention to the Globe Distribution Network (GDN), like OCN, a content distribution network for freely redistributable software. Its design specifically addresses the problem of deviants abusing the network to distribute other people's copyrighted works and illicit content. In particular, it requires all content published to be digitally traceable to the publisher. If, after publication, someone finds that this content is not free software the content will be removed and its publisher blocked from the network.

The GDN furthermore offers a scalable solution to the problem of finding the nearest replica (i.e., a scalable URI resolver service in OCN terms), and facilities for dynamically replicating content in areas with many downloaders.

Publications on the GDN, the underlying Globe middleware, and its initial implementation (BSD license) can be found on http://www.cs.vu.nl/globe. The best description of the anti-abuse measures of GDN are found in the paper titled ``A Law-Abiding Peer-to-Peer Network for Free-Software Distribution'' published at the IEEE NCA'01 Conference.

No way!

[IamTheRealMike]

They stole my name! http://theoretic.com/?action=history&id=Mike/OCN-F AQ

I was thinking about this only 3 or 4 days ago.....

hehe. Ah well, I'm glad somebody else is doing it really, I have more than enough on my plate right now. Perhaps they should check out the Creative Commons?

Logi$tic problems of root CA

[yerricde]

The network operators can kinda identify who they issued the certificate to.

Which means the network operators will have to make deals with notary public offices in every major metropolitan area in all 180-odd independent countries order to be able to certify that people are who they say they are. This can become expensive, and the total cost of maintaining a certificate may rise up to $200 per cert per year, making this situation no better than the SSL cert situation.

Re:Logi$tic problems of root CA

[Orasis]

Our initial approach is actually much simpler. We will simply certify a number of domains that are trusted to host mostly open source content, such as kernel.org, debian.org, etc, and use a network of metadata proxies that extract the secure hash information for file verification. Idealy each of the sites will run their own meta-data proxy so that the secure hash information is trustworthy.

Safety in intention

[sterno]

Actually the OCN should be safe from attack. The reason is that from the get go is intended to distribute content that is totally legal to distribute. Now certainly some may abuse it, but the problem that Napster ran into was that it so clearly built it's business model on make a profit from contributing to piracy.

The courts are unlikely to shut down a network like this that makes a good faith effort to be legitimate. Most other P2P services establish themselves as trading points for all manner of illegal content. They try to cover this up to look good to the courts but there's no doubt that Kazaa, etc, wouldn't be this popular were it not for piracy.

BitTorrent

[bramcohen]

BitTorrent enables downloaders to send pieces to each other when they have an incomplete file, making almost unlimited scaling possible. Simple multi-source downloading can be good for performance, but still is limited by the server's upload capacity.

We've had several large deployments of files which are a couple hundred megabytes and up, getting sustained downloads of a couple hundred downloaders at once, serving off a dsl line, and it's worked well.

By the way, BitTorrent, Swarmcast, and OCN all check secure hashes under the hood, so data integrity isn't an issue.

This already exists.... it's called USENET

[mbourgon]

I'll probably get modded down for this again, but why not just use Usenet?

Set up an alt.binaries.geektoys and post all the Videogame Demos, Distros, Open Source Software, Movie Trailers and the like that we're all interested in. You can use RARs, PARs, SFVs, etc to make sure the file is downloaded properly. Then the only issue is making sure the checksum matches up with what you got off the original web site.

ISPs already carry Usenet, so the infrastructure is set up, and this is definitely a useful, non-infringing use of Usenet.

"What was I downloading? Why, the latest version of Mandrake!" Sounds good to me...

Finally!! Thank you!

[zilym]

I've been hoping someone would do this for quite a long time now. No more playing with slow mirror ftp sites when I want to get my Linux upgrades.