Slashdot Mirror
Clay Shirky Defends P2P
richard writes: "Clay Shirky has responded to Jon Katz's article, Does P2P Suck?, (and a WSJ article published the same day) in an article titled "Backlash!" on OpenP2P.com. Shirky says: "P2P means many things to many people. PC users don't have to be second-class citizens. PCs can be woven directly into the Internet. Content can be provided from the edges of the network just as surely as from the center. Millions of small computers can be more reliable than one giant server. Millions of small CPUs can do the work of a supercomputer. ... These are sloppy ideas, ideas that don't describe a technology or a business model, but they are also big ideas, and they are also good ideas.""
As far as IMG being the GUI, without question it laid the way for GUI approaches to the net. Witness clickable image maps, the server side of which followed pretty closely on IMG's tail. Without IMG there was no need for GUI approaches so I for one don't mind them being synonymous.
I think that's more than a mindset; I think that's what most of the public wants to be.
In fact, that could be the disconnect at a lot of the dot-com woes. And possibly at the stalling of a lot of the "systems" in first- and second-world countries.
As /. users, we are almost all active participants. If there's a conversation going on, we want to be a part of it. If something new is on the horizon, we want to hear about it. We want our thoughts to be feedback.
Joe and Jane Sixpack don't. They don't want Tivo to help them program their own network; they want to switch on the box and have the network programmed for them. They don't want to worry about schools; they want to send their kid to school and have him or her come back educated. They don't want power generation in their backyard; they want the switch the light switch and to have the light come on.
They make a few decisions about where they live, what they do with their time, etc. but for the most part, they don't want to be heavily involved. They know the schools are broken, but they don't want to be the ones to fix them. They know politics is broken, but their reaction is to drop out and not vote. They just want things to work.
The active 10% is out here trying to bring new approaches to them, but they don't really care. Too much change is interesting to us, a whirlwind world we enjoy. We have an advantage in it, because we understand a lot more of it. But change is a problem to them. They don't care for instability; even if it's a pain in the ass, they want to get in their car and go pick up groceries and haul them home, because that's what they've always done and doing what they've always done is appealing in a topsy-turvy world. The more we, the active, push for innovation, the more the passives resist.
A friend of mine is a passive. He's even a computer guy. Back in 1986 I asked him why he didn't get a PC. "I use computers all day at work, I don't want one at home," he said. In 1993 I showed him Usenet. "It's a lot of trouble to read all that stuff, and then have to write back," he said. It was like more work to him. Now he has Internet access, I asked him why he didn't check out /. and other such high-profile sites. But I already knew: he's a passive. A good guy, even a competent, intelligent guy, but he wants life to come to him.
If we can understand these people, we can improve their lives (and they can pay us dearly to do that), but we can't expect them to take an active role. And so when you say to a passive "...all you have to do to run Linux is..." don't be surprised if the answer you get is "No, I don't have to do anything." So the challenge of not only the open source world, but the entire world, is to create paths of least resistance. If they have to do something, they won't. You have to figure out how to make them want to do something.
Ugh, I'll stop now before I become Katz.
Part of the problem is the public's mindset of 'consumer'. The Internet has the ability to promote the average person to an active role, but is being strangled by those who believe it is nothing more than television with a keyboard.
A lot of the community oriented services such as IRC and email have begun to erode that image, but there really haven't been any new truly successful technologies stressing community in years. The public has been sucked into this thing called 'the web' and been taught that's all there is to this Internet thing.
The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
The risk of pandemic infection is present in any environment without diversity. In the computing world, we have more diversity than I think any sane person needs.
Of course, with every major advance we make there is always an increasing risk of mass destruction. There is an end to all things. Do nothing and an outside force will destroy everything. Do something and at least you are in control.
The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
I never particularly liked the term peer-to-peer. I'm not exactly sure who originally coined it, but it seems to cause a lot of confusion with other technologies which sometimes piggyback ontop of it.
P2P, Distributed Aggregation and Distributed Computing are three separate but related things.
Peer-to-peer is simply a type of network where all nodes on the system are on equal standing with each other. There are no dedicated server machines, no dedicated client machines, but rather everyone is both a server and a client and they communicate with eachother as equals.
This type of system lends itself to a very interesting change in the way someone finds information. Instead of going to a place (e.g. slashdot.org) to get information, you go to the information to get a place.
Distributed aggregation is a method of intelligently locating and well, aggregating resources distributed among nodes across a network. Whether these resources are files, CPU time or disk space, the method of aggregation should remain basically the same. This fits in very well with the peer-to-peer model to provide each node with a simple way of locating resources on other machines.
Distributed computing is a method of using resources distributed among nodes across a network. Distributed aggregation can be thought of a part of distributed computing as you have to be able to find the resources to use them, but not all distributed computing systems provide or even need a method of handling dynamic changes in the network. Of course, distributed computing systems are not typically peer-to-peer. Individual nodes on the network rarely communicate with each other to share information, but instead handle jobs in batch fashion and push the results up to a central server.
Many have argued that peer-to-peer has existed on the Internet since time began and that all things are basically peer-to-peer. This is quite true in some respects. At the protocol level, machines communicate with other machines in a manner that can be considered peer-to-peer, but historically at the application level there have been a very clear line between servers and clients.
We currently live in a world where the majority of computers are nothing more than glorified dumb terminals utilizing only a small fraction of their computing power. My hope is that one day, the average person won't "use" the Internet, but instead "be" the Internet.
Of course, that's just my opinion. I could be wrong.
The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
Well obviously since UDP is a connectionless protocol, it solves your problem by...
:)
Uhh
Uhh, oh hell it actually makes the problem worse.
I like the last one in particular:
As a testement to that my main "workstation" is a K6 233. Workstation is an exageration. I use the 233 mostly for www, and email. I have a more powerfull machine, but it is for games. What do you mean by "be the Internet"? I thought at first that it mean that each person can serve information directly to others. I think that we are pretty close to that now. Many of those people who want to, run good presonal web sites. So I have decided that "to be" the Internet must be something greater. Something that the cyberpunk authors haven't predicted yet. I think that in order "to be" the Internet our interfaces (mostly www browers) have to change completely. In my mind "to be" anything is an active role. Currently I find much of the Internet to be passive in nature. I think our interfaces encourage that.just some random after bed time thoughts
Disclaimer: I really don't know who the heck Clay Shirky is and I haven't read much of his writing. This is just an observation.
Its was in fact Mosaic that created "Web interface as GUI", by making clickable pictures.
-clay
There are two basic solutions for dual NAT commnication. Unfortunately neither solution is very usefull for a majority of P2P applications.
:)
1) Use a proxy that is not behind a NAT firewal. This opens up problems all its own, which is fairly obvious.
2) Use UDP. For a number of reasons, a lot of people hate this idea as well.
The only hope (currently) is one of two things. First, that NAT firewall vendors will implement a suitable solution to bypass this problem (unlikely) and second, that we all get fixed IP addresses out the ass (about as unlikely).
So, in short, break out the wallet for those beefy relay servers!
(or use UDP
The ALPINE Network uses a flat direct connection network for searching/discovery operations.
:/
There are a few tweaks which improve the efficiency of this type of network, such as a reputation/affinity value attached to peers to keep you connected to the best, while quickly filtering out the worst or dissimilar.
The communication is multiplexed over a single UDP port and can handle hundreds of thousands of concurrent connections at the lowest layer. (higher level ALPINE connections require more overhead, and are restricted to 10,000 to 100,000 depending on user preference)
At any rate, my point is that you can use a simple packet routing architecture like IP to accomplish a flat, large, directly connected network that is usable.
If you want higher performance, more efficiency, and greater throughput you would need to start experimenting with some of the advanced network architectures you mention. However, the chance of such a network reaching the masses any time soon is pretty slim.
Ok, I should have been more verbose. Here is how it works with UDP through NAT.
1) Peer_A opens a UDP socket. Sends a packet to a well known server, or servers, that simply send a reply that contains the source IP and port of the packet.
2) Peer_A records this source IP and Port, as it is what the NAT gateway is masquerading its connection as.
3) Peer_B does the same. It know has its masqueraded IP and PORT from its NAT gateway.
4) Peer_A and Peer_B can now send packets to each other at their respective masqueraded IP and PORTs.
They recently changed their terms of service mandating that Juno can use its customers' computers for that very purpose. They can terminate your service if you don't leave your computer on 24x7 so that it can do the processing and dial in to Juno at whim for more data. The fun details are at www.byte.com/column/BYT20010222S0004.
Naturally they portray this as a benevolent thing and a chance to be part of their "Virtual Supercomputing Project," which claims to be completely voluntary, despite the fact that their Terms of Service directly contradict this:
Happy computing! :)
Cheers,
Katz is a tabloid journalist. He writes whatever will get the best ratings^W^Wmost comments. People respond accordingly. Now someone from the side he's attacked defends themselves, and we have the same reactions; some defend their position, some attack it.
I fail to see how any of this is really that remarkable. So p2p is the latest buzzword. So what? So long as we have marketroids who have to make quotas or journalists with deadlines, there'll be buzzwords. And wherever there are buzzwords there will be people to attack or defend them.
Personally, I'm going back to play with some cool bleeding edge stuff that might just be involved with a buzzword or two. I don't care, since I think it's cool in and of its own right.
Just because you're paranoid doesn't mean they're NOT after you.
IT's the normal everyday sheeple that don't get this. THe fact that it's true doesn't make them understand any better...
You and I and lots of others know how the internet works.. we don't like the 'centralized broadcast' way it's starting to be used.. and don't like how people insist that p2p is something 'new'.. but think about this.
For mom & pop jones out there.. it IS something new. Sure they could have always done it.. but are just now realizing it. To them, it's NEW. The applications are new... everything is new. So it'
s good to have articles like this....
By definition, is SETI (and distributed.net) considered to be P2P? I would have thought of it as distributed computing; then again, so is P2P...
once the shoe shine boys and taxicab drivers start talking about stocks, its time to get out
:)
Nice, I never thought of it that way
P2P is a revolution in the making, and tradition businesses are trying to crush it... It's as simple as that.
It will succeed, however, simply because it's gained enough momentum that it cannot be stopped. And because it cannot be controlled.
You sound so upset P2P systems have gained such popularity that you're no longer 37337 with IRC and ICQ, etc. I dismiss that as immaturity. Listen up and learn, maybe.
Clay Shirky is a savvy guy, and he has a point that P2P is a good idea, albeit a sloppy one just now. The main thesis of his article is that power resides not only in centered servers (this is Sun's doomed wishful thinking, exactly), but also exists and is growing on the edges, as widespread use of easily acquired and highly capable software triggers the law of large numbers and a shift to topple a tipping point, to utterly overwhelm those few evil hegemonists who seek to exert centralized _control_ in order to extract artificial scarcity based revenues from the large mass of networked connected people. Your anger is better directed at those who charge $15 per CD, $9 per movie.
Seeing what's going down and writing intelligently about it are two different things. I'm not entirely convinced you did the former, but you are convicted by your own posts of being incapable of the latter. Let me put it in simple terms for you. We're having a discussion here about the potential of peer-to-peer services at the network edge as contrasted with centralized client-server models. It's a current and rather interesting topic. Your negative posts are not advancing the conversation. Please take it outside.
Difficult, but far from impossible. Mesh routing - which is really what you're talking about here - is a much-studied field and pretty reasonable solutions to all of the major problems are known. Check out Routing in the Internet by Christian Huitema for a pretty good overview of the relevant theory and practice.
IMO one of the big problems with P2P is that too many P2P implementors are either ignorant or disdainful of related work in this and other areas - usually both, which is a bad combination. I went to the O'Reilly P2P conference in SF a couple of months ago, and overall it was fantastic, but I did notice one thing. Everyone there seemed very sophisticated about crypto and security etc. but at the same time most were stunningly ignorant about routing, protocol design, performance management, and a bunch of other fields. There were exceptions, of course, don't get me wrong, but there's still way too much fad-following in the P2P community and not enough solid science or engineering.
Slashdot - News for Herds. Stuff that Splatters.
That looks like a nice paper. Thanks!
Slashdot - News for Herds. Stuff that Splatters.
I beg to differ. One of the major things differentiating supercomputers from anything else is the presence of *huge* internal memory and I/O bandwidth. That will never be duplicated by a distributed group of machines, and so for a certain very large and important class of programs distributed computation will never be a substitute for supercomputers. Fortunately, there are enough problems out there that *do* partition very easily, so that distributed computing is still worthwhile.
Slashdot - News for Herds. Stuff that Splatters.
I think one of the things people need to get straight in their heads wrt P2P is the difference between temporary and permanent roles. Obviously, if node A is always and forever a server, and node B is always and forever a client, and they're totally incapable of switching roles (e.g. neither even has the code to do so) then that's not P2P. Just as obviously, if there's not even a notion of client and server, if every node is necessarily able to perform in either role at any instant, that's about as P2P as you can get. (Note, however, that with respect to an individual transaction there is still one node acting as client/requester and another acting as server/responder).
The real battleground is the area in between, where a node may be a server one moment, or a client the next, changing according to the needs of the network (e.g. nodes entering and leaving). Is that P2P? The P2P purists ("peerier than thou", to use Dr. Shirky's term) would say no. More practical people would say yes, or close enough to yes that it doesn't matter. One trend that more and more people are noticing is that many P2P protocols/applications are developing ideas of "supernodes" or "reflectors" or "defenders" (my candidate for stupidest term yet) that, because of their superior resources, are given additional responsibilities. In other cases, certain functions have been partly or completely centralized within a mostly P2P framework, because nobody could figure out how to make that particular piece - usually a location, searching, or indexing piece - scale within a pure P2P paradigm.
The important thing about P2P is not "oh my god, there's a server, we must eliminate such heresy from our design!" What's important is decentralization and automatic reconfiguration, to avoid bottlenecks and single points of failure. Those are the problems we're trying to solve, remember? If the system is flexible so that work can be redistributed seamlessly from one place to handle either overload or failure, that's "P2P enough for me" even if a picture of the system at any one point in time shows some nodes in server roles and others in client roles. That generally means that each node must be capable of performing the different roles - i.e. the code must be present, the protocols must support it - but whether a given node actually does ever perform a given role doesn't matter.
Slashdot - News for Herds. Stuff that Splatters.
I don't think the author truly understands the issues or the words he is using. He claims that millions of computers will be more reliable than one big computer but what is his definition of reliable? Unless you are talking about non-serilizable or non-consistent clusters of computers, the point is mute. Consider a cluster of databases...are they more reliable? No, if one of the computer in the cluster fail, the entire cluster fails because of the basic unsolvability of group membership. And if he is talking about non-consistent clusters of computers, then how will answers be guaranteed?
Millions of CPU doing a job of a supercomputer...what do you think a supercomputer is? It is thousands of CPU clustered. Only difference is that a supercomputer is pre-packaged. To have million "regular" PC clusters, it simply won't be able to scale for most applications (with notable exception being SETI@HOME), since the cost of routing information quickly overwelms the useful information that is being passed arround.
He also claims that PCs are second class citizens and they need to be servers. But does he have any idea what this will entail? Think of the security issues...think of the privacy issues...think of the performance issue it will bring to the entire internet. Even if the cost of routing scales linearly, that still sucks because number of computers that are being connected is increasing exponentially.
I'm not here to dismiss p2p - I think p2p will have a great future for some applications. But I think the author needs to think hard about his statements - which I think has very little meaning at all. The article seems too much hot air religious fervor about p2p.
I really thought that his point about making PCs first-class citizens of the 'Net was the most important, and one that needs to be driven home as much as possible. I still think that true freedom for web users (who must always rely on corporate connection providers) will derive from widespread, mainstream adoption of something like Freenet. I argued this point in an article I wrote for freshmeat called the World Free Web. I had hoped that we could jump-start that process by integrating Freenet with web browsers, effectively using Freenet as a huge, decentralized backup to the web - on that was out of any entity's control. I'm still working on getting people to work on this idea, so email me if you are interested...
Can your IM do this?
The only solution I can see is to add a layer between TCP and IP. Call it NATCP (or NCP, if you like nested TLAs). The NATCP data would be a chain of DWORD used to map back to a machine on the network. In the simplest implementations, it would be the IP address of the machine on the inner network, or be completely separate from the local IP address. 0.0.0.0 would be reserved for termination.
For those that fear that this takes away from the protection that NAT give the machines on the inner network: that is not the intention of NAT. NAT is used to give many machines access to the internet from a single IP address. Protection is provided by the firewall (which will often be you NAT server).
Note, that while I put this bewteen TCP and IP, it could easily go on top of TCP (although it more logically goes between the two).
For those that say, "why not just switch to IPv6", IPv6 doesn't solve this problem. ISPs will still only dole out a single IP address, because they want you to pay for each machine you connect at home (or at work). In this case, you get a single IP address, and you chain the NATCP address together, so even if you ISP is behind a NAT, their NATCP address gets prepended to yours. You can thus chain NAT networks with ease. This may be a better solution to the IP address shortage and allocation issues than IPv6.
-no broken link
Until P2P comes up with a solution to the NAT problems, it will continue to suck more and more. NATte home networks are going to become more and more commonplace, as handhealds and kid's computers get 802.11b in them and Airports or AirStations are sold to home consumers
-no broken link
1. I am sure there are many terabytes of movies, music and books available that are no longer protected by copyright (they have been liberated) laws. Distribution via a server based method might be better, more reliable etc, however, with no possibility for profit it would not exist.
2. It might be possible to use some sort of P2P system to distribute the traffic from high traffic websites more evenly around the internet. (what i am thinking about is kind of like caching, however each web browser would become a cache for machines around it... don't know if it is possible)
I am sure there are more...
Need a website host? Try out http://WebQualityHost.net
I hate to be the bearer of bad news but PCs will never be first class citizens on the NET until everyone either has Cable, DSL or Fiber Optic running to their house and their PC is reliably connected to the NET. That is why up 'till now PC's have always remained second class citizens, its not the PC's fault. Processing power is not a problem here, it CONNECTIVITY. Imagine if your PC and everyone elses were connected to the NET reliably and 24/7, also each PC had it own unique IP address or someway of identifying it (pray for IP V6). Think of the power, the possibilities, that is where we want to be but until we get the reliable bandwidth and permanent "on-all-the-time" connections, this is nothing more than a pipe dream.
Nathaniel P. Wilkerson
Domain Names for $13
Nathaniel P. Wilkerson
www.haidacarver.com
Enter Napster. Suddenly, everyone has a reason to think of their computer as no different from all the other computers (even if they are going through a centralized server). It becomes clear that there is great utility in being connected, and having access to other machines, both upstream and downstream. Now that the populus has gotten a taste of this, I doubt they'll go back. Napster will be re-implemented as Espra over Freenet, and given the much more generalized architecture, Peer to Peer networks will branch out into all kinds of new spheres of influence. I can't wait to watch it happen!
---
--
If we don't change direction soon, we'll end up where we're going.
Did you post this to the decentralization list? I don't think I've seen it there.
Actually that was Joseph Kennedy, who sold in the summer of 1929, supposedly because a shoeshine boy gave him a stock tip. And the story is probably totally apocryphal - Kennedy himself denied it.
--
--
Ikaruga scoreboard (supports netranking)
Shirkey says "The invention of the image tag, as part of the Mosaic browser (ancestor of Netscape), brought a GUI to the previously text-only Internet in exactly the same way that, a decade earlier, Apple brought a GUI to the previously text-only operating system." History simplified to the point of almost being wrong does us no service. The image tag was neither a GUI nor invented for Mosaic. It came out of CERN where the physicists wanted to put illustrations in their documents. Mosaic was just an early implementation of a browser that could run in existing GUI OS's and that implemented the CERN standards.
A few paragraphs later his theme is "big, sloppy ideas." Yeah, fine, he gets big and sloppy about his ideas of the past, and then parallels that distortion to a present he doesn't begin to define, and this passes for analysis? In a really vague way he may be waving his arms in the right direction, but why are we even trying to listen to someone whose prattling skirts close to the edge of intellectual dishonesty? It's like those old "make millions from the Internet" spams. Sure, you could make millions back in the day, by not by following the advice in those missives. It's because the likes of Shirky have been listened to by too many VCs and editors that the tech economy is so shakey now - false intelligence is more dangerous than ignorance.
"with their freedom lost all virtue lose" - Milton
now that joe fucknuts can do it, it's a revolution
Not to be rude, but I would say that that defines a revolution for almost anything. Once anyone can publish books (printing press) or own a telephone etc it does become a sort of revolution eventually changing society. Of course, plenty of stuff becomes super widely available but doesn't really change our lives, but...
I dunno - for all the the high-brow talk, is there another use for distributed file sharing? Gnutella and Freenet, as far as I can tell, consist of illegally shared MP3s and the same porn that's been passed around Hotline for years. (How many copies of gym.mpg do I need?) Besides being slow, buggy and unstable.
To my mind, if you have text documents you want to share, you put them on a web page. If you want to distribute demo MP3's for your band, a web page is definitely the way to go. If you're a political dissident, it seems to me that sharing from your computer is the last thing you do. You send the files to someone in a free country to make them available -- on a web page or FTP site.
Am I missing something? Of course, everyone here had all sorts of pious explanations for what they were using Napster for. Uh, yeah right.
If there are interesting, non-warez uses for P2P file sharing that are better than server-based methods, please enlighten me! I'm going to bed now, though, so flames and accusations of being a paid RIAA agent will go unnoticed (unless you're Freddy Krueger).
Unsettling MOTD at my ISP.
Unfortunately, the media is associating P2P with Napster and stealing. On the other hand, the little man loves his free music, so he might get freenet! It's a double edged sword, folks!
Douglas Adams
1952-2001 :(
I fail to understand the hype about this "new" peer-to-peer stuff. In 1983/84, I set up a group of Apollo workstations. There was no server. Symbolic links allowed the hard disks on the workstations to act like one big disk, so everything appeared local to everyone. There was no server. That was over 17 years ago. What's "new" about it?
Oh, goody. PCs won't be "second-class citizens" anymore. Hmm. In 1996, the PCs (mostly Pentiums, but including a 486 or two) in my small company were all connected to the net. One of the Win95 machines ran an IRC server. Linux boxes ran FTP and various Java client and server apps and bots. The only tasks reserved for the "big servers" (Suns) were DNS, RealAudio (which I later ran from a Pentium as well), and the main Web site. Everything else was distributed, and mostly to PCs.
So it's not new, and it's not innovative. What's the big deal?
If you look at the internet as our collective brain, with each node representing a neuron (broad metaphor i know but go with me for second on this) then client-server is like the motor cortex - you can map out a direct link between certain neurons and a particular service of your body - the use of your left leg for example. Damage a particular part of your motor cortex, and your leg wont work. Turn off the slashdot server, and nobody will see the site. But things like memory and thought in the brain are more distributed, just like a p2p system - even if you kill off a bunch of the neurons, the system can still function and even relearn what it has lost. Even if you kill off half the gnutella nodes, you will probably retain more than half the info that was accesible on them. Since distributed processes like thought and memory are considered "higher functions" of the brain, I would state that P2P does not suck and in fact it is like a higher function of the internet (when you look at the net as an extension of our brains) and so of course it is only the intelligencia (the frontal cortex) that will find it of much use - so be it! If the rest of the world cant figure it out or doesn't want to, thats fine - more bandwidth for me!
amount Chaucer got paid to write The Canterbury Tales = $0.00
...amount of St. Mathew's Passion which is "borrowed" work from other composers = 1/5
amount Van Gogh received from sales of all but 1 painting = $0.00
copyright fees collected by Bach = $0.00
amount they received in sampling/licensing fees = $0.00
amount (currency converted to USD and adjusted for inflation)
When in doubt, have a man come through a door with a gun in his hand.
You know, you're right, it is stealing. Guess what else, I don't care. I don't say "yesterday I got two songs off of Napster" to my friends; it would probably be more like, "yesterday I stole two songs off of Napster, fuck the RIAA!" Fair Use is bullshit, so what? The corporation is the most disgusting invention in the history of mankind. It allows us to escape blame for our actions and has no compassion for individual humans. I'd never steal from a person, a local store or even a large but not corporately owned company, but a corporation? Every single time I get the chance. I don't think of this in terms of right and wrong because I don't believe that such things are static and unchanging, I simply no longer care about the welfare of corporations. Oh and by the way, calling us dorks doesn't show much in the wittiness department.
"A witty saying proves nothing." - Voltaire