Evolving Swarms with Swarmstreaming

Orasis writes "Applications like Bittorrent have broadly validated swarming technology in the real-world. Now, the inventor of swarming has released a new technology called swarmstreaming that allows smooth progressive playback of content, skipping ahead, and random access without downloading the entire file. It's an HTTP proxy, so browsers, podcasting, and RSS apps should be able to use it transparently. "

Comments

In other news...

[k4_pacific]

In other news, the inventor of swarming was attacked by killer bees.

Re:In other news...

[nofx_3]

Were they africanized bees?

-kaplanfx

Re:In other news...

[Anubis350]

get with the times, its obvious the bees decided that killing him wasnt enough, hired a lawyer, and are now suing. They are claiming that they own the intellectual property associated with swarming and use of the word. The are joined in their lawsuit by wasps, flies, and locusts who have all jointly formed the SIAA (Swarming Insects Association of America).

in a statement issued by the SIAA they call humans who use swarming technology of violating copyright and tarnishing their image as insects by using swarming for illegal activites......

Re:In other news...

[maxbang]

No. They were placebo bees.

Re:In other news...

[G-funk]

I'm cured!

I mean, ow!

Re:In other news...

[dolmen.fr]

Bugs also decided to join the SIAA in the combat against the so-called "swarming technology". They are already preparing to invest software implementations.

Re:In other news...

[talaphid]

There should be an active movement to get the next revolutionary protocol (or this one, if possible) renamed zerg, so it's "file zerging". Blizzard should be on board with this, because they've taken from the Bittorrent thoughtshare, so returning a word as thanks shouldn't be a big deal... and then, years from now, on the news, we'll actually hear, "Today two teenagers were arrested at Minnesota Heights High school for file zerging..." and then my dream of a real life Kent Brockmanism will come true... nouning a verbed noun, and using the word zerg.

Re:In other news...

[dolmen.fr]

Are you saying that swarming is already a buzz word?

Re:In other news...

[dolmen.fr]

Just to correct myself:

• Buzzword in just one word.
• the right link is here.

Re:In other news...

[Anonymous Coward]

He is not the inventor of "swarming". It is an application of the "ant algorithm" and associated techniques developed by Marco Dorigo.

http://iridia.ulb.ac.be/~mdorigo/ACO/ACO.html

Re:In other news...

[erpbridge]

Or were they European? ...the real question, what is the airspeed...

BZZZ!
Ow!

OK, OK, Sor-Ree!

Re:In other news...

[mibus]

Ooh, I bet that's gonna sting... ;)

Re:In other news...

[SirBruce]

What's next? Will he release the dogs with bees in their mouths so when they bark they shoot bees at you??

Bruce

Re:Forget about browsers and RSS

[buro9]

Since when did you want to pause pr0n?

You must be doing it wrong... you are masturbating at the same time right?

Well, the linked site isn't using it...

[datastalker]

...since it's Slashdotted after one comment. :(

Google Cache

Re:Well, the linked site isn't using it...

[EnronHaliburton2004]

...since it's Slashdotted after one comment. :(

Ironically, isn't this the sort of thing that Swarmstreaming is supposed to protect against?

No more slashdotting?

[X0563511]

So, if this works 'transparently' to browsers, ect, does this mean slashdotting a site will be much harder?

Re:No more slashdotting?

[Anonymous Coward]

sure we only need a slashdot-effect-proof proxy server.

Re:No more slashdotting?

[Pxtl]

No. Heaven forbid anyone ever try and improve HTTP - that's blasphemy, even though it seems to be the source of most of the weakness of the internet (can't home serve due to lack of NAT traversal, can't serving large files is suicide because of lack of swarming, no way to differentially update content or inform the user of updates so you have users mashing "refresh" over and over again and redownloading the same goddamn html).

No, http is perfect, that's why nobody ever ever bothers to change it.

But I'm not bitter.

Re: No more slashdotting?

[Alwin Henseler]

"So, if this works 'transparently' to browsers, ect, does this mean slashdotting a site will be much harder?"

Well... if you consider 'swarming' to mean: "every recipient is potentially a distributor as well", then yes. 100 Million downloaders/readers, a little time -> 100 million mirror sites. Slashdotting then becomes a non-issue.

Site getting slow... here ya go, you can use the copy in my browser's cache, if only I knew what URL it has.

Re:No more slashdotting?

[ColaMan]

What the?

can't home serve due to lack of NAT traversal
Isn't that a transport layer issue?

can't serving large files is suicide because of lack of swarming
Isn't that a Phat Pipe issue?

no way to differentially update content or inform the user of updates
Heard of HTTP code 304 - Not Modified? Servers and browsers already know how to use it, and they do it transparently to the user. And maybe you should consider putting a "last modified" timestamp on your web page, to help stop clueless/hopeful people from overriding their browser and downloading something they arleady have.

HTTP has it's faults, but none of the problems you mention can be seriously blamed on HTTP.

Re:No more slashdotting?

[NoMoreNicksLeft]

can't home serve due to lack of NAT traversal
Isn't that a transport layer issue?

Agreed.

can't serving large files is suicide because of lack of swarming
Isn't that a Phat Pipe issue?


Rather elitist. Strange that a network originally meant to be decentralized is now so dependent on seperating it into big pipes and "economy class".


no way to differentially update content or inform the user of updates
Heard of HTTP code 304 - Not Modified? Servers and browsers already know how to use it, and they do it transparently to the user. And maybe you should consider putting a "last modified" timestamp on your web page, to help stop clueless/hopeful people from overriding their browser and downloading something they arleady have.

Well, it may be true that 304 is the first step in one the original poster suggests... but think about this. If the webpage is 1 meg in size, and he's only changing 2k worth of bits here and there, wouldnt it be better to send a difference file? How to implement it might be a challenge, but it seems like it would be a useful tool.

Something like this

[Anonymous Coward]

is going to bring the internet to a grinding halt someday.

Re:Something like this

[jacksonj04]

Someone mod this one up - swarming is all well and good unless people start to use it for everything. When every machine is streaming copies of the internet (OK, bits of it) there are going to be so many packets flying around the switches are just going to fall over.

Re:Something like this

[cryptochrome]

Seriously. All those college servers aren't blocking BitTorrent over the content - they're blocking it over the sheer cost and congestion of all that traffic.

That's the problem with flat-rate unlimited bandwidth for the consumer. If they can use as much as they want, they will use as much as you let them. Like how when you widen road, the traffic situation doesn't improve because more and bigger cars start using it.

Filesharing of video is a good example. Used to be a 30 minute episode would end up encoded with Real Media at low rates, and turn out to be maybe 50MB total. Then people started getting broadband and DivX was invented, and suddenly all the files ballooned to 175MB or even 225MB. Likewise audio - what was once 96kbs mp3s became 320kbs or even losslessly compressed. As transfer rates improve, don't be surprised to see people trading 1GB+ video files in HDTV resolution.

Re:Something like this

[ScrewMaster]

So what? It's up to the people that are maintaining and improving the infrastructure to give us what we want. You might as well say that, gee, back when people only had 64K of RAM they only used 64K of RAM, and when suddenly machines had 64 MB of memory, why, gosh ... they used that too! Huge surprise. Flat rate isn't the problem, it's the big ISPs milking the infrastructure they have to get the last penny from their users before they invest in upgrading. And upgrade it they will: there's just way too much money to be made in the content distribution business to leave things as they stand.

Re:Something like this

[NardofDoom]

The problems: Cable and telecom companies given government-mandated monopolies and laws like this one.

If we're going into a brand new "information" economy, whoever controls the medium over which the information flows controls the economy. That's why, until recently, electricity and telephone were publicly controlled, as well as the roads. If *everyone* is dependent on the Internet, why should we pay private corporations to provide the service? Shouldn't the "Information Superhighway" be like the real superhighways; owned by everyone?

Re:Something like this

[Anonymous Coward]

That's funny, I honestly believe attitudes like your's are going to do the same thing to the whole damn world.

I know change is scary, just try to remember to breathe until you really understand what's going on.

Re:Something like this

[Anonymous Coward]

Are you willing to pay for what you want?

I mean, you could get 1000 Mbit/s to your home, but since you won't pay the price for it, nobody bothers.

Re:Something like this

[ScrewMaster]

Well ... we had a national telephone system for almost a century: it was called "Ma Bell" and was composed of AT&T and 13-odd Regional Bell Operating Companies. It was an artificial monopoly instituted by the Federal Government for the express purpose of providing reasonably-priced phone service to all regardless of location. And it worked. But about twenty five years ago it was deemed that such a monopoly was no longer in the public interest, in that the public was not being given access to the latest technology had to offer. Monopolies are more interested in the status-quo than in offering the best service. So, old AT&T was broken up and the entire phone system was privatized. Whether that was ultimately a good thing (or not) is not an easy question to answer.

Ironically, the reason that a private sector organization like AT&T was created to implement and maintain the phone system was because the Federal Government recognized that it was ill-equipped to perform the same service in an efficient manner. That hasn't changed, I'm afraid, and I'd hate to see what a government owned-and-operated Internet would be like. It's easy to say "owned by everyone" but the reality is that the government is hideously inefficient in everything it does, and the very last thing we want is to have a nationalized communications infrastructure. Better to have a heavily regulated communications industry (as AT&T once was) with high quality-of-service standards (like AT&T used to have.)

Re:Something like this

[ScrewMaster]

Sure ... now. But look at the history of data communications. A few years ago the best I could get was a 56k connection, and some years before that I ran a major multinode BBS where we had top-of-the-line 9600 bps modems. Before that, I had 1200, and before that 300 bps. Now I have a 4 megabit per second connection. Why is that? Because there was a demand (i.e., people with money to spend) and a supply (i.e. companies with the brains to figure out how to give us what we wanted for what we had to spend.) Nothing has changed, really. Our need for speed hasn't even slowed down, so if providers want to keep getting our money they will have to keep finding ways to get us what we want. It's called capitalism in action.

Inventor?

[Anonymous Coward]

Now, the inventor of swarming has released a new technology

Uh, so the killer bees are inventing technology now, and nobody is alarmed? I, for one, welcome our new technology-wielding killer bee overlords.

Re:Inventor?

[carabela]

The idea of killer bees swarming software gives the phrase killer app a whole new dimension.

Re:Inventor?

[Anonymous Coward]

In korea, only old killer bees swarm.

Uhoh

[Anonymous Coward]

This is what his server looks like...

It's been done before.

[sakusha]

I can't quite tell what this app is all about, this is surely a record, ZERO comments and the site's already slashdotted.

But all those features mentioned in the ./ summary are already available in QuickTime Streaming Server.

Re:It's been done before.

[ravenspear]

this is surely a record, ZERO comments and the site's already slashdotted

You must be new here.

Re:It's been done before.

[OblongPlatypus]

Um... they're also available when I play a video directly from my hard drive, so what? The features mentioned are trivial when there's a single data source.

Either you missed the word "swarming" here, or I've missed what exactly the QuickTime Streaming Server does.

Re:It's been done before.

[Refrozen]

mentioned in the ./ summary

The ./ summary? Is that like a new site? DotSlash.org

How is this new?

[ravenspear]

smooth progressive playback of content, skipping ahead, and random access without downloading the entire file

Quicktime has had all that for several years. Apple called it "Instant On". I think both Real and Microsoft already use something similar.

Re:How is this new?

[Snarph]

Quicktime has had all that for several years. Apple called it "Instant On". I think both Real and Microsoft already use something similar.

I may be mistaken, but I doubt those have anything to do with swarming. We're talking about swarming here, ie: the technique used by bittorrent.

Re:How is this new?

[kakos]

Jesus, man. Okay... so you didn't read the article because it is /.ed, but you can at least read the summary! It's streaming using swarming and it provides smooth progressive playback of content. Streaming itself is very old. But streaming while swarming is news.

Re:How is this new?

[mrvis]

That's what it's called when your 1 computer talks to that 1 server on the internet.

The whole bittorrent thing is the new part.

Re:How is this new?

[Agret]

Quicktimes "Instant On" doesn't let you skip to anywhere you want in the file until it has actually got to that part. It downloads it progressively.

Re:How is this new?

[ravenspear]

Agreed but I was referring to the part of the summary that says "a new technology called swarmstreaming that allows smooth progressive playback of content, skipping ahead, and random access without downloading the entire file." The implication seemed to be that this "new technology" was giving us these new capabilities, which I was just pointing out weren't all that new. This is just a new implementation of those that happens to use swarming.

Re:How is this new?

[ravenspear]

That is incorrect. Instant On has to do with streaming, not progressive downloading. The two are different.

Progressive downloading is where you download something like http://www.whatever.com/movie.mov in a web browser and it starts to play as soon as part of it is downloaded. You can then skip to wherever you want once you have downloaded that part (because at that point all you are doing is scrubbing through a movie file stored on your local machine.)

Streaming is where you load something like rtsp://stream.whatever.com/something.mov into a video player and it streams it to you. At no time during that process is anything stored on your local mahcine aside from what you are currently viewing and whatever the client has buffered ahead of that. Instant On instructs the server to skip the stream to another section.

Re:How is this new?

[ravenspear]

Yeah I realize swarming is the new part. It was just that (at least to me) the article summary made it sound like those capabilities were new to streaming.

Re:How is this new?

[Orasis]

Technically, swarmstreaming is more closely related to progressive playback than RTSP style streaming, since it relies on a full file being available at an origin web server.

Re:How is this new?

[Anonymous Coward]

Reading comprehension wasn't your strong suit, huh?

Wow...

[naztafari]

Now you can click in the middle of the media player bar and play the streaming file and have it play from that point with less wait... amazing...

Re:Wow...

[aminorex]

What's amazing is that you can do it for thousands
or millions of consumers without a DOD-sized hosting
budget. I can see this getting very rapid uptake
with porn startups.

Re:Forget about browsers and RSS

[buddhaseviltwin]

Think about what this means for pr0n!

Is it just me, or isn't that the default implication for any new video streaming technology?

Oh wait... You must be new here.

Here's a picture of their webserver

[Anonymous Coward]

here

So...

[Mindwarp]

...will this become SwarmPorning or PornStreaming do you think?

Re:So...

[Anonymous Coward]

I already do plenty of "pornstreaming"...

Re:So...

[Anonymous Coward]

RealDollPlayer?

Re:So...

[mollyhackit]

I choose SwarmPorning... "Verbing nouns weirds the language"

Re:So...

[athakur999]

swr4m?

Re:So...

[Anonymous Coward]

/. points for the C&H reference :D

Re:So...

[krammit]

The idea of "PornStreaming" is just a little too vivid for my tastes. And "SwarmPorning" invokes 80's era gang bang images for me. Sorry. How about we just call it a "Fault Tolerant Honeypot" or something? I'd probably give you something a little better but I can't get past the idea of a porn stream. Someone fetch me my boots.

How does this work?

[koreaman]

No amount of swarming will ever get around the fact that a piece of something has to be in your local system before you can view it. "Skipping ahead" Will skip to a part of the clip that you may not have. This=lag. What's more, usually you cannot download one second of movie in one second of time, unless you have a crazy tricked out connection. This means that if you skip to a part you haven't seen yet, you will have to wait even longer for buffering. This is hardly worth it.

Re:How does this work?

[OblongPlatypus]

What exactly is your definition of "crazy tricked out"? You can easily stream movies in real-time on a common megabit ADSL connection.

(Most movie releases these days are Xvid-encoded to fit on a single CD. Assume 93 minutes of movie and 700 megabyte file size, and it works out to just about exactly 1Mbps.)

Re:How does this work?

[ravenspear]

"Skipping ahead" Will skip to a part of the clip that you may not have. This=lag

The technology to eliminate lag already exists and has been implemented. I have used it myself.

What's more, usually you cannot download one second of movie in one second of time, unless you have a crazy tricked out connection.

What nonsense. Have you ever downloaded a trailer from here? If the trailer starts to play immediately when you start downloading (i.e. the gray progress bar proceeds faster than the location marker), then you are downloading 1 second of movie in a time faster than 1 second. I can assure you that millions of people have a connection fast enough to do this.

This means that if you skip to a part you haven't seen yet, you will have to wait even longer for buffering.

Again, not necessarily. Buffering is when the streaming software requires that you download x amount of content ahead of the time you actually view it to account for inconsistencies in the stream or packet loss. If those can be eliminated, and connections made fast enough, there is no empirical reason why buffering must continue to be utilized.

Re:How does this work?

[Neophytus]

But millions of people don't have large enough upload capacity to support millions of other people streaming at that speed. With many domestic broadbands the ratio can be as bad as 12:1.

And that's before you even deal with people needing to set up port forwarding.

Re:How does this work?

[Pulzar]

The technology to eliminate lag already exists and has been implemented. I have used it myself.

Can you provide any details? I don't understand how you can see something without a lag if you can't download it in real time?

What nonsense. Have you ever downloaded a trailer from here?

Those trailers are reduced to very small resolutions, so that you can watch them right away. They are hardly good enough to replace regular TV or way off from DVD or HD content.

Re:How does this work?

[Pulzar]

Most movie releases these days are Xvid-encoded to fit on a single CD. Assume 93 minutes of movie and 700 megabyte file size, and it works out to just about exactly 1Mbps.

Do you often achieve the maximum download rate, and then sustain it for an hour and a half? If you do, you have a much better connection than most average ADSL users.

Also, if we're talking about swarming, it'll be even tougher to sustain the max download rate while upload clients are dropping in and out of the swarm.

Re:How does this work?

[ravenspear]

I don't understand how you can see something without a lag if you can't download it in real time?

You can. That was the point I was trying to make. There are plenty of connections out there fast enough to download video in real time, provided the data rate is reasonable.

They are hardly good enough to replace regular TV or way off from DVD or HD content.

Well that's obvious, but I wasn't referring to DVD or HD content. Streams at those data rates are basically nonexistent on the net today because very few connections can utilize them. I was referring to the more standard video formats that are used for streaming today, some of which can yield very good quality at a data rate far below what DVD or HD uses.

Re:How does this work?

[coldmist]

The current (before today) tech could do it from one single fat source, ie one of apples server for example, with a scaled down video source.

This can be getting the bits from 100 different sources, achieving the same effect, for potentially larger (height/width, such as HD content) video clips.

Now, how does this "already exist"?

Re:How does this work?

[chris_mahan]

from apple I can do the small and medium like you describe. Large view area doesn't download fast enough. I think full-screen is out of the question. Time Warner residential cable ISP in west san-fernando valley here.

Is there a divx player for firefox as a plugin? (please say yes)

Re:How does this work?

[thirdworld]

There is plenty of video content out there that requires under 1Mbps. I get well over that (on a bad) day on my home cable modem.

Re:How does this work?

[OblongPlatypus]

Yes, I regularly achieve the practical max of my connection, and sustaining it is never a problem. I do realize there are wildly varying levels of quality in the ADSL offerings out there though.

As for swarming making it tougher, have you ever actually used BitTorrent? I have never seen any sort of evidence of that. Although of course that would depend entirely on the quality and number of seeders in your particular torrent, so I guess once again there are people less lucky than myself.

But what I'm trying to say is that "swarmstreaming" movies in real time is perfectly technically possible for me, and I'm pretty average with no "crazy tricked out" connection. The only thing that's stopping me from doing it right now is that the defacto standard is BitTorrent, which requires you to wait for the entire download to finish before watching.

Re:How does this work?

[ravenspear]

Now, how does this "already exist"?

The capability to eliminate lag exists. I didn't say it can currently be applied to all forms of serving video or all data rates. Swarming might help there.

The grandparent post seemed to be taking the position that buffering would be a perpetual necessity, i.e. that as long as there was streaming, there would be lag. This is not necessarily the case. It is true that currently there is no easy way to avoid it for very large video sizes and data rates. However that is just a limitation of connection speeds and processing power.

Re:How does this work?

[Pulzar]

As for swarming making it tougher, have you ever actually used BitTorrent? I have never seen any sort of evidence of that.

I pretty much have something going through torrent at all times. I see wildly varying download rates during a download of a same item. It can go anywhere from 100KB/s to 30KB/s and back up in one hour. And, this is with very active torrents with 100+ clients.

The inactive torrents are actually a lot more stable, because there are only a few seeds and they are not being leetched to death.

I find this to be the norm in the circles that I go in, but I understand that the experiences vary with regions and ISPs. I can't say what the overall "world" average is.

Re:How does this work?

[joshv]

What's more, usually you cannot download one second of movie in one second of time, unless you have a crazy tricked out connection.

1.5 Mbps DSL here. Downloading compressed movies from Starz online movie service I regular have download times that are 2/3rds to 1/2 the playing time of the movie. Granted, these are heavily compressed, but they are better than VHS quality, and I would not call my connection "tricked out"

-josh

Re:How does this work?

[EvilAlien]

"crazy tricked out" = modem with giant spoiler, large R decal, and neon lights on the bottom. Spinner RJ45 plugs are optional for crazy tricked out bling bling connections.

Re:How does this work?

[Oopsz]

Don't forget the european-style Altezza activity lights..

Re:How does this work?

[jCaT]

Assuming you're talking about a 700kbit stream and most users have between 128-384kbps of upload capacity, you're really only talking about 4:1 or 2:1 ratio. While that doesn't mean everybody can just serve each other, it does reduce the load on the server by 25 or 50 percent. Being able to serve twice as many users with a fixed initial cost would definitely pay out in the long run.

Re:How does this work?

[aminorex]

that's what swarming is for: you get it from 12 different people, and voila, your incoming link is saturated.

Re:How does this work?

[aminorex]

> I wasn't referring to DVD

I will. Most of my viewing is MPEG video running
in the vicinity of 700kb/sec. I could run two
computers simultaneously real-time streaming NTSC
resolution video on a single consumer broadband
link. Many cable connections support 2Mb/sec, which
is more than adequate for a single real-time stream
of well-compressed video at DVD quality levels.

Re:How does this work?

[Wesley Felter]

Which means 11 other people are getting nothing. If the aggregate upstream bandwidth of the swarm is less than the aggregate downstream bandwidth, it's impossible to saturate all the downstream links. But swarming is still better than nothing.

Re:How does this work?

[ctr2sprt]

There are other codecs than MPEG2. XVID, for example, can do DVD quality at about half the bitrate. If you knock down the resolution a notch, like from 720xwhatever to 640xwhatever, you can cut the bitrate even further. 1.5Mb/s XVID (including audio) is very, very good on a computer monitor at that resolution. Since I get 3Mb/s on my cable, we could even stand to bump up the resolution to, say, 800x600 (which is what most HDTVs actually are).

You'd still probably want your nice expensive HDTV for stuff where quality really matters, but as far as delivering high-quality video over the Internet the capacity is definitely there. Well, it's there on the user side: if a million people suddenly started downloading 2.6GB files all at once I can imagine a few of the server's routers running shrieking from the data center while on fire.

Re:How does this work?

[42forty-two42]

The amount of time needed to decode a frame of media is non-zero. This will always be the lower bound for a buffer.

Re:How does this work?

[42forty-two42]

mplayer or vlc plugins

Re:How does this work?

[Kjella]

"Skipping ahead" Will skip to a part of the clip that you may not have. This=lag

The technology to eliminate lag already exists and has been implemented. I have used it myself.

Which is of course, plain absurd. If you don't have it, you need to download it. The only way you could not have lag is to have such an insanely fast speed, and an insanely fast source ready and willing to send you that expact piece as fast as all hell, regardless of whether you have anything to send back. In that case I could leech you at max speed all the time by simply claiming that I must have this part NOW.

If you look at BitTorrent, it works by grabbing the parts it can. Unless you have a massive seeder such as Apple (which would make the "swarm" a puny add-on), chances are you'll be downloading at 6%, 17%, 43% and 86% while being stuck waiting for the piece at 1% to arrive.

In short, you want it both ways. You want to have a swarm instead of a fat seed, yet you want it to behave like a fat seed (which is always willing to send you whichever piece you want next, and fast too). It works in a magic fairy world where everyone is playing nice, but in reality it is the tit-for-tat which keeps BT working. And this system can't do the same and be "live".

Kjella

Not pause!

[Anonymous Coward]

Jump ahead! Haven't you ever seen a scene you wanted to skip past? Well, now you can... even before it's done downloading.

Re:Not pause!

[jester22c]

So wait... you can pause bees in mid-air? Or are there bees in the pr0n that we're pausing? Either way this sounds dangerous.

I dunno...

[rackhamh]

It seems to me that the types of media that swarming is commonly used for won't benefit much from being able to skip forward.

I mean, if you're downloading a feature film or TV show, do you really want to watch the middle before the beginning?

Re:I dunno...

[dJOEK]

no, you wanna start seeing it, then people bother you, then later when you have more free time you can watch the rest

Re:I dunno...

[rackhamh]

I was referring to the part in the post that referred to "skipping ahead, and random access without downloading the entire file."

Congratulations on completely ignoring my point.

Re:I dunno...

[Orasis]

Its not just for media. Imagine a 10GB CAD file that you want to grab a 10MB object out of. You could grab just the bytes you need and skip the rest.

Re:I dunno...

[jensen404]

You can host multiple files on a single torrent. If you have a BT client like Azereus, you can specify which files you want.

Re:I dunno...

[Anonymous Coward]

You talkin' bout porn, right ?

Re:I dunno...

[aminorex]

Or a stream of nattering anonymous coward posts. You could just cherry-pick the +5 posts.

Re:I dunno...

[AtomicBomb]

Not really. Skip forward makes sense in many cases.

I am a regular listener of the RTHK radio archive (a Hong Kong government funded radio station). The subscription is free. The audio clip contains the news, government ad (like don't throw rubblish blahblah) and the radio show itself... Not one bothers to cut the junk out. In a typical 2-hr session, 20-25 mins are news and other junk... It is just odd and a waste of time sit still and listen to "news" several months or even years ago...

Re:I dunno...

[dJOEK]

Hold your horses, chester.

If i've already seen the first part, i do not want to download that again.

How to get it

[Kipsaysso]

Does anyone have a Torrent link to download it?

Were we just beta testers?

[DeckardJK]

If so... go back to the drawing board Justin...

Re:Were we just beta testers?

[Orasis]

Heh, thanks. Actually, I think this whole situation highlights just how much current client/server web architecture sucks. It just puts a fire up under our asses to get this technology adopted on a wide scale that much faster :)

Re:Forget about browsers and RSS

[Anonymous Coward]

Not pausing, but skipping the boring parts

i wonder how long

[bill11082]

wonder how long until the RIAA/MPAA uses the DMCA to declare this technology illegal

We have a new record!

[xv4n]

16 mentions to the word "swarm" and it's derivatives in 4 paragraphs!

Re:We have a new record!

[Frogbert]

Incidently, both are powers of two.

It needs work

[AndroidCat]

The 400+ hits on my port 1147 today by that "validated swarming technology" show that it doesn't play well with the rest of the Internet. Poor social skills, needs more effort.

Deeper Link

[ethzer0]

Here's a link to Onion Networks' Swarmstream product. It appears that to use the product, you need to purchase a Swamstream SDK, though.

Re:Deeper Link

[ethzer0]

If I could only express the look of horror on my face as I hit 'submit' without including the tag. *sigh*

The text... Google's cache isn't up to date.

[DeckardJK]

December 13, 2004 Swarmstreaming: Swarming Downloads Evolved I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly .swarm files. This is probably our most exciting advancement since the original invention of swarming. The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!. Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago. The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.

Server slowing down.. article text:

[Folmer]

Swarmstreaming: Swarming Downloads Evolved
I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly .swarm files. This is probably our most exciting advancement since the original invention of swarming.

The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!.

Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix

Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago.

The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.

He links to http://onionnetworks.com/technology/swarming/#swar mstreaming

Re:Server slowing down.. article text:

[Anubis350]

how far we've taken swarming since its humble beginnings five years ago.
you mean bees can now make honey and milk?

/.ed

[ProtoStar]

Perhaps he should implement his own technology so he doesn't get /.ed. Oh the irony of an anti-slashdotting technology getting slashdotted.

Re:/.ed

[Anonymous Coward]

Oh the tedium of the predictable /. replies.

The text (formatted better)

[DeckardJK]

December 13, 2004

Swarmstreaming: Swarming Downloads Evolved

I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly .swarm files. This is probably our most exciting advancement since the original invention of swarming.

The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!.

Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix

Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago.

The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.

Re:The text (formatted better)

[dknj]

http://onionnetworks.com/technology/swarming/#swar mstreaming

This sounds very useful, hopefully we will see these features adopted widely in the future

-dk

Re:The text (formatted better)

[DeckardJK]

http://www.poe-news.com/features.php?feat=31845

A nice link from April 29th on his site... just figured I'd share... picture intensive... but its the "Rumsfeld Fighting Technique"

Re:The text (formatted better)

[Anonymous Coward]

Great attempt at karma whoring twice there, buddy. It appears to be working.

Re:The text (formatted better)

[DeckardJK]

Oh it'll be modded down in a couple minutes anyway... it was my first attempt at a nicely formatted post on /. Gimme a break...

Re:Forget about browsers and RSS

[Televisor]

Yeah, I find stuff like moving the mouse pretty much impossible anyway.

Crichton

[bwy]

Whenever I hear swarm in an IT context, I can't help but think about Crichton's Prey.

Re:Crichton

[Anonymous Coward]

When I hear Crichton in a writing context, I can't help but think Dan Brown seems a bit more talented

Re:Crichton

[Kehvarl]

Whenever I hear "Dan Brown" I think "Bloody customers can't even remember the author or the title of The DaVinci Code." Then I think "glad I got out of that mindless business" then I think "Dan Brown's work really sucks".

After that, I usually go on to something else.

Re:Crichton

[Anonymous Coward]

That book was so lame. Everything was so completely improbable, and the story wasn't very good to balance any of it out. I highly recommend staying away from it.

)/1 Killer Bees

[Roger_Wilco]

The communal mind produces a savage strategy, yet no one could predict that this vicious crossbreed would unravel the secret of steel.

Re:Forget about browsers and RSS

[iamzack]

Since when did you want to pause pr0n?

For the times when you stop and think, is he really peeing on that girl?

Lets not forget the price of entry.

[Bruha]

SwarmStream Development Suite Features

* Object code for the entire suite of SwarmStream(TM) APIs, including WebRAID(TM), DirectCache(TM), Throttling, and THEX.

* Visualization tools to perform live inspections and demonstrations of what SwarmStream is doing during your application run time.

* One full license for WAN Transport(TM) Server (normally $2950), an HTTP server specifically designed provide advanced SwarmStream features such as self-healing downloads and automatic mirror discovery.

* One full day of developer training

* 20 hours of ongoing support

* One year of free upgrades for all of the above software.

* Unlimited right to use and implement SwarmStream technology for testing, prototyping, demonstrations, or creation of reference designs or applications. Production deployment requires an additional Deployment License.

* One-time fee: $25,000

Re:Lets not forget the price of entry.

[Anonymous Coward]

Via his blog:

Swarmcast will be completely free and provides all of the functionality mentioned above since it embeds the full SwarmStream engine. So any user, blogger, or web server administrator that wants to use swarmstreaming doesn't have to pay a dime for it nor do they have to modify their web pages or generate .torrent files in order to swarm-enable their content.

The SwarmStream SDK package that you mention is our complete bundle including a day of training, support, and a whole suite of supporting technologies for application developers that want to add swarming capabilities to their own applications.

Posted by Justin Chapweske at December 13, 2004 04:13 PM

Re:Lets not forget the price of entry.

[Lord_Dweomer]

Yeah, and then lets remember the fact that someway, somehow, an open source solution will arise, and in the mean time, pirates will have this a week before its released.

Re:Lets not forget the price of entry.

[Anonymous Coward]

Yeah, and then lets remember the fact that someway, somehow, an open source solution will arise

It already has. Dijjer is open source, and was developed by the creator of Freenet. It is still alpha, but is developing rapidly.

These guys must be pretty pissed that someone got slashdotted weeks before they did with some software that is entirely free, and does at least as much as what they claim their non-free software does.

Orasis?

[EVuL_C]

Anyone care that Orasis (the story author) = Justin Chapweske?

http://www.advogato.org/person/orasis/

-c

Re:Orasis?

[Mantorp]

good digging, he could have at least made up a new ID for submitting this. What a slacker. No drive at all.

Re:Orasis?

[jonbrewer]

Anyone care that Orasis (the story author) = Justin Chapweske?

Shameless self-promotion of closed-source software, $25k USD for a dev-kit, and Taco fell for it. uugh.

Re:Orasis?

[idlemachine]

Shameless self-promotion [...] and Taco fell for it. uugh.

The question is if this kind of crap is happening as regularly as it appears to be, are the editors "falling" for it or actively encouraging it?

Re:Orasis?

[utexaspunk]

i agree, although one must wonder exactly how successful this shameless self-promotion is when, as far as i can tell so far, every comment is either making a joke, saying how it's nothing new, or dissing it completely? perhaps justin's running with the "there's no such thing as bad publicity" mantra...

The Finale...

[Anonymous Coward]

You can now easily skip ahead to the CUM shot!! :-)

more linkage

[Glog]

Here is some info on the new technology from the guy's company's website: http://onionnetworks.com/products/swarmstream/

On a sidenote, I seriously doubt that he is the very first one to have thought of swarming. Swarming has been around since before 1999 (when he claims he invented it). He *may* be the first one to have applied it to p2p/networking however.

Re:more linkage

[Havokmon]

On a sidenote, I seriously doubt that he is the very first one to have thought of swarming. Swarming has been around since before 1999 (when he claims he invented it). He *may* be the first one to have applied it to p2p/networking however.

I rememeber the CIO of the bank I worked for wanting to spread data around on everyone's HD instead of getting a new RAID for the server. People were getting 4GB HD's, and only using 500MB or so, this was probably back in '97.

It wasn't a bad idea, there just wasn't a good way to implement it (on Win95 desktops even - shudder).

Re:more linkage

[Anonymous Coward]

I seriously doubt that he is the very first one to have thought of swarming. Swarming has been around since before 1999 (when he claims he invented it). He *may* be the first one to have applied it to p2p/networking however.

Not even that. BitTorrent is based on the old Mojo Nation system (another early mutli-source downloading system) and I know that the MN guys gave a demo to the bay area cypherpunks back in late 1999 or early 2000. Since they probably did not re-write the system overnight to create a "swarming" app I am guessing that it has been around the p2p world for a bit longer than this guy claims...

Re:more linkage

[Wesley Felter]

Actually, Mangosoft Medley did exactly what you describe in 1997. And yet today I don't think you can get such technology for love or money.

Swarming + streaming

[Anonymous Coward]

When you download something via BitTorrent, it's downloaded in random order, as pieces become available. While this works, it means you've got a huge file on your hard disk, but it's completely useless because random pieces are utter garbage bytes. For example, unlike with a straight download, you can't start watching a video file that's still being saved to disk.

The only thing swarmstreaming changes is that it tries to download data in order, so you can use it more quickly, like any other conventional stream-oriented protocol (which is basically anything that uses TCP, along with various streaming media protocols). Now, the innovation is putting together streaming media with the power of swarming--imagine being able to feed a live TV feed from a single stream from the "seed". This is basically what multicast promised, but due to infrastructure problems, has yet to deliver.

Now, the devil is in the details. You're going to have problems with a distributed application that tries to deliver the same data to all nodes in the network at once, since you don't get all those nice properties of randomized distribution of different pieces. Some lossiness would definitely be desirable, meaning you don't really want to use it like a Web proxy. Furthermore, it's physically impossible to deliver data around the planet without many tens or hundreds of milliseconds of latency, so it's not good for interactive applications.

It might be a big win for TV-on-the-Web, though. Imagine if just anyone with a couple hundred kbps could serve a worldwide audience... all those Internet radio stations that are begging for donations to pay bandwidth costs could slash their total bandwidth needs, while upgrading service as well.

I'm not sure if this particular product is going to do the trick (swarmstreaming isn't a new idea, after all, and lots of people have been working on it), but anything that gets people thinking about it should help in the long run.

Re:Swarming + streaming

[Orasis]

Yup, this guy hit the nail on the head. We've actually been doing swarming+progressive downloads since 2001. The big change with swarmstreaming is that this stuff scales to huge numbers of files in a single cache and supports out-of-order random access, which turns out to be much harder than progressive playback is. So if you have an application that only needs to read small ranges of bytes in a huge file, you can now use swarmstreaming to do the trick.

Re:Swarming + streaming

[hackstraw]

Torrents (AFAIK) don't download in random order but close enough for a simple explanation. They certainly do not download from beginning to end. One fix might be to dl in dynamic chunks that where the chunks are in order, but within the chunk it may not be in order. By dynamic, I mean that the chunk size could be dynamic according to the speed of the dl.

A kick ass technology would be to put a swarming like technology into a transparent caching web proxy server. Typically this would be installed by the ISP or at the outside edge of a network where the UL and DL speeds are symetric and FAST.

Re:Swarming + streaming

[snookums]

When you download something via BitTorrent, it's downloaded in random order, as pieces become available. While this works, it means you've got a huge file on your hard disk, but it's completely useless because random pieces are utter garbage bytes. For example, unlike with a straight download, you can't start watching a video file that's still being saved to disk.

I would contest this on three grounds

1) If you use an operating system and filesystem which support sparse files then the missing blocks will be zeros, not random garbage, and the file will only be as big as the data downloaded.

2) A media file or archive with random chunks missing is far from useless. So long as the file header is available (see next point), you should be able to preview a media file for content or quality; and view the listing of an indexed archive or extract some of the contained files.

3) Advanced Bittorrent clients such as Azureus can prioritise specific parts of files -- i.e. the beginnings of files where the metadata usually live.

That said, I have to agree that having "I want this bit" built in to the protocol would have to be an advantage. Imagine having this "swarmstreaming" built into an archive manager or media editor. Download the header/metadata, then grab just the archived files or sample clip that you want. Awesome!

Re:Swarming + streaming

[Orasis]

You're in luck: Swarmcast is a transparent caching web proxy server.

Re:Swarming + streaming

[spiko-carpediem]

>You're in luck: Swarmcast is a transparent caching web proxy server.
Take that, Internet

Re:Swarming + streaming

[Hast]

This is basically what multicast promised, but due to infrastructure problems, has yet to deliver.

Even if you have multicast "swarming" techniques are still a big bonus. Combining swarming downloads and multicast allow you to "join" a multicast broadcast at any time and still get the entire file. This would be particularly useful if you use error coding like Swarmcast did as it allows you to begin assembling the beginning of the file independently of what parts you are getting. (That's a simplification naturally.)

It will be interesting to see if this will become popular right of the bat or if it will take a "SwarmTorrent" before the masses takes notice.

related presentation on javalobby

[Anonymous Coward]

There is a link to a presentation about this stuff on javalobby:

http://www.javalobby.com/eps/swarmstream/

Re:Forget about browsers and RSS

[evilmousse]

since? the proper question is UNTIL. ..until they implement slow-motion, that is.

offtopic: (really though, why aren't more digital video players capable of slow motion? seems to me, regardless of format, it could be accomplished with an extra modifying value to set the played duration of a recorded second.)

What about the MPAA

[X0563511]

Have they had anything to say yet?

This would make it possible to watch a rip of some movie, without waiting to download it first. That is the cheif reason (in my impression) that they are not quite as bad as the RIAA just yet.

They may be able to use this to push some new legislation making it even more of a pain in the ass to practice Fair Use in the USA. I really don't care how it affects piracy, either way.

What i care about is getting caught in the crossfire.

Re:Forget about browsers and RSS

[Anonymous Coward]

Hehe :)

Re: Forget about browsers and RSS

[Anonymous Coward]

Not pausing, but skipping the boring parts

You mean skipping the entire pr0n movie?

Anyone notice the time this story was posted

[Anonymous Coward]

24 hour PST - 13:37.

Re:Anyone notice the time this story was posted

[kesuki]

interesting, but not really news.. Posted by CmdrTaco on Monday December 13, @13:37
We all know that articles are posted by crontabs, and there are no real humans posting articles... a few hours of setting up the days crontabs and then off to play with the computer the rest of the day or whatever... so clearly taco thought this article was 1337 enough to be posted at 13:37 PST.

Re:Anyone notice the time this story was posted

[idlemachine]

so clearly taco thought this article was 1337 enough to be posted at 13:37 PST.

I really hope you're mistaken, otherwise I'm clearly hanging out on the wrong site.

Well, it looked fine

[Tjoppen]

Until I tried the simulator running a T1 line at which point it chewed up a good deal of my RAM and 100% CPU. I gave up after a couple of minutes of waiting. Wonder what'd happen in real life in such a situation...
But if you can afford the 1Gbps line and $25k license fee - sure, why not?

It's quite similar to an idea me and another guy has had for a couple of years but never implented due to the time-bandit that is school. Except it wouldn't require static source material(which this does from what I've read).

Wonder how resistant it is to DoS attacks. Get a gang of people submitting false data down the line would certainly cause a lot of trouble before being detected..

Translation: You got to pay to play

[denis-The-menace]

If that is the case, the technology is DOA.
Especially if it's patented.
Nobody will want to touch it.

Re:Translation: You got to pay to play

[BarryNorton]

This is charming!

The site says "I have no animosity towards Bittorrent"... no animosity, huh?

How generous not to begrudge people developing for free!

Re:Translation: You got to pay to play

[aminorex]

Justin essentially invented swarming downloads, but he over-engineered it with FEC, and the first implementation suffered for requiring a JRE download at a time when the JRE was rather slovenly and broadband had much less penetration and computers were rather slower and smaller. Bram picked up the
ball, and wrote a much more practically useful application, which got a lot of fame and uptake, by deftly avoiding all those issues. (I haven't seen Bram around much for at least a year now. Wonder what he's up to.) Anyhow, I think Justin's point (SPECULATION) is that he doesn't mind that Bram stole his thunder with a less technically virtuosic but more market-appropriate product, and he doesn't have a not-invented-here hatred for the protocol design, nor does he think it is badly designed or executed in such a way as to justify animosity. (For example, if it used UDP with no TCP-friendly throttling -- if Bram had foolishly opted to do that, which many people of less foresight have done, given the proportion of IP traffic devoted to BT today, the Internet would no longer be transporting TCP packets with usable reliability -- he would have destroyed the Internet as we know it, which would have very much justified extreme animosity.)

Really, Justin is a cool, clever fellow, and your sneering comment is quite unjustified, except perhaps by the assumption-laden brevity of the expression to which you reacting.

Re:Translation: You got to pay to play

[BarryNorton]

Really, Justin is a cool, clever fellow, and your sneering comment is quite unjustified, except perhaps by the assumption-laden brevity of the expression to which you reacting.

Whatever cool means in this context, he ought to be clever enough to realise that 'getting there first' is only half the story and doesn't imply ownership. I was hardly sneering - if anyone's tone is off it's his; witness also "[n]owadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent". I didn't need a lecture from him or from you - I'm not so ignorant and can make up my own mind about where credit is due and not without being talked down to...

similar

[minus_273]

I actually wored on somehting similar in college. The difference being that i trtied to pay attention to the netowrk structure and chose hosts close to me. I have noticed more than the avalibility of bandwidth, it is the usage of it that makes the big difference in networks like these. In other word the overlay topology should be such that it tries to find the host that is closest and tries to take advantage of existing connections on common links. Unfortunatly, so far, here is so far nothing that compares to multicast on the router level and most systems do not support it.

Re:similar

[cyngus]

I worked on something similar as well. What I always found funny was reading the papers that said they had solved the problem by using hashing techniques to assign IDs and data to nodes. Of course, while it was easy to see that the number of hops was low, the node IDs had little or no relation to "closeness" between nodes in either a latency or bandwidth sense. The great trick that no one has really figured out is how to make the node IDs (a the overlay network) build itself in a way that accounts for resource availability between nodes AND deal with nodes coming and going. Its cool stuff to think about, particularly because it always seems like the idea is just around the corner.

Re:similar

[Anonymous Coward]

I actually wored on somehting similar in college.

That'd only be true if they paid you. If you do it for free, it's considered slutting.

Re:similar

[ganhawk]

"In other word the overlay topology should be such that it tries to find the host that is closest and tries to take advantage of existing connections on common links."

Amen to that! I worked on something similar as well. I tried using small groups of JXTA networks. Each host connects to the nearest ultra peer and form small clusters of nearby hosts.

Unfortunatly I could never spend enough time to get it working and it stagnated. Interesting acedamic execrcise though! You can read more at p2pbridge

Now Diijer sounds promising.

Swarmcast is Free

[Anonymous Coward]

The application itself is free for everyone to use, even as a proxy. The license above is if you want to develop an application that includes SwarmStream.

Is his real name ZIM!

[artoo]

Obligatory Zim quote

TAK: I should have been an invader! I should have been part of the Great Assigning! I didn't HAVE to be stealing this planet from YOU!
ZIM: You're after my robot bee!
TAK: NO!!!

the right tool for the right job

[Anonymous Coward]

Some things are better served swarmed, others are better served whole.

I prefer single-source if the server can serve me as fast as I can download. On a modem, that's just about any non-slashdotted site.

For things like linux distros, a download manager using a system of mirrors is usually good enough.

I use torrents when demand swamps the ftp and http servers.

the concept is appealing BUT...

[__aailob1448]

First of all, where is the proof of concept??? I've been loking that the website and I see no sample that I can download using this "awesome swarmstreaming technology". What the hell is up with that? I launched the simulation and the java thingie is downloading at 7 k/s.....can you say underwhelmed?

Second, there is a lot of boasting , marketspeak and references to patents, business and whatnot. We're far, far away from GPL territory here. At least bittorrent is opensource.

Re:the concept is appealing BUT...

[jpop32]

First of all, where is the proof of concept???

Yup. I must say that the whole thing looks like snake-oil to me. Lotsa big words, talk of 'incredibly complex technologies', ready with a couple of MagicTech(tm) names. And a $25k price tag. And a web site that buckles under 5 minutes of slashdotting.

To many red flags.

Re:Lets not forget the price of entry. A BARGAIN!

[Sai Babu]

If you have the use for it.

Consider internet radio or TeeVee

Streaming the same packets to each IP wanting them gets to be a real mess real fast. The beauty of this system is that so long as a recipients have adequate upload bandwidth to accomodate the stream bandwidth plus some delta (bigger delta will mean lower latency as parallism will increase with fewer stes away from the source) than the 'broadcaster' only needs enough bandwidth to get the stream out to a few people in order to each millions and millions. Don't forget, radio and TeeVee delays of a few seconds or minutes are easily tolerable when the alternative is no program at all.

Imagine a 'fee free' version of this. Anyone could reach as many people as clear channel radio for the expenst of a megabit or two of outbound bandwidth!

If I had programming to deliver and felt it would interest a few hundred thousand people, the on etime $25K would be a drop in the bucket considering what I'd have to pay to reach these same people by traditional radio, TV, or buying enough bandwisth for 100k streams.

Think about it in radio terms. If I'm running a 128kbps MP3 stream and 100k people want it, I need 12.8gbps and the hardware to stuff it. Hell, a 45mpbs and $25K, one time, is a BARGAIN!!!

I predict we will see some serious challanges to big media corps from this and it won't take long. Just watch how fast the PORN guys snap this stuff up!.

Also, imagine running live feeds from public events. A laptop, this application, a WiFi connection, and there's no limit to the number of people who can join in to 'attend' the event.

Suprnova?

[Lord_Dweomer]

I think we might just be witnessing the straw that breaks the camels back in terms of people shifting their viewing from the tv to the computer.

Imagine if instead of having to wait a few hours downloading torrents off of Suprnova, you could simply browse through their catalogue (which I swear is bigger than Blockbuster's, and has music and tv shows), click something you wanted to watch, and BAM, its on.

Welcome to the future of Internet TV. I just hope the law doesn't fuck it up.

Re:Suprnova?

[advocate_one]

that only works as long as several people have got the file and have left themselves seeding it... and it's very easy for it to be strangled by isps giving their customers highly assymetric down up ratios... In order to pull the stuff down at the fair speed required for live playback would require a ridiculous number of people to remain seeding several weeks after the initial upload.

Re:Suprnova?

[burns210]

Would be a killer dedicated app. A bittorrent(well, swarmstream) client, that does a tivo/tv guide menu and presents a nice big video screen... Like a quicktime+tv guide+tivo...

Would be a KILLER app. Being able to download and save, schedule, find 'if you like this you might also like...' shows. Works with tv and movies.

Man, that would be a powerful use of a couple megs of harddrive space.

Re:Suprnova?

[jpop32]

Would be a KILLER app. Being able to download and save, schedule, find 'if you like this you might also like...' shows. Works with tv and movies.

Well, a couple of scripts, BT client, RSS feeds, broadband connection and some disk space and you can roll your own, today. I'm betting someone is already working on this, if not done.

not free

[nedric]

Let's see... click... browse... oh, it's a "product". Move along.

The blog entry compares this to Bittorrent. That's great that they've exceeded BT's abilities, but BT would never have taken off if you had to pay for it. That's the whole point. That's like coming up with a wicked new IP algorithm and trying to sell it. Good luck with that.

Re:not free

[dbacher]

It's targetted at a different market than BitTorrent.

Imagine for example that you are a company distributing a maintenence release of a 40m application.

You seed this on a web server on your US east server, and you have the "swarm" running on US west, EU, Asia Pacific, etc.

Users connect to the proxy, but the proxy can use bandwidth from all of those sites. Assuming most users upgrade during the day, you're probably paying for a lot of bandwidth you're not using, that you could use to distribute the content.

That, I believe, is the target market from reading this. Think about Microsoft, with hundreds of network centers, most of which are empty at any given time. They would need a lot less combined bandwidth if they could distribute a service pack this way.

The issue with Torrent is that Torrent requires client software. This system runs in a proxy on the web server itself, so there is no client side isntallation required.

Re:Fascist republicans.

[Anonymous Coward]

We've seen this phonomenon on Fox News

Video game I don't want to be in

[CrazyJim1]

720: Skate or Die

Skate like a pro and win tournaments. Or else a swarm of evil, intelligent, killer bees will hunt you down.

Swarmstream audio/video presentation available

[otisg]

Here is a good presentation of this technology:
Justin's presentation (Flash). I just watched it earlier today.

Le Duh!

[TiggertheMad]

You can't get it off of Bit Torrent, duh! You have to get it off of a swarm torrent clien....uh, never mind.

Please for the love of God

[Profane MuthaFucka]

Won't someone invent something useful for porn?

Oh wait a second...

Prior Art here

[Anonymous Coward]

I think both Real and Microsoft already use something similar.

Only they call it

buffering.....buffering....buffering...buffering.. . ad infinitum

Swarming Downloads

[Orasis]

Swarming in the context of AI has of course existed for a long long time and I of course did not invent that. What I invented was swarming downloads with Swarmcast, which is the technique for P2P/grid file transfer.

BEWARE - Patents at work!

[BoldAndBusted]

As the original inventor, Onion Networks, Inc. holds a strong intellectual property position in Swarming and Swarmstreaming(TM) through multiple pending patents.

Shield your eyes if you want to make a GPL'd interoperable client (or similar service). This stuff looks like "poison" for those who work within the Free Software community. Let these folks charge their $25K for their SDK and such. All 10 customers will be very happy, I'm sure.

Re:Fascist republicans.

[Anonymous Coward]

Intentional invoking of Godwin's law!

Prevention of Downloading Bogus Files!

[dos4who]

This looks like it could be the next big thing in preventing the download of large bogus files.

Currently, in p2p programs (ala Kazaa, etc), you'd have to download the entire 600 MB file "Lord of the Rings.avi" (or "Busty Nurses. avi".. depending on your cinematic preferences), only to realize that someone has posted a bogus video in it's place.

Swarming the file (ie: "Lord of the Rings.avi"), would allow you to preview various portions of the file to ensure it's integrity... (personal integrity aside) before downloading the entire file to your local PC.

This is going to really pi$$ off the MPAA

Re:Prevention of Downloading Bogus Files!

[Anonymous Coward]

If you trust where you got the .torrent file, then BitTorrent already deals with this. Thanks to SHA-1 bad pieces are rejected (and some clients close the connections where they got the bad piece from). Still, it would be nice to, "watch videos while they are still be downloaded" (on their site, in bold).

Re:Prevention of Downloading Bogus Files!

[216pi]

Or you simply use vlc which perfectly plays not yet completely downloaded movies on my system since years.

Re:Prevention of Downloading Bogus Files!

[Anonymous Coward]

shareaza.com plays partials also

Re:Prevention of Downloading Bogus Files!

[maccam94]

dos4who: "Currently, in p2p programs (ala Kazaa, etc), you'd have to download the entire 600 MB file" uh... NO. Shareaza [www.shareaza.com] can preview almost any file while you are downloading, including movies.

Re:Prevention of Downloading Bogus Files!

[m50d]

Oh for goodness sake, have they never heard of avi preview? Just download about 30mb and run that, or just use mplayer on the partial file, then you can see if it's what you think it is.

So, a Dijjer you must pay for

[Anonymous Coward]

So, this sounds like Dijjer, except you have to pay for it.

Good luck with that!

Re:Please for the love of God - See 11076829

[dos4who]

See http://slashdot.org/comments.pl?sid=132653&cid=110 76829 above...

His mistake...

[Anonymous Coward]

Once or twice a year there is at least one slashdotter who actually RTFA and the poor dude was caught by EVuL_C who decided to read an article for the first time in many years.

Bandwidth

[cbr2702]

Never underestimate the bandwidth of a 747 filled with CD-ROMs.

I would fill it with 300GB drives.

But BT doesn't even allow you to watch first

[cryptochrome]

No, you want to watch from the beginning. However, BitTorrent is not designed for this - it specifically targets the worst represented portions of the file to help make the swarm as diverse as possible, so that as seeds disappear major bottlenecks do not arise. There are implementations like Azureus that can favor the first chunks, but the result is sketchy at best. Specifically delivering a desired chunk has merit anyway - but only for streamable content, and only if it can deliver fast enough for playback.

Also, BT's speed is often bottlenecked by ISPs which cap uploads, therefore penalizing you on the downloading side.

It's right there on the website though - due to the nature of the technology you have to update your playback/reader/displayer app to take advantage of this. BT just works as is. If the BT developers really wanted to they could update the protocol to allow all this, or even more interesting features - like parchive file repair and recovery, which would effectively eliminate the problem of swarms with lost seeds.

Re:But BT doesn't even allow you to watch first

[LiquidCoooled]

I see the problem with ISPs, and I think the one thing they *could* do to assist without increasing the bandwidth costs is to unlock local inter-ISP connections.

For instance, I could communicate with another NTL user just as fast as my cable modem could carry the data, but once I step outside their walled garden, I am restricted to my prescribed bandwidth.

Re:But BT doesn't even allow you to watch first

[Anonymous Coward]

*Intra-ISP

buffering

[ad0gg]

Stop using Real player.

Hierarchical Multimedia Streaming

[Anonymous Coward]

Yes, it's like comparing apples and oranges, but the nodezilla hierarchical streaming is really cool. In theory anyhow, have not tested it.

http://evl.sourceforge.net/rtp_guide.html

Ant Algorithms came first

[verbs_an_action_word]

He isn't the inventor of "swarming". I have followed Marco Dorio and his work for some years. It is an application of the "ant algorithm". http://iridia.ulb.ac.be/~mdorigo/ACO/ACO.html

Re:Ant Algorithms came first

[Orasis]

The concept of "swarming AI" is completely different than "swarming downloads". I used to always fully qualify it as "swarming downloads", but people have just taken to calling it swarming nowadays.

The concept of swarming in AI has indeed been around for years and I think its some great stuff. However, ant optimization algorithms have nothing to do with swarming downloads or P2P content delivery.

Re:Ant Algorithms came first

[verbs_an_action_word]

I though that this could be considered as an optimizing problem. It seemed that part of the problem was to determine an optimal sequence of visits to random nodes from which to retrieve parts of a distributed file. I thought this could be considered as an instance of the traveling salesman problem, to which ant algorithms have been applied.

Another interesting site to view... and listen to

[dolmen.fr]

There is a very interesting online presentation (slides with speech) here : http://www.javalobby.org/eps/swarmstream/ (Flash player required)

Just check the file while downloading !!

[Anonymous Coward]

www.videolan.org

even for divx and the like!

Creater of Swarming

[fugl]

"Now, the inventor of swarming has released a new technology called swarmstreaming that allows smooth progressive playback of content, skipping ahead, and random access without downloading the entire file."

Wow, He's releasing software now, is He?

iceswarm?

[Doc Ruby]

So, can I stick it between my Icecast server and my WinAmp? Do I have to get funky with the SDK to get that going? Since the Icecast server doesn't take anything from the HTTP connection but the initial request (no native advance/return), will I have to get even funkier with the Icecast source code?

Re:iceswarm?

[Orasis]

This is exactly what Brandon Wiley's Alluvium is doing, except that it uses Oggs instead of MP3. Its basically an implementation of the whole "Judo Radio" concept where you download and cache the files ahead of time and just receive a playlist that tells you the order the files should be streamed.

Re:iceswarm?

[Doc Ruby]

Do any of these distributed streaming solutions let me install *only* WinAmp, XMMS or iTunes on the client machine, and hit the distributed server network?

Re:iceswarm?

[Wesley Felter]

That can lead to a tragedy of the commons where no one installs the swarming software, thus there are no servers, thus there is no swarming. I don't know about Alluvium specifically, but it sounds like Swarmcast and Dijjer are HTTP proxies, so you could always hit the origin server.

Re:iceswarm?

[Doc Ruby]

But if it's truly a transparent HTTP proxy, the content providers will each install one to protect their servers. That might make it more of a server network, but it would be much easier to grow, without consumers installing new software. Different from simple P2P, but with complementary benefits.

File Swarming != Ant Algorithms

[Anonymous Coward]

They aren't the same thing, one is downloads, the other is intelligent agents.

Something just doesn't.... feel right

[deimors]

I dunno what it is.

See, there's this one part of my brain going "sweet! this could be great for a whole slew of things, like streaming software installation and tv shows!".

But then there's this little dark corner going "Oy! What about security?!?". If you can't verify the source of the data (like today's p2p's, it's coming from everywhere), and the data is being used as it comes in, how hard would it be to exploit security holes by doctoring up some hostile data to give a certain md5 sum and launching it out into the swarm?

Maybe it's not very possible, but it still seems like a spooky idea.

Could be used to crush some unexpected businesses

[dlantier]

Interesting stuff, this swarming system. The primary thing I see it doing is taking down companies like Intraware that have built an entire business on delivering software without its box. See, Intraware prides itself on being able to accommodate the large software vendors' needs in licensing entitlements, upgrade and patch services, and purchasing download accessibility. It doesn't sound like much of a deal at first but then you realize that these big software vendors have a lot to trace when they need to get critical software to their customers in a reliable way. Perhaps this notion sounded good to a venture capitalist or two back in the day, but the world has advanced and anyone with a little foresight ought to have been able to see that such a service could be built in-house without too much difficulty. Really Intraware is just a very specific sort of outsourcing host. With Bit Torrent technology and now swarming, all the nice features Intraware was focusing on, could be more efficiently replicated via Onion Network's technology.

Re:Forget about browsers and RSS

[aminorex]

I'm not a porn consumer, but I can imagine people wanting to pause while they talk to their boss on the phone, and more to the point, wanting to skip the "dramatic development" scenese.

eDonkey was first with swarming

[burris]

Not to diminish Justin's formidable skillz, but eDonkey 2000 was out and had tens of thousands of users before Swarmcast was ever finished. Furthermore, nobody ever actually used Swarmcast as it never really made it beyond Open Cola's dot.com dog and pony show.

Or Coral cache...

[Ayanami Rei]

and I'm surprised no one has linked to the NYUD Coral cache of the slowed-down website yet... -_-

See also Roland Piquepaille...

[Ayanami Rei]

I'm sure you can think of other recent examples.

It sounds like...

[Ayanami Rei]

bittorrent could use an API wherein you can suggest it to fetch certain pieces, or determine whether pieces have been verified yet. That's half the battle (aside from figuring out how to effectively USE the API)

bittorent must die

[Anonymous Coward]

Why of why do I want to give someone else my bandwidth. First thing I did was block the bittorent directory server ports.

Take note - Swarmcasting isn't even out yet.

[Anonymous Coward]

I'm sure you guys followed your own advice and Read the F'n Article, because the link to Swarmcast 3.0 is down, and apparently, "Will be up again soon."

Of course, I hate vaporware, and I hate it when companies sit on a project, do nothing with it, and only want to get back into the game once thier forgotten idea becomes mainstream ...

I.e., BitTorrent does what Swarmstreaming envisioned.

And this announcement of "swarmCASTING" isn't proven yet.

There is a software development kit available, but no actual software for neither the client nor server that implements anything beyond - or even up to par - with what BitTorrent does currently.

Re:Lets not forget the price of entry. A BARGAIN!

[adpowers]

There is a much better way to send out video or radio broadcasts over the internet en masse. It already exists and I have used it (albeit, on a local network). The answer is multicast. We just need to get the internet backbone companies to put multicast support on their routers, and we'll be all set. I believe Sprint's network already supports it, as do many others, I'm sure.

Andrew

Distributed Streaming

[pixelcort]

While the user can't skip ahead, they can efficiantly broadcast to an infinite number of other users.

PeerCast

There are other distributed streaming solutions out there, but this one seems to work the best for me.

Issues with "StreamTorrent" of live broadcast

[Krellan]

I've often thought about how a "StreamTorrent" protocol would work. This technique goes a long way towards solving the problem, but still has a way to go.

What of live radio/TV?

An "ordered BitTorrent" design, as described in this paper, could work if it's a static file being streamed. People watching late in the clip would send data they've already seen to newcomers who are watching early in the clip. However, this would not work for live content. There would be no "early" or "late" listeners, as everybody would be receiving content at the same point in time!

Also, there is another concern. If content is being broadcast live, it would most likely originate from a single source. It will become desirable to get as close as possible to the source, to get data that has passed through as few hands as possible, for reasons of timeliness and reliability. Getting and holding onto a low-generation connection would be desirable. BitTorrent's ranking/evaluation system would need to be extended in order to sort out who gets the scarce low-generation connections.

There are other issues, but these two come to the forefront. Perhaps long-term buffering (TiVo-style) could be used to allieviate some of these concerns. Listeners of live content, if their connection is judged inadequate to receive a live stream (or they just didn't get in early enough), would receive a "tape-delayed" stream from other listeners. Clients would be required to save the past hour or so of live content, so that an appropriate delay could be assigned to each new listener, giving the data a chance to percolate throughout the system. Content would eventually fan out throughout the network, going from low-generation to high-generation connections.

It's a toss-up: receiving live data, as direct as posssible from the source, is the best for timely content (and less chance of getting a degraded connection caused by somebody else refusing to send you data), but receiving time-buffered content is best for overall reliability (giving you a long time to retry and successfully re-download the data from different sources before your playback buffer starts to drop out).

Surprisingly, the large streaming vendors (RealPlayer, Windows Media Player, etc.) are surprisingly stuck in the mud: their recent innovations in playback protocols are slow or nonexistent. For the most part, they're still using a single basic TCP or UDP connection! A heroic third-party plugin will have to be the answer. I look forward to seeing "StreamTorrent" show up in RealAudio 20.0 or Windows Media Player 30.0 or so :)

To check out the current (sad) state of streaming online radio, check my signature below, and try to listen to Dr. Demento on Sunday nights when the servers overload.... :(

Re:Issues with "StreamTorrent" of live broadcast

[m50d]

I think the long term buffering thing is the way to go. Have everyone limited to 10 outgoing connections, or less by upstream bandwidth. First ten people to click the link get direct connections, next 100 people get referred to those 10, next 1000 get those 100. Maybe have everyone connect to three others, but two of them only send rsync-like checksums unless the connection gets dropped, at which point you switch to one of the people you were getting checksums from and add a new checksum host. In fact this could be done with actual rsync on each block. You'd need a fairly big buffer so you don't have a break when you lose a connection, but you're needing a big buffer anyway, so it ought to work.

Fancy having a go at implementing it? I do Python, C, and bad Java.

amazing

[_ph1ux_]

Thats pretty amazing, actually I have been thinking of something for a time wondering if it could ever happen;

A true virtual machine whereby all content data and processing is done "packetized" and never sits in one location.

You "store" a file on the system and it keeps it moving around in a swarm, and is accessed by streaming the information - but its never downloaded.

Additionally all the processing would by done on shared CPU time slicing..

what about p2p radio

[Venexiano]

p2pradio.sf.net accourding to the site:

P2P-Radio is a program that can distribute audio and video streams (MP3/Ogg Vorbis/NSV) over the Internet in a peer-to-peer manner. It's possible to create your own internet radio station with P2P-Radio!

P2P-Radio is programmed in Java and runs on Windows, Mac OS X, Linux and many other systems.

Cool application

[karji]

People in countries with good television, such as Norway, could use it to stream live television to the rest of the world.

Re:Lets not forget the price of entry. A BARGAIN!

[Hast]

First off multicast is not in wide use, and it most likely will never be for IP4. With IP6 we have a bit of a head start and perhaps can use that to ensure that it is standard when new products are sold. The problem is that most network equipment doesn't support multicast and thus it is not useable IRL.

Besides, multicast doesn't really solve all your problems. Eg it ownly works in the basic way if all users begin their download at the same time. For other situations it is not as useful. To get around it you'd, again, have to use technology like SwarmCasting (see their old papers for how to apply it on multicast systems).

Freenet

[JackJudge]

Sounds very very similar to Freenet (http://freenet.sourceforge.net/) but without the anonymity and inherent problems of that system.
But all the features the author describes are already in Freenet.

Re:Forget about browsers and RSS

[Kjella]

[slashdot stereotype]
When your mom comes down to the basement?
[/slashdot stereotype]

Re:Forget about browsers and RSS

[Anonymous Coward]

Why do you think the Internet evolved at the speed it did back in the days? Wait for it... wait for it...

Re:Forget about browsers and RSS

[dr. greenthumb]

In order to have slow motion you need alot more frames. Most clips have no more than 25 fps, which will just result in a slide-show bein played any slower ...

Re:Forget about browsers and RSS

[evilmousse]

that's fine with me!

we're talking pr0n here, remember?

ANYthing where there's just too much motion per instant that you want to see in detail. i'd want the same slow-slideshow to try to figure out some of the fliptricks from skating videos.

Re:Lets not forget the price of entry. A BARGAIN!

[adpowers]

I know it is not in wide use, but neither is IPv6. I guess you could say we'll all be using multicast over IPv6 networks to play Duke Nukem Forever on our machine with the WinFS.

The post I was replying to mentioned Internet TV and radio as a big use of this SwarmCasting, but it would be better to just use multicast. This would work wonders for things like Steve Jobs keynotes because they have thousands of users. Imagine a megabit stream video encoded in H.264 being multicasted over the the internet. I think multicast could be used more than it might seem. You want the latest Steam or World of Warcraft update? Turn on your client, it starts grabbing the file in the middle at high speed, continues to grab the file as the stream ends and starts over from the beginning, and then pieces it together. The WoW servers would just need to multicast the update in a loop for a few weeks, and then they could move to a unicast system if needed. Multicast would be ideal but it is a huge obstacle to overcome (getting all the routers and such to support it).

One more thing, are you sure most network equipment doesn't support it? I used it over, I believe, a consumer access point at a SeattleWireless meeting once (it was a local stream, just to test). Or did you mean big backbone routers? I imagine core routers would support it, or could be made to support it with a software update. I imagine it is all a matter of configuring the necessary settings, but I don't know.

Andrew