Slashdot Mirror
Evolving Swarms with Swarmstreaming
Orasis writes "Applications like Bittorrent have broadly validated swarming technology in the real-world. Now, the inventor of swarming has released a new technology called swarmstreaming that allows smooth progressive playback of content, skipping ahead, and random access without downloading the entire file. It's an HTTP proxy, so browsers, podcasting, and RSS apps should be able to use it transparently. "
In other news, the inventor of swarming was attacked by killer bees.
Unknown host pong.
Since when did you want to pause pr0n?
You must be doing it wrong... you are masturbating at the same time right?
...since it's Slashdotted after one comment. :(
Google Cache
Find out about the Lexus Rx400h Hybrid!
So, if this works 'transparently' to browsers, ect, does this mean slashdotting a site will be much harder?
For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
Now, the inventor of swarming has released a new technology
Uh, so the killer bees are inventing technology now, and nobody is alarmed? I, for one, welcome our new technology-wielding killer bee overlords.
This is what his server looks like...
Think about what this means for pr0n!
Is it just me, or isn't that the default implication for any new video streaming technology?
Oh wait... You must be new here.
...will this become SwarmPorning or PornStreaming do you think?
The gift of death metal does not smile on the good looking.
No amount of swarming will ever get around the fact that a piece of something has to be in your local system before you can view it. "Skipping ahead" Will skip to a part of the clip that you may not have. This=lag. What's more, usually you cannot download one second of movie in one second of time, unless you have a crazy tricked out connection. This means that if you skip to a part you haven't seen yet, you will have to wait even longer for buffering. This is hardly worth it.
Le français vous intéresse?
Does anyone have a Torrent link to download it?
This is another way of starting a sig with this and ending it with that.
If so... go back to the drawing board Justin...
wonder how long until the RIAA/MPAA uses the DMCA to declare this technology illegal
DANGER! 10,000 Ohms
no, you wanna start seeing it, then people bother you, then later when you have more free time you can watch the rest
Exercise caution when modding this message up: the author acts like a jerk when his karma is excellent.
Um... they're also available when I play a video directly from my hard drive, so what? The features mentioned are trivial when there's a single data source.
Either you missed the word "swarming" here, or I've missed what exactly the QuickTime Streaming Server does.
-- If no truths are spoken then no lies can hide --
So wait... you can pause bees in mid-air? Or are there bees in the pr0n that we're pausing? Either way this sounds dangerous.
16 mentions to the word "swarm" and it's derivatives in 4 paragraphs!
Here's a link to Onion Networks' Swarmstream product. It appears that to use the product, you need to purchase a Swamstream SDK, though.
Swarmstreaming: Swarming Downloads Evolved .swarm files. This is probably our most exciting advancement since the original invention of swarming.
r mstreaming
I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly
The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!.
Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix
Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago.
The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.
He links to http://onionnetworks.com/technology/swarming/#swa
December 13, 2004
.swarm files. This is probably our most exciting advancement since the original invention of swarming.
Swarmstreaming: Swarming Downloads Evolved
I'm proud to finally unveil swarmstreaming our third generation of swarming algorithms that are designed for the fastest downloads of web content and multimedia without any special server software or silly
The technology improves swarming by ensuring that the bytes that the user wants next are scheduled to be received next. So if they're playing back a video file, the bytes from the front of the file will be received first. If the user (or application) skips forward to the middle of the file, the bytes at the middle of the file will be prioritized. Thus, unlike first generation swarming systems like Swarmcast or Bittorrent, you don't have to wait for the entire file to download to do something useful with it!.
Under the covers it is almost unimaginably more complicated than this because it also provides Self-Healing Downloads, implements a full-blown, scalable, Web Proxy Cache, and actively works to ensure that the video playback never studders or buffers by constantly monitoring and adapting to changing network conditions. For a raw feature dump, check out the SwarmStream SDK Feature Matrix
Nowadays, because of its immense popularity, most people have only heard of swarming because of Bittorrent. I have no animosity towards Bittorrent because it has done more than any application to prove the value of swarming to the general public. But if people are impressed by Bittorrent, they're going to be absolutely blow away by swarmstreaming and how far we've taken swarming since its humble beginnings five years ago.
The best source of information right now on swarmstreaming is Onion Networks SwarmStream SDK, so check it out and let me know what you think.
Whenever I hear swarm in an IT context, I can't help but think about Crichton's Prey.
Quicktimes "Instant On" doesn't let you skip to anywhere you want in the file until it has actually got to that part. It downloads it progressively.
Have you metaroderated recently?
SwarmStream Development Suite Features
* Object code for the entire suite of SwarmStream(TM) APIs, including WebRAID(TM), DirectCache(TM), Throttling, and THEX.
* Visualization tools to perform live inspections and demonstrations of what SwarmStream is doing during your application run time.
* One full license for WAN Transport(TM) Server (normally $2950), an HTTP server specifically designed provide advanced SwarmStream features such as self-healing downloads and automatic mirror discovery.
* One full day of developer training
* 20 hours of ongoing support
* One year of free upgrades for all of the above software.
* Unlimited right to use and implement SwarmStream technology for testing, prototyping, demonstrations, or creation of reference designs or applications. Production deployment requires an additional Deployment License.
* One-time fee: $25,000
Anyone care that Orasis (the story author) = Justin Chapweske?
http://www.advogato.org/person/orasis/
-c
That is incorrect. Instant On has to do with streaming, not progressive downloading. The two are different.
Progressive downloading is where you download something like http://www.whatever.com/movie.mov in a web browser and it starts to play as soon as part of it is downloaded. You can then skip to wherever you want once you have downloaded that part (because at that point all you are doing is scrubbing through a movie file stored on your local machine.)
Streaming is where you load something like rtsp://stream.whatever.com/something.mov into a video player and it streams it to you. At no time during that process is anything stored on your local mahcine aside from what you are currently viewing and whatever the client has buffered ahead of that. Instant On instructs the server to skip the stream to another section.
Here is some info on the new technology from the guy's company's website: http://onionnetworks.com/products/swarmstream/
On a sidenote, I seriously doubt that he is the very first one to have thought of swarming. Swarming has been around since before 1999 (when he claims he invented it). He *may* be the first one to have applied it to p2p/networking however.
When you download something via BitTorrent, it's downloaded in random order, as pieces become available. While this works, it means you've got a huge file on your hard disk, but it's completely useless because random pieces are utter garbage bytes. For example, unlike with a straight download, you can't start watching a video file that's still being saved to disk.
The only thing swarmstreaming changes is that it tries to download data in order, so you can use it more quickly, like any other conventional stream-oriented protocol (which is basically anything that uses TCP, along with various streaming media protocols). Now, the innovation is putting together streaming media with the power of swarming--imagine being able to feed a live TV feed from a single stream from the "seed". This is basically what multicast promised, but due to infrastructure problems, has yet to deliver.
Now, the devil is in the details. You're going to have problems with a distributed application that tries to deliver the same data to all nodes in the network at once, since you don't get all those nice properties of randomized distribution of different pieces. Some lossiness would definitely be desirable, meaning you don't really want to use it like a Web proxy. Furthermore, it's physically impossible to deliver data around the planet without many tens or hundreds of milliseconds of latency, so it's not good for interactive applications.
It might be a big win for TV-on-the-Web, though. Imagine if just anyone with a couple hundred kbps could serve a worldwide audience... all those Internet radio stations that are begging for donations to pay bandwidth costs could slash their total bandwidth needs, while upgrading service as well.
I'm not sure if this particular product is going to do the trick (swarmstreaming isn't a new idea, after all, and lots of people have been working on it), but anything that gets people thinking about it should help in the long run.
There is a link to a presentation about this stuff on javalobby:
http://www.javalobby.com/eps/swarmstream/
24 hour PST - 13:37.
The application itself is free for everyone to use, even as a proxy. The license above is if you want to develop an application that includes SwarmStream.
First of all, where is the proof of concept??? I've been loking that the website and I see no sample that I can download using this "awesome swarmstreaming technology". What the hell is up with that? I launched the simulation and the java thingie is downloading at 7 k/s.....can you say underwhelmed?
Second, there is a lot of boasting , marketspeak and references to patents, business and whatnot. We're far, far away from GPL territory here. At least bittorrent is opensource.
Imagine if instead of having to wait a few hours downloading torrents off of Suprnova, you could simply browse through their catalogue (which I swear is bigger than Blockbuster's, and has music and tv shows), click something you wanted to watch, and BAM, its on.
Welcome to the future of Internet TV. I just hope the law doesn't fuck it up.
Buy Steampunk Clothing Online!
Its not just for media. Imagine a 10GB CAD file that you want to grab a 10MB object out of. You could grab just the bytes you need and skip the rest.
It's targetted at a different market than BitTorrent.
Imagine for example that you are a company distributing a maintenence release of a 40m application.
You seed this on a web server on your US east server, and you have the "swarm" running on US west, EU, Asia Pacific, etc.
Users connect to the proxy, but the proxy can use bandwidth from all of those sites. Assuming most users upgrade during the day, you're probably paying for a lot of bandwidth you're not using, that you could use to distribute the content.
That, I believe, is the target market from reading this. Think about Microsoft, with hundreds of network centers, most of which are empty at any given time. They would need a lot less combined bandwidth if they could distribute a service pack this way.
The issue with Torrent is that Torrent requires client software. This system runs in a proxy on the web server itself, so there is no client side isntallation required.
If your code is acting bloated, and is running rather slow, it's likely and predicted that some loops you will unroll.
This looks like it could be the next big thing in preventing the download of large bogus files.
Currently, in p2p programs (ala Kazaa, etc), you'd have to download the entire 600 MB file "Lord of the Rings.avi" (or "Busty Nurses. avi".. depending on your cinematic preferences), only to realize that someone has posted a bogus video in it's place.
Swarming the file (ie: "Lord of the Rings.avi"), would allow you to preview various portions of the file to ensure it's integrity... (personal integrity aside) before downloading the entire file to your local PC.
This is going to really pi$$ off the MPAA
"Yes, I have a Disaster Recovery Plan. It's called my Resume"
No, you want to watch from the beginning. However, BitTorrent is not designed for this - it specifically targets the worst represented portions of the file to help make the swarm as diverse as possible, so that as seeds disappear major bottlenecks do not arise. There are implementations like Azureus that can favor the first chunks, but the result is sketchy at best. Specifically delivering a desired chunk has merit anyway - but only for streamable content, and only if it can deliver fast enough for playback.
Also, BT's speed is often bottlenecked by ISPs which cap uploads, therefore penalizing you on the downloading side.
It's right there on the website though - due to the nature of the technology you have to update your playback/reader/displayer app to take advantage of this. BT just works as is. If the BT developers really wanted to they could update the protocol to allow all this, or even more interesting features - like parchive file repair and recovery, which would effectively eliminate the problem of swarms with lost seeds.
---If you can't trust a nerd, who can you trust?
So what? It's up to the people that are maintaining and improving the infrastructure to give us what we want. You might as well say that, gee, back when people only had 64K of RAM they only used 64K of RAM, and when suddenly machines had 64 MB of memory, why, gosh ... they used that too! Huge surprise. Flat rate isn't the problem, it's the big ISPs milking the infrastructure they have to get the last penny from their users before they invest in upgrading. And upgrade it they will: there's just way too much money to be made in the content distribution business to leave things as they stand.
The higher the technology, the sharper that two-edged sword.
This is exactly what Brandon Wiley's Alluvium is doing, except that it uses Oggs instead of MP3. Its basically an implementation of the whole "Judo Radio" concept where you download and cache the files ahead of time and just receive a playlist that tells you the order the files should be streamed.
Not really. Skip forward makes sense in many cases.
I am a regular listener of the RTHK radio archive (a Hong Kong government funded radio station). The subscription is free. The audio clip contains the news, government ad (like don't throw rubblish blahblah) and the radio show itself... Not one bothers to cut the junk out. In a typical 2-hr session, 20-25 mins are news and other junk... It is just odd and a waste of time sit still and listen to "news" several months or even years ago...
Well ... we had a national telephone system for almost a century: it was called "Ma Bell" and was composed of AT&T and 13-odd Regional Bell Operating Companies. It was an artificial monopoly instituted by the Federal Government for the express purpose of providing reasonably-priced phone service to all regardless of location. And it worked. But about twenty five years ago it was deemed that such a monopoly was no longer in the public interest, in that the public was not being given access to the latest technology had to offer. Monopolies are more interested in the status-quo than in offering the best service. So, old AT&T was broken up and the entire phone system was privatized. Whether that was ultimately a good thing (or not) is not an easy question to answer.
Ironically, the reason that a private sector organization like AT&T was created to implement and maintain the phone system was because the Federal Government recognized that it was ill-equipped to perform the same service in an efficient manner. That hasn't changed, I'm afraid, and I'd hate to see what a government owned-and-operated Internet would be like. It's easy to say "owned by everyone" but the reality is that the government is hideously inefficient in everything it does, and the very last thing we want is to have a nationalized communications infrastructure. Better to have a heavily regulated communications industry (as AT&T once was) with high quality-of-service standards (like AT&T used to have.)
The higher the technology, the sharper that two-edged sword.