Slashdot Mirror

I was once in a startup trying to deliver streaming video to tablets/laptops at sporting events via 802.11 wireless networks. Within 30-50 feet of an access point, it worked great, but any further than that and packet loss was too high to decode the video stream. Before our investors pulled the plug, we looked into technology from a company called Digital Fountain. They use a type of forward error correction called raptor codes to make data transmission high resilient to packet loss on unreliable networks. It's a commercial product, but I assume the OP's employer can afford it if they're transmitting "mission critical" files for the government.

Agreed, but while Reed Solomon is well suited for small scratches on a CD or small losses of streamed data, it is rather inefficient in case of massive losses that can destroy a whole (contiguous) block of the data, as it operates on small blocks, and so its range of effect is limited. Fortunately there exist very robust FEC codes that can protect larger blocks. With Raptor or LDPC codes, one can protect very large blocks of data with redundant codes that yield a high probability of recovery given an overall loss rate.

In the case of critical data storage, I'd advocate in favor of large block codes such as Raptor or LDPC with a redundancy of at least 100% (meaning that one can recover the whole data is half of it is destroyed). Note that this need not be done on the physical layer, rather one could FEC-encode files with varying levels of redundancy.

Note that data recovery on damaged physical storage is no different than on unreliable transmission channels such as Wifi, 3G or IP multicast. For information, Raptor codes have been chosen for the DVB-H standard as the preferred FEC scheme for IP datacasting (based on the FLUTE protocol), and LDPC is used in DVB-S2.

For those interested by LDPC, there exists an excellent LGPL'd library by the french INRIA here (info) and here (download), and best of all it's a patent-free technology. As for Raptor, it is unfortunately proprietary and patented by Digital Fountain, but you can however find a lot of enlightening info on their Web site (worth a read if you're interested in FEC technologies).

Once you have a bi-directional UDP packet exchange going, (it's not a connection like TCP) but it is in some sense an unreliable connection. You can then route TCP over it (grab packets from /dev/tap or /dev/tun) , or use a user space TCP stack connection or use something like my ECIP (http://www.ecip.com/ [ecip.com]) over it.

In "some sense an unreliable connection"? In every sense it's an unreliable "connection". There is 1) no connection 2) no reliability. What more do you need?

ECIP sounds very much like what Digital Fountain is doing http://www.digitalfountain.com/entry.asp?PageID=11 9

>Kind of. There's tricks you can do, for example carousel,
> where you continously send the same file out again and
> again. So people can start listening at any point, receive
> to the end of the file in the current sending, then listen
> for the first half when it's broadcast again.

What? So I should watch the last half of a show to see the ending and THEN watch the first half of it? That is completely pointless.

Not quite. I've worked with some software from Digital Fountain. Pretty neat stuff. Think about it like this. Take a two hour movie, break it into about forty pieces. Each of those pieces is going to be a multicast group, which is constantly running. Each of those pieces contain data from all portions of the movie, but in slightly different degrees. So, piece #1 would be about 70% from the first three minutes, 15% from the next three minutes, 5% from the next ten minutes, 5% from the next 20 minutes, and 5% from the rest of the movie. Piece #35 might be something more like 70% from the last 30 minutes, 20% from the last hour, and 10% from the first hour. The algorithm to actually split it up is quite a bit more complicated, but that's the general idea. Now, when you start up a movie, you wait about 10 minutes for the buffer to fill. Then it starts playing, from the beginning, and keeps on downloading in the background, filling in the areas you haven't got to yet. In the end, you're going to see the entire film, with just a 10 minute buffer to wait through at the beginning (and it'll probably be filled with advertising or previews of other films, if it's a commercial venture), and it's all multicasted. The hosting company is basically spilling out bandwidth for a single copy of the movie (plus some overhead) constantly, which can then get to any number of users simultaneously. It's very cool technology, and worked extremely well three years ago when I was playing around with it. I can only imagine they've improved on it since then.

Why do you need to guess what it's about when it's all there in the paper linked to by the article? I've skimmed it, gotten the gist of it, and I think their technique is quite clever. And the paper seems to give full details, so anyone can implement it.

Basically, similar coding schemes make scheduling of data in a swarm easier (so there's no choking/unchoking a la BitTorrent, data just flows) and minimize the risk of a file piece being owned by only one peer (if he leaves, downloading is over). These encoding schemes, through linear combinations of pieces using XOR, combat this (I'm generalizing here). The most attractive, I think, are Rateless and Raptor codes, which have similar performance. (Incidentally, the former was developed by Petar Maymounkov, who was actually one of the inventors of Kademlia.)

Anyway, a few months ago I read the Rateless paper, and thought "Gee, I should code this and release it under the GPL... It would be great for P2P apps!" But soon after I finished its implementation, I discovered that all the ideas authored in the Rateless paper were actually covered by patents of Digital Fountain, meaning that Petar's company, Rateless, had to develop a different, proprietary coding mechanism that is outside the patents of DF, and I can't release my code!

So, getting back to my original point, the paper says, "Network coding can be seen as an extension or generalization of the Digital Fountain approach since both the server and the end-system nodes perform information encoding." Meaning that it might not be covered by DF's patents, and thus should be welcomed by the P2P community, and not immediately disregarded blindly by prejudice. I mean, if it's a 20% improvement, why not give it a chance, huh?

- shadowmatter

The guys ofer at Digital Foutnain have had something that does this for a year or two now. And thiers works over satelite too. Wonder if they will have any IP issues. (Intelectual Property, no pun intended).

What they are really going to have problems with is that some rateless error codes are copyrighted/patened. And the company that did this have already made products (for a year or two now). So long as their code does not violate them they are fine. But if they do, I'm not sure they can GPL the code.

I don't think any commercial broadband wired services would be viable if everyone used all their available bandwidth all the time. For example in the past year or so most cable companies have started putting download caps into effect, for very good reason: you cannot sell bandwidth that costs you, e.g. $600 a month for a T1 to consumers for $30 a month. Never mind that the coax they use cannot supply that much bandwidth to more than a few folks per neighborhood.

A more realistic TCP/IP-by-satellite involves intermittent (on-the-go) usage or more efficient multicast broadcasts. No, it's not a T1-type tarrifed service anymore!

dominik@schnitzer.at, mozparty-at-subscribe@relax.ath.cx, dominik@schnitzer.at, david_markvica@web.de, johannes_richter@gmx.net, kairo@kairo.at, rossi@chello.at, markush@world-direct.com, cbiesinger@web.de, jenskager@gmx.net, jo-at-mt@gmx.net, johann.petrak@gmx.at, dviper01@gmx.net, simon@simonschwaighofer.net, dreckskerl@glump.at, wt-lists@trexler.at, dusty@strike.wu-wien.ac.at, kasparhauserjr@hotmail.com, b.schallar@gmx.net, mutato@libero.it, phil@goli.at, diddalick@gmx.net, studio@paw8.com, croco@utanet.at, petru@paler.net, jlemmerer@node.at, bigkub@time2change.at, patrick@seher-it.at, ronald@hartwig.at, mozilla_party@webterminate.com, stefan@kleinhans.it, horst.jens@gmx.at, jjan@gibts.net, mjahn@agency.at, gpoul@gnu.org, green@eggs.ham, gerhard.hipfinger@openforce.at, mailto:moz@moz.org>, florianweinwurm@yahoo.com, christian@precht-jensen.dk, Bill_Gates@microsoft.com, Tux_the_penguin@linux.rules.microsoft.sux.open.sou rce.is.the.way.to.go.net, domi@schnitzer.at, joe_ringmaster@gmx.at, sifu@isohypse.org, dk@perm.ru, nobandwidth@bigpond.com, nobandwidth@bigpond.com, luke@strangemonkey.com, mrundataker@optushome.com.au, mcgarry@tig.com.au, chris@think.net.au, Mathias.Burbach@Bigfoot.com, acuteparanoia@optushome.com.au, syzh401@cse.unsw.edu.au, maillist@jasonlim.com, ram@digitalmethod.org, jason@sydneypubguide.net, geek@digitalone.com.au, curious@ihug.com.au, bill@maidment.com.au, kristof@staesis.org, bill@microsoft.com, belle@netset.net.au, ksosez@softhome.net, jruderman@hmc.edu, andyed@surfmind.com, down8@yahoo.com, mozparty@sigkill.com, bulbul@ucla.edu, gavin-mozparty@doughtie.com, roger@digitalfountain.com, matt@linuxschooltorrance.com, mozparty@ventura.nu, rombouts@compuserve.com, ian@freenetproject.org, tristanreid@yahoo.com, groovefx@yahoo.com, jj@lacasabonita.com, gmoudry@hotmail.com, eyezero@yahoo.com, ian@primewave.net, jlawson7@adelphia.net, el_arturo@att.net, janie@freenetproject.org, 145371217@numenor.net, infinite_8_monkey@yahoo.com, charshman@divus.org, mozparty@shadowlurker.net, john@marinapacific.com, ilanterrell@yahoo.com, aafes@psu.edu, bustamam98@yahoo.com, mozparty@myunixbox.com, yaten@sbcglobal.net, joelinux@pacificnet.net, dgc@penguino.net, poserskater69@yahoo.com, lheartb@hotmail.com, ncmother@zimage.com, daniel@likeicare.com, digital.evil@lycos.com, cjeburke@yahoo.com, jblow@hotmail.com, zachary.anthony@verizon.net, boogah@23.org, mebelost@yahoo.com, nickkricheff@netscape.net, mikemcg@ucla.edu, gogomozilla@denofslack.net, mike@mm1.com, seanmcoleman@attbi.com, jsm@bigfoot.com, hoarycripple@crippl3.net, mozparty@nslu.x.myxomop.com, mozparty@camworld.com, mozpartyNYC@isoga.net, ccarlen@netscape.com, h@rediffmail.com, lefever@rcn.com, tedjackson@accounting.org, darren@ny.com, marlon@nyc.com, plui@hyperreal.org, dzeluff@zeluff.com, joel@natividads.com, ken@bigbadapple.com, treebeard@treebeard.net, florent@nyc.com, chad@macristy.com, spud@montelshow.com, gbman_of_gvill@yahoo.com, eam-mozparty@learningpatterns.com, pkrause@primavera.com, tossoffus@yahoo.com, ryan@pantz.com, nichomof@eecs.tulane.edu, billg@microsoft.com, DevilsRejection@msn.com, petergunn@hotmail.com, bagerj@sullcrom.com, isaac@structuredsystems.net, bobk@panix.com, ngellner@hotmail.com, luke@sigterm.org, vivake@yahoo.com, jon@mediavortex.com, groovefx@yahoo.com, brendan@sighup.net, jds@panix.com, bluerose@bluerose.com, chris@allermann.net, dimkal@yahoo.com, preppyl@yahoo.com, blujoker@blujoker.net, nowell_h@hotmail.com, aragorn@cs.stanford.edu, treed@cpr.com, brt204@nyu.edu, andreas@antonopoulos.com, dj@randomwalks.com, lists@pote.com, mike@mhudack.com, reliable57@yahoo.com, jared@geek-boy.com, ondadl@mac.com, floss@myrealbox.com, xod@thestonecutters.net, mozilla@sectae.net, tywonm@screamingmedia.com, Odin_NT@hotmail.com, crooney@panix.com, bg25222@binghamton.edu, eugenem@brainlink.com, dave@downneck.net, romspace@mac.com, sdaejo@yahoo.com, masseo1@yahoo.com, jim@fearandloathing.net, mike@mjoy.us, miles@openly.com, LuciferSD@hotmail.com, nsdilwor@intertechmedia.com, chrisdowden@yahoo.com, pgs10@columbia.edu, sbrennan@ovid.com, lthomiso@rcn.com, paralox@paralox.ath.cx, Jester_458@yahoo.com, jsadove@beltion.net, stuehmke@yahoo.com, mike@realfx.com, alex@risky-roosky.com, shava@efn.org, kra10@columbia.edu, saihung@ix.netcom.com, gropo@mac.com, scottnym@yahoo.com, shaas@vibe.com, roon_toon@hotmail.com, ajaygautam@yahoo.com, jhdaly@mindspring.com, manuel@sphinx.ms, very_itchy_rash@yahoo.com, emeldrum@drew.edu, jeld@mindless.com, as867@columbia.edu, slams@penguin.rutgers.edu, wassa@columbia.edu, tony@vegan.net, zilla@bibliotrack.com, zeno_lee@hotmail.com, fosh@fishnet.cx, linux@gpl.us, jblow@hotmail.com, dkrook@hotmail.com, ivesti@yahoo.com, arek@arekwyderka.com, bljoechang@yahoo.com, brian@tribrothers.com, sparky@marklife.org, charles@softwareprototypes.com, scottkundla@hotmail.com, ccharabaruk@meldstar.com, ian@pottinger.ca, netdemonz@yahoo.com, diatribe@mailcity.com, nick@tomkinet.com, shawnlin@yahoo.com, sculley@pathcom.com, herd.killing@rogers.com, dave@renouf.com, aliyamin@hotmail.com, aswitzer@ispgn.com, netm0nkey@ispgn.com, hyakugei@hotmail.com, geduggan.mozparty@peri.csclub.uwaterloo.ca, lwhite@darkfires.ca, jorel@the-wire.com, js@tap.net, davew@tap.net, tmh@whitefang.com, vid_mozillaparty@zooid.org, anon@foolswisdom.org, morris_mk@yahoo.ca, colinmc@idirect.com, marcus.brubaker@utoronto.ca, akish@kishcom.com, nconway@klamath.dyndns.org, jason@thegeekcave.com, rampaging_simian@hotmail.com, garret@sirsonic.com, piowie@myrealbox.com, m5m5m@yahoo.com, ivan.brovko@net-sweeper.com, returnofthedorks@hotmail.com, axxackall@yahoo.com, tednye@sympatico.ca, darren.fuller@bell.ca, jbailey@nisa.net, swangeo@yahoo.ca, Hercynium@yahoo.com, cinetron@passport.ca, jotaroh@hotmail.com, aghajani@principle.com, fzv@yahoo.com, rocketmail_com@rocketmail.com, foo@bar.com, wolfe@alt.net, drew@xyzzy.dhs.org, jimmiejaz@nixhelp.net, bofh@swma.net, nilesh_mehta@email.com, mslack@rogers.com, m-cahill@rogers.com, tworkowski@sympatico.ca, george@openlight.com, irina@openlight.com, ilia@lobsanov.com, rjs@tao.ca, paul-mp@it.ca, alvarolists@aycuens.com, xan@dimensis.com, ike@lab.org, miguel@asiinfo.net, marevalo@marevalo.net, iolalla@yahoo.com, peluz0n@justice.com, weeddeveloper@yahoo.com, alfonsobugs@terra.es, sgala@apache.org, z_gringo@hotmail.com, santiz@madritel.es, murphy@litio.net, fox@mozilla.gr.jp, party@mozilla.org.uk, danj@fledgeling.com, fun@thingy.apana.org.au, moz@the-allens.net, onelists@hotmail.com, joel@fysh.org, simon.mozilla-party-if-its-in-central-london@rumbl e.net, bigboyjim@excite.com, andrew.and.friends.iff.central.london@sent.freeser ve.co.uk, itwillbecentrallondon@mozilla.org.uk, noahsark2x2@tiscali.co.uk, mmm-central-london@smileyben.com, jonathan-for-central-london@peepo.com, dave-Party-in-Central-London@dgta.co.uk, DJGMOL@netscape.net, srick@europe.yahoo-inc.com, moz-party@zpok.demon.co.uk, moz-party-central-london@trickofthelight.org, marc@brosystems.com, party@budge.net, rillian@telus.net, uphillsurfer@hotmail.com, edward@debian.org, mozilla@robertbrook.com, reagan@technomoose.com, lew@saltbeefsandwich.co.uk, osama@afghanistan.com, barking@insaneworld.org.uk, john@billabong-media.com, leith@cs.bu.edu, mozparty@noseynick.org, jonasj@jonasj.dk, bugzilla@kenneth.dk, chr_damsgaard@hotmail.com, alring@email.com, hp.grondal@get2net.dk, martin@marquentein.dk, Lovechild@foolclan.com, Kim@schulz.dk, kl@vsen.dk, mbendix@dunghill.dk, schnitzer.at@tange.dk, tommy@svindel.net, moz10@pbb.dk, dezral@despammed.com, nick@tioka.com, ask@fujang.dk, gecko@c.dk, spam@deck.dk, bugzilla@gemal.dk, b@bogdan.dk, kenneth@gnu.org, jee@email.dk, daniel@rtfm.dk, umfalvo@yahoo.com, christian@ostenfeld.dk, xor@ivwnet.com, Jason@screaminweb.com, alex@spamcop.net, dustym@riseup.net, rmcgee1@earthlink.net, dr_zeus@hotmail.com, chris.lozano@myrealbox.com, looney_binn@yahoo(dot)com, apendell@attbi.com, dantrevino@wrevolution.org, fireball1244@mac.com, tommyo@hargray.com, natas@redtailboa.net, emmett_in_dallas@yahoo.com, razzbuten@yahoo.com, igdavis@truculent-telephone.org, foobar@null.net, bob@kludgebox.com, cgrimland@yahoo.com, ghamlett@swbell.net, bgood@inceptual.com, slot0k@pogox.org, kwhudson@netin.com, jimjamjoh@softhome.net, jimmys@utdallas.edu, charlesv@mfos.org chris@focus2.com jest6r@hotmail.com steve@ncc.com, usrg@mail.utexas.edu, steve@deltos.com, alex@avengergear.com, mkoenecke@alum.haverford.edu langley@hex.net mordred@inaugust.com swapan@yahoo.com drosoph@hotmail.com, goulash1@mac.com, ean@brainfood.com, vj@vj.com lpret42@hotmail.com bugoff@hotmail.com chad@digitaltriage.net, stewart@digitaltriage.net scottvr01@yahoo.com adam@dfwuptime.com dsaint@gnumatt.org naltrexone42@yahoo.com, webmaster@bast.net, tommyo@hargray.com, ladd@kryp.to, jtaylor5@bayou.uh.edu, jgschmitz@linuxmail.org, enslaver@enslaver.com edfierro@yahoo.com, moz@photonsphere.com, rayw@fuckmicrosoft.com, rfmobile@swbell.net, kevin@unif.com trident5@bigfoot.com Erik_Osterholm@ieee.org, tmunson@houston.rr.com, alessi_brand@hotmail.com, rballa1@lsu.edu, wasted@kewlhair.com, jofficer@martinapparatus.com, idiot@mylinuxisp.com, j0sh01@ev1.net faust@wintermarket.org bouncer@hotmonkeyporn.com tk-mozparty_@perljam.net janisch@students.zcu.cz, aha@pinknet.cz kuzi@atlas.cz scat@reboot.cz, petr@dousa.cz, ruzicka@core.cz, roman@management.cz, hojan@students.zcu.cz, tille@soti.org, cas.tuyn@hetnet.nl, aeon@pandora.be, sensi_millia2000@yahoo.com, crypto@shiftat.com, jan.fabry@vsknet.be, monkeyboy@fruru.com, adulau@foo.be, johan@linux.be, karu@pobox.com, soggie@soti.org nick@tomkinet.com, why_are_you_too_lazy_to_drive_1_hour_to_toronto@yo u_lazy.com try_grammer_class_a_while@get_a_life.com john@interlynx.ca asharp@axo.cc, unionstation@ryder.ca, prade@hotmail.com, 2600@hamilton2600.ca, chris.lozano@myrealbox.com, dantrevino@wrevolution.org, jksteinhauer@netscape.net, i_love_junk_email@yahoo.com, cmiller@surfsouth.com, jan@bestbytes.de, me@phillipoertel.com, sebastian@pixelsalon.de, ccozan@andtek.com, ben@itlib.de, martin.ament@gmx.de, pulsar@highteq.net, muid@gmx.de, cedi@zooomclan.org, soapy@soapy.ch, deep_blue_ocean@gmx.ch, stamp@zooomclan.org, hans@switzerland.com, milamber@zooomclan.org, mtettea@switzerland.com, cylander@zooomclan.org, duke@zooomclan.org, pegirun@gmx.ch, pilif@pilif.ch, mlati@yahoo.com, Mozillzooom@holophrastic.com, erichiseli@yahoo.com, la_burdet@yahoo.com, rkoerber@gmx.de, dotzmasta@hotmail.com, B.Eckstein@cli.de, rtfm@linux.de, info@phosmo.de, gz@disintegrated.de, byronbay@gmx.de, stiwi@mac.com, mage@koeln.netsurf.de, mozilla@portfolio16.de, wrede@fh-aachen.de, ilikemozilla@html.de, cloud@final-fantasy.de, sfricke@sfricke.de, info@flossbau.de, no@dom.de, julian.suschlik@gmx.net, omero@m4d.sm, lapo@lapo.it, alcor78@email.it, info@fuelcat.it, mutato@libero.it, ildella@inwind.it, a.marabini@spinthehumanfactor.com, uomoman@criticalbit.com, thefl74@netscape.net, elbardo@libero.it, clem131@libero.it, t-i-e@bigfoot.com, gng74@libero.it, moz.party.20.gnes@spamgourmet.com, ema.cerqui@libero.it, ubertob@tin.it, mozparty.20.anagoor@spamgourmet.com, gianpaolo@preciso.net, ian@deepsky.com, marco@porciletto.org, planetx2100@hotmail.com, billabong@tiscalinet.it, piofree@libero.it, skunkyboy@tiscalinet.it, vincenzo@mondopiccolo.net, macmatteo@interfree.it, contreras@jce.it, hereandnow@libero.it, pza@students.cs.mu.oz.au, caedwa@students.cs.mu.oz.au, mgi@students.cs.mu.oz.au, bah@humbug.net, mfp@cs.mu.oz.au, nospamplease@indevelopment.org, peter@simplyit.screaming,net, pmj@users.sf.net, xanni@sericyb.com.au, agh@kalcium-is.com, felicityconsult@ozemail.com.au, lucas@lucaschan.com, andrewg@nopninjas.com, andym@abnormal.com, ts@meme.com.au, jasonpell@hotmail.com, syngin@gimp.org, mhammond@skippinet.com.au, szutshi@devraj.org, rmoonen@bigpond.net.au, fawad@fawad.net, ufs@softhome.net, kotrade@yahoo.com, ben@benscorp.com, stevesmith@columbus.rr.com, kkimmelosu@yahoo.com, neal.lindsay@peaofohio.com, pat@linuxcolumbus.com, chrisbaker@iname.com, hiroki2c@yahoo.com, seth@remor.com, jsohn@columbus.rr.com, ross@nanonet.net, mark@cushman.net, swinghammer.2@osu.edu, roberto.12@osu.edu, farhat@hotmail.com, pgunn@dachte.org, jwagner@gcfn.org, bp@osc.edu, joepletch@postmark.net, dsherman@iwaynet.net, glenn@uniqsys.com, bernstein.46@osu.edu, trent_reznor@nothing.com, erikniklas@bobanddoug.com, walters@gnu.org, timo@bolverk.net, annek25@aol.com, jlamb@leader.com, bart@osc.edu, jason@mcvetta.org

Someone should write a P2P file exchanger using Digital Founain over multicast. I'm going to make the assumption that a lot of P2P is multiple people downloading the same file.

Moreover, if you are bothered by the NNTP, you can get a full (20-30 Mbps) news feed via satellite for just a few hundred per month.

Now this is interesting... My previous comment (parent of this one) was slightly wrong because the solution proposed by this company works differently. They use some FEC method that is better than Reed-Solomon codes, as I discovered later by reading their detailled technical documentation (available in their technology library.)

But I cannot understand why my comment which proposed a different method was rated three times as "Overrated (-1)", while its parent was rated twice as "Interesting (+1)". I do not understand why three different moderators would give exactly the same moderation, especially when the comment was already scored at 0 and the article contains many other comments could have been moderated up or down.

Strikes me as kind of odd that they need to make a piece of hardware to do this, especially when they say that it "relies instead on the computational power of standard PC processors" and "The translation is straightforward enough to be handled by a Pentium III processor" ... so why not just sell the software to do this? I suppose then they wouldn't get $70,000 a copy for it.
...
ok, I think I'm answering my own question (partially) .. according to their website the "box" they keep referring to in the article is a special server (1.5Gbps and 5 or 85 GB of storage) with their software on it. And you don't need two of these boxes for a complete connection. The server comes with a Fountain Client program to receive transfers. And they sell an SDK to make your own clients as well.

seems like they should've mentioned some of this in the article

For example, consider the transfer of a 1 GB file with an average Internet packet loss rate (2%) and global average RTT (just over 200 msec). For this case TCP has an inherent bandwidth limitation of 300 Kbps regardless of link speeds and thus will take more than 7 hours to transfer the file. A Digital Fountain server sending Meta-Content allows almost complete bandwidth utilization. With Digital Fountain, on a fully utilized 1.5 Mbps (T1) line the transfer of the 1 GB file will take about 1.5 hours and on a fully utilized 45 Mbps (T3) line the transfer time will be a little over 3 minutes.

I was a Skeptic, but I'm now that I've read the Digital Fountain white papers: Meta-Content Technology White Paper and TCP White Paper, I'm a True Believer.

I don't pretend to understand all the details, but the technology is based on the Luby transform which was invented by the company's founder. Essentially the content is broken into segments which are then randomly XORed to produce the metacontent. This provides redundancy since each segment of metacontent contains information from several of the original segments. Also the metacontent segments do not require sequential transmission.

For example, consider the transfer of a 1 GB file with an average Internet packet loss rate (2%) and global average RTT (just over 200 msec). For this case TCP has an inherent bandwidth limitation of 300 Kbps regardless of link speeds and thus will take more than 7 hours to transfer the file. A Digital Fountain server sending Meta-Content allows almost complete bandwidth utilization. With Digital Fountain, on a fully utilized 1.5 Mbps (T1) line the transfer of the 1 GB file will take about 1.5 hours and on a fully utilized 45 Mbps (T3) line the transfer time will be a little over 3 minutes.

I was a Skeptic, but I'm now that I've read the Digital Fountain white papers: Meta-Content Technology White Paper and TCP White Paper, I'm a True Believer.

I don't pretend to understand all the details, but the technology is based on the Luby transform which was invented by the company's founder. Essentially the content is broken into segments which are then randomly XORed to produce the metacontent. This provides redundancy since each segment of metacontent contains information from several of the original segments. Also the metacontent segments do not require sequential transmission.

A tech paper on their "tornado codes". Also, the link to their tech papers website.

Didn't have much chance to look over the algorithms, but we really should have compression used more frequently in network transport.

A tech paper on their "tornado codes". Also, the link to their tech papers website.

Didn't have much chance to look over the algorithms, but we really should have compression used more frequently in network transport.

This technology (from the write-up anyway) uses some kind of proprietary technique to re-map the data into another domain and send the information required to reproduce it. It sounds kind of like sending a waveform as a series of Fourier coefficients rather than as actual data samples.

Since it doesn't use TCP, I'll bet it won't have any problem handling the TCP part of "TCP/IP connections"... The networking end sounds really simple - send the amount of data to expect, wait for confirmation, send all of the data once *without* waiting for confirmation until it's all been sent. That's a lot easier than handling the overhead of tcp, which is the whole problem they're trying to solve.

BTW, it helps to read the article before posting "insightful" comments. :) There's a nice little demo at http://www.digitalfountain.com/technology/coreTech nology.htm

This technology is truly cool. Invented by Michael Luby, it addresses this very paradox, that the more popular the content is the more it costs to deliver with a one viewer per stream approach. The demo showed a server pushing an MPEG2 stream to about 20 different clients with different starting times. Watching the load on the server showed a negligible increase for each new client. It was astounding, so much so that I felt kind of sorry for all the folks pushing their one stream/one viewer media servers. I want one for delivering more riding and running videos...

As gets mentioned here from time to time, Digital Fountain addresses (or endeavors to address) the "recipients in sync" problem.

I'd be surprised if multicasting ever comes into very wide use; since the situations that it's useful under are limited...

Where I work, many of us kids like to listen to streaming radio broadcasts. We've been criticized for the strain we put on our Internet connection, and it's a valid point. Often several of us are listening to the same shoutcast stream (or whatever) at the same time, and it seems kinda silly that we consume N * 50-100kbps of bandwidth to receive the same content. But, hey, this is just a personal way in which multicast could help my life.

When most people think of multicast they think of 1-to-many transmission. There are also lots of applications involving many-to-many transmission. Chat is an obvious one; chat becomes particularly well suited for multicast when you're dealing with voice chat rather than text. A more interest application is in networked virtual environments (less grandiloquently, games). A couple other fellows and I wrote the networking part of an NVE that used many-to-many multicast: the world was partitioned into octrees, and each octree was assigned a multicast group. Octree nodes split and merged based on traffic, and there were different levels of groups for messages of different levels of detail (e.g., toe movements vs. explosions). (Well, this was the plan; we didn't finish all of it, but it was a cool demo). Peer-to-peer NVEs have many advantages over client-server systems, including reduced (and hopefully optimally minimal) latency and natural scalability. This book provides an overview of the subject, but there are many papers out there that are more in-depth and informative.

Finally, check out Kevin Almeroth's research in multicast applications. He has several good survey papers that address your synchronized play out gripe and explore the gamut of potential multicast applications.

Digital Fountain seeks to solve this problem.

For a good survey of the the past, present, and future of interdomain multicast routing, try this paper.

As others have mentioned, yes, flow control is a problem, and, yes, reliability is a problem. There are many solutions to both problems that have been researched (largely in academia), but flow control can't really be solved if you're trying to distribute RedHat over multicast to half a million people who are on disparate links. Someone a few posts up mentioned the Digital Fountain idea, but neglected to provide a link. Digital Fountain aims to solve the problems that are being discussed here for exactly the kinds of applications that are being discussed here. The paradigm is that random bytes are constantly flowing from the fountain (the multicaster) and the recipients fill up their buckets with the random bytes until a file is formed. Read their papers for a more mathematically rigorous explanation...