Slashdot Mirror
MBONE for Software Distribution?
Warren Vosper asks: "As I sit here twiddling my thumbs, waiting for the RedHat mirror sites
to finish pulling down RH7, I ponder the need for this. Why can't we
use the MBONE to update the mirrors? I could satisy my burning
need for instant gratification *so* much sooner. Hell, why couldn't
I tune in to an MBONE broadcast from RedHat and get it at the
same time as the mirror sites? As I looked over the ancient (5-6 years
ago) online info regarding MBONE I understand that it's used mostly for video and audio, but why not software distribution?"
Well, mbone as you describe it is not quite what multicast was originaly designed as. Only when an end client wanted content would that branch of the mbone be active. Also think of the requirement of synchronising the downloading to thousands of users whereas video/audio can be joined at any point. Now a auto-restarting/looping client would be smart enough to start saving at any point and when it reached the end and started the start again it would save itll it reached the point in the loop where it began and add the two halves. The way a true multicasting network works is that as a client wants a link it sends a request that that mcast address, the default gateway router would hear this and request an upstream link to that mcast IP. Then the next, and the next and soon untill the branch grew back to an allready recieving branch fo the tree. This is how mcasting was intended to operate. not a continual broadcast whether you want it or not type setup you described. The simple reason software dist of this nature doens't exist is because i don't think there are any truly inteligent tools to recieve it. Also as far as software the end client may not be able to recieve at the speed the root is sending out on the branch at. Now if mbone supported downsampling, say recieve every other, or third, or fourth packet it'd be better. but then you get into the problem of the router managing the rates it's downstream branches need or are limited too and getting a copy of each packet goign downstream. This method of software distribution is interesting and usefull for high download count sites. It just need the custom modifications to the mbone system and the software to do it with. I'd suggest starting with your own design, then present it to the IETF and hope you get accepted as a RFC then a couple years later ratified. hops this clears it up for everyone!
Just a couple of points. You were using multicasting, not MBONE - there's a difference, one is a protocol, the other is a network established to test large scale multicasting.
And secondly, Ghost killed your network because your switch is dumb, misconfigured or both. In order for multicasting to work well on a switched network, the switch has to listen in on the multicast (usually referred to as IGMP snooping) to determine which ports are part of the multicast. If your switch is unable to do this, whether due to design deficiency or misconfiguration, the multicast session devolves into a bandwidth sucking broadcast storm.
I've actually wrote a broadcast-oriented ftp around 6 or 7 years ago; it's currently used by a Fortune 500 company to xmit large chunks of data to 3000 locations, simultaneously. This is in a satellite WAN environment.
As for multicast: this is something I've been looking into lately. For those interested, a few pointers: Cisco has a few articles RFC draft, which has expired.
For a general discussion of IP Multicast, check out IPMulticast.com, especially the tech section which discuss reliable multicast protocols.
I should mention that there is a lot of work going into multicast protocols these days, for various reasons, and that, generally, multicast protocols aren't very generally applicable. Some, for instance, are useful for gaming, were the data is time sensitive, but the reliability of data transport doesn't have to be perfect. Data transport is obviously more important for an ftp-like system. Besides the more traditional use of UDP as a protocol base, some folks are implementing their own protocols on top of raw IP.
For a vendor's perspective on a product/implementation, check out Talarian, which has a reliable multicast product. [ You can even get code, if you register ] Note: this relies on the PGM protocol.
Check out Vaccine for an effort to create a multicast distro image distribution tool.
The Norton Anthology of English Literature, 4th Ed., Vol 2
Fcast seems way more complicated than it needs to be.
Say that you have a file that you want to send to a lot of people. These people are going to want to get the file as fast as possible, but they are also all going to have differing speed connections.
Now, as the sender of the file, we would like to minimize the number of packets we send, but we don't have to ensure that we only send each packet once, we just need to be better than sending every packet once to every recipient.
So, instead of using one multicast channel, use a bunch. Each channel broadcasts at some lowest common denominator speed which can be picked based on your intended recipient's networks (if you don't know, you need 14.4kbps or something like that). Then, compute the time it will take to transmit the entire file at that speed. Time shift each channel by channelNumber*totalTime/numberOfChannels and start broadcasting all of them continuously, at the prechosen speed.
Now, as a receiver, you know how much bandwidth the sender has to you (or at least you can figure it out). Simply subscribe to the largest number of channels you can w/o getting dropped packets over some threshold. You might get some duplicate packets from wrap around between the beginning and ending of the transmission, but those can be tossed.
If packets are lost, you could either request specific packets from the server (if you have only a few and the server isn't too loaded), or you could just jump on the channel that will have that packet soonest (and onto another channel if you miss it again, rinse, wash, repeat).
Assuming a constant base channel speed (which seems reasonable until broadband access is more wide spread), the trade off here is the number of channels. By increasing the number of channels, the sender has to repeat each packet more times, but the clients can have better maximum throughput and less time to wait to replace dropped packets.
There is probably some additional cost at the routing layer for all these people subscribing and unsubscribing from extra channels, but I assume (maybe incorrectly) that the routing layer would be able to handle this problem since it would be distributed across a whole bunch of routers.
First, what it really is. Multicast on a local level uses a special range of IP addresses (class D addresses) that are mapped into NIC addresses using a sort of hash. Modern NIC cards have filters so they can listen in to only traffic sent to them or to a multicast address placed in the filter. This keeps the card from having to process every single packet on the network.
It is possible to route multicast traffic by tracking mulitcast group subcribership and TTL. With this, traffic will only go onto a branch of a network if someone is there to listen to it. That is the whole concept for multicast, elminating redundant or unneeded traffic streams.
The MBONE, or Multicast Backbone, is a way of tieing together networks running multicast traffic over a non-multicast capable intermediate provider (usually your ISP). MBONE works by creating tunnels through the unicast network to carry multicast traffic. It's very inefficient, but it was a way to bootstrap people up. MBONE should have been a temporary stopgap measure. It would have gone away once the service providers upgraded their equipment to support multicast routing. Unfortunately, that has never happened.
Next, a little background. I first saw multicast and MBONE demonstrated at Networld+Interop in 1994. The demo was casting a stereo quality radio signal. I fell in love with the technology then and kept in eye on it for the next six year. It still amazes me it hasn't gone further. The biggest reason I can see for the lack of progress in a complete non-interest by Internet service providers. The is odd since it could save them on bandwidth costs in the long run.
As mentioned before, the most common uses for multicast are audio and video which can support a little bit of loss. Groups like broadcast.com would all but go out of business if multicast came into universal use. If you are sending a 28Kbps audio stream, you only need 28Kbps of outgoing bandwidth. The signal will just divide at routing points until it reaches the end subscribers. If nobody is subscribing to a particular broadcast down one branch, no traffic will go down that branch. Perhaps this scares the service providers since just about anyone could setup their own audio or video broadcasting service with a ISDN (or DSL) line for bandwidth.
With a little work, non-loss tolerant products can be moved via multicast as well. Forward error correction is one way. You could also have the routing points cache a piece of the stream to ask for a rebroadcast as well. In a worst case scenario, you connect clear back to the sender using unicast to request just the blocks of data you missed. From here, it would be possible to send back down via unicast or via the multicast channel so anyone else who missed that block could pick it up. It's all a matter of how creative the programmers want to get with their file transfer service.
Software updates are one file item to use for multicast. It would also be possible to come up with a multicast FTP that could allow several receivers to tap into a stream once the sender is already going. If someone comes in half way through, the sender would just start transmitting from the beginning again once the end of the file is reached. This combines both on demand transfer with streaming broadcast. Best of both worlds.
Usenet news is another good place where multicast could cut down on bandwidth use. Give major branches of the usenet tree their own multicast address. This way, if someone doesn't want the "alt.binaries.pictures.erotica" subtree, they just don't pick up that transmission.
Time sync was a planned use for multicast. ntpd has provisions for receiving packets via multicast. There is even a special address set aside (and defined in the "mcast.net" domain). A few master servers could keep the whole Internet on common time
Personally, I'm looking forward to the day when the entire Washington Post is pushed into my set top box via multicast at night. It could then bounce over to my PDA via BlueTooth and be ready for me to take a long to the train in the morning. Since the content is all advertising supported, the paper from a digital source could be free.
There is so much possibility here, it just needs to be tapped.
World Beach List, my latest project.
Since MBONE isn't for reliable data transfer, the mirrors could receive most of an iso image of the RedHat archive, then use RSync to "correct" any of the errors. Anyone who has built a debian image this way (download all the packages seperately, cat them together with 2048byte block padding, then use rsync to rearrange the files and add the rest of the iso data to the image). Very cool stuff.
There is a company, Digital Fountain, that has a system for doing bulk software distribution over multicast. (It's pretty cool--- they use an encoding that allows you to collect _any_ N distinct packets from the datastream and recreate the original file.) I don't know whether their software could be used over the MBONE, but hopefully the deployment of single-source multicast (which a lot of people are excited about) will help get things moving toward large-scale availability of multicast. Interested parties should probably check out the SIGCOMM '98 paper.
becuase with multicast, there is no flow control?
And no guaranteed delivery?
This is perfectly acceptable for media broadcast, where the codec can deal with dropped bytes.. but..
|-| == bandwidth to move it.
|----| == tarball size. 80s
|--| == bandwidth to move it.
|--------| == tarball size. 90s
|----| == bandwidth to move it.
Over the last 30 years of computing, we've always had more program than pipe to push it through. The only way to overcome this slowly increasing speed of affordable bandwidth, is to pay big bucks for a line that will be outdated in a few years.
In the not to distant past, todays game emulator ROMs used to be moved across the country in a game console containing a huuuuge amount of graphic hardware,. For the day, having a game that totaled more than 1 Meg in run time size was just gigantic, and now, we zip these same ROM images around the net in seconds.
In the very near future IP6 over Multi-Gps fixed wireless will make mirroring linus' balls of tar a trivial task. but, of course, by then, the "kernel" will be 200G ;).
The lesson here is that affordable bandwidth, slowly and stedily has increased over the history of computing, and I see no reason why it should jump.
See http://www.lucent.com/press/0597/970506.bla.html
Most multicast-native customers that make use of the MBONE have quite a bit bandwidth to toss around for video and data broadcasts, or it is part of their business model. (broadcast.com, NASA JPL, US DOE, etc.)
Now in regards to software distribution, it would not be feasible for RedHat to multicast a 600Mb ISO using the Internet multicast backbone as each provider that wanted access to that data would also subject their providers, and their providers providers' to receiving that data as well. So essentially you would have 600Mb flying through 6 transit networks to reach you. Imagine the waste of bandwidth. Do you think multicast providers would take this with an enthusiastic grin?
Currently, there are a few providers that use multicast for stream distribution to multiple servers on live events. You can be assured this is the case for large scalable video distribution houses like broadcast.com, possibly Akamai and others. Hope that provides some insight. I'm not an expert, I've just been performing a lot of multicasting research as of late. Cheers.
The data then branches off from there. This would be quite suitably for updating mirror sites, since
One problem I could see is that this method of distribution for data files (versus video and audio) wouldn't scale well. Imagine one site drops a packet. Well it can't very well start over, since that same packet did possibly reach all the other listening parties. They are all expecting the NEXT packet, not a retransmit.
On an fast, fault tolerant network (major backbones, and obviously intranets) this works great (we use ImageCast at work to simulcast drive images to multiple systems) the bandwidth used is no more than if a single system was done one at a time. But on any network where packet loss and latency are a problem, thing would seriously hamper to practibility of the system.
So I say, multicast to a few hundred major FTP mirrors from the master server (redhat in this case), and then good ol' traditional FTP from there.
For a good survey of the the past, present, and future of interdomain multicast routing, try this paper.
As others have mentioned, yes, flow control is a problem, and, yes, reliability is a problem. There are many solutions to both problems that have been researched (largely in academia), but flow control can't really be solved if you're trying to distribute RedHat over multicast to half a million people who are on disparate links. Someone a few posts up mentioned the Digital Fountain idea, but neglected to provide a link. Digital Fountain aims to solve the problems that are being discussed here for exactly the kinds of applications that are being discussed here. The paradigm is that random bytes are constantly flowing from the fountain (the multicaster) and the recipients fill up their buckets with the random bytes until a file is formed. Read their papers for a more mathematically rigorous explanation...
... and published on Doctor Dobb's Journal (yes, the source code is available). See http://www.ddj.com/articles/2000/0005/0005i/0005i. htm
The guy who wrote it works for Microsoft (so of course his implementation is Windows-dependant) but he makes some pretty good points on using multicast for file distribution, and naturally the idea and/or algorithms could be reimplemented in some D-O-S (Decent Operating System) like Linux...
Best Regards,
--
Durval Menezes.
Best Regards,
Durval Menezes.
I have never met a computer that didn't like me.
I teach classes on NT and various Unices. At the end of the week, all the machines have to be rebuilt. On NT, Norton's Ghost does a speedy multicast of a disk image. This is handy, so I've started writing an analous program for Unix. The plan is that it would be on a boot floppy. It is still new, but coming along nicely. The GPL'd source is at http://www.projectmanagersmba.com/casper.tar