Slashdot Mirror
MBONE for Software Distribution?
Warren Vosper asks: "As I sit here twiddling my thumbs, waiting for the RedHat mirror sites
to finish pulling down RH7, I ponder the need for this. Why can't we
use the MBONE to update the mirrors? I could satisy my burning
need for instant gratification *so* much sooner. Hell, why couldn't
I tune in to an MBONE broadcast from RedHat and get it at the
same time as the mirror sites? As I looked over the ancient (5-6 years
ago) online info regarding MBONE I understand that it's used mostly for video and audio, but why not software distribution?"
and yes, for large numbers of receivers it would be considerably more efficient
look at the SRM protocol and variants for an example.
why dont we use them then? because MBONE penetration is still dismally low...because no one has thought to ask for it. which is strange given the new unicast (wasteful) emphasis on streaming media..the internet had live video and audio more than 8 years ago..with open source tools
even if you have MBONE traffic its still very much a second-class citizen...its not uncommon to see loss rates in excess of 30% due to provider imposed rate limits
Well, mbone as you describe it is not quite what multicast was originaly designed as. Only when an end client wanted content would that branch of the mbone be active. Also think of the requirement of synchronising the downloading to thousands of users whereas video/audio can be joined at any point. Now a auto-restarting/looping client would be smart enough to start saving at any point and when it reached the end and started the start again it would save itll it reached the point in the loop where it began and add the two halves. The way a true multicasting network works is that as a client wants a link it sends a request that that mcast address, the default gateway router would hear this and request an upstream link to that mcast IP. Then the next, and the next and soon untill the branch grew back to an allready recieving branch fo the tree. This is how mcasting was intended to operate. not a continual broadcast whether you want it or not type setup you described. The simple reason software dist of this nature doens't exist is because i don't think there are any truly inteligent tools to recieve it. Also as far as software the end client may not be able to recieve at the speed the root is sending out on the branch at. Now if mbone supported downsampling, say recieve every other, or third, or fourth packet it'd be better. but then you get into the problem of the router managing the rates it's downstream branches need or are limited too and getting a copy of each packet goign downstream. This method of software distribution is interesting and usefull for high download count sites. It just need the custom modifications to the mbone system and the software to do it with. I'd suggest starting with your own design, then present it to the IETF and hope you get accepted as a RFC then a couple years later ratified. hops this clears it up for everyone!
Just a couple of points. You were using multicasting, not MBONE - there's a difference, one is a protocol, the other is a network established to test large scale multicasting.
And secondly, Ghost killed your network because your switch is dumb, misconfigured or both. In order for multicasting to work well on a switched network, the switch has to listen in on the multicast (usually referred to as IGMP snooping) to determine which ports are part of the multicast. If your switch is unable to do this, whether due to design deficiency or misconfiguration, the multicast session devolves into a bandwidth sucking broadcast storm.
There are a few problems with using any type of multicast network to do large scale distribution to end users.
The first is very few people have MBONE access. Most dialups certainly don't have it due to a good portion of the terminal servers simply not supporting IGMP. There is also the issue of ISPs getting access from their uplinks which is like pulling teeth with the bulk of tier 1 providers out there. Most routers have multicast disabled by default which hasn't helped much.
The second is the need for a reliable multicast protocol. The problem isn't getting a reliable multicast protocol, the problem is choosing between the 30 or so that were tailor made for specific configurations. The reliable multicast protocols are complex... they have to deal with slow clients, retransmits to individual hosts, ordering, etc. Getting an complete and open general purpose reliable multicast protocol for use on the Internet which can be made into a standard is the problem.
The complete lack of a 'killer application' to force end users to request multicast capable access to the internet is one of the reasons this stuff hasn't taken off. If it had, I'd like to think I'd be streaming HBO to my laptop and watching it right now (which could use existing UDP multicast streams).
As a side note, IPv6 has mandatory multicast support from what I've read which if it ever is actually deployed fully might finally make some of this stuff a reality.
The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
..like audio and video. Perhaps you could make clients close to the source receive most packets, but we, here, in the south of Brazil, would get only a few packets, since most of the data would be dropped on congested routers. If the server would try to compensate for this by resending some packets to each client who requested, then it is just easier to make unicast connections. Multicast is ok to Audio and Video, but not ok to connection oriented stuff.
Multicasting is a wonderful method of distributing data, and it is tragically under-used. Mostly because it's harder to tell which adverts are being read and by whom.
(Though, also because many backbone providers are elitist uber-snobs, who prefer keeping the fun stuff for themselves.)
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
There are a couple of efforts going though. Check out the Lightweight Reliable Multicast Protocol (LRMP). You may also want to search for data fountains.
--
I've actually wrote a broadcast-oriented ftp around 6 or 7 years ago; it's currently used by a Fortune 500 company to xmit large chunks of data to 3000 locations, simultaneously. This is in a satellite WAN environment.
As for multicast: this is something I've been looking into lately. For those interested, a few pointers: Cisco has a few articles RFC draft, which has expired.
For a general discussion of IP Multicast, check out IPMulticast.com, especially the tech section which discuss reliable multicast protocols.
I should mention that there is a lot of work going into multicast protocols these days, for various reasons, and that, generally, multicast protocols aren't very generally applicable. Some, for instance, are useful for gaming, were the data is time sensitive, but the reliability of data transport doesn't have to be perfect. Data transport is obviously more important for an ftp-like system. Besides the more traditional use of UDP as a protocol base, some folks are implementing their own protocols on top of raw IP.
For a vendor's perspective on a product/implementation, check out Talarian, which has a reliable multicast product. [ You can even get code, if you register ] Note: this relies on the PGM protocol.
Check out Vaccine for an effort to create a multicast distro image distribution tool.
The Norton Anthology of English Literature, 4th Ed., Vol 2
YES, it is!
How did I install Windows 2000 on my Cheap-O(TM) computers with crappy parts? I just dumped the installation files onto the hard drive and ran the installer. It found all of my hardware, with no problems. It even supported my Intel webcam (courtesy of DigiMarc, thanks) although Intel said that they don't work together.
How did I install Linux on a fairly standard computer with good two-year-old hardware? Most of the Debian install was very painless, even with two floppies and the rest being pulled over FTP. But to get it to work with my 3Com NIC (3Com!) NIC, I had to manually install kernel modules.
Granted, things were somewhat different after the install process, with the expected results...
--
Fcast seems way more complicated than it needs to be.
Say that you have a file that you want to send to a lot of people. These people are going to want to get the file as fast as possible, but they are also all going to have differing speed connections.
Now, as the sender of the file, we would like to minimize the number of packets we send, but we don't have to ensure that we only send each packet once, we just need to be better than sending every packet once to every recipient.
So, instead of using one multicast channel, use a bunch. Each channel broadcasts at some lowest common denominator speed which can be picked based on your intended recipient's networks (if you don't know, you need 14.4kbps or something like that). Then, compute the time it will take to transmit the entire file at that speed. Time shift each channel by channelNumber*totalTime/numberOfChannels and start broadcasting all of them continuously, at the prechosen speed.
Now, as a receiver, you know how much bandwidth the sender has to you (or at least you can figure it out). Simply subscribe to the largest number of channels you can w/o getting dropped packets over some threshold. You might get some duplicate packets from wrap around between the beginning and ending of the transmission, but those can be tossed.
If packets are lost, you could either request specific packets from the server (if you have only a few and the server isn't too loaded), or you could just jump on the channel that will have that packet soonest (and onto another channel if you miss it again, rinse, wash, repeat).
Assuming a constant base channel speed (which seems reasonable until broadband access is more wide spread), the trade off here is the number of channels. By increasing the number of channels, the sender has to repeat each packet more times, but the clients can have better maximum throughput and less time to wait to replace dropped packets.
There is probably some additional cost at the routing layer for all these people subscribing and unsubscribing from extra channels, but I assume (maybe incorrectly) that the routing layer would be able to handle this problem since it would be distributed across a whole bunch of routers.
DF's work is based on Tornado Codes, which use some really neat tricks with graphs and XORs to reconstruct an N-packet file from any N+e (small e) out of a stream of KN (for some non-trivial K, say 4) packets. John Byers is largely responsible for this invention - see his SIGCOMM98 and INFOCOM99 papers, among others. He's currently doing lots of work in the multicast/congestion control/next-gen network areas. (How do I know this? His office is maybe 20 yards from mine.)
You're right, but it's not really fair to say they're "just improved RS codes"; by adding epsilon to the reconstruction requirement, the cost of doing the initial encoding changes orders, which is a huge win considering how prohibitive the RS encoding algorithms are for non-trivially sized objects.
First, what it really is. Multicast on a local level uses a special range of IP addresses (class D addresses) that are mapped into NIC addresses using a sort of hash. Modern NIC cards have filters so they can listen in to only traffic sent to them or to a multicast address placed in the filter. This keeps the card from having to process every single packet on the network.
It is possible to route multicast traffic by tracking mulitcast group subcribership and TTL. With this, traffic will only go onto a branch of a network if someone is there to listen to it. That is the whole concept for multicast, elminating redundant or unneeded traffic streams.
The MBONE, or Multicast Backbone, is a way of tieing together networks running multicast traffic over a non-multicast capable intermediate provider (usually your ISP). MBONE works by creating tunnels through the unicast network to carry multicast traffic. It's very inefficient, but it was a way to bootstrap people up. MBONE should have been a temporary stopgap measure. It would have gone away once the service providers upgraded their equipment to support multicast routing. Unfortunately, that has never happened.
Next, a little background. I first saw multicast and MBONE demonstrated at Networld+Interop in 1994. The demo was casting a stereo quality radio signal. I fell in love with the technology then and kept in eye on it for the next six year. It still amazes me it hasn't gone further. The biggest reason I can see for the lack of progress in a complete non-interest by Internet service providers. The is odd since it could save them on bandwidth costs in the long run.
As mentioned before, the most common uses for multicast are audio and video which can support a little bit of loss. Groups like broadcast.com would all but go out of business if multicast came into universal use. If you are sending a 28Kbps audio stream, you only need 28Kbps of outgoing bandwidth. The signal will just divide at routing points until it reaches the end subscribers. If nobody is subscribing to a particular broadcast down one branch, no traffic will go down that branch. Perhaps this scares the service providers since just about anyone could setup their own audio or video broadcasting service with a ISDN (or DSL) line for bandwidth.
With a little work, non-loss tolerant products can be moved via multicast as well. Forward error correction is one way. You could also have the routing points cache a piece of the stream to ask for a rebroadcast as well. In a worst case scenario, you connect clear back to the sender using unicast to request just the blocks of data you missed. From here, it would be possible to send back down via unicast or via the multicast channel so anyone else who missed that block could pick it up. It's all a matter of how creative the programmers want to get with their file transfer service.
Software updates are one file item to use for multicast. It would also be possible to come up with a multicast FTP that could allow several receivers to tap into a stream once the sender is already going. If someone comes in half way through, the sender would just start transmitting from the beginning again once the end of the file is reached. This combines both on demand transfer with streaming broadcast. Best of both worlds.
Usenet news is another good place where multicast could cut down on bandwidth use. Give major branches of the usenet tree their own multicast address. This way, if someone doesn't want the "alt.binaries.pictures.erotica" subtree, they just don't pick up that transmission.
Time sync was a planned use for multicast. ntpd has provisions for receiving packets via multicast. There is even a special address set aside (and defined in the "mcast.net" domain). A few master servers could keep the whole Internet on common time
Personally, I'm looking forward to the day when the entire Washington Post is pushed into my set top box via multicast at night. It could then bounce over to my PDA via BlueTooth and be ready for me to take a long to the train in the morning. Since the content is all advertising supported, the paper from a digital source could be free.
There is so much possibility here, it just needs to be tapped.
World Beach List, my latest project.
Since MBONE isn't for reliable data transfer, the mirrors could receive most of an iso image of the RedHat archive, then use RSync to "correct" any of the errors. Anyone who has built a debian image this way (download all the packages seperately, cat them together with 2048byte block padding, then use rsync to rearrange the files and add the rest of the iso data to the image). Very cool stuff.
There is a company, Digital Fountain, that has a system for doing bulk software distribution over multicast. (It's pretty cool--- they use an encoding that allows you to collect _any_ N distinct packets from the datastream and recreate the original file.) I don't know whether their software could be used over the MBONE, but hopefully the deployment of single-source multicast (which a lot of people are excited about) will help get things moving toward large-scale availability of multicast. Interested parties should probably check out the SIGCOMM '98 paper.
I don't know about you, but I'm able to multicast financial data across our network to over 2000 hosts without causing significant problems at all. use good switches, that have capable backplanes. wirespeed is a good thing.
(www.foundrynet.com)
EOM
Quite a few of the financial data providers use multicast over private networks to transport realtime stock quotes and data nationally/internationally. one of these is www.bridge.com. The stock markets are moving to decimalized trading, which means things like .01 cents vs. 1/16 of a dollar, etc. this ends up with much larger amounts of data being whipped around. Doing research on this for our firm I found out several things, including the fact that bridge runs all their data across an ATM backbone, doing multicast. pretty damn slick. Various other providers do this as well, but they're the one I'm most familiar with (because I have to worry about it!) Thought that might be a good example for you who are looking for actual uses that aren't all just audio/video. they've got some great whitepapers on their site regarding decimalization and their network setup, as well. take a look if you get a chance.
EOM
And that's in an office using Ghost. ;0
Multicast can be tailored to not smash your network, but the only reason this is fairly effective in the office is because it's not lossy
becuase with multicast, there is no flow control?
And no guaranteed delivery?
This is perfectly acceptable for media broadcast, where the codec can deal with dropped bytes.. but..
I work for a company that uses multicast all over the place on a satellite based network, and we run into this all the time. The Mbone or multicast backbound is an adhoc network, built on top of the traditional internet, generally by building dvmrp tunnels between multicast enabled sites, allowing them to exchange traffic. More and more service providers are multicast enabling their network (this is generally a major pain in the a** depending on the equipment that you already have), because of the savings in bandwidth that can be achieved.
But I digress, Onto the question. What you need is a reliable multicast protocol. Most of these are based on a FEC(Forward Error Correction) scheme for making sure that all of the data gets to the intended procedure. You have programs like kencast that enable you to do this( there are others, but lately we are working on creating our own..) . One of the other posts suggested a Multicast version of TCP/IP. This makes no sense as TCP is a connection orientated protocal, whereas multicast is based on udp addressed to a class D internet address (224.0.0.0-239.255.255.255).
Any way hope that helps...
-doon
To E-mail me, replace the first period in my domain with an @
That's an acronym of DOS that would never have occured to me... :+)
Oh, I dunno - Descent Operating System possibly - although I tend to use mine as a DOOM2 operating System
--
-=DaveHowe=-
|-| == bandwidth to move it.
|----| == tarball size. 80s
|--| == bandwidth to move it.
|--------| == tarball size. 90s
|----| == bandwidth to move it.
Over the last 30 years of computing, we've always had more program than pipe to push it through. The only way to overcome this slowly increasing speed of affordable bandwidth, is to pay big bucks for a line that will be outdated in a few years.
In the not to distant past, todays game emulator ROMs used to be moved across the country in a game console containing a huuuuge amount of graphic hardware,. For the day, having a game that totaled more than 1 Meg in run time size was just gigantic, and now, we zip these same ROM images around the net in seconds.
In the very near future IP6 over Multi-Gps fixed wireless will make mirroring linus' balls of tar a trivial task. but, of course, by then, the "kernel" will be 200G ;).
The lesson here is that affordable bandwidth, slowly and stedily has increased over the history of computing, and I see no reason why it should jump.
The make system could still be aware of the other pieces you don't have so if you ever do need to grab some other stuff then it will tell you exactly code to download. If the kernel gets to 200G, this will be a necessity.
See http://www.lucent.com/press/0597/970506.bla.html
The client could also be smart and once it has most of a file, only tune into channels when the bits it is missing are "playing". As it gets towards the end, it could even get more agressive and tune into faster channels to get the chunks it needs (at the expense of your other bandwidth)
Rich
This is a VERY powerful system imager
http://systemimager.sourceforge.net/
The page says "Now supports: ext2, ext3, reiserfs, use without DHCP, and more!"
Worthwhile guys!
Ever need an online dictionary?
I was always under the impression that MBONE was designed for the types of communications that you can lose some data and still be ok. Streaming media... Files don't fall into this catagory. Lose a few bytes of a digital tv signal, and its nor really noticeable, maybe for a few seconds it is... Audio, you might hear some fuzz. Your kernel and binaries on the other hand... A little network congestion and all of the sudden you can't get online to fix the problems you just made.
Eh...
Multicast works by having the router duplicate and forward the packet to vaious designation from one source. The source is continously sending packet, regardless whether the packets has been properly received (acknowledged) by the listeners.
So if the server sends 10,000 packet for the whole file and you loose packet number 5,555... you'll probably have to wait till the next round packet number 5,555 comes. The server isn't gonna to resend the packet as is a direct TCP connection. This is tolerable in video & sound as you'll only loose a second or two. But for file, you can't loose too many bytes before it becomes unusable.
A solution for this is that the listener try to absorb as many bytes as possible during the duration of the multicast, and later initiate a TCP connection to request for the packets which the listener missed.
Gary Cho
In an optimised situation surely mbone would have some sort of star topology where the data is only transmitted once across the atlantic and then only once to each country and distributed from that countries main internet exchange. Am i right?
Anyway whether that's the case or not, there still must be an awful lot of excess data floating about if you only want to mirror the file onto say 100 sites. Given that people are also talking about tunnelling mbone connections too - it makes me cringe.
Point blank problem is that redhat dont have enough bandwidth.
Perhaps a better system of conventional ftp mirroring would be the answer. First of all put the file onto a restricted ftp server at redhat.com. Then only allow each contitenents fastest two servers to pull it, they can then distribute it to smaller ones, and finally (like 3 hours later) it's open to the world.
That way i'd probably find that Sunsite at imperial college london grab the file from redhat, my isp download it to their local mirror archive and then i can pull it over my cable modem at 55kbytes/s.
Most multicast-native customers that make use of the MBONE have quite a bit bandwidth to toss around for video and data broadcasts, or it is part of their business model. (broadcast.com, NASA JPL, US DOE, etc.)
Now in regards to software distribution, it would not be feasible for RedHat to multicast a 600Mb ISO using the Internet multicast backbone as each provider that wanted access to that data would also subject their providers, and their providers providers' to receiving that data as well. So essentially you would have 600Mb flying through 6 transit networks to reach you. Imagine the waste of bandwidth. Do you think multicast providers would take this with an enthusiastic grin?
Currently, there are a few providers that use multicast for stream distribution to multiple servers on live events. You can be assured this is the case for large scalable video distribution houses like broadcast.com, possibly Akamai and others. Hope that provides some insight. I'm not an expert, I've just been performing a lot of multicasting research as of late. Cheers.
The data then branches off from there. This would be quite suitably for updating mirror sites, since
One problem I could see is that this method of distribution for data files (versus video and audio) wouldn't scale well. Imagine one site drops a packet. Well it can't very well start over, since that same packet did possibly reach all the other listening parties. They are all expecting the NEXT packet, not a retransmit.
On an fast, fault tolerant network (major backbones, and obviously intranets) this works great (we use ImageCast at work to simulcast drive images to multiple systems) the bandwidth used is no more than if a single system was done one at a time. But on any network where packet loss and latency are a problem, thing would seriously hamper to practibility of the system.
So I say, multicast to a few hundred major FTP mirrors from the master server (redhat in this case), and then good ol' traditional FTP from there.
For a good survey of the the past, present, and future of interdomain multicast routing, try this paper.
As others have mentioned, yes, flow control is a problem, and, yes, reliability is a problem. There are many solutions to both problems that have been researched (largely in academia), but flow control can't really be solved if you're trying to distribute RedHat over multicast to half a million people who are on disparate links. Someone a few posts up mentioned the Digital Fountain idea, but neglected to provide a link. Digital Fountain aims to solve the problems that are being discussed here for exactly the kinds of applications that are being discussed here. The paradigm is that random bytes are constantly flowing from the fountain (the multicaster) and the recipients fill up their buckets with the random bytes until a file is formed. Read their papers for a more mathematically rigorous explanation...
... and published on Doctor Dobb's Journal (yes, the source code is available). See http://www.ddj.com/articles/2000/0005/0005i/0005i. htm
The guy who wrote it works for Microsoft (so of course his implementation is Windows-dependant) but he makes some pretty good points on using multicast for file distribution, and naturally the idea and/or algorithms could be reimplemented in some D-O-S (Decent Operating System) like Linux...
Best Regards,
--
Durval Menezes.
Best Regards,
Durval Menezes.
I have never met a computer that didn't like me.
- Download the 0.3.1 release, or check out the CVS.
- Modify your rc.local script (or equivalent) to run the node on boot. No, this in not optional!
- Use rar to break the ISO into a reasonable number of segments. I'd recommend a size of 10 megabytes each.
- Insert each file as a CHK (in bash):
- Insert the parts.txt file into freenet under the keytype of your choice. KSKs are human readable, but you have the option of using a SVK or SSK subspace, both of which are cryptographically signed by the publisher. Specifically, consider using an SSK if you intend to insert many related files.
Requesting your data is simple. You first request the list of parts -- KSK@my_inserted_data. You then request each of the parts in the list, concurrently if you've got the bandwidth! Obviously, if you need to request 50 files, you're going to want to write a simple script to do it.#for PART in *.rar ; do echo "$(./freenet_insert CHK@ $PART 2>&1 | grep "Inserted Key" | cut -b 18-) $PART" >>parts.txt ; done
This command will insert all the .rar files into Freenet and record their keys to parts.txt.
An example of inserting a KSK: #./freenet_insert KSK@my_inserted_data parts.txt
Say you only request 25 files at once, and they all come in at 3 kilobytes/sec (everyone else has a 56k modem). You thus download the file at 75kbyte/sec! Of course, most nodes run on fatter pipes, so speeds will be even better.
And did I mention that the files cache and mirror themselves automatically as demand increases?
I teach classes on NT and various Unices. At the end of the week, all the machines have to be rebuilt. On NT, Norton's Ghost does a speedy multicast of a disk image. This is handy, so I've started writing an analous program for Unix. The plan is that it would be on a boot floppy. It is still new, but coming along nicely. The GPL'd source is at http://www.projectmanagersmba.com/casper.tar