Slashdot Mirror
Low Cost Webcast Optimizations?
ChunKing asks: "I work for a small community broadcasting organization, and we operate a limited streaming media facility for a number of not-for-profit webcasters. It has always been an issue to optimize our streaming media infrastructure to most benefit our users. We operate a small cluster of servers from a data center with good connectivity and a highly-rated ISP, who will occasionally allow us to burst to unlimited bandwidth. For big webcasts, we will load balance the stream over a number of servers using round robin DNS. However, we still get problems with stream buffering and network drop-outs, particularly with streaming video. We cannot afford a network of edge delivery servers like Akamai, so in what ways can we further optimize our streaming media capacity to better produce smooth webcasts?"
(This probably won't be helpful to your present problem, unless something along this line of thought already exists.)
I wonder how well a method such as bittorrent adapted to streaming data would do - i.e. decentralized streaming.
The first few viewers receive the stream directly from the source, while acting as mirrors for subsequent viewers. It would dramatically cut down on the server load & overhead for the broadcasters.
"who will occasionally allow us to burst to unlimited bandwidth"
Well, unless your provider has "unlimited bandwidth" themselves, you're basing your webcast on a lie. Pretty much no one has unlimited bandwidth; even a couple hundred broadband streams can saturate an OC-3 connection.
What you need to do is plan your webcasts a little better and ask a bunch of questions: What's the real bandwidth your ISP can provide (with redundancy)? What's the buffer size that your client apps are using? (Settable in some clients, like Flash.) Does your ISP (who promised you unlimited bandwidth) even know what the hell they're doing?
Without going to a dedicated CDN like Akamai, it's still pretty easy to design a distributed service network with colocated servers scattered across the country. You might want to consider finding someone who knows this kind of thing and paying them a few bucks to fix your problems...
... at least for the most part. Encourage your viewers to download instead of stream, if at all possible.
Streaming? Are you for real, that is so 1997. Just more old media companies trying to push their way of doing things on the net. Give up on streaming. Produce several quailty versions of the same file support a few differnt codecs mp4, xvid, wmv... and setup torrents for each file. Then use your limited bandwidth to deliver the torrent files, run the tracker and with the remaining bandwidth seed the torrents with a good bittorrent client. Why suffer quailty and bandwidth requirements just so the show can be 'live'? I suspect you arn't even showing live events so why streaming? Stop thinking like a TV broadcaster and start thinking like an internet distributer.
Is it vitally important that they have to stream the data? You cannot just distribute downloads and Coral Cache em? Cut my bandwidth for files from 120GB/mo to ~15GB/mo
Your hair look like poop, Bob! - Wanker.
Multicast.
You mention a lot of factors, but you haven't said what area you're having the most problem with. Have you at least managed to identify what the actual bottleneck is? It could be bandwidth at your servers, or between your ISP and some peer (through which most of your traffic is going), or it could be CPU load on your servers, or it could be disk I/O limits on your servers, or it could be lack of memory on your servers (causing them to thrash), or it could be uneven load balancing to DNS-based load balancing being somewhat random, or it could be that you've reached the maximum capacity (in terms of hosts per response record in DNS) with DNS load balancing and you need to add another layer of load balancing.
The first step is going to be to check into all those areas and identify where the failure is currently happening. Once you know that, the solution will be more obvious. You might just need to add striping for your disks (to increase throughput by brute force) or RAM (to be able to cache all active streams) if the problem is that your disks can't keep up. But, if your ISP's connection to a major peer isn't sufficient, then you will need to do something much more involved (and will probably include gathering data during an actual event where the streaming can't keep up in order to prove that the ISP is the problem).
Stay away from CDNs if you're looking for performance (if it's pricing you're after, that's a seperate issue) - they are likely dealing with all the same issues that you are. Except that when something goes wrong, there's nothing you can do except shrug and say "it's the CDN's fault". My organization went to a CDN to save on ridiculous colo bills, and we heard all the claims about performance with skepticism, but we were shocked to find that despite all the "second generation edge servers" our CDN has, thruput actually went DOWN significantly when we switched to them. We didn't much care since it was saving us a lot of money and was usually unnoticeable or at least bareable...
:)
But to return to your original question - since you're dealing with not-for-profits, clearly the answer is a live streaming P2P "edge network". Let me know when you release that code
If you don't have the development resources for that, the best solution is probably to scale down your investment in your current colo and pop a few servers in a different geographic locale, a kind of DIY edge network. With 3 or 4 colos (say, Europe, US-East, US-West), and smart determination of which host to route to using Geo::IP or the like, you could probably save a lot of traffic and latency.
Upload video to Google video or YouTube.
Use their bandwidth to stream your video.
Use their API to embed the video on your website.
You lose control over the distribution of the content, but you save a lot of money in the process. The other option is Bit torrent.
Also, have you looked into DigitalBicycle? They are working on a PeerCasting system for Using Bit Torrent, RSS, and web community software
For video? I'd recommend XviD, and not just because I use it almost exclusively, but it creates a compliant MPEG-4 video stream (compresses nicely), which will hopefully be streamable by any player that supports MPEG-4 video.
Akamai is a premium priced streaming provider, there are many others, most of whom will do what you need at a much lower cost. We are currently streaming live streams (live television channels) with akamai, vitalstream, limelight, and others. We are pulling the seed streams from the middle east and south asia and broadcasting to thousands of users over a very wide geographic area. We have load tested to the 10s of thousands of users and none of the CDNS have even broken a sweat. I know of several sites that are streaming 24/7 channels that have literally hundreds of thousands of simultaenous viewers and viewerships ranking in the millions per month. (unique).
l
Do yourself a favor, call vitalstream or limelight they have starter streaming packs which you probably cannot even begin to use up and they run in the hundreds of dollars a month if I remember right.
To everyone that said xvid or any other format. xvid is a file format. he wants LIVE streams.
to everyone that said streaming is dead.. get a grip, iptv is the next television revolution..
http://www.cabledatacomnews.com/feb06/feb06-3.htm
http://ourmedia.org/ is setup for just such occassions. My video podcasts to a 100,000 people in the last month without complaint. CC friendly. And free.
This problem has been solved already. It's called Ultravox. It's a protocol that is format-agnostic and is used to efficiently multicast a broadcast within the data center up to the point it leaves your network. America Online and Cisco designed and implemented the protocol both in software and in the firmware of specialized Cisco routers and it is used for AOL Radio and for Winamp radio It's very interesting. All your viewers need is Winamp to hear or view your program.
http://ultravox.aol.com/
Kriston
Seriously, because ISPs generally do not support multicast (even though their hardware does, it's just disabled, it requires 5 seconds to turn on, and is likely native downstream of them anyway), it is better to do this the way NASA Select did with their TV broadcasts in the days of CU-SeeMe. (Great program, died an ugly death, and is now a zombie in the hands of a corporation.)
What NASA did was to multicast for anyone who could get multicast to receive, but ALSO to reflectors that supported unicast connections. Users without multicast could then hook up to the nearest reflector and get the broadcast with only marginal extra latency.
Because multicast reduces the number of streams (to one), you have the benefit of being able to boost the quality of the transmission. Typical webcasts are maybe 320x200 at 5 seconds a frame. If you're lucky. A typical MBone transmission is very close to NTSC resolution at 15 frames per second, because the users have that much extra bandwidth to play with. That kind of a dramatic improvement in quality draws attention, which means that if this is a payed service (multicasting supports those), you're likely to improve not only the output but the revenue as well.
Because this is a broadcast-only system, the flavour of multicasting most likely to be of interest is SSM (Source-Specific Multicasting) where only designated endpoints (in this case the webcast transmitter) can send, all others receive.
Remember, multicasting is your friend.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Check the load on the servers... that's not your problem. Check your bandwidth pipe. That is your bottle neck. Now just because you might have a 20 MB pipe doesn't mean you have 20 MB to the Internet. Bandwidth providers are famous for overselling pipes... that's how they make money! What you need to look into is getting a second ISP that feeds from a different data center than the one you have now. Now another option is to co-locate to a center that has multiple ISP feeds and place a publishing server at that location. Therefore you only use the bandwidth at your current site to send the feed to the co-location. They will most likely have three to four ISP feeds to the Internet and the cost is a lot less than adding more bandwidth and edge network services. With you being a non-profit you've got to have some leeway to get a better pricing deal.
It sounds like your problem is bandwidth. Either on the client-side, coming off your net to your ISP, or routing problems elsewhere on the Internet. The hard part is tracking down where the bottleneck is. Try to have multiple end-user test points. The company I work for has employees on a couple of different local ISPs and servers at various colo facilities. Try logging in to each machine and testing the stream. If it's across the board, you can safely bet it's your "unlimited" pipe to your ISP or a problem at your ISP. If it's a problem with a fraction of the clients, try doing traceroutes and see how the packets hop across the Internet and look for any similarities to see where the hang-ups can be. It's also possible that some clients just aren't getting the bandwidth they think they should be getting. Try everything you can to optimize your streaming application of choice. Reduce the number of FPS if you can. If it's only a talking head or something with little movement, try spacing out the keyframes more. Unless you're broadcasting a music video or movie, try using mono instead of stereo for audio. If it's a speech, you probably don't need CD-quality sound. Dropping the sound to 32, 24, or even 16kbps for a WMA/MP3/RA stream will still sound decent for something that doesn't have a lot of variation in sound.
Also, if it is possible, try to expirament around with variable bit-rate streams for video and/or audio. That can come in handy if your target audience spreads from AOL dial-up in the middle of nowhere to your urban 8 Mbps+ broadband connections.
In streaming, the big three big things that you have to worry about are the encoder's connectivity (to push the stream to the distribution point), the encoder's power (to make sure it can really compress all that video without overheating or dropping frames), and the distribution point's connectivity. Distribution servers don't use much bandwidth as all they do is simply regurgitate the data back out to clients without having to do any processing other than logging.
I've been doing streaming for about 5-6 years now. I've done everything from web cam streaming to large convention streaming that put out 300+ Mbps of connectivity. By no means am I an expert (I've done streaming--it's not what I do), but everything above is my experience with it. Most of all, just stick with what you know and are comfortable with. Your client base will determine most things. Personally (even being a BSD guy), my preferred streaming setup is using Windows Media Encoder at the source, Windows Media Server 2003 at the distribution point, and Windows Media Player (or anything else that can play Windows Media v9) at the end-user. In my experience, it has been rock-solid and can be deployed quickly. If you're already running Windows 2000/XP/2003, all the software mentioned above is provided free-of-charge.
Hope this helps.
First, I can give you unlimited bandwidth on a Dial up line but you're not going to stream video over it, so any ISP that is giving you "unlimited" probably isn't set up to host your video. ;-)
Second on YouTube and Google
I agree there is something to be said for "out sourcing" your video to a company like Google Video or YouTube, but if you don't mind I would like to throw my little Start-Up (Vidiac.com) into the ring since we've been doing free video hosting longer than either of those two without all the Web 2.0 Hype. Of course the difference with us is we give you a video hosting portal you can run on your website (Like video.YOURDOMAIN.com), so with us you can start your own "Flickr of Video" either free (ad supported) or pay-per stream. Sorry about the self promotion there.
Regarding your Dilemma
Here is my input on the question at hand based on my own experience (I'm streaming just over a million streams a day to 2.5 Million Unique visitors a month). First, as always, start with the needs of the users.
1.) Are they sophisticated? How would they react to a Torrent? From my experience, if someone has a torrent client installed then they're fine with it, but if they don't you have a large "fear factor" to overcome convincing a user to install a program to watch your video.
2.) Do they want to click and Watch, (stream), or do they want to download? If you produce content like a video podcast, then I highly recommend giving your users a download option over streaming. If you run a portal-site, like Big-Boys.com that aggregates content then you're better off streaming as you want to be viewed as more than an FTP site.
3.) How big is your Audience? This is probably the single biggest factor for capacity planning. If they all hit your site at once you need a ton of bandwidth.
4.) Is this a scheduled broadcast (like a weekly show), or is this content that will be downloaded over time? Scheduled events can cause massive spikes. 10,000 users all at once is harder on a server than a Million users spread out over a month.
5.) What video quality will your users tolerate? What Software can they use to view it? Windows Media? Quicktime? Flash? Size? Bit Rate? If your users are watching these videos on an iPod they'll accept a lower quality than if they're viewing it on their Plasma HDTV.
Based on the above you can get a firm handle on your
1.) Delivery Method (Direct Download, stream, Torrent)
2.) Video Size (Format, Quality, etc)
3.) Pipe Size (How much do you need to stream at once)
3.) Bandwidth Utilization profile (spikes and valley vs. flat)
Compare the above to your current solution. If your users are buffering, is it because...
Your upload pipe is saturated? Upgrade to a larger pipe (100Mbps is about the minimum if you have more than 10 people hitting your video at the same time)
Is your Disk I/O saturated? SATA and IDE are fine for most sites, but if you have 50 viewers requesting 30 different videos you need some strong I/O. we were burning through SATA drives on a weekly basis before switching to a SCSI SAN.
Does your ISP have proper Network access to get to your end users? Consider pulling in a pipe from an appropriate network. (AT&T strongly peers with Cox cable for instance, MCI has fantastic AOL connectivity).
Anyhow, I hate to answer a question with more questions, but there are so many ways to deliver video, there is no one silver bullet solution. You just have to do what works for your users and company. From experience though capacity planning for an online video application takes a lot of work, but once you wrap your head around the bandwidth issue, it gets easy. Good Luck!
-Adam
When ever you're in need of a streaming solution look and see what the Pr0n industry is up to. Awhile back Wired ran an article on an adult site who claim to offer P2P streaming Pr0n...great way to save on bandwidth if it works.
First, forget round robin DNS. It simply doesn't work. You cannot rely on the DNS clients to select the round-robin IP addresses in order except under very rare circumstances.
Second, if it absolutely has to work, you can rent services from Akamai, who have a huge and under-used network for exactly this service. Let them deal with distributing the load from one feed to thousands of servers worldwide: they think it will make them lots of money. They will play more clever DNS games than merely round-robining.
Third, forget streaming live if you can. Set up a download site for a Bittorrent feed. If lots of people want to see the download, let them distribute the load among themselves.
Fourth, forget streaming Windows Media. Use Quicktime or Real if you have to stream: they both put much less load on your servers and your server can handle a lot more clients.
Contact the folks at Carnegie Mellon University's "End System Multicast" gang. Its a P2P video streaming sysytem!
It's at http://esm.cs.cmu.edu/
Think again. Three copies means you chew up three times the bandwidth to get the same file out. If you multicast, you can: (a) transmit to all three the same information at the bandwidth of a single person, (b) transmit to all three three times the information at no extra bandwidth, (c) anywhere in between.
The first option basically means you can run a broadcast station and be doing other things on the Internet (surfing, downloading, whatever). Because broadcasts eat a lot of bandwidth, cutting out two redundant copies frees up a lot of the broadcaster's resources. Alternatively, as bandwidth costs money, the broadcaster can now reduce the speed of service they've bought, save money, AND still transmit to their entire audience.
The second option is also an important one. If you're doing a video feed, three times the bandwidth can mean the difference between b/w or full colour, or between low res or high res. For audio (such as Internet Radio operations), the difference can be between 8 bit mono and CD-quality MP4-AAC with full Dolby-like stereo.
The other important consideration is that multicast scales. If you go from an audience of 3 to an audience of 300, unicast uses 100 times the bandwidth. What could be done on a cheapo DSL line now takes a fractional T3 line, after which the broadcaster jumps off a bridge from crippling debt. With multicast, you could go from 3 to 3 million and not increase your bandwidth consumption by so much as a single byte.
(If you're running a mobile Internet broadcasting station, using nearby wifi access points or a nearby phone line plus a 56K modem, you'd better damn well have multicast if you expect to reach an audience of more than one.)
Should every router on the Internet participate in the creation of PIM trees or allow the inclusion of hundreds of thousands of /32 routes under the 224 prefix? They shouldn't.
Dunno where this comes from. If there's nobody downstream in the group, then the router doesn't carry that group. Plain and simple. Routers only send to routers to which there is at least one connection with a user subscribed to that group. Routers do not need to know every single group, they need only know of those groups they are explicitly forwarding. In consequence, multicasting is actually very light on the router requirements. (With IGMPv3, the tree can be further reduced. If more than half of the connections want the stream, then the router can invert the logic and transmit to all EXCEPT for those who don't want it. Routers can also filter by source, if instructed to do so, so the tree doesn't even have to carry excess traffic any more.)
How do I know this for sure? Well, aside from having run multicast routers myself, I know that the Internet backbone has been running native multicast for the better part of eight to ten years. Has the Internet obvious suffered from this? Not that I'm aware of. Sure, there have been partial meltdowns of the Internet, but those are invariably unicast, not multicast sessions. Of those, had multicasting been used instead of unicast, the Internet would have barely noticed the impact.
It sounds to me as though you've been learning about multicasting from some highly uninformed sources who aren't aware of the distinction between a broadcast protocol (where everyhing is sent to everyone) and multicasting (where only specifically requested streams within specifically selected groups are forwarded only to explicitly interested parties).
If that is the case, feel free to point them in my direction and I'll be more than happy to go through in meticulous detail every aspect of multicasting that they could possibly want to know about. And more. The same is true of anyone else on Slashdot - I am more than happy to provide detailed information on any aspect o
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)