Slashdot Mirror
Low Cost Webcast Optimizations?
ChunKing asks: "I work for a small community broadcasting organization, and we operate a limited streaming media facility for a number of not-for-profit webcasters. It has always been an issue to optimize our streaming media infrastructure to most benefit our users. We operate a small cluster of servers from a data center with good connectivity and a highly-rated ISP, who will occasionally allow us to burst to unlimited bandwidth. For big webcasts, we will load balance the stream over a number of servers using round robin DNS. However, we still get problems with stream buffering and network drop-outs, particularly with streaming video. We cannot afford a network of edge delivery servers like Akamai, so in what ways can we further optimize our streaming media capacity to better produce smooth webcasts?"
"who will occasionally allow us to burst to unlimited bandwidth"
Well, unless your provider has "unlimited bandwidth" themselves, you're basing your webcast on a lie. Pretty much no one has unlimited bandwidth; even a couple hundred broadband streams can saturate an OC-3 connection.
What you need to do is plan your webcasts a little better and ask a bunch of questions: What's the real bandwidth your ISP can provide (with redundancy)? What's the buffer size that your client apps are using? (Settable in some clients, like Flash.) Does your ISP (who promised you unlimited bandwidth) even know what the hell they're doing?
Without going to a dedicated CDN like Akamai, it's still pretty easy to design a distributed service network with colocated servers scattered across the country. You might want to consider finding someone who knows this kind of thing and paying them a few bucks to fix your problems...
Streaming? Are you for real, that is so 1997. Just more old media companies trying to push their way of doing things on the net. Give up on streaming. Produce several quailty versions of the same file support a few differnt codecs mp4, xvid, wmv... and setup torrents for each file. Then use your limited bandwidth to deliver the torrent files, run the tracker and with the remaining bandwidth seed the torrents with a good bittorrent client. Why suffer quailty and bandwidth requirements just so the show can be 'live'? I suspect you arn't even showing live events so why streaming? Stop thinking like a TV broadcaster and start thinking like an internet distributer.
You mention a lot of factors, but you haven't said what area you're having the most problem with. Have you at least managed to identify what the actual bottleneck is? It could be bandwidth at your servers, or between your ISP and some peer (through which most of your traffic is going), or it could be CPU load on your servers, or it could be disk I/O limits on your servers, or it could be lack of memory on your servers (causing them to thrash), or it could be uneven load balancing to DNS-based load balancing being somewhat random, or it could be that you've reached the maximum capacity (in terms of hosts per response record in DNS) with DNS load balancing and you need to add another layer of load balancing.
The first step is going to be to check into all those areas and identify where the failure is currently happening. Once you know that, the solution will be more obvious. You might just need to add striping for your disks (to increase throughput by brute force) or RAM (to be able to cache all active streams) if the problem is that your disks can't keep up. But, if your ISP's connection to a major peer isn't sufficient, then you will need to do something much more involved (and will probably include gathering data during an actual event where the streaming can't keep up in order to prove that the ISP is the problem).
Stay away from CDNs if you're looking for performance (if it's pricing you're after, that's a seperate issue) - they are likely dealing with all the same issues that you are. Except that when something goes wrong, there's nothing you can do except shrug and say "it's the CDN's fault". My organization went to a CDN to save on ridiculous colo bills, and we heard all the claims about performance with skepticism, but we were shocked to find that despite all the "second generation edge servers" our CDN has, thruput actually went DOWN significantly when we switched to them. We didn't much care since it was saving us a lot of money and was usually unnoticeable or at least bareable...
:)
But to return to your original question - since you're dealing with not-for-profits, clearly the answer is a live streaming P2P "edge network". Let me know when you release that code
If you don't have the development resources for that, the best solution is probably to scale down your investment in your current colo and pop a few servers in a different geographic locale, a kind of DIY edge network. With 3 or 4 colos (say, Europe, US-East, US-West), and smart determination of which host to route to using Geo::IP or the like, you could probably save a lot of traffic and latency.
Upload video to Google video or YouTube.
Use their bandwidth to stream your video.
Use their API to embed the video on your website.
You lose control over the distribution of the content, but you save a lot of money in the process. The other option is Bit torrent.
Also, have you looked into DigitalBicycle? They are working on a PeerCasting system for Using Bit Torrent, RSS, and web community software
This problem has been solved already. It's called Ultravox. It's a protocol that is format-agnostic and is used to efficiently multicast a broadcast within the data center up to the point it leaves your network. America Online and Cisco designed and implemented the protocol both in software and in the firmware of specialized Cisco routers and it is used for AOL Radio and for Winamp radio It's very interesting. All your viewers need is Winamp to hear or view your program.
http://ultravox.aol.com/
Kriston
It sounds like your problem is bandwidth. Either on the client-side, coming off your net to your ISP, or routing problems elsewhere on the Internet. The hard part is tracking down where the bottleneck is. Try to have multiple end-user test points. The company I work for has employees on a couple of different local ISPs and servers at various colo facilities. Try logging in to each machine and testing the stream. If it's across the board, you can safely bet it's your "unlimited" pipe to your ISP or a problem at your ISP. If it's a problem with a fraction of the clients, try doing traceroutes and see how the packets hop across the Internet and look for any similarities to see where the hang-ups can be. It's also possible that some clients just aren't getting the bandwidth they think they should be getting. Try everything you can to optimize your streaming application of choice. Reduce the number of FPS if you can. If it's only a talking head or something with little movement, try spacing out the keyframes more. Unless you're broadcasting a music video or movie, try using mono instead of stereo for audio. If it's a speech, you probably don't need CD-quality sound. Dropping the sound to 32, 24, or even 16kbps for a WMA/MP3/RA stream will still sound decent for something that doesn't have a lot of variation in sound.
Also, if it is possible, try to expirament around with variable bit-rate streams for video and/or audio. That can come in handy if your target audience spreads from AOL dial-up in the middle of nowhere to your urban 8 Mbps+ broadband connections.
In streaming, the big three big things that you have to worry about are the encoder's connectivity (to push the stream to the distribution point), the encoder's power (to make sure it can really compress all that video without overheating or dropping frames), and the distribution point's connectivity. Distribution servers don't use much bandwidth as all they do is simply regurgitate the data back out to clients without having to do any processing other than logging.
I've been doing streaming for about 5-6 years now. I've done everything from web cam streaming to large convention streaming that put out 300+ Mbps of connectivity. By no means am I an expert (I've done streaming--it's not what I do), but everything above is my experience with it. Most of all, just stick with what you know and are comfortable with. Your client base will determine most things. Personally (even being a BSD guy), my preferred streaming setup is using Windows Media Encoder at the source, Windows Media Server 2003 at the distribution point, and Windows Media Player (or anything else that can play Windows Media v9) at the end-user. In my experience, it has been rock-solid and can be deployed quickly. If you're already running Windows 2000/XP/2003, all the software mentioned above is provided free-of-charge.
Hope this helps.