Slashdot Mirror
Experimenting w/ High Performance Computing and Multicasting?
jessemersonuy asks: "Multicasting plays an important role in the design, development, operation, and application of next generation networks that rely on the efficient delivery of packets to their destinations. Due to the advent of broadband, wireless and web-based system design technologies, it has become possible and feasible to design and construct large scale, heterogeneous and complex wireline and wireless communication networks that can support
multimedia conferencing, streaming media distribution,
distributed data sharing, distance learning, 'push'- oriented, and QoS for wired-cable and wired-wireless applications. Now, we have a small High Performance Computing system in our university campus and I would really like to use that system for testing multicasting applications. However, I do not know what would be the best way to use the cluster for multicasting purposes. Has anyone experimented with this before? What might be the best multicasting application to use to be able to fully utilize the power of the cluster?"
Jesus H. Christ in a bucket, let's deconstruct what this idiot "Asks Slashdot".
>creamofsomeyoungguy asks:
> "Multicasting plays an important role in the design, development, operation,
>and application of next generation networks that rely on the efficient
>delivery of packets to their destinations.
We want to multicast.
> Due to the advent of broadband, wireless and web-based system design technologies,
Because the internet exists,
> it has become possible and feasible to design and construct large scale,
>heterogeneous and complex wireline and wireless communication networks
>that can support multimedia conferencing, streaming media distribution,
>distributed data sharing, distance learning, 'push'- oriented, and QoS
>for wired-cable and wired-wireless applications.
we can hook different computers together and do lots of cool things.
> Now, we have a small High Performance Computing system in our university campus
>and I would really like to use that system for testing multicasting applications.
I know of a computer or two I'd like to try multicasting on.
> However, I do not know what would be the best way to use the cluster for
>multicasting purposes. Has anyone experimented with this before?
Anybody have any tips?
> What might be the best multicasting application to use
>to be able to fully utilize the power of the cluster?"
Know of any multicast programs that will bring it to its knees?
C'mon, people, speak English.
I've spent a little time setting up and using multicast networks. I can't necessary say where a cluster would fit into the mix. I'll outline a couple of the uses we have found/use multicasting for, maybe there is something in here that you can get ideas to use your cluster for:
1. Music/sound broadcast - using a MP3 streamer you can pump stations out on multicast addresses then using mp3 players that support multicast streams you can latch on and listen to the stream. For business purposes this might be to distribute recordings of company meetings or perhaps training information that does not need video.
2. Video - The setup of stations to re-run important videos of events, etc. can be useful. In most cases these are just like TV statations - you join a multicast stream then you see whatever is on that stream at the curren time. Good for just about anything that you would like to rebroadcast to a large number of users using minimal bandwidth.
3. Applications - Under windows there are a couple applications available that will distribute applications across a campus (or any multicast network) to remote users. Broadcast once, receive many. I haven't used this function much but for file distribution that happens on a schedule to the desktop or to multiple servers it could be useful. Not sure of what support is available under Linux for this currently.
4. Video Conferencing - This is one of the applications that we have been experimenting using multicasting for. Being able to broadcast one video conference to say, 80 different sites but in a network only having to distribute a small number of streams is great stuff. An example of this is we have customers who have say 50 sites, all of the sites are connected via sattelite, if we streamed to all 50 sites it would consume a massive amount of bandwidth (say 768k x 60) - using multicasting we can cut that to one transmission that every site receives (only 768k). There are some other rules - like the returning data must follow a unicast path, etc. but overall the bandwidth used is still around 1 T1 instead of the 25+ it would normally take to do this type of streaming.
Someone else stated that this is largely a router/switch thing. It really is, proper configuration of the multicast network at on the routers/switches in the network is crucial. If you are going to be distributing across a large network you need to choose the proper protocols for your needs (sparse or dense mode protocols) and setup the routing so everything flows across the links you desire. A misconfiguration of the equipment could mean that you end up distributing the same data across multiple links to the same site, not distributing the data at all or having it traverse the wrong link which could end up destroying network performance rather than enhancing it.
-Alan
furthermore, multicasting has fuck all to do with servers. multicasting is accomplished within a routing/switching infrastructure. the server sends out a SINGLE data stream to a SINGLE SPECIAL ip address; this SINGLE SPECIAL ip address actually identifies a GROUP of hosts. routers & switches know which UNIQUE hosts belong to said GROUP and selectively forward the data. the whole point of this is to reduce load on the server, and prevent multiple identical data streams on the network infrastructure.
k /i cs/cs011.htm
a few notes from cisco :
http://www.cisco.com/univercd/cc/td/doc/cisintw
"Multicast---Multicast applications send each packet to a multicast group address. Hosts that want to receive the packets indicate that they want to be members of the multicast group. This type of application expects that networks with hosts that have joined a multicast group will receive multicast packets. Multicast applications and underlying multicast protocols control multimedia traffic and shield hosts from having to process unnecessary broadcast traffic."
" IP multicasting applications use Class D addresses to address packets. The high-order four bits of a Class D address are set to 1110, and the remaining 28 bits are set to a specific multicast group ID. Class D addresses are typically written as dotted-decimal numbers and are in the range of 224.0.0.0 through 239.255.255.255."
" The Internet Group Management Protocol (IGMP) uses IP datagrams to allow IP multicast applications to join a multicast group. Membership in a multicast group is dynamic---that is, it changes over time as hosts join and leave the group.
Multicast routers that run IGMP use IGMP host-query messages to keep track of the hosts that belong to multicast groups. These messages are sent to the all-systems group address 224.0.0.1. The hosts then send IGMP report messages listing the multicast groups they would like to join. When the router receives a packet addressed to a multicast group, it forwards the packet to those interfaces that have hosts that belong to that group. If you want to prevent hosts on a particular interface from participating in a multicast group, you can configure a filter on that interface by using the ip igmp access-group interface configuration command."
As gets mentioned here from time to time, Digital Fountain addresses (or endeavors to address) the "recipients in sync" problem.
I'd be surprised if multicasting ever comes into very wide use; since the situations that it's useful under are limited...
Where I work, many of us kids like to listen to streaming radio broadcasts. We've been criticized for the strain we put on our Internet connection, and it's a valid point. Often several of us are listening to the same shoutcast stream (or whatever) at the same time, and it seems kinda silly that we consume N * 50-100kbps of bandwidth to receive the same content. But, hey, this is just a personal way in which multicast could help my life.
When most people think of multicast they think of 1-to-many transmission. There are also lots of applications involving many-to-many transmission. Chat is an obvious one; chat becomes particularly well suited for multicast when you're dealing with voice chat rather than text. A more interest application is in networked virtual environments (less grandiloquently, games). A couple other fellows and I wrote the networking part of an NVE that used many-to-many multicast: the world was partitioned into octrees, and each octree was assigned a multicast group. Octree nodes split and merged based on traffic, and there were different levels of groups for messages of different levels of detail (e.g., toe movements vs. explosions). (Well, this was the plan; we didn't finish all of it, but it was a cool demo). Peer-to-peer NVEs have many advantages over client-server systems, including reduced (and hopefully optimally minimal) latency and natural scalability. This book provides an overview of the subject, but there are many papers out there that are more in-depth and informative.
Finally, check out Kevin Almeroth's research in multicast applications. He has several good survey papers that address your synchronized play out gripe and explore the gamut of potential multicast applications.