Slashdot Mirror
Experimenting w/ High Performance Computing and Multicasting?
jessemersonuy asks: "Multicasting plays an important role in the design, development, operation, and application of next generation networks that rely on the efficient delivery of packets to their destinations. Due to the advent of broadband, wireless and web-based system design technologies, it has become possible and feasible to design and construct large scale, heterogeneous and complex wireline and wireless communication networks that can support
multimedia conferencing, streaming media distribution,
distributed data sharing, distance learning, 'push'- oriented, and QoS for wired-cable and wired-wireless applications. Now, we have a small High Performance Computing system in our university campus and I would really like to use that system for testing multicasting applications. However, I do not know what would be the best way to use the cluster for multicasting purposes. Has anyone experimented with this before? What might be the best multicasting application to use to be able to fully utilize the power of the cluster?"
Jesus H. Christ in a bucket, let's deconstruct what this idiot "Asks Slashdot".
>creamofsomeyoungguy asks:
> "Multicasting plays an important role in the design, development, operation,
>and application of next generation networks that rely on the efficient
>delivery of packets to their destinations.
We want to multicast.
> Due to the advent of broadband, wireless and web-based system design technologies,
Because the internet exists,
> it has become possible and feasible to design and construct large scale,
>heterogeneous and complex wireline and wireless communication networks
>that can support multimedia conferencing, streaming media distribution,
>distributed data sharing, distance learning, 'push'- oriented, and QoS
>for wired-cable and wired-wireless applications.
we can hook different computers together and do lots of cool things.
> Now, we have a small High Performance Computing system in our university campus
>and I would really like to use that system for testing multicasting applications.
I know of a computer or two I'd like to try multicasting on.
> However, I do not know what would be the best way to use the cluster for
>multicasting purposes. Has anyone experimented with this before?
Anybody have any tips?
> What might be the best multicasting application to use
>to be able to fully utilize the power of the cluster?"
Know of any multicast programs that will bring it to its knees?
C'mon, people, speak English.
I've spent a little time setting up and using multicast networks. I can't necessary say where a cluster would fit into the mix. I'll outline a couple of the uses we have found/use multicasting for, maybe there is something in here that you can get ideas to use your cluster for:
1. Music/sound broadcast - using a MP3 streamer you can pump stations out on multicast addresses then using mp3 players that support multicast streams you can latch on and listen to the stream. For business purposes this might be to distribute recordings of company meetings or perhaps training information that does not need video.
2. Video - The setup of stations to re-run important videos of events, etc. can be useful. In most cases these are just like TV statations - you join a multicast stream then you see whatever is on that stream at the curren time. Good for just about anything that you would like to rebroadcast to a large number of users using minimal bandwidth.
3. Applications - Under windows there are a couple applications available that will distribute applications across a campus (or any multicast network) to remote users. Broadcast once, receive many. I haven't used this function much but for file distribution that happens on a schedule to the desktop or to multiple servers it could be useful. Not sure of what support is available under Linux for this currently.
4. Video Conferencing - This is one of the applications that we have been experimenting using multicasting for. Being able to broadcast one video conference to say, 80 different sites but in a network only having to distribute a small number of streams is great stuff. An example of this is we have customers who have say 50 sites, all of the sites are connected via sattelite, if we streamed to all 50 sites it would consume a massive amount of bandwidth (say 768k x 60) - using multicasting we can cut that to one transmission that every site receives (only 768k). There are some other rules - like the returning data must follow a unicast path, etc. but overall the bandwidth used is still around 1 T1 instead of the 25+ it would normally take to do this type of streaming.
Someone else stated that this is largely a router/switch thing. It really is, proper configuration of the multicast network at on the routers/switches in the network is crucial. If you are going to be distributing across a large network you need to choose the proper protocols for your needs (sparse or dense mode protocols) and setup the routing so everything flows across the links you desire. A misconfiguration of the equipment could mean that you end up distributing the same data across multiple links to the same site, not distributing the data at all or having it traverse the wrong link which could end up destroying network performance rather than enhancing it.
-Alan
Multicasting doesn't really equate with needing high-end hardware. In fact, the requirements would be much lower for multicasting, as opposed to having to do the equivalent number of unicasts.
Multicasting has more to do with switch and router configuration.
I've also been interested in mbone (multicasting backbone) for a while since I got broadband access. It's seems simple enough by installing a special virtual router on my firewall (mrouted).
However I have not been able to find any way to hook up my home router to the actual mbone network.
Are there any official contacts for the mbone network that can assist in finding a connection point?
//Pingo
--- Linux or FreeBSD, it's like blondes or brunettes. I like both. ---
The whole point of multicast is that it requires very little in terms of resources. Having a high performance computing cluster isn't going to do much if you're trying to test out multicast.
More likely, you'll want a sniffer, or else access to routers, either directly or through SNMP. Because what you're really going to want to find out is bandwidth utilization. That's where you're going to see gains going with multicast.
You should head over to IETF's website and start looking at RFCs about multicast. RFCs are usually boring to read, but very insightful.
A better question to ask the Slashdot crowd would be: "Which multicast protocol should I be using?". (FWIW, I recommend going with PIM sparse-mode. See also SSM.) In terms of apps, you haven't really told us much... Are you looking for UNIX, Mac, NT? Audio, video, text, file transfer? High bandwidth, low bandwidth, reliable transport? Without this information, the question posed is so open as to be unanswerable.
furthermore, multicasting has fuck all to do with servers. multicasting is accomplished within a routing/switching infrastructure. the server sends out a SINGLE data stream to a SINGLE SPECIAL ip address; this SINGLE SPECIAL ip address actually identifies a GROUP of hosts. routers & switches know which UNIQUE hosts belong to said GROUP and selectively forward the data. the whole point of this is to reduce load on the server, and prevent multiple identical data streams on the network infrastructure.
k /i cs/cs011.htm
a few notes from cisco :
http://www.cisco.com/univercd/cc/td/doc/cisintw
"Multicast---Multicast applications send each packet to a multicast group address. Hosts that want to receive the packets indicate that they want to be members of the multicast group. This type of application expects that networks with hosts that have joined a multicast group will receive multicast packets. Multicast applications and underlying multicast protocols control multimedia traffic and shield hosts from having to process unnecessary broadcast traffic."
" IP multicasting applications use Class D addresses to address packets. The high-order four bits of a Class D address are set to 1110, and the remaining 28 bits are set to a specific multicast group ID. Class D addresses are typically written as dotted-decimal numbers and are in the range of 224.0.0.0 through 239.255.255.255."
" The Internet Group Management Protocol (IGMP) uses IP datagrams to allow IP multicast applications to join a multicast group. Membership in a multicast group is dynamic---that is, it changes over time as hosts join and leave the group.
Multicast routers that run IGMP use IGMP host-query messages to keep track of the hosts that belong to multicast groups. These messages are sent to the all-systems group address 224.0.0.1. The hosts then send IGMP report messages listing the multicast groups they would like to join. When the router receives a packet addressed to a multicast group, it forwards the packet to those interfaces that have hosts that belong to that group. If you want to prevent hosts on a particular interface from participating in a multicast group, you can configure a filter on that interface by using the ip igmp access-group interface configuration command."
Install Napster across the network.
--
--
#nohup cat
First I would not call 8 Athlon 650 machines a "High Performance Computing system", also if you check all the machines are plugged into a singe 24-port switch. Where does the multi-casting come into play? Your testbed was designed for distributed computing, not mult-casting. I have to assume you just started in your lab. Go talk to your professor and read up on the material before asking for Slashdot to help you out.
There is plenty of room for making good use of
multicasting in high performance computing.
However, most clusters really don't use it
because either A) The switches don't support
it (other than 1-to-all) or B) TCP/IP doesn't
support it.
The message passing libraries probably being used on your cluster (either PVM or some variant of an MPI implementation probably have multicasting statements, but they are in actuality probably
implemented as a sequence of normal one-one communications. Now if you wanted to implement
your own simulated multicast in the cluster, where
you used some kind of tree structure to forward
messages to all the right nodes, have at it,
but you'll probably have to modify your message
passing library...
Dan
As gets mentioned here from time to time, Digital Fountain addresses (or endeavors to address) the "recipients in sync" problem.
I'd be surprised if multicasting ever comes into very wide use; since the situations that it's useful under are limited...
Where I work, many of us kids like to listen to streaming radio broadcasts. We've been criticized for the strain we put on our Internet connection, and it's a valid point. Often several of us are listening to the same shoutcast stream (or whatever) at the same time, and it seems kinda silly that we consume N * 50-100kbps of bandwidth to receive the same content. But, hey, this is just a personal way in which multicast could help my life.
When most people think of multicast they think of 1-to-many transmission. There are also lots of applications involving many-to-many transmission. Chat is an obvious one; chat becomes particularly well suited for multicast when you're dealing with voice chat rather than text. A more interest application is in networked virtual environments (less grandiloquently, games). A couple other fellows and I wrote the networking part of an NVE that used many-to-many multicast: the world was partitioned into octrees, and each octree was assigned a multicast group. Octree nodes split and merged based on traffic, and there were different levels of groups for messages of different levels of detail (e.g., toe movements vs. explosions). (Well, this was the plan; we didn't finish all of it, but it was a cool demo). Peer-to-peer NVEs have many advantages over client-server systems, including reduced (and hopefully optimally minimal) latency and natural scalability. This book provides an overview of the subject, but there are many papers out there that are more in-depth and informative.
Finally, check out Kevin Almeroth's research in multicast applications. He has several good survey papers that address your synchronized play out gripe and explore the gamut of potential multicast applications.