Slashdot Mirror
Experimenting w/ High Performance Computing and Multicasting?
jessemersonuy asks: "Multicasting plays an important role in the design, development, operation, and application of next generation networks that rely on the efficient delivery of packets to their destinations. Due to the advent of broadband, wireless and web-based system design technologies, it has become possible and feasible to design and construct large scale, heterogeneous and complex wireline and wireless communication networks that can support
multimedia conferencing, streaming media distribution,
distributed data sharing, distance learning, 'push'- oriented, and QoS for wired-cable and wired-wireless applications. Now, we have a small High Performance Computing system in our university campus and I would really like to use that system for testing multicasting applications. However, I do not know what would be the best way to use the cluster for multicasting purposes. Has anyone experimented with this before? What might be the best multicasting application to use to be able to fully utilize the power of the cluster?"
I have access to broadcast feeds for all major North American, Asian and European exchanges (including options data from OPRA). It averages about 7,500 messages per second during trading hours, and even compresses down to less than a T1 in realtime. Thats hardly going to stress a high speed network. Really badly written and inefficient proprietary market data distribution systems are another matter entirely :-)
A paper on an object based distributed shared memory system using IP multicast. It runs on Linux, IRIX, and SunOS. ftp://munra.com/pub/coraldoc
Jesus H. Christ in a bucket, let's deconstruct what this idiot "Asks Slashdot".
>creamofsomeyoungguy asks:
> "Multicasting plays an important role in the design, development, operation,
>and application of next generation networks that rely on the efficient
>delivery of packets to their destinations.
We want to multicast.
> Due to the advent of broadband, wireless and web-based system design technologies,
Because the internet exists,
> it has become possible and feasible to design and construct large scale,
>heterogeneous and complex wireline and wireless communication networks
>that can support multimedia conferencing, streaming media distribution,
>distributed data sharing, distance learning, 'push'- oriented, and QoS
>for wired-cable and wired-wireless applications.
we can hook different computers together and do lots of cool things.
> Now, we have a small High Performance Computing system in our university campus
>and I would really like to use that system for testing multicasting applications.
I know of a computer or two I'd like to try multicasting on.
> However, I do not know what would be the best way to use the cluster for
>multicasting purposes. Has anyone experimented with this before?
Anybody have any tips?
> What might be the best multicasting application to use
>to be able to fully utilize the power of the cluster?"
Know of any multicast programs that will bring it to its knees?
C'mon, people, speak English.
Hmm, THAT would explain why this sorry ass of a question got posted. However, since the editors don't like to correct mistakes in the submissions, I doubt they would add links in the submitted text. That would not be right.
I also thought it was strange when I first heard about it but after reading some of the documentation some systems had found ways around this.
Disk Imaging - software has been doing this for awhile but the image only streams to the workstations at the speed of the slowest machine accepting data.
File Distribution - I haven't actively used any of these in production environments but I have played with them.
From what I have read what these apps do is the workstations return verification of received data on a unicast path. That way there is verification not at a UDP/TCP level but at the application level. It is still quite a bit more efficient on a reliable transport medium than sending every ounce of data to every computer. You do get some overhead on the unicast path but you still do not have seperate data streams for every computer.
I could be completely wrong about this but the applications I have read about this is how they documented keeping data integrity while multicasting.
Have a good one.
-Alan
I've spent a little time setting up and using multicast networks. I can't necessary say where a cluster would fit into the mix. I'll outline a couple of the uses we have found/use multicasting for, maybe there is something in here that you can get ideas to use your cluster for:
1. Music/sound broadcast - using a MP3 streamer you can pump stations out on multicast addresses then using mp3 players that support multicast streams you can latch on and listen to the stream. For business purposes this might be to distribute recordings of company meetings or perhaps training information that does not need video.
2. Video - The setup of stations to re-run important videos of events, etc. can be useful. In most cases these are just like TV statations - you join a multicast stream then you see whatever is on that stream at the curren time. Good for just about anything that you would like to rebroadcast to a large number of users using minimal bandwidth.
3. Applications - Under windows there are a couple applications available that will distribute applications across a campus (or any multicast network) to remote users. Broadcast once, receive many. I haven't used this function much but for file distribution that happens on a schedule to the desktop or to multiple servers it could be useful. Not sure of what support is available under Linux for this currently.
4. Video Conferencing - This is one of the applications that we have been experimenting using multicasting for. Being able to broadcast one video conference to say, 80 different sites but in a network only having to distribute a small number of streams is great stuff. An example of this is we have customers who have say 50 sites, all of the sites are connected via sattelite, if we streamed to all 50 sites it would consume a massive amount of bandwidth (say 768k x 60) - using multicasting we can cut that to one transmission that every site receives (only 768k). There are some other rules - like the returning data must follow a unicast path, etc. but overall the bandwidth used is still around 1 T1 instead of the 25+ it would normally take to do this type of streaming.
Someone else stated that this is largely a router/switch thing. It really is, proper configuration of the multicast network at on the routers/switches in the network is crucial. If you are going to be distributing across a large network you need to choose the proper protocols for your needs (sparse or dense mode protocols) and setup the routing so everything flows across the links you desire. A misconfiguration of the equipment could mean that you end up distributing the same data across multiple links to the same site, not distributing the data at all or having it traverse the wrong link which could end up destroying network performance rather than enhancing it.
-Alan
People have been talking about multicasting for a long time . I first heard about mbone in 1994 but I have yet to see it in widespread use.
Multicasting made sense when you looked at the internet, and the amount of high bandwidth content was very limited, but the user base was growing very rapidly.However, content grew at a rate that was probably not forseen. In order for multicasting to make sense, you have to assume that many people downstream of the signal are watching the exact same content exactly in sync (presumably live content) in the same format.
With a few notable exceptions (The Victoria Secret Fashion Shows come to mind) this situation doesn't come up that often. So although it's an attractive idea, neither the ISP's not the content providers have pushed very hard for it, so essentially nothing has happened.
One possible use content that is provided by one person with relativly low bandwidth that can reach many people. This comes at a cost of convienience; you can't provide information "on demand," but only in pseudo-broadcast style. Additionally, ISPs generally don't approve of "little people" providing content to many many people. They want to reserve that ability for those who are willing to pay (a lot) for it.
I'd be surprised if multicasting ever comes into very wide use; since the situations that it's useful under are limited, and the econmoic incentives to get it going don't seem that strong to me. OTOH, the trend in online advertising advertising has been to make it more like television, so maybe a shift is in the works.
Multicastings real killer app is amplification per node and cacheing remotely on nodes. I'll bet $50 US that I can multicast as successfully if not better (depending on switch) with a pentium 75 from a home DSL line than you can with your cluster.
The reason is that this is a throughput and amplification problem, not a computing one. Your cluster has to communicate through soda straws of bandwidth compared to the througput you can get on one mainboard. 1 Gbs is not as fast as 6 Gbs, which is slower than a G4's mainboard.
Once the output stream leaves the box (this really is just I/O) you send it to a node which sends it to a bunch of nodes, then those nodes do the same ad infinitum. Kinda like in "Waynes World" when Wayne does the whole "telephone" speech, with nifty visuals.
In short...using your cluster for this is a waste of computing power, and is hobbled by the connections between boxen.
"Let him go, Ralph. He knows what he's doing." --Otto Mann (simpsons)
The only thing I could think of in a jiffy that would max out a high-perf network would be a setup something like the following:
This is a little tricky to imagine, but here's what I'd so.
Suppose you select 5 nodes on this network.. Each node is equipped with a display, a camera, a mic, and a few local & exported NFS filesystems. Each display has a window on it showing what the camera sees. Point the camera towards the monitor in a video feedback sort of fashion, so you get a picture within a picture within a picture.. Make this stream available to others on the network via NFS. Similarly, have every other node do the same thing, making their streams available as well. The separate displays of each node should be displayed on each node. Now, do the same trick for audio.
As the network begins to reach saturation, you should begin to see a lag in terms of recorded image versus displayed image, and recorded sound versus perceieved sound. Use this lagtime as a barometer for how well your network is able to handle the traffic. The worse the conditions on the network are, the more noticable the delay will be between your live, real-world actions, and the perceptions of those actions when viewed from other nodes.
Now, with this sort of setup in place, you can then design or tweak your network so as to minimize or eliminate this lagtime. Protocol choice will have alot to do with it, but dont neglect the obvious..A chain is only as strong as it's weakest link.
Bowie J. Poag
Bowie J. Poag
I bet you dollars to doughnuts that VALinux has a "partnership" with them. Kinda like with FatbRain.com, yah know? Dying companies need to hang together to weather the storm.
Blar.
Multicasting doesn't really equate with needing high-end hardware. In fact, the requirements would be much lower for multicasting, as opposed to having to do the equivalent number of unicasts.
Multicasting has more to do with switch and router configuration.
There are multiple problems with multicast, not the least of which will be setting up your routers to actually do multicast routing. In the words of an IETF routing area director: "We've spent how many years on this problem, and we still can't do inter-domain multicast routing?"
You've got two choices in the LAN environment: DVMRP and PIM-DM. Router vendors converged on PIM-SM, because it fits better with its inter-domain big brother, PIM-SM (and partly the result of IETF multicast standards politics.) However, I have not come across an open source version of PIM-SM for either FreeBSD or Linux. If you know of such a critter and it ACTUALLY WORKS, I'd be interested in finding out about it.
If you're not interested in inter-domain type multicast routing, then DVMRP works just fine. There are a number of open source implementations out there, although, you should be advised that it will probably go the way of T Rex in the future.
Once you get multicast routing to work, then you'll need to consider things like reliable multicast transport. TCP doesn't work for reasonably obvious reaons apparent to even undergrads. You'll also want to look for IETF RFCs from the LSMA working group that talk about how to apply multicast to large scale simulations. And you will have to carefully analyze what makes sense to multicast in your application environment, i.e. where one-to-many partial computation replication actually makes sense. For example, does your application have one-to-many partial computations that form a staged pipeline?
Multicast has the potential for being a fine tool, but it solves one set of problems at a fairly low network level. There's a lot of extra thinking one has to do to use it effectively at the application level, above and beyond content streaming.
As others have said - multicasting itself doesn't require that high end hardware. So, use it for something CPU intensive that you can then stream perhaps? Maybe realtime video compression? I'm not even sure THAT would require as much CPU as you've got. Perhaps a giant realtime 'net "radio station"? I dunno'...
Why exactly did you associate high end computing with multicasting?
Build it, Drive it, Improve it! Hybridz.org
OK. I'll pretend to take the question seriously for a while. About 6 years ago when I was working on military simulations, multicast was a hot topic in the DIS (Distributed Interactive Simulation) community.
If you're lucky a Google search on DIS and multicast might come up with something interesting.
I've also been interested in mbone (multicasting backbone) for a while since I got broadband access. It's seems simple enough by installing a special virtual router on my firewall (mrouted).
However I have not been able to find any way to hook up my home router to the actual mbone network.
Are there any official contacts for the mbone network that can assist in finding a connection point?
//Pingo
--- Linux or FreeBSD, it's like blondes or brunettes. I like both. ---
Here is a a explanation of unicast vs. multicast:
Unicast means you (your computer) sends out N identical packets to N different (destination) IP addresses.
Multicast means you (your computer) sends out 1 packet to 1 special multicast IP address, and downstream router(s) duplicate that 1 packet into N identical packets to the N different (real) IP addresses which the special multicast IP address corresponds to.
Imagine the potential for DDoS attacks...;-)
Well, both cluster MPI implementations (MPICH and LAM) only support simulated multicasting (a tree approach). However, most switches used in clusters (including the 3com superstack 3300 which we use) support a couple of multicasting protocols on the switch. This would allow you to use multicasting. If MPICH or LAM supported this, it could increase overall bandwidth in certain circumstances, and certainly improve latency. I hope that there will be suport for this in the not-too-distant future, as a lot of people would like to use it, and there are many codes that could benefit from it.
I can understand that some people question the articles that get posted to Slashdot. But if the admins listend to all those whiners, nothing would ever be posted.
--
+1 Insightful, -1 Troll. What can I say, I'm an Insightful Troll.
Uh .. there IS reliable multicast. There are quite some different approaches to it.
...
.. you'll find some !
One would be to to cache some of the data in the routers (or on computers near the routers), which can be retransmitted if someone down the leaf sends NACK's
STFW
I seconds this. Spread is a pretty nifty way of doing reliable and fast multicast of data. Very nice for writing distributed network software (depending on your goals.)
/dev/psychic: No medium found
One application of multicast-IP is broadcasting Interactive Digital Television.
- Multicast an MPEG video stream.
- Multicast the EPG information.
- Multicast interactive applications & data. Massively multi-player games, quizzes (International-Interactive "Who Wants to be a millionaire" ?)
Unicast wouldmn't work because the load on the server would be a factor of the number of consumers, multicast-IP is much more efficient.
It's possible for practically anybody to multi-cast 'Pirate' TV broadcasts, (i.e. "Eyes Only" from "Dark Angel") currently the limitation factor is not the technology, but the high bandwidth cost of content insertion, compared to the [low] numbers of people with broadband connections, to exploit the content.
Soon, real soon :)
The possibilities are awsome!
"We control the horizontal; We control of the vertical; this is the Multicast-Zone"
Check this link out for a cool example of this technology in action. http://www.kit.tv
You know...broadband may be available to everyone else, but where I live they always ask do you run "some version of windows". If the price of a faster connection is to sell my soul, then "no thanks!"
Time is what keeps everything from happening all at once.
Although most of your current multicast applications are for streaming audio and video, multicast can do SOOOOOOOO much more.
Any application where you basically have one system sending identical information to a bunch of other systems at the same time is a potential multicast application. Right now, the common perception is for audio and video technologies. Makes sense...that's what broadcast radio and television is all about. And streaming media does work very well over multicast. But here's a few other apps that also could work well with multicast:
Online games. Especially games which have hundreds, if not thousands of people interacting together. How do you really know where everyone is in a game? You don't, because most online games can't spare the bandwidth to tell you where everyone is. With multicast, a server would be sending a single update on the world to everyone playing. Big bandwidth savings. Also, think about the little guys making games. They might not be able to support the kind of games they want to make, because they can't afford the bandwidth necessary. Multicast saves on bandwidth. You could end up with a couple of guys in their basement hosting games to thousands of people.
Pushing data to cache servers. Think about a company like Akamai, that's got hundreds of these web caching servers all over the country. Reuters comes out with a news flash. They could push out that flash to all these cache boxes out there with multicast. Every cache server gets it at the same time, so whomever hits that cache box sees the story when everyone else does. Also useful for...
Stock market tickers. Everyone wants to see how their stocks are doing, and they want to see it in real-time. You don't want to give an advantage to someone just because they know information sooner than you do. That's where multicast comes in. Using 5 kbps or less of bandwidth, a brokerage company can update millions of people with the stock ticker.
Sys admin. Ever want to deploy new software in your company, but dread having to install it on 500 workstations. You could multicast the files, without completely bringing the LAN to its knees, and it would only take as long as it does to transfer one file.
Popular application updates. Linux kernel 2.6.0 gets released, and instantly FTP servers around the world are hit. They could multicast the software, using a couple of different bandwidth streams to hit modem and T3 users alike, and do it with a reliable multicast protocol that makes sure everyone gets the entire file without loss. Big bandwidth savings, and much happier users who won't have to wait several days for the initial burst of traffic to subside.
Multicast can do a lot of things. Streaming media is just a small part of what multicast can do.
The whole point of multicast is that it requires very little in terms of resources. Having a high performance computing cluster isn't going to do much if you're trying to test out multicast.
More likely, you'll want a sniffer, or else access to routers, either directly or through SNMP. Because what you're really going to want to find out is bandwidth utilization. That's where you're going to see gains going with multicast.
You should head over to IETF's website and start looking at RFCs about multicast. RFCs are usually boring to read, but very insightful.
A better question to ask the Slashdot crowd would be: "Which multicast protocol should I be using?". (FWIW, I recommend going with PIM sparse-mode. See also SSM.) In terms of apps, you haven't really told us much... Are you looking for UNIX, Mac, NT? Audio, video, text, file transfer? High bandwidth, low bandwidth, reliable transport? Without this information, the question posed is so open as to be unanswerable.
The basic problem is that the most popular content on the Internet has a share of maybe 0.1%, compared to, say, 20% for top-rated TV shows. Even if TV eventually moves to the Internet, it will probably be video on demand, not multicasting. If there were any content stream that could draw a sizable market share on the Internet, somebody would have found it during the dot-com boom. Nobody did.
If you really wanna have some fun, write your own applications using multicasting, on your own protocols. See what you can do with mpeg streaming to the machines on your cluster. If thats too much work, which it probably is, play with Spread. It will do the low-level networking and reliability for you, and let you concentrate on what you want it to do.
I am *not* an Atomic Playboy, but I *am* a cheese-eating surrender-monkey!
You know what the whole thing sounds like? Like one of those questions that's written by a parent and given to a kid to ask into a microphone at a school assembly featuring a 'special guest speaker.' I can just hear this being read in the monotone of a little six year old kid.
Vintage computer games and RPG books available. Email me if you're interested.
You can test whether your network is multicast enabled using the multicast tester applet. If you are, information about the content available can be found at the Internet 2 Multicast Calendar web site.
Sorry. Should have been more specific. It's actually Tibco's Rendezvous product. They used to have a neat Mandelbrot generator set up on their web site to demonstrate, but it seems to be gone now.
Fuck 'im up, Tim! His views are invalid! -Pirate Corp$
I'm not sure why this was modded down, so I'll repost it. NCS white paper in PostScript format.
Fuck 'im up, Tim! His views are invalid! -Pirate Corp$
You could try implementing some parallel computing test using some middle ware like Tibco. Tibco is expensive, but it meant for exactly this purpose. There may be other similar middleware applications out there, possibly even Free/Open ones.
Fuck 'im up, Tim! His views are invalid! -Pirate Corp$
furthermore, multicasting has fuck all to do with servers. multicasting is accomplished within a routing/switching infrastructure. the server sends out a SINGLE data stream to a SINGLE SPECIAL ip address; this SINGLE SPECIAL ip address actually identifies a GROUP of hosts. routers & switches know which UNIQUE hosts belong to said GROUP and selectively forward the data. the whole point of this is to reduce load on the server, and prevent multiple identical data streams on the network infrastructure.
k /i cs/cs011.htm
a few notes from cisco :
http://www.cisco.com/univercd/cc/td/doc/cisintw
"Multicast---Multicast applications send each packet to a multicast group address. Hosts that want to receive the packets indicate that they want to be members of the multicast group. This type of application expects that networks with hosts that have joined a multicast group will receive multicast packets. Multicast applications and underlying multicast protocols control multimedia traffic and shield hosts from having to process unnecessary broadcast traffic."
" IP multicasting applications use Class D addresses to address packets. The high-order four bits of a Class D address are set to 1110, and the remaining 28 bits are set to a specific multicast group ID. Class D addresses are typically written as dotted-decimal numbers and are in the range of 224.0.0.0 through 239.255.255.255."
" The Internet Group Management Protocol (IGMP) uses IP datagrams to allow IP multicast applications to join a multicast group. Membership in a multicast group is dynamic---that is, it changes over time as hosts join and leave the group.
Multicast routers that run IGMP use IGMP host-query messages to keep track of the hosts that belong to multicast groups. These messages are sent to the all-systems group address 224.0.0.1. The hosts then send IGMP report messages listing the multicast groups they would like to join. When the router receives a packet addressed to a multicast group, it forwards the packet to those interfaces that have hosts that belong to that group. If you want to prevent hosts on a particular interface from participating in a multicast group, you can configure a filter on that interface by using the ip igmp access-group interface configuration command."
instead of trying to find the project to suit your equipment, doesn't it make more sense to have an idea for a project, and then make the choice of technologies to suit that particular project?
Your question makes no sense!
.com grammar)
Here's the answer: Get yerself an eSolution Webified Scalable Broadband e-Business Streaming LAN Architecture Multiplexor with e-Server e-Integration e-Strategy Streaming OptiRanger for MultiMedia e-LoadBalancing.
WHY MUST WE ALL USE "WEBWORDS"?!?!?!?!?!?!? Speak english for crying out loud, and perhaps your question would make sense!
("e-WebWords" are the worst faction of
Skiers and Riders -- http://www.snowjournal.com
Install Napster across the network.
--
--
#nohup cat
Besides technical issues related to multicast protocols (ex. core placement) there is one big problem which has stopped widespread use of multicast on the net: the lack of security mechanisms .
To take a simple example imagine pay-per-view TV implemented using multicast. We need a mechanism to restrict access to users who paid for the access. If such a mechanism does not exist, there will be very few commercial applications that use multicast.
To understand the difficulty of multicast access control, imagine the pay-per-view scenario. You encrypt traffic with a key. Then you give the key to the members who paid for access. However each time a member is added or removed from the group, you have to change that key. Otherwise if you use the same key, members joining the group will have access to past data and members leaving the group will still have access to future data. Now imagine that you have a million members, and one of them is removed from the group. How can you scalably send a new key to a million members while you make sure that the removed member does not recieve it?
Well, this is still a research area. Though there are some quite clever ideas, none of them provides a completely satisfactory solution.
Even for applications which don't require access control, there remains authentication issues. Perhaps CNN might be interested in TV multicasting, however, how can you make sure that the stream that you receive comes from CNN and not a bunch of hackers sending some kinky video? Well for live realtime streams this requires some form of authentication that needs to be fast and tolerate some data loss, etc... Again no perfect solution exists yet.
So I don't think you will see widespread use of multicast until these issues are solved. On the other hand, multicast will probably be used internaly in corporations, intranets and such.
Ubero will be releasing some software soon that does just this using a simple set of Java APIs.
http://www.ubero.net
Microsoft invented advertising too.
How about real time stock quotes? A few companies have multicast backbones for this sole purpose (e.g., Savvis). Requirement is then to never miss a beat from opening to closing of the market(s). You also end up have to deal with multiple streams that can be very bursty.
Frankly, it all depends.
A small audience can already be taken care of today using individual connections. What's better, the receivers don't have to be synchronized-- each one can start their session at leisure.
The most interesting application of multicast is actually its one-to-many semantics. The first important use of this is wide-area discovery, where nodes who wish to find others with a similar interest can do so in a distributed fashion with a very low bandwith requirement.
For example, multicast in the wide-area network would be great for discovery of members in a peer-to-peer network. In say Gnutella, the current discovery system relies on a few centralized servers, which are easy targets for attacks. With multicast, new peers could discover the existing nodes without a single centralized point.
~
Tsunami -- You can't bring a good wave down!
Do a search on freshmeat. Install the software. Pick a video, stream it.
It wasn't modded down, but when you click on post anonymously it gets posted at 0 instead of 1
Slashdot: Proof that a million monkeys at a million typewriters can create a masterpiece
also if you check all the machines are plugged into a singe 24-port switch. Where does the multi-casting come into play
Explain exactly what the hell having a 24-port switch has to do with multicasting, and why (as you imply) he can't do multicasting if he has it? Or are you just trying to get modded up by trying to sound like you know what you are talking about by throwing in a few technical terms? What exactly is your argument? Are you trying to say that the switch will somehow prevent him from multicasting? I'm not sure how this could possibly be. If its a cheapo switch it doesn't support IGMP snooping, and the machines will hog bandwidth, assuming each is multicasting out on a different multicast session. But most switches support IGMP snooping. Do *you* know if his does? Either way, your point doesn't seem to make any sense, unless you already have some idea how the system is going to be used and why it won't work because there is a switch? WTF are you trying to say man?
Come on, *make your argument*. Don't throw in a technical term or two and just hope we're going to just assume that you know what you're talking about and that we just don't understand. If you do have a point, I'm all for hearing it, go for it, explain it.
"multicasting has fuck all to do with servers" .. next sentence .. "the server sends out a SINGLE data stream"
Everything else you've said is pretty accurate, but I'm not sure what you meant by "has fuck all to do with servers". Multicast apps tpyically involve a single server casting out to n clients.
Mostly only the cheapest, bottom-of-the-range switches don't support IGMP-snooping. I know the one in our office doesn't, but its the only one of about a dozen in its range from 3com that doesn't support it. Of course, the majority of small businesses probably buy the cheapest switches.
As for TCP/IP 'not supporting it', thats hardly valid. TCP/IP stacks of more than 99% of OS's in use support it. It's implemented on UDP, by the way, so it has nothing to do with TCP itself, it's UDP/IP, with addresses in the 224.x.x.x range. For LANs, this is mapped to a certain range of special MAC addresses. The multicast group management uses IGMP protocol, these packets are what switches need to monitor to keep track of the MAC addresses in the sessions.
I know of some fairly major networking communications libraries that *do* make use of IP multicast, e.g. HLA.
The only need I've ever had for multicasting was when we would use Ghost to put a new image on every identical lab machine on our campus. We used to do it all from the server but it became such a strain as the labs grew.
Absolutely nothing, and that's my point. What do 8 headless boxes have to do with multicasting? Nothing.
You didn't check the technical specifications on the equipment they are using. They have 8 Athlon machines plugged into a Intel 410T Ethernet switch. This switch does not provide anything beyond level 2 switching (based on MAC address). Their setup is identical to what you would find in a small office or computer lab, save 7 of the machines are headless.
Now lets look at this fellow's question:
I would really like to use that system for testing multicasting applications. However, I do not know what would be the best way to use the cluster for multicasting purposes. Has anyone experimented with this before? What might be the best multicasting application to use to be able to fully utilize the power of the cluster?
His question is similar to asking "I have a waffle iron and I want to look into the effects of microwave energy on the human eye". The tools have nothing to do with the application. If he had been familiar with the subject area he would have known that. He might be able to use it for load testing, but I am sure others have more important uses for it (like distributed/parallel processing).
First I would not call 8 Athlon 650 machines a "High Performance Computing system", also if you check all the machines are plugged into a singe 24-port switch. Where does the multi-casting come into play? Your testbed was designed for distributed computing, not mult-casting. I have to assume you just started in your lab. Go talk to your professor and read up on the material before asking for Slashdot to help you out.
There is plenty of room for making good use of
multicasting in high performance computing.
However, most clusters really don't use it
because either A) The switches don't support
it (other than 1-to-all) or B) TCP/IP doesn't
support it.
The message passing libraries probably being used on your cluster (either PVM or some variant of an MPI implementation probably have multicasting statements, but they are in actuality probably
implemented as a sequence of normal one-one communications. Now if you wanted to implement
your own simulated multicast in the cluster, where
you used some kind of tree structure to forward
messages to all the right nodes, have at it,
but you'll probably have to modify your message
passing library...
Dan
When are people going to get it?
Multicasting is, also, not going to work.
Why? Think about it. How many 1 Mbps multicasts can a 10 Mbps pipe carry? 10. Ooh Aah.
Work the numbers boys and girls. There just ain't enough bandwidth to carry more than a handful of broadcast shite in the backbones...
As gets mentioned here from time to time, Digital Fountain addresses (or endeavors to address) the "recipients in sync" problem.
I'd be surprised if multicasting ever comes into very wide use; since the situations that it's useful under are limited...
Where I work, many of us kids like to listen to streaming radio broadcasts. We've been criticized for the strain we put on our Internet connection, and it's a valid point. Often several of us are listening to the same shoutcast stream (or whatever) at the same time, and it seems kinda silly that we consume N * 50-100kbps of bandwidth to receive the same content. But, hey, this is just a personal way in which multicast could help my life.
When most people think of multicast they think of 1-to-many transmission. There are also lots of applications involving many-to-many transmission. Chat is an obvious one; chat becomes particularly well suited for multicast when you're dealing with voice chat rather than text. A more interest application is in networked virtual environments (less grandiloquently, games). A couple other fellows and I wrote the networking part of an NVE that used many-to-many multicast: the world was partitioned into octrees, and each octree was assigned a multicast group. Octree nodes split and merged based on traffic, and there were different levels of groups for messages of different levels of detail (e.g., toe movements vs. explosions). (Well, this was the plan; we didn't finish all of it, but it was a cool demo). Peer-to-peer NVEs have many advantages over client-server systems, including reduced (and hopefully optimally minimal) latency and natural scalability. This book provides an overview of the subject, but there are many papers out there that are more in-depth and informative.
Finally, check out Kevin Almeroth's research in multicast applications. He has several good survey papers that address your synchronized play out gripe and explore the gamut of potential multicast applications.
Simple, get multiple video feeds and stream 'em. Charge if you want to. That oughta get some cycles going. If you really want to push it, use one of those free java streaming video apps that'll bog the server AND the client!
2) Multicast campus events that have a limited capacity at the site, like sporting events, concerts and plays.
3) Multicast or unicast of a remote event to the campus, via point-to-point connection, which is then multicast to the students. Again, special lectures, sporting events and such could be covered.
That or you could just multicast parties at the local frats and sororities for entertainment value.
If you can't beat them, embrace and extend them.
By the way, you can download TIB/Rendezvous software, which is what most of the big trading floors use to deliver market data to professional traders (the download is a limited version). This is a good way to experiment with multicasting (but you don't get the market data - you have to pay big $ for that).
hahaha, your techno babble is top drawer. I can you're going places
If you don't say anything, you won't be called on to repeat it. -- Calvin Coolidge
A San Francisco startup called Sonicity is building a business around this very topic. Check out some product information here. I'm not vouching for them--- I just happened to read their tech papers a few weeks ago.
I don't like to complain man, but there is tons of information out there on the subject. Is 'Ask Slashdot' all of a sudden a 'luxury' version of a search engine??
In other words: google->multicast->first site: IP multicast initiative->couple of links down: startdust.com
This is just being lazy.
Impatiently waiting for my popcorn to pop in the microwave
What might be the best multicasting application to use to be able to fully utilize the power of the cluster?
When you think about it, the applications of network multicasting (of content anyway) are pretty straightforward, and pretty much come down to 1-to-many streamed audio/visual content. Most of your ideas (distance learning, conferencing, etc.) assume that you have content to stream. As for distributed file share, etc., those all seem to me to be more applicable to unicast technologies.
If you're interested in multicast technologies, there are some very interesting low bandwidth applications. Heartbeats for distributed system applications are a good example. See the Linux-HA (High Availability) project for an application of this: linux-ha.org.
Invisible Agent
Invisible Agent
This post is a mirror; when a monkey stares in, no hacker gazes out.
Multicasting is designed to require very little hardware. I doubt that you'll be able to put much of a strain on your equipment at all. Now, Seti@home is a different story....
Kind thoughts do not change the world
On a more serious notes than my other post, I don't think this m0r0n understands what multicasting is. I just think he needs to get in his Corporate Buzzword Bingo quota in for his term paper.
He just better make sure that he champions a value-added, retention-based, focus-group tested, peer-to-peer, internet-enabled, disenfranchised, anti-pornography, rectal concept that enhanced shareholder value, motivates employees, and makes a mean bagna cauda.
That is all.
------ 1001001