Slashdot Mirror


Pro-Active VoIP Management Solutions?

Adeptus_Luminati asks: "I've been running a 1000 user Mitel VoIP phone (to the desk) network which encompasses 20 buildings glued together by our Telco's _private_ fibre backbone (no Internet involved here). Once in a while we have voice quality degradation issues caused by excess latency, jitter, bandwidth saturation, QoS mis-configurations, and so forth. I've been using Ixia Chariot software to simulate VoIP calls over the WAN between our various offices and collect data of the problems, but this is only useful AFTER the problem is reported by our users, and after I am lucky enough to be around and catch the problem happening in real time; otherwise, I have no way of proving to our Telco that there IS a problem. What solutions have other network admins come up with to pro-actively manage similar private VoIP networks?" "I am looking for some sort of solution to allow me to pro-actively monitor or simulate 24/7 VoIP calls between offices and then report back to me immediately when certain thresholds of voice quality degradation have been exceeded and accumulate significant info that I can forward my Telco and get them to deal with the problem, right away. FYI, bandwidth is free on my office WAN links, we're mostly 100Mbit fibre, and we have QoS from end to end (except small parts of the telco backbone)."

30 comments

  1. Simulating voice calls by hummassa · · Score: 2, Interesting

    Computer #1 in one building, #2 in another.
    Cron job:
    Computer #1 voice-calls computer #2 and plays a complex and long sound.
    Computer #2 records the sound it received.
    Computer #2 compares the sound it received with the original file.
    Log errors; if error-rate > x, page you, sleep short time, repeat cron job.
    Simple, ain't it?

    --
    It's better to be the foot on the boot than the face on the pavement. ~~ tkx Kadin2048
    1. Re:Simulating voice calls by pyrrhonist · · Score: 1
      Simple, ain't it?

      And wrong. Telephony systems aren't like file systems. You can't just receive a file, compare the output with the original, and assume that there's a problem if it doesn't pass. In telephony systems, there may be format translations or echo cancellation done on the voice channel, which may change the output, but are not indicative of a problem. Depending on how his network is set up, the voice channel may even be routed through analog equipment. If this is the case, you won't ever get the same sound as what you put into the other end.

      --
      Show me on the doll where his noodly appendage touched you.
    2. Re:Simulating voice calls by pete-classic · · Score: 1

      Is measuring the "error rate" between a known sound sample and a re-sampled version trivial? You sure make it sound like it is!

      -Peter

    3. Re:Simulating voice calls by Ramses0 · · Score: 2, Informative

      Talk to the ogg-vorbis people, and check their mailing list archives. I believe they have some tools that do the moral equivalent of:

      $ compress foo.wav > foo.ogg

      $ compare foo.wav foo.ogg
      18% different

      Some interesting quick googling turned up the following: http://www.abde.net/projects/ogg_mp3/

      Original google search:
      http://www.google.com/search?hl=en&lr=&q=ogg+vorbi s+quantitative&btnG=Search

      Term I seem to recall is "quantitative" comparision of audio quality (vs. "qualitative" ... ie: "it sounds better").

      When doing audio "optimization" there are a few types of tests, one that can be done by computers (comparing data and formulas to other data), and the other that has to be done by people.

      If you only did the "computer" type tests, you might have something that is as close to accurate as possible, but would still sound "off" in the "wrong way" to human ears (ie: the computer might have "optimized out" all the bass in order to be more accurate on the treble, but few people would accept "qualitatively" the results of that compression, even though quantitatively it might be "closer to accurate" than the other).

      Anyway, I am not an audio researcher, but you might start there.

      --Robert

    4. Re:Simulating voice calls by Marxist+Hacker+42 · · Score: 1

      Assuming it's all digital- and Voice over IP is- then yes, it is relatively trivial. Especially if you get to pick the sound- what you want is a siren. That way you get a big blast of sound running through all the frequencies you're interested in, and you can use the (relatively) quiet to loud transition as a good zero indicator. From there, it's just a byte comparison.

      --
      SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
    5. Re:Simulating voice calls by ldspartan · · Score: 1

      Except that, if this guy is smart, he's compressing the voice over the line. I'm guessing a siren run through G.729a won't sound exactly like you say.

      Remember these codecs are specifically designed to compress speech with minimal losses of intelligibility. Good luck measuring how well a human can understand speech without a mark I human ear.

      As for the OPs actual question, Cisco Callmanager can be configured to collect call statistics on every call (jitter, delay, etc.). It can also be set to simulate calls between two physical handsets (i.e., 2 SCCP phones do the work and are unavailable as a result). I am defiitely Cisco-biased, but its possible the Mitel the OP is using has similiar features. If not, I'm sure Cisco would be happy to sell him a replacement :).

      --
      lds

    6. Re:Simulating voice calls by tzanger · · Score: 1

      Sorry, he isn't wrong.

      You don't do exact byte-for-byte comparisons. You do regular signal analysis and see how close the audio is to each other. If it exceeds a given deviation, you alert them.

      And yes, it's not dead as in doornail simple, but it is a fairly straightforward problem.

    7. Re:Simulating voice calls by pyrrhonist · · Score: 1
      Sorry, he isn't wrong. You don't do exact byte-for-byte comparisons. You do regular signal analysis and see how close the audio is to each other. If it exceeds a given deviation, you alert them.

      First of all, the standard for speech quality is PESQ or Perceptual Evaluation of Speech Quality (ITU-T P.862). In PESQ, you don't just, "see how close the audio is to each other". There's a lot of things that need to be done to both the input and reference signals before you can begin analysis.

      First, the input and reference signal levels are aligned to a constant sound pressure level of 79dB. Secondly, filtering is performed to take into account things like the effect of the electrical and acoustic components of the handset. Next, delay is taken into account by detecting voice activity and aligning the parts of the signal that match. Then the signals are transformed to match certain properties of the human ear to get a sensation surface, and any loud sections are realigned. Finally, the parts of the signal with negligable effect on quality are equalized. Once all that is done, then the sensation surfaces of the signals are analyzed to find the MOS.

      It's not just sending files around and diffing them like the OP implied. Oh, yeah, and you need to license the algorithm.

      --
      Show me on the doll where his noodly appendage touched you.
    8. Re:Simulating voice calls by Anonymous Coward · · Score: 0

      WAY TO GO YOU PEDANT!

  2. :gag: by Anonymous Coward · · Score: 4, Funny

    Pro-Active VoIP Management Solutions?

    You're going to hell.

  3. SNMP by QuantumRiff · · Score: 3, Interesting
    Ask your telco for "SNMP read" access to their routers that they use. Setup an MRTG page that shows traffic and latency. Is this pure fiber from building to building? or are there a bunch of Cat5 (or other cabling) to fiber converters along the path? Most Telco's offer SLA (Service Level Agreements) that garuntee a certain amount of bandwith, latency, and availabiltiy. Also, I know on our metro fiber ring we are moving to, it is all ethernet over fiber, and each company gets their own VLAN. Is your connection pure ethernet all the way through? (if you live in a big city, some of the big players give you your own wavelength, instead of VLAN.. Much nicer)

    there is also the option of turning down the audio quality between buildings. (ie, 128Kb stream inside the building, 64kb stream between them.) While slightly more noisy, it still works, and uses less bandwith. I know with our old Cisco VOIP at my old job, department to department calls were low bandwith, and customer calls were setup for highest bandwith. (clearest)

    --

    What are we going to do tonight Brain?
  4. Network General InfiniStream by arnie_apesacrappin · · Score: 2, Informative
    The box is a sniffer with a huge array of disks. It records all traffic that you send it. I have used the product before, but not for VoIP. Here is what the Network General site says about the VoIP option for the InfiniStream:

    The Voice Option is a value-added package that integrates with InfiniStream Network Management to provide additional insights into voice- and video-over-IP converged traffic. Voice-over-IP (VoIP) Experts automatically detect and help resolve key problems seen on VoIP networks--jitter, packet loss, packet-sequencing errors, and latency. These VoIP Experts and call-tracking capabilities, along with the traditional Expert system, help ensure successful VoIP network rollouts while maintaining "toll-quality" voice and high-quality data for all users.

    The product URL is here

    They make a couple of versions. The last time I looked, the 1 TB version was around 25K and the 4 TB version was around 95K. I didn't buy one, but it was a fun toy to play with.

    --

    Still, with a plan, you only get the best you can imagine. I'd always hoped for something better than that. -CP

  5. Also by QuantumRiff · · Score: 2

    to reply to my post, This looks interesting NQMS

    --

    What are we going to do tonight Brain?
  6. Commercial Product: NEXVU by breadbot · · Score: 1

    It sounds like you're talking about monitoring your network for application performance and watching for telltales that precede degredation. You might check out a product from this company:

    NEXVU Technologies

    Because that's exactly what they do :)

    Unlike a general packet sniffer or network monitor, they aim exactly at your kind of problem.

    Disclaimer: until I entered the glorious realm of academic programming, I was employed by NEXVU, and I still have stock and stock options. Even though they no longer benefit from my obvious genius, I think they have a great product. If enough VoIP administrators agree, then those options may be worth something in the future!

  7. Cisco IOS SLA on routers might help. by Mordant · · Score: 1

    Check here.

  8. Ach, lad... by jo42 · · Score: 1

    You shoulda stayed with the olde way of doing things. This new fangled VOIP cr*p is nothing but a load of POS foisted upon yee by sales and marketing droids to give them a reason to exist. And sell yee more cr*p.

  9. The phone company by arrow · · Score: 1

    If you have to prove to them that their is a problem, they aren't likely to do anything about it when you do come up with evidence.

    Sorry to burst your bubble, but unless you call your sales rep and threaten to leave, your not going to get anywhere.

    --
    symetrix. We are building a religion, a limited edition.
  10. Similar problem by Omega1045 · · Score: 2, Interesting
    A friend had a similar problem. They were sure that the only available telco in the area was not providing the level of service to which they had agreed. They could not get the telco to help at all.

    His solution? He got his board of directors to approve the purchase of some wifi radio equipment, which they mounted on nearby towers. I am not a hardware or radio guy, but this was not Linksys crap that I run in my home. He got some professional stuff. Each office had LOS to a local tower, and the towers to each other. Last I heard, they are running all of their voice and all of their data over their new links. Routers at both ends are configured for QoS, and thing are running very well. The cost of the equipment has already been paid for with the savings since what they pay for the towers is a fraction of the cost of the circuits they were running between offices. They maintain a few landlines that the phone systems on each end can use in the case of emergency to route voice traffic, and I believe he also has a couple of redundant DSL lines for data.

    --

    Great ideas often receive violent opposition from mediocre minds. - Albert Einstein

    1. Re:Similar problem by Anonymous Coward · · Score: 0


      I'm sure it sounds great in a thunderstorm or heavy downpour.

    2. Re:Similar problem by qqtortqq · · Score: 1

      It was probably microwave. I don't know much about it, but I have seen it in plenty of places and understand it is a good way to handle digital video signals, so I have to assume it would do well with VoIP also.

  11. What I would do by DDumitru · · Score: 1

    I don't know of an existing tool for this, but you want to measure "known traffic" across the route and report when it degrades.

    Setup an "application ping server" on the far end. This will be a C program that posts a UDP listen on a port in the RTP range. When it gets an inbound packet, it returns exactly that same packet to whom sent it to them. This needs to be in C (or similar) because the latency needs to be very low. It should also run on a very low utilization server.

    On the near end, write a similar program that sends a UDP packet to the far end and measures round-trip delay. If you want to really act like a VOIP call, send 200 byte packets at 50/sec (or whatever the SIP/RTP payload is for your particular codec).

    The question is then how to analyze the data. I would probably have the program spit out peak/avg "numbers" for latency and packet loss into a text file, perhaps at 1/minute. You could then have MRTG (or your favorite grapher) read those and draw you pretty stuff. Once you see the types of degredation that actually matters, you can program in your thresholds to page/email/whatever you.

    Things to remember when setting this up. Make sure the traffic is handled by the routers exactly like your VOIP packets. Run on the same UDP ports with the same QOS tags. In other terms, make your test traffic indistinguishable from the actual voice traffic (at least from a router's/switch's voicepoint).

    Of course, someone probably has a package that does this. If not, build it and open a sourceforge project.

  12. Cheap options and Expensive options by kasparov · · Score: 3, Informative
    Cheap option: Linux box hooked up to an ethernet tap at interconnects with the telco's lines. Run ethereal's tethereal in ring buffer mode (making sure that individual files are under 2GB). You are only limited by hard drive space in how much you can store. When viewing the dumps, use etheral > 0.10.10 and go to Statistics->Voip Calls. It will allow you to choose specific calls and even graph things such as latency, jitter, etc. Since you will be dealing with lots of very large files, I recommend using tcpslice (which usually ships in distros with tcpdump) to grab specific chunks that you would like to look at.

    Expensive option:Empirx Hammer XMS. It does all of the above with a nice web interface plus it gives you RTP quality metrics like r-factor and MOS. It's not cheap, but I've used and it does a good job (it is basically a SuSE Linux box with some networking gear running their network monitoring software).

    All of the above I have tested only with SIP/RTP traffic. If you youse MGCP or H.323, I can't personally vouch for either of the above solutions, though both support them.

    --
    There's no place I can be, since I found Serenity.
  13. Prognosis make something like that. by GrpA · · Score: 1

    http://www.prognosis.com/ Also, consider IP SLA (Cisco) GrpA

    --
    Enjoy science fiction? "Turing Evolved" - AI, Mecha, Androids and rail-gun battles. What more could you want?
  14. Fibre vs. Fiber by ericlakin · · Score: 1

    Fibre = Fibre Channel protocol as used in a SAN

    Fiber = Fiber optic (as in the physical cabling)

    Sorry. It's an annoying habit of mine.

    1. Re:Fibre vs. Fiber by Anonymous Coward · · Score: 0

      Sorry. It's an annoying habit of mine.

      Even more annoying, it's also incorrect! The root word, fibre is of French origin and is therefore used in French, English and several other European languages. America, unable to use the Euro/English version of anything had to change it to fiber. The Americans did similar changes with centre, tyre, colour, neighbour, flavour, defence, offence and many many other words.

      There is no difference in the definition of fibre or fiber, only the spelling is different. Your annoying habit could use a dictionary. I would recommend that you use an Oxford dictionary.

  15. Smokeping by subreality · · Score: 1

    Give Smokeping a try. Smokeping:ping::MRTG:bandwidth.

    http://people.ee.ethz.ch/~oetiker/webtools/smokepi ng/

    It shows latency, loss, and jitter in a combined easy to read graph. After using it for a while, you can spot many normally invisible network anomolies on these graphs long before they become visible to users. They're also great for post mortem analysis.

    They don't have anything to do specifically with VoIP, but I think they're invaluable tools for any network admin.

  16. YACP: Yet Another Commercial Product by bferrell · · Score: 1

    http://edgewaternetworks.com/

    It's specifically designed for VOIP quality monitoring.

    And as a disclaimer, I do some work for the company.

    1. Re:YACP: Yet Another Commercial Product by Anonymous Coward · · Score: 0

      You can also use a software product from Viola Networks called NetAlly - soft agents that will make nXvoice-calls periodically and raise an email or SNMP alert if the quality drops off.

      In Australia you can get it from http://www.telcosi.com/ or from www.violanetworks.com globally

  17. Get a Real Backbone! by Anonymous Coward · · Score: 0

    100mbit backbone are you joking?
    QoS partially implemented, are you joking?

        Think about it, 100mbit is just about adequate for the Store and Forward world of data packeting and is hardly adequate for the real time packet delivery world of Voice. Unless its just you and that hot admin assistant exchanging essential personal data!

  18. Use the tool you have... by mindslip · · Score: 1

    Chariot does both monitoring and call load simulation... exactly what you want.

    I'm in charge of network assessments for a very large voip hardware manufacturer... we've used this tool to do what you're describing.

    Give tech support a call and get them to walk you through it. It's a great tool.

    Make sure you've got QoS properly set up on all your devices too, regardless if it's across the internet or not. You still need QoS!