Pro-Active VoIP Management Solutions?

Adeptus_Luminati asks: "I've been running a 1000 user Mitel VoIP phone (to the desk) network which encompasses 20 buildings glued together by our Telco's _private_ fibre backbone (no Internet involved here). Once in a while we have voice quality degradation issues caused by excess latency, jitter, bandwidth saturation, QoS mis-configurations, and so forth. I've been using Ixia Chariot software to simulate VoIP calls over the WAN between our various offices and collect data of the problems, but this is only useful AFTER the problem is reported by our users, and after I am lucky enough to be around and catch the problem happening in real time; otherwise, I have no way of proving to our Telco that there IS a problem. What solutions have other network admins come up with to pro-actively manage similar private VoIP networks?" "I am looking for some sort of solution to allow me to pro-actively monitor or simulate 24/7 VoIP calls between offices and then report back to me immediately when certain thresholds of voice quality degradation have been exceeded and accumulate significant info that I can forward my Telco and get them to deal with the problem, right away. FYI, bandwidth is free on my office WAN links, we're mostly 100Mbit fibre, and we have QoS from end to end (except small parts of the telco backbone)."

Comments

Simulating voice calls

[hummassa]

Computer #1 in one building, #2 in another.
Cron job:
Computer #1 voice-calls computer #2 and plays a complex and long sound.
Computer #2 records the sound it received.
Computer #2 compares the sound it received with the original file.
Log errors; if error-rate > x, page you, sleep short time, repeat cron job.
Simple, ain't it?

Re:Simulating voice calls

[Ramses0]

Talk to the ogg-vorbis people, and check their mailing list archives. I believe they have some tools that do the moral equivalent of:

$ compress foo.wav > foo.ogg

$ compare foo.wav foo.ogg
18% different

Some interesting quick googling turned up the following: http://www.abde.net/projects/ogg_mp3/

Original google search:
http://www.google.com/search?hl=en&lr=&q=ogg+vorbi s+quantitative&btnG=Search

Term I seem to recall is "quantitative" comparision of audio quality (vs. "qualitative" ... ie: "it sounds better").

When doing audio "optimization" there are a few types of tests, one that can be done by computers (comparing data and formulas to other data), and the other that has to be done by people.

If you only did the "computer" type tests, you might have something that is as close to accurate as possible, but would still sound "off" in the "wrong way" to human ears (ie: the computer might have "optimized out" all the bass in order to be more accurate on the treble, but few people would accept "qualitatively" the results of that compression, even though quantitatively it might be "closer to accurate" than the other).

Anyway, I am not an audio researcher, but you might start there.

--Robert

:gag:

[Anonymous Coward]

Pro-Active VoIP Management Solutions?

You're going to hell.

SNMP

[QuantumRiff]

Ask your telco for "SNMP read" access to their routers that they use. Setup an MRTG page that shows traffic and latency. Is this pure fiber from building to building? or are there a bunch of Cat5 (or other cabling) to fiber converters along the path? Most Telco's offer SLA (Service Level Agreements) that garuntee a certain amount of bandwith, latency, and availabiltiy. Also, I know on our metro fiber ring we are moving to, it is all ethernet over fiber, and each company gets their own VLAN. Is your connection pure ethernet all the way through? (if you live in a big city, some of the big players give you your own wavelength, instead of VLAN.. Much nicer)

there is also the option of turning down the audio quality between buildings. (ie, 128Kb stream inside the building, 64kb stream between them.) While slightly more noisy, it still works, and uses less bandwith. I know with our old Cisco VOIP at my old job, department to department calls were low bandwith, and customer calls were setup for highest bandwith. (clearest)

Network General InfiniStream

[arnie_apesacrappin]

The box is a sniffer with a huge array of disks. It records all traffic that you send it. I have used the product before, but not for VoIP. Here is what the Network General site says about the VoIP option for the InfiniStream:

The Voice Option is a value-added package that integrates with InfiniStream Network Management to provide additional insights into voice- and video-over-IP converged traffic. Voice-over-IP (VoIP) Experts automatically detect and help resolve key problems seen on VoIP networks--jitter, packet loss, packet-sequencing errors, and latency. These VoIP Experts and call-tracking capabilities, along with the traditional Expert system, help ensure successful VoIP network rollouts while maintaining "toll-quality" voice and high-quality data for all users.

The product URL is here

They make a couple of versions. The last time I looked, the 1 TB version was around 25K and the 4 TB version was around 95K. I didn't buy one, but it was a fun toy to play with.

Also

[QuantumRiff]

to reply to my post, This looks interesting NQMS

Similar problem

[Omega1045]

A friend had a similar problem. They were sure that the only available telco in the area was not providing the level of service to which they had agreed. They could not get the telco to help at all.

His solution? He got his board of directors to approve the purchase of some wifi radio equipment, which they mounted on nearby towers. I am not a hardware or radio guy, but this was not Linksys crap that I run in my home. He got some professional stuff. Each office had LOS to a local tower, and the towers to each other. Last I heard, they are running all of their voice and all of their data over their new links. Routers at both ends are configured for QoS, and thing are running very well. The cost of the equipment has already been paid for with the savings since what they pay for the towers is a fraction of the cost of the circuits they were running between offices. They maintain a few landlines that the phone systems on each end can use in the case of emergency to route voice traffic, and I believe he also has a couple of redundant DSL lines for data.

Cheap options and Expensive options

[kasparov]

Cheap option: Linux box hooked up to an ethernet tap at interconnects with the telco's lines. Run ethereal's tethereal in ring buffer mode (making sure that individual files are under 2GB). You are only limited by hard drive space in how much you can store. When viewing the dumps, use etheral > 0.10.10 and go to Statistics->Voip Calls. It will allow you to choose specific calls and even graph things such as latency, jitter, etc. Since you will be dealing with lots of very large files, I recommend using tcpslice (which usually ships in distros with tcpdump) to grab specific chunks that you would like to look at.

Expensive option:Empirx Hammer XMS. It does all of the above with a nice web interface plus it gives you RTP quality metrics like r-factor and MOS. It's not cheap, but I've used and it does a good job (it is basically a SuSE Linux box with some networking gear running their network monitoring software).

All of the above I have tested only with SIP/RTP traffic. If you youse MGCP or H.323, I can't personally vouch for either of the above solutions, though both support them.