Pro-Active VoIP Management Solutions?

Adeptus_Luminati asks: "I've been running a 1000 user Mitel VoIP phone (to the desk) network which encompasses 20 buildings glued together by our Telco's _private_ fibre backbone (no Internet involved here). Once in a while we have voice quality degradation issues caused by excess latency, jitter, bandwidth saturation, QoS mis-configurations, and so forth. I've been using Ixia Chariot software to simulate VoIP calls over the WAN between our various offices and collect data of the problems, but this is only useful AFTER the problem is reported by our users, and after I am lucky enough to be around and catch the problem happening in real time; otherwise, I have no way of proving to our Telco that there IS a problem. What solutions have other network admins come up with to pro-actively manage similar private VoIP networks?" "I am looking for some sort of solution to allow me to pro-actively monitor or simulate 24/7 VoIP calls between offices and then report back to me immediately when certain thresholds of voice quality degradation have been exceeded and accumulate significant info that I can forward my Telco and get them to deal with the problem, right away. FYI, bandwidth is free on my office WAN links, we're mostly 100Mbit fibre, and we have QoS from end to end (except small parts of the telco backbone)."

Comments

Simulating voice calls

[hummassa]

Computer #1 in one building, #2 in another.
Cron job:
Computer #1 voice-calls computer #2 and plays a complex and long sound.
Computer #2 records the sound it received.
Computer #2 compares the sound it received with the original file.
Log errors; if error-rate > x, page you, sleep short time, repeat cron job.
Simple, ain't it?

SNMP

[QuantumRiff]

Ask your telco for "SNMP read" access to their routers that they use. Setup an MRTG page that shows traffic and latency. Is this pure fiber from building to building? or are there a bunch of Cat5 (or other cabling) to fiber converters along the path? Most Telco's offer SLA (Service Level Agreements) that garuntee a certain amount of bandwith, latency, and availabiltiy. Also, I know on our metro fiber ring we are moving to, it is all ethernet over fiber, and each company gets their own VLAN. Is your connection pure ethernet all the way through? (if you live in a big city, some of the big players give you your own wavelength, instead of VLAN.. Much nicer)

there is also the option of turning down the audio quality between buildings. (ie, 128Kb stream inside the building, 64kb stream between them.) While slightly more noisy, it still works, and uses less bandwith. I know with our old Cisco VOIP at my old job, department to department calls were low bandwith, and customer calls were setup for highest bandwith. (clearest)

Similar problem

[Omega1045]

A friend had a similar problem. They were sure that the only available telco in the area was not providing the level of service to which they had agreed. They could not get the telco to help at all.

His solution? He got his board of directors to approve the purchase of some wifi radio equipment, which they mounted on nearby towers. I am not a hardware or radio guy, but this was not Linksys crap that I run in my home. He got some professional stuff. Each office had LOS to a local tower, and the towers to each other. Last I heard, they are running all of their voice and all of their data over their new links. Routers at both ends are configured for QoS, and thing are running very well. The cost of the equipment has already been paid for with the savings since what they pay for the towers is a fraction of the cost of the circuits they were running between offices. They maintain a few landlines that the phone systems on each end can use in the case of emergency to route voice traffic, and I believe he also has a couple of redundant DSL lines for data.