Measure and Manage Voice Performance Across Networks

Posted by on September 23, 2014

VoIP adoption continues to grow rapidly, with business deployment nearing 50%. The packet-switching approach has several advantages over the traditional telephone services, mostly because it’s a less expensive solution and it’s easier to manage. The rapid growth of VoIP makes it a critical application for operations teams to understand, monitor and manage. Despite the increasing importance, we hear from operations teams that dealing with VoIP is often frustrating.

Wrangling with VoIP

First, migrating or deploying an entire communication system can be overwhelming. System administrators will need to cooperate with network designers, network security staff and end-users to understand their needs and concerns prior deploying a VoIP solution. Is our IP network ready for VoIP? Are the routers prioritizing VoIP traffic? Is the firewall letting it go through? Is VoIP traffic going to be affected by traffic surges at regular office hours? How about VoIP calls between different branch offices? Enterprises have many unified communications options, including both on-premises and UCaaS deployments from vendors such as Cisco, Avaya and Microsoft as well as SIP trunking for their VoIP deployments.

Second, troubleshooting an already existing VoIP infrastructure can lead to mutual finger-pointing between all the involved teams, from the end-users (who are definitely not using the system in the right way), to the Chief Security Officer (that is most certainly enforcing some extreme security policy and blocking VoIP traffic), to the network administrator (who failed to configure the network devices properly), and all the way to the network provider (that is not able to handle our voice traffic). Without technical expertise and more insight into the problem, this usually results in a lot frustration and no solutions.

Our Take on VoIP Monitoring

Given these frustrations, we’ve created a new take on VoIP monitoring that aims to clearly identify VoIP problems so that operations teams can better solve them. No guesswork or mutual finger-pointing. Our VoIP tests don’t require an existing VoIP solution to be in place, so you can even use it in a pre-deployment phase to check if the IP network infrastructure is ready. Of course, you can also set up periodic tests to monitor the performance of an existing VoIP solution and go back to them to troubleshoot any performance issues. For a great summary, see Amy Arnold’s article about ThousandEyes VoIP capabilities from Networking Field Day 8.

First, you set up a test from one or more ThousandEyes Agents (sources) to another ThousandEyes Agent (target). These tests periodically generate synthetic VoIP calls between the caller agents and the target. The tests are performed in a single direction using a stream of UDP packets that carry RTP (Real-time Transport Protocol) audio frames. A complete VoIP call consists in two independent voice streams (one for each direction), so in order to fully test the voice quality in both directions two independent tests should be created (we are planning to add bidirectional VoIP tests in the future).

Since enterprise VoIP calls typically come to or from one of the enterprise branch offices, we provide the following scenarios depending on whether you’re using Enterprise Agents on your own corporate network or Cloud Agents in Internet POPs.

  1. If the target agent is a ThousandEyes Enterprise Agent, the remaining agents (sources) can be either Enterprise or Cloud Agents.
  2. If the target agent is a ThousandEyes Cloud Agent, then all the remaining agents (sources) must be Enterprise Agents.

After defining the source agents and the target, you’d set up the test periodicity (from 5 minutes to 1 hour) and Alert Rules. Tests also support several audio codecs and frame rates (e.g., G.711 at 64Kbps or G.729a at 8 Kbps), as well as a custom IP DSCP field (e.g., Expedited Forwarding), and a specific static de-jitter buffer size. The VoIP performance can then be monitored in the Voice Metrics screen (Figure 1).

Figure 1: Overview of voice metrics from global callers to a target in New York.
Figure 1: Overview of voice metrics from global callers to a target in New York.

On the top right side we can see a timeline, which shows the average MOS across all agents at that particular time slice. MOS stands for Mean Opinion Score and it’s a combined metric (from 1-5) that reflects the voice quality of a call as perceived by a human. We calculate the MOS-CQE by using the E-Model (see ITU-T G.107), which takes into account several impairment factors, such as packet loss and discards (audio frames were dropped because they arrived too late to be played back), one-way latency, and codec-specific impairment factors. We also show the DSCP as received by the target, to check if the QoS policies are correctly implemented, and the 99.9th percentile of the packet delay variation (PDV), which is useful to determine de-jitter buffer sizes. Generally speaking, the de-jitter buffer should be large enough so that it doesn’t drop too many late packets (if any) and small enough so that it doesn’t add too much delay to the mouth-to-ear latency. By making it about the size of PDV, we ensure that the buffer is large enough such that there are no discards.

Troubleshooting VoIP with ThousandEyes

We can see that three agents have low MOS values and high packet loss, which would account for the drop in voice quality. Furthermore, we can use X-Layer to correlate information from different views, giving us a wider perspective about the problem. An important perspective for troubleshooting VoIP issues is the Path Visualization view (Figure 2).

Figure 2: Path trace of traffic from callers on the left to the target on the right, with packet loss indicated in red.
Figure 2: Path trace of traffic from callers on the left to the target on the right, with packet loss indicated in red.

Immediately we are presented with all the routes from the source agents to the target while performing the VoIP tests; these routes were obtained with the same type of RTP traffic used in the voice metrics. In particular, we can see that one of the hops was marked in red because it had more than 25% of forwarding loss. In fact, it is a common hop in the route from the three agents with high packet loss, which suggests that it was dropping voice packets from those three locations, and therefore affecting the voice quality or MOS. We can also see mismatched DSCP values when the DSCP value seen by the target differs from the one used by the sources. View how DSCP changes at every hop by hovering over each node. In this case all sources have mismatched DSCP since the agents are sending packets with Expedited Forwarding (EF) but packets are actually going through the public Internet and ISPs are resetting the DSCP header.

Check our shared link for this test where you can interact with the visualization. If you’d like to chat with us during live Q&A, join our VoIP webinar on Tuesday September 30th. And if you’re interested in trying out voice testing, join the Voice Beta.

You can also watch the video and accompanying slides below where I demo our new VoIP tests at the Network Field Day 8.