(Accedian Networks') EtherNEWS Blog

Teletimes published this Feature Article in their current edition: download original Teletimes backhaul article.

Mobile Carrier Ethernet backhaul services from network operators, and the advent of packet based mobile systems, are predicted to provide mobile operators with scalable and more cost efficient solutions for handling both the increasing number of mobile devices attached to their networks and the traffic volumes they generate.

A major reason for the increased use of Carrier Ethernet in wireless backhaul applications is the ability to use a diverse physical infra­structure to deliver Carrier Ethernet to the base station. The physical delivery mecha­nisms for Carrier Ethernet include:

• Ethernet over copper
• Ethernet over fiber (both dark fiber and over SONET/SDH)
• Ethernet over bonded copper
• Ethernet over radio (microwave)
• Ethernet over PON (Passive Optical Networks)

Low latency is the key to delivering reliable, high-performance backhaul for 3G and 4G wireless networks. Real-time communications, transactional applications, high-speed roaming, and media streaming are all delay-sensitive. Latency increases of just a few milliseconds can result in dropped calls, garbled voice and unresponsive applications, and can mean significant losses in financial trading.

At times, service providers over-provide bandwidth to keep latency and jitter in check. While increasing bandwidth can sometimes reduce latency, it often has little effect. In packet-based networks the relationship between latency and bandwidth is complex and varied. Consider the four main sources of latency, categorized as:

• Serialization delay: time required for a port to transmit a packet, related to frame size and bit-rate;
• Propagation delay: limita­tions imposed by the laws of physics (speed of light, path length, circuit design);
• Congestion delay: the time a frame idles in the output queue of a network element (NE) while a backlog of packets is being transmitted. Congestion delay can be caused by traffic bursts, larger ingress vs. egress bandwidth (e.g. oversub­scribed aggregation), or due to network congestion resulting in paused trans­mission (flow control).
• Forwarding delay: the time required for the Network Element (NE) to analyze, process and forward a packet in a congestion-free scenario; a function of NE architecture and packet-processing requirements (the number and complexity of operations performed on a packet between receipt and transmission, e.g. service mapping, switching, rate limiting, shaping, etc).

Of these components, serialization delay is the most constant, having only a small influence on end-to-end latency. Propagation delay, typically stable in circuit-switched networks, can be irregular and introduce jitter over routed networks due to path variation; overall, its contribution is usually small, even under heavy utilization.

The more important sources of latency – congestion and forwarding delay – are not entirely independent: as a NE is subject to heavy load (conges­tion delay), it may need additional queue time to handle and process the increased volume of traffic (forwarding delay). Depend­ing on the NE’s design, forwarding delay can be significant when advanced functions such as traffic shaping and multi-flow Ethernet OAM (Operations Administration & Mainte­nance) are enabled.

As network congestion can have a large impact on end-to-end latency – impacting both forwarding and pure conges­tion (queuing-related) delay -reducing traffic bottlenecks is a key part of network manage­ment and design. Increasing capacity (available band­width) should, at least in theory, help reduce congestion when applied to network “pinch points”. However, increasing throughput does not always lead to the expected decrease in latency, even if congestion is reduced. Results will vary depending on implementation, network architecture, traffic patterns, and a number of other factors explored here.

Bandwidth vs. Media Rate

Although Ethernet offers easy access to bandwidth on demand, “dialing up” through­put often has little effect on latency if the link was origi­nally sized correctly. Consider a cell tower connected to an access platform using a 100FX optical link, with a CIR (Committed Information Rate, enforced by rate limiting) of 20 Mbps. If tower traffic never exceeds 20 Mbps, increasing the throughput to 30, 50, or even 100 Mbps will have no effect on latency. Why? Although capacity may have increased, each packet is still bound to the physical link speed negotiated between ports (e.g. 100 Mbps for 100FX media). This means the latency (time of flight) is the same as long as the physical link remains the same.

Dialing-up bandwidth can get more data across a link quicker (a file downloads faster), but each individual packet still travels at the same speed. This is analogous to adding more lanes to a highway without changing the speed limit: unless there is congestion, time to travel from A to B remains the same. Instead of increas­ing the CIR, the media needs to connect at a faster rate to reduce delay, in this example, changing from the 100FX to a GbE interface would reduce latency ten-fold. In many cases this requires replacing or upgrading network equipment, an expensive and time-consuming option.

Congestion: Access vs. End-to-End Throughput

When congestion is present, increasing bandwidth will reduce latency, but only if throughput is increased end-to-end. For example, if Ethernet-over-SONET/SDH (EoS) is used for transport, increasing the media of the last mile will have little effect on overall latency if packets are still carried over the same TDM “container” (e.g. DS3 / E3). This is analogous to the Ethernet media rate effect: the rate of traffic entering the network has no effect on latency if TDM / core provisioning remains unchanged.

Likewise in all-packet net­works (e.g. MPLS core), if the end-to-end network doesn’t have sufficient capacity, packets will become congested in the core instead of the access network, simply displacing the problem elsewhere. When this occurs, increasing access bandwidth may result in even longer delays, as already busy core NEs add more traffic to their queues and processing load.

Stability Uncertain

Latency is likely to change over time, even in the best-designed networks. Increased oversubscription, new subscribers, changing traffic patterns and increased service usage can all introduce congestion and delay. Latency can also increase suddenly if a link fails and a protected path is required. In microwave links, inclement weather and radio issues also affect delay. As no service, subscriber base or user behavior is constant, so latency (and jitter) will vary over time.

Flow-Specific Latency

Not all packets are created equal. Critical applications and services typically traverse the network with a higher priority than best-effort traffic, as maintaining low latency and jitter is required for these applications’ quality of service (QoS). However, networks do not always respect or maintain a packet’s priority. Some network elements discard or alter the specified class of service (CoS), or re-map Ethernet CoS to the IP equivalent (DSCP), or vice-versa, when bridging layer 2 or 3 network segments. Errors in NE configuration, firmware, and service class mapping rules often result in latency issues, which are commonly intro­duced when new services or traffic patterns require NEs to be reconfigured.

Optimizing Latency Performance

Making Latency Visible

The causes of latency are complex and non-deterministic. To maintain overall QoS, latency and jitter need to be continuously monitored on a per-service, application, SLA or VLAN-basis. Monitoring latency using coarse port-level or software-based methods (e.g. ping), fail to identify latency issues affecting specific traffic classes, and are unable to isolate whether latency originates at the IP or Ethernet layer.

Likewise, monitoring latency round-trip is insufficient for SLA reporting and trouble­shooting delay issues in networks where traffic is often asymmetrical; one-way latency and jitter measurements are required. To provide a complete view of latency performance, measurements should also span the complete service path, end-to-end (the demarcation point and at key nodes in-between). This allows operations staff to isolate latency issues to access, metro, core networks – or perhaps more importantly -the customer’s network.

Latency measurements need sufficient precision to detect subtle changes in delay to proactively identify latency drift that can lead to QoS issues and SLA violations. Precision should be at least one order of magnitude greater than the target threshold to be useful (e.g. sub-millisecond if an SLA specifies a 10ms latency maximum), otherwise measurement error will mask delay issues, create false warnings, or even result in negative latencies in customer SLA reports!

Similarly, measurements need sufficient granularity (fre­quency) to detect short-term latency problems, which can indicate micro-bursting, ineffective traffic management practices, or mis-configured NEs. The ability to measure each second, for example, provides sufficient information to diagnose intermittent QoS issues.

In summary, to provide complete visibility into latency issues, measurements should:

• be conducted per-flow, application, service or VLAN, not simply per-port;
• be conducted one-way, as opposed to round-trip;
• measure latency end-to-end to accurately account for all sections of the network, as well as to key intermediate nodes for problem isolation;
• offer sufficient precision to detect gradual shifts in latency, and to provide meaningful  alarm  and reporting values;
• Provide high granularity if required to troubleshoot transient, delay-related issues.

Validating Congestion

Before trying to correct latency by adding bandwidth, it’s important to first determine if and where congestion is occurring in the service path. As provisioning additional bandwidth is time-consuming and expensive from a network perspective, it is best increased where it will most improve performance.

Per-flow packet loss is a good indication of congestion, and can normally be measured along with latency using non-intrusive active testing. As good as packet-loss measure­ments are at identifying network bottlenecks, they cannot be used to describe the impact congestion is having on customer throughput, or on the latency of a particular application or flow.

In-Service throughput testing is a unique technique that allows providers to determine the capacity available over a path, which can then be compared to a customer’s OR to determine the level of oversubscription present in the network. As in-service throughput testing has no impact on customer traffic, it can be conducted on-demand when issues are identified, or during peak traffic hours to accurately assess a service under worst-case conditions.

Based on the RFC-2544 throughput test, in-service throughput testing outputs latency, jitter, packet loss and throughput as measured one-­way over the precise path a customer’s traffic traverses. When used with long-term latency monitoring, in-service throughput testing can be a valuable tool in maintaining QoS and optimizing network design.

Optimize at the Edge

We’ve explored how latency issues can be displaced from one section of a network to another as link capacity and parameters are changed. From a network efficiency perspec­tive, it is better to optimize latency, bandwidth usage and traffic priority as close to the service demarcation point as possible. This ensures the best-possible QoS end-to-end, as intermediate NEs have less traffic to manage. Core and aggregation NEs will have reduced forwarding delay as processing requirements are shifted to the network edge, as well as lower congestion-related delays as less traffic will be queued at all nodes across the network.

Effective traffic management is best accomplished using hierarchical QoS (H-QoS), a combination of per-flow rate limiting (bandwidth policy enforcement), filtering and traffic shaping implemented against a service priority hierarchy.   To ensure that H-QoS does not impact the latency it aims to improve, it is critical that the NE offers ultra-low pass-through forwarding delay as well as hardware-based shaping performance. As classifying traffic into service flows is required to apply H-QoS, the unit should also have versatile traffic classification and priority marking capabilities. This may involve defining as many as 500 unique flows for Ethernet wholesale hand-offs.

When properly applied, H-QoS can greatly reduce latency by ensuring critical traffic is identified and handled with sufficient priority, reducing packet loss (and re-transmission) through effective shaping methods, and limiting ingress traffic to each service’s particular bandwidth profile. A key benefit of H-QoS is that latency for critical applications can often be greatly improved without any increase in access or core bandwidth.

To summarize the key elements of effective latency and QoS optimization:

• establish per-service priority and bandwidth as close to the edge as possible;
• apply H-QoS using a hardware-based device with ultra-low forwarding latency;
• Classify traffic as granularly as possible to ensure each application is given its correct position in the service hierarchy.

Using NIDs to Improve Latency & Overall QoS

Network Interface Devices (NIDs), provide a cost-effective monitoring and traffic optimization   solution. Typically installed at customer premises, cell sites and inter-carrier handoff points, NIDs are uniquely positioned to provide an end-to-end view of the network. When installed at key aggregation and switching locations, integrated test capabilities map out the health of the entire network to deliver per-flow visibility for all key QoS and SLA parameters. If the NIDs also feature embedded H-QoS functionality, a provider not only sees the performance of each service, but can precisely tailor it to meet individual customer and service requirements.

Accedian Networks has developed a line of EtherNID™ & Metro NID™ packet assurance demarcation units that were designed to address all of these requirements, while also integrating Ethernet OAM and MEF-certified Carrier Ethernet service mapping functionality. Based exclusively on a dedicated silicon, all-hardware design, these units’ Fast-Thru™ packet processing engine offers forwarding latency and jitter of 3-3 and 0.1 microseconds, respectively, eliminating the delays associated with most NEs’ network processor-based, store-and-forward architectures.

These devices are capable of high-density H-QoS on full line-rate traffic, can monitor up to 100 flows at layer 2 or 3, perform in-service through­put testing, and establish or terminate MEF-certified E-Line and E-LAN services. A unique remote clock synchronization technique provides one-way delay and jitter measurements to 1 microsec­ond resolution, even over geographically diverse, multi-technology, multi-vendor and multi-operator networks. Tests can be conducted to up to 100 remote sites from each NID, over point-to-point, hub-and-spoke, and full-mesh unicast or multicast services.

A zero-latency traffic shaping feature ensures that no jitter or latency is added to critical traffic, while minimizing packet loss and queuing delays for lower-priority flows. With real-time traffic classification, the units combine rate limiting and shaping into a sophisticated H-QoS function that is currently used in critical, delay-sensitive applications such as financial trading, cellular backhaul, media delivery and large enterprise business services.

A performance monitoring capability is needed to truly monitor each NID and its performance. One leader in this field is Accedian who has also developed the comple­mentary EchoVault SLA EMS performance monitoring platform efficiently collects measurements and traffic statistics from each NID deployed to produce a real­-time view into network, circuit or service performance with per-second visibility. Flexible service thresholds and tolerances provide early indication when latency or other SLA parameters start to degrade, as well as hard fault notification for service outages or QoS faults. Service availability and other service statistics are calculated over selected time intervals for trending, baselining and graphing, presented by percentile, max, min and average values to allow quick problem detection. The integrated web portal engine allows service providers to quickly setup secure online SLA reporting sites for each customer, showing only the metrics they need to confirm their services are functioning within spec.

Tags: 3g, 4g, backhaul, carrier ethernet, jitter, low latency, lte, NID, OAM, Service Assurance, SLA Assurance, SLA Monitoring, wimax, wireless backhaul, y.1731

This entry was posted on Tuesday, February 16th, 2010 at 3:54 pm and is filed under Applications, News, Solutions. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.