(Accedian Networks') EtherNEWS Blog

A recent MEF session offered Ethernet Operations Administrations & Maintenance (OAM) Lessons Learned from the trenches at Verizon’s Interoperability Forum (VIF).  The VIF is where multiple vendors’ Ethernet gear gets connected together and stressed out by some of the most talented network engineers on a mission – to test how well today’s implementations of OAM work together, where there are gaps in the standards, misunderstandings, shortcoming and otherwise disconnection in this important step in Ethernet’s road to adulthood.

The OAM standards are designed to bring simplicity in management, fault notification and performance monitoring to Ethernet, making it more like TDM – reliable, resilient, carrier-grade.  This is no small feat considering the various topologies, technologies, equipment capabilities and the number of non-deterministic factors that affect packets as they attempt to cross the Ether.  In addition to connectivity fault management (CFM) specified in IEEE 802.1ag & ITU-T Y.1731, the latter also specifies measurements for performance monitoring (PM) including frame loss ratio, frame delay, delay variation (jitter), throughput and (soon) availability.

The theory has it that once these capabilities are implemented on all Ethernet-able elements, providers will easily be able to detect, isolate, troubleshoot and correct any common fault in an Ethernet Virtual Circuit (EVC), even if the route spans multiple vendors’ equipment and multi-operator networks.  A hierarchy of alarms ensures operators have the information they need to fix an issue, while providers using these peered networks for transport or last mile access are notified of the issue – allowing them to reroute traffic over alternative routes.  In theory.

The Scorecard

So how are we doing as a vendor community?  Conclusions from the first phase of VIF testing (EVPL) offer mixed reviews.  With nine vendors in the mix, “Most vendors still have issues but some level of interoperability was achieved” when the basic loopback, link trace and continuity check message functionality was put to the test.

Various failure scenarios revealed difficulty configuring unlike equipment and issues with protocol priorities.  A general need for management systems to automate OAM provisioning was a central issue, as was problematic auto-discovery resulting in elements being added to the wrong maintenance association (a logical management group of links between elements).  Not surprisingly the 802.1ag and Y.1731 standards for CFM and link trace functions don’t interoperate properly due to slight protocol differences.  Perhaps most concerning, most vendors don’t support continuity check messages at sub-second frequency, key to delivering 99.999% reliability.

So how big a mess is this?  Not enough to call it quits.  The same level of interoperability issues plagues any new technology – a similar multi-year pain-period occurred when GFP, LCAS and VCAT emerged to efficiently transport Ethernet over TDM.  It may be that the standards will have to be adjusted, and the basic software approach that vendors have taken to enable OAM in their elements will need to be enhanced, upgraded, or replaced by dedicated hardware to provide a useful level of performance.

So nothing unexpected, but certainly disappointing to service providers who are counting on OAM to simplify and open up the Ethernet wholesale market.  For providers looking to get a standards-based head-start into the market, Accedian Networks offers solutions that work today and that are evolving with the standards, while also providing integrated performance monitoring at layer 3.  You can learn all about Ethernet OAM, its strengths and shortcomings in the video overview available at www.Accedian.com/oam.

My previous post about traffic micro-bursting explained the adverse impact these transient bandwidth spikes have on business-critical Ethernet and IP/MPLS access links supporting financial trading and other time-sensitive applications. But just like it’s hard to leave your work at the office, micro-bursting has followed us home – and that’s bad news for ISPs, telcos and cable MSOs serving up residential broadband.

In a presentation at the SCTE Conference on Emerging Technologies recently held in Washington, DC, Carol Ansley, the Director of IP Engineering at Arris showed how streaming video delivery from popular sites has changed dramatically over the last several months. Crying wolf? Yes – but justifiably so – a big one just arrived!

Streaming Video Download Bandwidth Usage Evolution

 These graphs she captured from a cable modem accessing a “leading video streaming site” shows just how quickly and significantly download traffic profiles are changing. In the first graph a video is being slowly (peak 1.5Mbps) and steadily being loaded into a browser. In the second graph, recorded less than a year later, the same video is being delivered in occasional micro-bursts approaching 30 Mbps (20x greater peak bandwidth)!

Carol pointed out that despite numerous inquiries to service providers and the video content sites themselves, no one could clearly explain why videos are now being delivered in short, sharp bursts compared to the steady flows from less than a year ago. What is clear, however, is that with video traffic increasing at a rapid rate, servers, content delivery networks, cloud-computing and caching resources are straining to keep up.

And this could explain what we’re seeing.

Servers using preemptive multitasking – a technique where equal slices of processing time are allocated to each thread – are handling so many flows concurrently that the time-slice dedicated to each end user is becoming ever shorter. This would result in micro-bursts as servers attempt to keep up with thousands of downloads by shooting out streaming data as fast as possible within the short time allocated to each end-user.

The cause may be open for speculation, but whatever the reason, there’s no sign of download patterns returning to the moderate flows of the past. Increasingly online videos are encoded in HD formats, and selection is growing to include more feature-length films and complete TV shows as all camps from CNN to Google compete for the almighty surfer’s attention.

This is definitely a problem if you’re a broadband provider. Quality of Service and network performance are going to take a big hit very soon if this goes unchecked. As more and more users download videos, there’s a greater chance that not only will link capacity to each home not support these data rates, the aggregated micro-bursts will start to coincide together, causing massive packet loss in HFC, DSL, and FTTx access networks. Imagine a cable loop or DSLAM supporting 50 customers, each watching a video delivered in micro-bursts, combined traffic peaks could exceed 2.5Gbps. An unlikely worst case? Maybe, but ever more likely as over-the-top video delivery dynamics continue to shift.

The Fix That Fits

One way operators can effectively address micro-bursts is through high-performance, hardware-based traffic shaping applied in the downstream direction – ideally at each CMTS or DSLAM in bandwidth-limited cable and DSL networks. Hardware-based processing is required because standard shaping offered by most network elements places significant strain on the shared network processor, causing noticeable latency, extensive buffering and packet loss. Dedicated hardware-based shapers, available in the latest generation of Network Interface Devices (unconcerned with complex switching tasks), provide near-zero latency pass-through for real-time flows like VoIP, while efficiently shaping lower-priority traffic. This effectively smoothes out the traffic to squeeze bursts into the limited capacity of the access network. Applying shaping to streaming downloads would have no visible effect to the end user, while assuring QoS for all other applications sharing the network.

You can watch a video of our advanced traffic shaping in action from our lab at www.Accedian.com/shaping.