Improve Customer Experience: Design-Deliver-
Operate & Optimize
By Mack Coulibaly, Director, Technical Services, Customer Advocacy, Cisco Systems, Inc.

Customer experience is not just product experience. It encompasses all touch points including how well the product is designed (Design), the customer’s expectations of the product (Delivery), and the product’s ability to sustain operations with minimal disruption (Operate & Optimize). In this article, you will learn how Cisco is developing an integrated methodology to compute performance indicators that measure and improve the customer experience. 

Performance Indicators for Customer Experience

Background

Metrics such as "Mean-Time-Between Failures" or "Network Availability" have been used with a varying degree of success. For infrastructure such as IT and Telecom, it is critical that there be a common set of basic methods in place to measure performance, and hence, provide a benchmark of shareholder value. Cisco Systems is leading the industry by developing an integrated methodology to measure a set of performance indicators (xPI) in order to gauge the customer experience. These indicators provide a benchmark for measuring product performance (hardware, software and services) and set the basis for equivalence comparison amongst products, networks, and customers.

Components of the “Whole” Customer Experience

The customer experience encompasses all touch points including how well the product performs and how the product delivers on expectations. Also, it must include the experience of operating and optimizing the product to sustain business operations. The race to develop new capabilities to support the next generation of tools that go beyond “break-fix” and deliver genuine proactive ongoing customer interactions has already started. For companies across the industry, and especially in networking, this is the next frontier in achieving differentiated performance objectives for service and business operations.

At a macro level, the “Design Deliver Operate & Optimize (DDO)” concept recognizes that customer experience is influenced by three components which form the interactions between customer and provider. The link between these three components is impactful to the customers' operations, therefore, the implementation described herein uses “disruption to operations” as the common unit of measure.

Customer Experience Influenced by Three Components

Design: The ‘Design’ component measures product performance and suggests whether design improvements are required to meet customer needs.

Deliver: The ‘Deliver’ component measures the effectiveness of the organization's delivery of products & services to the end customer. This is where customer expectations of products, services, support and maintenance are set.

Operate & Optimize: Here, the effectiveness of products and services are gauged by measuring disruption (or lack of it) to operations. This implicitly measures such things as technical abilities and operational processes and challenges encountered by the customer.

This article focuses on the ‘Design’ and ‘Operate& Optimize’ side of the model.

Successes and Limitations of Traditional Network Performance Measurements

Before we dive into disruption and the new set of performance indicators, let’s take a quick look at widely used metrics.

Mean-Time-Between-Failure (MTBF)
Mean time between failures is a common unit to measure reliability of a part, a component or a product. MTBF is the average expected interval between failures of a product in steady state. A potential mis-application of the MTBF metric is using it to specify an average time when the likelihood of failure equals the likelihood of not having a failure. In fact, this is only true for certain symmetric distributions. As a result, many industries have called into question the accuracy of the MTBF’s application to real systems.

Availability
The availability percent of "number of nines" ‘N’ describes a system which is available a fraction ‘A’ of the time: N = - \log_10(1 - A)

The illusion of the ‘nines’ is the implicit assumption that if the system is operating 99.9999% of the time, then business or application is operating 99.9999% of the time. It is now well understood that a machine outage of one minute can cause a business operation ‘disruption’ of hours.

Mean-Time-To-Repair (MTTR)
MTTR is another factor that determines network availability. MTTR is the average time it takes the operations staff to diagnose, isolate, remove, and replace a failed system and then restore full functionality to the network. It is obvious that MTTR might vary substantially based on operational processes and capabilities.

Network Availability
Network Availability (NA) is defined as MTBF divided by the sum of the MTBF and MTTR. Network equipment with a MTBF of 100,000 hours and with an MTTR of 4 hours has a network availability of 99.996 as illustrated below:

This implies that a vendor or service provider who offers a service level agreement (SLA) based on network availability of 99.99% is indicating that it expects a failure to occur, on average, every 45,000 hours if we assume a system with MTTR of 4 hours (one failure every 5 years).

How does this translate to disruption of customers' operations?

Using Disruption to Evolve Existing Customer Metrics

Today’s networks are a clever design of asymmetrical engineering components composed of highly varied and intelligent devices. Any assessment of that environment must include:

• Design considerations: No two networks are the same (product mix, configurations, software, line card mixes, processes, applications, etc…)

• Operational Considerations: Maturity of the network and the people who design and support networks. The “IT Capability Maturity Model” provides a common means of assessing the overall process maturity.

The Disruption algorithm (Cisco proprietary) is a measure relative to the impact of a failure or maintenance event. It provides a new way of assessing performance and gauging customer experience. Disruption can be thought of as impact to business operations or impact to support teams.

The computation for Disruption includes:

• Age (time it takes to resolve the issue). This is relevant because it theoretically represents how long the customer was impacted by the effect of the issue.
• Severity (what is the perceived gravity of the issue 1 to 4 scale). This captures the perceived impact of the issue on the customer.
• Escalation (was there a change in severity). The customer is relatively more impacted or otherwise frustrated if during the time when the issue is open, he or she requests an increase in severity, priority or an escalation of the issue.
• Bugs (was the issue caused by a bug). The vendor or the support company determines whether the issue is caused by a bug, defect or an operator error. This is the only parameter set by the provider.
• Outage (outage occurs as a result i.e. users lost the ability to work). Here, outage is simply defined as the "inability or severely limited ability to operate”

Except for Bugs, all the parameters of Disruption are elements of operational experience of the customer. Hence, the magnitude of the Disruption is a good gauge of what happened to the customer.

The best implementation is a tight integration with CRM, ERP and Services portfolio to provide the basis for equivalence performance comparison, aggregation and normalization at the service contract, product, network, or vertical market level.

Examples of New Measurements based on Disruption

Using Disruption, we created the Product Disruption Index and the Network Disruption Index. The first measures the propensity of a product to induce disruption in a network. The second is the propensity of a network to have disruptive issues.

Some of the suggested indicators are:

Aggregate Network Disruption (AND): Weekly summation of all Disruption in a network

figure 1: Aggregate Network Disruption
figure 1: Aggregate Network Disruption

Interpretation: Network is relatively stable after an initial increase in disruption between August and December.

Relative Network Disruption (RND): The rate of change (percentage) between the magnitude of Disruption on the network from 90 days ago and that of today. (Could be monthly)

Interpretation: Below zero indicates that network is doing better than it 90 days ago.

figure 2: Relative Network Disruption
figure 2: Relative Network Disruption

Critical Network Disruption: The percentage of the aggregate Disruption that is caused by network outage. This is a way to discriminate Disruption by the impact of events.

figure 3: Relative Network Disruption
figure 3: Relative Network Disruption

Interpretation: Customer A has a more stable and predictable network than Customer B. We could as well compare customer A against the vertical segment.

Product Disruption Index (PDI): The propensity of a product to induced Disruption in operations.

figure 4: Product Disruption Index (Hardware)
figure 4: Product Disruption Index (Hardware)

Interpretation: The lower the PDI, the better the performance. Product A is stable with a relatively high disruption propensity. Product D is stable with a relatively low disruption propensity. Product B has the pattern of a new product. It has an unstable disruption pattern.

figure 5: Product Disruption Index (Software)
figure 5: Product Disruption Index (Software)

Interpretation: Software Version 1 is older and more mature. Its disruption pattern is lowering. Software version 2 is less than a year old. Its disruption pattern is stabilizing at a higher level of disruptive propensity. Software version 3 is about six month old and beginning to stabilize at the same level as the more mature version 1.

A composite Network Disruption Index (NDI) can be computed to represent a listing of products (hardware, software and service), and a statistic reflecting the composite disruption value of its components. This can be used as a tool to represent the characteristics of its component products, all of which bearing some commonality such as being used by a certain customer segment, running similar software, or having similar place in the network, etc. This index can be used to benchmark the performance of a customer’s product portfolio.

Benefits of Disruption-Based Performance Indicators

The core value proposition of Disruption-based performance indicators is the fact that they enable and foster a strategic relationship with the customer. The set of various performance indicators (xPI) are especially focused on the customer’s experience and allow for cross comparison across customers, networks, vertical segment, and services. They can be used to set benchmarks and measurement frameworks which can in turn be used to set joint organizational goals and measure improvements in customer experience. Finally, the disruptive performance in the field can be used to monitor and improve products design and supportability.

References:
IT Service Capability Maturity Model, Software Engineering Institute at Carnegie-Mellon University

Product Development Capability Model, Kenneth Crow, http://www.npd-solutions.com/cmm.html

.........................................................................................................

About the Author

Mack M. Coulibaly is Director of Technical Services in Customer Advocacy at Cisco Systems, Inc. He is an inventor and a published author with 16 years of experience in various technical fields ranging from telecom research, software development, hardware design, network engineering, and support. In 2000, he published the book titled Cisco IOS Software Releases: The Complete Reference. He also wrote several white papers including the widely distributed Cisco IOS Reference Guide.

Prior to Cisco Systems, Inc., Mr. Coulibaly worked for Cable & Wireless as Senior Network Design Engineer and for Citigroup as Network Configuration Manager. Mr. Coulibaly hold a BS in Electrsical Engineering (UDC), a MA eq. in Math & Physics (U. Abidjan) and an MBA in Technology Management (UOP).

Mr. Coulibaly can be reached at mcoulibaly@gmail.com