Metrics, metrics everywhere! Sorting out good, bad, and useless
By Francoise Tourniaire 

Last week a client showed me the metrics he’s putting together as part of an ambitious knowledge management project. He had a simple, easy-to-read spreadsheet with a handful of columns with enticing names: KB usage, customer satisfaction, new documents, etc. The usage numbers even had an attractive green shading to show the threshold had been met.
His question was: Are these the “right” metrics? And the conversation got more interesting from there.

Are you measuring the right things?

Stop looking at numbers for just a moment. What kinds of signs would you watch for to make sure your knowledge management initiative is working? You’re on the right track if you target activity in document creation and maintenance, whether customers are using it, and whether it actually resolves customer issues. Limiting yourself to just one dimension would give you a lopsided and ultimately inaccurate view. You could have a fast-growing, well-maintained knowledge base that no one uses.

Tip: Create a balanced view. For knowledge base metrics: usage, effectiveness, and document maintenance are probably the right choices.

Are you measuring outcome or process?

Metrics can multiply to the point that they become confusing, and even useless to their recipients. The current vogue of dashboards is linked to this desire to simplify, and that’s a good thing for the most part. (I have serious qualms about dashboards when they try to reduce a complex function to just one number. What’s the one number that will tell you how well your knowledge management solution is working? I say there is no such number.)

A good way to reduce excessive metrics is to distinguish between outcome and process metrics. Outcomes are what you report outside the organization, the deliverables of the effort. For knowledge management, outcome metrics are usage and effectiveness. If people use the knowledge base and solve problems with it, knowledge management is working.

If you’re responsible for the knowledge base, outcome metrics aren’t enough. You need to know how the supporting processes are working: how many documents were created this week, how fast are documents moving through the publishing workflow, how old documents are purged. Such process metrics are very important for hands-on managers, but they shouldn’t be needed by executives outside the organization.

Tip: Separate outcome metrics from process metrics.

Can you trust the underlying data?

Beware of running statistics on bad underlying data. With the simple tool my client uses for creating and maintaining knowledge, it’s possible to count how many documents were created, reviewed, and published in a given week, but it’s very difficult to measure how long it takes for documents to go through the publishing workflow. Until he upgrades the tool, my recommendation is to forget about detailed workflow metrics: just watch the backlog and dive into the system manually on a regular basis.

Tip: Don’t try to run metrics on non-existent or shaky data.

Are the measurements meaningful?

This is the tricky part. Even with solid underlying data, you can easily create meaningless metrics. For example, my client wants to measure the effectiveness of the knowledge base. One metric he selected is the average rating of documents by customers. Sounds good. Well, only 2% of customers bother to rate documents. With the ratings based on such a small sampling, the average is meaningless. (Note that the ratings themselves aren’t without meaning. For example, documents with poor ratings should be reviewed and improved.)
A better way to measure effectiveness would be to track whether customers who use the knowledge base on a particular day go on to log a case on the same day. This would work well for my client since his customers are all on support contracts and need to log in to access the knowledge base. Matching knowledge base records with case-tracking records will do the trick.

Another example of a meaningless metric is measuring the hit rate on knowledge base documents: yes, more visits mean more hits, but more hits can also mean that customers can’t find what they want. It’s better to measure the number of unique visits.

Tip: Make sure there is a direct connection between the metric and the behavior you are trying to capture.

Are the metrics easy to understand?

After passing all the previous tests (measuring the right things, separating outcome and process metrics, using sound data, and using meaningful measurements) all you need to do is package the metrics in an easy-to-use report. That shouldn’t be too difficult since you won’t have more than a handful of numbers to display. You don’t need to be fancy: my client’s humble spreadsheet is effective, although I would recommend using a graph to show performance over time.

Finally, make sure that the recipients of the metrics understand them the way you intend them to be understood. A short training session and a few targeted footnotes can make a big difference. If you measure usage by counting knowledge base sessions, label that column “KB Sessions” rather than something vague like “Usage”.

Tip: Periodically check the recipients’ understanding of the metrics.

Francoise Tourniaire is the founder and principal of FT Works, a consulting firm that helps technology companies create and grow their support operations. She is the author of “Best Practices for Support Metrics”, a practical guide to creating meaningful metrics for support organizations. For more information, visit www.ftworks.com or call 650 559 9826.

How do you handle price increases to your support maintenance?
› View Answer