The Deceptive Nature of Metrics
By ALAN NICOL, Executive Member, AlanNicolSolutions
Most of us can’t, won’t and don’t operate our businesses, or our specific corners of a business, without metrics. Metrics are important. However, we must remain vigilant against the misleading potential of metrics and should establish behaviors to guard against metrics-driven mistakes.
Before I point out some of the hazards we face with our metrics, let me say that I too believe that metrics are important and necessary. We need some way of measuring progress, setting alarms, and sometimes for driving behavior.
Metrics can also cause trouble. The problem is that metrics are not perfect. As much as we would like them to represent the truth, metrics are not the truth. They are just a small still-picture taken through a small window pointed at some piece of the truth.
The statistician Dr. David Cox stated one of my favorite quotable phrases. He said that, “All models are wrong; some models are useful.” He was talking about mathematical models and simulations. I find that the same holds true for metrics.
Let me provide a thought exercise to explain why. Metrics are simply a measure of something that we collect periodically in order to gage progress or performance. Given that, let’s use an example measurement that is very plain and easy to perceive. Let’s just talk physical dimensions.
Imagine that we have a business of making certain objects. The size of these objects is important. We want to know if the size of the objects is changing and we want to ensure that the objects are produced to the correct size.
We decide that the easiest thing to measure is length, so we begin measuring the length of the objects. We make a-lot of them and it is time consuming to measure them all, so we measure only every 20th piece. We accept that we won’t know everything, just every 20th opportunity. Sound reasonable? Good we have a metric.
Soon, however, someone points out that there is more to size than just length. So we start measuring width and height as well. We decide since we are measuring more stuff that we’ll measure randomly at certain time intervals in order to keep our measurements pragmatic. Great, now our measurements are less often, but more comprehensive.
But, wait, there’s more to size than 3 dimensions, there are also weight or mass and volume. We could weigh each piece, which requires all new measurement equipment. That could give us even more detail.
What if our pieces aren’t a simple shape like a block or a cylinder? What if we are measuring lima beans? We could drop a measured piece into a water tank and measure its displaced volume. That would give us an accurate volume, but we can’t replace our dimensional measurements with the water tank because something out of specification for one of those measurements could still have a correct overall volume.
While we are debating the best way to optimize our metric for size, our metrics are magically improving. We find that our business is lending credence to the observation that merely measuring something improves performance. Quantum physicists suspect that the mere act of measuring quantum particles changes their behavior. Guess what? People work that way too.
As we begin to have an in-our-face glimpse of piece quality, we improve our processes to make more parts pass muster. This is exactly the kind of behavior we want from measuring our performance.
In order to make the measuring process as expeditious and simple as possible those performing the task constructed a jig that quickly indicates if the piece is either too great in any direction or too small. They also learned that if they seat a piece in the jig just right it can either be helped to pass or fail. Since life is generally better for everyone when pieces pass than when they fail, more pass.
Thus, our metric is much like a quantum particle. The very act of measuring changes behavior. In fact the subject of how metrics drive or change behavior is worthy of a detailed book and some psychoanalysis; more than we will discuss here.
Something as simple as measuring size so we can have a picture of how well our product is melting, expectations has exploded into something very complex. The effort required to get a true picture of what is going on requires so much data, and measurement process, and analysis, and behavior management that it becomes impractical. All we really wanted was a simple measurement.
That last statement is the key to keeping metrics in perspective: When determining or designing metrics, begin with the need in mind. Answer the question, “What do we want to know?” If you want to know the truth, the details of performance, you must go forth and see it for yourself, but even then, the mere act of standing there watching will modify peoples’ behavior.
Metrics will not tell the whole story. Understand this so that when designing metrics you can accept that they will be imperfect. They don’t need to tell you the truth; they just need to tell you enough to make an intelligent and informed decision. They only need to be useful.
I’ll share a quick story to make a final couple of points. A friend and colleague of mine recently placed in charge of upgrading his business’ test systems with some automated test equipment. It sounds like it was a unanimous leadership decision to go forward with the initiative, but since one functional leader is responsible for the test systems, that leader needed a solid business case to justify the project.
My friend went through the motions of putting together some estimates of money that would be saved in a number of different ways by upgrading equipment and automating some tests. He also received a great deal of input from numerous other leaders and colleagues in the form of advice concerning how to calculate some of the savings.
He presented his calculations to the functional leader who dismissed most of them, including those the other leaders were certain he wanted. In the end, he would only accept a single calculation based on the elimination of re-tests. The added profits from shorter product development cycles was too confounded with too many contributing or influencing factors for him to accept, and so were several others.
It didn’t much matter because the calculation desired showed that the equipment would pay for itself quickly enough to justify moving forward. My friend, however, might have been disappointed if his bonus at the end of the year depended on how much he saved the company with the project.
I won’t start an argument here. I don’t need to in order to make some observations or points. I offer the following:
- The purpose of metrics is to help us make decisions, but in this case one was created to help rationalize a decision that was already made.
- In this tale, established metrics concerning test performance might have saved my friend a great deal of research and analysis.
- Not everyone likes the same metric and we often drive a great deal of wasted effort when different people ask for different calculations.
- Managers and leaders are just as prone to mess with metrics as those who make the parts or collect the measurements.
- One person’s “golden metric” can be another person’s curse once responsibility and accountability come into play.
So, based on the example, the short story, and the listed observations, let me offer the following suggestions. I hope that they are helpful.
Make it well known and understood that metrics are, by nature, imperfect. Focus on what you need them to indicate and keep them as simple as possible to fulfill the purpose. Generally we want to know when something is changing or, alternatively, when something is not changing.
Don’t try to know everything about that something. Just give yourself an indication that a change of a critical parameter has occurred. In terms of monitoring for change I’m a big fan of control charts. They prevent us from overreacting to natural fluctuations and help us prove that change has genuinely occurred.
Hold people accountable for the quality of the metric, but be cautious about holding people accountable for the actual metric value. It’s common to incentivize performance and behavior with metrics-based goals. It’s just as common for those metrics to be skewed or manipulated in a favorable direction, or for people to be punished for performance outcomes that are beyond their control.
Conflicting metrics, such as speed or volume vs. quality or yield are a common way of trying to prevent our natural drive to improve one performance from sacrificing another important performance. This is a good practice. Keep it simple. Be cautious of the other points identified herein.
Metrics are useful for monitoring and eliminating waste, but the very practice of collecting metrics is wasteful, especially if they are always changing or if they try to measure too much. Focus only on the critical measures.
Try to catch yourself and your peers when the desire to rationalize a decision turns into a discussion of metrics. Chances are, any decision already made can be justified by some invented metric or measure. It is in our business culture for us to justify our decisions with solid business rationale. So be it. Be sure that your calculations provide evidence. Don’t invent rationale to justify.
In my friend’s story above, it sounds like his functional leader might have done right to demand a inarguable calculation that indicated business payoff for the investment and ignored all of the other calculations that might have painted a more glorified, but more arguable business case. After all, if everyone intuitively knows it’s the right thing to do, a simple explanation similar to what follows should be enough.
“This one calculation shows that we’ll pay off our investment in 16 months. We know that other savings and revenue are probable and the payoff might be sooner. We should do this.”
Finally, if you will be driving decisions with metrics, or if you have or desire a metrics-based business culture, standardize your metrics. Make everyone collect and answer to the same metrics. Eliminate the inefficiency of constantly changing metrics and over-measured processes. Doing so also eliminates many arguments that lead to wasteful indecision.
Metrics should be simple, but they aren’t. Simple or complex, they are always imperfect. Know it. Understand it. Communicate it. Accept that metrics are imperfect snapshots of limited perspective. Collect only what you need to monitor your critical performance and make educated decisions. Make sure your metrics are indicating what you need to know. Keep them simple. Standardize them as much as possible, and be careful about hanging peoples’ careers on imperfect measurements of performance. Use them wisely.
Stay wise, friends.
For more information, please visit www.bizwizwithin.com.