Making Meaningful Metrics Matter More

We’ve been talking a lot about dashboard metrics and making them meaningful. If you manage to accomplish this, you can get people to do some amazing things. It’s a way to provide clear direction, show progress, create impact in the right areas and have something to work toward besides raw financial success.

The downside is that a metric that no one can clear affect, no matter what impact it actually has, will either get ignored or acting on in an unintended way.

For example, at work we have a recycling indicator that measures recycling as a percentage of waste.

It’s likely easier to affect this indicator for us all to bring our recycling to work than to reduce our waste or recycle more of our waste. So, there is an odd incentive to control the indicator with unintended methods.

Food waste is likely our largest source of waste, by far. And, while I don’t know what the cafes are doing with composting, it’s not included in the indicator at all, and yet would probably have a much bigger impact to support a composting program (we also have gardens on site now, they would certainly benefit as well). Also, the output of the cafes is counted, but that’s essentially a different business, so we have influence there, but not control over their processes.

Lastly, there’s a serious problem of my own impact. I produce very little waste personally. And, I’m not alone. It’s very likely that this is a “long tail” problem, where a small minority of the population produces the majority of the waste. The awareness of our waste stream is great, but it’s not clear what most of us can or should do to improve the situation and meet the goal.

No indicator is perfect, and so it’s important to really think through the implications of setting a goal based on a calculation. And, sometimes it’s easier to measure something as a proxy indicator to measure something else. However, it’s important to recognize the difference between the proxy measure and the real measure.

A prime example of this is seen in measuring the country’s economic health. This has been a leading story for quite some time, and we use the Dow Jones Industrial Average as a central measure for the overall economy. Admittedly, it’s not a horrible proxy measure, but it’s definitely not really measuring the overall economy.

DJIA vs. S&P 500 - close correlation despite very little overlap 

The reason it’s a good proxy is that the stock market tends to self-correlate, meaning that, on average, all stocks perform about the same. (Note the “on average” in that statement; a single stock doesn’t have this behavior). So, to get an idea of how the entire market is doing, you only need to average a handful of stocks together to get the general trend. This is exactly what DJIA does, averaging (very crudely) the top 30 companies on the market.

The reason it’s a bad proxy is that the stock market doesn’t necessarily represent the economic health of the nation though. DJIA just hit an all-time high, and yet I don’t think anyone would say that we’re seeing an economic boom right now. Again, there is likely a long tail when considering the population as a whole. The stock market is probably a great indicator of how the upper class is doing (and actually, is probably a very direct measure of this). But, this is a measure of how wealth is doing, which is not evening distributed. So, the increasing divide in wealth would explain why the stock market can sore and the majority of us don’t feel it. I digress; it’s another case of a bad indicator.

I’ve given two examples of bad indicators (and I could keep going). In both examples, a contributing problem is a long tail, which essentially concentrates what’s being measured with a minority of stakeholders. This isn’t a bad thing in itself. After all, our intension is to recycle more of our waste at work, so the minority of “worst-offenders” should get more attention. However, if the point of the indicator is to change behavior across a large population then the contribution of each individual needs to be considered.

To measure and change behavior, a much better recycling indicator would measure each person’s contribution. Maybe something like this:

This is not a perfect measure. But, it’s one that measures individual contribution much better, and gives each of us a very clear direction on how we can change the indicator.

So, making indicators can get incredibly involved. The impact you want to have with them can be subtle, perverse, powerful, or ignored. The focus on what that impact should be is very important when making these. Sometimes the impact of making a change is the most important piece, sometimes awareness and behavior is more important. I hope I’ve shown you that there’s much more to this that just coming up with a fancy formula.