Damn lies and statistics (part I)

The Be Excellent blog recently posted an article on what gets watched gets improved, quoting Bob Parsons (CEO of GoDaddy.com) to the effect that

anything of significance that is measured and watched, improves

This is a variation of the mantra in business circles that “what gets measured, gets managed.”

There is, of course, a healthy amount of wisdom in this mantra, helping account for its popularity. However, the mantra is also rather concerning.

The first concern is its apparent implication that what doesn’t get measured with hard and fast numerical metrics doesn’t get managed.

Obviously it’s good to measure things, when you can. But some things that are measured don’t translate into effective management – for example, profitability is almost always measured, but companies aren’t almost always profitable. Sometimes they are measuring how rapidly they are losing money, and simply knowing that does not necessarily lead to clear and appropriate actions to take to address that situation.

What you need to ‘measure’ is what makes a causal difference to your business, for example who your customers are and what they want from you and why they get it from you. You need to know what the causal factors are around your diminishing profitability, and then measure the drivers that relate to that. You need to measure the right sort of things. And you need to know ‘what you’re going to do’ with that information – how you will translate measurement into action.

But sometimes these important things you need to assess are qualitative. For example, a customer survey can be highly quantitative and tell you what % of customers care about what predefined issues. But such a quantitative survey might not surface new and important issues and concerns, simply because the predefined questions don’t ask about it. A qualitative survey or focus group with open questions might yield far better insight into customer concerns.

Even more difficult to manage with clear and valid metrics are intangibles such as the organisational culture or level of creativity. Sure, you can conduct psychometric testing, perform climate and culture surveys, etc – but these tend to be superficial, date easily, and not give you the depth of insight that a qood intuition and qualitative assessment from a seasoned manager could yield.

What is important is not measurement, it’s assessment – specifically assessment of the right sort of ‘things’ to enable appropriate decision making and course correction.

It might be a highly qualitative ‘finger on the pulse’ of what’s happening in the market, or it might be detailed numerical metrics. It might be highly intuitive, leveraging management’s years and decades of experience, or it might be highly methodical and formalised quantitative analysis. Both are important, and both have their place.

The word “measurement” is too narrow. Perhaps it should be ‘what is monitored, gets managed’. And monitoring could include both quantitative and qualitative activity.

Sorry, comments are closed for this post.

Damn lies and statistics (part I)

The Be Excellent blog recently posted an article on what gets watched gets improved, quoting Bob Parsons (CEO of GoDaddy.com) to the effect that

anything of significance that is measured and watched, improves

This is a variation of the mantra in business circles that “what gets measured, gets managed.”

There is, of course, a healthy amount of wisdom in this mantra, helping account for its popularity. However, the mantra is also rather concerning.

The first concern is its apparent implication that what doesn’t get measured with hard and fast numerical metrics doesn’t get managed.

Obviously it’s good to measure things, when you can. But some things that are measured don’t translate into effective management – for example, profitability is almost always measured, but companies aren’t almost always profitable. Sometimes they are measuring how rapidly they are losing money, and simply knowing that does not necessarily lead to clear and appropriate actions to take to address that situation.

What you need to ‘measure’ is what makes a causal difference to your business, for example who your customers are and what they want from you and why they get it from you. You need to know what the causal factors are around your diminishing profitability, and then measure the drivers that relate to that. You need to measure the right sort of things. And you need to know ‘what you’re going to do’ with that information – how you will translate measurement into action.

But sometimes these important things you need to assess are qualitative. For example, a customer survey can be highly quantitative and tell you what % of customers care about what predefined issues. But such a quantitative survey might not surface new and important issues and concerns, simply because the predefined questions don’t ask about it. A qualitative survey or focus group with open questions might yield far better insight into customer concerns.

Even more difficult to manage with clear and valid metrics are intangibles such as the organisational culture or level of creativity. Sure, you can conduct psychometric testing, perform climate and culture surveys, etc – but these tend to be superficial, date easily, and not give you the depth of insight that a qood intuition and qualitative assessment from a seasoned manager could yield.

What is important is not measurement, it’s assessment – specifically assessment of the right sort of ‘things’ to enable appropriate decision making and course correction.

It might be a highly qualitative ‘finger on the pulse’ of what’s happening in the market, or it might be detailed numerical metrics. It might be highly intuitive, leveraging management’s years and decades of experience, or it might be highly methodical and formalised quantitative analysis. Both are important, and both have their place.

The word “measurement” is too narrow. Perhaps it should be ‘what is monitored, gets managed’. And monitoring could include both quantitative and qualitative activity.

Sorry, comments are closed for this post.