This is when KPIs fail

The effect of unexpected events can be devastating to the usability of KPIs. We've talked about it before, but now we'll zoom in a little bit more. In the end KPIs are used to make sure you take the right action at the right time (that is, hopefully before disaster strikes). In theory the threshold is chosen wisely and the indicator shows you what way it is going, so you can take appropriate action when needed. 

"In theory", because in practice many things can happen. Let's say there is a very special turkey living in US that read "How to measure anything", a bestseller on KPIs by Douglas Hubbard. The turkey defines a KPI that measures his general well being from day to day. This is what  his dashboard would look like.

In his book "The Black Swan" Nassim Taleb uses this example in order to explain the effect of unexpected events. For those unfamiliar with the idea of Black Swans, here's a small list of elements that make an event a Black Swan (based on the criteria as stated by Taleb himself).

1. The event is a surprise (to the observer).
2. The event has a major effect.
3. After the first recorded instance of the event, it is rationalized by hindsight, as if it could have been expected; that is, the relevant data were available but unaccounted for in risk mitigation programs. The same is true for the personal perception by individuals.

Taleb used the Black Swans metaphor because for decades the existence of such a bird was presumed non-existing. The discovery of a black swan was probably a surprising event. By now we know plenty of unexpected, surprising and impact full examples (not only in the Financial sector).

The thing with these Black Swans is that they can really mess up your business strategy (which progress was nicely being measured by KPIs). In Banking for example a regulator can unexceptionally ask you to comply to new regulation after they themselves reacted to an unexpected financial Black Swan. Or your customer satisfaction KPI drops because of a negative story on you company exploded on Twitter. Black Swans most often have a negative effect on your results (destroying the predictive power of your KPIs).

Taleb mentions several reasons why we tend to miss these kind of events.

1. We are terrible in predicting the future

2. If you think the likelihood of something happening is very low; it probably isn't

3. Don't ask the expert, he or she doesn't know either

4. The world is complex, don't think it isn't because you have a predictive model

5. Your intuition is very bad in statistics; don't let your strategy depend on it

6. Just believing a fact true doesn't make it so (see also previous blog on confirmation bias)

How to cope with Black Swans? One really effective way is doing a so-called Pre-Mortem. In this exercise you ask several people to imagine themselves in the future (e.g one year from now). Now draft the future situation where everything went terribly wrong. Whatever you tried to accomplish did not happen. Even worse, your work is perceived by the whole company as a complete disaster. The atmosphere among all people involved is terrible. Everybody is blaming each other and nobody is talking to each other (out of disappointment or anger). You even consider quitting your job because you can't cope with the shame every time you meet senior management.

Now aks everybody in the room to write down what went wrong. No pausing, just put whatever comes to mind on paper. Think of the most unexpected events that made it an disaster (remember the turkey!). Try to think beyond the "normal" things (e.g. death, viruses  financial crisis, people getting fired, fights, etc). 

Collect all output and write them down on a large poster. Share the poster with everybody in the organization  Make sure that from now on you start looking for signs that show some event on the poster is happening.

Next time: what can we learn from the Millennium Goals?

7 Cognitive Biases that influence the usage of KPIs

Lists apparently do well in blog titles. A total of 865 times my blogs were read (not counting my own clicks). The last blog called The five most overrated KPIs broke a record and was viewed 60 times (I say viewed because of course I can't tell whether they were actually read). I don't count the number of views as a KPI, but just to be sure I again present a list today

Remember one of my first blogs "Why we (really) use KPIs"? I talked about the workings of our brains and the two systems that make it operate. There was the "automatic" pilot governing our behavior most of the time (making the brain the efficient and effective organ it is). But when we have a more difficult taks to fulfill the non-automatic system jumps in (using it is tiresome though).

The problem is that we tend to think that we make our decisions deliberately and after solid reasoning. Unfortunately the brain system that we use most takes shortcuts, is lazy, loves stereotypes, and likes to go on as soon as possible. It is the price we pay for the enormous task we set the brain to do. 

Researchers already know for years that we make mistakes all the time without even noticing. Cognitive biases are the tendencies of our brain to make all sorts of mistakes. Mostly we are not aware of these biases. And nobody is immune to them. We cannot turn of our automatic pilot and therefor we cannot be completely bias free.

Here is a list of 10 cognitive biases that might diminish the effectiveness of KPIs*:

1. Cognitive Dissonance
We tend to ignore, ridicule or downplay information that conflicts with our beliefs or convictions. The stronger the belief, the stronger the effect. It takes a lot of effort to objectively look at conflicting information. This can have an effect on the usage of KPIs on several levels. Especially when "red flags" are ignored because we think it will turn green again soon.

2. Risk Aversion
In general our decisions tend to be risk averse, especially if we have to make them in a split second. This effect is studied thoroughly and is found in many social situations. With regards to KPIs this could result in choosing thresholds that are too low (playing safe) resulting in many false positives. But Risk Aversion can also lead to thresholds that are set too high avoiding the risk of getting the status red (and the risk of tough discussions with your management)

3. Confidence Bias
Our fast and automatic brain system is not prone on doubt. It hates doubt and will construct a story that makes what it sees true and coherent. So even when a KPI is indicating to an obvious wrong number, our over-enthousiastic brain will at first try to make it true. Only with effort we are able to see the wrongness for what it really is.

4. Causation Bias
Our hasty brain sees patterns all around us (even when there are no there). One them is the causation pattern. When we some events that correlate, we tend to apply some causal thinking. In the past this has led to many wrong assumptions and mistakes. When creating KPIs this can lead (among others) to selecting useless indicators, as it is wrongly assumed that they measure the underlying causal mechanism for performance.

5. Availability bias
Make a list of three situations where you showed assertiveness. Next, evaluate how assertive you are. Of course you are biased answering the second question. The three situations might come easy and therefor might give you the impression that you are assertive indeed (I'm not saying you're not). But the easier you can come up with a long list of something, the more it will affect your judgements later on. For KPIs this could mean that you will choose those indicators that easily come to mind and think that they must be good because of the fact that they came to mind that easy.

6. Anchoring
How many calories are there in MacDonald's Big Mac Burger (7.6 oz)**? Just take a guess and formulate your answer before you read on.


Is your answer around 850 or maybe 60? Then you where the victim of Anchoring. I deliberately added the numbers 856 and 60 in the intro of my blog. These numbers tend to stick for a while and they influence decisions later on. 

7. The illusion of understanding

Our brain has to deal with tons of information every minute (even when we are asleep). In order not to get completely insane our brain starts from the default position that the world outside in general makes sense and information we receive is coherent an unambiguous.  Because we think we know the past, we assume that we know the future. And with a great deal of "I knew it all along" attitude we arrogantly think we understand it all. We are in general ignorant of our own ignorance. KPIs in general are based on our past experience and by applying them think that we can control or predict future events.

Next time we'll zoom in on this last illusion by talking about the effect of unexpected events.


* The list and some of the text is retrieved from the book Thinking Fast and Slow by Daniel Kahneman
**The number of calories in a Big Mac is about 550 (calorieking.com). 

The five most overrated KPIs

There are KPIs for

Accounting, Sustainability, Corporate Services, Finance, Governance, Compliance, Risk, Human Resources, Information Technology, Knowledge & Innovation, Management, Marketing & Communications, eCommerce, Project Management, Portfolio Management, Commerce, Production Management, Quality Management, Sales & Customer Service, Supply Chain, Procurement, Distribution, SHOP BY INDUSTRY, Agriculture, Arts & Culture, Construction & Capital Works, Education & Training, Financial Institutions, Government, Local Government,  Healthcare, Hospitality & Tourism, Infrastructure Operations, Manufacturing, Media, Non-profit / Non-governmental, Postal & Courier Services, Professional Services, Publishing, Real Estate / Property, Resources, Retail, Sport Management, Sports, Telecommunications / Call Center, Transportation, and Utilities*.

Of course this is a non-limitative list. There are many different types of KPIs but fortunately many KPI experts already made some choices for you as to which ones are the BEST. Just Google KPI and you'll find some gurus telling you the TOP 5 KPIs everybody should use. That triggered me to list the TOP 5 most overrated KPIs. Here they are.

5. School grades

Schoolchildren are constantly assessed throughout the year by their teachers, and report cards are issued to parents at varying intervals. Generally the scores for individual assignments and tests are recorded for each student in a grade book, along with the maximum number of points for each assignment. In the US most often these scores are translated to a letter grade. In other countries (in Europe) the 1 to 10 scale is used.

At the end of the year most often an average is calculated to give an indication on the average performance of the kid. It is all too easy to assume that aggregate or average marks give a reliable assessment of overall performance or that the process is as objective as counting. Fortunately many teachers will tell parents this. They know that it is only an indication or a "photo" and is not telling anything on future performance. It is however difficult for parents to not see these grades and think that their kids are either "doomed" or "future professors". Especially in high-school much depends on these grades (status within the group, development of future plans, possibilities for universities). The system is hard on children that bloom on a late age.

4. Net Promoter Score

Would you recommend our company to a friend or colleague? That is the question many companies will ask their customers on a regular basis. Why? Because the answer is apparently telling you all about your customers feelings towards your company or products. A Net Promoter Score is generated based on this question, ranging from 1 to 10. The resulting score is supposed to indicate whether there is a huge risk of losing customers or whether there are loyal. The NPS has become one of the most important drivers in the area of customer intimacy strategy.

You might remember the blog on the APGAR score that suggested to make your KPI as stupid and simple as possible. The NPS is indeed simple and easy to understand. However the performance it is trying to measure is by far too complex to capture via this simple score. Especially when it is used for complex strategic choices which on their turn might effect future results. Furthermore the score is most often bases on what a sample of customers is saying. I won't go into detail but many issues arise when sampling your customer base.

In a White Paper called “The “Net-Net” on the Net Promoter Score” the authors surmise that “the NPS approach is incomplete at best, and potentially misleading at worse. It is unwise to rely solely on one survey item (likelihood to recommend) to establish customer loyalty strategies. While [the creators of NPS] provide sound advice on some aspects of customer loyalty measurement and management, he seriously overstates the case for relying on that “one number” to grow a business.”

3. Key Risk Indicators

When people are asked to give three examples of the most disruptive innovation of the last decades they come up with Computer, Internet, or the Mobile phone. All these innovations had a huge impact, but were all unpredicted, unplanned and their impact was underestimated at the time. Same goes for most manifestations of risks. When we try to predict risks we use risk models to predict likelihood of occurrence and the impact the particular risk will have if it actually manifests itself. Unfortunately all actual impactful events of the past decades were most often not predicted and if someone was lucky enough to have mentioned them, the impact was underestimated at the time. When we were in the midst of them the impact was not recognized by experts. Consider for example the latest project you were involved in (could be any type of project). Of course things went wrong, they always do. Would you have been able to predict them upfront? In other words: the gross of (impactful) risks come from outside the predictive models.

2. Employee Satisfaction

People can be satisfied with their jobs for several reasons. Asking for these reasons is a valid and useful thing to do. However using these results to measure performance is risky, especially when questionnaires are used. Even if the survey anonymous, employees might not wish to reveal the information or they might think that they will not benefit from responding (thinking perhaps even to be penalised by giving their real opinion). Even if employees fill in the survey honestly, you are still measuring individual opinions and not really their behaviour. 

It is already difficult enough to objectively understand and know your own intentions and motivations, let alone answering questions about them from someone who is paying your salary. Furthermore an anonymous survey is likely to reveal warts and all.  Management should be prepared for discovering that the top down view can differ from the bottom up view.

1. Stock price

The overall idea is that the stock price of a certain company is telling you something about how that company is doing. This is because it is assumed that investors digest all possible information about a company and this will be reflected in the price. This is what is known as the Efficient Market Hypothesis (EMH). The EMH assumes that all investors perceive all available information in precisely the same manner. However this is of course not the case. Furthermore it is impossible to say what information is already incorporated into the price. Not all information is available and the most impactful events are difficult to predict and therefor a surprise for everybody (see also the Key Risk Indicator paragraph).  

Investopedia.com summarized it as follow “Companies live and die by their stock price, yet for the most part they don't actively participate in trading their shares within the market. If performance of its stock is ignored, the life of the company and its management may be threatened with adverse consequences, such as the unhappiness of individual investors and future difficulties in raising capital”.

*list is extracted from kpiinstitute.org

NEXT TIME: Cognitive Biases and their impact on KPIs