This is when KPIs fail

The effect of unexpected events can be devastating to the usability of KPIs. We've talked about it before, but now we'll zoom in a little bit more. In the end KPIs are used to make sure you take the right action at the right time (that is, hopefully before disaster strikes). In theory the threshold is chosen wisely and the indicator shows you what way it is going, so you can take appropriate action when needed. 

"In theory", because in practice many things can happen. Let's say there is a very special turkey living in US that read "How to measure anything", a bestseller on KPIs by Douglas Hubbard. The turkey defines a KPI that measures his general well being from day to day. This is what  his dashboard would look like.

In his book "The Black Swan" Nassim Taleb uses this example in order to explain the effect of unexpected events. For those unfamiliar with the idea of Black Swans, here's a small list of elements that make an event a Black Swan (based on the criteria as stated by Taleb himself).

1. The event is a surprise (to the observer).
2. The event has a major effect.
3. After the first recorded instance of the event, it is rationalized by hindsight, as if it could have been expected; that is, the relevant data were available but unaccounted for in risk mitigation programs. The same is true for the personal perception by individuals.

Taleb used the Black Swans metaphor because for decades the existence of such a bird was presumed non-existing. The discovery of a black swan was probably a surprising event. By now we know plenty of unexpected, surprising and impact full examples (not only in the Financial sector).

The thing with these Black Swans is that they can really mess up your business strategy (which progress was nicely being measured by KPIs). In Banking for example a regulator can unexceptionally ask you to comply to new regulation after they themselves reacted to an unexpected financial Black Swan. Or your customer satisfaction KPI drops because of a negative story on you company exploded on Twitter. Black Swans most often have a negative effect on your results (destroying the predictive power of your KPIs).

Taleb mentions several reasons why we tend to miss these kind of events.

1. We are terrible in predicting the future

2. If you think the likelihood of something happening is very low; it probably isn't

3. Don't ask the expert, he or she doesn't know either

4. The world is complex, don't think it isn't because you have a predictive model

5. Your intuition is very bad in statistics; don't let your strategy depend on it

6. Just believing a fact true doesn't make it so (see also previous blog on confirmation bias)

How to cope with Black Swans? One really effective way is doing a so-called Pre-Mortem. In this exercise you ask several people to imagine themselves in the future (e.g one year from now). Now draft the future situation where everything went terribly wrong. Whatever you tried to accomplish did not happen. Even worse, your work is perceived by the whole company as a complete disaster. The atmosphere among all people involved is terrible. Everybody is blaming each other and nobody is talking to each other (out of disappointment or anger). You even consider quitting your job because you can't cope with the shame every time you meet senior management.

Now aks everybody in the room to write down what went wrong. No pausing, just put whatever comes to mind on paper. Think of the most unexpected events that made it an disaster (remember the turkey!). Try to think beyond the "normal" things (e.g. death, viruses  financial crisis, people getting fired, fights, etc). 

Collect all output and write them down on a large poster. Share the poster with everybody in the organization  Make sure that from now on you start looking for signs that show some event on the poster is happening.

Next time: what can we learn from the Millennium Goals?

7 Cognitive Biases that influence the usage of KPIs

Lists apparently do well in blog titles. A total of 865 times my blogs were read (not counting my own clicks). The last blog called The five most overrated KPIs broke a record and was viewed 60 times (I say viewed because of course I can't tell whether they were actually read). I don't count the number of views as a KPI, but just to be sure I again present a list today

Remember one of my first blogs "Why we (really) use KPIs"? I talked about the workings of our brains and the two systems that make it operate. There was the "automatic" pilot governing our behavior most of the time (making the brain the efficient and effective organ it is). But when we have a more difficult taks to fulfill the non-automatic system jumps in (using it is tiresome though).

The problem is that we tend to think that we make our decisions deliberately and after solid reasoning. Unfortunately the brain system that we use most takes shortcuts, is lazy, loves stereotypes, and likes to go on as soon as possible. It is the price we pay for the enormous task we set the brain to do. 

Researchers already know for years that we make mistakes all the time without even noticing. Cognitive biases are the tendencies of our brain to make all sorts of mistakes. Mostly we are not aware of these biases. And nobody is immune to them. We cannot turn of our automatic pilot and therefor we cannot be completely bias free.

Here is a list of 10 cognitive biases that might diminish the effectiveness of KPIs*:

1. Cognitive Dissonance
We tend to ignore, ridicule or downplay information that conflicts with our beliefs or convictions. The stronger the belief, the stronger the effect. It takes a lot of effort to objectively look at conflicting information. This can have an effect on the usage of KPIs on several levels. Especially when "red flags" are ignored because we think it will turn green again soon.

2. Risk Aversion
In general our decisions tend to be risk averse, especially if we have to make them in a split second. This effect is studied thoroughly and is found in many social situations. With regards to KPIs this could result in choosing thresholds that are too low (playing safe) resulting in many false positives. But Risk Aversion can also lead to thresholds that are set too high avoiding the risk of getting the status red (and the risk of tough discussions with your management)

3. Confidence Bias
Our fast and automatic brain system is not prone on doubt. It hates doubt and will construct a story that makes what it sees true and coherent. So even when a KPI is indicating to an obvious wrong number, our over-enthousiastic brain will at first try to make it true. Only with effort we are able to see the wrongness for what it really is.

4. Causation Bias
Our hasty brain sees patterns all around us (even when there are no there). One them is the causation pattern. When we some events that correlate, we tend to apply some causal thinking. In the past this has led to many wrong assumptions and mistakes. When creating KPIs this can lead (among others) to selecting useless indicators, as it is wrongly assumed that they measure the underlying causal mechanism for performance.

5. Availability bias
Make a list of three situations where you showed assertiveness. Next, evaluate how assertive you are. Of course you are biased answering the second question. The three situations might come easy and therefor might give you the impression that you are assertive indeed (I'm not saying you're not). But the easier you can come up with a long list of something, the more it will affect your judgements later on. For KPIs this could mean that you will choose those indicators that easily come to mind and think that they must be good because of the fact that they came to mind that easy.

6. Anchoring
How many calories are there in MacDonald's Big Mac Burger (7.6 oz)**? Just take a guess and formulate your answer before you read on.


Is your answer around 850 or maybe 60? Then you where the victim of Anchoring. I deliberately added the numbers 856 and 60 in the intro of my blog. These numbers tend to stick for a while and they influence decisions later on. 

7. The illusion of understanding

Our brain has to deal with tons of information every minute (even when we are asleep). In order not to get completely insane our brain starts from the default position that the world outside in general makes sense and information we receive is coherent an unambiguous.  Because we think we know the past, we assume that we know the future. And with a great deal of "I knew it all along" attitude we arrogantly think we understand it all. We are in general ignorant of our own ignorance. KPIs in general are based on our past experience and by applying them think that we can control or predict future events.

Next time we'll zoom in on this last illusion by talking about the effect of unexpected events.


* The list and some of the text is retrieved from the book Thinking Fast and Slow by Daniel Kahneman
**The number of calories in a Big Mac is about 550 (calorieking.com). 

The five most overrated KPIs

There are KPIs for

Accounting, Sustainability, Corporate Services, Finance, Governance, Compliance, Risk, Human Resources, Information Technology, Knowledge & Innovation, Management, Marketing & Communications, eCommerce, Project Management, Portfolio Management, Commerce, Production Management, Quality Management, Sales & Customer Service, Supply Chain, Procurement, Distribution, SHOP BY INDUSTRY, Agriculture, Arts & Culture, Construction & Capital Works, Education & Training, Financial Institutions, Government, Local Government,  Healthcare, Hospitality & Tourism, Infrastructure Operations, Manufacturing, Media, Non-profit / Non-governmental, Postal & Courier Services, Professional Services, Publishing, Real Estate / Property, Resources, Retail, Sport Management, Sports, Telecommunications / Call Center, Transportation, and Utilities*.

Of course this is a non-limitative list. There are many different types of KPIs but fortunately many KPI experts already made some choices for you as to which ones are the BEST. Just Google KPI and you'll find some gurus telling you the TOP 5 KPIs everybody should use. That triggered me to list the TOP 5 most overrated KPIs. Here they are.

5. School grades

Schoolchildren are constantly assessed throughout the year by their teachers, and report cards are issued to parents at varying intervals. Generally the scores for individual assignments and tests are recorded for each student in a grade book, along with the maximum number of points for each assignment. In the US most often these scores are translated to a letter grade. In other countries (in Europe) the 1 to 10 scale is used.

At the end of the year most often an average is calculated to give an indication on the average performance of the kid. It is all too easy to assume that aggregate or average marks give a reliable assessment of overall performance or that the process is as objective as counting. Fortunately many teachers will tell parents this. They know that it is only an indication or a "photo" and is not telling anything on future performance. It is however difficult for parents to not see these grades and think that their kids are either "doomed" or "future professors". Especially in high-school much depends on these grades (status within the group, development of future plans, possibilities for universities). The system is hard on children that bloom on a late age.

4. Net Promoter Score

Would you recommend our company to a friend or colleague? That is the question many companies will ask their customers on a regular basis. Why? Because the answer is apparently telling you all about your customers feelings towards your company or products. A Net Promoter Score is generated based on this question, ranging from 1 to 10. The resulting score is supposed to indicate whether there is a huge risk of losing customers or whether there are loyal. The NPS has become one of the most important drivers in the area of customer intimacy strategy.

You might remember the blog on the APGAR score that suggested to make your KPI as stupid and simple as possible. The NPS is indeed simple and easy to understand. However the performance it is trying to measure is by far too complex to capture via this simple score. Especially when it is used for complex strategic choices which on their turn might effect future results. Furthermore the score is most often bases on what a sample of customers is saying. I won't go into detail but many issues arise when sampling your customer base.

In a White Paper called “The “Net-Net” on the Net Promoter Score” the authors surmise that “the NPS approach is incomplete at best, and potentially misleading at worse. It is unwise to rely solely on one survey item (likelihood to recommend) to establish customer loyalty strategies. While [the creators of NPS] provide sound advice on some aspects of customer loyalty measurement and management, he seriously overstates the case for relying on that “one number” to grow a business.”

3. Key Risk Indicators

When people are asked to give three examples of the most disruptive innovation of the last decades they come up with Computer, Internet, or the Mobile phone. All these innovations had a huge impact, but were all unpredicted, unplanned and their impact was underestimated at the time. Same goes for most manifestations of risks. When we try to predict risks we use risk models to predict likelihood of occurrence and the impact the particular risk will have if it actually manifests itself. Unfortunately all actual impactful events of the past decades were most often not predicted and if someone was lucky enough to have mentioned them, the impact was underestimated at the time. When we were in the midst of them the impact was not recognized by experts. Consider for example the latest project you were involved in (could be any type of project). Of course things went wrong, they always do. Would you have been able to predict them upfront? In other words: the gross of (impactful) risks come from outside the predictive models.

2. Employee Satisfaction

People can be satisfied with their jobs for several reasons. Asking for these reasons is a valid and useful thing to do. However using these results to measure performance is risky, especially when questionnaires are used. Even if the survey anonymous, employees might not wish to reveal the information or they might think that they will not benefit from responding (thinking perhaps even to be penalised by giving their real opinion). Even if employees fill in the survey honestly, you are still measuring individual opinions and not really their behaviour. 

It is already difficult enough to objectively understand and know your own intentions and motivations, let alone answering questions about them from someone who is paying your salary. Furthermore an anonymous survey is likely to reveal warts and all.  Management should be prepared for discovering that the top down view can differ from the bottom up view.

1. Stock price

The overall idea is that the stock price of a certain company is telling you something about how that company is doing. This is because it is assumed that investors digest all possible information about a company and this will be reflected in the price. This is what is known as the Efficient Market Hypothesis (EMH). The EMH assumes that all investors perceive all available information in precisely the same manner. However this is of course not the case. Furthermore it is impossible to say what information is already incorporated into the price. Not all information is available and the most impactful events are difficult to predict and therefor a surprise for everybody (see also the Key Risk Indicator paragraph).  

Investopedia.com summarized it as follow “Companies live and die by their stock price, yet for the most part they don't actively participate in trading their shares within the market. If performance of its stock is ignored, the life of the company and its management may be threatened with adverse consequences, such as the unhappiness of individual investors and future difficulties in raising capital”.

*list is extracted from kpiinstitute.org

NEXT TIME: Cognitive Biases and their impact on KPIs

Creating a KPI: What can possibly go wrong?

On July 8, 2009, Christopher Westley blogged a paper titled The Financial Crisis and the Systemic Failure of the Economics Profession published in Critical Review, by Colander, Goldberg, Haas, Juselius, Kirman, Lux, and Sloth with the following abstract:

Economists not only failed to anticipate the financial crisis; they may have contributed to it–with risk and derivatives models that, through spurious precision and untested theoretical assumptions, encouraged policy makers and market participants to see more stability and risk sharing than was actually present. Moreover, once the crisis occurred, it was met with incomprehension by most economists because of models that, on the one hand, downplay the possibility that economic actors may exhibit highly interactive behavior; and, on the other, assume that any homogeneity will involve economic actors sharing the economist’s own putatively correct model of the economy, so that error can stem only from an exogenous shock. The financial crisis presents both an ethical and an intellectual challenge to economics, and an opportunity to reform its study by grounding it more solidly in reality. (Source: Ludwig von Mises Institute)

In other words you might conclude in the run-up to the recent Financial Crisis, all financial and risk KPIs failed grotesquely. As a response you can say that it is easy to judge with hindsight. But are we sure that we today are not making the exact same mistakes and falling in the exact same pitfalls?

In the past 6 blogs I addressed the different steps needed to create KPIs. Reading back those blogs you can see that creating good KPIs is not a given. You only need to glitch one or two times and your KPI will be useless (resulting in an illusion rather then a steering tool).

And remember that these are the things that can go wrong when building them. We're not even using them yet. Here is a short summary of the possible pitfalls we encountered so far.

Step 1 Determine the goal you want to achieve

• Goals are too narrow or too vague
• Too many goals are defined
• Long term goals are ignored
• Short term goals are ignored
• Goals on changing behavior are very tricky

Step 2 Choose the KEY performance that influences your succes

• Too many performance indicators, but no KEY performance indicators.
• The indicators chosen are not the ones measuring the factors influencing performance
• KEY indicators are chosen just because everybody does so.

Step 3 Develop the indicator that measures the performance

• Chosen model is too complex
• Chosen model is too simple
• Data is not available for chosen method
• Chosen method does not reflect reality

Step 4 Choose the threshold that tells you how you are doing

• Thresholds chosen do not reflect the goals set
• Thresholds are fixed
• No thresholds are set upfront
• No tolerance level is considered
• Thresholds are copied

Step 5 Implement the KPI

• Complexity of changing behavior is underestimated
• Frequency is to high/low
• Number of KPIs on the dashboard is too high/low
• Balance between frequency rate and number of KPIs is not set right
• Owner, distributor and user are not in line with each other
• The outcome is not made actionable
• Look and feel of the dashboard does not fit the audience
• Wrong tools are chosen
• Complex KPIs are cropped into oversimplified "traffic lights"

Next time: The Top 5 most overrated KPIs

Step 5: Deploying the KPI (part II)

Here is a funny exercise. Type the words "KPI Dashboard" in Google Pictures and look at the first 20 results. I bet there are at least 15 dashboards shown like the one shown here (from ontimec.com). This is the way many consultant firms wants us to think of KPIs. Tidy and comprehensive dashboards, complete with meters, stopping lights, pictures and what have you. Most of them are “real time” and promise to drive your business to the sky and above.

I always wondered how many companies actually use these kind of fancy dashboards. In my whole career I didn’t see anyone using them. But that, of course, doesn’t say they aren’t. Last time I spoke of the impact on behaviour when deploying KPIs. The “look and feel” of a KPI Dashboard is another aspect to consider when distributing your KPIs. The graphical interface should be designed with the audience in mind. As said before one should keep it Stupid and Simple. One picture says more than 1000 words (or complex formulas). But cramping complex KPIs in a fancy stopping light isn’t going to work either. Keep in mind where the KPIs will be used and who is looking at the dashboard. A daily call is something different then the Board of Directors meeting. The fancier you make your dashboard, the more it will distract from the message you want to tell. The more detailed information you give, the more people will loose themselves into those details (or drop out). But there is more to be considered than the impact and the look and feel. Here are some more elements.

Frequency rate

Some suppliers of KPI Dashboards promote their Real Time Functionality. I assume not because it is particularly useful, but because it sounds nice in sales-pitches. I don’t question whether the supplier can actually deliver this functionality, but most often the data needed for such a dashboard is not available in real time. And increasing the frequency by which your KPI Dashboard is presented comes with a price. There is a converse relation between the frequency and the number of KPIs you can present.  If done right the number of KPI’s on your dashboard should decrease as soon as you increase its frequency. 

Consider a call center where you might want to show some KPIs on a big screen (e.g. the number of customers waiting and the time they are waiting). Of course these KPIs should be presented in real time. But what happens when you start increasing the number of indicators on the screen? Agents probably start being distracted and focused on the screen in stead of the call they are having (whether it is wise to put KPIs in a call center in the first place is another discussion we will have in another blog). 

So finding the right balance between Frequency and Number of KPIs is key. One can imagine that an insurance company that has a 50 page thick KPI document that is discussed each month by Senior Management isn’t deploying their KPIS very effectively. By the time the document is created, agreed upon, distributed and ready for discussion, it is time to start with the next months report. One thing is most important here. Don’t just copy the frequency rate just because it was always done so. Or because some department within the organisation requires it to be so (“our report is sent to Senior Management each quarter so could you please aggregate your daily KPI into a quarterly dashboard?”)

Creation and Distribution

Every KPI should have a (documented) Initially Intended Purpose (IIP) set by the KPI Owner (hopefully you a have one). Of course this Owner can “outsource” the creation and distribution of the KPI to someone else. In that case a mutual agreement should exist between these two parties. If not changes are that the KPI will at some time change and deviate from the IIP. Often the KPI Owner and the KPI User are one an the same person (or department). But there might be other Users too (Senior Management, Compliance, Risk, Finance, Audit, etc). Again these “second hand” Users should take notice of the IIP, otherwise KPIs will be used out of context, leading to wrong conclusions and actions. Especially when Owner, Provider, and User are all different people or departments.

Actionability

A KPI is more than just an indicator. As said in the previous blog a KPI should at least influence future behaviour if appropriate. As a result it is important that before implementing the KPI one should think in what way actions can be extracted from the KPIs and how the follow up is organised. Is there a Issue Management process in place, by which actions can be defined, allocated, solved and monitored? Are roles and responsibilities documented? Do the people involved aware of what is expected from them?

Let’s say that you have implemented a KPI that measures the effect of a marketing campaign of some sort (e.g. number of customers per week that used the promo-code online).  What corrective actions should be planned upfront in case the indicator shows “underperformance”.  Is a new mailing ready for distribution? And do we already know who to mail and how many? Will the business case still be valid and who will make this decision? Again, all these questions should be addressed and planned upfront.

Tools

The tool most used to create KPIs is probably Excel, later to be copied into nice PowerPoint slides with colorful graphs and matrixes. But using MS Office as your deployment toolkit is time-consuming and maintenance is difficult (we all know how we miss our Key Excel Guru as soon as he is on unexpected sick leave).

There are many suppliers of Dashboards in the market that developed “of the shelf” applications sometimes even providing you with a set of KPIs “ready to use”. These applications are often developed specifically for certain industries or topics (Risk, Data Quality, Finance, Customer Satisfaction etc). Downside of these tools is that the work best (or only) if you use the KPIs provided by the supplier. And most often you have to provide a very specific set of data in order to implement them properly. But even if you would be able to do so, the question is whether these “of the shelf” KPIs are best measuring your specific goals. Remember all the previous blogs where we described the four steps. All things considered you might say that each KPI is in the end very customized for a specific goal. It would be impossible for a pre-set KPI to meet all the requirements, regardless what the supplier claims.

Next time I will summarize the past 6 blogs (the five steps). And after that I will address the five most overrated KPIs.