This is when KPIs fail

The effect of unexpected events can be devastating to the usability of KPIs. We've talked about it before, but now we'll zoom in a little bit more. In the end KPIs are used to make sure you take the right action at the right time (that is, hopefully before disaster strikes). In theory the threshold is chosen wisely and the indicator shows you what way it is going, so you can take appropriate action when needed. 

"In theory", because in practice many things can happen. Let's say there is a very special turkey living in US that read "How to measure anything", a bestseller on KPIs by Douglas Hubbard. The turkey defines a KPI that measures his general well being from day to day. This is what  his dashboard would look like.

In his book "The Black Swan" Nassim Taleb uses this example in order to explain the effect of unexpected events. For those unfamiliar with the idea of Black Swans, here's a small list of elements that make an event a Black Swan (based on the criteria as stated by Taleb himself).

1. The event is a surprise (to the observer).
2. The event has a major effect.
3. After the first recorded instance of the event, it is rationalized by hindsight, as if it could have been expected; that is, the relevant data were available but unaccounted for in risk mitigation programs. The same is true for the personal perception by individuals.

Taleb used the Black Swans metaphor because for decades the existence of such a bird was presumed non-existing. The discovery of a black swan was probably a surprising event. By now we know plenty of unexpected, surprising and impact full examples (not only in the Financial sector).

The thing with these Black Swans is that they can really mess up your business strategy (which progress was nicely being measured by KPIs). In Banking for example a regulator can unexceptionally ask you to comply to new regulation after they themselves reacted to an unexpected financial Black Swan. Or your customer satisfaction KPI drops because of a negative story on you company exploded on Twitter. Black Swans most often have a negative effect on your results (destroying the predictive power of your KPIs).

Taleb mentions several reasons why we tend to miss these kind of events.

1. We are terrible in predicting the future

2. If you think the likelihood of something happening is very low; it probably isn't

3. Don't ask the expert, he or she doesn't know either

4. The world is complex, don't think it isn't because you have a predictive model

5. Your intuition is very bad in statistics; don't let your strategy depend on it

6. Just believing a fact true doesn't make it so (see also previous blog on confirmation bias)

How to cope with Black Swans? One really effective way is doing a so-called Pre-Mortem. In this exercise you ask several people to imagine themselves in the future (e.g one year from now). Now draft the future situation where everything went terribly wrong. Whatever you tried to accomplish did not happen. Even worse, your work is perceived by the whole company as a complete disaster. The atmosphere among all people involved is terrible. Everybody is blaming each other and nobody is talking to each other (out of disappointment or anger). You even consider quitting your job because you can't cope with the shame every time you meet senior management.

Now aks everybody in the room to write down what went wrong. No pausing, just put whatever comes to mind on paper. Think of the most unexpected events that made it an disaster (remember the turkey!). Try to think beyond the "normal" things (e.g. death, viruses  financial crisis, people getting fired, fights, etc). 

Collect all output and write them down on a large poster. Share the poster with everybody in the organization  Make sure that from now on you start looking for signs that show some event on the poster is happening.

Next time: what can we learn from the Millennium Goals?