Creating a KPI: What can possibly go wrong?

On July 8, 2009, Christopher Westley blogged a paper titled The Financial Crisis and the Systemic Failure of the Economics Profession published in Critical Review, by Colander, Goldberg, Haas, Juselius, Kirman, Lux, and Sloth with the following abstract:

Economists not only failed to anticipate the financial crisis; they may have contributed to it–with risk and derivatives models that, through spurious precision and untested theoretical assumptions, encouraged policy makers and market participants to see more stability and risk sharing than was actually present. Moreover, once the crisis occurred, it was met with incomprehension by most economists because of models that, on the one hand, downplay the possibility that economic actors may exhibit highly interactive behavior; and, on the other, assume that any homogeneity will involve economic actors sharing the economist’s own putatively correct model of the economy, so that error can stem only from an exogenous shock. The financial crisis presents both an ethical and an intellectual challenge to economics, and an opportunity to reform its study by grounding it more solidly in reality. (Source: Ludwig von Mises Institute)

In other words you might conclude in the run-up to the recent Financial Crisis, all financial and risk KPIs failed grotesquely. As a response you can say that it is easy to judge with hindsight. But are we sure that we today are not making the exact same mistakes and falling in the exact same pitfalls?

In the past 6 blogs I addressed the different steps needed to create KPIs. Reading back those blogs you can see that creating good KPIs is not a given. You only need to glitch one or two times and your KPI will be useless (resulting in an illusion rather then a steering tool).

And remember that these are the things that can go wrong when building them. We're not even using them yet. Here is a short summary of the possible pitfalls we encountered so far.

Step 1 Determine the goal you want to achieve

• Goals are too narrow or too vague
• Too many goals are defined
• Long term goals are ignored
• Short term goals are ignored
• Goals on changing behavior are very tricky

Step 2 Choose the KEY performance that influences your succes

• Too many performance indicators, but no KEY performance indicators.
• The indicators chosen are not the ones measuring the factors influencing performance
• KEY indicators are chosen just because everybody does so.

Step 3 Develop the indicator that measures the performance

• Chosen model is too complex
• Chosen model is too simple
• Data is not available for chosen method
• Chosen method does not reflect reality

Step 4 Choose the threshold that tells you how you are doing

• Thresholds chosen do not reflect the goals set
• Thresholds are fixed
• No thresholds are set upfront
• No tolerance level is considered
• Thresholds are copied

Step 5 Implement the KPI

• Complexity of changing behavior is underestimated
• Frequency is to high/low
• Number of KPIs on the dashboard is too high/low
• Balance between frequency rate and number of KPIs is not set right
• Owner, distributor and user are not in line with each other
• The outcome is not made actionable
• Look and feel of the dashboard does not fit the audience
• Wrong tools are chosen
• Complex KPIs are cropped into oversimplified "traffic lights"

Next time: The Top 5 most overrated KPIs