Creating a KPI: What can possibly go wrong?

On July 8, 2009, Christopher Westley blogged a paper titled The Financial Crisis and the Systemic Failure of the Economics Profession published in Critical Review, by Colander, Goldberg, Haas, Juselius, Kirman, Lux, and Sloth with the following abstract:

Economists not only failed to anticipate the financial crisis; they may have contributed to it–with risk and derivatives models that, through spurious precision and untested theoretical assumptions, encouraged policy makers and market participants to see more stability and risk sharing than was actually present. Moreover, once the crisis occurred, it was met with incomprehension by most economists because of models that, on the one hand, downplay the possibility that economic actors may exhibit highly interactive behavior; and, on the other, assume that any homogeneity will involve economic actors sharing the economist’s own putatively correct model of the economy, so that error can stem only from an exogenous shock. The financial crisis presents both an ethical and an intellectual challenge to economics, and an opportunity to reform its study by grounding it more solidly in reality. (Source: Ludwig von Mises Institute)

In other words you might conclude in the run-up to the recent Financial Crisis, all financial and risk KPIs failed grotesquely. As a response you can say that it is easy to judge with hindsight. But are we sure that we today are not making the exact same mistakes and falling in the exact same pitfalls?

In the past 6 blogs I addressed the different steps needed to create KPIs. Reading back those blogs you can see that creating good KPIs is not a given. You only need to glitch one or two times and your KPI will be useless (resulting in an illusion rather then a steering tool).

And remember that these are the things that can go wrong when building them. We're not even using them yet. Here is a short summary of the possible pitfalls we encountered so far.

Step 1 Determine the goal you want to achieve

• Goals are too narrow or too vague
• Too many goals are defined
• Long term goals are ignored
• Short term goals are ignored
• Goals on changing behavior are very tricky

Step 2 Choose the KEY performance that influences your succes

• Too many performance indicators, but no KEY performance indicators.
• The indicators chosen are not the ones measuring the factors influencing performance
• KEY indicators are chosen just because everybody does so.

Step 3 Develop the indicator that measures the performance

• Chosen model is too complex
• Chosen model is too simple
• Data is not available for chosen method
• Chosen method does not reflect reality

Step 4 Choose the threshold that tells you how you are doing

• Thresholds chosen do not reflect the goals set
• Thresholds are fixed
• No thresholds are set upfront
• No tolerance level is considered
• Thresholds are copied

Step 5 Implement the KPI

• Complexity of changing behavior is underestimated
• Frequency is to high/low
• Number of KPIs on the dashboard is too high/low
• Balance between frequency rate and number of KPIs is not set right
• Owner, distributor and user are not in line with each other
• The outcome is not made actionable
• Look and feel of the dashboard does not fit the audience
• Wrong tools are chosen
• Complex KPIs are cropped into oversimplified "traffic lights"

Next time: The Top 5 most overrated KPIs

Step 5: Deploying the KPI (part II)

Here is a funny exercise. Type the words "KPI Dashboard" in Google Pictures and look at the first 20 results. I bet there are at least 15 dashboards shown like the one shown here (from ontimec.com). This is the way many consultant firms wants us to think of KPIs. Tidy and comprehensive dashboards, complete with meters, stopping lights, pictures and what have you. Most of them are “real time” and promise to drive your business to the sky and above.

I always wondered how many companies actually use these kind of fancy dashboards. In my whole career I didn’t see anyone using them. But that, of course, doesn’t say they aren’t. Last time I spoke of the impact on behaviour when deploying KPIs. The “look and feel” of a KPI Dashboard is another aspect to consider when distributing your KPIs. The graphical interface should be designed with the audience in mind. As said before one should keep it Stupid and Simple. One picture says more than 1000 words (or complex formulas). But cramping complex KPIs in a fancy stopping light isn’t going to work either. Keep in mind where the KPIs will be used and who is looking at the dashboard. A daily call is something different then the Board of Directors meeting. The fancier you make your dashboard, the more it will distract from the message you want to tell. The more detailed information you give, the more people will loose themselves into those details (or drop out). But there is more to be considered than the impact and the look and feel. Here are some more elements.

Frequency rate

Some suppliers of KPI Dashboards promote their Real Time Functionality. I assume not because it is particularly useful, but because it sounds nice in sales-pitches. I don’t question whether the supplier can actually deliver this functionality, but most often the data needed for such a dashboard is not available in real time. And increasing the frequency by which your KPI Dashboard is presented comes with a price. There is a converse relation between the frequency and the number of KPIs you can present.  If done right the number of KPI’s on your dashboard should decrease as soon as you increase its frequency. 

Consider a call center where you might want to show some KPIs on a big screen (e.g. the number of customers waiting and the time they are waiting). Of course these KPIs should be presented in real time. But what happens when you start increasing the number of indicators on the screen? Agents probably start being distracted and focused on the screen in stead of the call they are having (whether it is wise to put KPIs in a call center in the first place is another discussion we will have in another blog). 

So finding the right balance between Frequency and Number of KPIs is key. One can imagine that an insurance company that has a 50 page thick KPI document that is discussed each month by Senior Management isn’t deploying their KPIS very effectively. By the time the document is created, agreed upon, distributed and ready for discussion, it is time to start with the next months report. One thing is most important here. Don’t just copy the frequency rate just because it was always done so. Or because some department within the organisation requires it to be so (“our report is sent to Senior Management each quarter so could you please aggregate your daily KPI into a quarterly dashboard?”)

Creation and Distribution

Every KPI should have a (documented) Initially Intended Purpose (IIP) set by the KPI Owner (hopefully you a have one). Of course this Owner can “outsource” the creation and distribution of the KPI to someone else. In that case a mutual agreement should exist between these two parties. If not changes are that the KPI will at some time change and deviate from the IIP. Often the KPI Owner and the KPI User are one an the same person (or department). But there might be other Users too (Senior Management, Compliance, Risk, Finance, Audit, etc). Again these “second hand” Users should take notice of the IIP, otherwise KPIs will be used out of context, leading to wrong conclusions and actions. Especially when Owner, Provider, and User are all different people or departments.

Actionability

A KPI is more than just an indicator. As said in the previous blog a KPI should at least influence future behaviour if appropriate. As a result it is important that before implementing the KPI one should think in what way actions can be extracted from the KPIs and how the follow up is organised. Is there a Issue Management process in place, by which actions can be defined, allocated, solved and monitored? Are roles and responsibilities documented? Do the people involved aware of what is expected from them?

Let’s say that you have implemented a KPI that measures the effect of a marketing campaign of some sort (e.g. number of customers per week that used the promo-code online).  What corrective actions should be planned upfront in case the indicator shows “underperformance”.  Is a new mailing ready for distribution? And do we already know who to mail and how many? Will the business case still be valid and who will make this decision? Again, all these questions should be addressed and planned upfront.

Tools

The tool most used to create KPIs is probably Excel, later to be copied into nice PowerPoint slides with colorful graphs and matrixes. But using MS Office as your deployment toolkit is time-consuming and maintenance is difficult (we all know how we miss our Key Excel Guru as soon as he is on unexpected sick leave).

There are many suppliers of Dashboards in the market that developed “of the shelf” applications sometimes even providing you with a set of KPIs “ready to use”. These applications are often developed specifically for certain industries or topics (Risk, Data Quality, Finance, Customer Satisfaction etc). Downside of these tools is that the work best (or only) if you use the KPIs provided by the supplier. And most often you have to provide a very specific set of data in order to implement them properly. But even if you would be able to do so, the question is whether these “of the shelf” KPIs are best measuring your specific goals. Remember all the previous blogs where we described the four steps. All things considered you might say that each KPI is in the end very customized for a specific goal. It would be impossible for a pre-set KPI to meet all the requirements, regardless what the supplier claims.

Next time I will summarize the past 6 blogs (the five steps). And after that I will address the five most overrated KPIs.