The five most overrated KPIs

There are KPIs for

Accounting, Sustainability, Corporate Services, Finance, Governance, Compliance, Risk, Human Resources, Information Technology, Knowledge & Innovation, Management, Marketing & Communications, eCommerce, Project Management, Portfolio Management, Commerce, Production Management, Quality Management, Sales & Customer Service, Supply Chain, Procurement, Distribution, SHOP BY INDUSTRY, Agriculture, Arts & Culture, Construction & Capital Works, Education & Training, Financial Institutions, Government, Local Government,  Healthcare, Hospitality & Tourism, Infrastructure Operations, Manufacturing, Media, Non-profit / Non-governmental, Postal & Courier Services, Professional Services, Publishing, Real Estate / Property, Resources, Retail, Sport Management, Sports, Telecommunications / Call Center, Transportation, and Utilities*.

Of course this is a non-limitative list. There are many different types of KPIs but fortunately many KPI experts already made some choices for you as to which ones are the BEST. Just Google KPI and you'll find some gurus telling you the TOP 5 KPIs everybody should use. That triggered me to list the TOP 5 most overrated KPIs. Here they are.

5. School grades

Schoolchildren are constantly assessed throughout the year by their teachers, and report cards are issued to parents at varying intervals. Generally the scores for individual assignments and tests are recorded for each student in a grade book, along with the maximum number of points for each assignment. In the US most often these scores are translated to a letter grade. In other countries (in Europe) the 1 to 10 scale is used.

At the end of the year most often an average is calculated to give an indication on the average performance of the kid. It is all too easy to assume that aggregate or average marks give a reliable assessment of overall performance or that the process is as objective as counting. Fortunately many teachers will tell parents this. They know that it is only an indication or a "photo" and is not telling anything on future performance. It is however difficult for parents to not see these grades and think that their kids are either "doomed" or "future professors". Especially in high-school much depends on these grades (status within the group, development of future plans, possibilities for universities). The system is hard on children that bloom on a late age.

4. Net Promoter Score

Would you recommend our company to a friend or colleague? That is the question many companies will ask their customers on a regular basis. Why? Because the answer is apparently telling you all about your customers feelings towards your company or products. A Net Promoter Score is generated based on this question, ranging from 1 to 10. The resulting score is supposed to indicate whether there is a huge risk of losing customers or whether there are loyal. The NPS has become one of the most important drivers in the area of customer intimacy strategy.

You might remember the blog on the APGAR score that suggested to make your KPI as stupid and simple as possible. The NPS is indeed simple and easy to understand. However the performance it is trying to measure is by far too complex to capture via this simple score. Especially when it is used for complex strategic choices which on their turn might effect future results. Furthermore the score is most often bases on what a sample of customers is saying. I won't go into detail but many issues arise when sampling your customer base.

In a White Paper called “The “Net-Net” on the Net Promoter Score” the authors surmise that “the NPS approach is incomplete at best, and potentially misleading at worse. It is unwise to rely solely on one survey item (likelihood to recommend) to establish customer loyalty strategies. While [the creators of NPS] provide sound advice on some aspects of customer loyalty measurement and management, he seriously overstates the case for relying on that “one number” to grow a business.”

3. Key Risk Indicators

When people are asked to give three examples of the most disruptive innovation of the last decades they come up with Computer, Internet, or the Mobile phone. All these innovations had a huge impact, but were all unpredicted, unplanned and their impact was underestimated at the time. Same goes for most manifestations of risks. When we try to predict risks we use risk models to predict likelihood of occurrence and the impact the particular risk will have if it actually manifests itself. Unfortunately all actual impactful events of the past decades were most often not predicted and if someone was lucky enough to have mentioned them, the impact was underestimated at the time. When we were in the midst of them the impact was not recognized by experts. Consider for example the latest project you were involved in (could be any type of project). Of course things went wrong, they always do. Would you have been able to predict them upfront? In other words: the gross of (impactful) risks come from outside the predictive models.

2. Employee Satisfaction

People can be satisfied with their jobs for several reasons. Asking for these reasons is a valid and useful thing to do. However using these results to measure performance is risky, especially when questionnaires are used. Even if the survey anonymous, employees might not wish to reveal the information or they might think that they will not benefit from responding (thinking perhaps even to be penalised by giving their real opinion). Even if employees fill in the survey honestly, you are still measuring individual opinions and not really their behaviour. 

It is already difficult enough to objectively understand and know your own intentions and motivations, let alone answering questions about them from someone who is paying your salary. Furthermore an anonymous survey is likely to reveal warts and all.  Management should be prepared for discovering that the top down view can differ from the bottom up view.

1. Stock price

The overall idea is that the stock price of a certain company is telling you something about how that company is doing. This is because it is assumed that investors digest all possible information about a company and this will be reflected in the price. This is what is known as the Efficient Market Hypothesis (EMH). The EMH assumes that all investors perceive all available information in precisely the same manner. However this is of course not the case. Furthermore it is impossible to say what information is already incorporated into the price. Not all information is available and the most impactful events are difficult to predict and therefor a surprise for everybody (see also the Key Risk Indicator paragraph).  

Investopedia.com summarized it as follow “Companies live and die by their stock price, yet for the most part they don't actively participate in trading their shares within the market. If performance of its stock is ignored, the life of the company and its management may be threatened with adverse consequences, such as the unhappiness of individual investors and future difficulties in raising capital”.

*list is extracted from kpiinstitute.org

NEXT TIME: Cognitive Biases and their impact on KPIs

Creating a KPI: What can possibly go wrong?

On July 8, 2009, Christopher Westley blogged a paper titled The Financial Crisis and the Systemic Failure of the Economics Profession published in Critical Review, by Colander, Goldberg, Haas, Juselius, Kirman, Lux, and Sloth with the following abstract:

Economists not only failed to anticipate the financial crisis; they may have contributed to it–with risk and derivatives models that, through spurious precision and untested theoretical assumptions, encouraged policy makers and market participants to see more stability and risk sharing than was actually present. Moreover, once the crisis occurred, it was met with incomprehension by most economists because of models that, on the one hand, downplay the possibility that economic actors may exhibit highly interactive behavior; and, on the other, assume that any homogeneity will involve economic actors sharing the economist’s own putatively correct model of the economy, so that error can stem only from an exogenous shock. The financial crisis presents both an ethical and an intellectual challenge to economics, and an opportunity to reform its study by grounding it more solidly in reality. (Source: Ludwig von Mises Institute)

In other words you might conclude in the run-up to the recent Financial Crisis, all financial and risk KPIs failed grotesquely. As a response you can say that it is easy to judge with hindsight. But are we sure that we today are not making the exact same mistakes and falling in the exact same pitfalls?

In the past 6 blogs I addressed the different steps needed to create KPIs. Reading back those blogs you can see that creating good KPIs is not a given. You only need to glitch one or two times and your KPI will be useless (resulting in an illusion rather then a steering tool).

And remember that these are the things that can go wrong when building them. We're not even using them yet. Here is a short summary of the possible pitfalls we encountered so far.

Step 1 Determine the goal you want to achieve

• Goals are too narrow or too vague
• Too many goals are defined
• Long term goals are ignored
• Short term goals are ignored
• Goals on changing behavior are very tricky

Step 2 Choose the KEY performance that influences your succes

• Too many performance indicators, but no KEY performance indicators.
• The indicators chosen are not the ones measuring the factors influencing performance
• KEY indicators are chosen just because everybody does so.

Step 3 Develop the indicator that measures the performance

• Chosen model is too complex
• Chosen model is too simple
• Data is not available for chosen method
• Chosen method does not reflect reality

Step 4 Choose the threshold that tells you how you are doing

• Thresholds chosen do not reflect the goals set
• Thresholds are fixed
• No thresholds are set upfront
• No tolerance level is considered
• Thresholds are copied

Step 5 Implement the KPI

• Complexity of changing behavior is underestimated
• Frequency is to high/low
• Number of KPIs on the dashboard is too high/low
• Balance between frequency rate and number of KPIs is not set right
• Owner, distributor and user are not in line with each other
• The outcome is not made actionable
• Look and feel of the dashboard does not fit the audience
• Wrong tools are chosen
• Complex KPIs are cropped into oversimplified "traffic lights"

Next time: The Top 5 most overrated KPIs

Step 5: Deploying the KPI (part II)

Here is a funny exercise. Type the words "KPI Dashboard" in Google Pictures and look at the first 20 results. I bet there are at least 15 dashboards shown like the one shown here (from ontimec.com). This is the way many consultant firms wants us to think of KPIs. Tidy and comprehensive dashboards, complete with meters, stopping lights, pictures and what have you. Most of them are “real time” and promise to drive your business to the sky and above.

I always wondered how many companies actually use these kind of fancy dashboards. In my whole career I didn’t see anyone using them. But that, of course, doesn’t say they aren’t. Last time I spoke of the impact on behaviour when deploying KPIs. The “look and feel” of a KPI Dashboard is another aspect to consider when distributing your KPIs. The graphical interface should be designed with the audience in mind. As said before one should keep it Stupid and Simple. One picture says more than 1000 words (or complex formulas). But cramping complex KPIs in a fancy stopping light isn’t going to work either. Keep in mind where the KPIs will be used and who is looking at the dashboard. A daily call is something different then the Board of Directors meeting. The fancier you make your dashboard, the more it will distract from the message you want to tell. The more detailed information you give, the more people will loose themselves into those details (or drop out). But there is more to be considered than the impact and the look and feel. Here are some more elements.

Frequency rate

Some suppliers of KPI Dashboards promote their Real Time Functionality. I assume not because it is particularly useful, but because it sounds nice in sales-pitches. I don’t question whether the supplier can actually deliver this functionality, but most often the data needed for such a dashboard is not available in real time. And increasing the frequency by which your KPI Dashboard is presented comes with a price. There is a converse relation between the frequency and the number of KPIs you can present.  If done right the number of KPI’s on your dashboard should decrease as soon as you increase its frequency. 

Consider a call center where you might want to show some KPIs on a big screen (e.g. the number of customers waiting and the time they are waiting). Of course these KPIs should be presented in real time. But what happens when you start increasing the number of indicators on the screen? Agents probably start being distracted and focused on the screen in stead of the call they are having (whether it is wise to put KPIs in a call center in the first place is another discussion we will have in another blog). 

So finding the right balance between Frequency and Number of KPIs is key. One can imagine that an insurance company that has a 50 page thick KPI document that is discussed each month by Senior Management isn’t deploying their KPIS very effectively. By the time the document is created, agreed upon, distributed and ready for discussion, it is time to start with the next months report. One thing is most important here. Don’t just copy the frequency rate just because it was always done so. Or because some department within the organisation requires it to be so (“our report is sent to Senior Management each quarter so could you please aggregate your daily KPI into a quarterly dashboard?”)

Creation and Distribution

Every KPI should have a (documented) Initially Intended Purpose (IIP) set by the KPI Owner (hopefully you a have one). Of course this Owner can “outsource” the creation and distribution of the KPI to someone else. In that case a mutual agreement should exist between these two parties. If not changes are that the KPI will at some time change and deviate from the IIP. Often the KPI Owner and the KPI User are one an the same person (or department). But there might be other Users too (Senior Management, Compliance, Risk, Finance, Audit, etc). Again these “second hand” Users should take notice of the IIP, otherwise KPIs will be used out of context, leading to wrong conclusions and actions. Especially when Owner, Provider, and User are all different people or departments.

Actionability

A KPI is more than just an indicator. As said in the previous blog a KPI should at least influence future behaviour if appropriate. As a result it is important that before implementing the KPI one should think in what way actions can be extracted from the KPIs and how the follow up is organised. Is there a Issue Management process in place, by which actions can be defined, allocated, solved and monitored? Are roles and responsibilities documented? Do the people involved aware of what is expected from them?

Let’s say that you have implemented a KPI that measures the effect of a marketing campaign of some sort (e.g. number of customers per week that used the promo-code online).  What corrective actions should be planned upfront in case the indicator shows “underperformance”.  Is a new mailing ready for distribution? And do we already know who to mail and how many? Will the business case still be valid and who will make this decision? Again, all these questions should be addressed and planned upfront.

Tools

The tool most used to create KPIs is probably Excel, later to be copied into nice PowerPoint slides with colorful graphs and matrixes. But using MS Office as your deployment toolkit is time-consuming and maintenance is difficult (we all know how we miss our Key Excel Guru as soon as he is on unexpected sick leave).

There are many suppliers of Dashboards in the market that developed “of the shelf” applications sometimes even providing you with a set of KPIs “ready to use”. These applications are often developed specifically for certain industries or topics (Risk, Data Quality, Finance, Customer Satisfaction etc). Downside of these tools is that the work best (or only) if you use the KPIs provided by the supplier. And most often you have to provide a very specific set of data in order to implement them properly. But even if you would be able to do so, the question is whether these “of the shelf” KPIs are best measuring your specific goals. Remember all the previous blogs where we described the four steps. All things considered you might say that each KPI is in the end very customized for a specific goal. It would be impossible for a pre-set KPI to meet all the requirements, regardless what the supplier claims.

Next time I will summarize the past 6 blogs (the five steps). And after that I will address the five most overrated KPIs.

Step 5: Deploying the KPI (part I)

1. Lose Weight
2. Getting Organized
3. Spend Less, Save More
4. Enjoy Life to the Fullest
5. Staying Fit and Healthy
6. Learn Something Exciting
7. Quit Smoking
8. Help Others in Their Dreams
9. Fall in Love
10. Spend More Time with Family

Recognize any of these items? They are the top ten new year resolutions made by people in the US. There are other lists available, but most of them have a huge overlap. Less than half of the people who say they make resolutions maintain them through the first 6 months (46%). (University of Scranton. Journal of Clinical Psychology)

Changing behaviour is though and often thought too lightly. The underlying purpose of a KPI should be to change behaviour. That is to say to root out actions that antagonize performance and boost those actions that increase it. To my (humble ;-) opinion the deployment step is the most underestimated step in creating KPIs. Nobody will argue that this step is redundant, but the execution is most not done accordingly to its importance. I will explain here why I think this is the case. Here are the most common mistakes that are being made when implementing KPIs in the organization

Just activating a KPI might give you the impression that behaviour will change accordingly all by itself. But setting an indicator to see whether performance changed over time will not change the behaviour itself needed to accomplish the increase in performance. It like with the resolutions. They also are  meant to motivate people to change their behaviour.  Timothy Pychyl, a professor of psychology at Carleton University in Canada, argues however that people aren't ready to change their habits, particularly bad habits, and that accounts for the high failure rate. Another reason, says Dr. Avya Sharma of the Canadian Obesity Network, is that people set unrealistic goals and expectations in their resolutions.

These effects have been studied to great extend and we can learn from these studies in relation to best implement KPIs (so they actually do what they should do). These are eight common tips that are given to make resolutions work better.

1. Focus on the deployment of one (or just a few) KPIs at a time
2. Set realistic, specific expectations of the speed by which the performance should increase;
3. Don't wait till things almost go wrong to make resolutions. Make it a yearlong process, every day;
4. Take small steps. Many people quit because the goal is too big requiring too big a step all at once;
5. Have someone close to the department (but not part of) that you have to report progress;
6. Celebrate your success between milestones. Don't wait the goal to be finally completed
7. Focus your thinking on new ways of doing things (out of the box). You have to create new pathways to change habits (what got you here, won’t get you there;
8. Focus on the present. What's the one thing you can do today, right now, towards your goal?

This is part one of the blog on Deployment. Next time I will discuss some other things to consider when deploying a KPI (like target audience, responsibilities, format, frequency, etc).

Step 4: Raising the bar and avoiding pitfalls

Every large company has a department that deals with internal and external fraud cases. Depending on the business, the type of customers and the size of the company there can be many or just a few cases to investigate each month. Companies gather all sorts of data relating fraud. Number of (reported, investigated and solved) cases, the money (potentially) lost and recovered, number of customers involved (both as fraudsters and as victim). From a management perspective each and one of these cases is special, but for an investigator a case can be one of many and even "routine". The fact that people can have different perspectives on the impact of a single case is important when building KPIs.

Let's look at an example. You work at a Fraud Department at a large insurance company. Your job is to report the number of fraud cases per month to senior management and indicate whether the increase/decrease should result in "immediate action" (red), "monitor closely" (amber), or "business as usual" green. At some point in time you report a small increase for three months in a row. There is no certain reason why it increases and the cases are no different in modus operandi as in the past. Because of this you cannot say whether this is a trend or coincidence. What RAG-status will you choose? From the investigation departments view it is "business as usual". But from a senior management perspective this is the least likely status they expect when fraud cases are increasing. If you report green (as you probably should), you are own senior management a good explanation. From experience I know that it is very hard to explain however that even something like fraud can be "business as usual". If you report amber (playing safe), you will only delay the discussion to next month (what will you do if the number increases again by a few cases?). And as soon as you report red (as management might expect), you know for certain that the next question will be: what actions will you take? Problem is that you will not be able to indicate any mitigation actions, as you don't have a proper cause identified for the increase. If you report amber (safest choice), you will only pros pone the discussion to next month (what will you do if the number increases again by a few cases?).

KPIs are meant to keep you awake and alert. They should make sure that you take appropriate actions whenever they indicate that performance is declining. This leads to a very crucial question you have to answer: when is the appropriate time? You don't want to be in panic-mode every day, but on the other hand you don't want to miss important signs either. If you set the bar too high, you will be dulled a sleep. If it too low, you will never reach a "business as usual" status. The Red-Amber-Green coloring is the most used way of indicating the status of an indicator. In later blogs we will discuss what I would call "Green Field Management" where all (good and bad) measures are taken to stay at a green status (including altering the threshold or ignoring/including certain outliers). But for now we will focus on the pitfalls in choosing the thresholds in the first place.

Pitfall 1: Predicting is hard, especially the future
It is good to realise that setting thresholds above or below which certain behaviour is expected is like predicting the future. When you start off with your fresh KPI and start collecting data points, you don't know where the statistics will lead over time. You might have an idea where it ideally should lead, but it is not to say that it will. That makes setting thresholds especially difficult.

Pitfall 2: No thresholds are set upfront Because of pitfall 1 this is happens more often than not. Because it is very hard to determine the RAG-thresholds upfront, it is done as soon as new data comes along (like in the fraud example). This might lead to numerous problems like opportunistic and ad-hoc decisions. On the other hand don’t be too rigid. You should set you thresholds upfront, but make sure they are not carved in stone. If you change them however, this should be well done after close consideration of all data, the goals involved and well documented.

Pitfall 3: What goes up, must go down
Every series of increases is inevitably followed (at some point in time) by a decrease. You have to consider this upfront and what “fluctuation” you tolerate before setting a threshold.

Pitfall 4: Regression to the mean
This phenomenon results directly from pitfall 3. In statistics, regression toward (or to) the mean is the phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement—and, paradoxically, if it is extreme on its second measurement, it will tend to have been closer to the average on its first. To avoid making incorrect inferences, regression toward the mean must be considered when interpreting data (Wikipedia). In the case of a KPI it is therefore important to know what the baseline (or average) is. And whether movement towards this baseline is “good” or “bad”.

Pitfall 5: (again) what got you here, won’t get you there
I could also have called this pitfall, the Copy-Paste pitfall. Over and over again companies copy-paste old thresholds into new ones, reasoning that we have been using them for years. Or KPI’s (including their thresholds) are copied, because everybody in the industry is using them.

Bernard Marr, a lead expert on KPIs, said it like this: “A lot of companies fall into the trap of thinking they can just use their existing metrics and “retro fit” their objectives around them. This is dangerous for many reasons and will often leave you without real insight into the things that matter the most. Success involves effort and you should be willing to spend time thinking carefully about what information you need, where to find it, how to gather it and why it will be of benefit”.

Next time the final step: Implementing the KPI

Step 3: Keeping them Stupid and Simple

It goes without saying that (to-be) parents want to give birth to a healthy baby. From a KPI perspective that is a clear and unambiguous goal. And of course there is no real discussion on what the KEY performance indicator would be; Health. But how do you measure the health of a newly born baby? What indicates the well being of a little human that is breathing for the first time? For decades it was gut-feeling of the person delivering the baby. This led babies to suffer from brain injuries because important signs were not seen or ignored.

This was the case until 1953. In that year it was Virginia Apgar who published her proposal for a new method of evaluation of the newborn infant. Having considered several objective indicators pertaining to the condition of the infant at birth she selected five. These indicators were heart rate, respiratory effort, reflex irritability, muscle tone and color. Sixty seconds after the complete birth of the baby a rating of zero, one or two was given to each sign, depending on whether it was absent or present. The purpose of the Apgar test was (and still is) to determine quickly whether a newborn needs immediate medical care; it was not designed to make long-term predictions on a child's healthVirginia Apgar. A twelve-institution study involving 17,221 babies, established that the Apgar Score, especially the five-minute score, can predict neonatal survival and neurological development. Some ten years after the initial publication, a backronym for APGAR was coined: Appearance (skin color), Pulse (heart rate), Grimace (reflex irritability), Activity (muscle tone), and Respiration.

The APGAR score is still being used all over the world. As a rough-and-ready, simple, but broadly accurate measure, it has saved countless lives since its introduction.  What can we learn about the story of Virginia Apgar (apart from the fact that innovative thinking combined with science can lead to great things)?

Keep it simple
Those who have experienced the birth of a child up close, know that it is a hectic situation. A complex and complicated KPI won't work. The APGAR score was created with the end-user in mind. Thousands of medical personal should be able to understand and apply the methodology in a few seconds. Each of the five indicators are scored with zero, one or two. Simply adding the individual scores gives you the APGAR score. Even a distressed farther-to-be can understand that. Simple tests like this are effective because they tend to be accurate enough, and crucially are simple enough that busy people actually use them. Simple formulas are as reliable as complex formulas or even expert judgement in many cases.

You might think that a simple 0,1 or 2 is an oversimplification that can't possibly be used as an indicator. However research shows that statistically sophisticated or complex methods do not necessarily provide more accurate forecasts than simpler ones (Makridakis and Hibon, 1999). The problem is that we focus on the rare occasions when these indicators work and almost never on their far more numerous failures (Taleb, 2010). Once you have defined a clear goal and selected the KEY performance indicators, it is not necessary to develop complex and complicated models that can be used as indicators (In a later blog we will discuss the issue of complex predictive models in more detail). Simple and easy-to-understand indicators will also help when implementing the KPI. Keeping it Stutip and Simple (KISS) helps people to relate to the indicator and apply the outcome directly in their actions.

Integer or decimal

Probably not deliberately, but Virginia Apgar chose a decimal range of values for scoring health. A decimal range is a set of values with a fixed maximum and minimum value (in this case 0 and 10). You might recognize another example of a decimal range from questionnaires, were often an answer range is given between 1 and 5.

Integer ranges on the other hand (theoretically) have no minimum or maximum. Temperature measured in Celsius or Fahrenheit, Speed measured in miles per hour, or Age measured in years, are all examples of integer value ranges. If you would extract two integer values, the result is again an integer value and can be compared to other values in the range. For example, the difference between a 9-year-old and a 7-year-old results in exactly the same difference comparing someone who is 40 with another who is 42.

This however does not work for a decimal range. Let's say that you measure individual performance in a range between 1 and 5 (poor, sufficient, good, very good, excellent). The difference between 1 and 2 (poor and sufficient) is not quite the same as the difference between good and excellent. This is a much more subjective difference. Comparing integer with decimal, one could say that the first "contains" much more information than the latter. This is because information can be extracted from each value point (and even from between two value points). One could even calculate the average without "losing" information.This makes integer ranges more detailed and "rich"*.

That is not to say that a decimal range cannot be used as measurement range. As long as you understand its limitations. The information that can be retrieved from such a range is limited, but on the other hand it is easier to understand and interpret. An integer range on the other hand provides more detailed information, but its interpretation is more abstract because they tend to be more difficult to relate to the "real"  world. Which could be problematic as we have seen that KPIs best be stupid and simple.

Next time we'll discuss step 4: choosing the threshold.

*theoretically you could calculate an average for a decimal range, but this would be less representing a "real" value (e.g. what would an average of 2.7 say in the individual performance example above?)