Step 5: Deploying the KPI (part I)

1. Lose Weight
2. Getting Organized
3. Spend Less, Save More
4. Enjoy Life to the Fullest
5. Staying Fit and Healthy
6. Learn Something Exciting
7. Quit Smoking
8. Help Others in Their Dreams
9. Fall in Love
10. Spend More Time with Family

Recognize any of these items? They are the top ten new year resolutions made by people in the US. There are other lists available, but most of them have a huge overlap. Less than half of the people who say they make resolutions maintain them through the first 6 months (46%). (University of Scranton. Journal of Clinical Psychology)

Changing behaviour is though and often thought too lightly. The underlying purpose of a KPI should be to change behaviour. That is to say to root out actions that antagonize performance and boost those actions that increase it. To my (humble ;-) opinion the deployment step is the most underestimated step in creating KPIs. Nobody will argue that this step is redundant, but the execution is most not done accordingly to its importance. I will explain here why I think this is the case. Here are the most common mistakes that are being made when implementing KPIs in the organization

Just activating a KPI might give you the impression that behaviour will change accordingly all by itself. But setting an indicator to see whether performance changed over time will not change the behaviour itself needed to accomplish the increase in performance. It like with the resolutions. They also are  meant to motivate people to change their behaviour.  Timothy Pychyl, a professor of psychology at Carleton University in Canada, argues however that people aren't ready to change their habits, particularly bad habits, and that accounts for the high failure rate. Another reason, says Dr. Avya Sharma of the Canadian Obesity Network, is that people set unrealistic goals and expectations in their resolutions.

These effects have been studied to great extend and we can learn from these studies in relation to best implement KPIs (so they actually do what they should do). These are eight common tips that are given to make resolutions work better.

1. Focus on the deployment of one (or just a few) KPIs at a time
2. Set realistic, specific expectations of the speed by which the performance should increase;
3. Don't wait till things almost go wrong to make resolutions. Make it a yearlong process, every day;
4. Take small steps. Many people quit because the goal is too big requiring too big a step all at once;
5. Have someone close to the department (but not part of) that you have to report progress;
6. Celebrate your success between milestones. Don't wait the goal to be finally completed
7. Focus your thinking on new ways of doing things (out of the box). You have to create new pathways to change habits (what got you here, won’t get you there;
8. Focus on the present. What's the one thing you can do today, right now, towards your goal?

This is part one of the blog on Deployment. Next time I will discuss some other things to consider when deploying a KPI (like target audience, responsibilities, format, frequency, etc).

Step 4: Raising the bar and avoiding pitfalls

Every large company has a department that deals with internal and external fraud cases. Depending on the business, the type of customers and the size of the company there can be many or just a few cases to investigate each month. Companies gather all sorts of data relating fraud. Number of (reported, investigated and solved) cases, the money (potentially) lost and recovered, number of customers involved (both as fraudsters and as victim). From a management perspective each and one of these cases is special, but for an investigator a case can be one of many and even "routine". The fact that people can have different perspectives on the impact of a single case is important when building KPIs.

Let's look at an example. You work at a Fraud Department at a large insurance company. Your job is to report the number of fraud cases per month to senior management and indicate whether the increase/decrease should result in "immediate action" (red), "monitor closely" (amber), or "business as usual" green. At some point in time you report a small increase for three months in a row. There is no certain reason why it increases and the cases are no different in modus operandi as in the past. Because of this you cannot say whether this is a trend or coincidence. What RAG-status will you choose? From the investigation departments view it is "business as usual". But from a senior management perspective this is the least likely status they expect when fraud cases are increasing. If you report green (as you probably should), you are own senior management a good explanation. From experience I know that it is very hard to explain however that even something like fraud can be "business as usual". If you report amber (playing safe), you will only delay the discussion to next month (what will you do if the number increases again by a few cases?). And as soon as you report red (as management might expect), you know for certain that the next question will be: what actions will you take? Problem is that you will not be able to indicate any mitigation actions, as you don't have a proper cause identified for the increase. If you report amber (safest choice), you will only pros pone the discussion to next month (what will you do if the number increases again by a few cases?).

KPIs are meant to keep you awake and alert. They should make sure that you take appropriate actions whenever they indicate that performance is declining. This leads to a very crucial question you have to answer: when is the appropriate time? You don't want to be in panic-mode every day, but on the other hand you don't want to miss important signs either. If you set the bar too high, you will be dulled a sleep. If it too low, you will never reach a "business as usual" status. The Red-Amber-Green coloring is the most used way of indicating the status of an indicator. In later blogs we will discuss what I would call "Green Field Management" where all (good and bad) measures are taken to stay at a green status (including altering the threshold or ignoring/including certain outliers). But for now we will focus on the pitfalls in choosing the thresholds in the first place.

Pitfall 1: Predicting is hard, especially the future
It is good to realise that setting thresholds above or below which certain behaviour is expected is like predicting the future. When you start off with your fresh KPI and start collecting data points, you don't know where the statistics will lead over time. You might have an idea where it ideally should lead, but it is not to say that it will. That makes setting thresholds especially difficult.

Pitfall 2: No thresholds are set upfront Because of pitfall 1 this is happens more often than not. Because it is very hard to determine the RAG-thresholds upfront, it is done as soon as new data comes along (like in the fraud example). This might lead to numerous problems like opportunistic and ad-hoc decisions. On the other hand don’t be too rigid. You should set you thresholds upfront, but make sure they are not carved in stone. If you change them however, this should be well done after close consideration of all data, the goals involved and well documented.

Pitfall 3: What goes up, must go down
Every series of increases is inevitably followed (at some point in time) by a decrease. You have to consider this upfront and what “fluctuation” you tolerate before setting a threshold.

Pitfall 4: Regression to the mean
This phenomenon results directly from pitfall 3. In statistics, regression toward (or to) the mean is the phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement—and, paradoxically, if it is extreme on its second measurement, it will tend to have been closer to the average on its first. To avoid making incorrect inferences, regression toward the mean must be considered when interpreting data (Wikipedia). In the case of a KPI it is therefore important to know what the baseline (or average) is. And whether movement towards this baseline is “good” or “bad”.

Pitfall 5: (again) what got you here, won’t get you there
I could also have called this pitfall, the Copy-Paste pitfall. Over and over again companies copy-paste old thresholds into new ones, reasoning that we have been using them for years. Or KPI’s (including their thresholds) are copied, because everybody in the industry is using them.

Bernard Marr, a lead expert on KPIs, said it like this: “A lot of companies fall into the trap of thinking they can just use their existing metrics and “retro fit” their objectives around them. This is dangerous for many reasons and will often leave you without real insight into the things that matter the most. Success involves effort and you should be willing to spend time thinking carefully about what information you need, where to find it, how to gather it and why it will be of benefit”.

Next time the final step: Implementing the KPI

Step 3: Keeping them Stupid and Simple

It goes without saying that (to-be) parents want to give birth to a healthy baby. From a KPI perspective that is a clear and unambiguous goal. And of course there is no real discussion on what the KEY performance indicator would be; Health. But how do you measure the health of a newly born baby? What indicates the well being of a little human that is breathing for the first time? For decades it was gut-feeling of the person delivering the baby. This led babies to suffer from brain injuries because important signs were not seen or ignored.

This was the case until 1953. In that year it was Virginia Apgar who published her proposal for a new method of evaluation of the newborn infant. Having considered several objective indicators pertaining to the condition of the infant at birth she selected five. These indicators were heart rate, respiratory effort, reflex irritability, muscle tone and color. Sixty seconds after the complete birth of the baby a rating of zero, one or two was given to each sign, depending on whether it was absent or present. The purpose of the Apgar test was (and still is) to determine quickly whether a newborn needs immediate medical care; it was not designed to make long-term predictions on a child's healthVirginia Apgar. A twelve-institution study involving 17,221 babies, established that the Apgar Score, especially the five-minute score, can predict neonatal survival and neurological development. Some ten years after the initial publication, a backronym for APGAR was coined: Appearance (skin color), Pulse (heart rate), Grimace (reflex irritability), Activity (muscle tone), and Respiration.

The APGAR score is still being used all over the world. As a rough-and-ready, simple, but broadly accurate measure, it has saved countless lives since its introduction.  What can we learn about the story of Virginia Apgar (apart from the fact that innovative thinking combined with science can lead to great things)?

Keep it simple
Those who have experienced the birth of a child up close, know that it is a hectic situation. A complex and complicated KPI won't work. The APGAR score was created with the end-user in mind. Thousands of medical personal should be able to understand and apply the methodology in a few seconds. Each of the five indicators are scored with zero, one or two. Simply adding the individual scores gives you the APGAR score. Even a distressed farther-to-be can understand that. Simple tests like this are effective because they tend to be accurate enough, and crucially are simple enough that busy people actually use them. Simple formulas are as reliable as complex formulas or even expert judgement in many cases.

You might think that a simple 0,1 or 2 is an oversimplification that can't possibly be used as an indicator. However research shows that statistically sophisticated or complex methods do not necessarily provide more accurate forecasts than simpler ones (Makridakis and Hibon, 1999). The problem is that we focus on the rare occasions when these indicators work and almost never on their far more numerous failures (Taleb, 2010). Once you have defined a clear goal and selected the KEY performance indicators, it is not necessary to develop complex and complicated models that can be used as indicators (In a later blog we will discuss the issue of complex predictive models in more detail). Simple and easy-to-understand indicators will also help when implementing the KPI. Keeping it Stutip and Simple (KISS) helps people to relate to the indicator and apply the outcome directly in their actions.

Integer or decimal

Probably not deliberately, but Virginia Apgar chose a decimal range of values for scoring health. A decimal range is a set of values with a fixed maximum and minimum value (in this case 0 and 10). You might recognize another example of a decimal range from questionnaires, were often an answer range is given between 1 and 5.

Integer ranges on the other hand (theoretically) have no minimum or maximum. Temperature measured in Celsius or Fahrenheit, Speed measured in miles per hour, or Age measured in years, are all examples of integer value ranges. If you would extract two integer values, the result is again an integer value and can be compared to other values in the range. For example, the difference between a 9-year-old and a 7-year-old results in exactly the same difference comparing someone who is 40 with another who is 42.

This however does not work for a decimal range. Let's say that you measure individual performance in a range between 1 and 5 (poor, sufficient, good, very good, excellent). The difference between 1 and 2 (poor and sufficient) is not quite the same as the difference between good and excellent. This is a much more subjective difference. Comparing integer with decimal, one could say that the first "contains" much more information than the latter. This is because information can be extracted from each value point (and even from between two value points). One could even calculate the average without "losing" information.This makes integer ranges more detailed and "rich"*.

That is not to say that a decimal range cannot be used as measurement range. As long as you understand its limitations. The information that can be retrieved from such a range is limited, but on the other hand it is easier to understand and interpret. An integer range on the other hand provides more detailed information, but its interpretation is more abstract because they tend to be more difficult to relate to the "real"  world. Which could be problematic as we have seen that KPIs best be stupid and simple.

Next time we'll discuss step 4: choosing the threshold.

*theoretically you could calculate an average for a decimal range, but this would be less representing a "real" value (e.g. what would an average of 2.7 say in the individual performance example above?)