This is why KPIs fail

A few blogs back I wrote about when KPIs fail (see here for the blog) and the blog How to lie with your KPI? was about deliberate manipulations of KPIs and their outcome. But there are more reasons why KPIs in the end fail.

Over and over again I emphasized the fact that KPIs are meant to set change in motion when necessary. If no change is initiated, the set goals will not be met and the performance was measured for nothing. Here is a list of possible things that prevent you to act upon the outcome of a KPI.

1. Bad data quality
KPIs run on data. Without good information the KPI cannot be created. Data drives your KPI. That's why correct data is of upmost importance. Unfortunately all sorts of things can go wrong with you data. Especially when data was originally not created for the purpose you are using it for in your KPI. Or because default values can be entered (e.g. '999999'). This can be especially tricky with Financial KPIs. But also with commercial KPIs this can be bothersome. Can you really ensure that all the data you use is correct? A small error in your underlying data can have huge impact. This topic asks for a separate blog.

2. People don't see the importance
Without people knowing and understanding the importance of the specific KPI it won't fly at all. Not only must people be aware of them, they also have to see the consequences of not meeting the thresholds set.

3. People don't understand
In one of the first blogs I discussed the complexity of KPIs (Keeping them Simple and Stupid). This wasn't for no reason. People will not easily admit that they didn't understand the complex and complicated KPI you showed them. They will say they did, but that's only because they don't want to look stupid. Inevitably this will lead them to ignore the KPI as much as possible (to prevent looking stupid again). Or their actions are less effective as they could have been, had they understood the KPI better.

4. People are not interested
As a result of item 2 and 3 or because of other factors, people just might drop out. Some people just hate being measured and KPIs is the manifestation of this. Others just don't desire a product like KPIs because they don't see what's in it for them. And there is always a group of people that lose interest as soon as numbers are involved.

5. People were not involved
I think this is an underlying factor for people to be skeptical about the usefulness of the KPI. It is the "Not-Invented-Here" principle. People do like to have an influence, especially when it concerns their future and how they will be assessed.

6. After creation, the process was stopped
Creation is just the first phase when using KPIs (see my first few blogs on the creation process). There are three more phases that are just as (or maybe even more) important. These are Communicate, Consult and Control (together with the Create this is what I would call the Four C model). Item 2 -5 of the list above are a direct result if you do not communicate. But also "Consultancy" and advice on how to implement and use KPIs is important. It is not enough to just tell people that your KPI exists and how important it is. And last but not least, you have to check whether people adhere to the agreed actions. Are due-dates met and was the work done sufficient and correct?

Why do (almost) all projects fail?

You might find the question in the title a little bit dishonest as it suggests that most projects fail. And in that sense you are right, it is a wrong a question. But it is also wrong for a less more obvious reason. The problem with the question is that it does not tell you what is meant by "fail". If I would ask you to define project failing I guess you would come up with something like "delivered above budget" or "not delivered on time". It is true that project performance most often is measured via these two basic KPIs (see for example my blog on IT projects within Government).

But let's not ignore the fact that many projects do indeed fail to deliver on budget and on time. When was the last time you were involved in a project that was either on time or on budget (let alone both)? So even when it is the essence of the Project Managers job to keep their projects within the GREEN, it almost never happens. In my opinion this is because we are measuring the wrong things. My (maybe bold) statement is that these two KPIs are useless to measure project performance. Of course it does say something about the progress of the project but not about the actual performance. Focussing on just these two KPIs is like the mouse that is staring in the two headlights of a car. It blinds you from the real "danger". So what are these "real" dangers that we should focus on when executing projects?

For starters, it is safe to assume that your budget estimate was wrong in the first place. We are masters in "short-term-predictions". That's what our brains are doing all day long. However when it comes to long term predictions we are just terrible. In general we are biased towards optimism (optimism bias), we ignore obvious warning signals (confirmation bias), take previous events out of context (context effect) and tend to remember things more positively (Egocentric bias).

Secondly, the actual risks that materialize during your project (endangering your delivery time) are not the ones that you summed beforehand. The ones that you can think of beforehand, are the ones that were probably copied from the previous project starting documents and are most often already taken into account when defining the time window of the project. The real issue was already addressed in the blog "When KPIs fail". It's the problem of Black Swans, who are always unexpected but impactful.

So using time and budget as your KPIs is a recipy for failure. Unfortunately this is not without far-reaching consequences. Especially when the GREEN status of these KPIs

become the goal. Requirements are de-scoped or time goes before quality.

But apart from these hidden sides regarding budget and time there are more dangers that lurk in the dark (if not measured properly). When focussing on budget and time we tend to forget that the real "performance" of the project is measured by the quality of the thing it is implementing. How often is a Business Case drafted in the beginning of the project and never checked during the project? Or when it is checked it is altered to fit the new timelines and budget. Even the Business Case itself is most often as "light as a feather", presenting three "scenarios" to choose from: Doing Nothing, Doing Everything, Doing the Halfway Solution.  Furthermore most Business Cases do not take into account things like it-debt, increased complexity, maintenance costs, imbedding in Business As Usual, governance aspects, etc.

Projects implement change and change has an effect on people. All sorts of behavioural effects can take place (both inside and outside the project) that have an impact on the project results. Don't underestimate the behavioural effect. Coping with change is one of the hardest things to do for everybody. People can (directly or indirectly) sabotage the projects. Early adapters might lose interest (and the project loses a sponsor). Quality might go down when people feel the pressure to deliver. People within or outside the project do not believe in the change (even when you have a communication professional). People might mistrust the external people you hired. And so forth. But most importantly, people won't admit that they were wrong and keep on doing what they were doing, believing and assuming it is the right thing. This is especially true for sponsor, project manager, and project members as they have invested the most. Of course people will deny all of the above when you ask them.

So next time you do a project. Please make sure your KPIs are set on measuring the Business Case on a frequent basis and not only listen to the people involved, but observe what they actually do. And be brave: dare to stop projects.

How to lie with your KPI

There are lies, damn lies and KPIs

Often KPIs are used to manage and prioritize activities within the organization. It is expected that employees act upon the KPI outcome. If in the end the KPI does not show any progress, it might have an effect on the performance assessment of the people involved. Especially in an autocratic lead company these consequences of KPIs turning RED might be harsh (see also my blog on management styles and KPIs).

A too rigorous usage of KPIs in relation to people management might lead to a culture of fear. Employees will do their best to avoid the RED status. Most of them will make sure that the KPI is moving up (or down) by just doing their best (hoping the KPI turns or stays GREEN). Others might go a little bit further to keep the KPI in status GREEN. 

Manipulating techniques are not that hard. Knowing a few of them might even come in handy. Not to use them yourself of course! No, just to recognize them when encountered. In the end: it takes a thief to catch one. I've listed six techniques here. They are subtle and are meant to create a smokescreen around the real results. In other words, they help presenting the results better than they really are.

1. Play with thresholds
This trick was already mentioned several times in my previous blogs. It is very easy to manipulate the thresholds above (or below) which you indicator-status turns amber or red. As long as you stretch the threshold long enough, your status will stay green. See here an example I found depicted in an article called "Why Red Amber and Green (RAG)?" at intrafocus.com.

It doesn't take much fantasy to see that one could easily make the green area larger by increasing the lower threshold.

2. Lie with your graph
Playing with your threshold is not the only technique. There are many tricks you can play with the way you present the results. Especially line graphs are easy to manipulate by altering the Y-axis. Consider your starting point, end point and scale of your axis. Use suggestive labels or add chart junk. Use two Y-axis to confuse your readers (if you can't convince them, confuse them). This short blog however isn't the place to discuss all these different techniques. There are some really nice books that I can recommend (1) (among which is my own book on the misuse of statistics ;-)

3. Work with percentages
The percentage is a very popular statistic. That's because almost everybody above 10 has a basic understanding of what it stands for.  The Dutch author J. Bakker once said: "percentages are like bikinis. They bring you to all sorts of ideas, but hide the essence". That's probably why percentages are used in commercials all the time (31% less wrinkles! 70% less fat!). Most often they are as hollow as the claims they support. This is because only the percentage does not say anything. It's the absolute figures behind them that really count. An increase of 200% sounds very impressive, but could mean everything or nothing (going from 1 to 2 is also an increase of 200%).

4. Choose your average wisely
Let's say you launched a new website and you want to see how successful it is. Your KPI is wisely chosen. Not the number of hits, but the average time people stay at your website is your performance indicator. The longer the better. The picture below shows three possible outcomes on how many people stayed a certain amount of minutes on your website.

Now have a look at the average-measurement most often used: the mean. Depending on the "skewness" of the results your mean could be lower or higher. So let's say that most people only stay a short time on your website (represented by the graph on the right). Using the "mean" as your type of measurement however gives you the impression that that the amount is higher. This is because the few "fans" that stay at your site a long time, push the mean upward.

5. Leave out certain data

KPIs don't like extremes or outliers. These incidents might influence your indicator and result in a (temporary) RED or AMBER status. So one of the most tricks used is to just name these extremes "incidents" or "a coincidence" and remove them from your graph.

6. Aggregate your KPIs
Remember the cartoon from the "Manager styles and KPIs" blog? The one with the birds? What does the bird at the top see? Not much really. If there is a KPI set for each of the departments below the bird on the top, most likely the overall status will turn GREEN every time. This is because aggregating three values (RAG) will do that. Look at the picture on the left. Even if there is are many AMBER and RED departments throughout the organization, the top level KPI is green. The chance of the top level KPI turning RED is very small. This is because working with the RAG structure makes you limited in the way you aggregate them upwards.




(1) recommended reads on manipulation with graphs: "Van tofu krijg je geheugenverlies" - Coen de Bruijn and "How to lie with charts" - G.E. Jones