Approach for a fair comparison

Many benchmarks use the mean value of the comparison group as a reference value. However, the mean value is often regarded as the “mediocre average”. The median can be used as an alternative approach to achieve a fair comparison for customers and service providers.

“At the request of one of the parties, the parties shall conduct a joint benchmarking procedure”. This sentence is now almost a standard inclusion in many IT service contracts and will sooner or later result in a market price benchmark being established. The objectives of conducting the benchmarking process are varied – checking the service level, comparing the IT service structures, auditing the responsibilities matrix or, for example, conducting an IT strengths and weaknesses analysis.

A benchmarking process is usually initiated by the customer. Generally the customer also took the first step towards outsourcing – in this case a strategic decision was made to work with an external service provider and start the process of the request for proposal (RFP), hold talks with potential partners and award the order. Many management concepts recommend that decisions be reviewed and also advise that corrections be made if the initial objectives are not achieved. In other words the aim is to check and review outsourcing decisions just like any other decisions. Benchmarking is a foreseeable request during the term of the contract and should therefore be included in the contract from the very outset. The more precisely the rules are described the greater concentration can be dedicated to conducting the benchmarking process.

Many benchmarks use the mean value of the comparison group as a reference value. However, the mean value is often regarded as the “mediocre average”. From a mathematical point of view, the mean value is of course the average but no valuation is carried out in mathematics. The average value is rarely used in the objectives of a company, however. Every company strives to win one of the first places in terms of market share, customer satisfaction, turnover yields and profits. So when a benchmarking process is carried out and the mean value is used for comparison purposes, this does not appear to cover the actual objective of the exercise. “We want to be better than the average” is the most common argument when looking for alternatives to the mean value.

The characteristic figure which benefits most commonly from this situation is the first quartile. In simple terms the first quartile describes the values from the group of peer companies which are less than 75 percent of the remaining comparison group. In a typical benchmark comparison group of six to ten peers, the first quartile will therefore be between the second, third and fourth best comparison companies. Specific comparison values are defined for in around 30 percent of benchmarking agreements. Examples of customer’s own reference values include a mean value between the minimum and the first quartile and a mean value of the two best comparison companies. It is easily possible to go well over the target as the effects of the reference value “minimum minus 25 percent” does not become clear to the parties involved until the end of the project.

At first glance the customer can only win in this scenario since the benchmark will very likely be significantly below the current price level. However, serious questions must be asked as to whether the service provider can seriously compete at this level. The scope available to the service provider may be restricted to such an extent that he concentrates more on the services which are not covered by the contract and are charged separately than on day-to-day operations.

The median is a comparison value which is rarely used in benchmarking agreements. However, this value divides the comparison group in the middle and is less affected by statistical irregularities and therefore has a characteristic which is expressly desirable in benchmarking processes. Therefore the median can be used as an alternative approach to achieve a fair comparison which satisfies the requirements of both customers and service providers.

If the mean value is greater than the median, the mean value must be affected disproportionately by the high values in the comparison group (see graphic). In this case the maximum of the comparison group will be ignored. The benchmark reference value is then re-identified from this adjusted comparison group. In this case the mean value is again used. This value (the median-weighted mean value = MVw) then no longer suffers from the stigma of being “average“.

If the influence of the maximum is regarded as a distortion, the reverse conclusion must be drawn and the disproportionate influence of the minimum value also ruled out. Using the same procedure described above, the minimum from the comparison group is ignored if the mean value is less than the median. The (MVw) is then recalculated.

The requirement for a fair comparison is supported by everybody involved. Which calculation rules are ultimately used must be agreed by mutual consent and in transparent terms. Discussion about the advantages and disadvantages of the various reference values should be held in advance and result in a decision being made for a specific reference value. If various benchmarking values are included in the process there is a risk that attempts will be made to use the one that best fits what each party is attempting to achieve. Outsourcing partnerships which agree the details of a benchmarking clause when the contract is first concluded are well prepared and can concentrate their energies on executing a successful benchmark.

Karsten Tampier, Managing Consultant, Maturity

Go back