Recognising the pitfalls in quantitative success measurement, trustees and administrators have begun to change the way that service success is measured. New ways of looking at and interpreting data have seen administrators switch the focus to more qualitative service management. This can only be a good thing for the industry as a whole.

 

 

In the past, quantitative KPIs ruled the roost with measures like the number of enquiries dealt with, and the number of complaints raised, being used to give a feel for how ‘successful’ the service was. In reality this was a narrow view of how well it was meeting the requirements of the members.

 

It’s plain to see why this became the norm. Numbers like these are easy to obtain and measure, with little room for interpretation. They also give schemes an insight into performance that can be compared across the market. With the historical perspective that turnaround times in the industry have traditionally been poor, it comes as no surprise that this has naturally been an area of focus.

           

However, when it comes to measuring success and member experience, quantitative data used in isolation will never give the full picture.

Over past years, the need for more qualitative reporting has been driven largely as a result of the disillusionment of trustees, whose experiences didn’t tally with the successes being reported by their administrators. Trustees have often complained that the information reported didn’t align with the feedback they were receiving from members, and it didn’t fit with how they felt services were being delivered. Of course, everything might have happened within the timelines agreed, but were member expectations met, and was the right information and support provided?

More administrators also began to realise that there are more pieces to the jigsaw. Tunnel-vision for facts and figures has allowed analysis of what they are doing, but not how well it’s being done. To understand the overall performance of a scheme, qualitative information must be given just as much importance as quantitative data.

 

This has given rise to an overhaul in the way service management is managed and measured, and a new way of thinking about KPIs developed. New measures were introduced, with more administrators also reporting on factors such as member experience and feedback, internal error rates, levels of automation and self-service.

 

 

Measuring qualitative data is not without its problems and there are many obstacles to overcome if a scheme is to record its success through the use of both quantitative and qualitative data. One of the key reasons is because it’s more subjective, and therefore far more difficult to accurately measure when compared to the factual nature of quantitative data. Many also rely on a host of different inputs, many of which might be outside of an administrator’s direct control.

 

To really understand how well a pensions administration service is meeting its overall goals, schemes require effective systems to capture error and accuracy rates. Experience cannot be measured in a single metric. It spans everything from turnaround times, to communication clarity, to the standard of customer support provided over the telephone and online. So, it’s easy to see how it can be a struggle to measure the success or otherwise of the service and member experience.

 

One of the most obvious ways to gather feedback is, of course, the humble survey. Information collected via surveys is key in measuring member feedback, but its effectiveness is heavily reliant on joined-up thinking from the start. So, survey writers need to ensure that questions being asked, and the ways in which they are answered can be analysed in both a quantitative and qualitative way further down the line. They also need to be independent and not biased towards a pre-conceived result.

 

The effective analysis of member feedback through surveys also relies on two other important factors: whether members are willing to provide feedback, and how often they are willing to do so. In today’s world, we’re busier than ever, and our patience is at an all-time low, so it’s not unreasonable to expect that the vast majority of pension scheme members simply won’t have the time, or the inclination, to sit and fill out yet another customer experience survey.

 

Indeed, members tend not to provide feedback frequently enough for data to be statistically useful at a scheme level. And this problem is even more acute in the pensions industry. FluidSurveys has calculated that, as a whole, email surveys achieve a response rate of roughly 24.8%. But, when it comes to the pensions administration you’re lucky if you get a response rate higher than 10% of surveyed members.

 

However, well-targeted digital surveys that are short, timely and sent to a membership base that is motivated to fill them in can achieve higher response rates, according to SurveyGizmo.^[1] So, all is not lost in the search for qualitative member feedback. It’s just a case of ensuring that surveys are brief and offer motivation for respondents to answer.

If any scheme is to achieve its service goals, the importance of measuring success cannot be understated. Administrators have long relied on quantitative data in order to understand how well their processes met customer demand, but this alone will never give the full picture.

 

Neither qualitative, nor quantitative data can be used in isolation, but together these different types of information enable schemes to measure their success and understand whether or not strategic goals are being missed, met or surpassed.