The Trials and Tribulations of Learning Measurement
Big data is big news. It is essential in today’s business environment to demonstrate an impact from training initiatives.
The ROI Institute surveyed CEOs from several Fortune 500 companies and found that just 8% of respondents reported their business to currently measure training impact, yet 96% said it was their number one priority.
Likewise a Talent Pulse report published in 2015 from the Human Capital Institute and CGS Learning described how in 2012, U.S. organisations spent $164b on training and development, yet only 39% of practitioners reported that their organisation regularly reviews and improves the delivery of L&D programs and only 21% measure if learning is used on the job.
The measurement of L&D programs has been focused on tactical, present-focused metrics rather than anticipating future needs or evaluating the business impact. With the lack of measurement and infrequent strategy discussions, L&D functions can miss opportunity to add value to the business and its employees.
I was recently reading another article from Chief Learning Officer (CLO) Media around measurement. They frequently collaborate with IDC and survey the CLO magazine’s Business Intelligence Board on learning trends. They found that between 2010 and 2015, an overwhelming number of CLOs were dissatisfied with the tools, resources or data available to them to measure learning impact.
However, CLO media think that this landscape is changing. While L&D measurement seems to have fallen behind other areas for the past 5 years, a 2016 survey has found that more than 60% of CLOs report their measurement processes are “fully aligned” with their learning strategy, and more are now satisfied with their approach than dissatisfied.
Interestingly, CLO media found that there appears to be a strong correlation between effective measurement and the perception of impact. With poor measurement in place, most CLOs perceive that they have low influence in helping to achieve organisational priorities. Conversely, when organisations are satisfied with their measurement processes in place, they also believe that L&D plays a more important role in achieving organisational goals.
It remains challenging to gain access to reliable metrics and finding the time and resources to analyse the data. They found that CLOs often report a “lack of data or access to data” or “a lack of leadership support.”
CLO and IDC also found that organisations continue to try and correlate learning to organisational change. Yet a focus remains on happy sheets and the like, alongside knowledge quizzes, with less than 50% measuring retention. I was glad to read their affirmation that there is a clear opportunity to use measurement to establish the impact learning has on wider business outcomes. If you feel your happy sheets or “smile” sheets are due a makeover then Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form is a must read. I am loving learning from author, Will Thalheimer.
In the past, I have been fairly scathing about the industry-wide obsession with evaluation. Although it’s great to see some progress, it’s clear that measurement remains a challenge for many. Attempting to fudge, hide or ignore results by cherry picking what to evaluate is pointless. Genuinely evaluating success or failure against worthwhile objectives is not. Evaluation of success is an important part of any endeavour. After all, if we cannot establish whether we have met or exceeded our objectives then it is very hard to establish if something has been worthwhile or not. If we cannot work that out then we could be throwing good money after bad.
The trends in evaluation are suggesting that people are finally waking up to the fact that anything below Kirkpatrick’s level three evaluation is meaningless. The key to transformation and genuine training success is application or behavioural change.
Jack and Patti Phillips from the ROI Institute are leaders in the field of evaluation, having written 50 books on the subject. The Kirkpatricks have also made their mark in the area and it was Donald Kirkpatrick who first identified the importance of behavioural measures in evaluation.
While many think of the ROI Institute as being all about ROI it is much more than that. The Phillips’ business alignment model emphasises the importance of tying the evaluation to the objectives. For evaluation to be truly effective we must start with the end in mind. Only when we set the objectives at the start can we effectively evaluate against those objectives at the end.
With so much talk around the trials and tribulations of learning measurement, hopefully this gives you some food for thought and an opportunity to move measurement up your agenda this year.