Does your business know the true value that your training adds to the bottom line? If you are a training manager, do you find it hard to justify your training budget to the rest of the business? If you are a trainer or even in the operational side of your organisation, do you wonder whether the training you’re delivering (or participating in) really works?
It’s not what’s right or wrong, it’s what works!
Evaluation is one of those topics that stirs up strong emotion in training professionals. Experts in the field can often be found extolling the virtues of their preferred approach, and decrying any others as ‘wrong’. As you’d expect, we have a different view. We believe that the right evaluation strategy is the one that works for you, for your organisation and for the interventions that you are measuring.
We find that many of our clients, including many of the largest blue chip organisations, are just beginning to develop their processes for measuring their operational success. Naturally we work with these clients to help them develop their measurement capabilities, AND we find that they want to continue to measure the success of learning interventions in very specific ways. They expect approaches that suit their requirements at that time, rather than the prescriptive ‘bottom line’ measurements that some feel is the only way to evaluate training.
The Head of Customer Services in one of Europe’s largest energy suppliers was absolutely clear on how we were to measure the success of one intervention with her management team. “If I can see them doing more of the jobs that they should be doing then it will have been a success”. For her, no more scientific measures were needed. She would meet with them, and observe them, and if there were doing the job better, the training was a success.
Of course we have clients whose whole existence is based around measurement, and the approach we take to evaluating the success of interventions with them is very different. We have worked with clients to accurately measure the ‘£ sign’ savings of change interventions. We’ve also measured the savings from HR interventions and the cost reductions achieved by implementing the learning from Brain Friendly Learning Workshops in terms of reduced design and delivery times.
The common theme for me through all this is ensuring that the stakeholders, whether internal or external, get the information that they need to enable them to see that the interventions were successful. Applying the five principles of brain friendly evaluation will help you develop outstanding evaluation strategies specifically for your organisation and for your interventions.
Kaizen’s Big Five for Evaluation
Honour Uniqueness
Each organisation is different, as is each project and each training intervention. Why then do we try to apply one set ‘model’ for evaluation as a panacea for measuring training results? We agree that using Kirkpatrick or Phillips as a base approach to measuring the success of training can be useful. However we believe that evaluation strategies should be built around a set of tools and approaches that are right for that organisation or project, based on a factors such as outcomes, resources and expertise.
Keep It Real
How sophisticated is your organisation in collating and measuring data? Is every penny accounted for during projects or is it more important to see results in terms of customer feedback or product success? Our experience is that evaluation strategies rarely match the measurement culture in organisations. For example, in organisations where data collation and analysis is part and parcel of everyday working life, ‘evaluation’ often stops at gathering end of course reaction sheets. In other organisations, well-meaning training teams carry out full and detailed evaluations across varying levels, only to find that the business doesn’t value or understand the information. By matching your approach to evaluation to the organisations approach to measurement, you will ensure that you demonstrate your value without wasting valuable time and money.
Creation not just Consumption
Those of you who are graduates of our Brain Friendly Learning workshop will know that learning is about the creation of value, meaning and ACTION on the part of the learner, not merely the consumption of information. Anything generated by the learners will be many times more powerful than anything the trainer can create. This is also true for evaluation. If you can partner with your client group (whether internal or external) to develop the approach to evaluation, this can only be more powerful than a model developed by ‘the training department’ or some guru from the past. What measurement criteria does the organisation have in place already that you could use to measure the success of this intervention? Building a partnership approach to evaluation ensures that the business buys into your approach, and guarantees that you are able to deliver exactly what the organisation is looking for. In addition to this, you are taking a true consultative role by creating more ‘resourcefulness’ in your organisation.
Rich and Multi-Sensory
We already know that memories are stronger when multi-faceted, but what’s this got to do with evaluation? Well often the secret to a great evaluation strategy is in the richness of the exploration. It’s not just about achieving all of targets and meeting all of the needs; it’s about being able to show a range of customers the results that are important to them. What results will be most important to those taking part in the training? What is important for their line managers? What about the HR Manager or the Finance Director? Having explored and gathered the data you need to answer these questions how can you deliver the information in a memorable and powerful way? Even the most detailed and professional ‘evaluations’ tend to end with a report that is circulated around the organisation
There might even be a PowerPoint presentation to back up the report and this is presented in the usual ‘sage on the stage’ style to anyone who will listen. We believe that the results of an intervention should be shared in such a way that the success of the project is remembered as strongly as the intervention itself.
State is Everthing…(well, almost)
Have you ever considered what state you want to generate from your interventions as a whole? Even more than that, have you ever considered what state you want to generate through your approach to evaluation? Can you imagine what it would be like to generate a state of intense excitement through the results of your evaluation? We believe that the core outcome of your evaluation is to leave the learners, the organisation and you in a better state than before. This should be through a combination of all of the other principles and of course will be led by yourstate. Naturally results are vital, but how you arrive at these results can be just as important – as is how you share them, especially if the bottom-line results aren’t quite as positive as you’d have expected. We must use everything we already know about the power of state to ensure that our evaluation strategies have the maximum impact.
Some Models and Tools Behind Evaluation
Kirkpatrick: the best-known and most used model for evaluation
Level 1 – Reactions
Level 2 – Learning
Level 3 – Behaviour
Level 4 – Business Impact
Phillips
Perhaps the next best-known model. Similar to Kirkpatrick’s and includes some extra vital levels.
Level 1 – Participants’ Reaction and Satisfaction
Level 2 – Learning Results
Level 3 – On-the-Job Application of Training
Level 4 – Business Impact of Training
Level 5 – Return on Training Investment
Intangible benefit
The ROI Calculation
The best-known formula for describing the financial benefits of training.
Return on Investment = Net Training Benefits * 100 = divided by costs of training = ROI (as a percentage)
The Benefit-Cost Ratio
An alternative for evaluating the improvements generated by interventions.
This formula compares the annual financial benefits of an intervention to the cost of the intervention by a ratio.
BCR = Training Benefits
Training Costs
An Alternative To Happy Sheets?
We’re often asked if there is an alternative way to collect this ‘did the course go OK?’ data. It’s recognised by many as having limited value AND is key to ensuring that the learning experience was a positive one for those taking part. Here’s a very simple alternative that some clients find valuable.
Close to the end of the intervention create three flipcharts for learners to add information to. Write a heading on each, ‘What did you love?’ ‘What would have made it even more valuable’? ‘What will you do differently as a result of this training?’
Collate the data and feed it back to the stakeholders in an appropriate way (see ‘Rich and Multisensory’.)
Can the effects of training be isolated from other factors?
This is a question that is regularly asked when discussing evaluation with clients and, while I think it is technically possible, creating an absolute isolation of the impact of training from other initiatives that usually coincide with large scale training interventions requires significant time and resource investment. In fact implementing the evaluation strategy itself could make a significant dent in the ROI provided by the training! Add to this the impact that ‘effects’ such as Hawthorne and Pygmalion can have, I believe that it would be rare for an organisation to want to explore this complex area in significant detail.
Of course should the occasion arise we’ve got just the tools to enable you to do it! Here are a few of our favourites.
Control Groups
Select at least two random groups and collect performance data relative to the training for both groups. One group (the experimental group) benefits from the learning intervention, another (the control group) does not.
The groups’ performance is measured and evaluated. The data is then processes to draw differences in performance since the intervention. This approach takes significant steps towards isolating the effects of the training, as it is the comparison between the groups that is most important rather than simple increases in performance.
Trendline Analysis
Effectively this is a process where historical data is used to predict future performance. The actual performance following the learning intervention is then plotted against the projected results. While this isn’t an exact science and there are many factors that could affect the trend projection it is a valuable tool as part of a detailed strategy and is particularly useful as it is simple, time and cost efficient.
Other Tools for Collecting Data at Different Levels
Line Manager Interviews
360 degree feedback
Forecasting
Participant Interviews
Customer Survey
Action Planning
Peer Coaching
Staff Survey
Delivery against agreed outcomes
Workplace Observation
Line Manager Surveys
Participant Surveys
Case Studies
Business Games
Participant Knowledge Tests
