“Hire character, train skill.” – Peter Schutz
I was asked recently, “what is the value of training?” After recovering from the challenge of the question, we had a great discussion of my friend’s perceptions of training (which were negative based on recent experiences) and of how effective training could be different. This led to a discussion of how to measure the impact of training, for which there is one key reference (and deservedly so).
The four levels of evaluating training programs
Why do we need to evaluate training (or other change management processes)? Put simply, we need to evaluate the return on investment (or business contribution) of training and coaching programs, as well as continuously determine how we can improve them in the future.
Donald Kirkpatrick first developed his training evaluation framework in 1959, and it is still the most widely used today (although not widely enough). The framework is very straightforward, comprising four levels with the highest level reflecting business results. The framework provides a clear and structured way to think about evaluating any program of change management, and ca be used far beyond training (for instance to evaluate marketing programs or market research studies). It is also a useful way to think about measuring customer relationship management (CRM).
The approach is comprehensive, and based on four levels which build on each other: Reaction, Learning, Behaviour and Results. Some users also add a fifth level which is really only an elaboration of the fourth.
The first level focuses on evaluating reaction, which is essentially the same thing as measuring customer satisfaction. Evaluation is typically conducted through ‘happy sheets’ and is the most common form of training evaluation. Measuring immediate reaction is very important and should not be underestimated. It gives valuable feedback on training and how it can be improved, it provides reassurance that the trainers appreciate the need to help participants, it is a good mechanism for providing feedback to internal customers (ie line managers) and it provides a framework for comparing standards of performance across training.
Here are some simple guidelines for using evaluating reactions:
- Be clear what you need to find out (and what you don’t)
- Encourage written comments and suggestions
- Get 100% immediate response (at the time of the training)
- Get honest responses
Learning is the next stage and is defined as the extent to which participants change attitudes, improve knowledge and/or increase skill as a result of the training. Some training programs (and marketing programs) aim to change attitudes, some technical programs focus on improving knowledge and/or skills, and some might combine all these. Learning here is defined slightly differently from behaviour (see below), but for the purposes of the second level of evaluation I would consider learning to have take place if attitudes are changed, skills are improved or knowledge is increased.
These are necessary pre-requisites of the next stage and relatively easy to evaluate. Moreover, it is important to evaluate this stage, as measuring behavioural outcomes is more difficult and, more importantly, is affected by other influences – even if someone demonstrates real learning, they may not be able to change behaviour if they are working in the wrong climate and are not encouraged to change.
Examples of measuring learning are to measure skills or knowledge before and after the training (but always consider measuring a control group in order to determine the real impact of the training). Paper and pen tests can be used to measure attitudes and knowledge and performance tests to measure skill.
The third level focuses on the extent to which a participant’s behaviour changes as a result of attending the training. Generally, in order for someone to change behaviour four conditions are needed:
- They must have the desire to change
- They must know what to do and how to do it
- They must work in the right climate
- They must be rewarded for changing
While training can help with the first and second of these, it is not enough to ensure a change in behaviour, which also depends on the participants supervisor, colleagues and management providing a supportive environment and intrinsic or extrinsic rewards (which is why it is so important to get all stakeholders to buy in to training programs). Importantly, behavioural change also takes time (and opportunities to use the learning), and repeated practice of the required behaviour. For example, Tapestry Works includes ongoing coaching in some training programs, to ensure continued practice and appropriate feedback to embed learning and translate it into behavioural outcomes.
The fourth and final level evaluates the final results that occurred because of a training program. These could cover a wide range of business objectives: improve productivity, decrease costs, increase sales, higher profits or other goals. It is important to think of these ultimate outcomes when planning training programs, even if they are long term objectives, and to consider how they can be measured.
To some extent, these long term outcomes are easier to objectively measure than behavioural outcomes as long as they are explicitly stated in advance. Productivity, costs, sales and profits can all be accurately measured and tracked over time, and even used to evaluate the ROI of training when linked to the training’s cost (versus other programs or alternative business strategies).
Good training programs always pay off, and the pay off can be measured. As with all change management, pay off is best when the business needs are well understood, target audience identified, desired outcomes agreed in advance, program well constructed to meet the objectives, and training follows the best practice based on adult learning theory (as we have written about previously). Measuring outcomes is straightforward when you are clear on which outcomes your business needs.
Evaluating Training Programs: The Four Levels by Donald & James Kirkpatrick