SEPARATION: HOW TO EVALUATE THE COACHING JOURNEY
At all phases of the coaching relationship, both the WECoach and the coachee should feel motivated and confident that each is contributing toward shared goals. It’s common, once the coaching relationship has served its purpose and the long-term goals have been achieved, that the relationship comes to an end.
The relationship may also end for other reasons, such as: the relationship has achieved its objective, you feel that your mentee is confident and ready to move on, you have tried but the mentee is not responding, the programme is coming to a close, the relationship isn’t working successfully and both parties wish to move on. But both the WECoach and coachee have the responsibility to make the relationship end happily. The coachee may feel abandoned, deceived or unprepared in case of premature separation. WECoaches may feel used or deceived if the coachees do not consult them or seek guidance anymore.
The separation stage is not only important but also necessary. It marks the end of the coaching relationship and assesses the coaching effectiveness. There are several ways to evaluate a coaching journey, but the most important aspect to be taken into consideration is that it should be done along the mentoring process – in this way changes can be included during the implementation phase like changes of mentors/mentees, communication updates or even cessation of the relation.
WECoaches have specific responsibilities in the separation phase:
- Be sensitive to when the relationship has run its course,
- After formal mentoring relationship is finished, they need to follow up on their coachee’s successes,
- Provide a summative evaluation of the experience,
- Say “thank you” and give credit where credit is due,
- Learn from their experience when coaching others.
Defining outcomes and evaluation measures
Measuring the impact and results of the coaching programs is key for the success of the relationship. Without it, it’s not possible to gauge how successful or beneficial the program has been and nor the areas of improvement are.
Coaching is a non-formal way of learning and is not linked to nor based on any specific programme. This can lead to an apparent lack of clear aims and objectives at the start of the process which causes difficulties for the evaluation of the learning process. Defining learning outcomes in coaching processes is of the utmost importance in order to set up the evaluation measures (Carter, 2006).
Since coaching involves individuals’ growth and development, each person involved will get different things out of it. Whereas in formal training all involved parties know the same thing to the same level. This affects majorly the evaluation stage pf the coaching process since there are no universal nor predefined criteria or indicators to evaluate the parts involved. However, this does not mean that there is no evaluation.
Demonstrating the effectiveness and real work-related contribution of a coaching process can be a harsh task. In fact, coaching works at a non-formal level and in deep individual circumstances which makes evaluation a complex task. But the evaluation stage is a crucial moment for the overall coaching process that cannot be skipped nor underestimated in importance since it proves the actual achievement of the learning objectives and outcomes pre-established.
Coaching is also a less traditional form of learning and thus needs to be further sustained if put in comparison with more established forms of learning which are supported by extensive literature and academic research. There are not many forms of traditional learning which can impact the behaviour of the individual as in the case of coaching, thus it is worthy to explore more in deep a valid evaluation method that can be easily associated with coaching. Since traditional learning is not as challenged as coaching in terms of value and respect among the widest public, addressing evaluation methods in coaching can influence its credibility and success.
The WECAN project provides a framework that can be used to develop a well-rounded evaluation plan to gauge how impactful the coaching has been. Based on the widely used Kirkpatrick Evaluation Model, this guide teaches a technique to evaluate the WECAN coaching program.
Donald L Kirkpatrick's evaluation model
Donald Kirkpatrick, Professor Emeritus at the University of Wisconsin, published his model in 1959 and then revised it in 1975, and again in 1993. In 1993 he also published his most famous work Evaluating Training Programs.
In the Kirkpatrick’s evaluation model each successive level represents a more precise measure of the effectiveness of a training program. In 2016 the model was revised again by his son and his son’s wife who emphasised the importance of making training relevant to people's everyday jobs.
The four levels of Kirkpatrick's evaluation model essentially measure:
His measures are recommended for full and meaningful evaluation of all learning outcomes and although Kirkpatrick's evaluation model is addressed to evaluate training programmes, it can be easily adapted to coaching.
Kirkpatrick's four levels of training evaluation in detail
This table[1] illustrates Kirkpatrick's structure in detail, and particularly its more modern interpretation, usage, implications, and examples of tools and methods.
Level I - What is measured: REACTION | ||
|---|---|---|
|
Evaluation description and characteristics |
Examples of Evaluation Tools and Methods |
Relevance, Practicality and Application |
|
Reaction evaluation is how the learners felt, and their personal reactions to the training or learning experience, for example: Did the learners like and enjoy the training? Did they consider the training relevant? Was it a good use of their time? Did they like the venue, the style, timing, domestics, etc? Was the level of participation sufficient? Ease and comfort of experience. Level of effort required to make the most of the learning. Perceived practicality and potential for applying the learning. |
Typically, 'happy sheets'. Feedback forms based on subjective personal reaction to the training experience. Verbal reaction which can be noted and analysed. Post-training surveys or questionnaires. Online evaluation or grading by learners. Subsequent verbal or written reports given by learners to managers back at their jobs. |
Can be done immediately the training ends. Very easy to obtain reaction feedback Feedback is not expensive to gather or to analyse for groups. Important to know that people were not upset or disappointed. Important that people give a positive impression when relating their experience to others who might be deciding whether to experience same. |
Level II - What is measured: LEARNING | ||
|---|---|---|
|
Evaluation description and characteristics |
Examples of Evaluation Tools and Methods |
Relevance, Practicality and Application |
|
Learning evaluation is the measurement of the increase in knowledge or intellectual capability from before to after the learning experience: Did the learners learn what was intended to be taught? Did the learners experience what was intended for them to experience? What is the extent of advancement or change in the learners after the training, in the direction or area that was intended? |
Typically assessments or tests before and after the training. Interview or observation can be used before and after although this is time-consuming and can be inconsistent. Methods of assessment need to be closely related to the aims of the learning. Measurement and analysis is possible and easy on a group scale. Reliable, clear scoring and measurements need to be established, so as to limit the risk of inconsistent assessment. Hard-copy, electronic, online |
Relatively simple to set up, but more investment and thought required than reaction evaluation. Highly relevant and clear-cut for certain training such as quantifiable or technical skills. Less easy for more complex learning such as attitudinal development, which is difficult to assess. Cost escalates if systems are poorly designed, which increases work required to measure and analyse. |
Level III - What is measured: BEHAVIOUR | ||
|---|---|---|
|
Evaluation description and characteristics |
Examples of Evaluation Tools and Methods |
Relevance, Practicality and Application |
|
Behaviour evaluation is the extent to which the learners applied the learning and changed their behaviour, and this can be immediately and several months after the training, depending on the situation: Did the learners put their learning into effect when back on the job? Were the relevant skills and knowledge used? Was there noticeable and measurable change in the activity and performance of the learners when back in their roles? Was the change in behaviour and new level of knowledge sustained? Would the learner be able to transfer their learning to another person? Is the learner aware of their change in behaviour, knowledge, skill level? |
Observation and interview over time are required to assess change, relevance of change, and sustainability of change. Arbitrary snapshot assessments are not reliable because people change in different ways at different times. Assessments need to be subtle and ongoing, and then transferred to a suitable analysis tool. Assessments need to be designed to reduce subjective judgement of the observer or interviewer, which is a variable factor that can affect reliability and consistency of measurements. The opinion of the learner, which is a relevant indicator, is also subjective and unreliable, and so needs to be measured in a consistent defined way. 360-degree feedback is useful method and need not be used before training, because respondents can make a judgement as to change after training, and this can be analysed for groups of respondents and learners. Assessments can be designed around relevant performance scenarios, and specific key performance indicators or criteria. Online and electronic assessments are more difficult to incorporate - assessments tend to be more successful when integrated within existing management and coaching protocols. Self-assessment can be useful, using carefully designed criteria and measurements. |
Measurement of behaviour change is less easy to quantify and interpret than reaction and learning evaluation. Simple quick response systems unlikely to be adequate. Cooperation and skill of observers, typically line-managers, are important factors, and difficult to control. Management and analysis of ongoing subtle assessments are difficult, and virtually impossible without a well-designed system from the beginning. Evaluation of implementation and application is an extremely important assessment - there is little point in a good reaction and good increase in capability if nothing changes back in the job, therefore evaluation in this area is vital. Behaviour change evaluation is possible given good support and involvement from line managers or trainees, so it is helpful to involve them from the start, and to identify benefits for them, which links to the level 4 evaluation below. |
Level IV - What is measured: RESULTS | ||
|---|---|---|
|
Evaluation description and characteristics |
Examples of Evaluation Tools and Methods |
Relevance, Practicality and Application |
|
Results evaluation is the effect on the business or environment resulting from the improved performance of the learner - it is the acid test. Measures would typically be business or organisational key performance indicators, such as: Volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organisational performance, for instance; numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc. |
It is possible that many of these measures are already in place via normal management systems and reporting. The challenge is to identify which and how relate to the learner's input and influence. Therefore, it is important to identify and agree accountability and relevance with the learner at the start of the training, so they understand what is to be measured. This process overlays normal good management practice - it simply needs linking to the training input. Failure to link to training input type and timing will greatly reduce the ease by which results can be attributed to the training. For senior people particularly, annual appraisals and ongoing agreement of key business objectives are integral to measuring business results derived from training. |
Individually, results evaluation is not particularly difficult; across an entire organisation it becomes very much more challenging, not least because of the reliance on line-management, and the frequency and scale of changing structures, responsibilities and roles, which complicates the process of attributing clear accountability. Also, external factors greatly affect organisational and business performance, which cloud the true cause of good or poor results. |
The Kirkpatrick model is an effective evaluation tool and is particularly effective in coaching evaluation because the coachee can play an active part in the design of the evaluation process and can collaborate with the WECoach in the evaluation stage.
Applying the Kirkpatrick model to a coaching process
Although the Kirkpatrick model has been designed to analyse and evaluate the value of a training to the business, it is easily adaptable to show the effectiveness of a coaching programme.
The four levels of evaluation represent the fundamentals of the Kirkpatrick Model. Each level that makes the system requires more resources than the previous one and each level is the foundation of the next one. The deeper the evaluation goes into the levels, the more time-consuming and demanding the evaluation it becomes.
Level I: Reactions
The objective of the first level is to understand whether the coachee has liked the coaching session/programme and how she feels about it. The first level is thus quite straightforward, since it aims at posing questions directed at evaluating the impressions and thoughts of the coachee. This is why this first level is often associated to a smile sheet.
This first level cannot be skipped, after all, all the subsequent evaluation steps depend on this one. In fact, if the result of this first evaluation is poor, it would signal that the coachee level of participation is low and she will most likely drop out or invest too little effort.
One way to get Level 1 feedback for the WECoach is to send out a survey to the coachees with questions such as:
- What was the most meaningful part of the coaching session/programme?
- On a scale from 1 to 5, how valuable has your coaching experience been for your entrepreneurial career?
- Did you enjoy participating in the programme?
Image 1 Level one: Reaction[2]
This first level of assessment does not tell whether the entire session or programme has been successful. However, it is a first important hint to have an understanding of the experience of the coachee in the program and it can early uncover negative issues.
The downsize of this first level is that it can be situational and the coachee can be easily influenced by her mood when replying to the questions. That’s why a second level evaluation is required.
Level II: Learning
After having detected the initial reactions of the coachee, in Level II the WECoach will seek to measure how she has learned over the course of the coaching session/programme. It is worth underlying that the learning outcomes of a coaching programme are not always technical or skill-based. Most of the time the important learning that goes on in a coaching programme is attitudinal.
Image 2 Level 2: Learning[3]
In order to understand whether the coachee has acquired new knowledge, skills or attitudes and whether she has learned or not, the second level evaluation brings forward techniques. One way could be using a pre- and post-survey or just a final self-assessment.
Level III: Behaviour
The third level of Kirkpatrick’s model seeks to measure the behavioural change of the participants in the coaching programme. In order to understand which behavioural transformations to address during the sessions and how to measure them, the WECoach needs to establish clear goals since the very beginning of the programme with their coachees. Behaviour changes are important in coaching programmes that focus on leadership development or entrepreneurial-related development.
Coachees may start to make behavioural changes during the duration of the coaching process. If this does not happen, then WECoaches might need to investigate what is not working during the sessions and try to address new forms of learning.
Image 3 Level 3: Behaviour[4]
An example of third level evaluation would be to understand whether the coachee is learning or not about a skill she needed to learn during her coaching journey. Let’s say the coachee needs to acquire new leadership skills as a result of the coaching programme. Most likely, she won’t apply these new skills in her entrepreneurial career if she doesn’t support the leadership perspective presented in the coaching programme.
A commonly used method to measure behavioural transformation is via interviews or observations. A good idea would be to get others involved in providing information. For example, someone who works directly with the coachee or a close friend might be able to provide behavioral observations that can be helpful in measuring the success of the coaching programme.
Assessing the behavioural transformations makes it possible to understand if the new acquired knowledge, mind-set or skills are actually being applied. This evaluation level is considered to be the truest and deeper way to understand whether the learning programme has been successful. This kevel presents many difficulties though. In fact, it is quite complicated to understand when the coachee will start to use the knowledge acquired making it complicated to know when and how to measure the behavioural changes.
Level IV: Results
In the fourth level of the Kirkpatrick model, the WECoach can measure the results of the programme. These results should be coherent with the objectives of the coaching programme that were set in the initiation stage. This is why, although challenging, it is of the utmost importance to define the goals and expected outcomes of the coaching programme right at the beginning of the relationship.
Image 4 Level 4: Results[5]
This level is the one that is most commonly avoided in the trainings evaluations. It is actually the primary objective of the model, but somehow is always skipped by assessors and evaluators. For WECoaches it is interesting to determine whether their coachees have achieved successful outcomes in their entrepreneurial careers such as getting a grant or an investor, achieving a more balanced work-life routine or more self-esteem. Although this is critical to determine, WECoaches will find this exercise useful for their learning process as coaches as well. Assessing a coachee represents a great tool to understand whether the delivered coaching programme has achieved the expected goals.
Timing of evaluation
Evaluation directly after the coaching experience may not be appropriate as the benefits may only accrue later on when the WECoach or the coachee fully reflects on the benefits that the relationship has brought them. In this sense it can be seen as similar to evaluating careers guidance. Although this is true also of other forms of training, whereby you may not get to use your skills until a later date, it is more likely that you will be able to anticipate or recognise where training will help.
With coaching, the outcomes of what is learnt or discussed may not be immediately obvious, either to the WECoach or to coachees, unless they are linked into a wider programme (e.g. leadership development, culture change). As the defined success criteria may not be agreed upon until the coaching process has begun, it is difficult to measure pre‐ and post‐coaching intervention to assess the impact. The second issue is that information with which to evaluate may not be readily accessible in a coaching situation (e.g. because of the confidential nature of the coaching context).
Summary of Kirkpatrick’s model
The Kirkpatrick Model is a great tool to shape and design the coaching process. In fact, the model provides an insight on what is happening during the course of the process: if the expected results are not being accomplished, the Model will tell the WECoach in due time and she will have the chance to take corrective measures.
There are of course other models which work fine as in the case of the one chosen in this methodology. WECoaches can opt for other models which make them feel more comfortable with. What is more important for WECoaches, is to correctly use the models to have a clear idea of the results being achieved. Only in this way the coaching process can be successful.
