Apply Kirkpatrick’s 4 Level Evaluation Program to Organizational Change Initiatives
In TRDV 433 Organization Change, students study various interventions that can improve effectiveness at the individual, team, and organization-wide levels when used correctly. Most change agents agree that selecting the correct intervention is both an art and a science. Since billions of dollars were spent on organizational change initiatives over the last two decades, many in the field believe it is time to emphasize the “science” component to ensure that we can document change in a way that justifies expenditures. One way to evaluate change, particularly from the perspective of the training profession, is to apply Kirkpatrick’s model to the change process.
Kirkpatrick’s Evaluation Model
As depicted below, Kirkpatrick’s Four-Level Evaluation Program involves the assessment of four unique dimensions. Therefore, when applied to a change initiative, we can use the model to answer such questions as:
- At level 1, how did members of the organization react to the change? We may expand this question in terms of effects on job satisfaction, intent to leave, relationships with coworkers and leaders, etc.
- At level 2, consider what was learned from the change initiative. More importantly, ask how we can use lessons learned to support the change and build a learning organization.
- At level 3, consider the sustainability of the change effort, measuring its continued effects over time.
- Finally, at level 4, consider the results of the change initiative in measurable terms and based on the original objectives or reasons to initiate change. Did we achieve what we set out to achieve?
Kirkpatrick’s model offers a comprehensive approach to the measurement of change process outcomes. It provides a structure that ensures that change outcomes are evaluated from a multifaceted perspective. Rather than relying only on financial results, applying the four levels leads us to consider how change initiatives affect the individual worker, the learning organization, whether they are sustainable, and ultimately whether they produce the desired outcomes.
Questions for Discussion:
- Why do you think that organizations fail to measure the results of change initiatives accurately?
- What strategies might organizations use to conduct a level 4 evaluation of a change initiative?
RU Training @ Roosevelt University in Chicago 
Organizations sometimes fail to measure the results of change initiatives accurately by failing to fully gather context at the outset–often in Level 1. For example, I recently observed an organization survey its staff on changes to work from home policies due to COVID-19. The survey results were interpreted by leadership as “most staff feel it is more difficult to work from home than in the office”. However the survey failed to account for other difficulties (i.e. childcare) that have made working from home more difficult in the past year. So, management’s reading of the results of the survey does not fully reflect the view of the employees’ who have been working from home, which in turn feeds distrust of management.
Excellent point about the need to collect valid data. In this situation, it may have helped to begin with focus groups or interviews to gather information about the issues at hand.
Agreed Kathleen. Absent robust data it is very difficult to solve for the problem(s). Interviews would have been ideal.
I suspect It takes precision, time and in-depth knowledge to ensure the correct method is selected for specific situations. There are such a selection of methods to choose from and choosing the wrong one may be the death knoll for any organization’s change process.
For this area to grow even further, it is important that the science is highlighted. Companies will not continue their dollar spends if there is no emphasis placed on the science component.
Replying to Kevin’s comment here: agreed. The expenses are not to be underestimated. Investment into solutions should be backed up by data.
I think organizations fail to measure the results of change initiatives accurately because there is no clear and compelling information for change; people do not understand and face anxiety and resistance. Information about change should be clear so employees can understand the implications and can feel engaged during the process of change. Lack of trust and empowerment are factors that contribute for organizations to measure results.
Lety, intriguing thought re: “clear and compelling information for change”. I agree, the nature of most organizational structures do not communicate a full case for change to all employees. Do you think it would help the change process if that communication were more robust? What implications might come along with that, since so many factors are unknown–even to leadership–at the beginning of the change process?
I agree Lety it is difficult to think about evaluation when the initial objectives are not clear.
I agree Lety, I think Trust is a key word for many organizational changes. When people learn there is a change in management or a merge is happening, people start to worry about their jobs. If an employee has the trust towards their department from the beginning, I don’t see that there would be any issue in adjusting to the changes. An organization should prepare for the team that the change will effect. They should get to know them and understand their capabilities and productivity before making the change. This will assist later when comparing the before and after.
– MAttia
I think organizations fail to measure the results of change because the change is unclear. When people don’t understand why change is necessary, anxiety, cynicism and resistance inevitably build.
Most organization fail to measure change solely because they are very comfortable in the way they do things. In order to be successful at the level 4 stage the organization must become uncomfortable in order to meet the goals at level 4.
I believe companies fail to measure the results of change accurately because they may not ask enough people about their results. They also may not get enough detail about the value of what has changed. They also have to take into consideration if their are any outside factors that may prohibit the accuracy of the results. Employers should get feedback from all employees from various job titles and backgrounds to get different point of views about the changes.
My current company is incredibly reactive and does not presently have a standard for performing evaluations.
The Kirkpatrick Model is an excellent and simple tool for anyone to follow.
I really appreciated the detailed explanation on how to make this applicable within todays organizations. If more organizations followed through to step 4 they would be more successful in their project and development goals.
In regards to this question: Why do you think that organizations fail to measure the results of change initiatives accurately?
In my opinion, companies do not want to bear the extra expense that it takes to follow up and confirm that the training worked. It takes time and effort along with a well laid out plan to correctly measure results. From my experience as a trainer, companies want the training to occur. They are less concerned about what happens after the training is complete. The emphasis is on having the training and then hoping that it works. In addition, as a trainer, it can be challenging to measure results. For example, I do training on soft skills like leadership, communication and team work. There is no easy way to measure any of those areas. It takes observation and time to measure improvement. In addition, there are a variety of factors that can subsequently affect their retention of material. I always say – if you don’t use it, then you will lose it. So if a leaner is not given the opportunity to use what they learned, then they will lose it and this would negatively impact the ability to measure the change of initiatives accurately.
With Level 1 of the Kirkpatrick framework, how can we take into account learning styles when wanting to gauge reactions to the content. Thinking outside of pen and paper and quiz?
Based on experiences as a learner and trainer within organizations, I think they fall short of measuring initiatives due to a lack of interest or viewing it as an ineffective use of time and money. I worked for an organization that transitioned to a new CEO and with that, an onslaught of change in employee role structure and businesses processes occurred. There was no change strategy in place, nor any measurement of outcomes. The lack of interest resulted in confusion, distrust and dedicated employees seeking roles outside of the organization. By not investing in evaluating change, it ended up costing the organization money in the long run.
As someone who has had a myriad of jobs in my professional career it feels like leadership at the top level sets the tone, as it should be. But in my experience it isn’t. uncommon for someone to be promoted because they are good at their job not necessarily leading people. From there it is hard to implement change if you are not sure how to lead.
Additionally there is a certain amount of setting side of the ego when you decide to make change as you have to face that things are not working.
Why do you think that organizations fail to measure the results of change initiatives accurately? I think quanitfying the impact of change initiatives is a struggle because timing, participants, and other factors can all vary greatly once the initiative is live.
What strategies might organizations use to conduct a level 4 evaluation of a change initiative? Organizations and companys can compare pre-tests to post tests, evaluate work output, and review survey results.
Organizations fail to measure the results of change initiatives because they are interested in short-term effects. Leaders often forget that there must be some follow-through to measure if the change initiative met the desired performance for change to be successful.
What strategies might organizations use to conduct a level 4 evaluation of a change initiative?
If the change initiative ties to a metric, organizations can use KPIs to measure performance. Additionally, interviews can elicit employee feedback to gather data on the change and if they see results.
I think organizations fail to measure the results of change accurately because short-term success/results allows the box to be checked and show completion. Once this is done, many organizations move on to a different issue and don’t maintain a focus on the previous change.
Why do you think that organizations fail to measure the results of change initiatives accurately?
I believe the article stated it best, most times when results are being measured organizations look at the financial results over the individual worker. I believe the finances in this situation are the smoke, and the individual worker is the fire. For change to occur and a true measure of the results organizations must focus on the worker over all else.
I think organizations fail to measure the results of change initiatives accurately because they do not know how to properly measure the results of the change initiative. Additionally, I think organizations fail to measure the results accurately because they lose focus of the change initiative after they see a short-term reward of the initiative, rather than seeing it all the way through.
This is a great article and resource is such a wonderful way to utilize this model in applying these methods for Groups, Team or Individuals.
Here is also a great article as well, “Employing Kirkpatrick’s Evaluation Framework to Determine the Effectiveness of Health Information Management Courses and Programs”
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3070232/
I think organizations fail to measure the results of change initiatives because they don’t properly plan for evaluations. They may not of done a proper needs assessment that showed were the base line was, truly this is the first evaluation and sets up how the results will be measured. Then they need to incorporate evaluations into the design of the training and then follow up with evaluations at intervals that makes sense for that type of change. A general rule of thumb is the 30, 60 and 90 day marks. This gives them a nice set of comparative measurements to use to gauge the results of the effectiveness of the change initiative.
I think there is likely a lack of education around evaluation in any organization. I have been in T&D for ten years. Before starting my Master’s Program, I had never had an in-depth conversation about Kirkpatrick with any training professional inside or outside the organization. We don’t know what we don’t know… Based on preliminary discussions internally at my organization, I think that convincing stakeholders that the effort to measure is worthwhile is another challenge. There needs to be organizational buy-in at the level that measurement needs to occur, AND this must be discussed before any intervention is developed, training or otherwise.
Thanks for sharing your experience with us. It’s great that you are able to bring this important practice to your organization.
What strategies might organizations use to conduct a level 4 evaluation of a change initiative?
One of the strategies which can be used is a measurement of a ‘change curve’ by deploying the same survey at 3 and 9 months once the change was introduced. The survey will measure perceptions of the results of the change. For example, if a new org structure was introduced with a goal to streamline redundancies and implement efficiencies, the survey questions should target that measurement in terms of specific business process efficiencies. If one of anticipated change outcomes was business ownership of certain practices/processes, the survey questions should be designed to measure the difference in perception of this outcome at 3 and 9 months into this change. The 3- and 9-months surveys need to be compared to draw conclusions if the change resulted in anticipated outcomes. The challenge with this strategy is how well the survey questions are designed, the questions should be targeted towards measurable outcomes, vs. individual feelings and perceptions towards the change.
I think in large part organizations fail to measure change initiatives because their objectives of what they are trying to measure are likely poorly defined. If you want to measure change you also need to know what is the baseline. How are organizations measuring the baseline? Unlike learning, change can seem more subjective especially as it relates to people and their attitudes. A good amount of thought needs to be put into how to measure those before and after as well as clearly defining the goal trying to be reached. Process changes are much simpler to measure as long as that measurement is defined prior to starting.
My response to the second question:
Kirkpatrick’s Level 4 evaluation focuses on the results of a training or change initiative. To evaluate at this level, organizations can Measure key performance indicators (KPIs) before and after the initiative, Use surveys to gauge the organizational perception of change impact, Benchmark against industry standards or competitors, Calculate the initiative’s return on investment (ROI), Monitor long-term effects to ensure sustainability, Document case studies highlighting positive outcomes, Interview key stakeholders for insights, Commission external audits for an unbiased review, Employ data analytics for real-time monitoring, Organize focus groups for qualitative feedback, By combining various methods, organizations gain a comprehensive view of the initiative’s effectiveness and areas needing refinement.