Evaluating Change: Moving from Assumptions to Evidence
Despite decades of investment in change management, many organizations still rely on intuition or anecdotal feedback to assess success. Even when they use quantitative measures, the focus tends to remain narrow, often limited to metrics like revenue. Organizations need a more comprehensive approach to evaluating change initiatives. A practical option is Kirkpatrick’s Four-Level Evaluation Model, a flexible and widely respected framework suitable for many contexts.
But how can we thoughtfully adapt a model initially created for training evaluation to the broader, more complex landscape of organizational change?
Originally developed to assess training effectiveness, Kirkpatrick’s Model evaluates outcomes across four dimensions:
1. Reaction: how employees respond to the change, including whether they find it relevant, understandable, and worth supporting,
2. Learning: measures what knowledge they developed during the process, such as new systems, policies, or expectations,
3. Behavior: How their on-the-job actions changed, and
4. Results: Focuses on broader organizational outcomes, such as improved performance, employee retention, or service quality.
Consider a practical example from a recent project:
We applied Kirkpatrick’s Four Levels to evaluate and measure outcomes in a change initiative. Our client, a well-known hotel chain, had acquired an independent hotel and retained the existing staff. The goal was to integrate the property into the company’s culture, systems, and service model without disrupting operations or alienating employees. To support this transition, we introduced strategies to increase employee involvement, implemented targeted learning interventions to build understanding of the parent company’s culture and policies, created pathways for continued learning and advancement, and established a reward system to reinforce performance improvements. Next, we applied Kirkpatrick’s Model to evaluate the success of our intiaitives.
- At Level 1, we conducted surveys and focus groups to understand how staff responded to the change initiative. We looked at indicators such as job satisfaction, intent to leave, and the quality of relationships with coworkers and supervisors. This helped surface concerns early and guided our communication strategy.
- At Level 2, we examined the new knowledge developed during the initiative, including new expectations, procedures, and aspects of the company’s culture.
- At Level 3, we assessed the sustainability of the change. We tracked whether behaviors introduced during the integration, such as adherence to more rigorous service and performance standards and cross-team collaboration, continued over time and became embedded in daily operations.
- At Level 4, we evaluated outcomes against the original goals of the initiative. Metrics included reduced staff turnover, improved customer satisfaction scores, and progress toward fully aligning the acquired property with the company’s broader operational and cultural standards.
Kirkpatrick’s model offers a comprehensive approach to the measurement of change process outcomes. It provides a structure that ensures that change outcomes are evaluated from a multifaceted perspective. Rather than relying only on financial results, applying the four levels leads us to consider how change initiatives affect the individual worker, the learning organization, whether they are sustainable, and ultimately whether they produce the desired outcomes.
Questions for Discussion:
- Why do you think that organizations fail to comprehensively measure the results of change initiatives?
- What strategies might organizations use to conduct a level 4 evaluation of a change initiative?
Organizations sometimes fail to measure the results of change initiatives accurately by failing to fully gather context at the outset–often in Level 1. For example, I recently observed an organization survey its staff on changes to work from home policies due to COVID-19. The survey results were interpreted by leadership as “most staff feel it is more difficult to work from home than in the office”. However the survey failed to account for other difficulties (i.e. childcare) that have made working from home more difficult in the past year. So, management’s reading of the results of the survey does not fully reflect the view of the employees’ who have been working from home, which in turn feeds distrust of management.
Excellent point about the need to collect valid data. In this situation, it may have helped to begin with focus groups or interviews to gather information about the issues at hand.
Agreed Kathleen. Absent robust data it is very difficult to solve for the problem(s). Interviews would have been ideal.
I suspect It takes precision, time and in-depth knowledge to ensure the correct method is selected for specific situations. There are such a selection of methods to choose from and choosing the wrong one may be the death knoll for any organization’s change process.
For this area to grow even further, it is important that the science is highlighted. Companies will not continue their dollar spends if there is no emphasis placed on the science component.
Replying to Kevin’s comment here: agreed. The expenses are not to be underestimated. Investment into solutions should be backed up by data.
I think organizations fail to measure the results of change initiatives accurately because there is no clear and compelling information for change; people do not understand and face anxiety and resistance. Information about change should be clear so employees can understand the implications and can feel engaged during the process of change. Lack of trust and empowerment are factors that contribute for organizations to measure results.
Lety, intriguing thought re: “clear and compelling information for change”. I agree, the nature of most organizational structures do not communicate a full case for change to all employees. Do you think it would help the change process if that communication were more robust? What implications might come along with that, since so many factors are unknown–even to leadership–at the beginning of the change process?
I agree Lety it is difficult to think about evaluation when the initial objectives are not clear.
I agree Lety, I think Trust is a key word for many organizational changes. When people learn there is a change in management or a merge is happening, people start to worry about their jobs. If an employee has the trust towards their department from the beginning, I don’t see that there would be any issue in adjusting to the changes. An organization should prepare for the team that the change will effect. They should get to know them and understand their capabilities and productivity before making the change. This will assist later when comparing the before and after.
– MAttia
I think organizations fail to measure the results of change because the change is unclear. When people don’t understand why change is necessary, anxiety, cynicism and resistance inevitably build.
Most organization fail to measure change solely because they are very comfortable in the way they do things. In order to be successful at the level 4 stage the organization must become uncomfortable in order to meet the goals at level 4.
I believe companies fail to measure the results of change accurately because they may not ask enough people about their results. They also may not get enough detail about the value of what has changed. They also have to take into consideration if their are any outside factors that may prohibit the accuracy of the results. Employers should get feedback from all employees from various job titles and backgrounds to get different point of views about the changes.
My current company is incredibly reactive and does not presently have a standard for performing evaluations.
The Kirkpatrick Model is an excellent and simple tool for anyone to follow.
I really appreciated the detailed explanation on how to make this applicable within todays organizations. If more organizations followed through to step 4 they would be more successful in their project and development goals.
In regards to this question: Why do you think that organizations fail to measure the results of change initiatives accurately?
In my opinion, companies do not want to bear the extra expense that it takes to follow up and confirm that the training worked. It takes time and effort along with a well laid out plan to correctly measure results. From my experience as a trainer, companies want the training to occur. They are less concerned about what happens after the training is complete. The emphasis is on having the training and then hoping that it works. In addition, as a trainer, it can be challenging to measure results. For example, I do training on soft skills like leadership, communication and team work. There is no easy way to measure any of those areas. It takes observation and time to measure improvement. In addition, there are a variety of factors that can subsequently affect their retention of material. I always say – if you don’t use it, then you will lose it. So if a leaner is not given the opportunity to use what they learned, then they will lose it and this would negatively impact the ability to measure the change of initiatives accurately.
With Level 1 of the Kirkpatrick framework, how can we take into account learning styles when wanting to gauge reactions to the content. Thinking outside of pen and paper and quiz?
Based on experiences as a learner and trainer within organizations, I think they fall short of measuring initiatives due to a lack of interest or viewing it as an ineffective use of time and money. I worked for an organization that transitioned to a new CEO and with that, an onslaught of change in employee role structure and businesses processes occurred. There was no change strategy in place, nor any measurement of outcomes. The lack of interest resulted in confusion, distrust and dedicated employees seeking roles outside of the organization. By not investing in evaluating change, it ended up costing the organization money in the long run.
As someone who has had a myriad of jobs in my professional career it feels like leadership at the top level sets the tone, as it should be. But in my experience it isn’t. uncommon for someone to be promoted because they are good at their job not necessarily leading people. From there it is hard to implement change if you are not sure how to lead.
Additionally there is a certain amount of setting side of the ego when you decide to make change as you have to face that things are not working.
Why do you think that organizations fail to measure the results of change initiatives accurately? I think quanitfying the impact of change initiatives is a struggle because timing, participants, and other factors can all vary greatly once the initiative is live.
What strategies might organizations use to conduct a level 4 evaluation of a change initiative? Organizations and companys can compare pre-tests to post tests, evaluate work output, and review survey results.
Organizations fail to measure the results of change initiatives because they are interested in short-term effects. Leaders often forget that there must be some follow-through to measure if the change initiative met the desired performance for change to be successful.
What strategies might organizations use to conduct a level 4 evaluation of a change initiative?
If the change initiative ties to a metric, organizations can use KPIs to measure performance. Additionally, interviews can elicit employee feedback to gather data on the change and if they see results.
I think organizations fail to measure the results of change accurately because short-term success/results allows the box to be checked and show completion. Once this is done, many organizations move on to a different issue and don’t maintain a focus on the previous change.
Why do you think that organizations fail to measure the results of change initiatives accurately?
I believe the article stated it best, most times when results are being measured organizations look at the financial results over the individual worker. I believe the finances in this situation are the smoke, and the individual worker is the fire. For change to occur and a true measure of the results organizations must focus on the worker over all else.
I think organizations fail to measure the results of change initiatives accurately because they do not know how to properly measure the results of the change initiative. Additionally, I think organizations fail to measure the results accurately because they lose focus of the change initiative after they see a short-term reward of the initiative, rather than seeing it all the way through.
This is a great article and resource is such a wonderful way to utilize this model in applying these methods for Groups, Team or Individuals.
Here is also a great article as well, “Employing Kirkpatrick’s Evaluation Framework to Determine the Effectiveness of Health Information Management Courses and Programs”
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3070232/
I think organizations fail to measure the results of change initiatives because they don’t properly plan for evaluations. They may not of done a proper needs assessment that showed were the base line was, truly this is the first evaluation and sets up how the results will be measured. Then they need to incorporate evaluations into the design of the training and then follow up with evaluations at intervals that makes sense for that type of change. A general rule of thumb is the 30, 60 and 90 day marks. This gives them a nice set of comparative measurements to use to gauge the results of the effectiveness of the change initiative.
I think there is likely a lack of education around evaluation in any organization. I have been in T&D for ten years. Before starting my Master’s Program, I had never had an in-depth conversation about Kirkpatrick with any training professional inside or outside the organization. We don’t know what we don’t know… Based on preliminary discussions internally at my organization, I think that convincing stakeholders that the effort to measure is worthwhile is another challenge. There needs to be organizational buy-in at the level that measurement needs to occur, AND this must be discussed before any intervention is developed, training or otherwise.
Thanks for sharing your experience with us. It’s great that you are able to bring this important practice to your organization.
What strategies might organizations use to conduct a level 4 evaluation of a change initiative?
One of the strategies which can be used is a measurement of a ‘change curve’ by deploying the same survey at 3 and 9 months once the change was introduced. The survey will measure perceptions of the results of the change. For example, if a new org structure was introduced with a goal to streamline redundancies and implement efficiencies, the survey questions should target that measurement in terms of specific business process efficiencies. If one of anticipated change outcomes was business ownership of certain practices/processes, the survey questions should be designed to measure the difference in perception of this outcome at 3 and 9 months into this change. The 3- and 9-months surveys need to be compared to draw conclusions if the change resulted in anticipated outcomes. The challenge with this strategy is how well the survey questions are designed, the questions should be targeted towards measurable outcomes, vs. individual feelings and perceptions towards the change.
I think in large part organizations fail to measure change initiatives because their objectives of what they are trying to measure are likely poorly defined. If you want to measure change you also need to know what is the baseline. How are organizations measuring the baseline? Unlike learning, change can seem more subjective especially as it relates to people and their attitudes. A good amount of thought needs to be put into how to measure those before and after as well as clearly defining the goal trying to be reached. Process changes are much simpler to measure as long as that measurement is defined prior to starting.
My response to the second question:
Kirkpatrick’s Level 4 evaluation focuses on the results of a training or change initiative. To evaluate at this level, organizations can Measure key performance indicators (KPIs) before and after the initiative, Use surveys to gauge the organizational perception of change impact, Benchmark against industry standards or competitors, Calculate the initiative’s return on investment (ROI), Monitor long-term effects to ensure sustainability, Document case studies highlighting positive outcomes, Interview key stakeholders for insights, Commission external audits for an unbiased review, Employ data analytics for real-time monitoring, Organize focus groups for qualitative feedback, By combining various methods, organizations gain a comprehensive view of the initiative’s effectiveness and areas needing refinement.
Hi,
Great post!
In my opinion, organizations often fail to measure change initiatives accurately because they focus too much on short-term financial results and overlook harder-to-measure aspects like employee engagement or cultural shifts. Without clear objectives from the start, it becomes difficult to track success. Resistance to change and lack of leadership support can also weaken efforts to properly assess the results.
To conduct a level 4 evaluation, I believe organizations should set specific objectives that go beyond financials, including employee satisfaction and productivity. Using both qualitative data, like surveys, and quantitative metrics ensures a more complete picture of success. Ongoing evaluations are also key to ensuring the change is sustainable over time.