Work Backwards & Define Results First: Measuring Informal Learning Strategically
by Tom Ford, MATD candidate
Informal learning is one of the greatest learning tools of the 21st century and also one of the biggest headaches for the modern trainer to evaluate effectively. Kirkpatrick’s 4 Levels of Evaluation (Reaction, Learning, Behavior and Results)[1] offers a starting point for tackling this problem but lacks a clear implementation framework. The problem is further exacerbated by inconsistent organizational policies around the use of one of the biggest informal learning platforms in existence today: social media. Therefore, how do you create a comprehensive evaluation framework to capture a wide variety of informal learning possibilities that exist in an organization while also promoting the adoption of an informal learning program? To begin, always start by working backward through the 4 levels:
- Create a clear, company-wide definition of what is and what is not informal learning that everyone is upheld to in the company in order to ensure that managers do not discredit or discourage employee time spent on informal learning. Creating this definition will also help with the development of organizationally relevant informal learning groups. As Mattox writes, these groups include: communities of practice, virtual knowledge sharing, performance support, and mentoring. [2] Understand what the end results or goals are and work with leadership to remove organizational barriers to achieving these goals before designing your program.
- Develop a qualitative feedback system with quantitative components for managers to evaluate their employees’ behavioral growth from informal learning. This could be as simple as sending out a survey with data showing the groups of informal learning employees have taken during a quarter that includes a numerical ranking system for managers to indicate the perceived impacts, or lack thereof, that specific informal learning groups have had on the performance of their direct reports. Both managers and employees should have a clear understanding of what the groups mean and how they are used in the evaluation process.
- Gather feedback from employees on what will motivate them to document their informal learning experiences so that they will be willing to A) briefly explain what they learned and B) briefly describe how they plan to practice their new skills. Motivation to do so could come in the form of a points system. For example, every time an employee documents an informal learning experience they will earn 100 points redeemable for purchases in an online store that sells merchandise, gift cards, and paid time off! As always, set the expectations with the employees at the beginning of the program.
- Establish a system where employees can document their reactions to different learnings and use the groups to classify their informal learning experiences. This system of record could be an LMS like Moneyball for Sales or it could be an HRIS like Workday.
When creating any evaluation program, it is essential to start with a strategic approach by defining the desired organizational results first. Once you know the expectations, the challenges of analyzing behavior, documenting learnings, promoting program adoption, and choosing the right system of record to become much easier to manage. Has your organization recently implemented an informal learning program? If so, how are you measuring the results and what challenges have you encountered?
[1] Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating Training Programs: The Four Levels. Oakland: Berrett-Koehler Publishers, Inc.
[2]Mattox, John R., II. (2012). Measuring the effectiveness of informal learning methodologies. T + D, 66(2), 48-53,8. Retrieved from http://ezproxy.roosevelt.edu:2048/login?url=http://search.proquest.com/docview/921536572?accountid=28518
RU Training @ Roosevelt University in Chicago 
Wow who knew you could learn or teach so much by going backwards. But this not the first method doing this I have seen. I took a Teaching science class years ago and the instructor had us do this.
Thanks for reading my post and for sharing your story! Sounds like it was a great class! 🙂
Great post, Tom! Beginning with the end results desired is the way to implement an evaluation framework.
Thanks for your kind words, Stuart!
Hi Tom,
I love this article. The concept of working backward is a great idea in evaluation process. You did a great job.
Nneka.
Thanks for reading, Nneka! 🙂
Hi Tom,
Thank you for a great article. I just recently learned about Kirkpatrick’s 4 Levels of Evaluation and it was interesting to see you perspective on it.
Hi Joanna,
Thank you for reading – I’m glad you enjoyed it! 🙂
Be well,
Tom
Hello Tom,
You did a great job on your blog! Congrats on making it to the website. 😊
Great description of the evaluation process.
Sonia
Thank you for your kind words and for reading, Sonia! 🙂
Hi Tom,
I very much appreciated specific and practical advice of your post – thank you! One question I have is how to strike a balance of putting enough structure around something that is supposed to be informal, and not “over-engineer” it to a degree when it will feel too formal and possibly even forced. Informal learning is supposed to be organic and almost effortless for the learners (or we are being told so…). Do you think there is any risk in creating feedback mechanisms that potentially may take the value away from the experience?
Thanks!
Olga
Hi Olga,
Thanks for reading my post! I think it’s always risky to apply structure to organic efforts because of exactly what you said: it could inadvertently discourage participation. However, I think emphasizing to your participants why structure for informal learning is important while having managers/leadership reinforce this messaging can mitigate the risks. Here are two examples that could help with a pitch to employees / end users:
1) Feedback over time should start to show trends around what types of informal learning actually impacts the organization. Eventually, this will allow the organization to recommend different types of informal learning based on an individuals’ job role and/or job aspirations. Without this feedback, the potential to spend (waste?) time on low quality, low impact informal learning increases.
2) Structure and definition allows for the organization to validate informal learning as an important part of an employees’ time; without structure, informal learning may be largely absent from certain departments within the organization because of the wide variety of “opinions” on what informal learning is, how much time should be spent on informal learning activities, etc.
Keeping the structure of the program simple is also important but this is also why it’s so important to define results first. Once you know the end goal(s), there might be 20 different informal learning components that you could incorporate but only 5 that are actually important to management. Start with those 5 and build/refine over time.
I hope this helps, Olga. Every organizations’ challenges will be different and unique to some degree but if you explain the “why” of the program to the point of absurdity and keep it simple enough that a grandfather or a mother could do it, you’ve got your “in.” 🙂
Best of luck,
Tom Ford
Hi Tom,
thank you so much for your reply. I really appreciate the ideas and talking points that can be used in a pitch for evaluation of informal learning.
Happy Holidays!
Olga
My pleasure, Olga! It’s always so nice to interact with you. Happy Holidays to you, too! 🙂
Even though there are many ADDIE-type processes out there for companies to use, many follow a forward-thinking approach. As the author Covey says, “begin with the end in mind.” I think it is really important to articulate the end result of any change. Does the company want to reduce staffing? Does the company want to outfit its factories with new equipment for competitive purposes? Once the end goal is identified, the procedures of needs analysis, design, development, etc. become much more focused. It may be discovered that the change is department-specific versus company-wide. This can also make clear some opportunities for informal learning to share with the internal stakeholders.
Thanks for this post. It is the marriage of TRDV and common sense.
Thanks for your kind words, Brian! Well said!
Hi, Thanks for posting this. At my previous company we attempted several times time to launch an informal learning program. I think the issue was that if people saw training assigned, they would be compelled to complete it, or it would seem that Managers would make the informal training required. We constantly had to tell people this is not the case. I will say that right before I left the company we did create an online learning library, and people did checkout books. I think that this will probably be the safest and be use for informal learning.
Hi Kay,
Thanks for reading my post! I’m curious: what were the expectations that were set with the employees around the goals of the informal learning program? I ask because we also have a similar issue at my current company with eLearning based sales training. The content was good and the feedback was solid but the person who originally owned the program oversold what the program was capable of, resulting in low user adoption because people thought they were sold a scam learning opportunity. I did try to win back managers and employees but by the time I entered, people were already over the original program and had moved on to newer, better structured learning options.
I recognize that many of us are not fortunate enough to land a job where training needs to be built from the ground up; many of us inherit disasters that we have to somehow “fix.” What did you do in your scenario to try and get ahead of the misconceptions around users needing to complete trainings as soon as they saw they were assigned? Did you encounter internal resistance with implementing changes?
Looking forward to your thoughts! Thanks for your time!
-Tom
Hi Tom,
I too am a Steven Covey follower. I have always begun any project that I have had the opportunity to lead and develop with the end up front. When “working backwards” it is easier to make sure that all the steps needed are accounted for and that within the Kirkpatrick model, if there are segways that are not addressed (as you indicated in your article) you can make the necessary adjustments almost in a “real time” environment. Thanks again for a great read and congratulations on your soon to be completion of the program at Roosevelt.
Sincerely,
Andrea Kehl
Hi Andrea,
Thanks for your kind words! You are spot on about how working backwards allows you to make adjustments in “real time.” I’ve certainly worked on projects where planning was lacking and it caused so much redundant work because we would constantly have to go back and adjust previous work based on new information, the unexpected, or just a lack of understanding around what the original expectations were.
Thanks for reading my post and be well,
Tom
I agree, defining results is very important. As I was reading this, I was taken back to my days as a sales rep where one of the most important steps of the process was to manage expectations. Being clear to the customer on what they should expect and anticipate created a much better experience for them as it eliminated surprises. I feel as though defining results is in a similar vein as when things are made clear, it allows everyone to be on the same page and disallows surprises and disappointments.
That’s exactly where a lot of my inspiration came from for my first post. I’ve worked in sales for years, especially on the customer success side. It’s a night and day difference when you’re working with a client where the expectations were set vs. a client who thinks they bought a penthouse but are really only renting a remodeled apartment. Anything that can help people get to and stay on the same page is always a win for everyone! 😉
Be well,
Tom
I believe your concept is brilliant and well argued within your blog. Sometimes we do have to start with the results to find a true assessment of the needs within the performance gaps. As a student I do obtain more information on social media than actually reading textbooks nowadays. You definitely put a new perspective into my mind about the evaluation process. Once again, excellent job on your blog!
Hi Steph!
Thank you for reading! I also get a lot of my information from various online platforms today although you still can’t beat a good book! 🙂
Thanks for reading and for your comments,
Tom
This was very helpful. I think by setting a clearly defined company level of what is and what is not informal will help get rid of any discrepancies between people that have learned either way and not the other. It’s important that what is taught is cohesive and that everyone gets the same information in the most qualitative way.
Thanks for your kind words! I also agree that it’s important to discuss what should be evaluated as a result of the program, even if it creates some healthy conflict among the stakeholders.
Thanks for writing this post, it was very informative Tom! How common is a system like this used in the industry? How likely are employees to take the time to document experiences? Are incentives usually involved? Thanks.
Niké
Hi Niké!
My pleasure – thanks for reading it! Workday is growing but a lot of people still use systems like Oracle, SuccessFactors/SAP, and a host of other tools. In terms of LMS’, Moneyball is a smaller tool but has been rolled out successfully to sales organizations with over 1,500 employees. I see Cornerstone being purchased by lots of clients – it seems though that every week, a new LMS springs up with more or less the same features. What really seems to make one system stand out from another is the level of customization and integration available “out of the box.”
A benefit of having learning funcations incorporated in to an HRIS (Human Resources Information System) is it creates one less system for employees to learn, thus making the documentation hurdle easier to overcome. If they’re already in a system filling out time sheets and requesting PTO, a section for documenting “learning experiences” is easier to incorporate in to daily employee responsibilities. I do find that incentives help people engage with new programs but just throwing incentives at people rarely works. The challenge becomes identifying incentives that the people at your organization actually want to work for as well as giving them a range of limited options to choose from. Surveys help figure the basics out but even then, any incentive driven program should be developed in iterations so employee input can help shape how the program evolves and stays fresh.
Thanks again for reading and hope you are well,
Tom Ford 🙂
Thanks so much Tom. That was a really helpful reply. I always learn something new from you! Take care.
Thanks Niké! The feeling is mutual! 🙂
Hi Tom,
Wonderful work on your post. I’m currently taking Evaluation Research TRDV 434, and it ties in perfectly. Wiggins and McTighe in Understanding By Design also began with the end in mind. Your idea of seeking employee input on what would motivate them to document their informal learning is a great approach. Thanks for sharing.
Thank you for reading, Stuart!
I just finished that class. I’ve had a few people reference Understanding by Design and just added it to my Amazon wish list. I think employee input is key to the success of any program. You may not be able to accommodate everyone’s needs but you can at least often spot general trends and preferences to be mindful about.
In solidarity,
Tom Ford
This blog post came at a good time for me, as I have just finished the Evaluation Research class, which was one of the more challenging classes I have taken in this program. I felt like I really needed to see/work with Kirkpatrick’s concepts in order to keep that information fresh and relevant in my mind. This post helped me think of it again and in a relevant way. Thanks for posting!
Patty
Thank you for your kind words Patty! I’m glad the post helped refresh the information for you in a practical context! 🙂
Tom,
Love the article! I so agree! Starting with a vision of what the ideal end and accomplishments look like is the best way to begin. It allows designers to have a measuring stick so-to-speak to align all components of the course development against. Sometimes that measuring stick is malleable also…as we go through the development, we can sometimes improve the vision of the final product. This also makes it easier to develop the training evaluation plan to ensure the outcomes are what were planned.
I love that you also stated having a corporate strategy in mind for what is considered informal and informal learning, and working with management to define what the desired results are from a more global perspective. So true!
Thanks!
Melissa
Hi Melissa!
Thanks for your kind words and for reading. I think working closely with management to do a solid risk analysis and develop a cohesive implementation plan is key before too much design work goes in to developing the program. It’s tempting to jump ahead and develop a program based on agile principles but since training programs can largely be influenced by politics, having leadership involved in the beginning can really impact how any program is received as well as adopted by an organization.
Hope you are well,
Tom Ford 🙂
Tom you did great giving an honest evaluation of Kilpatrick’s levels of evaluation.
Thank you Joi!
Tom this is fantastic. I, too, am a big fan of backwards planning (I use Understanding By Design for my classroom work almost daily). And your link to Workday may well me the tool I have been missing my work as I manage several budgets/grants/work plans. Thanks!
Thanks for your kind words, Maureen! I hope Workday is what you’ve been looking for!
I love the concept of working backward Tom. You did a great job of putting Kirkpatrick into a well thought out plan. Thank you for writing this.
Thanks so much for reading, Marady!
I believe it is Stephen Covey that stated, start with the end in mind. I think that principle is applicable here as well.
Agreed Michael! Thanks for reading my post!