Training Evaluation Series Part 3: Measuring Intangible Benefits
When considering the word “intangible” terms like ethereal, ghostly, spectral, unearthly and supernatural come to mind. Not exactly how we think of the efficient and grounded process of calculating the return on investment (ROI) of training.
In the Phillips ROI Methodology (see diagram below), intangible benefits are those that not readily converted to a monetary value; outcomes that are difficult to value. Examples of intangible benefits include:
- Job satisfaction
- Self-efficacy or confidence
- Lower Stress
Although the ROI calculation usually limits the inclusion of intangible benefits, going for only “hard” numbers like profit, turnover, and errors, they represent many the most difficult challenges faced by organizations today. Can we accurately calculate the value of training without considering the critical contributions of intangibles like improved satisfaction, commitment, and communication?
The key to making intangibles more tangible lies in how we measure them. Since most trainers have neither the time or the skill-set to embark on an extensive validation process for each “construct” they measure, the most timely and accurate way to measure intangibles is through the use of validated tests. When developing tests, researchers carefully analyze each question ensuring that it adds unique value and that it measures what it is supposed to measure (validity) accurately over time (reliability).
Locating valid and reliable tests requires more than Googling terms. The simplest way to begin your search is to use an online database that compiles published tests. Here are instructions that will get you started:
First, identify the construct you want to measure; what intangible benefit would you expect your training or intervention to reap?
Once you have a construct or outcome in mind, begin your search by visiting the “Mental Measurements Yearbook” (instructions on how to access this database at the end of the article) available free and fully online through the RU Library.
Enter search terms and locate instruments that both measure the outcomes you have identified and do so in the correct population group (adults, workplace).
Evaluate the instruments by considering information about reliability and validity.
For example, f you were to conduct a training intervention around stress management and wanted to measure results, you might enter the terms stress and burnout and locate the Maslach Burnout Inventory (MBI) which you may purchase through Mind Garden. Or if you were delivering training or an intervention focused on teams, you might use the OTCI to measure orientation toward team culture. While the MBI is on of the most researched measures of stress and burnout the OTCI, on the other hand, demonstrates reliability in adult work-based populations but is lacking in proven validity, so use this instrument with caution.
In this brief article, you’ve learned about intangible benefits and how to locate and select them. Stay tuned for the next article in this series to find out more about measuring intangibles with test resources in research publications.
Can you name additional intangible benefits that might arise from training and should be measured?
Accessing the Mental Measurements Yearbook:
Log in to the RU Library. See instructions on how to do this from off campus
Enter the search term “mental measurements yearbook” and click on the link to the database
Enter appropriate search terms and locate information about tests. Note the appropriate age and setting (adult, workplace versus children, educational setting) and information about reliability and validity.