Training Evaluation Series Part 4: Measurement Tools from Scholarly Publications
Last week, in the post, “Measuring Intangible Benefits,” you were introduced to strategies to locate tests that allow you to quantify the intangible outcomes of training by using the Mental Measurements Yearbook.
As you will find, the “Yearbook” reviews commercially available tests, but there are many other tests in published research studies.
The best database available to locate tests is PsychTests, which is relatively new and unfortunately not yet available in the RU Library system. But there is a workaround that allows you to locate tests without this resource. It’s a bit like prospecting for gold–you have to sift through a great deal of rubble before hitting pay dirt–but it’s worth the time and trouble to locate valid and reliable instruments to measure the intangible benefits of training.
Follow the instructions in last week’s article to log into the RU library system. Then, as before, go to the Database section. This time, choose Google Scholar, By first logging into and going through the RU library system you will have access to full-text articles in Google Scholar beyond those available to the general public.
In the search box, enter the construct you want to measure and add additional search terms like “instrument,” or “measure,” or “validation,” or “scale” and hit search. For example, say you are delivering a program or intervention designed to increase service to internal customers (or coworkers) within an organization, and you want to measure changes in the attitude of your trainees. One “intangible” benefit or outcome might be increased organizational citizenship, which relates to willingness to go above and beyond job expectations. When I enter the terms organizational citizenship and instrument, I see a lengthy list of articles related to this topic. In the list, I select the article titled “Organizational citizenship behavior: Construct redefinition, measurement, and validation” because it has the terms measurement and validation, which should yield actual test questions. I click on the PDF link to the right and see the entire article. As I scroll through the article, I hit pay-dirt when I come to Table 3 (Page 781) and find a series of questions measuring various aspects of citizenship. I probably wouldn’t use all the questions to measure citizenship but will choose those most appropriate to my audience and purpose. Rarely am I so lucky to find a rich resource like this one so quickly, but often have to enter a variety of search terms and sift through the results.
As you probably realize, a critical step in this entire process is the identification of the intangible benefits that you want to measure that are logical outcomes of your training program. Here are some possibilities:
Leader-Member Exchange: a brief measure to assess relationships between workers and their supervisors after leadership or supervisory skills training.
Job Satisfaction: a measure to assess overall satisfaction with the workplace after a change initiative, reorganization, or merger.
Workplace Thriving: an attitude of feeling energized and alive (vitality) and a sense of continually improving and getting better at what one does (learning) that might be used following a large-scale training initiative or a lengthy new hire training program.
Take the challenge. Identify a training program that you might deliver in your current or past organization, come up with one or more possible intangible benefits, and use Google Scholar (by way of the RU Library) to locate potential instruments that you might use to measure your intangible. Most importantly, share your findings with us in the comment box.
RU Training @ Roosevelt University in Chicago 
Another great resource to measure intangibles! In my former role, I facilitated site director trainings for our after school site directors. It was a training designed to help employees in those roles feel equipped to carry out the day-to-day operational duties within their respective after school sites. In short, it developed KSAs. I did my search on the intangible benefit of employee empowerment. I was always curious on how our trainings helped our site directors feel empowered. I found a useful write-up by Bob Hayes on, “How to Measure Empowerment”. The author goes in-depth on how to construct and develop an EEQ (Employee Empowerment Questionnaire). There was an ample example that provided questions to ask tailored to empowerment (attitude). The majority of these questions could transfer to an EEQ for the site directors. The author also gave examples on how to quantify empowerment.
I provided the link here in case anyone was curious. I will definitely be using the resource in the prior posting and in this one. It takes some of the guesswork out of developing questionnaires for intangible benefits.
Click to access How-to-Measure-Empowerment.pdf
This article was extremely helpful. It is good to know the tricks of the trade, in terms of finding information using databases like google scholar. I was interested in finding instruments that measured job satisfaction, so I plugged in some very general search terms in google scholar, including measurement, job satisfaction, and instrument. I also limited my search to articles published since 2018. Even this, though, yielded far too many articles to sort through—16,900 to be exact! Even so, I found one very quickly that provided an instrument created after an extensive view of existing Job Satisfaction Surveys. Published in 2019 in the International Journal of Business and Economic Sciences Applied Research, it offers a 38-item survey designed to measure job satisfaction in relation to organizational commitment. The authors found their survey to meet standards of both validity and reliability. I think I would need to do more research before settling on their questionnaire, but I enjoyed the process of searching for the article and seeing the different possibilities. Here is the article in case anyone is interested:
Dalkrani, Maria and Efstathios Dimitriadis (2019). “The Effect of Job Satisfaction on Employee Commitment.” International Journal of Business and Economic Sciences Applied Research 11, no. 3, pp. 16-23.
Dare to be Aware
Truth be told, I am a stranger in performing a training evaluation in the manner in the real world. However, I am learning about training evaluation in my Master’s program in Training and Development, which increased my knowledge and confidence level in conducting it. Allow me to share a snippet of what I am learning in my Training Evaluation class with websites and a textbook to reference.
One model I have come to learn about is Kirkpatrick’s 4 Levels of Evaluation, which consists of 4 levels. According to Evaluating Training Programs (3rd Edition), The Four Levels by Donald L. Kirkpatrick and James D. Kirkpatrick on p. 21 states, “The four levels represent a sequence of ways to evaluate program. Each level is important and has an impact on the next level.” The below information is provided by the following web link: https://www.watershedlrs.com/blog/kirkpatrick-learning-evaluation-model
1. Reaction. Did participants enjoy the training?
2. Learning. Did they pass the assessment?
3. Behavior. Do they work better?
4. Results. Did business metrics improve?
Another model I come to learn is the Phillip’s ROI Process Model. This model is considered an extension of the Kirkpatrick’s Model. This model focuses on ROI (Return On Investment) and is without a doubt, in my opinion, a comprehensive one. It involves 6 levels (Kirkpatrick’s levels involved) and phases. The below information is provided by
https://www.ginaabudi.com/articles/using-roi-to-measure-the-results-of-business-process-improvement-initiatives/.
1. Reaction to a new process initiative introduced within the organization
a. Level 1 evaluation: Do employees feel positive about the new process and their involvement in rolling out the new process?
2. Confidence in utilizing the new process
a. Level 2 evaluation: Do employees feel confident in their training on the new process and that they have
acquired the necessary skills to implement the new process?
3. Proof that employees can apply the new process.
a. Level 3 evaluation: Have employees been utilizing the new process in their roles?
4. Business impact of the new process.
a. Level 4 evaluation: What impact does the new process have on the business?
5. Return on Investment (ROI)
a. What is the ROI of the new process?
6. Intangible benefits
a. What other benefits (not measured in a monetary value) were gained, e.g.:
i. More confidence/less stress among employees who are using the new process
ii. Increased job satisfaction among employees
The above evaluation models are considered best practices when gathering as much detailed data as possible to show the client and stakeholders the worth of having a training program. Learning about these evaluation models and their processes allows me to communicate on a higher level with the funders and clients about the effects of a training program.
1. Have anyone of you experience any of these levels during an evaluation? If so, what are your thoughts?
2. Which evaluation model do you prefer and why?
3. What types of data intangible or tangible will bring satisfaction to the CEO of an organization and why?
By Sixto Lopez Jr.
Training Evaluation series part 4: measurement tools from scholarly publications, I feel this way because sometimes even with my busy life including work, school, my son and my family it can be hard at times to balance them all, trying to figure out each one like a scale 25 to one 40 here and 50 here, and whatever is left over than i may try to spare to share myself with others. this can be so stressed yet, I still try to hold myself up for the most part with everyone especially my son which is three.
Thank you for sharing your “gold nuggets” with us! I’m glad that you were able to locate useful information using the databases.
Kathy
Following the prompts, under RU Library, I logged into Databases and searched Mental Measurements Yearbook. In the search field I entered Intangible Benefits and a list of research topics posted for me to review.
I chose Organization and Team Culture Indicator which led me to the article and evaluation study Review of the Organizational and Team Culture by Stephen B Johnson. This was not only a fascinating read but the basis for my future doctoral studies. It addresses many several theorists and theories, one of which is Emotional Intelligence in the Workplace.
You can find the entire study on http://www.alliedacademies.org/articles;jocccvol19no12014.pdf.
The evaluation is based on organizational politics influence on the organization’s climate and intangible benefits in the workplace. Several areas of focus including political behavior, which is the lack of political action of the individual and potential conflict that arises in the workplace and how staff avoids such conflict. It also focused on payments and promotion which is how the organization rewards political maneuvering; meaning human resources tend to reward those individuals who match certain behaviors and influence and then penalize those who fail to demonstrate the same type of behavior or conduct. Last it focuses on expressions of employees feelings and their personal contributions to the organization.
Here are some highlights to the type of questions used during the evaluation categories on intangible benefits:
General Political Behavior
1. People in this organization achieve promotion by overcoming those who challenge them.
2. Here has always been an influential group in this department that controls everything.
Political Behavior of Permanency
1. Employees are encouraged to speak out even when they are critical of well-established ideas.
2. There is no place for yes-men around here; good ideas are welcomed even if it means disagreeing with superiors.
3. Agreeing with powerful others is the best alternative in this organization.
4. Sometimes it is easier to remain quiet than to fight the system.
Payments and Promotions
1. Since I have worked in this department, I have never seen the pay and promotion policies applied in a bias manner.
2. I can’t remember when a person received a pay increase or promotion that was inconsistent with the published policies.
3. The stated pay and promotion policies have nothing to do with how pay raises and promotions are determined.
Expression of Own Feelings and Personal Contributions
1. My boss is flexible in relation to the fulfillment of my goals.
2. My boss supports my ideas and how I do things.
3. My boss gives me the authority to do things as I think best.
I encourage everyone to read this entire study and hope you found it to be as fascinating and interesting as I have during my readings.
I followed the prompts and steps above researching the intangible benefit of work-life balance. Albeit, this is a bit of an umbrella term further complicating the task of measuring it, but I did find two surveys that seem to do a decent job of it.
International Social Survey Programme (ISSP)
European Social Survey (ESS)
There is a great deal of overlap between the two. Samples of questions from both surveys can be found below:
I have come home from work too tired to do the chores which need to be done?
It has been difficult for me to fulfill my family responsibility because of the amount of time I spend on my job?
I have arrived at work too tired to function well because of the household work I had done?
I have found it difficult to concentrate at work because of my family responsibilities?
(A) Using this card, how often do you…
1. … keep worrying about work problems when you are not working? (‘worrying’)
2. … feel too tired after work to enjoy the things you would like to do at home? (‘too tired’)
3. … find that your job prevents you from giving the time you want to your partner or family? (‘prevents’)
(B) How often do you …
4. … find that your partner or family gets fed up with the pressure of your job? (‘fed up’)
5. … find it difficult to concentrate on work because of your family responsibilities? (‘concentrate’)
While I may not use all the questions in each, I might create a condensed version of my own that is an amalgamation of both surveys.
The link to this is below though you may not be able to access it without logging in:
https://link.springer.com/article/10.1007/s11205-008-9297-5