Training Evaluation Series Part 4: Measurement Tools from Scholarly Publications

Last week, in the post, “Measuring Intangible Benefits“, you were introduced to strategies to locate tests that allow you to quantify the intangible outcomes of training by using the Mental Measurements Yearbook.

As you will find, the “Yearbook” reviews commercially available tests, but there are many other tests in published research studies.

The best database available to locate tests is PsychTests, which is relatively new and unfortunately not yet available in the RU Library system. But there is a work around that allows you to locate tests without this resource. It’s a bit like prospecting for gold–you have to sift through a great deal of rubble before hitting pay dirt–but it’s worth the time and trouble to locate valid and reliable instruments to measure the intangible benefits of training.

Follow the instructions in last week’s article to log into the RU library system. Then, as before, go the Database section. This time, choose Google Scholar, By first logging into and going through the RU library system you will have access to full-text articles in Google Scholar beyond those available to the general public.

In the search box, enter the construct you want to measure and add additional search terms like “instrument,” or “measure,” or “validation,” or “scale” and hit search. For example, say you are delivering a program or intervention designed to increase service to internal customers (or coworkers) within an organization, and you want to measure changes in the attitude of your trainees. One “intangible” benefit or outcome might be increased organizational citizenship, which relates to willingness to go above and beyond job expectations. When I enter the terms organizational citizenship and instrument, I see a lengthy list of articles related to this topic. In the list, I select the article titled “Organizational citizenship behavior: Construct redefinition, measurement, and validation” because it has the terms measurement and validation, which should yield actual test questions. I click on the PDF link to the right and see the entire article. As I scroll through the article, I hit pay-dirt when I come to Table 3 (Page 781) and find a series of questions measuring various aspects of citizenship. I probably wouldn’t use all the questions to measure citizenship but will choose those most appropriate to my audience and purpose. Rarely am I so lucky to find a rich resource like this one so quickly, but often have to enter a variety of search terms and sift through the results.

As you probably realize, a critical step in this entire process is the identification of the intangible benefits that you want to measure that are logical outcomes of your training program. Here are some possibilities:

Leader-Member Exchange: a brief measure to assess relationships between workers and their supervisors after leadership or supervisory skills training.

Job Satisfaction: a measure to assess overall satisfaction with the workplace after a change initiative, reorganization, or merger.

Workplace Thriving: an attitude of feeling energized and alive (vitality) and a sense of continually improving and getting better at what one does (learning) that might be used following a large-scale training initiative or a lengthy new hire training program.

Take the challenge. Identify a training program that you might deliver in your current or past organization, come up with one or more possible intangible benefits, and use Google Scholar (by way of the RU Library) to locate potential instruments that you might use to measure your intangible. Most importantly, share your findings with us in the comment box.

Advertisements

3 comments

  • Thank you for sharing your “gold nuggets” with us! I’m glad that you were able to locate useful information using the databases.

    Kathy

  • Following the prompts, under RU Library, I logged into Databases and searched Mental Measurements Yearbook. In the search field I entered Intangible Benefits and a list of research topics posted for me to review.

    I chose Organization and Team Culture Indicator which led me to the article and evaluation study Review of the Organizational and Team Culture by Stephen B Johnson. This was not only a fascinating read but the basis for my future doctoral studies. It addresses many several theorists and theories, one of which is Emotional Intelligence in the Workplace.

    You can find the entire study on http://www.alliedacademies.org/articles;jocccvol19no12014.pdf.

    The evaluation is based on organizational politics influence on the organization’s climate and intangible benefits in the workplace. Several areas of focus including political behavior, which is the lack of political action of the individual and potential conflict that arises in the workplace and how staff avoids such conflict. It also focused on payments and promotion which is how the organization rewards political maneuvering; meaning human resources tend to reward those individuals who match certain behaviors and influence and then penalize those who fail to demonstrate the same type of behavior or conduct. Last it focuses on expressions of employees feelings and their personal contributions to the organization.

    Here are some highlights to the type of questions used during the evaluation categories on intangible benefits:

    General Political Behavior
    1. People in this organization achieve promotion by overcoming those who challenge them.
    2. Here has always been an influential group in this department that controls everything.

    Political Behavior of Permanency
    1. Employees are encouraged to speak out even when they are critical of well-established ideas.
    2. There is no place for yes-men around here; good ideas are welcomed even if it means disagreeing with superiors.
    3. Agreeing with powerful others is the best alternative in this organization.
    4. Sometimes it is easier to remain quiet than to fight the system.

    Payments and Promotions
    1. Since I have worked in this department, I have never seen the pay and promotion policies applied in a bias manner.
    2. I can’t remember when a person received a pay increase or promotion that was inconsistent with the published policies.
    3. The stated pay and promotion policies have nothing to do with how pay raises and promotions are determined.

    Expression of Own Feelings and Personal Contributions
    1. My boss is flexible in relation to the fulfillment of my goals.
    2. My boss supports my ideas and how I do things.
    3. My boss gives me the authority to do things as I think best.

    I encourage everyone to read this entire study and hope you found it to be as fascinating and interesting as I have during my readings.

  • I followed the prompts and steps above researching the intangible benefit of work-life balance. Albeit, this is a bit of an umbrella term further complicating the task of measuring it, but I did find two surveys that seem to do a decent job of it.

    International Social Survey Programme (ISSP)

    European Social Survey (ESS)

    There is a great deal of overlap between the two. Samples of questions from both surveys can be found below:

    I have come home from work too tired to do the chores which need to be done?
    It has been difficult for me to fulfill my family responsibility because of the amount of time I spend on my job?
    I have arrived at work too tired to function well because of the household work I had done?
    I have found it difficult to concentrate at work because of my family responsibilities?

    (A) Using this card, how often do you…
    1. … keep worrying about work problems when you are not working? (‘worrying’)
    2. … feel too tired after work to enjoy the things you would like to do at home? (‘too tired’)
    3. … find that your job prevents you from giving the time you want to your partner or family? (‘prevents’)

    (B) How often do you …
    4. … find that your partner or family gets fed up with the pressure of your job? (‘fed up’)
    5. … find it difficult to concentrate on work because of your family responsibilities? (‘concentrate’)

    While I may not use all the questions in each, I might create a condensed version of my own that is an amalgamation of both surveys.

    The link to this is below though you may not be able to access it without logging in:

    https://link.springer.com/article/10.1007/s11205-008-9297-5

Please post a comment

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s