What to consider when choosing psychometrics: Four key questions to ask a test publisher
This week’s article is all about what do you need to know from your suppliers before putting your money where your mouth is and purchasing your psychometric tools. It follows on from part one - ‘Why use them in the first place?’ and part two - ‘Five key questions to ask yourself before buying psychometrics’,
The reality is that once you’ve got past that first hurdle of figuring out what type of measure you want, you then have a veritable minefield of different suppliers to choose from – all of whom will tell you that their psychometrics are better than all the others!!! So, to help you navigate through these tricky waters here are some key questions to ask when making contact:
1. How much do the products cost?
Different test publishers have different pricing schedules and structures - with some more easy to navigate around that others.
My advice here is that when speaking to a potential supplier have in mind how many respondents you want to assess, both immediately and approximately over the forthcoming 12 months - and then ask them to provide you a total price based on these numbers. This will be the simplest way of getting the information you need about costs.
Hidden (or not so hidden extra) costs to watch out for, that will be additional to the actual costs of the materials you want to purchase, are:
Licence fee (this may be a one-off or annual)
Set-up costs (with an online system you may expect some reasonable charge for set up but watch out for being charged just for having access to basic functionality)
User Manuals
Training (I’ll come to this in more detail shortly)
Access to different norm groups
Being expected to purchase large quantities of 'credits' at a time
2. How do your psychometrics tools deter and manage cheating and faking?
This is something customers always want to ask so I decided I might as well cover it here too. When asking this question ideally your potential supplier will be telling you the following things:
Ability tests.
Test items are randomised to reduce the possibility of cheat sheets (if online)
The assessment includes clear deterrents as part of the introduction to the test, i.e., honesty contracts
There are alternative materials available if you want to re-test (e.g., parallel forms, paper and pencil/online equivalents) and ‘verify’ functionality if online
Personality questionnaires.
Rigorous validation to eliminate items that appear to be subject to respondents demonstrating high levels of social desirability
That there are ways of measuring social desirability (i.e., the extent to which an individual appears to respond in a way which presents them in a favourable light), e.g. inclusion of a Social Desirability scale
The instructions for the instrument include ‘motivators’ which highlight the benefits of answering honestly
3. What training will we need to be able to use your products?
Different tools require different types or amounts of training. Here is a rundown of some of the most common forms:
Test User Training.
Starting with the industry recognised ones, Test User Training (ability and personality) certification is available from the British Psychological Society (BPS). Formerly known as Level A and Level B training, the certifications will provide a good foundation knowledge of psychometrics, as well as covering how to administer and interpret tests.
You can typically complete the training for ability tests and personality questionnaires in about 3 days each, or a combined course covering both within 5 days.
Conversion training.
This typically only applies to personality questionnaires, and when you have already received some training (Test User: Personality) but need to learn how to use that test publisher’s particular instrument. However, this should only really apply in situations where the instrument you are looking to use is quantifiably different to the personality questionnaire you have already received training on. For example, you are trained to use a trait-based instrument, e.g. OPQ32, Saville Wave, Lumina Spark, etc. and you want to start using a type-based instrument, e.g. MBTI.
The length of conversion training is usually 1-2 days.
Product-specific training.
Due to the time and cost commitment that can be involved in Test User training, some test publishers will offer courses that focus on their specific instruments only. Typically, this will involve all of the skills and knowledge required to use that tool, but does not go into the same depth or breadth that a Test User training qualification would.
Product specific training typically lasts anywhere between half a day and two days.
4. What is the reliability and validity of your products?
We are slightly venturing into psychometric ‘geek land’ here if you are not naturally inclined to contemplate the statistics that underpin the development of an ability test or personality questionnaire. However, this question is really important as this information fundamentally tells you whether or not that instrument actually works!
Reliability.
This provides information on whether or not the psychometric can and will measure the constructs you are interested in (i.e., ability, extraversion, conscientiousness, etc) in a consistent and reliable way.
This ability to be consistent is measured using correlations, and if I’m starting to get too techie and loose you already, the bit you really want to know is can they show you a figure that is 0.7 or above.
Ideally, a test publisher shouldn’t even be publishing an instrument that doesn’t meet 0.7 as a minimum, but there are a few who do so it’s always worth asking.
Validity.
This is all about does the instrument measure what it’s meant to measure, i.e., is the tool valid for the job?
Now there isn’t a straight forward single number that a test publisher can show you with this one. Unfortunately! However, what you can do is be requesting to have a look at the test or questionnaire you are interested in and seeing whether the content ‘looks right’, i.e., does that numerical ability test consist of the need to carry out calculations and reason with numerical data? Ultimately, your common sense needs to kick in here.
If you fancy delving into validity in a bit more depth then you can ask about construct validity and criterion validity.
The first is what work the test publisher did to check that the tool is really measuring the underlying construct – this is done by getting a group of people to complete their instrument and an established one (that measures the same thing) and correlating the results.
In contrast, criterion validity looks to correlate performance on the ability test or personality questionnaire with an external measure, such as interview results or job performance. Test publishers have to work with existing clients to do this and it is primarily with the aim of seeing whether performance on the psychometric can predict performance in the job.
Summary
So, there you have four fairly hard-hitting questions to ask any potential supplier of your ability tests and/or personality questionnaires. These range from the practical (how much does it cost) through to checking that the tool will actually work (reliability and validity).
If you haven’t read them already, do check out Part 1: Why use them in the first place? and Part 2: The five questions you need to ask yourself before speaking to a supplier .
Find out more
If you would like find out more about how Psychology Works uses psychometrics as part of our work in the areas of assessment, leadership development, or talent, please do get in touch.
In the meantime, if you’ve enjoyed Maria’s ramblings why not check out some of our other blogs and thought pieces. For real time updates and insights you can also find us on LinkedIn, our social media platform of choice: @PsychologyWorks and @mariagardner