Red Flags. What do you really need to know when buying psychometrics?
Last week we ran an event in collaboration with the Association for Business Psychology (ABP) for over 150 psychologists, HR professionals, coaches and OD experts. The topic of this event asked the question "Red flags. What do you really need to know when buying psychometrics?"
In the UK, the psychometrics industry is now relatively mature, and with numerous test publishers offering ability tests, personality questionnaires and situational judgement tests, amongst others. But with so much choice on offer across the full price spectrum, how easy is it to really make an informed decision?
Here is our rundown of five red flags, which should trigger a 'Buyer Beware' response:
1. The test publisher won't provide you with any reliability and/or validity information
This is a huge red flag and should trigger an immediate 'thanks, but no thanks' response on your part. If the test publisher won't give you access to the technical manual for the tool(s) you are interested in, this a serious concern as it denies you the ability to independently verify the quality of the tool. An 'Oh, the Technical Manual is still in development', is similarly not good enough. Walk away.
A good test publisher will not hesitate to send you the technical manual, and ideally will offer a call to talk you through it. Furthermore, they will be able to point you in the direction of at least one independent review of the assessment, whether it be a review by the BPS* or link to a peer-reviewed paper.
2. You only seem to be able to talk to a salesperson
Whilst it's entirely reasonable for large test publishers to employ a team of sales people, they should also be able offer you the opportunity to speak to a consultant or psychologist. The sales consultants, whilst having a reasonable level of knowledge, will never have the deep technical knowledge a trained psychologist will.
With such a wide range of scenarios within which you can use a psychometric, and with such choice on offer, it really is worth taking the time to speak to a consultant. In particular, get them to talk you through how their tool would be well suited to meet your specific needs. Discuss administration, reports, interpretation, etc.
If you are unable to speak to a properly trained psychologist, seriously consider walking away.
3. The tool is presented as 'cure for all ills'
Psychometrics work best when they are used for the specific reason that they were designed for. Less reputable test publishers will sell their '16 item' (or some other ridiculously short assessment) as being suitable for every possible scenario, and offer approximately 47 different reports.
Selection. Tick. Leadership development. Tick. Team building. Tick. Redundancy. Tick. Employee engagement. Tick. A bad back. Tick. World peace. Tick.
If the output on offer, relative to the amount of information being collected, seems too good to be true, it probably is.
4. The technical information isn't up to scratch.
When it comes to the quality of a psychometric, there are a couple of basic tests it should be able to pass:
Reliability coefficient over 0.7
Construct validity – this uses correlations to compare their tool with existing ones on the market, to check that it is measuring what it claims to measure
A large representative norm group of at least 150
I'm sure my psychometrician colleagues and peers would add a few others to this list, but these three are a good test of the basic robustness of a tool.
5. No requirement for test user training
Whilst there has been a real push in recent years amongst test publishers to make their tools more accessible and user friendly, there also have to be some limits. If they are offering you their products without checking your existing training and experience in using psychometrics, this should also trigger a 'Buyer Beware' response.
A little bit of knowledge is a dangerous thing when it comes to psychometrics. Interpreting and/or feeding back the information incorrectly is a real risk and with real world consequences.
A good test publisher will check if you've previously completed any BPS* approved test user training (ability and/or personality). They will then be able to confirm if your existing training gives you the appropriate level of knowledge and experience needed to use their tool straight away. If not, they will offer you the appropriate training.
To be clear, a good test publisher will not allow you to use their tool, without the right training. So, whilst this might appear to be a bit frustrating, a cheap money-making exercise or them putting obstacles in the way of you using their tool, in reality this is a basic duty of care on their part.
And one more for good luck….
They are unable to clearly explain what they've done to make sure their tools are accessible and avoid adverse impact.
In this day and age there is absolutely no excuse for an online assessment not meeting basic accessibility standards. For example:
What guidance do they provide and/or reasonable adjustments do they recommend for those with neurodiverse conditions, such as dyslexia?
Is their online assessment compatible with screen readers, giving the candidates opportunity to change text size and screen contrast, etc.?
What work did they do during the test development phase to assess for adverse impact and ensure that the tool does not inherently discriminate against any particular group based on gender, ethnicity, etc.?
What are they doing to keep their assessments and norm groups up to date? How are they continuing to monitor matters related to diversity and inclusion?
What to do if you spot a red flag?
Firstly, we would say, slow down the buying process and resist any pressure the test publisher might be putting on you to purchase. The risk of using the wrong or substandard tool is not worth the risk of losing out on the 'one time only' super discount deal you are being offered.
Secondly, if your 'Buyer Beware' button is triggered, it's not to say that you absolutely shouldn’t buy that tool. Instead, dig into what is being offered in a lot more detail and at the very least get a second opinion from an expert test user that you trust.
*British Psychological Society (BPS)
The encore.
If you’ve made it this far, and still want more, then why not check out these other articles and thought pieces….
Find out more…
If you would like to know more about how Psychology Works uses psychometrics in our assessment and leadership development work please don’t hesitate to contact us.
In the meantime if you’ve enjoyed Maria’s ramblings why not check out some of our other blogs and thought pieces.
For real time updates and insights you can also find us on LinkedIn, our social media platform of choice: @PsychologyWorks and @mariagardner