There’s an old adage along the lines of “The more you know, the more you know what you don’t know.” I think it might have originally come from a Peanuts strip where Charlie Brown was going through his Nietzsche phase and staring into the abyss. Be that as it may, this concept came to mind recently when I was helping my employer go through a changeover to a new version of our recruiting website. I-O Psychologists and our professional neighbors know an awful lot about recruitment, but as I tried to help define how our jobs website would work, it occurred to me that we may not know the answer to one of the most important questions, like how the heck people use job boards.
I’m thinking about basic stuff here. How do people search for jobs? By keyword, by location, by job title, by salary range, by educational requirements? What kinds of factors make people more likely to come back to a website and look for newly posted jobs, or to sign up for e-mail or text alerts? Do they search or browse? How many clicks in the application process before someone decides it’s not worth it? What are the effects of different kinds of information in job postings and the presentation thereof? What are the effects of noting (or not noting) selection systems like drug screens, physicals, pre-employment test, or background checks on applicant reactions or perceptions of the company?
Companies who roll out new internet job boards have a LOT of questions about these basics, and the answers dictate how things are going to be configured and the quality of experience job-seekers will have when they come looking for new opportunities. If you do it better than your competition, you’re probably going to have a real advantage in the marketplace, just like any other superior recruiting activity. On the other hand, if you do things haphazardly, only the really determined (or desperate) will make it through the gauntlet.
As I said, some of these questions could probably be answered by synthesizing information from the recruiting literature and company culture literature. We know, for example, something about how people react to drug screens or to diversity statements in job postings. That’s research that can be put to use. But I think there’s big chunks of the solution missing in that what we don’t know about are the nuts and bolts about how people use websites like these. What do people like, dislike, want, never use, et cetera? There are no shortage of experts on web design and web usability –indeed, it’s grown into an entire industry. And while I’m sure someone could offer to tell you what color palate to use, what size to make your font, and where on the page to put your logo, I’m not sure anyone has sat down to tell you the best way to get people to search for jobs that match their qualifications or how long an online application can be before casual job seekers wander off.
Some colaboration is needed here. Of course, maybe despite the considerable time I spent with Google looking into this issue, there is a body of research out there and I just don’t know about it. If you know that to be the case, please let me know!
I’ve been thinking a bit about retesting policies lately. You know, if someone takes your employment test, do you let them take it again? When? How many times? Do you poke them with a stick first?
Based on what I’ve seen and heard from talking to other colleagues, the only thing that people seem to agree on is that they’re needed. After that, recommendations get either vague or militantly specific. There do seem to be a few things that most of the experts agree you need to keep in mind.
First, how long do you make people wait before retesting? I call this “the cooldown timer” but that’s just the World of Warcraft geek in me. The main concerns here are drains on company resources (in terms of how expensive it is to give a test) and a practice effect for test-takers. If your test is a hands-on work sample that takes 6 hours to complete and can only be administered one-on-one, then you may want to keep people from retesting as often as you might if you’re talking about a 40-minute, paper-based test that can be given to dozens of people at a time.
The practice effect is a thornier problem. If a person is allowed to take the same test over and over again her score may have too much to do with practice and not enough to do with the validity of the test. Problem. This may be especially true of timed tests, or with things like tests of reading comprehension where the test taker would benefit from repeated exposure to the material. The nature of the test will have to inform your decision, but generally you can combat this by either having alternate forms of your test (expensive!) and/or having them wait a month or more between attempts so that they have a chance to forget (cheap!).
On the other hand, sometimes practice isn’t a bad thing. Some skills (say, data entry or physical abilities) may be expected to change with practice, and if they improve that’s a GOOD thing and you may not want to discourage people. Again, the nature of the test and what constructs it measures should inform your decision.
On the third hand, there are arguably some constructs that are highly unlikely to change over time. Personality is stable by definition. General mental ability doesn’t change much in adults. In these cases allowing retesting may have more to do with controlling perceptions of fairness than with validity.
The second thing you want to address is which test results are a person’s “official” ones. If you just work with a pass/fail result where a person has to pass a certain cut score in order to be put in an applicant pool, this is easy. Once a person passes the test, no retesting is needed or allowed. But if you use top-down selection or banding, things get trickier. Candidates may want to retest to move to the top of the pile and better their chances of getting the job. This is going to be particularly true if you always let their highest score be their official one.
My suggestion? Always make the most recent score the one that’s used for any selection decision. People may try to improve their score through retesting, but if they backslide it’s just part of the risk inherent to the process. Life is like a game of Chutes and Ladders that way.
Finally, you need to consider how long test results are good for. In other words, do they curdle like milk and expire if given long enough? Again, this is something where the nature of the test is going to have to be your guide. In general, aptitude, personality, and general mental ability tests aren’t going to change, but tests of physical strength, skill, or even job knowledge are susceptible to the ravages of time and may call for quicker expiration dates.