Artificial intelligence is transforming the hiring process. As we previously reported, tools like ChatGPT are making it easier for job seekers to generate resumes and cover letters that are likely to get a look. And employers are increasingly relying on more layers of technology to winnow down applications, verify applicant qualifications and make a match.

Pre-employment screening tests are becoming more common. They might require demonstrating skills like coding, completing a job simulation or something called a psychometrics assessment, which aims to quantify characteristics we often think of as intangible: personality, attitudes, integrity or “emotional intelligence.” 

The principles of psychometrics have been studied for more than a century, and they have a new spin in the digital age.

James Klusaritz is all too familiar with the assessments. In fact, he’ll never be able to look at a balloon the same way again.

“Yeah, it’s like PTSD,” he said.

About a year ago, he was graduating from the University of Pennsylvania with degrees in economics and philosophy, and looking to join the corporate ranks. He put in dozens of applications at big consulting firms like McKinsey & Co. and PwC, and he’d often get a similar automated reply:

“‘OK, to complete your application, you have to, you know, spend 30 minutes playing these fun games,'” like the balloon game, Klusaritz said, part of an assessment from the company Pymetrics, which is now part of Harver.

The balloon game asks you to click on a digital balloon to incrementally inflate it, collecting more money the bigger it gets. You can move on at any time, but if you click too many times it will ultimately pop and you lose all the money. To complicate things, there are different colors of balloons that inflate to different volumes before they pop.

Klusaritz couldn’t immediately tell what the test was measuring or what he could do to ace it, which is by design, said Tomas Chamorro-Premuzic, an organizational psychologist, author and chief innovation officer at Manpower Group. He said these assessments are about revealing inner potential: “predicting whether somebody can do a job that they haven’t done before, or whether somebody is likely to learn a skill that they haven’t actually displayed or learned before.”

Psychometric tests date back to World War I, when the military used them to identify recruits at risk of developing shell shock. The methods filtered into business, but were often impractical.

“In the old days, you would have to put somebody through an assessment center,” said Chamorro-Premuzic. “They would have to spend four hours in the office doing an exercise or a simulation.”

Today’s psychometric assessments are shorter and usually digital. The Traitify personality test from Paradox — used by large employers like McDonald’s — touts itself as the world’s fastest, clocking in at about 90 seconds.

“It is very challenging to develop an assessment that is that short but also has predictive value,” said Heather Myers, chief industrial-organizational psychologist at Paradox who leads the team that developed the Traitify test. It asks users to swipe through images — like a driver in traffic with the caption “unfazed” — and mark whether they identify or not.

“To see kind of what the behavioral fit for an individual is and the roles that they’re looking to apply to,” Myers said.

Traitify uses a mobile game format, not AI, to speed up its psychometric assessments, but many other platforms rely on AI. The technology can make tests more efficient, said Beth Bynum with Human Resources Research Organization.

With machine learning, companies can analyze huge amounts of data to find the exact permutations of traits predictive of success for any one job.

“The challenges are with transparency,” said Bynum. “I think the more data you have, the more risk there is to not understanding what’s going into your prediction, and where you may have something that’s not job-relevant being included.”

An example often cited by researchers is an algorithm that was trained to identify images of huskies and wolves, but was trained on images in which all of the photos of wolves had snow in the background. The machine mistakenly learned that “snow” meant “wolf.”

In the employment context, Reuters first reported on an experiment at Amazon testing artificial intelligence for recruiting that was shut down after the algorithm was found to score women applicants systematically lower based on inputs like attendance at a women’s college. The AI had apparently learned from past data that men were more likely to be employed at the tech company.

Ben Porr, an organizational psychologist and executive at Harver, said their algorithms are not self-learning. They’re set and then monitored by real people. He said algorithms can now be tuned to maximize for those profiles predictive of success while at the same time minimizing adverse impacts, like bias.

“We’re not saying that we’re going to be completely bias free, right? Because it is humans developing this,” Porr said. “But we can quickly identify if there is any bias and correct for that,” unlike the traditional hiring process, he said, where bias often goes unmeasured.

All of the platforms we talked to for this story have teams of scientists that test their assessments for validity, reliability and bias, often with third parties or peer review. They say the tools don’t replace human decisions, and they provide standardized and evidence-based metrics that can make hiring decisions more fair and meritocratic.

But the demand for these tests is growing fast, without much legal oversight in the United States. 

And there are plenty of ways AI can go wrong, said Manpower’s Tomas Chamorro-Premuzic.

“There is a real risk that actually, at least in the short term, this increases inequality,” he said.

Recent college grad James Klusaritz never found out how he did on the Pymetrics balloon test. He’s shifted his focus from finding a corporate job to doing comedy on TikTok.

“My best videos are ones that sort of make light of things people hate or things I hated,” said Klusaritz.

A sketch about the balloon test and other psychometric games has about 20,000 likes.