New York City businesses that use artificial intelligence to help find employees must now prove the process was free of sexism and racism.
a new law, which enters into force on wednesday, is believed to be the first of its kind in the world. Under New York’s new rule, hiring software that relies on machine learning or artificial intelligence to help employers pick preferred candidates or weed out bad ones (called an automated employment decision tool, or AEDT) must pass an audit by an outside company to show you are free of racist or sexist bias.
Companies running AI recruiting software are also required to publish those results. Companies that use third-party AEDT software can no longer legally use such programs if they have not been audited.
The companies are using more and more automated tools in their recruitment processes. Cathy O’Neil, chief executive of Orcaa, a consulting firm that has been conducting audits of hiring tools for companies that want to keep up with New York’s new law, said the rise of tools that automatically judge recruiters candidates has become necessary because job seekers also use tools that submit large numbers of applications.
“In the age of the Internet, it is much easier to apply for a job. And there are tools for candidates to streamline that process. Like ‘Give us your resume and we’ll apply for 400 jobs,’” O’Neil said. “They get too many applications. They have to select the list somehow, so these algorithms do it for them.”
AI-infused hiring programs have come under scrutiny, especially over whether they end up exhibiting bias based on the data they are trained on. Studies have long found that programs using machine learning or artificial intelligence often exhibit racism, sexism and other biases.
As flashy generative AI apps like ChatGPT and Midjourney have gained popularity, federal lawmakers and even many tech company executives have repeatedly called for regulation. But so far, Congress has little sense of what that might look like.
Experts say that while the New York law is important for workers, it is still very limited.
Julia Stoyanovich, a professor of computer science at New York University and a founding member of the city’s Automatic Decision Systems Task Force, said it’s an important but still very limited start.
“First of all, I’m very happy that the law is on the books, that now there are rules and we’re going to start enforcing them,” Stoyanovich said.
“But there are also many gaps. So, for example, the bias audit is very limited in terms of categories. We don’t look at discrimination based on age, for example, which in hiring is a big problem, or disabilities,” she added.
It is also unclear how or to what extent the law will be enforced.
The New York Department of Consumer and Worker Protection, charged with enforcing the law, “will collect and investigate complaints” from companies accused of violating it, an agency spokesperson said.
Jake Metcalf, an AI researcher for Data & Society, a nonprofit group that studies the effects of technology on society, said the wording of the law, which defines AEDT as technology that “will substantially assist or replace the discretionary decision-making,” has led lawyers advising big business not to take it seriously.
“There are quite a few New York employment law firms that advise their clients that they don’t have to comply, given the letter of the law, even though the spirit of the law appears to apply to them,” Metcalf said. .
“It’s very difficult to understand what ‘substantially’ means,” he said.
Even if audits help reduce some bias against job candidates, it’s still not clear that AI job selection is particularly good at what it seeks to do.
“I don’t think they’re improving the overall system for people who are in the system, whether it’s for candidates or hiring managers,” Orcaa’s O’Neil said. “It’s just information overload and there’s not a lot of reason to rely on these tools to make things better.”
Stoyanovich went further, saying that some automated recruiting tools he’s studied just don’t work.
“One of the things this law doesn’t protect us against is these pointless detection methods. I can tell you about some of the ones I’ve audited, along with a team of collaborators, where it’s utter nonsense. So it’s not going to be biased, because it’s just random, as far as I can tell. So it will be just as absurd for men and women, blacks and whites,” Stoyanovich said.
