Skip to main content

How a Machine Learns Prejudice

Artificial intelligence picks up bias from human creators—not from hard, cold logic

If artificial intelligence takes over our lives, it probably won’t involve humans battling an army of robots that relentlessly apply Spock-like logic as they physically enslave us. Instead, the machine-learning algorithms that already let AI programs recommend a movie you’d like or recognize your friend’s face in a photo will likely be the same ones that one day deny you a loan, lead the police to your neighborhood or tell your doctor you need to go on a diet. And since humans create these algorithms, they're just as prone to biases that could lead to bad decisions—and worse outcomes.

These biases create some immediate concerns about our increasing reliance on artificially intelligent technology, as any AI system designed by humans to be absolutely "neutral" could still reinforce humans’ prejudicial thinking instead of seeing through it. Law enforcement officials have already been criticized, for example, for using computer algorithms that allegedly tag black defendants as more likely to commit a future crime, even though the program was not designed to explicitly consider race.

The main problem is twofold: First, data used to calibrate machine-learning algorithms are sometimes insufficient, and second, the algorithms themselves can be poorly designed. Machine learning is the process by which software developers train an AI algorithm, using massive amounts of data relevant to the task at hand. Eventually the algorithm spots patterns in the initially provided data, enabling it to recognize similar patterns in new data. But this does not always work out as planned, and the results can be horrific. In June 2015, for example, Google’s photo categorization system identified two African Americans as “gorillas.” The company quickly fixed the problem, but Microsoft AI researcher Kate Crawford noted in a New York Times op-ed that the blunder reflected a larger “white guy problem” in AI. That is, the data used to train the software relied too heavily on photos of white people, diminishing its ability to accurately identify images of people with different features.


On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.


The recent spate of fake stories inundating Facebook users’ news feeds also highlights the AI bias problem. Facebook’s trending news algorithm was prioritizing stories based on engagement—how often users click on or share. Veracity was not considered at all. In early November several news outlets revealed that a group of Macedonian teenagers had fooled Facebook’s News Feed algorithm into promoting blatantly false stories that appealed to right-wing voters during the U.S. election. Facebook says it has modified the algorithm since then and has announced plans for Snopes, Factcheck.org, ABC News and PolitiFact to help weed out obviously false articles.

“It's a bit like the ‘Russian tank problem,’” says Hal Daumé III, an associate professor of computer science at the University of Maryland. This legend—apocryphal but illustrative, and oft-related by computer science teachers—dates from machine learning’s early days in the 1980s. The story says the U.S. military tried training a computer to distinguish between Russian and American tanks in photos. “They got super-high classification accuracy—but all the photos of Russian tanks were blurry, and American tanks were high-definition,” Daumé explains. Instead of identifying tanks, the algorithm learned to distinguish between grainy and high-quality photos.

Despite such known limitations, a group of researchers recently released a study asserting that an algorithm can infer whether someone is a convicted criminal by assessing facial features. Xiaolin Wu and Xi Zhang, researchers at China’s Shanghai Jiao Tong University, trained a machine-learning algorithm on a dataset of 1,856 photos of faces— 730 convicted criminals and 1,126 non-criminals. After looking at 90 percent of the pictures, the AI was able to correctly identify which ones in the remaining 10 percent were the convicted criminals.

This algorithm correlated specific facial characteristics with criminality, according to the study. Criminals, for example, were more likely to have certain spatial relationships between the positions of eye corners, lip curvature and the tip of the nose, Wu says—although he notes that having any one of those relationships does not necessarily indicate that a person is more likely to be a criminal. Wu also found that the criminals’ faces differed from one another more, while non-criminals tended to share similar features.

Wu continued testing the algorithm using a different set of photos it had not seen before, and found that it could correctly spot a criminal more often than not. The researchers attempted to avoid bias by training and testing their algorithm using only faces of young or middle-aged Chinese men with no facial hair or scars.

“I set out to prove physiognomy was wrong,” Wu says, referring to the centuries-old pseudoscience of assessing character based on facial features. “We were surprised by the results.” Although the study might appear to validate some aspects of physiognomy, Wu acknowledges that it would be “insane” to use such technology to pick someone out of a police lineup, and says there is no plan for any law enforcement application.

Other scientists say Wu and Zhang's findings may be simply reinforcing existing biases. The subjects’ criminality was determined by a local justice system run by humans making (perhaps subconsciously) biased decisions, notes Blaise Agüera y Arcas, a principal scientist at Google who studies machine learning. The central problem with the paper is that it relies on this system “as the ground truth for labeling criminals, then concludes that the resulting [machine learning] is unbiased by human judgment,” Agüera y Arcas adds.

Wu and his colleagues “jump right to the conclusion that they found an underlying pattern in nature—that facial structure predicts criminality. That’s a really reckless conclusion,” says Kyle Wilson, an assistant professor of mathematics at Washington College who has studied computer vision. Wilson also says this algorithm may be simply reflecting the bias of the humans in one particular justice system, and might do the same thing in any other country. “The same data and tools could be used to better understand [human] biases based on appearance that are at play in the criminal justice system,” Wilson says. “Instead, they have taught a computer to reproduce those same human biases.”

Still others say the technology could be improved by accounting for errors in the patterns computers learn, in an attempt to keep out human prejudices. An AI system will make mistakes when learning—in fact it must, and that's why it's called “learning,” says Jürgen Schmidhuber, scientific director of the Swiss AI Lab Dalle Molle Institute for Artificial Intelligence. Computers, he notes, will only learn as well as the data they are given allows. “You cannot eliminate all these sources of bias, just like you can’t eliminate these sources for humans,” he says. But it is possible, he adds, to acknowledge that, and then to make sure one uses good data and designs the task well; asking the right questions is crucial. Or, to remember an old programmer's saying: “Garbage in, garbage out.”