Word2vec was able to provide numeric vector representations of words, and those vectors could be used to make calculations like: “If you take the word ‘king’ and subtract ‘man’ and add ‘woman,’ what do you get?” The result is obviously the word “queen,” and these vector embeddings provide similar meaning through almost the entire English language.
The embeddings also allow us to make simple calculations to identify complex relationships. For instance, given the word problem, “Paris is to France as Rome is to ____,” simple vector math will tell you “Italy” as the answer.
Word2vec was widely accepted and many pre-trained vector embeddings were then incorporated into all sorts of other software, including recommendation engines and job-search systems.
Is the Machine Sexist?
However, when researchers started to dig into the word2vec trained on Google News (known as w2vNEWS), they found some more troubling correlations. Correlations like, “Man is to woman as computer programmer is to homemaker,” and “Man is to woman as boss is to receptionist.”
The trouble wasn’t the machine itself — it was working perfectly. Nor did the engineers’ intentional or unintentional bias make its way into the software. Instead, the problem was that the program picked up on the biases inherent in the news being reported on Google News. The word “homemaker” appeared more frequently with the pronoun, “she” than with “he,” so it was more closely associated with female-ness than male-ness.
"Any time you have a dataset of human decisions, it includes bias," said Roman Yampolskiy, director of the Cybersecurity Lab at the University of Louisville. "Whom to hire, grades for student essays, medical diagnosis, object descriptions, all will contain some combination of cultural, educational, gender, race, or other biases."
While that’s a bit troubling, it doesn’t seem like an earth shattering problem. Until you think about all of the systems that use word embeddings to understand language — including chatbots, recommendation algorithms, image-captioning programs, and translation systems. Studies have shown that bias in training data can result in everything from job-search advertisements showing women lower paying jobs than men to predictive policing software that disproportionately rates people of color as likely to commit crimes.
Kathryn Hume, of artificial-intelligence company Integrate.ai, calls this the “time warp” of AI: “Capturing trends in human behavior from our near past and projecting them into our near future.”
Debiasing Word Embeddings
A simplistic approach to solve this problem would be to simply remove the “gender dimension” from all the words to essentially take all gender out of your data. But gender differences aren’t inherently a problem. The kind of difference that makes “king” and “queen” different isn’t so troubling. But there are definite problems with “computer programmer” and “homemaker.”
Instead of removing all gender from the words, researchers wanted to identify the difference between a legitimate gender difference and a biased gender difference.
So researchers set out to disentangle the biased gendered terms from the unbiased ones. They explain it like this: if there’s a “gender dimension,” then that dimension can have shape to it. They treat bias like a warp on the gender space that affects different types of words. They set out to identify the terms that are problematic and exclude them while leaving the unbiased terms untouched. This method is called “hard debiasing.” By doing this, they were able to remove a lot of bias from their dataset while keeping the structure of their data.
To learn more about their approach and the results, check out the full study here.
Great Power, Great Responsibility
In researching this study, I talked to Jeff McGehee, director of engineering at Very. I asked him if Google fixed the problem with w2vNEWS, and he said it would be impossible to know if all the vector embeddings people use in their software would be fixed even if Google corrected the bias.
But Jeff said there’s a good chance that data scientists using w2vNEWS in their own software had already identified this hidden bias and taken action in order to reduce its impact. He said, “If I was building something where gender bias would have significant impact, that’s the first question I would’ve asked.”
This highlights the importance of having trained data scientists building products that leverage machine learning. Bias in training data can be mitigated, but only if someone recognizes that it’s there and knows how to correct it.
Today, it’s easier than ever for any software engineer to add natural language processing or facial recognition to their products. It’s also more important than ever to remember that the products we build can project the biases of today onto the world we live in tomorrow.