Bias in AI: why models can get it wrong

There's a comforting idea that a machine is objective because it has no feelings. It sounds logical, but it's false. An AI model isn't born knowing anything; it learns by copying patterns from mountains of data that we, the people, generate. And if that data is crooked, the model learns the same crookedness, just at massive scale and wearing the face of neutrality.

Understanding bias isn't a matter of abstract philosophy. More and more decisions that affect your life —who gets a loan, who gets called for an interview, who a camera identifies— pass through some algorithm. It's worth knowing why it sometimes gets things wrong.

What bias in AI means

Bias, in this context, means a system produces systematically worse or unfair results for certain groups of people. It's not an isolated error that hits everyone equally; it's a pattern that punishes some more than others. And the root is almost always in the same place: the data the model learned from.

A majority-white dataset does not produce accurate results for dark faces; training a system using only images of men does not guarantee it works for women.

The facial recognition case

The most studied example is facial recognition. In 2018, researcher Joy Buolamwini of the MIT Media Lab published the Gender Shades project, testing commercial systems from IBM, Microsoft and Face++. The finding was stark: they classified the gender of lighter-skinned men with an error of just 0.8%, but the error climbed to 34.7% for darker-skinned women. The same system, near-perfect for some and badly wrong for others.

It wasn't a one-off. In 2019 NIST, the U.S. standards agency, evaluated 189 algorithms from 99 developers using over 18 million images. Its lead author, Patrick Grother, reported that many algorithms returned false positives for Asian and African American faces 10 to 100 times more often than for Caucasian faces. NIST also noticed something revealing: algorithms developed in Asia didn't show that gap on Asian faces, likely because their training data included more Asian faces.

The hiring case

Bias doesn't live only in cameras. Amazon built a tool to screen résumés, trained on those of employees who had been successful at the company in the past. Because those employees were mostly men, the model learned to penalize résumés that contained the word "women" or that mentioned women's colleges. The machine wasn't sexist out of malice; it simply reproduced, amplified, an inequality that already existed in the historical data.

Where bias comes from

Looking at the examples, the problem is almost never the code itself, but what goes in and how we use it. The sources agree on a few recurring causes.

• Unrepresentative data: if a group barely appears in training, the model learns poorly about it.
• Unfair history packaged as truth: if the data reflects biased human decisions, the model inherits them as if they were the norm.
• Human labels and criteria: someone decided what counted as a "good candidate" or a "match," and that judgment carries bias too.
• No per-group testing: if nobody measures performance broken down by age, gender or skin tone, the gap stays invisible.

What this means for an ordinary business

You don't have to train a giant model for this to touch you. If you use AI tools to classify customers, filter messages, or recommend actions, a healthy dose of skepticism helps. Ask the vendor what data it was trained on, test the tool with your own real clientele, and watch whether it treats groups differently when it shouldn't. The good news is that bias, once measured, can be corrected; the danger is not looking at it.

That's why it pays to pick tools that are honest about their limits. A conversational agent like Lidia, for example, should be judged by how well it serves all your customers equally, not just the average customer who showed up most in the examples.

Why measuring by group matters

There's a practical lesson hidden in the NIST case that almost nobody mentions. When they measured average performance, several algorithms looked excellent; the problem only became visible once they broke the results out by group. A system can be 98% accurate overall and still fail badly for a minority that weighs little in the average. The average, without meaning to, hides the people who show up least.

For a business, the moral is direct: don't settle for the headline number the vendor shows you. Ask how it performs across different types of customer, different accents, different contexts. If a tool only brags about its average and gets uncomfortable when you ask for the per-group detail, that discomfort already tells you something.

Takeaway

AI is neither magical nor evil: it's a mirror of the data we feed it. When that data is partial, the model will be too, with the trap that it looks impartial. Cases like Gender Shades, NIST and Amazon don't prove AI is useless; they prove it needs human oversight, careful data, and the honesty to measure whether it treats everyone well. That vigilance isn't a brake on technology; it's what makes it trustworthy.

Sources

• MIT Media Lab — https://www.media.mit.edu/articles/study-finds-gender-and-skin-type-bias-in-commercial-artificial-intelligence-systems/
• NIST — https://www.nist.gov/news-events/news/2019/12/nist-study-evaluates-effects-race-age-sex-face-recognition-software
• MIT Sloan — https://mitsloan.mit.edu/ideas-made-to-matter/unmasking-bias-facial-recognition-algorithms
• CSIS — https://www.csis.org/blogs/strategic-technologies-blog/problem-bias-facial-recognition
• Wikipedia (Algorithmic bias) — https://en.wikipedia.org/wiki/Algorithmic_bias