Screening CVs With AI, Watch Out for Bias Getting Automated Too

One job opening, three hundred applications. Reading them all can take days, so it's natural to think: have AI read every CV, score them, take the top ten. Fast, cheap, and it feels objective.

The trouble is that "feels objective" is exactly the trap. AI learns from patterns, and if the patterns contain bias, AI applies that bias very consistently, to hundreds of applicants at once, without anyone noticing.

Two often-cited examples

Amazon, 2018. Reuters reported that Amazon scrapped an AI recruiting tool it had been developing since 2014 (republished by Euronews). The model was trained on CVs submitted over ten years, mostly from men. As a result, the system learned to downgrade CVs containing the word "women's" and graduates of all-women's colleges. Efforts to fix it didn't guarantee fair results, and the project was abandoned. Amazon said its recruiters never relied solely on the tool.

University of Washington, 2024. UW researchers tested three large language models with more than 550 real CVs and more than 500 job listings across nine occupations, more than three million comparisons in total. The CVs were the same, only the names changed. The models favored names associated with white people in 85% of comparisons, and female names in only 11% of comparisons, versus 52% for male names.

Comparisons where the models favored white-associated names (UW, 2024)

85%

Comparisons where female names were chosen, versus 52% for male names

11%

Total CV and job listing comparisons in the study

3M+

The context is American, with American names. But the mechanism applies anywhere: a model picks up signals that have nothing to do with ability to do the job. In Indonesia, CVs often include a photo, age, marital status, address, and the name of the university. Any of those can become a signal that gets "scored" without anyone intending it.

Relevant rules

In Indonesia, Article 10(1) of the Personal Data Protection Law (UU PDP) gives data subjects the right to object to decisions based solely on automated processing, including profiling, that have legal consequences or a significant impact on them. Further details are to be set by government regulation. Being rejected for a job can reasonably count as a significant impact on an applicant, so the final decision should stay with a person. This is orientation, not legal advice.

For comparison, Annex III of the EU AI Act classifies AI systems for recruitment and selection, including those that analyze and filter applications and evaluate candidates, as high-risk. That's a European rule, but it shows how regulators view AI in hiring.

How to use AI in hiring more safely

AI can still be very helpful, as long as its role is right. Here's what I'd suggest:

1. Write the criteria first, based on the job. The skills, experience, and requirements that are actually needed. Without written criteria, AI will guess for itself what counts as "good".
2. Use AI to summarize, not to produce a final ranking. Ask it to match each CV against the written criteria and show the evidence, not to give a score from 1 to 100.
3. Remove irrelevant information before processing: name, photo, age, marital status, and address.
4. A person reads every shortlisted candidate, plus a random sample of rejections. If there's a strong candidate in the rejected pile, the filter needs checking.
5. Run a simple bias test. Take one CV, change the name, age, or university, and see whether the result changes. If it does, don't use it yet.
6. Tell applicants that AI helps with screening, and offer a way to ask for human review.
7. Protect applicant data. Don't paste CVs into personal AI accounts, keep only what you need, and delete it once the process is done.

On personal accounts and sensitive data, I go into more detail in Want to Summarize Your Company's Financial Data in ChatGPT? Is It Dangerous?. And the split between AI and humans used here is the same as in At What Point Should Humans Still Check AI and Automation Results: decisions with a big impact on people stay with people.

Wrapping up

AI can cut the time it takes to read hundreds of CVs, but don't let it quietly cut people's chances too. Let AI summarize and match against clear criteria, remove irrelevant signals, test the results, and let a person make the decision.

If you want to streamline hiring with AI without adding risk, tell me about your current process first.