II. Development of the model

Bias and Fairness in Machine Learning with Supervision

 

You previously learned that issues in training data can lead to biased models. Machine learning is now being used in an increasing number of application areas – including particularly sensitive domains such as healthcare, recruitment, or criminal justice. Biased model outcomes in such areas can have significant negative impacts on people’s lives. Below you will find some examples:

 
Correctional Offender Management Profiles for Alternative Sanctions (COMPAS)
COMPAS was one of two leading commercial tools in the U.S. used to assess the likelihood that an incarcerated person would reoffend – a term used to describe offenders who commit crimes again.

However, an analysis conducted by ProPublica showed that Black defendants were significantly more likely to be falsely classified as high risk of reoffending than white defendants. Conversely, white defendants were far more often falsely assessed as low risk than Black defendants. Despite this, the tool was increasingly used at early stages of the criminal justice process – for example, in decisions about pretrial detention and sentencing, the so-called “front end” of the justice system.

READ MORE: Mattu, Julia Angwin, Jeff Larson, Lauren Kirchner, Surya. “Machine Bias.” ProPublica. Accessed March 6, 2024. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.

 
AI recruitment tool 

Amazon developed an AI hiring tool that was trained on résumés of previous successful applicants, who were unfortunately predominantly male. The model learned patterns that reflected this imbalance, resulting in men being favored over women. Although this tool was later discontinued, similar AI systems continue to be developed and used in practice.

The problem is that these tools can learn and reinforce gender bias. For example, Amazon’s system downgraded applicants who used words such as “women,” for instance in “women’s chess club captain,” and even favored hobbies that are typically associated with men. Other AI hiring tools show similar problems and sometimes award higher scores for wording that occurs more frequently in men’s résumés.

Alamy Woman on an AI video interview

READ MORE: Charlotte Lytton (2024). AI hiring tools may be filtering out the best job applicants. BBC. https://www.bbc.com/worklife/article/20240214-ai-recruiting-hiring-software-bias-discrimination

As you can see, in sensitive application areas, biased models can further reinforce existing prejudices and inequalities against historically disadvantaged groups, such as Black people or women.

Therefore, in such areas it is particularly important to develop machine learning models using bias-reduction methods in order to avoid ethical issues.


A machine learning model can be classified as unfair or biased if its decisions are skewed in favor of a particular group of people.
In the context of machine learning, fairness is defined as “the absence of prejudice or favoritism toward an individual or group based on their inherent or acquired characteristics”.