Definitions of fairness in machine learning explained through examples

Updated Sep 07, 2026

Before judging whether an algorithm is fair, specify what you want the judgement to mean. Equal outcomes, equal error rates and comparable probability estimates are different requirements.

Sahil Verma and Julia Rubin's 2018 paper, Fairness Definitions Explained, illustrates these distinctions using a logistic-regression classifier and the German Credit Dataset: 1,000 applicants described by 20 attributes. This is a structured credit-data example, not sentiment analysis of applicant descriptions or a study of students.

Five distinctions that matter

Their example meets some criteria and fails others. A claim of fairness therefore depends on the selected definition. The full paper explains additional measures and their assumptions.

Applying the distinction in education

Our editorial recommendation is to write down the decision being evaluated before selecting a metric. If a system flags comments for review, distinguish the proportion flagged from the proportion incorrectly flagged. Ask what a missed case means and what happens after a flag.

Then discuss whose experience the evaluation should represent, how the comparison groups are defined and whether the evaluation material is suitable for that purpose. Those choices need their own justification. A mathematical measure cannot establish that the chosen task or decision process is appropriate, and this credit example does not validate an educational application.

Correction, 7 September 2026: The previous summary confused conditional use accuracy with error-rate criteria and omitted the score condition from calibration. It also described a text-analysis application that was not part of the dataset or study. These errors and the associated FAQ claims have been removed.