Fairness Measures

Datasets and software for detecting algorithmic discrimination







Code (on GitHub) 🔗




We provide software implementing the following measures, as found in the literature on algorithmic discrimination. You can find the software in our GitHub repository.
In the following, positive outcome is considered as the desired one for a certain candidate in a group, e.g. being granted a loan given a certain credit score (target score). We are continuously expanding the list of available measures.

Absolute Measures

Mean Difference

The difference between the target score mean values of the protected vs. unprotected groups. No difference = no discrimination

Normalized Difference

The mean difference for binary classification normalized by the rate of positive outcomes. At a given positive outcome rate, maximum possible discrimination is taken into account

Impact Ratio

The ratio of positive outcomes for the protected group over the non-protected group.

Odds Ratio

The association between exposure and outcome

Statistical Tests

Difference of Means a.k.a. Welch-Test

For a null hypothesis that the means of the two groups (protected and non-protected) are equal.

Difference in proportions for two groups a.k.a. Fisher’s exact test

For a null hypothesis that the rates of positive outcomes within the two groups are equal.


Currently we implement measures described in the following references:

Žliobaitė, Indrė. “Measuring discrimination in algorithmic decision making” Data Mining and Knowledge Discovery 31, no. 4 (July 31, 2017): 1060-089. doi:10.1007/s10618-017-0506-1. bibtex