Algorithm Fairness Case Study

A Study on COMPAS Recidivism Dataset from ProPublica

In reality, calculating FPR and FNR requires knowing the probabilities of classifier outputs conditioned on the eventual outcomes. In other words, it requires knowing things like “what fraction of loan applicants was denied among those who would have repaid if approved?” Naturally, such possibilities are hard to obtain empirically, and in general may require randomized trials, while no such experiments were performed in the COMPAS analysis. Chouldechova represents the positive class with a risk score S, asserting that a score S that is greater than some threshold sHR would be interchangeable with a positive prediction.

Chouldechova points out that a score that is calibrated may not have predictive parity, illustrated by the image below, where the distributions of risk scores for the majority group deviates from that for the minority group.

In addition to the individual and group fairness criteria studied in the four papers, another interesting discussion covers the ways to interpret the classifiers to ensure fairness since machine learning algorithms often operate in a black box fashion without much information being publicly available. Chouldechova and Hardt provided two ways to further improve the classifiers. Chouldechova proposes to: “1). allow unequal false negative rates to retain equal PPVs and achieve equal false positive rates, 2). Allow unequal false positive rates to retain equal PPVs and achieve equal false negative rates, 3). Allow unequal PPVs to achieve equal false positive and false negative rates” (Chouldechova). Chouldechova focused on the broader application: having a context-specific framework that is more flexible and intuitive to understand. Chouldechova believes that acting as a better benchmark at eliminating potential discriminations, the error rate should be more emphasized when it comes to practice life decision. This way ensures that no specific group is having significantly higher FPR, or being favored. Hardt, on the other hand, the notion of Oblivious measures, which singles out the correlation between the predictor and the joint distribution.

Despite the fact that race as group membership isn’t an input to the scoring function, COMPAS score turned out to be well calibrated between groups. Considering calibration requires context, this is not such a surprise. In fact, Lab 5 has let us explore different scenarios of data manipulation through calibration to authenticate data that is merely seemingly fair. One plausible solution to ensure the scores are calibrated is calibration by the group, implementing different metrics within each group when necessary.

Related posts:

A family of ducks lived near a lake. These ducks did not yet live at the lake. Instead, they kept a nest in a tree, since the ducklings were still young. Now, the day came that the mother deemed it…

When it comes to marketing on the various social media platforms, everyone wants in on the action. Social media marketing, on the surface, seems like a relatively simple concept. Just create your…

What type of cars have low insurance costs for young drivers and also which insurance providers have the best deals for new drivers? Thanks in advance! ANSWER: I would recommend that you try this web…