Fair for everyone?

The idea

Biased data leads to unfair results, so fairness means checking the results for each group and acting on what the checks show.

Is 90 out of 100 enough?

A school app predicts which students need extra help in maths. It is right for 90 students out of every 100. Is that enough to call it fair?

• Yes, 90 out of 100 is a very good score
• Not yet. I would also want to see how it does for each group
• No, any machine that makes mistakes is unfair

Fair or unfair?

Each card describes what a school app does. Decide whether the result is fair or unfair for the students it affects.

Some cards are tricky. Tap a group. Read the reason, then press Next card.

Check every group

Look at the quiz-score card. A score of 94 out of 100 hid a group the app served badly. In the unfair cards, the cause was in the data: a group was thin, or old patterns were copied.

Fair does not mean perfect. It means no group gets clearly worse results, and people keep checking.

Biased data leads to unfair results, so fairness means checking the results for each group, not only the overall score.

Teams call this a bias check.

Notes

Biased data leads to unfair results, so fairness means checking the results for each group, not only the overall score.

Teams can add varied data, test every group, and keep a person in charge. Anyone can tell a trusted adult about a result that looks unfair.

Check yourself

A bus-route app often suggests poor routes to people who live at the edge of town. Its data came mostly from trips in the town centre. What is the likely cause?

An app is tested on 180 town students and 20 village students. It is right for 162 town students and 8 village students. Overall, how many out of every 100 students does it get right?

Answer: 85 out of 100

162 + 8 = 170 right out of 200 students, which is 85 out of every 100. That looks good. But the village group got only 40 out of 100. The overall score hid the gap.

A team finds that its app works worse for one group. What is the best next step?

Joseph thinks a school app gave his friend an unfair result. What is a good thing for him to do?

• Few trips from the edge of town were in its data — correct. Yes. A machine learns from the examples it was given. Few examples from a place means weaker results for the people there.
• People at the edge of town choose their routes badly. Their choices are fine. An app should work for the people who use it, wherever they live.
• Routes at the edge of town are simply harder to plan. Those routes are not harder in themselves. The app learned mostly from town-centre trips, so it has little to learn from for the edge.
• Leave it alone, because the overall score is high. A high average can hide a group that is treated badly. That group still gets worse results.
• Drop that group from the tests so the problem disappears. Hiding a problem does not remove it. The group will still get worse results once the app is in use.
• Add varied data, re-test every group, and keep people checking — correct. Yes. Fix the data, check again for every group, and keep people in charge. That is how a team turns a bias check into a fix.
• Accept it, since the app was tested before the school bought it. A test before buying does not prove the app is fair to every group. Results can still be unfair, and you are allowed to ask questions.
• Tell a teacher, and show exactly what happened — correct. Yes. A clear example helps people check the result for that group and fix it. Telling a trusted adult is the right first step.
• Post it online to embarrass the app's makers. Posting can spread anger, but it does not help anyone review the result. Speak to someone who can look into it.