5.3 Computing biasBiased data in, biased results out

Computing innovations can reflect existing human biases because of biases written into the algorithms or biases in the data used by the innovation. A program that learns from data learns whatever patterns are in that data, unfair ones included. The program never decides to be unfair. It just becomes very good at copying what it was shown.
?Wait, what's bias in data, from Unit 2?
Bias can be embedded at all levels of software development: which data is collected and from whom, which features the program pays attention to, what goal it's trained to hit, and who it's tested on before it's released. The CED names three areas where real cases have appeared: criminal risk assessment tools, facial recognition systems and job applicant screening.
Here's a real example of a stand-in clue. A hospital program studied in 2019 tried to predict which patients needed extra care, and it used how much money had been spent on each patient's care in the past as its measure of need. But for patients who were equally sick, less had historically been spent on Black patients, so the program rated them as healthier than they were. Nobody typed race into the program. The bias came in through the choice of what to measure.
?Wait, what's a percent of a group again?
What should programmers do? Programmers should take action to reduce bias in algorithms used for computing innovations as a way of combating existing human biases. That means collecting data that represents everyone the program will be used on, testing results separately for each group, questioning whether past decisions are the right thing to copy, and keeping people involved in high-stakes decisions. Beyond the course, as a bonus: a 2019 study by the U.S. National Institute of Standards and Technology tested many face recognition programs and found that most of them made more errors for some demographic groups than for others.
| Where the bias came from | What programmers can do |
|---|---|
| Some groups missing or rare in the training data | Collect data that represents everyone the program will be used on |
| Data that records past unfair decisions | Question whether the past is the right target to copy |
| A stand-in clue (zip code, club name, spending) that tracks a group | Test outcomes by group; removing one input is not enough |
| Testing only on people like the developers | Test on diverse users before release, and keep testing after |
| Nobody checking the decisions | Keep humans reviewing high-stakes decisions |
I thought about it. Show me
- (a) Error rate = errors ÷ tests. A: 12 ÷ 1,000 = 1.2%. B: 35 ÷ 1,000 = 3.5%. C: 80 ÷ 1,000 = 8.0%.
- Compare: 8.0 ÷ 1.2 ≈ 6.7, so group C sees almost 7 times as many errors as group A.
Why?
Putting a number on the gap makes the argument concrete. On a written answer, specific beats vague. - (b) The groups with the fewest training photos have the most errors. The system learned faces mostly from group A, so it is worse at the groups it saw least: biased training data.
- (c) Step 1: collect more training photos of groups B and C, until the data represents everyone who will use the system.
- Step 2: before release, require error rates to be close for every group, and keep testing by group after launch. A person can also check any failed check-in rather than locking someone out.
Why?
A good fix addresses the cause (the data) and checks the result (testing by group). "Use a better computer" fixes neither.
Answer: (a) 1.2%, 3.5% and 8.0%. (b) Groups B and C were underrepresented in the training data. (c) Collect representative data, and test and compare error rates by group before and after release.
Stuck? Get a hint
- For (a), divide each group's errors by 1,000 and turn it into a percent.
- For (b), compare the error rates with each group's share of the training photos.
- For (c), one step should fix the data; another should check the results.
- AThe computer running the program is too slow
- BThe program decided on its own to prefer some applicants
- CThe program learned patterns from past decisions that were themselves biased
- DAlgorithms cannot be biased, so the applicants must be less qualified
Show the answer
C. The program copies the patterns in its training data, and the past hiring decisions carried bias. The second choice is the trap: a program does not "decide" to be unfair. The fourth repeats the myth that algorithms are neutral math.
- Innovations can reflect human biases through biased data or biased design.
- Bias can enter at every stage: data, features, goals, testing and use.
- Real cases: criminal risk assessment, facial recognition, job applicant screening.
- Programmers should act to reduce bias: representative data, testing by group, human review.