AP Computer Science Principles · Unit 5 lecture · Part 4 of 11 · 9 min

5.3 Computing biasBiased data in, biased results out

Mr. Thompkins
People often say a computer can't be biased because it's "just math". Let me show you why that's wrong, and where the bias actually comes from. Spoiler: it comes from us.

Computing innovations can reflect existing human biases because of biases written into the algorithms or biases in the data used by the innovation. A program that learns from data learns whatever patterns are in that data, unfair ones included. The program never decides to be unfair. It just becomes very good at copying what it was shown.

?Wait, what's bias in data, from Unit 2?
In Unit 2 you saw that data can be biased if it doesn't represent the whole group it's supposed to describe, for example a survey that only reaches people who are online. Collecting more data of the same kind doesn't fix that. Only more representative data does.

Bias can be embedded at all levels of software development: which data is collected and from whom, which features the program pays attention to, what goal it's trained to hit, and who it's tested on before it's released. The CED names three areas where real cases have appeared: criminal risk assessment tools, facial recognition systems and job applicant screening.

collect datapick featurestraintestusewho is missing?stand-in clues?copies the pasttested on whom?used on whom?bias can enter at every stagethe program never decides to be unfairit learns what the data and design give it
Five stages of building a program that learns from data, and the bias question to ask at each one.

Here's a real example of a stand-in clue. A hospital program studied in 2019 tried to predict which patients needed extra care, and it used how much money had been spent on each patient's care in the past as its measure of need. But for patients who were equally sick, less had historically been spent on Black patients, so the program rated them as healthier than they were. Nobody typed race into the program. The bias came in through the choice of what to measure.

Share of training photos (%)70Group A25Group B5Group CErrors per 1,000 tests12Group A35Group B80Group C
Invented numbers for a face check-in system. Group C was only 5% of the training photos and gets almost 7 times as many errors as group A.
?Wait, what's a percent of a group again?
Divide the part by the whole, then multiply by 100. 35 errors out of 1,000 tests is 35 ÷ 1,000 = 0.035, which is 3.5%. Comparing rates instead of raw counts matters when groups are different sizes.

What should programmers do? Programmers should take action to reduce bias in algorithms used for computing innovations as a way of combating existing human biases. That means collecting data that represents everyone the program will be used on, testing results separately for each group, questioning whether past decisions are the right thing to copy, and keeping people involved in high-stakes decisions. Beyond the course, as a bonus: a 2019 study by the U.S. National Institute of Standards and Technology tested many face recognition programs and found that most of them made more errors for some demographic groups than for others.

Where the bias came fromWhat programmers can do
Some groups missing or rare in the training dataCollect data that represents everyone the program will be used on
Data that records past unfair decisionsQuestion whether the past is the right target to copy
A stand-in clue (zip code, club name, spending) that tracks a groupTest outcomes by group; removing one input is not enough
Testing only on people like the developersTest on diverse users before release, and keep testing after
Nobody checking the decisionsKeep humans reviewing high-stakes decisions
Where bias comes from, and what programmers can do about each source.
Pause and think
A student says, "Just delete the column that says which group each person belongs to, and the program can't be biased." Predict whether that works, and why.
I thought about it. Show me
Usually not. Other inputs can act as stand-ins for the deleted one: a zip code, a school name, a club, a spending history. The program can still learn the same pattern through them. You find out by testing the outcomes for each group, not by trusting that one column is gone.
Analyze the data
A company tests its face check-in system on 1,000 photos from each of three groups. It gets 12 errors for group A, 35 for group B and 80 for group C. The training photos were 70% group A, 25% group B and 5% group C. (a) Find each group's error rate. (b) Identify the most likely cause of the difference. (c) Describe two specific steps to reduce it.
  1. (a) Error rate = errors ÷ tests. A: 12 ÷ 1,000 = 1.2%. B: 35 ÷ 1,000 = 3.5%. C: 80 ÷ 1,000 = 8.0%.
  2. Compare: 8.0 ÷ 1.2 ≈ 6.7, so group C sees almost 7 times as many errors as group A.
    Why?
    Putting a number on the gap makes the argument concrete. On a written answer, specific beats vague.
  3. (b) The groups with the fewest training photos have the most errors. The system learned faces mostly from group A, so it is worse at the groups it saw least: biased training data.
  4. (c) Step 1: collect more training photos of groups B and C, until the data represents everyone who will use the system.
  5. Step 2: before release, require error rates to be close for every group, and keep testing by group after launch. A person can also check any failed check-in rather than locking someone out.
    Why?
    A good fix addresses the cause (the data) and checks the result (testing by group). "Use a better computer" fixes neither.

Answer: (a) 1.2%, 3.5% and 8.0%. (b) Groups B and C were underrepresented in the training data. (c) Collect representative data, and test and compare error rates by group before and after release.

Stuck? Get a hint
  1. For (a), divide each group's errors by 1,000 and turn it into a percent.
  2. For (b), compare the error rates with each group's share of the training photos.
  3. For (c), one step should fix the data; another should check the results.
Check yourself
A job screening program was trained on ten years of a company's past hiring decisions, and it now ranks some groups of qualified applicants lower. What is the most likely source of the bias?
  1. AThe computer running the program is too slow
  2. BThe program decided on its own to prefer some applicants
  3. CThe program learned patterns from past decisions that were themselves biased
  4. DAlgorithms cannot be biased, so the applicants must be less qualified
Show the answer

C. The program copies the patterns in its training data, and the past hiring decisions carried bias. The second choice is the trap: a program does not "decide" to be unfair. The fourth repeats the myth that algorithms are neutral math.

  • Innovations can reflect human biases through biased data or biased design.
  • Bias can enter at every stage: data, features, goals, testing and use.
  • Real cases: criminal risk assessment, facial recognition, job applicant screening.
  • Programmers should act to reduce bias: representative data, testing by group, human review.