AP Computer Science Principles · Unit 2 lecture · Part 9 of 11 · 8 min

2.3 Data challenges: cleaning, bias and scaleMessy, biased and huge

Mr. Thompkins
Real data is messy. People skip questions, type "ninth" instead of 9, and enter the same form twice. If you analyze messy data, you get confident wrong answers. So before the analysis, there's the cleanup.

Data sets pose challenges regardless of their size. Data can be incomplete (values missing), invalid (values that can't be right, like a negative number of minutes), or inconsistent (the same thing written different ways). Combining data from different sources adds more chances for mismatches.

Cleaning data is a process that makes the data uniform without changing its meaning. For example, replacing "ninth", "9th" and "Gr 9" with the single value 9, so a program counting ninth graders finds all of them. Cleaning can also mean removing exact duplicates or setting invalid entries aside to check them. What cleaning must never do is change what the data says.

StudentGradeMinutes readingProblem
A01930none
A02ninth45inconsistent: should be 9
A0310incomplete: value missing
A0411-20invalid: minutes cannot be negative
A01930duplicate of the first row
A0512600possible, but check: 10 hours a day?
A survey before cleaning. Each problem needs a different fix, and the last row needs a human check, not an automatic change.
?Wait, what makes a value invalid rather than just unusual?
An invalid value is impossible given what the data means: negative minutes, a 13th month, an age of 400. An unusual value is possible but surprising, like 600 minutes of reading in a day. Invalid values must be fixed or removed; unusual ones should be checked before anyone changes them.

Bias in data

Problems of bias are often caused by the type or source of the data being collected. If a survey about school lunches is only posted on a website that students check from home computers, students without home internet are left out, and the results lean toward one group. Here's the part that surprises people: bias is not eliminated by simply collecting more data. Ten times as many responses through the same website are still missing the same students.

?Wait, what's a sample?
A sample is the group you actually collect data from, used to learn about a bigger group, the population. A sample is biased if the way it was chosen leaves out or over-counts some part of the population.

Scale

The size of a data set affects the amount of information that can be extracted from it, and how it has to be processed. A class survey fits in a spreadsheet. A billion trip records don't. The ability to process data depends on the capabilities of the users and their tools, and large data sets may require parallel systems: many computers working on parts of the data at the same time (you'll meet parallel computing in Unit 4). Being able to keep working as the data grows is called scalability.

Pause and think
A fitness app wants to know how much exercise teenagers get. It collects data only from teens who wear its watch. After a year it has 2 million teens in its data. Is the data unbiased now?
I thought about it. Show me
No. It's still only teens who own and wear that watch, who may be more active, or wealthier, than teens in general. A bigger sample from the same biased source is still biased. Size can't fix how the data was collected.
Clean it, then judge it
Look at the survey table above. (a) Describe how to clean each of the first five problem rows. (b) The survey was given only to students in the library during lunch. Explain whether the cleaned data can tell you how much all students read.
  1. A02: change "ninth" to 9, so every grade uses the same format.
    Why?
    Making data uniform without changing meaning is exactly what cleaning is.
  2. A03: the missing value is incomplete data. Leave it out of averages (don't invent a number), or follow up with the student.
  3. A04: -20 minutes is invalid. Set it aside or ask, but don't include it in calculations.
  4. The repeated A01 row: same ID, same answers, so it's a duplicate. Remove one copy so that student isn't counted twice.
    Why?
    Duplicates quietly double-count people and skew totals.
  5. (b) No. Students who spend lunch in the library are probably more likely to read than students who don't, so the sample is biased. Cleaning fixes errors in the values; it doesn't fix who was asked.
    Why?
    Bias comes from the source of the data, and neither cleaning nor more data from the same source removes it.

Answer: (a) Make "ninth" a 9, leave the missing value out of averages, set aside the negative value, remove the duplicate row. (b) No: the sample only includes library students at lunch, so it is biased toward readers, and cleaning can't fix that.

Stuck? Get a hint
  1. For each row, is the problem the format, a missing value, an impossible value, or a repeat?
  2. Cleaning should never change what the data means.
  3. For (b), who was never asked?
Check yourself
A spreadsheet of state names contains "TX", "Tex." and "texas". Which action is an example of cleaning the data?
  1. ADeleting every row that mentions Texas
  2. BChanging all three versions to the same value, "TX"
  3. CCollecting more responses from Texas
  4. DAdding a new column with each person's email address
Show the answer

B. Cleaning makes data uniform without changing its meaning, so all three versions become "TX". Deleting valid rows changes the results, and adding emails is collecting more (and more personal) data, not cleaning.

Check yourself
An online poll about a town's new park is shared only on a social media page followed mostly by dog owners. The town gets 5,000 responses. Which statement is most accurate?
  1. AWith 5,000 responses, the results must represent the whole town.
  2. BThe results may be biased toward dog owners, and more responses from the same page would not fix that.
  3. CThe data must be cleaned before it can be biased.
  4. DBias only happens in small data sets.
Show the answer

B. The source of the data, a page followed mostly by dog owners, causes the bias. Collecting more data the same way repeats the same bias. Large size is the trap in the first choice.

  • Data can be incomplete, invalid or inconsistent. Cleaning makes it uniform without changing its meaning.
  • Bias often comes from the type or source of the data. Collecting more data does not remove it.
  • Large data sets may need more powerful tools or parallel systems. Scalability is handling growth.
  • Clean first, then analyze, then question who the data leaves out.