2.3 Data challenges: cleaning, bias and scaleMessy, biased and huge

Data sets pose challenges regardless of their size. Data can be incomplete (values missing), invalid (values that can't be right, like a negative number of minutes), or inconsistent (the same thing written different ways). Combining data from different sources adds more chances for mismatches.
Cleaning data is a process that makes the data uniform without changing its meaning. For example, replacing "ninth", "9th" and "Gr 9" with the single value 9, so a program counting ninth graders finds all of them. Cleaning can also mean removing exact duplicates or setting invalid entries aside to check them. What cleaning must never do is change what the data says.
| Student | Grade | Minutes reading | Problem |
|---|---|---|---|
| A01 | 9 | 30 | none |
| A02 | ninth | 45 | inconsistent: should be 9 |
| A03 | 10 | incomplete: value missing | |
| A04 | 11 | -20 | invalid: minutes cannot be negative |
| A01 | 9 | 30 | duplicate of the first row |
| A05 | 12 | 600 | possible, but check: 10 hours a day? |
?Wait, what makes a value invalid rather than just unusual?
Bias in data
Problems of bias are often caused by the type or source of the data being collected. If a survey about school lunches is only posted on a website that students check from home computers, students without home internet are left out, and the results lean toward one group. Here's the part that surprises people: bias is not eliminated by simply collecting more data. Ten times as many responses through the same website are still missing the same students.
?Wait, what's a sample?
Scale
The size of a data set affects the amount of information that can be extracted from it, and how it has to be processed. A class survey fits in a spreadsheet. A billion trip records don't. The ability to process data depends on the capabilities of the users and their tools, and large data sets may require parallel systems: many computers working on parts of the data at the same time (you'll meet parallel computing in Unit 4). Being able to keep working as the data grows is called scalability.
I thought about it. Show me
- A02: change "ninth" to 9, so every grade uses the same format.
Why?
Making data uniform without changing meaning is exactly what cleaning is. - A03: the missing value is incomplete data. Leave it out of averages (don't invent a number), or follow up with the student.
- A04: -20 minutes is invalid. Set it aside or ask, but don't include it in calculations.
- The repeated A01 row: same ID, same answers, so it's a duplicate. Remove one copy so that student isn't counted twice.
Why?
Duplicates quietly double-count people and skew totals. - (b) No. Students who spend lunch in the library are probably more likely to read than students who don't, so the sample is biased. Cleaning fixes errors in the values; it doesn't fix who was asked.
Why?
Bias comes from the source of the data, and neither cleaning nor more data from the same source removes it.
Answer: (a) Make "ninth" a 9, leave the missing value out of averages, set aside the negative value, remove the duplicate row. (b) No: the sample only includes library students at lunch, so it is biased toward readers, and cleaning can't fix that.
Stuck? Get a hint
- For each row, is the problem the format, a missing value, an impossible value, or a repeat?
- Cleaning should never change what the data means.
- For (b), who was never asked?
- ADeleting every row that mentions Texas
- BChanging all three versions to the same value, "TX"
- CCollecting more responses from Texas
- DAdding a new column with each person's email address
Show the answer
B. Cleaning makes data uniform without changing its meaning, so all three versions become "TX". Deleting valid rows changes the results, and adding emails is collecting more (and more personal) data, not cleaning.
- AWith 5,000 responses, the results must represent the whole town.
- BThe results may be biased toward dog owners, and more responses from the same page would not fix that.
- CThe data must be cleaned before it can be biased.
- DBias only happens in small data sets.
Show the answer
B. The source of the data, a page followed mostly by dog owners, causes the bias. Collecting more data the same way repeats the same bias. Large size is the trap in the first choice.
- Data can be incomplete, invalid or inconsistent. Cleaning makes it uniform without changing its meaning.
- Bias often comes from the type or source of the data. Collecting more data does not remove it.
- Large data sets may need more powerful tools or parallel systems. Scalability is handling growth.
- Clean first, then analyze, then question who the data leaves out.