2.3 Extracting information from dataFrom numbers to knowledge

Information is the collection of facts and patterns extracted from data. Data provide opportunities for identifying trends, making connections and addressing problems. A bike-share company with a year of trip records can see which stations run out of bikes at 8 a.m., when demand peaks, and where to add new stations.
Often a single source doesn't contain the data needed to draw a conclusion, so it may be necessary to combine data from a variety of sources. Trip data plus weather data can show how rain changes riding. Trip data plus repair records can show which bikes are busy and overdue for service.
| Bike ID | Trips this week (app data) | Last repair (shop data) | Combined insight |
|---|---|---|---|
| B-104 | 41 | 3 weeks ago | Busy and recently fixed |
| B-221 | 38 | 8 months ago | Busy and overdue: repair first |
| B-317 | 4 | 1 week ago | Rarely used: move it to a busier station |
Metadata
Metadata are data about data. For a photo, the data is the pixels, and the metadata is when and where it was taken, the camera, and the file size. Metadata are used for finding, organizing and managing information: your phone can show every photo from a trip because each one carries a location and a date. And changes to the metadata don't change the primary data: renaming a song or fixing its genre tag doesn't change a single sound sample.
| Part of a song file | Data or metadata? |
|---|---|
| The sound samples you hear | Data: the song itself |
| Title and artist | Metadata |
| Length (3:42) | Metadata |
| Date you added it | Metadata |
| Genre tag | Metadata |
?Wait, is a file's name data or metadata?
Correlation is not causation
Digitally processed data may show a correlation between variables: when one goes up, the other tends to go up (or down) too. A correlation found in data does not necessarily mean that a causal relationship exists, that one thing makes the other happen. Additional research is needed to understand the exact nature of the relationship.
Of course not. Rain causes both: people buy umbrellas, and wet roads cause crashes. A hidden third factor like this is the most common reason a correlation isn't a cause. Sometimes the cause runs the other way, and sometimes a correlation is a coincidence. Big data sets make it easy to find patterns; they don't make it easy to explain them.
?Wait, what would it take to show a cause?
I thought about it. Show me
- (a) Data: the arrival time of bus 12 at Elm Street. Metadata: the upload time and device for that record.
Why?
Metadata describe the records; they aren't the bus information itself. - (b) It can conclude that newer engines and on-time arrival are correlated in this data.
- It can't conclude the engines cause the on-time arrivals. A third factor might: perhaps newer buses are assigned to shorter, less crowded routes.
Why?
Correlation does not imply causation. Name a possible third factor to show you understand why. - To learn more, the city could combine sources (route length, traffic data) or try newer engines on the slow routes and compare.
Answer: (a) Data: an arrival time. Metadata: when and by which device a record was uploaded. (b) Newer engines and punctuality are correlated, but that doesn't prove the engines cause it; route differences could explain it.
Stuck? Get a hint
- Is the item about the buses, or about the records of the buses?
- Could something else make newer buses arrive on time?
- Correlation is what the data shows. Causation needs more investigation.
- AChanging the genre changes the audio data in the file.
- BThe genre is metadata, so changing it does not change the song's audio.
- CThe genre is the primary data, so the song must be re-recorded.
- DMetadata cannot be changed once a file is created.
Show the answer
B. The genre describes the song, so it is metadata, and changes to metadata don't change the primary data, the sound samples.
- Information is the facts and patterns extracted from data. Combining sources often reveals more.
- Metadata are data about data. Used to find, organize and manage data; changing them doesn't change the data.
- A correlation in data does not necessarily mean causation. More research is needed.
- Look for a hidden third factor before you trust a cause-and-effect claim.