Theory
The average that was confidently wrong
First pass on the raw survey: mean screen time 171 minutes. Ready for the dean?
Look closer. Roll 117 appears twice (a form scanned two times). One city reads " Surat" with a leading space, another "surat": R counts three Surats as three cities. And one screen-time entry says 1620 minutes: twenty-seven hours a day.
Every statistic you have learned will happily average garbage. Cleaning is the unglamorous step that makes the glamorous steps true.
Theory
Washing before cooking
No cook throws unwashed vegetables into the pot because "the recipe is the skilled part".
Data cleaning is the washing: remove the doubles (stones), standardise the labels (peel), question the impossible (the rotten one). The recipe (mean, SD, charts) only deserves trust if the washing happened: and professionals spend MORE time washing than cooking. The checklist has four stations.
Practical
The four-station cleaning line
survey <- read.csv("survey.csv")
# STATION 1: duplicates
sum(duplicated(survey)) # how many repeat rows? e.g. [1] 1
survey <- survey[!duplicated(survey), ] # keep first copies only
# STATION 2: inconsistent text
table(survey$city) # reveals " Surat", "surat", "Surat"...
survey$city <- trimws(survey$city) # strip stray spaces
survey$city <- tolower(survey$city) # one consistent case
table(survey$city) # confirm: one spelling per city
# STATION 3: impossible values (range check)
survey[survey$screen < 0 | survey$screen > 1440, ]
# roll 121: screen = 1620 (more minutes than a day has!)
survey$screen[survey$screen > 1440] <- NA # unknown, not zero
# STATION 4: transformation (derived columns, last lesson)
survey$hours <- survey$screen / 60
# save as a NEW versioned file: never overwrite the raw original
write.csv(survey, "survey_clean.csv", row.names = FALSE)
Theory
The two discovery tools
You cannot fix what you have not seen. Two commands expose ninety percent of dirt:
- table(df$city): every distinct value with its count: misspellings, stray spaces and case chaos line up for inspection. Run it before AND after fixing: the after-table is your proof.
- summary(df): min and max per numeric column: a minimum of -20 or a maximum of 1620 screams data entry error.
Habit: table() every categorical column, summary() every numeric one, before any statistics leave your desk.
Quiz
duplicated(survey) returns FALSE FALSE TRUE FALSE for four rows. What does the TRUE mean, and what does survey[!duplicated(survey), ] keep?
- Row 3 repeats an earlier row; the ! keeps rows 1, 2 and 4: first occurrences survive, repeats drop
- Row 3 is the original and must be deleted
- Rows 1, 2 and 4 are duplicates of row 3
- The command deletes all four rows to be safe
Show the answer
Row 3 repeats an earlier row; the ! keeps rows 1, 2 and 4: first occurrences survive, repeats drop
duplicated() marks TRUE from the second appearance onward: the first copy stays FALSE, so negating with ! keeps exactly one copy of everything. Option B inverts the convention (the original is never flagged): the distinction matters because keeping-the-first is what makes the operation safe. Counting first with sum(duplicated(...)) tells you the size of the problem before you fix it.
Think first
Judge the two outliers
Two suspicious screen times survive the range check debate: roll 121 with 1620 minutes, and roll 108 with 600 minutes. Before tapping: which one gets corrected/NA-ed, which one stays, and what is the principle?
Show the answer
1620 is impossible: a day has 1440 minutes: this is a data-entry error (likely 162 or 620): fix it if the original form exists, else set NA.
600 stays: ten hours is extreme but possible (the binge-watcher from the central-tendency lesson was real).
The principle: clean errors, keep outliers. Impossible = remove/repair; improbable-but-real = retain and let robust statistics (the median!) handle it. Deleting inconvenient real values is not cleaning: it is fabrication.
Watch out
The cleaner's code of conduct
Never overwrite the raw file: clean the memory copy, save as survey_clean.csv: anyone must be able to re-run your cleaning from the original (reproducibility).
Never delete real outliers because they spoil the mean: that is the median's job.
Prove your fixes: the before/after table() pair is the evidence an examiner (or auditor) wants.
Theory
Where the hours go in real jobs
Ask any data analyst: 60-80% of project time is cleaning, and every scandalous "study retracted" story hides a dirty dataset. You now hold the whole line: discover (table, summary), deduplicate, standardise, range-check, transform, version. Next lesson confronts the value we created today with survey$screen[...] <- NA: what NA really is, and how statistics computes around holes.
Summary
Key takeaways
- Statistics on dirty data is confidently wrong: clean first, compute second.
- duplicated() flags repeats (from the second copy); df[!duplicated(df), ] keeps firsts.
- trimws + tolower standardise text; table() before/after is discovery and proof.
- Range checks catch impossible values (screen > 1440): fix or set NA, never silently keep.
- Clean errors, keep real outliers: deletion of true extremes is fabrication.
- Clean the memory copy; save as a NEW versioned file: the raw original is sacred.
- Memory hook: wash the vegetables before cooking.