Session 1: R Basics, Importing, and Inspecting Data

Objects, functions, packages, and project folders, then your first dataset

NoteIn this session you will learn to
  1. Create objects, use vectors, and call functions
  2. Install packages (once) and load them (every session)
  3. Open the workshop’s project folder and explain why here() makes file paths work on any computer
  4. Import a data file with import() and here()
  5. Inspect a dataset and write down a list of the problems that need cleaning

Before you start: complete the Setup and open student_scripts/Session 1 - R Basics, Importing, and Inspecting_STUDENT.R.

Where this session fits. Most data projects go through the same steps, whatever the topic. Today covers the first two, Import and Inspect (highlighted): getting the data into R, and looking at it carefully before changing anything. You could do these steps in Excel or SPSS too. What R adds is that every step is written down in a script, so you (or a colleague) can check it, rerun it on next month’s data, and share it.

  1. Import
  2. Inspect
  3. Clean
  4. Combine
  5. Reshape
  6. Summarize
  7. Visualize
  8. Save and share
  9. Automate

Part 1: R basics

Before we open any data, a few building blocks you’ll use in every line of R. Type these in your script and run each line with Cmd + Return (Mac) or Ctrl + Enter (Windows); the result appears in the Console.

Objects and the assignment arrow <-

<- stores a value in an object so you can use it later. Read it as “gets”. The object appears in the Variables pane.

my_age <- 35
my_age
[1] 35
my_age + 5
[1] 40
Note🔍 What this code does
  • my_age <- 35 creates an object called my_age and stores the number 35 in it. Nothing prints: saving is silent. my_age now appears in the Variables pane.
  • my_age on its own prints what is stored: [1] 35. The [1] means “this is the first value”. It helps you keep your place when the output runs over several lines.
  • my_age + 5 uses the object in a calculation and prints 40. my_age itself is still 35: to change it, you would save again with <-.

Vectors

c() combines values into a vector. Every value in a vector has the same type.

ages <- c(2, 18, 56, 61)
mean(ages)
[1] 34.25
Note🔍 What this code does
  • c(2, 18, 56, 61) combines four numbers into one vector, and ages <- saves it.
  • mean(ages) gives the whole vector to mean(), which adds the values and divides by how many there are: (2 + 18 + 56 + 61) / 4 = 34.25.
  • Most R functions work on a whole vector at once, so you rarely handle values one by one.
hospitals <- c("Central Hospital", "Military Hospital", "Other")
length(hospitals)
[1] 3
Note🔍 What this code does
  • The hospital names are text, so each one is in quotes. Without quotes, R would look for an object called Central and give an error.
  • length(hospitals) counts the values in the vector: 3. It counts values, not letters.

Functions and arguments

A function does something to its inputs (arguments): function_name(argument1, argument2, ...). Arguments can be named.

round(3.14159)
[1] 3
round(3.14159, digits = 2)
[1] 3.14
Note🔍 What this code does
  • round(3.14159) has one argument, the number. The digits argument wasn’t given, so round() uses its default, 0, and rounds to a whole number: 3.
  • round(3.14159, digits = 2) adds a second, named argument and gives 3.14.
  • Arguments you leave out use their default value. The help page (?round) lists every argument and its default.
TipGetting help

Type ?round in the Console to open the help page for any function.

Data frames

A data frame is a table: each column is a variable (a vector), each row is an observation. (Later you’ll also see tibbles: the tidyverse version of a data frame, which prints more neatly.)

patients <- data.frame(
  name = c("Ana", "Ben", "Cho"),
  age  = c(34, 51, 27)
)
patients
  name age
1  Ana  34
2  Ben  51
3  Cho  27
patients$age        # $ pulls out one column
[1] 34 51 27
Note🔍 What this code does
  • data.frame() builds a table from vectors. Each column_name = c(...) becomes one column: name holds text and age holds numbers.
  • Every column must have the same length (3 here), because each row is one patient.
  • patients$age pulls out the age column as a vector: 34 51 27. You can give it to any function, for example mean(patients$age).
WarningCommon mistakes
  • R is case-sensitive: Age and age are different
  • Text needs quotes ("Central Hospital"); object names do not (my_age)
  • Every ( needs a ) — Positron highlights the matching pair
  • # starts a comment; R ignores the rest of the line
Important✏️ TRY IT 1 (2 min): objects and functions

Store three temperatures, 38.5, 39.1, and 37.9, in an object called my_temps. Then find their mean().

✅ Check: the mean is 38.5

my_temps <- c(38.5, 39.1, 37.9)
mean(my_temps)
[1] 38.5
Note📖 Good to know
  • Script or Console? The Console runs a line once and forgets it. Your script is the record of what you did, so type your code in the script and run it from there.
  • Objects live in R’s memory, not on your disk. They disappear when you close Positron or restart R. That’s fine: rerun your script and they’re back. If you’re ever asked whether to save the workspace, choose No.
  • <- or =? Both can store a value, but this course always uses <-, so that = only ever means “name an argument” (as in digits = 2).
  • A vector holds one type. If you mix text into numbers, R turns everything into text: c(5, 12, "unknown") is three pieces of text. You’ll see this in the practice data, where a few ages say “unknown”.

Part 2: Installing and loading packages

A package is a collection of functions someone else wrote. There are two steps:

Step Function How often Analogy
Install install.packages() Once per computer Buying a book
Load library() Every time you open R Taking it off the shelf

Install everything for the whole workshop series (skip this if you did the Setup):

install.packages(c("tidyverse", "here", "rio", "janitor", "skimr", "gtsummary", "flextable", "scales"))

Load only what you need today:

library(tidyverse)   # data wrangling (dplyr) and plotting (ggplot2)
library(here)        # file paths that work on any computer
library(rio)         # import() and export() for almost any file type
library(skimr)       # skim() for a quick data summary
library(janitor)     # get_dupes() today; clean_names() in Session 2
Note🔍 What this code does
  • Each library() line makes one package’s functions available until you close Positron or restart R. That’s why every session starts with these lines.
  • library(tidyverse) loads a whole family of packages at once (dplyr, ggplot2, tidyr, readr, stringr, forcats, lubridate, tibble, and purrr) and lists them.
  • Its Conflicts message says that a few tidyverse functions share a name with a base R function, for example filter(). From now on, filter() means the tidyverse version. This is normal, not an error.
  • The text after # is a comment: a note for people. R ignores it.
WarningCommon mistakes
  • Package names need quotes in install.packages() but not in library().
  • Red text is not always an error! Messages like “Attaching core tidyverse packages” are normal. An error starts with Error.
Note📖 Good to know
  • Packages are shared on CRAN, R’s official package library, which has more than 20,000 of them. install.packages() downloads from there.
  • The tidyverse is a family of packages designed to work together, with the same style of function names and arguments. That’s why one library(tidyverse) loads several of them.

Part 3: Project folders and here()

Why keep everything in one project folder?

  • Organization: data, scripts, and outputs live in one folder with a predictable structure (raw_data/, processed_data/, student_scripts/)
  • Reproducibility: the project works on any computer — yours, a collaborator’s, future you’s
  • Shareable: zip the folder and send it, or put it on GitHub

Open the project folder for this class

  1. Download the workshop folder and save it somewhere you can find it (not your Downloads folder)
  2. In Positron, choose File → Open Folder…, select the cwml-r-workshop folder, and click Open. Positron uses that folder as the working directory
  3. Check the Explorer on the left: it should list the workshop files (raw_data, student_scripts, …)
TipYour own projects later

Make a new folder for each project (with subfolders like raw_data/ and scripts/), and open it in Positron with File → Open Folder…. Add an empty file called .here at the top so here() always finds it.

Relative vs absolute paths

  • Bad (absolute): "/Users/mwegener/Desktop/maybe_here/data.csv" — only works on one computer
  • Good (relative): here("raw_data", "data.csv") — works on any computer that has the project folder

here() finds the top of the project folder (it looks for a .here file, which the workshop folder has) and builds the path from there.

here::here()                                   # where is the top of my project?
Note🔍 What this code does
  • here::here() prints the full path to the top of the project folder: wherever you saved cwml-r-workshop on your computer. It finds the top by looking for the .here file.
  • If the path isn’t your workshop folder, the folder isn’t open in Positron. Choose File → Open Folder… and open it.
here::here("raw_data", "linelist_raw.xlsx")    # full path to a file inside it
Note🔍 What this code does
  • here::here("raw_data", "linelist_raw.xlsx") adds a folder and a file name to that path, joined with /. R uses / on every computer, Windows included.
  • It only builds the path, as text. It doesn’t check that the file exists or open it: that’s the job of import(), next.
  • Each piece is in quotes because it’s text, not the name of an object.

here::here() means “the here() function from the here package”. The :: lets you use a function before loading its package.

Important✏️ TRY IT 2 (1 min): a path to the practice data

Build the path to the practice data, raw_data/measles_raw.csv.

✅ Check: the output ends in .../raw_data/measles_raw.csv

here::here("raw_data", "measles_raw.csv")
Note📖 Good to know
  • The working directory is the folder R reads files from and saves files to. Opening the workshop folder in Positron sets it for you.
  • Older tutorials often start with setwd("C:/Users/..."). That path only exists on one computer, so the script breaks everywhere else. A project folder with here() avoids this.
  • here() finds the top of the project by looking for a marker: a .here file (the workshop folder has one), an .Rproj file, or a .git folder.

Part 4: Importing data

import() + here()

rio::import() reads almost any file type (.csv, .xlsx, .txt, SPSS, Stata, SAS, …). It works out the type from the file extension, so you only need to learn one function. Save the result with <-, or it only prints to the Console and is gone.

linelist_raw <- import(here("raw_data", "linelist_raw.xlsx"))
Tip🧩 Reading a function call

Here is the line you just ran, taken apart:

linelist_raw <- import(here("raw_data", "linelist_raw.xlsx"))

Piece What it is What it does
linelist_raw the name of a new object holds the result
<- the assignment arrow saves the result in linelist_raw
import(...) a function from the rio package reads a file into R
here(...) a function from the here package, used as the argument of import() builds the full path to the file
"raw_data", "linelist_raw.xlsx" two text arguments of here() the folder, then the file name

R runs it from the inside out: first here() builds the path, then import() reads the file at that path, and finally <- saves the result. Reading code aloud in the same order helps: “linelist_raw gets: import the file at here, raw_data, linelist_raw.xlsx.”

Note🔍 What this code does
  • import() saw the .xlsx ending and read the file as an Excel workbook (the first sheet, unless you ask for another).
  • Nothing prints. linelist_raw appears in the Variables pane with 6604 rows and 12 columns.
  • The result is a data frame: the same kind of table as patients in Part 1, only bigger. Each row is one patient (a case), and each column is one piece of information about them.

The same function works for other file types:

case_info <- import(here("raw_data", "case_info.csv"))   # comma-separated
hosp_info <- import(here("raw_data", "hosp_info.txt"))   # tab-separated text
Note🔍 What this code does
  • Same pattern, different file types: import() chooses how to read each file from its ending. .csv is comma-separated values; this .txt file is separated by tabs.
  • case_info has 5888 rows and 7 columns of extra details about each case. We join it to the linelist in Session 3.
  • hosp_info is a small look-up table (6 rows): the correct name and some facts about each hospital. We use it in Session 2 to fix the hospital names.
WarningCommon mistakes
  • File names must match exactly, including capitals and the extension.
  • “Error: No such file” means the folder or file name is misspelled, or the workshop folder isn’t open (check the Explorer on the left).
TipOther import options you may see
  • Base R: read.csv() — older, slower, and needs more settings
  • readr: read_csv() — the tidyverse version, fast and returns a tibble
  • haven: read_sav(), read_dta(), read_sas() — SPSS, Stata, SAS files
  • Excel files with several sheets: import(here("raw_data", "file.xlsx"), which = "Sheet2")
Important✏️ TRY IT 3 (2 min): import the practice data

Import the practice data raw_data/measles_raw.csv and call it measles_raw.

✅ Check: in the Variables pane, measles_raw shows 1215 rows and 11 columns

measles_raw <- import(here("raw_data", "measles_raw.csv"))
Note📖 Good to know
  • import() passes the work to a specialist package based on the file ending: readxl for Excel, data.table for CSV and text files, and haven for SPSS, Stata, and SAS. You only need to remember one function.
  • import() returns a plain data frame, which prints every row. That’s why this lesson wraps long output in head(). Tidyverse functions return a tibble, a data frame that prints only the first 10 rows.
  • Excel stores dates as date-times, so most date columns arrive as <dttm>. One column, date.onset, arrives as text: Excel files often hold surprises like this, which is why you inspect before you clean.

Part 5: Inspecting data

Always look before you clean. Inspecting tells you whether the data imported correctly and what needs fixing.

Size and structure

view(linelist_raw)       # opens the Data Explorer (or click the table icon next to it in the Variables pane)
dim(linelist_raw)        # rows, columns
[1] 6604   12
names(linelist_raw)      # column names
 [1] "Case ID"         "infection date"  "date.onset"      "hosp date"      
 [5] "date of outcome" "OUTCOME"         "Gender"          "hospital"       
 [9] "age"             "age_unit"        "wt_kg"           "ht_cm"          

Spaces, dots, capitals, and a mix of styles (Case ID, date.onset, OUTCOME). They are hard to type and need special handling. We fix all of them in Session 2 with one function.

head(linelist_raw)       # first 6 rows
  Case ID infection date date.onset  hosp date date of outcome OUTCOME Gender
1  5fe599     2014-05-08 2014-05-13 2014-05-15            <NA>    <NA>      m
2  8689b7           <NA> 2014-05-13 2014-05-14      2014-05-18 Recover      f
3  11f8ea           <NA> 2014-05-16 2014-05-18      2014-05-30 Recover      m
4  b8812a     2014-05-04 2014-05-18 2014-05-20            <NA>    <NA>      f
5  893f25     2014-05-18 2014-05-21 2014-05-22      2014-05-29 Recover      m
6  be99c8     2014-05-03 2014-05-22 2014-05-23      2014-05-24 Recover      f
                              hospital age age_unit wt_kg ht_cm
1                                Other   2    years    27    48
2                                 <NA>   3    years    25    59
3 St. Mark's Maternity Hospital (SMMH)  56    years    91   238
4              Regional Medical Center  18    years    41   135
5                    Military Hospital   3    years    36    71
6              Regional Medical Center  16    years    56   116
glimpse(linelist_raw)    # every column, its type, and the first few values
Rows: 6,604
Columns: 12
$ `Case ID`         <chr> "5fe599", "8689b7", "11f8ea", "b8812a", "893f25", "b…
$ `infection date`  <dttm> 2014-05-08, NA, NA, 2014-05-04, 2014-05-18, 2014-05…
$ date.onset        <chr> "2014-05-13", "2014-05-13", "2014-05-16", "2014-05-1…
$ `hosp date`       <dttm> 2014-05-15, 2014-05-14, 2014-05-18, 2014-05-20, 201…
$ `date of outcome` <dttm> NA, 2014-05-18, 2014-05-30, NA, 2014-05-29, 2014-05…
$ OUTCOME           <chr> NA, "Recover", "Recover", NA, "Recover", "Recover", …
$ Gender            <chr> "m", "f", "m", "f", "m", "f", "f", "f", "m", "f", "m…
$ hospital          <chr> "Other", NA, "St. Mark's Maternity Hospital (SMMH)",…
$ age               <chr> "2", "3", "56", "18", "3", "16", "16", "0", "61", "2…
$ age_unit          <chr> "years", "years", "years", "years", "years", "years"…
$ wt_kg             <dbl> 27, 25, 91, 41, 36, 56, 47, 0, 86, 69, 67, 84, 68, 4…
$ ht_cm             <dbl> 48, 59, 238, 135, 71, 116, 87, 11, 226, 174, 112, 18…

In glimpse(), the type is shown in angle brackets: <chr> text, <dbl> number, <date> date, <dttm> date-time.

Note🔍 What this code does
  • view() opens the data like a spreadsheet in the Data Explorer. You can sort and filter there to look around; that doesn’t change the data.
  • dim() gives the size: rows first, then columns (6604 12).
  • names() lists the 12 column names exactly as they’re spelled. Code has to use exactly this spelling.
  • head() shows the first 6 rows. head(linelist_raw, 10) shows 10.
  • glimpse() turns the table on its side: one line per column, with its type and the first few values. It’s the easiest way to see every column of a wide dataset.
  • date.onset is <chr> (text) but the other dates are <dttm> (date-time)
  • age is <chr> — it should be a number
  • There is an age_unit column — are all ages in years?
Important✏️ TRY IT 4 (3 min): size and structure

Use dim() and glimpse() on measles_raw.

  1. Which column should be a number but is stored as text (<chr>)?
  2. Which column names will need backticks?

✅ Check: dim() gives 1215 11

dim(measles_raw)
[1] 1215   11
glimpse(measles_raw)
Rows: 1,215
Columns: 11
$ `Patient ID`       <chr> "MS-0931", "MS-0113", "MS-0975", "MS-0297", "MS-015…
$ `Report Date`      <IDate> 2024-04-13, 2024-04-28, 2024-03-25, 2024-03-24, 2…
$ District           <chr> "West", "North", "Central", "Nroth", "South", "West…
$ SEX                <chr> "Female", "Male", NA, NA, "Female", "F", "Male", "M…
$ `Age (years)`      <chr> "4", "5", "7", "1", "10", "7", "4", "2", "0", "3", …
$ `Vaccinated?`      <chr> "No", "Yes", "No", "No", "No", "NO", "NO", "Unknown…
$ Fever              <chr> "yes", "yes", "yes", "yes", "yes", "yes", "yes", "y…
$ Rash               <chr> "yes", "no", "no", "no", "yes", "yes", "yes", "no",…
$ `Temp (C)`         <dbl> 38.3, 38.4, 39.3, 38.0, 38.9, 38.5, 38.7, 38.5, 39.…
$ `Days in Hospital` <int> NA, NA, NA, 4, NA, 5, 6, 5, 9, NA, 3, 1, 4, NA, NA,…
$ Outcome            <chr> "Died", "recovered", "Recovered", "Recovered", "Rec…
  1. Age (years) is <chr> — it contains “unknown” and the code “999”
  2. Names with spaces or symbols need backticks: `Patient ID`, `Report Date`, `Age (years)`, `Vaccinated?`, `Temp (C)`, `Days in Hospital`

Summary of every column

skim(linelist_raw)
Data summary
Name linelist_raw
Number of rows 6604
Number of columns 12
_______________________
Column type frequency:
character 7
numeric 2
POSIXct 3
________________________
Group variables None

Variable type: character

skim_variable n_missing complete_rate min max empty n_unique whitespace
Case ID 130 0.98 6 6 0 5888 0
date.onset 286 0.96 10 10 0 580 0
OUTCOME 1493 0.77 5 7 0 2 0
Gender 317 0.95 1 1 0 2 0
hospital 1509 0.77 5 36 0 13 0
age 100 0.98 1 2 0 75 0
age_unit 0 1.00 5 6 0 2 0

Variable type: numeric

skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100 hist
wt_kg 0 1 52.69 18.59 -11 41 54 66 111 ▁▃▇▅▁
ht_cm 0 1 125.25 49.57 4 91 130 159 295 ▂▅▇▂▁

Variable type: POSIXct

skim_variable n_missing complete_rate min max median n_unique
infection date 2315 0.65 2012-04-09 2015-04-27 2014-10-04 538
hosp date 0 1.00 2012-04-20 2015-04-30 2014-10-15 570
date of outcome 1061 0.84 2012-05-14 2015-06-04 2014-10-26 575

Look at the n_missing column, the minimum of wt_kg (below zero!), and the maximum of ht_cm (almost 3 meters!).

Note🔍 What this code does
  • skim() summarizes every column at once, grouped by type: text (character), numbers (numeric), and date-times (POSIXct).
  • n_missing is the number of missing values in each column; complete_rate is the share that is filled in.
  • For numbers, p0 is the smallest value, p50 the median, and p100 the largest. Here the smallest weight is -11 kg and the tallest height is 295 cm: both impossible.
  • For text, n_unique is the number of different values. A high number in a column that should have only a few categories is a sign of misspellings.

Categories: count()

count() shows every distinct value in a column and how often it appears. It’s the fastest way to find misspellings.

The data goes first, then the column: count(DATA, COLUMN).

count(linelist_raw, hospital)
                               hospital    n
1                       Central Hopital   11
2                      Central Hospital  457
3                            HMO Health  288
4                      Military Hopital   32
5                     Military Hospital  800
6                      Mitylira Hopital    1
7                     Mitylira Hospital   79
8                                 Other  907
9                        Premier Health  288
10                          Regional MC   48
11              Regional Medical Center 1756
12 St. Mark's Maternity Hospital (SMMH)  417
13   St. Marks Maternity Hopital (SMMH)   11
14                                 <NA> 1509
count(linelist_raw, Gender)
  Gender    n
1      f 3140
2      m 3147
3   <NA>  317
count(linelist_raw, OUTCOME)
  OUTCOME    n
1   Death 2898
2 Recover 2213
3    <NA> 1493
count(linelist_raw, age_unit)
  age_unit    n
1   months   37
2    years 6567
Note🔍 What this code does
  • count(linelist_raw, hospital) lists each different value in hospital and how many rows have it (n). Values are sorted alphabetically, with missing values (NA) last.
  • The hospital list shows the same hospital spelled several ways, for example “Central Hospital” (457 rows) and “Central Hopital” (11). R treats every spelling as a different hospital, so these have to be fixed before counting cases per hospital. 1509 rows have no hospital at all.
  • count(linelist_raw, age_unit) shows that 37 ages are in months, not years.
Note🔁 The base R way: table()

You will often see base R’s table() in other people’s code. It counts too, but it hides missing values unless you ask for them:

table(linelist_raw$OUTCOME)                   # where are the missing outcomes?

  Death Recover 
   2898    2213 
table(linelist_raw$OUTCOME, useNA = "ifany")  # there they are: 1493

  Death Recover    <NA> 
   2898    2213    1493 
Note🔍 What this code does
  • table() takes one column, pulled out with $, instead of the data and a column name.
  • Without useNA = "ifany", it shows only Death (2898) and Recover (2213). The 1493 missing outcomes are left out without a warning, so any percentages you worked out from it would be wrong.
  • useNA = "ifany" adds an <NA> count whenever there are missing values.

count() always shows NA. With table(), add useNA = "ifany". See Base R ↔︎ tidyverse for more translations.

Important✏️ TRY IT 5 (3 min): categories
  1. count() District, SEX, and `Vaccinated?` in measles_raw. How many different ways is “North” written? How many ways is “No” written?
  2. Now try table(measles_raw$District). What is missing compared with count()?

✅ Check: a) count(District) shows 18 rows (including NA) b) table() has no NA: 34 cases disappear

count(measles_raw, District)
           District   n
1           Central  95
2  Central District   6
3           Cetnral   9
4              East 215
5     East District   7
6             North 318
7    North District   9
8             Nroth  19
9             South 273
10   South District  11
11             West 167
12    West District   6
13          central   3
14             east  11
15            north  10
16            south  13
17             west   9
18             <NA>  34
count(measles_raw, SEX)
     SEX   n
1      F  81
2 Female 415
3      M  86
4   Male 509
5 female  37
6   male  47
7   <NA>  40
count(measles_raw, `Vaccinated?`)
  Vaccinated?   n
1           N  40
2          NO  44
3          No 608
4     Unknown 183
5           Y  32
6         Yes 231
7          no  49
8         yes  28
table(measles_raw$District)                   # b) no NA!

         central          Central Central District          Cetnral 
               3               95                6                9 
            east             East    East District            north 
              11              215                7               10 
           North   North District            Nroth            south 
             318                9               19               13 
           South   South District             west             West 
             273               11                9              167 
   West District 
               6 
table(measles_raw$District, useNA = "ifany")

         central          Central Central District          Cetnral 
               3               95                6                9 
            east             East    East District            north 
              11              215                7               10 
           North   North District            Nroth            south 
             318                9               19               13 
           South   South District             west             West 
             273               11                9              167 
   West District             <NA> 
               6               34 

“North” is written 4 ways (“North”, “north”, “Nroth”, “North District”), and so is “No” (“No”, “no”, “N”, “NO”). table() silently drops the 34 cases with no district; useNA = "ifany" brings them back. It also sorts lower-case values first, so “central” and “Central” are listed apart.

Zooming in: filter() and arrange()

  • filter() keeps rows that meet a condition
  • arrange() sorts rows (desc() for largest first)

Each one takes the data first. Column names with spaces need backticks: `Case ID`. (The backtick key is at the top left of most keyboards.) Another reason to clean the names!

linelist_raw is a regular data frame, so R prints every row it gets — all 6,604 of them, with 12 columns each! To do two things, such as filtering and then showing only the first 10 rows, put one function inside another. R runs the inside first:

# How many patients have ages in months?
nrow(filter(linelist_raw, age_unit == "months"))
[1] 37
# Show the first 10 of them
head(filter(linelist_raw, age_unit == "months"), 10)
   Case ID infection date date.onset  hosp date date of outcome OUTCOME Gender
1   e1c081     2014-10-13 2014-10-14 2014-10-14      2014-10-27   Death      m
2   9d8358     2014-10-01       <NA> 2014-10-18            <NA> Recover   <NA>
3   df9f64     2014-10-06 2014-10-26 2014-10-27      2014-11-06   Death      f
4   f76807           <NA> 2014-10-27 2014-10-28      2014-11-02   Death      f
5   58b05a           <NA> 2014-12-04 2014-12-05            <NA> Recover   <NA>
6   5a2c39     2014-12-17 2014-12-25 2014-12-25      2015-01-16 Recover      m
7   092cd0     2014-12-22 2015-01-02 2015-01-03      2015-01-17   Death      f
8   04a78f           <NA> 2015-01-15 2015-01-17      2015-01-26 Recover      f
9   833a20     2014-09-20 2014-09-30 2014-10-02      2014-10-05 Recover      m
10  5439ab     2014-09-20 2014-10-04 2014-10-05            <NA>    <NA>      f
                  hospital  age age_unit wt_kg ht_cm
1  Regional Medical Center   24   months    68   138
2                     <NA> <NA>   months    22    32
3  Regional Medical Center    7   months    49    72
4  Regional Medical Center    7   months    52   120
5        Military Hospital   24   months    49   123
6                    Other   24   months    48    74
7                    Other    6   months    72   223
8        Military Hospital   36   months    43    65
9  Regional Medical Center    4   months    40    65
10                    <NA>    4   months    58   125
# Impossible weights: sort smallest first, then show the first 10 rows
head(arrange(linelist_raw, wt_kg), 10)
   Case ID infection date date.onset  hosp date date of outcome OUTCOME Gender
1   6056ba           <NA> 2014-10-20 2014-10-21      2014-11-05   Death      f
2   6056ba           <NA> 2012-10-20 2012-10-21      2012-11-05   Death      f
3   7400b3     2014-10-07 2014-10-12 2014-10-13      2014-10-18   Death      f
4   0fde59     2014-09-01 2014-09-04 2014-09-08      2014-09-14 Recover      f
5   fca629           <NA> 2014-12-04 2014-12-07      2014-12-30 Recover      f
6   eb4d8f     2014-10-02 2014-10-06 2014-10-07      2014-10-11    <NA>      f
7   4fb46f     2014-08-12 2014-08-21 2014-08-25      2014-09-01 Recover      f
8   db9404     2015-01-18 2015-01-21 2015-01-24      2015-01-27   Death      f
9   78b0f7     2014-10-29 2014-11-05 2014-11-05      2014-11-19 Recover      f
10  35a38e     2014-09-11 2014-09-16 2014-09-17      2014-09-26 Recover      f
                  hospital age age_unit wt_kg ht_cm
1                     <NA>   0    years   -11    24
2               HMO Health   0    years   -11    24
3  Regional Medical Center   0    years   -10    13
4                     <NA>   0    years   -10    18
5         Central Hospital   0    years    -9    24
6                    Other   0    years    -8    22
7  Regional Medical Center   0    years    -7    13
8  Regional Medical Center   0    years    -7    16
9                     <NA>   0    years    -6    19
10 Regional Medical Center   0    years    -6    23
# Tallest patients: desc() sorts largest first
head(arrange(linelist_raw, desc(ht_cm)), 10)
   Case ID infection date date.onset  hosp date date of outcome OUTCOME Gender
1   ebb14f     2014-08-19 2014-09-01 2014-09-07      2014-09-13 Recover      m
2   610c15           <NA> 2015-02-04 2015-02-06      2015-02-20    <NA>      m
3   0b32d8     2015-01-07 2015-01-13 2015-01-14      2015-01-15   Death      m
4   b06109     2014-11-13 2014-11-28 2014-11-30      2014-12-05   Death      m
5   b21b3e           <NA> 2015-01-11 2015-01-12      2015-01-14   Death      m
6   9c3a56     2015-01-04 2015-01-20 2015-01-21      2015-01-27   Death   <NA>
7   389175     2015-03-13 2015-03-24 2015-03-25      2015-04-22 Recover      f
8     <NA>     2015-03-13 2015-03-24 2015-03-25      2015-04-22 Recover      f
9   ae0163     2014-08-26 2014-08-29 2014-09-06      2014-09-19    <NA>      m
10  4b158a           <NA> 2014-11-07 2014-11-09      2014-12-08 Recover   <NA>
                             hospital age age_unit wt_kg ht_cm
1  St. Marks Maternity Hopital (SMMH)  84    years   111   295
2                   Military Hospital  69    years   105   287
3                   Military Hospital  72    years    94   272
4             Regional Medical Center  68    years    85   268
5                                <NA>  49    years    80   266
6                                <NA>  76    years   101   266
7                                <NA>   6   months    95   266
8                                <NA>   6   months    95   266
9                                <NA>  60    years    94   263
10                               <NA>  69    years    87   262
Note🔍 What this code does
  • filter(linelist_raw, age_unit == "months") keeps only the rows where age_unit is exactly “months”. == asks “is it equal?” for every row, and filter() keeps the rows where the answer is TRUE.
  • nrow() counts the rows that are left: 37. head(..., 10) shows only the first 10 of them.
  • arrange(linelist_raw, wt_kg) sorts all rows by weight, smallest first, so the impossible weights (down to -11 kg) come to the top. desc(ht_cm) sorts largest first, so the 295 cm height comes to the top.
  • None of these change linelist_raw. They show a changed copy and then forget it. To keep a result, save it with <-.
NoteReading code from the inside out

head(arrange(linelist_raw, wt_kg), 10) reads: take linelist_raw, arrange it by weight, then take the head. Two steps are manageable, but with three or four the brackets pile up. In Session 2 you’ll learn the pipe %>%, which writes the same steps in order, from left to right.

NoteConditions you can use in filter()

== equal · != not equal · > < >= <= · %in% c(...) one of several values · is.na() missing · & and · | or

Use == (two equals signs) to compare. One = is for naming arguments.

Important✏️ TRY IT 6 (3 min): zooming in

Sort measles_raw by `Temp (C)`, largest first, and show only the top 10 rows. What is the highest temperature? What do you think happened?

Pattern: head(arrange(DATA, desc(COLUMN)), 10)

✅ Check: the top value is 104.7

head(arrange(measles_raw, desc(`Temp (C)`)), 10)
   Patient ID Report Date District    SEX Age (years) Vaccinated? Fever Rash
1     MS-0663  2024-03-15    South   Male          14          No   yes  yes
2     MS-0444  2024-02-26     East   Male           8     Unknown   yes  yes
3     MS-0908  2024-03-03    Nroth   Male           1         yes   yes  yes
4     MS-0523  2024-04-21     East Female          13          No   yes  yes
5     MS-0132  2024-03-18     <NA>   Male           1         Yes   yes  yes
6     MS-0052  2024-02-01    South   Male           5         Yes   yes  yes
7     MS-1027  2024-03-21     East   Male           3     Unknown   yes  yes
8     MS-0167  2024-04-10  Central Female           6          No   yes  yes
9     MS-0110  2024-04-22    South   Male           4          No   yes  yes
10    MS-0943  2024-04-11    South Female          23     Unknown   yes  yes
   Temp (C) Days in Hospital   Outcome
1     104.7               NA Recovered
2     104.0               NA Recovered
3     104.0               NA Recovered
4     104.0               NA Recovered
5     103.5               NA Recovered
6     103.3               NA Recovered
7     103.3               NA Recovered
8     103.3               NA Recovered
9     102.9               NA Recovered
10    102.6               NA Recovered

104.7 is about 40 °C written in Fahrenheit. A few temperatures were entered in °F instead of °C.

Missing values and duplicates

# Missing values in each column
colSums(is.na(linelist_raw))
        Case ID  infection date      date.onset       hosp date date of outcome 
            130            2315             286               0            1061 
        OUTCOME          Gender        hospital             age        age_unit 
           1493             317            1509             100               0 
          wt_kg           ht_cm 
              0               0 
# How many rows have no Case ID? Count the rows that filter() keeps
nrow(filter(linelist_raw, is.na(`Case ID`)))
[1] 130
# How many rows are exact copies of an earlier row?
sum(duplicated(linelist_raw))
[1] 25
# Look at the duplicated rows (shows every copy, plus a dupe_count column)
get_dupes(linelist_raw)
Note🔍 What this code does
  • is.na(linelist_raw) checks every cell and gives TRUE where a value is missing. colSums() adds up each column, counting each TRUE as 1. The result is the number of missing values per column, for example 1493 in OUTCOME.
  • filter(linelist_raw, is.na(`Case ID`)) keeps the rows with no Case ID, and nrow() counts them: 130.
  • duplicated() marks every row that is an exact copy of an earlier row (every column the same), and sum() counts them: 25. The first copy of each row isn’t counted.
  • get_dupes() (janitor) shows every row that has a copy, side by side, with a dupe_count column. Here that’s 50 rows: each of the 25 duplicates and its original.
Important✏️ TRY IT 7 (3 min): missing values and duplicates

In measles_raw:

  1. How many rows have a missing `Patient ID`?
  2. How many rows are exact duplicates?
  3. Which column has the most missing values? Is that a mistake, or does it make sense?

✅ Check: a) 37 b) 15

nrow(filter(measles_raw, is.na(`Patient ID`)))
[1] 37
sum(duplicated(measles_raw))
[1] 15
colSums(is.na(measles_raw))
      Patient ID      Report Date         District              SEX 
              37               29               34               40 
     Age (years)      Vaccinated?            Fever             Rash 
               0                0                0                0 
        Temp (C) Days in Hospital          Outcome 
               0              766               68 
  1. Days in Hospital has the most (766). That makes sense: patients who were never hospitalized have no hospital days. Missing is not always an error!

Part 6: The problem list

Note📖 Good to know: a checklist for any new dataset

Use these questions on your own data, too. The answers become your problem list.

  1. How many rows and columns are there? Is that what you expected?
  2. Are the column types right (numbers, dates, text)?
  3. Which columns have missing values, and do they make sense?
  4. Are the categories spelled consistently (count())?
  5. Are there impossible values (skim(), arrange())?
  6. Are there duplicate rows?

Here is everything we found in linelist_raw. This list is the to-do list for Session 2.

# Problem How we found it
1 Column names have spaces, dots, and capitals names()
2 Date columns are different types (date.onset is text) glimpse()
3 age is stored as text, and 37 ages are in months not years glimpse(), count()
4 Hospital names are misspelled (“Hopital”) or inconsistent (“Regional MC”), and many are missing count()
5 Gender and outcome have missing values count(), skim()
6 Impossible weights (negative, 0) and heights (4 cm, 295 cm) skim(), arrange()
7 130 rows have no Case ID is.na()
8 25 rows are duplicates duplicated()
Important✏️ TRY IT 8 (10 min): the measles problem list

Write a numbered problem list for measles_raw in your student script, using everything you found in the TRY ITs. Save it — you will fix these problems in Session 2!

✅ Check: you should find at least 8 problems

  1. Column names have spaces, capitals, and symbols (? and parentheses)
  2. Age (years) is stored as text because of “unknown” (27 rows) and “999” (4 rows)
  3. District is misspelled (“Nroth”, “Cetnral”), has lower-case versions, and some say “… District”
  4. SEX is written 6 different ways (Male / male / M, Female / female / F)
  5. Vaccinated? is written 8 different ways (Yes / yes / Y, No / no / N / NO, Unknown)
  6. Outcome has inconsistent capitalization and 68 missing values
  7. Some temperatures were entered in Fahrenheit (values around 97–105)
  8. 37 rows have no Patient ID
  9. 15 rows are duplicates
  10. Report Date, District, and SEX have some missing values
Tip⭐ Bonus: your own data

Brought your own dataset? Put the file in the raw_data/ folder, then: import() it; look at it with dim(), glimpse(), and skim(); count() its text columns to look for misspellings; check for missing values and duplicates; and write its problem list.

Key points

  • <- saves a value in an object; c() makes a vector; functions take arguments.
  • Install a package once with install.packages(); load it every session with library().
  • A project folder plus here() makes file paths work on any computer.
  • import(here("folder", "file")) reads almost any file type.
  • Look before you clean: glimpse(), skim(), count(), filter(), arrange(), is.na(), and duplicated() find the problems.
  • To do two steps, put one function inside another: head(arrange(data, column), 10). Wrap long output in head().

Next: Session 2 — Cleaning. We start with the pipe %>%, then fix the problem list with clean_names(), mutate(), if_else(), and case_when().