6A: Data Manipulation with dplyr

Readings

From R for Data Science by Drs. Hadley Wickham, Garrett Grolemund, and Mine Çetinkaya-Rundel:

Data Transformation

From Tidy Hurricanes: Analyzing Tropical Storms with Tidyverse Tools by Dr. Hadley Wickham:

From Stat 331/531 Statistical Computing with R by Dr. Emily Robinson:

Data Cleaning and Manipulation

Topics

dplyr and the grammar of data manipulation
The pipe %>% and |> operators

`dplyr` and the grammar of data manipulation

Data wrangling, also known as data munging, is the process of cleaning, transforming, and manipulating data so that the data are usable for visualization, analysis, and modeling.
Much of data wrangling involves subsetting, e.g., removing unwanted rows and/or columns, modifying columns, summarizing data by groups.
dplyr (dee-ply-er, “d” for data and “plyr” for pliers or tools), is a popular R package in the tidyverse that implements the grammar of data manipulation. It includes the following verbs:
- slice() to subset rows using their positions
- filter() to keep rows that match a condition
- select() to keep or drop columns
- mutate() to modify existing variables or add new variables
- arrange() to reorder rows using column names
- summarize() to reduce variables to values
- group_by() to perform grouped (aggregated) operations
Note that these functions never modify the original data set unless we choose to do so. In addition, extracting a variable using the dollar $ operator is not necessary.

`classmates` data set

For this lesson, we will work with the following small data set:

library(tidyverse)

classmates <- tibble(
  name = c('Alice', 'Brianna', 'Carlos', 'David', 
           'Emmett', 'Fiona', 'Grace', 'Helen',
           'Issac', 'Jill', 'Leila', 'Nate'),
  gender = c('female', 'female', 'male', 'male',
             'male', 'female', 'female', 'female',
             'male', 'female', 'female', 'male'),
  height = c(5.4, 5.9, 5.2, 6.0, 5.6, 5.8, 5.3, 6.1, 5.5, 5.10, 5.4, 6.2),
  weight = c(130, 165, 120, 185, 140, 170, 125, 160, 135, 170, 128, 210),
  age = c(19, 21, 20, 22, 18, 23, 19, 21, 20, 22, 18, 23),
  major = c('Math', 'CS', 'DS', 'CS', 'Math', 'DS', 'CS', 'Math', 'CS', 'CS', 
            'Math', 'DS')
)

classmates

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Alice   female    5.4    130    19 Math 
 2 Brianna female    5.9    165    21 CS   
 3 Carlos  male      5.2    120    20 DS   
 4 David   male      6      185    22 CS   
 5 Emmett  male      5.6    140    18 Math 
 6 Fiona   female    5.8    170    23 DS   
 7 Grace   female    5.3    125    19 CS   
 8 Helen   female    6.1    160    21 Math 
 9 Issac   male      5.5    135    20 CS   
10 Jill    female    5.1    170    22 CS   
11 Leila   female    5.4    128    18 Math 
12 Nate    male      6.2    210    23 DS

Note that classmates has no real references!

`slice()` to subset rows

slice() allows us to select rows based on index position.

# row 1
slice(classmates, 1)

# A tibble: 1 × 6
  name  gender height weight   age major
  <chr> <chr>   <dbl>  <dbl> <dbl> <chr>
1 Alice female    5.4    130    19 Math

# row 3
slice(classmates, 3)

# A tibble: 1 × 6
  name   gender height weight   age major
  <chr>  <chr>   <dbl>  <dbl> <dbl> <chr>
1 Carlos male      5.2    120    20 DS

# rows 5, 11, 9
slice(classmates, 5, 11, 9)

# A tibble: 3 × 6
  name   gender height weight   age major
  <chr>  <chr>   <dbl>  <dbl> <dbl> <chr>
1 Emmett male      5.6    140    18 Math 
2 Leila  female    5.4    128    18 Math 
3 Issac  male      5.5    135    20 CS

# rows 2, 3, 4, 5, 6
slice(classmates, 2:6)

# A tibble: 5 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Brianna female    5.9    165    21 CS   
2 Carlos  male      5.2    120    20 DS   
3 David   male      6      185    22 CS   
4 Emmett  male      5.6    140    18 Math 
5 Fiona   female    5.8    170    23 DS

Recall that negative indexing means “all except”.

# all except row 1
slice(classmates, -1)

# A tibble: 11 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Brianna female    5.9    165    21 CS   
 2 Carlos  male      5.2    120    20 DS   
 3 David   male      6      185    22 CS   
 4 Emmett  male      5.6    140    18 Math 
 5 Fiona   female    5.8    170    23 DS   
 6 Grace   female    5.3    125    19 CS   
 7 Helen   female    6.1    160    21 Math 
 8 Issac   male      5.5    135    20 CS   
 9 Jill    female    5.1    170    22 CS   
10 Leila   female    5.4    128    18 Math 
11 Nate    male      6.2    210    23 DS

# all except row 3
slice(classmates, -3)

# A tibble: 11 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Alice   female    5.4    130    19 Math 
 2 Brianna female    5.9    165    21 CS   
 3 David   male      6      185    22 CS   
 4 Emmett  male      5.6    140    18 Math 
 5 Fiona   female    5.8    170    23 DS   
 6 Grace   female    5.3    125    19 CS   
 7 Helen   female    6.1    160    21 Math 
 8 Issac   male      5.5    135    20 CS   
 9 Jill    female    5.1    170    22 CS   
10 Leila   female    5.4    128    18 Math 
11 Nate    male      6.2    210    23 DS

# all except rows 5, 11, 9
slice(classmates, -5, -11, -9)

# A tibble: 9 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Alice   female    5.4    130    19 Math 
2 Brianna female    5.9    165    21 CS   
3 Carlos  male      5.2    120    20 DS   
4 David   male      6      185    22 CS   
5 Fiona   female    5.8    170    23 DS   
6 Grace   female    5.3    125    19 CS   
7 Helen   female    6.1    160    21 Math 
8 Jill    female    5.1    170    22 CS   
9 Nate    male      6.2    210    23 DS

# all except rows 2, 3, 4, 5, 6
slice(classmates, -(2:6))

# A tibble: 7 × 6
  name  gender height weight   age major
  <chr> <chr>   <dbl>  <dbl> <dbl> <chr>
1 Alice female    5.4    130    19 Math 
2 Grace female    5.3    125    19 CS   
3 Helen female    6.1    160    21 Math 
4 Issac male      5.5    135    20 CS   
5 Jill  female    5.1    170    22 CS   
6 Leila female    5.4    128    18 Math 
7 Nate  male      6.2    210    23 DS

slice_head() allows us to select the first few rows by specifying the number n or the proportion prop.

# first row
slice_head(classmates)

# first two rows
slice_head(classmates, n = 2)

# first 30% of the rows
slice_head(classmates, prop = 0.3)

slice_tail() is for the last few rows.

# last row
slice_tail(classmates)

# last four rows
slice_tail(classmates, n = 4)

# last 30% of the rows
slice_tail(classmates, prop = 0.3)

slice_sample() randomly selects rows.

# pseudo random generation
set.seed(101)

# 1 random row
slice_sample(classmates)

# 6 random rows
slice_sample(classmates, n = 6)

# 12 random rows
slice_sample(classmates, n = 12)

# a random sample of 80% of rows
slice_sample(classmates, prop = 0.8)

slice_min() selects rows with the smallest of a variable.

# classmate with the smallest weight
slice_min(classmates, weight)

# 2 classmates with the smallest weight
slice_min(classmates, weight, n = 2)

# 12 classmates with the smallest weight
slice_min(classmates, weight, n = 12)

# half of classmates from small to large weight
slice_min(classmates, weight, prop = 0.5)

slice_max() selects rows with the largest of a variable.

# classmate with the largest weight
slice_max(classmates, weight)

# 2 classmates with the largest weight
slice_max(classmates, weight, n = 2)

# 12 classmates with the largest weight
slice_max(classmates, weight, n = 12)

# half of classmates from large to small weight
slice_max(classmates, weight, prop = 0.5)

💻 Hands-On

Use slice() and related functions to return rows of the classmates data set.

# row 4

# row 11

# rows 4, 5, 6, ..., 10

# rows 7, 8, 9, ..., 12

# rows 2, 10, 1, 5, 9

# rows 12, 10, 2, 11, 6

# all except row 5

# all except rows 6, 7, 8, 9

# first 5 rows

# last 3 rows

# about four youngest classmates

# about three shortest classmates

# about five oldest classmates

# about two tallest classmates

# a random sample of 7 rows

# a random sample of two third of the rows

Answer

# row 4
slice(classmates, 1)

# row 11
slice(classmates, 11)

# rows 4, 5, 6, ..., 10
slice(classmates, 4:10)

# rows 7, 8, 9, ..., 12
slice(classmates, 7:12)

# rows 2, 10, 1, 5, 9
slice(classmates, 2, 10, 1, 5, 9)

# rows 12, 10, 2, 11, 6
slice(classmates, 12, 10, 2, 11, 6)

# all except row 5
slice(classmates, -5)

# all except rows 6, 7, 8, 9
slice(classmates, -(6:9))

# first 5 rows
slice_head(classmates, n = 5)

# last 3 rows
slice_tail(classmates, n = 3)

# about four youngest classmates
slice_min(classmates, age, n = 4)

# about three shortest classmates
slice_min(classmates, height, n = 3)

# about five oldest classmates
slice_max(classmates, age, n = 5)

# about two tallest classmates
slice_max(classmates, height, n = 2)

# a random sample of 7 rows
slice_sample(classmates, n = 7)

# a random sample of two third of the rows
slice_sample(classmates, prop = 2/3)

`filter()` to select rows with conditions

filter() allows us to subset rows that satisfy some conditions.

# CS majors
filter(classmates, major == 'CS')

# A tibble: 5 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Brianna female    5.9    165    21 CS   
2 David   male      6      185    22 CS   
3 Grace   female    5.3    125    19 CS   
4 Issac   male      5.5    135    20 CS   
5 Jill    female    5.1    170    22 CS

# CS majors who are taller than 5.2
filter(classmates, major == 'CS', height > 5.2)

# A tibble: 4 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Brianna female    5.9    165    21 CS   
2 David   male      6      185    22 CS   
3 Grace   female    5.3    125    19 CS   
4 Issac   male      5.5    135    20 CS

# CS majors who are taller than 5.2 and are female
filter(classmates, major == 'CS', height > 5.2, gender == 'female')

# A tibble: 2 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Brianna female    5.9    165    21 CS   
2 Grace   female    5.3    125    19 CS

💻 Hands-On

Use filter() return rows of the classmates data set.

# male classmates

# male classmates who weigh less than 200 lbs

# male classmates who weigh less than 200 lbs and are below 21

# classmates shorter than 6 ft

# classmates shorter than 6 ft and at least 21

# Math majors

# Math majors that are female

Answer

# male classmates
filter(classmates, gender == 'male')

# A tibble: 5 × 6
  name   gender height weight   age major
  <chr>  <chr>   <dbl>  <dbl> <dbl> <chr>
1 Carlos male      5.2    120    20 DS   
2 David  male      6      185    22 CS   
3 Emmett male      5.6    140    18 Math 
4 Issac  male      5.5    135    20 CS   
5 Nate   male      6.2    210    23 DS

# male classmates who weigh less than 200 lbs
filter(classmates, gender == 'male', weight < 200)

# A tibble: 4 × 6
  name   gender height weight   age major
  <chr>  <chr>   <dbl>  <dbl> <dbl> <chr>
1 Carlos male      5.2    120    20 DS   
2 David  male      6      185    22 CS   
3 Emmett male      5.6    140    18 Math 
4 Issac  male      5.5    135    20 CS

# male classmates who weigh less than 200 lbs and are below 21
filter(classmates, gender == 'male', weight < 200, age < 21)

# A tibble: 3 × 6
  name   gender height weight   age major
  <chr>  <chr>   <dbl>  <dbl> <dbl> <chr>
1 Carlos male      5.2    120    20 DS   
2 Emmett male      5.6    140    18 Math 
3 Issac  male      5.5    135    20 CS

# classmates shorter than 6 ft
filter(classmates, height < 6)

# A tibble: 9 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Alice   female    5.4    130    19 Math 
2 Brianna female    5.9    165    21 CS   
3 Carlos  male      5.2    120    20 DS   
4 Emmett  male      5.6    140    18 Math 
5 Fiona   female    5.8    170    23 DS   
6 Grace   female    5.3    125    19 CS   
7 Issac   male      5.5    135    20 CS   
8 Jill    female    5.1    170    22 CS   
9 Leila   female    5.4    128    18 Math

# classmates shorter than 6 ft and at least 21
filter(classmates, height < 6, age >= 21)

# A tibble: 3 × 6
  name    gender height weight   age major
  <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
1 Brianna female    5.9    165    21 CS   
2 Fiona   female    5.8    170    23 DS   
3 Jill    female    5.1    170    22 CS

# Math majors
filter(classmates, major == 'Math')

# A tibble: 4 × 6
  name   gender height weight   age major
  <chr>  <chr>   <dbl>  <dbl> <dbl> <chr>
1 Alice  female    5.4    130    19 Math 
2 Emmett male      5.6    140    18 Math 
3 Helen  female    6.1    160    21 Math 
4 Leila  female    5.4    128    18 Math

# Math majors that are female
filter(classmates, major == 'Math', gender == 'female')

# A tibble: 3 × 6
  name  gender height weight   age major
  <chr> <chr>   <dbl>  <dbl> <dbl> <chr>
1 Alice female    5.4    130    19 Math 
2 Helen female    6.1    160    21 Math 
3 Leila female    5.4    128    18 Math

`select()` to subset columns (variables)

select() allows us to select columns (variables) by names.

# column name
select(classmates, name)

# columns name, gender, major
select(classmates, name, gender, major)

# a range of columns
select(classmates, height:major)

Although less common, it is also possible to select by index positions.

# column name
select(classmates, 1)

# columns name, gender, major
select(classmates, 1, 2, 6)

# a range of columns
select(classmates, 3:6)

Recall that negative indexing means “all except”.

# all except column name
select(classmates, -name)

# all except columns name, gender, major
select(classmates, -name, -gender, -major)

# all except a range of columns
select(classmates, -(height:major))

💻 Hands-On

Use select() to retrieve columns of the classmates data set.

# column weight

# column major

# columns weight, weight, weight 

# columns weight, major, weight, major

# columns name, height, weight, age

# all except column height

# all except column age

# all except columns name, height, weight, age

Answer

# column weight
select(classmates, weight)

# A tibble: 12 × 1
   weight
    <dbl>
 1    130
 2    165
 3    120
 4    185
 5    140
 6    170
 7    125
 8    160
 9    135
10    170
11    128
12    210

# column major
select(classmates, major)

# A tibble: 12 × 1
   major
   <chr>
 1 Math 
 2 CS   
 3 DS   
 4 CS   
 5 Math 
 6 DS   
 7 CS   
 8 Math 
 9 CS   
10 CS   
11 Math 
12 DS

# columns weight, weight, weight 
select(classmates, weight, weight, weight)

# A tibble: 12 × 1
   weight
    <dbl>
 1    130
 2    165
 3    120
 4    185
 5    140
 6    170
 7    125
 8    160
 9    135
10    170
11    128
12    210

# columns weight, major, weight, major
select(classmates, weight, major, weight, major)

# A tibble: 12 × 2
   weight major
    <dbl> <chr>
 1    130 Math 
 2    165 CS   
 3    120 DS   
 4    185 CS   
 5    140 Math 
 6    170 DS   
 7    125 CS   
 8    160 Math 
 9    135 CS   
10    170 CS   
11    128 Math 
12    210 DS

# columns name, height, weight, age
select(classmates, name, height, weight, age)

# A tibble: 12 × 4
   name    height weight   age
   <chr>    <dbl>  <dbl> <dbl>
 1 Alice      5.4    130    19
 2 Brianna    5.9    165    21
 3 Carlos     5.2    120    20
 4 David      6      185    22
 5 Emmett     5.6    140    18
 6 Fiona      5.8    170    23
 7 Grace      5.3    125    19
 8 Helen      6.1    160    21
 9 Issac      5.5    135    20
10 Jill       5.1    170    22
11 Leila      5.4    128    18
12 Nate       6.2    210    23

# all except column height
select(classmates, -height)

# A tibble: 12 × 5
   name    gender weight   age major
   <chr>   <chr>   <dbl> <dbl> <chr>
 1 Alice   female    130    19 Math 
 2 Brianna female    165    21 CS   
 3 Carlos  male      120    20 DS   
 4 David   male      185    22 CS   
 5 Emmett  male      140    18 Math 
 6 Fiona   female    170    23 DS   
 7 Grace   female    125    19 CS   
 8 Helen   female    160    21 Math 
 9 Issac   male      135    20 CS   
10 Jill    female    170    22 CS   
11 Leila   female    128    18 Math 
12 Nate    male      210    23 DS

# all except column age
select(classmates, -age)

# A tibble: 12 × 5
   name    gender height weight major
   <chr>   <chr>   <dbl>  <dbl> <chr>
 1 Alice   female    5.4    130 Math 
 2 Brianna female    5.9    165 CS   
 3 Carlos  male      5.2    120 DS   
 4 David   male      6      185 CS   
 5 Emmett  male      5.6    140 Math 
 6 Fiona   female    5.8    170 DS   
 7 Grace   female    5.3    125 CS   
 8 Helen   female    6.1    160 Math 
 9 Issac   male      5.5    135 CS   
10 Jill    female    5.1    170 CS   
11 Leila   female    5.4    128 Math 
12 Nate    male      6.2    210 DS

# all except columns name, height, weight, age
select(classmates, -name, -height, -weight, -age)

# A tibble: 12 × 2
   gender major
   <chr>  <chr>
 1 female Math 
 2 female CS   
 3 male   DS   
 4 male   CS   
 5 male   Math 
 6 female DS   
 7 female CS   
 8 female Math 
 9 male   CS   
10 female CS   
11 female Math 
12 male   DS

`mutate()` to transform, add, or delete variables

mutate() allows us perform do the following:
- Transform existing columns (variables)
- Add new columns (variables)
- Delete existing columns (variables)
The most common use of mutate() is to transform variable(s).

# transform height from ft to cm
mutate(classmates, height = height * 30.48)

# transform weight from lbs to kg
mutate(classmates, weight = weight * 0.454)

# transform height from ft to cm and weight from lbs to kg at the same time
mutate(classmates, height = height * 30.48, weight = weight * 0.454)

Note that mutate() does not modify the original data unless we choose to do so.

classmates <- mutate(classmates, height = height * 30.48)

mutate() also allows us to add new variables.

# add height_cm
mutate(classmates, height_cm = height * 30.48)

# add height_cm and weight_kg at the same time
mutate(classmates, height_cm = height * 30.48, weight_kg = weight * 0.454)

mutate() can be used to delete variables.

# delete height_cm and weight_kg
mutate(classmates, height_cm = NULL, weight_kg = NULL)

💻 Hands-On

Use mutate() to add the following variables the classmates data set:

height_in gives height in inches (note that 1 foot = 12 inches)
can_drink shows TRUE if a classmate is at least 21 and FALSE otherwise

Answer

mutate(classmates, height_in = height * 12, can_drink = age > 21)

# A tibble: 12 × 8
   name    gender height weight   age major height_in can_drink
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>     <dbl> <lgl>    
 1 Alice   female    5.4    130    19 Math       64.8 FALSE    
 2 Brianna female    5.9    165    21 CS         70.8 FALSE    
 3 Carlos  male      5.2    120    20 DS         62.4 FALSE    
 4 David   male      6      185    22 CS         72   TRUE     
 5 Emmett  male      5.6    140    18 Math       67.2 FALSE    
 6 Fiona   female    5.8    170    23 DS         69.6 TRUE     
 7 Grace   female    5.3    125    19 CS         63.6 FALSE    
 8 Helen   female    6.1    160    21 Math       73.2 FALSE    
 9 Issac   male      5.5    135    20 CS         66   FALSE    
10 Jill    female    5.1    170    22 CS         61.2 TRUE     
11 Leila   female    5.4    128    18 Math       64.8 FALSE    
12 Nate    male      6.2    210    23 DS         74.4 TRUE

`arrange()` to order rows

arrange() allows us to order rows according to selected variables. Note that the order of variables matters.

# all female -> all male
arrange(classmates, gender)

# all female -> all male -> within female, sort by age -> within male, sort by age
arrange(classmates, gender, age)

# all 18 -> all 19 -> all 20 -> all 21 -> all 22 -> all 23 
arrange(classmates, age)

# all 18 -> all 19 -> all 20 -> all 21 -> all 22 -> all 23 
# -> within 18, sort by gender
# -> within 19, sort by gender
# and so on
arrange(classmates, age, gender)

By default, arrange() sorts rows in increasing order. To arrange rows in descending order, we need to use desc().

# all male -> all female
arrange(classmates, desc(gender))

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Carlos  male      5.2    120    20 DS   
 2 David   male      6      185    22 CS   
 3 Emmett  male      5.6    140    18 Math 
 4 Issac   male      5.5    135    20 CS   
 5 Nate    male      6.2    210    23 DS   
 6 Alice   female    5.4    130    19 Math 
 7 Brianna female    5.9    165    21 CS   
 8 Fiona   female    5.8    170    23 DS   
 9 Grace   female    5.3    125    19 CS   
10 Helen   female    6.1    160    21 Math 
11 Jill    female    5.1    170    22 CS   
12 Leila   female    5.4    128    18 Math

# all 23 -> all 22 -> all 21 -> all 20 -> all 19 -> all 18 
arrange(classmates, desc(age))

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Fiona   female    5.8    170    23 DS   
 2 Nate    male      6.2    210    23 DS   
 3 David   male      6      185    22 CS   
 4 Jill    female    5.1    170    22 CS   
 5 Brianna female    5.9    165    21 CS   
 6 Helen   female    6.1    160    21 Math 
 7 Carlos  male      5.2    120    20 DS   
 8 Issac   male      5.5    135    20 CS   
 9 Alice   female    5.4    130    19 Math 
10 Grace   female    5.3    125    19 CS   
11 Emmett  male      5.6    140    18 Math 
12 Leila   female    5.4    128    18 Math

💻 Hands-On

Use arrange() to order rows of the classmates data set.

# major

# major -> gender

# major -> gender -> height

# descending major

# descending major -> descending age

Answer

# major
arrange(classmates, major)

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Brianna female    5.9    165    21 CS   
 2 David   male      6      185    22 CS   
 3 Grace   female    5.3    125    19 CS   
 4 Issac   male      5.5    135    20 CS   
 5 Jill    female    5.1    170    22 CS   
 6 Carlos  male      5.2    120    20 DS   
 7 Fiona   female    5.8    170    23 DS   
 8 Nate    male      6.2    210    23 DS   
 9 Alice   female    5.4    130    19 Math 
10 Emmett  male      5.6    140    18 Math 
11 Helen   female    6.1    160    21 Math 
12 Leila   female    5.4    128    18 Math

# major -> gender
arrange(classmates, major, gender)

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Brianna female    5.9    165    21 CS   
 2 Grace   female    5.3    125    19 CS   
 3 Jill    female    5.1    170    22 CS   
 4 David   male      6      185    22 CS   
 5 Issac   male      5.5    135    20 CS   
 6 Fiona   female    5.8    170    23 DS   
 7 Carlos  male      5.2    120    20 DS   
 8 Nate    male      6.2    210    23 DS   
 9 Alice   female    5.4    130    19 Math 
10 Helen   female    6.1    160    21 Math 
11 Leila   female    5.4    128    18 Math 
12 Emmett  male      5.6    140    18 Math

# major -> gender -> height
arrange(classmates, major, gender, height)

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Jill    female    5.1    170    22 CS   
 2 Grace   female    5.3    125    19 CS   
 3 Brianna female    5.9    165    21 CS   
 4 Issac   male      5.5    135    20 CS   
 5 David   male      6      185    22 CS   
 6 Fiona   female    5.8    170    23 DS   
 7 Carlos  male      5.2    120    20 DS   
 8 Nate    male      6.2    210    23 DS   
 9 Alice   female    5.4    130    19 Math 
10 Leila   female    5.4    128    18 Math 
11 Helen   female    6.1    160    21 Math 
12 Emmett  male      5.6    140    18 Math

# descending major
arrange(classmates, desc(major))

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Alice   female    5.4    130    19 Math 
 2 Emmett  male      5.6    140    18 Math 
 3 Helen   female    6.1    160    21 Math 
 4 Leila   female    5.4    128    18 Math 
 5 Carlos  male      5.2    120    20 DS   
 6 Fiona   female    5.8    170    23 DS   
 7 Nate    male      6.2    210    23 DS   
 8 Brianna female    5.9    165    21 CS   
 9 David   male      6      185    22 CS   
10 Grace   female    5.3    125    19 CS   
11 Issac   male      5.5    135    20 CS   
12 Jill    female    5.1    170    22 CS

# descending major -> descending age
arrange(classmates, desc(major), desc(age))

# A tibble: 12 × 6
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Helen   female    6.1    160    21 Math 
 2 Alice   female    5.4    130    19 Math 
 3 Emmett  male      5.6    140    18 Math 
 4 Leila   female    5.4    128    18 Math 
 5 Fiona   female    5.8    170    23 DS   
 6 Nate    male      6.2    210    23 DS   
 7 Carlos  male      5.2    120    20 DS   
 8 David   male      6      185    22 CS   
 9 Jill    female    5.1    170    22 CS   
10 Brianna female    5.9    165    21 CS   
11 Issac   male      5.5    135    20 CS   
12 Grace   female    5.3    125    19 CS

`summarize()` to summarize values

summarize() allows us to build a summary table. Note that extracting a variable using the dollar $ operator is not necessary within summarize().
summarize() is most powerful when paired with group_by() to build summary tables for different groups.

# summary statistics for the whole data set
summarize(
  classmates,
  mean_height = mean(height),
  median_height = median(height),
  min_height = min(height),
  max_height = max(height),
  title = 'My Table'
)

# A tibble: 1 × 5
  mean_height median_height min_height max_height title   
        <dbl>         <dbl>      <dbl>      <dbl> <chr>   
1        5.62          5.55        5.1        6.2 My Table

💻 Hands-On

Use summarize() to build a summary table of the classmates data set that includes mean_weight, median_weight, min_weight, max_weight, mean_age, median_age, min_age, max_age.

Answer

summarize(
  classmates,
  mean_weight = mean(weight),
  median_weight = median(weight),
  min_weight = min(weight),
  max_weight = max(weight),
  mean_age = mean(age),
  median_age = median(age),
  min_age = min(age),
  max_age = max(age),
)

# A tibble: 1 × 8
  mean_weight median_weight min_weight max_weight mean_age median_age min_age
        <dbl>         <dbl>      <dbl>      <dbl>    <dbl>      <dbl>   <dbl>
1        153.           150        120        210     20.5       20.5      18
# ℹ 1 more variable: max_age <dbl>

`group_by()` to divide a data set by variables

group_by() allows us to quickly divide a data set into subgroups.

# female classmates and male classmates
group_by(classmates, gender)

# CS majors, DS majors, and Math majors
group_by(classmates, major)

# combination of gender and major
group_by(classmates, gender, major)

Pairing group_by() with summarize() is a convenient way to obtain summary statistics for each group.

# summary statistics by gender

summarize(
  group_by(classmates, gender),
  mean_height = mean(height),
  median_height = median(height),
  min_height = min(height),
  max_height = max(height)
)

💻 Hands-On

Use group_by() to obtain subsets of the classmates data set.

# by different ages

# by different names

Answer

# by different ages
group_by(classmates, age)

# A tibble: 12 × 6
# Groups:   age [6]
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Alice   female    5.4    130    19 Math 
 2 Brianna female    5.9    165    21 CS   
 3 Carlos  male      5.2    120    20 DS   
 4 David   male      6      185    22 CS   
 5 Emmett  male      5.6    140    18 Math 
 6 Fiona   female    5.8    170    23 DS   
 7 Grace   female    5.3    125    19 CS   
 8 Helen   female    6.1    160    21 Math 
 9 Issac   male      5.5    135    20 CS   
10 Jill    female    5.1    170    22 CS   
11 Leila   female    5.4    128    18 Math 
12 Nate    male      6.2    210    23 DS

# by different names
group_by(classmates, name)

# A tibble: 12 × 6
# Groups:   name [12]
   name    gender height weight   age major
   <chr>   <chr>   <dbl>  <dbl> <dbl> <chr>
 1 Alice   female    5.4    130    19 Math 
 2 Brianna female    5.9    165    21 CS   
 3 Carlos  male      5.2    120    20 DS   
 4 David   male      6      185    22 CS   
 5 Emmett  male      5.6    140    18 Math 
 6 Fiona   female    5.8    170    23 DS   
 7 Grace   female    5.3    125    19 CS   
 8 Helen   female    6.1    160    21 Math 
 9 Issac   male      5.5    135    20 CS   
10 Jill    female    5.1    170    22 CS   
11 Leila   female    5.4    128    18 Math 
12 Nate    male      6.2    210    23 DS

💻 Hands-On

Use group_by() and summarize() to build a summary table of the classmates data set that includes mean_weight, median_weight, min_weight, max_weight for each major.

Answer

summarize(
  group_by(classmates, major),
  mean_weight = mean(weight),
  median_weight = median(weight),
  min_weight = min(weight),
  max_weight = max(weight),
)

# A tibble: 3 × 5
  major mean_weight median_weight min_weight max_weight
  <chr>       <dbl>         <dbl>      <dbl>      <dbl>
1 CS           156            165        125        185
2 DS           167.           170        120        210
3 Math         140.           135        128        160

The pipe `%>%` and `|>` operators

In practice, data wrangling usually involves a combination of slice(), filter(), select(), mutate(), summarize(), group_by(), and other functions. Step-by-step computations can create extra variables or less human-readable code.
The pipe operators aim to address step-by-step computations in a forward manner. There are two pipe operators with the same functionality:
- %>% from the magrittr (muh-GREET-er) package
- |> from base R
In their most basic use, the pipe operators allow us to call a function with at least 2 arguments f(x, y) by x |> f(y).

# from classmates, extract row 3
classmates |> slice(3)

# from classmates, extract rows 5, 11, 9
classmates |> slice(5, 11, 9)

# from classmates, retrieve all CS majors
classmates |> filter(major == 'CS')

# from classmates, retrieve all CS majors who are taller than 5.2
classmates |> filter(major == 'CS', height > 5.2)

# from classmates, select column name
classmates |> select(name)

# from classmates, select columns name, gender, major
classmates |> select(name, gender, major)

# from classmates, transform height from ft to cm
classmates |> mutate(height = height * 30.48)

# from classmates, arrange data by gender
classmates |> arrange(gender)

# from classmates, arrange data by gender then by age
classmates |> arrange(gender, age)

# from classmates, summarize height 
classmates |> summarize(mean_height = mean(height))

# from classmates, group the data by gender
classmates |> group_by(gender)

However, the pipe operators are most powerful with step-by-step computations.

# retrieve all male classmates
classmates |>
  filter(gender == 'male')

# from those male classmates, look at their height, weight, and major
classmates |>
  filter(gender == 'male') |>
  select(height, weight, major)

# then sort them by height
classmates |>
  filter(gender == 'male') |>
  select(height, weight, major) |>
  arrange(height)

💻 Hands-On

Use the pipe operators to complete the following step-by-step computations

# retrieve all CS majors


# from those CS majors, look at their name, gender, and age


# then sort them by gender


# then group them by gender


# then obtain their mean weight


# with pipe operators

Answer

# retrieve all CS majors
classmates |> 
  filter(major == 'CS')

# from those CS majors, look at their name, gender, and age
classmates |> 
  filter(major == 'CS') |>
  select(name, gender, age)

# then sort them by gender
classmates |> 
  filter(major == 'CS') |>
  select(name, gender, age) |>
  arrange(gender)

# then group them by gender
classmates |> 
  filter(major == 'CS') |>
  select(name, gender, age) |>
  arrange(gender) |>
  group_by(gender)

# then obtain their mean age
classmates |>  
  filter(major == 'CS') |>
  select(name, gender, age) |>
  arrange(gender) |>
  group_by(gender) |>
  summarize(mean_age = mean(age))

Readings

Topics

dplyr and the grammar of data manipulation

classmates data set

slice() to subset rows

filter() to select rows with conditions

select() to subset columns (variables)

mutate() to transform, add, or delete variables

arrange() to order rows

summarize() to summarize values

group_by() to divide a data set by variables

The pipe %>% and |> operators

`dplyr` and the grammar of data manipulation

`classmates` data set

`slice()` to subset rows

`filter()` to select rows with conditions

`select()` to subset columns (variables)

`mutate()` to transform, add, or delete variables

`arrange()` to order rows

`summarize()` to summarize values

`group_by()` to divide a data set by variables

The pipe `%>%` and `|>` operators