The new dplyr and workarounds for the old one

Introduction

dplyr is the part of tidyverse that is concerned with choosing things from dataframes and reorganizing dataframes. It recently underwent an upgrade (to version 1.2 from version 1.1). The upgrade is definitely an improvement. You can read about some of the thinking behind the upgrade here. However, the version of R (and dplyr) on r.datatools has not kept pace,1 and until it is updated, you won’t be able to use these new functions there:

  • filter_out
  • when_any and when_all
  • recode_values, replace_values, replace_when

Here, I will tell you about the (less satisfactory) workarounds that will still work on the version of R that is on r.datatools. I will try to mimic the examples from lecture (on the Australian athletes data).

Packages and data

library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
my_url <- "http://ritsokiguess.site/datafiles/ais.txt"
athletes <- read_tsv(my_url)
Rows: 202 Columns: 13
── Column specification ────────────────────────────────────────────────────────
Delimiter: "\t"
chr  (2): Sex, Sport
dbl (11): RCC, WCC, Hc, Hg, Ferr, BMI, SSF, %Bfat, LBM, Ht, Wt

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
athletes

The athletes data, again.

Filtering out: filter_out

filter_out provides a natural way of saying “leave out these rows but give me all the others”. Using filter requires you to phrase things in terms of the rows you want to keep, which means reversing the logic in your condition. Here’s the example from lecture:

athletes %>% filter_out(Sport == "Tennis")

This gives us all the athletes except the tennis players. To express this with filter, think of it as “I want to keep all the rows where the sport is not equal to Tennis”. “Not equal” in R is !=:2

athletes %>% filter(Sport != "Tennis")

You can check that we get the same athletes both ways. The filter code is actually shorter, but it is less clear: your reader now has to work to figure out that we got rid of the tennis players (by keeping everyone except the tennis players).

Filtering on multiple things: when_all and when_any

In lecture, we saw this:

athletes %>% filter(when_all(
  Sport == "Tennis",
  RCC < 5))

This gives us all the tennis players whose RCC is less than 5: logically, “the athlete’s sport is tennis and their RCC is less than 5”. The way you specify an “and” without using when_all is a filter with multiple things:

athletes %>% filter(Sport == "Tennis", RCC < 5)

This is definitely shorter but also definitely less clear: how do you or your reader know that this means “and” and not something else, such as “or”? An alternative is two filters one after the other:

athletes %>% 
  filter(Sport == "Tennis") %>% 
  filter(RCC < 5)

This is a bit longer but a lot clearer: “give me all the tennis players, and then out of those, give me the ones whose RCC is less than 5”.

In lecture, we also saw when_any:

athletes %>% filter(when_any(
  Sport == "Tennis",
  RCC > 5
))

This shows us the athletes for whom any of these things are true: that is, either they play tennis, or their RCC is bigger than 5 (or both3).

To do this without using when_any, you need a way of saying “or”, which is |:

athletes %>% 
  filter(Sport == "Tennis" | RCC > 5)

Again, shorter, but less clear: you have to remember that | means “or”.

The last one of these we had was females whose RCC is bigger than 5, or males whose RCC is bigger than 6:

athletes %>% 
  filter(when_any(
    when_all(
      Sex == "female",
      RCC > 5
    ),
    when_all(
      Sex == "male",
      RCC > 6
    )
  ))

This one is a headache to do any other way. I think the shortest way is this:

athletes %>% 
  filter((Sex == "female" & RCC > 5) | 
         (Sex == "male"   & RCC > 6))

I couldn’t see a reasonable way to do this in this way without also using the shorthand & for “and”, which I was hoping to avoid. Again, shorter, but less clear to my mind (and the any and all help remind you about what is what: “when any of the following things are true” and so on). It is helpful here to put brackets around the constituent pieces; I think R does “and” before “or”, but it is much more helpful for you (and your reader) to be explicit about things.

Recoding and replacing values

Doing things without using recode_values, replace_values and replace_when involves using case_when with less convenience and more typing, and also paying attention to default values.

recode_values

Here’s how we recoded Sex to numeric values 1 and 2:

athletes %>% 
  mutate(sex_code = recode_values(
    Sex,
    "female" ~ 1,
    "male"   ~ 2
  )) %>% 
  select(Sex, sex_code)

and here’s how it goes with case_when:

athletes %>% 
  mutate(sex_code = case_when(
    Sex == "female" ~ 1,
    Sex == "male"   ~ 2
  )) %>% 
  select(Sex, sex_code)

We have to repeat Sex every time as the price we pay for case_when being more flexible (and thus more general) but less suited to this particular situation.

This was the next one we did:

athletes %>% 
  mutate(where_played = recode_values(
    Sport,
    c("BBall", "Netball", "Tennis") ~ "court",
    c("Row", "Swim", "WPolo")       ~ "water",
    c("Field", "T400m", "TSprnt")   ~ "track&field",
    default                         = Sport
  )) %>% 
  select(Sport, where_played)

There are two ways to approach this using case_when: one sport at a time (which means repeating things like court), or by using a new technique which I’ll get to shortly (now that I write this, there are a couple of other ways I can think of). Here’s the one-at-a-time approach, which is very repetitious:

athletes %>% 
  mutate(where_played = case_when(
    Sport == "BBall"   ~ "court",
    Sport == "Netball" ~ "court",
    Sport == "Tennis"  ~ "court",
    Sport == "Row"     ~ "water",
    Sport == "Swim"    ~ "water",
    Sport == "WPolo"   ~ "water",
    Sport == "Field"   ~ "track&field",
    Sport == "T400m"   ~ "track&field",
    Sport == "TSprnt"  ~ "track&field",
    .default           = Sport
  )) %>% 
  select(Sport, where_played)

It was painful to type all that, and I copy-pasted most of it!

Something that is more along the lines of what we did with recode_values uses the %in% operator. This means “in a set” or “is a member of a set”. Using this, we can do the trios of sports in one shot each:

athletes %>% 
  mutate(where_played = case_when(
    Sport %in% c("BBall", "Netball", "Tennis") ~ "court",
    Sport %in% c("Row", "Swim", "WPolo")       ~ "water",
    Sport %in% c("Field", "T400m", "TSprnt")   ~ "track&field",
    .default                                   = Sport
  )) %>% 
  select(Sport, where_played)

%in% returns TRUE or FALSE depending on whether the thing on the left (an athlete’s sport) is found somewhere in the collection of things on the right. case_when checks each of the conditions in turn, and the first one that returns TRUE is the one that gives its value to the new column. For example, if Sport is Swim, the first one of these lines to return TRUE is the second one, and in that case where_played gets the value water.

I said I had a third way. This is taking us off course rather, but so it goes. There is a package forcats (part of the tidyverse) that is for doing things with factors, that is to say, categorical variables. Sport is a categorical variable, and what we want to do is to combine some of its levels (categories) together, which is what fct_collapse does.4 Here’s how it goes:

athletes %>% 
  mutate(where_played = fct_collapse(
    Sport,
    court         = c("BBall", "Netball", "Tennis"),
    water         = c("Row", "Swim", "WPolo"),
    `track&field` = c("Field", "T400m", "TSprnt")
  )) %>% 
  select(Sport, where_played)

This is actually easier than I was expecting, but you’ll notice how things are backwards compared with recode_values (the new values are now on the left).

replace_values

Working around replace_values and replace_when, instead using case_when, means that we have to go back to dealing with defaults, because case_when expects us to have a new value for every one of the old values.

Here’s our replace_values:

athletes %>% 
  mutate(Sport = replace_values(
    Sport,
    c("T400m", "TSprnt") ~ "Track"
    ))

This was short because we were only making a few changes. Using case_when, we have to deal with the other possibilities, but it turns out not to be too bad:

athletes %>% 
  mutate(Sport = case_when(
    Sport %in% c("T400m", "TSprnt") ~ "Track",
    .default                        = Sport
    ))

I used the %in% idea here, but you could go with one sport at a time, or in fact fct_collapse works about as well as the replace_values does:

athletes %>% 
  mutate(Sport = fct_collapse(
    Sport,
    Track = c("T400m", "TSprnt")
    ))

because you only need to specify the ones that change: the others will be left alone.

replace_when

The last thing we need to worry about is replace_when. Our example in lecture was this:

athletes %>% 
  mutate(RCC = replace_when(
    RCC,
    RCC < 4   ~ NA,
    RCC > 4.5 ~ 4.5
  ))

With case_when, this one needs a default:

athletes %>% 
  mutate(RCC = case_when(
    RCC < 4   ~ NA,
    RCC > 4.5 ~ 4.5,
    .default  = RCC
  ))

where the last line says “any other values of RCC, leave them as they are.5”

Other things

The other things we saw in lecture were part of dplyr version 1.1, such as rows_update and the joins, so you should have no problems running those. (Your problems are more likely to be along the lines of figuring out what to do, rather than doing it once you have figured that out.)

Footnotes

  1. It is still on version 1.1 of dplyr.↩︎

  2. This is the same as in Python.↩︎

  3. In mathematics and coding, “or” includes “or both”. If you want the tennis players or the athletes with RCC greater than 5 but not both, you need an “exclusive or”.↩︎

  4. I don’t use forcats much, so I had to look up the syntax.↩︎

  5. Or, I suppose, ”replace them with the original value of RCC”, which is the same as saying ”don’t change them”.↩︎