```{r setup, include=FALSE}
library(tidyverse)
library(conflicted)

filter <- dplyr::filter

knitr::opts_chunk$set(echo = TRUE)
```

## 2020-07-14 Reading Data with `readr` and Tidying/Wrangling Data with `tidyr`

DESCRIPTION HERE

### `readr`

Up until now, we have been using data that comes with R or R packages. But when you start an analysis, you'll have data that you want to work with and you need to import it into R so you can do stuff to it. The tidyverse has a package `readr` for reading and writing files; to read files in you use the `read_*()` set of functions. To select the appropriate function, you need to know what your plain text file is delimited by, what character is used to separate the data. Then you can select the appropriate one of the three functions below:

- `read_delim()` = can read in any plain text file if you supply the delimiter
  - `read_csv()` = reads in plain text files separated by commas (csv = comma separated value)
  - `read_tsv()` = reads in plain text files separated by tabs, written as "\t" in plain text (tsv = tab separated value)
  
---

You can read a file in using either the file path or you can read a file using a URL. Since this is a CSV file (you can tell because the file ends in `.csv`), we could used `read_csv()` to read it in, but we'll use `read_delim()` instead to demonstrate.

```{r}
read_delim("https://raw.githubusercontent.com/WSJ/measles-data/master/all-measles-rates.csv",
           delim = ',') -> measles
```

Let's make a change and then save the table to our local machine using `write_tsv()`

```{r}
measles %>%
# Forgot to mention this in the dplyr demo; you can drop a column by putting 
# a - in front of the column name in select()
  select(-index) -> measles2

# To save, give the name of the table and the file name you'd like to save it as
write_tsv(measles2, 'measles_vaccination_rates.tsv')
```

Read the file we've just written back in.

```{r}
# If you don't know which file you want, you can use dir() to see the contents
# of the working directory
dir()

# Read the file back in
read_tsv('measles_vaccination_rates.tsv') -> measles_read
```

### `tidyr`

Data seldom follows tidy data principles, so the tidyverse development team created the `tidyr` package. (You can read more about it at its website <https://tidyr.tidyverse.org/>.) If you look at the table we read into R by running it in the chunk below, you can see that there are two ways it isn't tidy. First, each sample is it's own column and since "sample" is am individual variable that violates the tidy principle that each variable is a column. Second, each cell is a collection of four separate values separated by underscores `_`, which violates the tidy principle that cell should be a single value. We'll use `tidyr` functions to fix this.

---

The first example data table from tidyr is the cleverly named "table1". This table is tidy! The other tableNs in this dataset are untidy versions of this one to practice on. 

```{r}
table1
```

As you can see when you look at table1, each column is a variable and contains only values of the same type belonging to that variable. Going foward, we'll be tidying messed up versions of table1

### `pivot_wider()`

If you look at table 2 in the chunk below, it's not tidy because the column 'type' contains two variables, cases and population. Cases and population are separate measurements, so they should be their own columns. To make more columns from existing columns, use `pivot_wider()`. 

```{r}
table2

# You need to specify the column that new column names will come from (names_from)
# and the column the new values will come from (values_from)
table2 %>% pivot_wider(names_from = type, values_from = count)
```

### `pivot_longer()`

Looking at table4a in the chunk below, it's not tidy because the columns 1999 and 2000 are not variable descriptions, they're data (the year the measurement was made in). We need the column names to be values in a column. To do that we'll use `pivot_longer()`.

```{r}
table4a

# You need to pick the columns that will be used to fill in the two new columns
# and you need to specify the names of the two columns being created. Also, if
# you have columns that aren't named with words or have special characters in
# them you can use backticks (``) to make tidyverse/R treat them as column names
table4a %>% pivot_longer(names_to = 'year', values_to = 'cases', `1999`:`2000`)

# Instead of using the name of the column, you can use the number of the column
table4a %>% pivot_longer(names_to = 'year', values_to = 'cases', 2:3)
```

### `separate()`

In the chunk below, table3 is not tidy because it has multiple values separate by a backslash in the rate column. To split one column into two or more columns, use `separate()`

```{r}
table3

# You need to specify the column you're separating and the names of the columns 
# for after the separation. The other two arguments are optional, but a very 
# good idea. The sep argument tells what character to split on and convert checks
# the new data and changes the type of the column if necessary.
table3 %>% separate(rate, into = c('cases', 'population'), sep = '/', convert = TRUE)
```

`separate()` also lets you specify an interger position to split at

```{r}
# just so you know, you can also separate by specifying a position to separate after
table3 %>% separate(rate, into = c('cases', 'population'), sep = 3)
```

### `unite()`

In the chunk below, table5 is not tidy because the date is split between two columns, century and year. To combine columns, use the opposite of `separate()`, `unite()`

```{r}
table5

# You need to specify the name of the new column, the columns to join and you
# might want to specify how you separate them in the new column (_ is default)
table5 %>% unite(year, c('century', 'year'), sep = '')
```

### Combing `tidyr` Functions

You may have noticed that the rate column in table5 is the same as the rate column in table3 and contains two values. Chain together some of the tidyr functions above to completely tidy table5.

```{r}
table5 %>% 
  unite(year, c('century', 'year'), sep = '') %>% 
  separate(rate, into = c('cases', 'population'), sep = '/')
```

<br>

### Combining Tables: Binds and Joins

Binds and joins are in the dplyr package, but fit in better with data wrangling, so they were saved for this demo.

#### Binds

Binds attach tables together. In order to bind tables, the tables being bound together have to have the same number of rows OR the same number of columns (depending if you're binding by rows or columns).

#### `rbind()` / `bind_rows()`

Attach tables together by adding additional rows. When you bind by rows, the second table is added to the bottom of the first table. 

```{r}
table4a
table4b

# base R
rbind(table4a, table4b)

# not same number of columns doesn't work
rbind(table4a, table1)

# tidyverse, dplyr
bind_rows(table4a, table4b)

# not same number of columns gives NAs
bind_rows(table4a, table1)
```

The difference between `rbind()` and `bind_rows()` is that if you have a different number of columns, `rbind()` will refuse to work and return an error, while `bind_rows()` will keep all columns from all tables and add NAs where there are no values.

#### `cbind()` / `bind_cols()`

Attach tables by adding additional columns.

```{r}
table4a
table4b

# base R
cbind(table4a, table4b)

# not same number of rows, rows are recycled (see how the stuff from table4a is repeated)
cbind(table4a, table1)

# tidyverse, dplyr
bind_cols(table4a, table4b)
# notice that bind_cols() adds a 1 to columns that are repeated so that each column has a unique name

# not same number of rows doesn't work
bind_cols(table4a, table1)
```

#### Joins

Joins combine two tables together by common variables. The difference between binds and joins is that binds are intended to combine tables together that have exactly the same information, like for different samples, while joins are intended to combine together tables with complimentary but different information by common variables, like samples with their metadata. The `*_join()` functions combine either by the columns you specify or by auto-detecting columns with the same name. Mutating joins add additional columns, while filtering joins filter a table by another table.

We'll use the `band_*` tables from the `dplyr` package to practice joins. As you can see when you run the tables in the chunk below, the tables have information in common, musician names, but complimentary information, what band they belong to in one table and what instruments they play in the others.

```{r}
band_members
band_instruments
band_instruments2
```

The descriptions below refer to x and y in the join. x is the first table listed in the join and y is the second. The order tables are given in joins is important!

#### Mutating Joins

Mutating joins add columns onto a table from another table.

##### `inner_join()`

Joins tables by returning all rows where x and y equal each other (in the join columns) and all columns from both x and y.

```{r}
band_members
band_instruments
inner_join(band_members, band_instruments, by = c('name'))
```

##### `full_join()`

The opposite of `inner_join()`, `full_join()` joins tables by returning all rows all columns from both x and y. If there is no matching value between x and y, an NA will be returned.

```{r}
band_members
band_instruments
full_join(band_members, band_instruments, by = c('name'))
```

##### `left_join()`

Joins tables by return all rows from x and all columns from x and y. If there is no matching value in y for x, an NA will be returned.

```{r}
band_members
band_instruments
left_join(band_members, band_instruments, by = c('name'))

# order matters
left_join(band_instruments, band_members, by = c('name'))
```

##### `right_join()`

Joins tables by the opposite of `left_join()`, returning all rows from y and all columns from x and y. If there is no matching value in x for y, an NA will be returned.

```{r}
band_members
band_instruments
right_join(band_members, band_instruments, by = c('name'))
```

#### Filtering Joins

Filtering joins filter a table by another table.

#### `semi_join()`

Filters x by returning all rows from x that match values in y.

```{r}
band_members
band_instruments
semi_join(band_members, band_instruments, by = c('name'))
```

#### `anti_join()`

The opposite of `semi_join()`, `anti_join()` returns rows from x that DON'T match a value in y

```{r}
band_members
band_instruments
anti_join(band_members, band_instruments, by = c('name'))
```

#### What if my column names don't match?

Name and artist are the same variable, who the person is, in the tables below.

```{r}
band_members
band_instruments2
```

To specify the join by two differently named columns, specify that they equal each other. You MUST list the column names in the order the tables are given to the join. 'name' comes first below, because that's the column you want to join by in the first table.

```{r}
full_join(band_members, band_instruments2, by = c('name' = 'artist'))

# if there are no common names, it won't work if you don't tell the join that two columns are the same
full_join(band_members, band_instruments2)
```

### Dealing with NAs

If there's no value in a table, or R can't figure out what's supposed to be there, it will give you an `NA` as seen in the Star Wars table below

```{r}
starwars
```

NAs are bad not only because you're missing data, but also because many functions in R either refuse to run or return weird answers if there are NAs in the data. 

```{r}
mean(starwars$birth_year)
```

#### `na.omit()`

One strategy is just to drop all rows that contain NAs

```{r}
starwars
starwars %>% na.omit()
```

#### `replace_na()`

Sometimes you might want to replace NAs with a value though, which you can do using tidyr

```{r}
starwars
starwars %>% replace_na(list(hair_color = 'none', birth_year = 'unknown', gender = 'n/a'))
```


