## Merging datasets

Two or more data frames can be combined into a single one according to the values of a single variables using the `merge` function.

```{r}
students <- data.frame(name = c("Paul", "Peter", "Amy", "Megan", "Selena"),
                age = c(18, 19, 18, 20, 21),
                stringsAsFactors=FALSE)
grades <- data.frame(name = c("Amy", "Megan", "Peter", "Drew"),
                     grade = c("A", "B+", "A-", "B"),
                     stringsAsFactors=FALSE)

merge(students, grades)
# we can be more specific about how to merge
merge(students, grades, by="name")

# if the common variable has different names, we need to specify it
grades <- data.frame(student = c("Amy", "Megan", "Peter", "Drew"),
                     grade = c("A", "B+", "A-", "B"),
                     stringsAsFactors=FALSE)
merge(students, grades, by.x="name", by.y="student")

```

Note that by default `merge` will only return rows that are present in both dataset; but we can change that default behavior. For the non-matches, the returned value will be `NA`.

```{r}
merge(students, grades, by.x="name", by.y="student", all.x=TRUE)
merge(students, grades, by.x="name", by.y="student", all.y=TRUE)
merge(students, grades, by.x="name", by.y="student", all=TRUE)
```

What happens if names do not have exactly the same spelling? We can try some fuzzy string matching based on the Levenshtein (or edit) distance between strings. See `?adist`

```{r}
students <- data.frame(name = c("Paul", "Pete", "Amie", "Meghan", "Selena"),
                age = c(18, 19, 18, 20, 21),
                stringsAsFactors=FALSE)
grades <- data.frame(name = c("Amy", "Megan", "Peter", "Drew"),
                     grade = c("A", "B+", "A-", "B"),
                     stringsAsFactors=FALSE)

# probably there are better ways to code this...
students$grade <- NA

for (i in 1:nrow(students)){
  dist <- adist(students$name[i], grades$name)
  if (any(dist<4)){
    students$grade[i] <- grades$grade[which.min(dist)]
  }
}

students

students <- data.frame(name = c("Paul", "Peter", "Amy", "Megan", "Selena"),
                age = c(18, 19, 18, 20, 21),
                stringsAsFactors=FALSE)
merge(students, grades)

```

To learn more about merging datasets based on strings, see the `RecordLinkage` package.

Another package that is very useful is `countrycode`, which contains a data frame with pretty much every potential country code you may want to use.

## Reshaping datasets

We often want to convert the format in which a data frame is structured. For example, we may to compute the average value of a variable, grouped by values of a different variable. Let's see some examples using the `aggregate` function: 


```{r}
d <- read.csv("~/data/unvoting.csv", stringsAsFactors=F)

aggregate(idealpoint ~ CountryAbb, data=d, FUN=mean)

aggregate(idealpoint ~ CountryAbb, data=d, FUN=median)

aggregate(PctAgreeRUSSIA ~ Year, data=d, FUN=mean)
plot(aggregate(PctAgreeRUSSIA ~ Year, data=d, FUN=mean), type="l")

aggregate(idealpoint ~ CountryAbb, data=d, FUN=length)

```

The other type of reshaping process is transforming a dataset from long (our preferred format!) to wide; and the other way around.

```{r}
dd <- d[d$CountryAbb %in% c("HUN", "AUS") & d$Year %in% 2010:2012, 
        c("Year", "CountryAbb", "idealpoint")]

# converting to wide format
library(reshape)
(dc <- cast(dd, CountryAbb ~ Year, value="idealpoint"))

# converting back to long format
melt(dc, id.vars=c("CountryAbb", "Year"))
```


## Error handling

When collecting data from the web, we need to expect the best but always prepare for the worst. You don't want to leave your computer scraping data overnight just to wake up and discover there was an error with one the websites you were scraping and no data was downloaded.

To deal with these issues, we can use some of the options that R offers to handle errors.

```{r, eval=FALSE}
# this will give an error
for (i in 1:10){
  message(i)
  readLines("file.json")
}
```

We can use `try` to ignore any error that the function may return.

```{r}
# try, but not saving output of error
for (i in 1:10){
  message(i)
  try(readLines("file.json"))
}
```

But generally, `tryCatch` is better because it will let us store the error message and react in response to it

```{r}
for (i in 1:10){
  message(i)
  error <- tryCatch(readLines("file.json"), error=function(e) e)
  if (grepl("Error in file", as.character(error))){
    message("Error! Maybe the file does not exist?")
  }
}
```

Let's see it in the wild...

```{r, eval=FALSE}
library(tweetscores)
accounts <- c("realDonaldTru", "HillaryClinton", "BernieSanders", "tedcruz")
load("~/my_oauth")

while (length(accounts) > 0){
  # sample randomly one account to get tweets
  new.user <- sample(as.character(accounts), 1)
  message(new.user)

  error <- tryCatch(friends <- getFriends(screen_name=new.user,
                                          oauth=my_oauth, 
                                          verbose = TRUE), error=function(e) e)
  if (inherits(error, 'error')) {
    cat("Error! On to the next one...")
    next
  }
  fname <- paste0("~/data/", new.user, '.rdata')
  save(friends, file=fname)
  accounts <- accounts[-which(accounts %in% new.user)]
}



```








