### Scraping web data in unstructured format

A common scenario for web scraping is when the data we want is available in plain html, but in different parts of the web, and not in a table format. In this scenario, we will need to find a way to extract each element, and then put it together into a data frame manually.

The motivating example here will be the website `ipaidabribe.com`, which contains a database of self-reports of bribes in India. We want to learn how much people were asked to pay for different services, and by which departments.

```{r}
url <- 'http://ipaidabribe.com/reports/paid'
```

We will also be using `rvest`, but in a slightly different way: prior to scraping, we need to identify the CSS selector of each element we want to extract. 

A very useful tool for this purpose is `selectorGadget`, an extension to the Google Chrome browser. Go to the following website to install it: `http://selectorgadget.com/`. Now, go back to the ipaidabribe website and open the extension. Then, click on the element you want to extract, and then on the rest of highlighted elements that you do __not__ want to extract. After only the elements you're interested in are highlighted, copy and paste the CSS selector into R.

Now we're ready to scrape the website:

```{r}
library(rvest, warn.conflicts=FALSE)
bribes <- read_html(url) # reading the HTML code
amounts <- html_nodes(bribes, ".paid-amount span") # identify the CSS selector
amounts # content of CSS selector
html_text(amounts)
```

We still need to do some cleaning before the data is usable:

```{r}
amounts <- html_text(amounts)
(amounts <- gsub("Paid INR | |\r|\n|,", "", amounts)) # remove text, white space, and commas
(amounts <- as.numeric(amounts)) # convert to numeric
```

Let's do another one: transactions during which the bribe ocurred
```{r}
transaction <- html_nodes(bribes, ".transaction a")
(transaction <- html_text(transaction))
```

And one more: the department that is responsible for these transactions
```{r}
# and one more
dept <- html_nodes(bribes, ".name a")
(dept <- html_text(dept))
```

This was just for one page, but note that there are many pages. How do we scrape the rest? First, following the best practices on coding, we will write a function that takes the URL of each page, scrapes it, and returns the information we want.

```{r}
scrape_bribe <- function(url){
	bribes <- read_html(url)
	# variables that we're interested in
	amounts <- html_text(html_nodes(bribes, ".paid-amount span"))
	amounts <- as.numeric(gsub("Paid INR | |\r|\n|,", "", amounts))
	transaction <- html_text(html_nodes(bribes, ".transaction a"))
	dept <- html_text(html_nodes(bribes, ".name a"))
	# putting together into a data frame
	df <- data.frame(
		amounts = amounts,
		transaction = transaction,
		dept = dept,
			stringsAsFactors=F)
	return(df)
}
```

And we will start a list of data frames, and put the data frame for the initial page in the first position of that list.

```{r}
bribes <- list()
bribes[[1]] <- scrape_bribe(url)
str(bribes)
```

How should we go about the following pages? Note that the following urls had `page=XX`, where `XX` is 10, 20, 30... So we will create a base url and then add these additional numbers. (Note that for this exercise we will only scrape the first 5 pages.)

```{r}
base_url <- "http://ipaidabribe.com/reports/paid?page="
pages <- seq(0, 40, by=10)
```

And now we just need to loop over pages, and use the function we created earlier to scrape the information, and add it to the list. Note that we're adding a couple of seconds between HTTP requests to avoid overloading the page, as well as a message that will informs us of the progress of the loop.

```{r}
for (i in 2:length(pages)){
	# informative message about progress of loop
	message(i, '/', length(pages))
	# prepare URL
	url <- paste(base_url, pages[i], sep="")
	# scrape website
	bribes[[i]] <- scrape_bribe(url)
	# wait a couple of seconds between URL calls
	Sys.sleep(2)
}
```

The final step is to convert the list of data frames into a single data frame that we can work with, using the function `do.call(rbind, LIST)` (where `LIST` is a list of data frames).

```{r}
bribes <- do.call(rbind, bribes)
head(bribes)
str(bribes)
```

Let's get some quick descriptive statistics to check everything worked. First, what is the most common transaction during which a bribe was paid?

```{r}
tab <- table(bribes$transaction) # frequency table
tab <- sort(tab, decreasing=TRUE)	# sorting the table from most to least common
head(tab)
```

What was the average bribe payment?

```{r}
summary(bribes$amount)
```

And what was the average payment for each department?
```{r}
agg <- aggregate(bribes$amount, by=list(dept=bribes$dept), FUN=mean)
agg[order(agg$x, decreasing = TRUE),] # ordering from highest to lowest
```




