Extracting media text from newspaper's websites is a very frequent task in webscraping. One advantage of these sites is that they tend to offer an RSS feed that contains a list of all the stories they have published, which we can then use to more efficiently scrape them.

Parsing RSS feeds requires we learn a slightly different data format: XML, or eXtensible Markup Language, which predates (but is similar to) JSON. Just like HTML, it uses a series of tags and a tree structure. We will use the `xml2` and `rvest` packages to read data in XML format:

Let's look at an example:

```{r}
feed <- "http://www.spiegel.de/politik/index.rss"
library(xml2)
library(rvest)
rss <- read_xml(feed)
substr(as.character(rss), 1, 1000)
```

Just like with HTML, we can extract specific nodes of the XML file using a combination of `xml_nodes` and `xml_text`

```{r}
headlines <- xml_nodes(rss, 'title')
(headlines <- xml_text(headlines))
urls <- xml_nodes(rss, 'link')
(urls <- xml_text(urls))
```

Once we have the article URLs, we could go page by page, looking at their internal structure, and then scraping it. However, some packages exist that already compile a set of scrapers that generally work with any type of newspaper website -- one of these is `boilerpipeR`. It uses a combination of machine learning and heuristics to develop functions that should work for any newspaper website. Let's see how it works in this case:

```{r}	
library(boilerpipeR)
# read first URL -- note that all text needs to be into a single character vector
text <- readLines(urls[3])
text <- paste(text, collapse="\n")
# now let's try to parse it..
main_text <- ArticleExtractor(text)
cat(main_text)
```

Once we have prototype code, the last step is to generalize using a loop that will iterate over URLs.

```{r}
articles <- list()
for (i in 1:length(urls)){

	message(i, " of ", length(urls))
	text <- paste(readLines(urls[i]), collapse="\n")
	main_text <- ArticleExtractor(text)
	articles[[i]] <- data.frame(
		url = urls[i],
		headline = headlines[i],
		text = main_text,
		stringsAsFactors=F)

}
articles <- do.call(rbind, articles)

```


Of course, some times this standardized code will not work with specific websites. In those cases, it's easier to just develop our own code. Here I show you an example written by one of the students in my lab, Anthony Ramos.

The following two blocks of code to the following: 1) download the home page of the Spiegel in html format, 2) extract the URLs of all articles linked in the home page, and then download those, and finally 3) parse the html code in those pages.

```{r}

scrapeSpiegelOnline <- function(path) {

  html <- download.file("http://www.spiegel.de/",
    destfile=path)
  doc <-read_html(path)

  #get main articles
  title=html_nodes(doc,".article-title a")
  titles = xml_attr(title,"title")
  title_links = xml_attr(title,"href")

  title_links <- ifelse( grepl("https?://", title_links),
    title_links, paste0("http://www.spiegel.de", title_links) )

  df <- data.frame(title=titles, url=title_links, 
                   time=as.character(Sys.time()), stringsAsFactors=F)
  df <- df[!is.na(df$url),]
  df <- df[!duplicated(df$url),]
  
  return(df)
}


scrapeSpiegelArticle <- function(url, path) {
  
  html <- download.file(url, destfile=path, quiet=TRUE)

  article <- read_html(path)
  article_intro <- html_text(html_nodes(article,".headline-intro"))[1]
  article_title <- html_text(html_nodes(article,".headline"))[1]
  article_title <- paste(article_intro,article_title,sep=": ")
  
  date <- html_text(html_nodes(article,".article-function-date"))[1]
  date <- gsub("\t|\n|\r", "", gsub("\r", "", date))
  
  content <- html_text(html_nodes(article,"p"))
  summary <- content[1]
  
  content <- paste(content, collapse="\n\n")
  content <- gsub("^ *| *$", "", gsub("\n|\t|\r", "", content))
  
  summary <- gsub("^ *| *$", "", gsub("\n", "", summary))
  summary <- gsub(" {2,}", " ", summary)
  
  comments <- grep("insgesamt (.*) Beiträge",html_text(html_nodes(article, "span")),value=TRUE)
  comments <- gsub("^ *", "", gsub("\r|\n|\t", "", comments))
  
  
  article_df <- data.frame(url=url, date=date, summary=summary,
                           headline=article_title, text=content,
                           comments=ifelse(length(comments)==0, NA, comments),
                           stringsAsFactors=F)
                           
  return(article_df)

}

```


```{r}

today <- Sys.Date()

# filename for homepage, in html and csv format
homepage <- paste0("home-", today, ".html")
homepagecsv <- paste0("articles-", today, ".csv")

# folder where home articles will be stored
artfolder <- paste0("home-articles")
try(dir.create(artfolder, showWarnings=FALSE))

# scraping homepage and saving .html to "homepage" file
headlines <- scrapeSpiegelOnline(path=homepage)

# scraping 10 first articles linked in the homepage
articles <- list()
for (i in 1:10){
	message(headlines$title[i])
  artname <- paste0(artfolder, "/", 
		gsub("/", "_", gsub("http://www.spiegel.de/", "", headlines$url[i])) )
	# download html file and parse it
  error <- tryCatch(
		articles[[i]] <- scrapeSpiegelArticle(
			url = headlines$url[i], path=artname),
		error=function(e) e)
	if (inherits(error, "error")){ message("Error")}
	Sys.sleep(1)
}

articles <- do.call(rbind, articles)

# merging
headlines <- merge(headlines, articles, by="url", all.x=TRUE, sort=FALSE)

# writing to disk
write.csv(headlines, file=homepagecsv, row.names=FALSE)

```

