## Basics of character encoding in R

Detecting the encoding of your system.

```{r}
Sys.getlocale(category = "LC_CTYPE")
```

Let's now consider some text in German that contains a non-ASCII character.

```{r}
# some text in German
de <- "Einbahnstraße"
# all good!
message(de)
de
```

This worked because the file where we are writing is in UTF-8 encoding so it automatically recognizes the encoding.
```{r}
Encoding(de)
```

But what if the file is not in UTF-8 and when we save it and re-open it, it looks like this? As long as we set the right encoding, we can switch back and forth. 

```{r}
de <- "Einbahnstra\u00dfe"
Encoding(de)
message(de)
# this is the wrong encoding
Encoding(de) <- "latin1"
message(de)
# now back to the right encoding
Encoding(de) <- "UTF-8"
message(de)
```

We can also use the stringi package to fix this
```{r}
library(stringi)
stri_unescape_unicode("Einbahnstra\u00dfe")
```

If you want to translate a string from one encoding scheme to another in a single line of code, you can use `iconv`:

```{r}
de <- "Einbahnstra\xdfe"
iconv(de, from="windows-1252", to="UTF-8")
de <- "Einbahnstra\u00dfe"
iconv(de, from="UTF-8", to="latin1")
```

You're probably wondering now - how do we know the encoding of some text we want to analyze? Good question! Turns out it's a hard problem, but we can use the `guess_encoding` question in the `rvest` package (which uses `stri_enc_detect` in the `stringi` package) to try to figure that out...

```{r}
library(rvest)
de <- "Einbahnstra\xdfe"
stri_enc_detect(de)
guess_encoding(de)
iconv(de, from="ISO-8859-1", to="UTF-8")

de <- "Einbahnstra\u00dfe"
stri_enc_detect(de)
guess_encoding(de)
message(de) # no need for translation!
```

The same applies to websites... (Although you can also check the `<meta>` tag for clues.)

```{r}
url <- "http://www.presidency.ucsb.edu/ws/index.php?pid=96348"
guess_encoding(url)
url <- "http://www.spiegel.de"
guess_encoding(url)
url <- "http://www.elpais.es"
guess_encoding(url)
```


## Dealing with Unicode headaches

Unicode text can take different, and somewhat complicated, forms when you scrape it from the web. Here we'll see some of the most common and how to avoid getting errors when we parse text scraped from the web. We'll be using the `stringi` package for some of the code here.

```{r}
# what if it looks like this? (Unicode characters as HEX/bite codes)
# see: http://www.fileformat.info/info/unicode/char/00df/index.htm
de <- "Einbahnstra<c3><9f>e"
# this will not work:
guess_encoding(de)
iconv(de, from="ISO-8859-1", to="UTF-8")
stri_unescape_unicode(de)

# one solution from stack overflow:
# https://stackoverflow.com/questions/25468716/convert-byte-encoding-to-unicode
m <- gregexpr("<[0-9a-f]{2}>", de)
codes <- regmatches(de,m)
chars <- lapply(codes, function(x) {
    rawToChar(as.raw(strtoi(paste0("0x",substr(x,2,3)))), multiple=T)
})
regmatches(de,m) <- chars
de

# what is happening here? We're replacing:
codes
# with:
chars

# switching to a different language...
# what if it looks like this?
example <- c(
  "SAD DA POMOGNU RJE<U+0160>AVANJE POLITI<U+010C>KE KRIZE", 
  "PROBLEME GURAJU POD TEPIH", 
  "ODAO PRIZNANJE DR<U+017D>AVI")
# different representation of Unicode characters, e.g.:
# http://www.fileformat.info/info/unicode/char/0160/index.htm
# this will not work either:
guess_encoding(example)
iconv(example, from="ISO-8859-2", to="UTF-8")

# Things get even more complicated...
# One solution here:
# https://stackoverflow.com/questions/28248457/gsub-in-r-with-unicode-replacement-give-different-results-under-windows-compared
# we're basically going to convert to regular Unicode characters that
# R will be able to parse

trueunicode.hack <- function(string){
    m <- gregexpr("<U\\+[0-9A-F]{4}>", string)
    if(-1==m[[1]][1])
        return(string)

    codes <- unlist(regmatches(string, m))
    replacements <- codes
    N <- length(codes)
    for(i in 1:N){
        replacements[i] <- intToUtf8(strtoi(paste0("0x", substring(codes[i], 4, 7))))
    }

    # if the string doesn't start with a unicode, the copy its initial part
    # until first occurrence of unicode
    if(1!=m[[1]][1]){
        y <- substring(string, 1, m[[1]][1]-1)
        y <- paste0(y, replacements[1])
    }else{
        y <- replacements[1]
    }

    # if more than 1 unicodes in the string
    if(1<N){
        for(i in 2:N){
            s <- gsub("<U\\+[0-9A-F]{4}>", replacements[i], 
                      substring(string, m[[1]][i-1]+8, m[[1]][i]+7))
            Encoding(s) <- "UTF-8"
            y <- paste0(y, s)
        }
    }

    # get the trailing contents, if any
    if( nchar(string)>(m[[1]][N]+8) )
        y <- paste0( y, substring(string, m[[1]][N]+8, nchar(string)) )
    y
}

trueunicode.hack(example[1])
trueunicode.hack(example[2])
trueunicode.hack(example[3])

# and here's how we would convert back and forth...
# same text in Croatian
example <- "SAD DA POMOGNU RJEŠAVANJE POLITIČKE KRIZE"
Encoding(example) # UTF-8
# convert to ASCII and delete non-ASCII characters
iconv(example, "UTF-8", "ASCII", sub="")
# convert to latin1 and substitute to byte characters
(lat <- iconv(example, "UTF-8", "latin1", sub="byte"))

m <- gregexpr("<[0-9a-f]{2}>", lat)
codes <- regmatches(lat,m)
chars <- lapply(codes, function(x) {
    rawToChar(as.raw(strtoi(paste0("0x",substr(x,2,3)))), multiple=T)
})
regmatches(lat,m) <- chars
lat

```

And one final example...

```{r}
example <- "\U0001F602 \U0001F64C \U0001F602" # extended unicode character
message(example)
# you can search for the unicode representations of all these characters online
```

And now we can use this to search for these characters on Twitter!

```{r}
load("~/my_oauth")
library(tweetscores)
library(streamR)
```

```{r, eval=FALSE}
message("\U0001F926")
searchTweets(q="\U0001F926",
  filename="~/data/unicode-tweets.json",
  n=1000,
  oauth=my_oauth)
```

```{r}
tweets <- parseTweets("~/data/unicode-tweets.json")
message(sample(tweets$text, 5))
```

  
## Wordclouds with Japanese, Korean, and Chinese characters
  
```{r}
# reading into R
tweets <- streamR::parseTweets("~/data/japanese-tweets.json", simplify=TRUE)
library(quanteda)
tw <- corpus(tweets$text)
twdfm <- dfm_select(dfm(tw, remove_punct = TRUE, verbose=TRUE, remove_url=TRUE),
                    min_nchar=2)
topfeatures(twdfm, n=25)
```

What doesn't work:

```{r}
textplot_wordcloud(twdfm, rot.per=0, scale=c(3, .75), max.words=100)
```

But this should now work:

```{r}
pdf("wordcloud.pdf", family="Japan1")
textplot_wordcloud(twdfm, rot.per=0, scale=c(3, .75), max.words=100)
dev.off()
```

How to choose the family font? See `?postscriptFonts`.


