#### word2vec

Word embeddings offer a way to transform text into features. Instead of using vectors of word counts, words now are represented as positions on a latent multidimensional space. These positions are weights from an underlying deep learning models where the use of words are predicted based on the contiguous words. The idea is that words that have similar weights are likely to be used surrounded by the same words. 

`word2vec` is a method to compute word embeddings developed by Google. There are others (e.g. `Glove`, `BERT`, etc), but it is quite popular and we can use pre-trained models to speed up our analysis.

Let's see what we can do with it usign the `rword2vec` package in R. The examples here are based on the package materials, available [here](https://github.com/mukul13/rword2vec).

```{r}
#library(devtools)
#install_github("mukul13/rword2vec")
#install.packages("lsa")
library(rword2vec)
library(lsa)
```

This is how you would train the model. Note that this chunk of code will take a LONG time, so don't run it. There are different ways to train the model (see ?word2vec for details)

```{r, eval=FALSE}
model <- word2vec(
	train_file = "text8",
	output_file = "vec.bin",
	binary=1,
	num_threads=3,
	debug_mode=1)
```

To speed up the process, I'm providing a pre-trained model, available in the file `vec.bin`. We can now use it to run some analyses.

We'll start by computing the most similar words to a specific word, where _similar_ means how close they are on the latent multidimensional space (cosine similarity).

```{r}
distance(file_name = "../data/vec.bin",
		search_word = "princess",
		num = 10)
distance(file_name = "../data/vec.bin",
	search_word = "terrible",
	num = 10)
distance(file_name = "../data/vec.bin",
	search_word = "london",
	num = 10)
distance(file_name = "../data/vec.bin",
	search_word = "uk",
	num = 10)
distance(file_name = "../data/vec.bin",
	search_word = "philosophy",
	num = 10)
```

As an alternative way to think of embeddings, see this cool [online visualization](https://github.com/anvaka/word2vec-graph).

Where do these similarities come from? Let's extract the underlying word vectors.

```{r, eval=FALSE}
# Extracting word vectors
bin_to_txt("../data/vec.bin", "../data/vector.txt")
```

And now read them in R:
```{r}
library(readr)
data <- read_delim("../data/vector.txt", 
	skip=1, delim=" ",
	col_names=c("word", paste0("V", 1:100)))
data[1:10, 1:6]
```

That's the value of each word for each of the first five dimensions. We can plot some of these to understand better exactly what we're working with:

```{r}
plot_words <- function(words, data){
  # empty plot
  plot(0, 0, xlim=c(-2.5, 2.5), ylim=c(-2.5,2.5), type="n",
       xlab="First dimension", ylab="Second dimension")
  for (word in words){
    # extract first two dimensions
    vector <- as.numeric(data[data$word==word,2:3])
    # add to plot
    text(vector[1], vector[2], labels=word)
  }
}

plot_words(c("good", "better", "bad", "worse"), data)
plot_words(c("microsoft", "yahoo", "apple", "mango", "peach"), data)
plot_words(c("atheist", "agnostic", "catholic", "buddhist", "protestant", "christian"), data)
plot_words(c("government", "economics", "sociology", 
             "philosophy", "law", "engineering", "astrophysics",
             "biology", "physics", "chemistry"), data)

```

Once we have the vectors for each word, we can compute the similarity between a pair of words:

```{r}
similarity <- function(word1, word2){
	lsa::cosine(
		x=as.numeric(data[data$word==word1,2:101]),
		y=as.numeric(data[data$word==word2,2:101]))

}

similarity("australia", "england")
similarity("australia", "canada")
similarity("australia", "apple")
```

The final function provided by the package is `word_analogy`, which helps us find regularities in the word vector space:

```{r}
word_analogy(file_name = "../data/vec.bin",
	search_words = "king queen man",
	num = 1)

word_analogy(file_name = "../data/vec.bin",
	search_words = "paris france berlin",
	num = 1)

word_analogy(file_name = "../data/vec.bin",
	search_words = "man woman uncle",
	num = 2)

word_analogy(file_name = "../data/vec.bin",
	search_words = "building architect software",
	num = 1)

word_analogy(file_name = "../data/vec.bin",
	search_words = "man actor woman",
	num = 5)

word_analogy(file_name = "../data/vec.bin",
	search_words = "france paris uk",
	num = 1)

word_analogy(file_name = "../data/vec.bin",
	search_words = "up down inside",
	num = 2)

```

And we can see some examples of algorithmic bias (but really, bias in the training data):

```{r}
word_analogy(file_name = "../data/vec.bin",
	search_words = "man woman professor",
	num = 1)
word_analogy(file_name = "../data/vec.bin",
	search_words = "man doctor woman",
	num = 1)
```

