## Topic Modeling: LDA

While supervised learning is used when we know the categories we want to produce, unsupervised learning (including topic modeling) is used when we do not know the categories. In topic modeling, documents are not assumed to belong to one topic or category, but simultaneously belong to several topics. The topic distributions also vary over documents. 

The workhorse function for the topic model is `LDA`, which stands for Latent Dirichlet Allocation, the technical name for this particular kind of model. 

We will now use a dataset that contains the lead paragraph of around 5,000 articles about the economy published in the New York Times between 1980 and 2014. As before, we will preprocess the text using the standard set of techniques.

The number of topics in a topic model is somewhat arbitrary, so you need to play with the number of topics to see if you get anything more meaningful. We start here with 30 topics.

```{r message = FALSE}
library(topicmodels)
library(quanteda)

# reading data and converting to DFM
nyt <- read.csv("../data/nytimes.csv", stringsAsFactors = FALSE)
nyt_corpus <- corpus(nyt, text_field = "lead_paragraph")
toks <- tokens(nyt_corpus, 
               remove_punct = TRUE, remove_numbers = TRUE)
nyt_dfm <- dfm(toks)
nyt_dfm <- dfm_remove(nyt_dfm, stopwords("english"))
cdfm <- dfm_trim(nyt_dfm, min_docfreq = 2)

# estimate LDA with K topics
K <- 30
lda <- LDA(cdfm, k = K, method = "Gibbs", 
                control = list(verbose=25L, seed = 123,
                               burnin = 100, iter = 500))
```

We can use `get_terms` to the top `n` terms from the topic model, and `get_topics` to predict the top `k` topic for each document. This will help us interpret the results of the model.

```{r}
terms <- get_terms(lda, 15)
terms[,1]
topics <- get_topics(lda, 1)
head(topics)
```

Let's take a closer look at some of these topics. To help us interpret the output, we can look at the words associated with each topic and take a random sample of documents highly associated with each topic.

```{r}
# Topic 2
paste(terms[,2], collapse=", ")
sample(nyt$lead_paragraph[topics==2], 1)
# Topic 3
paste(terms[,3], collapse=", ")
sample(nyt$lead_paragraph[topics==3], 1)
# Topic 4
paste(terms[,4], collapse=", ")
sample(nyt$lead_paragraph[topics==4], 1)
# Topic 14
paste(terms[,14], collapse=", ")
sample(nyt$lead_paragraph[topics==14], 1)
```

You will that often some topics do not make much sense. They just capture the remaining cluster of words, and often correspond to stopwords. For example:

```{r}
# Topic 12
paste(terms[,12], collapse=", ")
sample(nyt$lead_paragraph[topics==12], 1)
# Topic 19
paste(terms[,19], collapse=", ")
sample(nyt$lead_paragraph[topics==19], 1)
```

In the case of date with timestamps, looking at the evolution of certain topics over time can also help interpret their meaning. Let's look for example at Topic 25, which appears to be related to the stock market.

```{r}
# Topic 25
paste(terms[,25], collapse=", ")
sample(nyt$lead_paragraph[topics==25], 1)
# add predicted topic to dataset
nyt$pred_topic <- topics
nyt$year <- substr(nyt$datetime, 1, 4) # extract year
 # frequency table with articles about stock market, per year
tab <- table(nyt$year[nyt$pred_topic==2])
plot(tab)
```

But we can actually do better than this. LDA is a probabilistic model, which means that for each document, it actually computes a distribution over topics. In other words, each document is considered to be __about a mixture of topics__. 

This information is included in the matrix `gamma` in the LDA object (`theta` in the notation we used for the slides). For example, article 1 is 21% about topic 27, 7% about topic 29, 4% about topic 2, and then less than 4% for each of the rest.

```{r}
round(lda@gamma[1,], 2)
```

So we can actually take the information in the matrix and aggregate it to compute the average probability that an article each year is about a particular topic. Let's now choose Topic 23, which appears to be related to the financial crisis.

```{r}
# Topic 23
paste(terms[,23], collapse=", ")
# add probability to df
nyt$prob_topic <- lda@gamma[,23]
# now aggregate at the year level
agg <- aggregate(nyt$prob_topic, by=list(year=nyt$year), FUN=mean)
# and plot it
plot(agg$year, agg$x, type="l", xlab="Year", ylab="Avg. prob. of article about topic 23",
     main="Estimated proportion of articles about the financial crisis")

```
