## Wordscores

Let's check an example of wordscores. Here we have tweets from a random sample of 100 Members of the U.S. Congress, as well as their ideal points based on roll-call votes. Can we replicate the ideal points only using the text of their tweets?

```{r}
cong <- read.csv("../data/congress-tweets.csv", stringsAsFactors=F)
# creating the corpus and dfm objects
library(quanteda)
ccorpus <- corpus(cong$text)
docnames(ccorpus) <- cong$screen_name
cdfm <- dfm(ccorpus, remove_punct=TRUE, remove=c(stopwords("english"), "t.co", "https", "rt", "amp", "http", "t.c", "can"))
cdfm <- dfm_trim(cdfm, min_docfreq = 2)
# running wordscores
ws <- textmodel(cdfm, cong$idealPoint, model="wordscores", smooth=.5)
ws
# let's look at the most discriminant words
sw <- sort(ws@Sw)
head(sw, n=20)
tail(sw, n=20)
```

Now let's split the data into training and test set and see what we can learn...

```{r}
set.seed(123)
test <- sample(1:nrow(cong), floor(.20 * nrow(cong)))
# extracting ideal points and replacing them with missing values
refpoints <- cong$idealPoint
refpoints[test] <- NA
# running wordscores
ws <- textmodel(cdfm, refpoints, model="wordscores", smooth=.5)
# predicted values (this will take a while...)
preds <- predict(ws, rescaling="lbg")
scores <- preds@textscores
# and let's compare
plot(scores$textscore_lbg[test], cong$idealPoint[test])
cor(scores$textscore_lbg[test], cong$idealPoint[test])
```

## Wordfish

To explore an unsupervised approach to ideological scaling, let's continue with our previous example tweets by Members of Congress. Can we recover a latent ideological dimension based on the text of their tweets?

```{r}
# note heavy feature selection!
cdfm <- dfm_trim(cdfm, min_docfreq = 25)
# running wordfish
wf <- textmodel(cdfm, dir=c(10, 8), model="wordfish")
wf
# let's look at the most discriminant words (note same notation as in slides)
sw <- data.frame(beta=wf@beta, word=wf@features)
sw <- sw[order(sw$beta),]
head(sw, n=20)
tail(sw, n=20)
# and now we can compare the estimate positions with the ideal points...
plot(wf@theta, cong$idealPoint)
cor(wf@theta, cong$idealPoint)
cor(wf@theta[cong$party=="R"], cong$idealPoint[cong$party=="R"])
cor(wf@theta[cong$party=="D"], cong$idealPoint[cong$party=="D"])

```


