We'll now turn to a different type of Twitter data -- static data, either recent tweets or user-level information. This type of data can be retrieved with Twitter's REST API. We will use the `tweetscores` package here -- this is a package that I created to facilitate the collection and analysis of Twitter data.

### Searching recent tweets

It is possible to download recent tweets, but only up those less than 7 days old, and in some cases not all of them.

```{r}
load("~/my_oauth")
library(tweetscores)
library(streamR)

searchTweets(q="halloween", 
  filename="../data/halloween-tweets.json",
  n=1000, until="2022-10-31", 
  oauth=my_oauth)

tweets <- parseTweets("../data/halloween-tweets.json")
```

What are the most popular hashtags?
```{r}
library(stringr)
ht <- str_extract_all(tweets$text, '#[A-Za-z0-9_]+')
ht <- unlist(ht)
head(sort(table(ht), decreasing = TRUE))
```

You can check the documentation about the options for string search [here](https://dev.twitter.com/rest/public/search).

### Extracting users' profile information

This is how you would extract information from user profiles:

```{r}
wh <- c("JoeBiden", "POTUS", "VP", "FLOTUS")
users <- getUsersBatch(screen_names=wh,
                       oauth=my_oauth)
str(users)
```

Which of these has the most followers?
```{r}
users[which.max(users$followers_count),]
users$screen_name[which.max(users$followers_count)]
```

Download up to 3,200 recent tweets from a Twitter account:
```{r}
getTimeline(filename="../data/uscpoir.json",
            screen_name="uscpoir", n=200, oauth=my_oauth)
```

What are the most common hashtags?
```{r}
tweets <- parseTweets("../data/uscpoir.json")
ht <- str_extract_all(tweets$text, '#[A-Za-z0-9_]+')
ht <- unlist(ht)
head(sort(table(ht), decreasing = TRUE))
```

### Building friend and follower networks

Download friends and followers:
```{r}
followers <- getFollowers("uscpoir", 
    oauth=my_oauth)
friends <- getFriends("uscpoir", 
    oauth=my_oauth)
```

What are the most common words that friends of the uscpoir account use to describe themselves on Twitter?

```{r, fig.height=6, fig.width=6}
# extract profile descriptions
users <- getUsersBatch(ids=friends, oauth=my_oauth)
# create table with frequency of word use
library(quanteda)
tw <- tokens(corpus(users$description[users$description!=""]),remove_punct=TRUE)
dfm <- dfm_remove(dfm(tw), c(stopwords("english"), stopwords("spanish"),
                                 "t.co", "https", "rt", "rts", "http"))
topfeatures(dfm, n = 20)
```

### Estimating ideology based on Twitter networks

The `tweetscores` package also includes functions to replicate the method developed in the Political Analysis paper [__Birds of a Feather Tweet Together. Bayesian Ideal Point Estimation Using Twitter Data__](https://doi.org/10.1093/pan/mpu011). For an application of this method, see also [this Monkey Cage blog post](http://www.washingtonpost.com/blogs/monkey-cage/wp/2015/06/16/who-is-the-most-conservative-republican-candidate-for-president/).

```{r}
# download list of friends for an account
user <- "DonaldJTrumpJr"
friends <- getFriends(user, oauth=my_oauth)
# estimating ideology with correspondence analysis method
(theta <- estimateIdeology2(user, friends, verbose=FALSE))
```

### Other types of data

The REST API offers also a long list of other endpoints that could be of use at some point, depending on your research interests.

1) You can search users related to specific keywords:

```{r}
users <- searchUsers(q="uscpoir", count=100, oauth=my_oauth)
users$screen_name[1:10]
```

2) If you know the ID of the tweets, you can download it directly from the API. This is useful because tweets cannot be redistributed as part of the replication materials of a published paper, but the list of tweet IDs can be shared:

```{r}
# Downloading tweets when you know the ID
getStatuses(ids=c('1454115859534950406', '1452687910055002115',
                  '1451896893743767555'),
            filename="../data/old-tweets.json",
            oauth=my_oauth)
parseTweets("../data/old-tweets.json")
```

3) Lists of Twitter users, compiled by other users, are also accessible through the API.

```{r}
# download user information from a list
govs <- getList(list_id="7560205", oauth=my_oauth)
head(govs)
```

This is also useful if e.g. you're interested in compiling lists of journalists, because media outlets offer these lists in their profiles.

4) List of users who retweeted a particular tweet -- unfortunately, it's limited to only 100 most recent retweets.

```{r}
# Download list of users who retweeted a tweet (unfortunately, only up to 100)
rts <- getRetweets(id='653733796408377344', oauth=my_oauth)
# https://twitter.com/HillaryClinton/status/653733796408377344
```

5) And one final function to convert dates in their internal Twitter format to another format we could work with in R:

```{r}
# format Twitter dates to facilitate analysis
tweets <- parseTweets("../data/uscpoir.json")
tweets$date <- formatTwDate(tweets$created_at, format="date")
hist(tweets$date, breaks="month")
```
