Scraping data from Twitter’s Streaming API
Pablo Barbera
June 24, 2019
Authenticating
Before we can start collecting Twitter data, we need to create an OAuth token that will allow us to authenticate our connection and access our personal data.
After the new API changes, getting a new token requires submitting an application for a developer account, which may take a few days. For teaching purposes only, I will temporarily share one of my tokens with each of you, so that we can use the API without having to do the authentication.
However, if in the future you want to get your own token, check the instructions at the end of this file.
library(ROAuth)
load("~/my_oauth")To check that it worked, try running the line below:
library(tweetscores)## Loading required package: R2WinBUGS## Loading required package: coda## Loading required package: boot## ##
## ## tweetscores: tools for the analysis of Twitter data## ## Pablo Barbera (LSE)## ## www.tweetscores.com
## ##getUsers(screen_names="LSEnews", oauth = my_oauth)[[1]]$screen_name## [1] "LSEnews"If this displays LSEnews then we’re good to go!
Some of the functions below will work with more than one token. If you want to save multiple tokens, see the instructions at the end of the file.
Collecting data from Twitter’s Streaming API
Collecting tweets filtering by keyword:
library(streamR)## Loading required package: RCurl## Loading required package: bitops## Loading required package: rjson## Loading required package: ndjsonfilterStream(file.name="~/data/trump-streaming-tweets.json", track="trump",
timeout=20, oauth=my_oauth)## Capturing tweets...## Connection to Twitter stream was closed after 20 seconds with up to 927 tweets downloaded.Note the options: - file.name indicates the file in your disk where the tweets will be downloaded
- track is the keyword(s) mentioned in the tweets we want to capture. - timeout is the number of seconds that the connection will remain open
- oauth is the OAuth token we are using
Once it has finished, we can open it in R as a data frame with the parseTweets function
tweets <- parseTweets("~/data/trump-streaming-tweets.json")## 913 tweets have been parsed.tweets[1,]## text
## 1 RT @bvkitsvage: la fille a gagné un procès contre un président wsh https://t.co/Q5fpJcuiNT
## retweet_count favorite_count favorited truncated id_str
## 1 20490 32920 FALSE FALSE 1062296260269559808
## in_reply_to_screen_name
## 1 <NA>
## source
## 1 <a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>
## retweeted created_at in_reply_to_status_id_str
## 1 NA Tue Nov 13 10:48:59 +0000 2018 <NA>
## in_reply_to_user_id_str lang listed_count verified
## 1 <NA> fr 0 FALSE
## location user_id_str
## 1 Clermont-Ferrand, France 985256540352458752
## description geo_enabled
## 1 Clf \U0001f3f4\n19yo\ndeaf\n•C17\U0001f339\n•G\U0001f4ab FALSE
## user_created_at statuses_count followers_count
## 1 Sat Apr 14 20:40:39 +0000 2018 3092 37
## favourites_count protected user_url name time_zone user_lang utc_offset
## 1 714 FALSE <NA> Memel NA fr NA
## friends_count screen_name country_code country place_type full_name
## 1 106 LinossierAmelie <NA> <NA> NA <NA>
## place_name place_id place_lat place_lon lat lon
## 1 <NA> <NA> NaN NaN NA NA
## expanded_url
## 1 https://twitter.com/navyofficiel/status/1061999888521744386
## url
## 1 https://t.co/Q5fpJcuiNTIf we want, we could also export it to a csv file to be opened later with Excel
write.csv(tweets, file="~/data/trump-streaming-tweets.csv", row.names=FALSE)And this is how we would capture tweets mentioning multiple keywords:
filterStream(file.name="~/data/politics-tweets.json",
track=c("graham", "sessions", "trump", "clinton"),
tweets=20, oauth=my_oauth)Note that here I choose a different option, tweets, which indicates how many tweets (approximately) the function should capture before we close the connection to the Twitter API.
We can also filter tweets in a specific language:
filterStream(file.name="~/data/spanish-tweets.json",
track="trump", language='es',
timeout=20, oauth=my_oauth)
tweets <- parseTweets("~/data/spanish-tweets.json")
sample(tweets$text, 10)And we can filter tweets by / retweeting / mentioning a specific user:
filterStream(file.name="~/data/trump-follow-tweets.json",
follow=25073877, timeout=10, oauth=my_oauth)
tweets <- parseTweets("~/data/trump-follow-tweets.json")
sample(tweets$text, 10)We now turn to tweets collect filtering by location instead. To be able to apply this type of filter, we need to set a geographical box and collect only the tweets that are coming from that area.
For example, imagine we want to collect tweets from the United States. The way to do it is to find two pairs of coordinates (longitude and latitude) that indicate the southwest corner AND the northeast corner. Note the reverse order: it’s not (lat, long), but (long, lat).
In the case of the US, it would be approx. (-125,25) and (-66,50). How to find these coordinates? You can use Google Maps, and right-click on the desired location. (Just note that long and lat are reversed here!)
filterStream(file.name="~/data/tweets_geo.json", locations=c(-125, 25, -66, 50),
timeout=30, oauth=my_oauth)## Capturing tweets...## Connection to Twitter stream was closed after 30 seconds with up to 485 tweets downloaded.We can do as before and open the tweets in R
tweets <- parseTweets("~/data/tweets_geo.json")## 710 tweets have been parsed.And use the maps library to see where most tweets are coming from. Note that there are two types of geographic information on tweets: lat/lon (from geolocated tweets) and place_lat and place_lon (from tweets with place information). We will work with whatever is available.
library(maps)
tweets$lat <- ifelse(is.na(tweets$lat), tweets$place_lat, tweets$lat)
tweets$lon <- ifelse(is.na(tweets$lon), tweets$place_lon, tweets$lon)
tweets <- tweets[!is.na(tweets$lat),]
states <- map.where("state", tweets$lon, tweets$lat)
head(sort(table(states), decreasing=TRUE))## states
## california texas florida
## 95 74 43
## ohio north carolina:main pennsylvania
## 29 28 28We can also prepare a map of the exact locations of the tweets.
library(ggplot2)
## First create a data frame with the map data
map.data <- map_data("state")
# And we use ggplot2 to draw the map:
# 1) map base
ggplot(map.data) + geom_map(aes(map_id = region), map = map.data, fill = "grey90",
color = "grey50", size = 0.25) + expand_limits(x = map.data$long, y = map.data$lat) +
# 2) limits for x and y axis
scale_x_continuous(limits=c(-125,-66)) + scale_y_continuous(limits=c(25,50)) +
# 3) adding the dot for each tweet
geom_point(data = tweets,
aes(x = lon, y = lat), size = 1, alpha = 1/5, color = "darkblue") +
# 4) removing unnecessary graph elements
theme(axis.line = element_blank(),
axis.text = element_blank(),
axis.ticks = element_blank(),
axis.title = element_blank(),
panel.background = element_blank(),
panel.border = element_blank(),
panel.grid.major = element_blank(),
panel.grid.minor = element_blank(),
plot.background = element_blank()) ## Warning: Removed 5 rows containing missing values (geom_point).
And here’s how to extract the edges of a network of retweets (at least one possible way of doing it):
tweets <- parseTweets("~/data/trump-streaming-tweets.json")## 913 tweets have been parsed.# subset only RTs
rts <- tweets[grep("RT @", tweets$text),]
library(stringr)
edges <- data.frame(
node1 = rts$screen_name,
node2 = str_extract(rts$text, 'RT @[a-zA-Z0-9_]+'),
text = rts$text,
stringsAsFactors=F
)
edges$node2 <- str_replace(edges$node2, 'RT @', '')
# plotting largest connected component
library(igraph)##
## Attaching package: 'igraph'## The following objects are masked from 'package:stats':
##
## decompose, spectrum## The following object is masked from 'package:base':
##
## uniong <- graph_from_data_frame(d=edges, directed=TRUE)
comp <- decompose(g, min.vertices=20)
plot(comp[[1]])
Finally, it’s also possible to collect a random sample of tweets. That’s what the “sampleStream” function does:
sampleStream(file.name="~/data/tweets_random.json", timeout=30, oauth=my_oauth)## Capturing tweets...## Connection to Twitter stream was closed after 30 seconds with up to 1634 tweets downloaded.Here I’m collecting 30 seconds of tweets. And once again, to open the tweets in R…
tweets <- parseTweets("~/data/tweets_random.json")## 2284 tweets have been parsed.What is the most retweeted tweet?
tweets[which.max(tweets$retweet_count),]## text
## 2532 RT @BTS_twt: 여행\U0001f97a\U0001f49c\U0001f495 \n#방탄6주년보라해 #HBD_BTS https://t.co/0SUszcLGRM
## retweet_count favorite_count favorited truncated id_str
## 2532 568000 1644160 FALSE FALSE 1142921315231260672
## in_reply_to_screen_name
## 2532 <NA>
## source
## 2532 <a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>
## retweeted created_at in_reply_to_status_id_str
## 2532 FALSE Sun Jun 23 22:24:10 +0000 2019 <NA>
## in_reply_to_user_id_str lang listed_count verified location
## 2532 <NA> ko 0 FALSE <NA>
## user_id_str
## 2532 1040813097546539009
## description
## 2532 \U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c\U0001f49c
## geo_enabled user_created_at statuses_count
## 2532 FALSE Sat Sep 15 04:02:34 +0000 2018 995
## followers_count favourites_count protected user_url
## 2532 7 1018 FALSE <NA>
## name time_zone user_lang utc_offset
## 2532 방탄에게 반\U0001f49c한 우영 NA <NA> NA
## friends_count screen_name country_code country place_type
## 2532 5 uNYYuEQubgNUGfk <NA> <NA> NA
## full_name place_name place_id place_lat place_lon lat lon
## 2532 <NA> <NA> <NA> NaN NaN NA NA
## expanded_url url
## 2532 <NA> <NA>What are the most popular hashtags at the moment? We’ll use regular expressions to extract hashtags.
library(stringr)
ht <- str_extract_all(tweets$text, '#[A-Za-z0-9_]+')
ht <- unlist(ht)
head(sort(table(ht), decreasing = TRUE))## ht
## #TeenChoice #BTS #1 #BTSinTokyoDome
## 12 8 6 5
## #EstrenoECNSE #EXO
## 5 5And who are the most frequently mentioned users?
handles <- str_extract_all(tweets$text, '@[0-9_A-Za-z]+')
handles_vector <- unlist(handles)
head(sort(table(handles_vector), decreasing = TRUE), n=10)## handles_vector
## @BTS_twt @weareoneEXO @SethFedeIin @YouTube
## 32 8 6 6
## @Camila_Cabello @charliekirk11 @cosmo_ph @ekrem_imamoglu
## 3 3 3 3
## @Jack_Septic_Eye @lofagmon
## 3 3How many tweets mention Justin Bieber?
length(grep("bieber", tweets$text, ignore.case=TRUE))## [1] 1These are toy examples, but for large files with tweets in JSON format, there might be faster ways to parse the data. For example, the ndjson package offers a robust and fast way to parse JSON data:
library(ndjson)
json <- stream_in("~/data/tweets_geo.json")
json## Source: local data table [710 x 1,417]
##
## # A tibble: 710 x 1,417
## contributors coordinates created_at place.url place.place_type
## <int> <int> <chr> <chr> <chr>
## 1 NA NA Thu Nov 1… https://… admin
## 2 NA NA Thu Nov 1… https://… admin
## 3 NA NA Thu Nov 1… https://… city
## 4 NA NA Thu Nov 1… https://… city
## 5 NA NA Thu Nov 1… https://… city
## 6 NA NA Thu Nov 1… https://… city
## 7 NA NA Thu Nov 1… https://… city
## 8 NA NA Thu Nov 1… https://… city
## 9 NA NA Thu Nov 1… https://… admin
## 10 NA NA Thu Nov 1… https://… city
## # … with 700 more rows, and 1,412 more variables: quote_count <dbl>,
## # reply_count <dbl>, user.verified <lgl>, entities.hashtags <int>,
## # entities.symbols <int>, retweet_count <dbl>, retweeted <lgl>,
## # entities.urls <int>, entities.user_mentions <int>,
## # favorite_count <dbl>, favorited <lgl>, source <chr>, text <chr>,
## # filter_level <chr>, geo <int>, timestamp_ms <chr>, truncated <lgl>,
## # user.contributors_enabled <lgl>, user.created_at <chr>, id <dbl>,
## # id_str <chr>, user.default_profile <lgl>,
## # user.default_profile_image <lgl>, in_reply_to_screen_name <chr>,
## # in_reply_to_status_id <dbl>, in_reply_to_status_id_str <chr>,
## # in_reply_to_user_id <dbl>, user.description <chr>,
## # user.favourites_count <dbl>, in_reply_to_user_id_str <chr>,
## # is_quote_status <lgl>, user.follow_request_sent <int>,
## # user.followers_count <dbl>, user.following <int>,
## # user.friends_count <dbl>, lang <chr>, place.attributes <int>,
## # user.geo_enabled <lgl>, user.id <dbl>,
## # place.bounding_box.coordinates.0.0.0 <dbl>,
## # place.bounding_box.coordinates.0.0.1 <dbl>,
## # place.bounding_box.coordinates.0.1.0 <dbl>,
## # place.bounding_box.coordinates.0.1.1 <dbl>, user.id_str <chr>,
## # user.is_translator <lgl>, place.bounding_box.coordinates.0.2.0 <dbl>,
## # place.bounding_box.coordinates.0.2.1 <dbl>, user.lang <chr>,
## # user.listed_count <dbl>, user.location <chr>, user.name <chr>,
## # place.bounding_box.coordinates.0.3.0 <dbl>,
## # place.bounding_box.coordinates.0.3.1 <dbl>, user.notifications <int>,
## # user.profile_background_color <chr>, place.bounding_box.type <chr>,
## # place.country <chr>, place.country_code <chr>, place.full_name <chr>,
## # user.profile_background_image_url <chr>,
## # user.profile_background_image_url_https <chr>, place.id <chr>,
## # place.name <chr>, user.profile_background_tile <lgl>,
## # user.profile_banner_url <chr>, user.profile_image_url <chr>,
## # user.profile_image_url_https <chr>, user.profile_link_color <chr>,
## # user.profile_sidebar_border_color <chr>,
## # user.profile_sidebar_fill_color <chr>, user.profile_text_color <chr>,
## # user.profile_use_background_image <lgl>, user.protected <lgl>,
## # user.screen_name <chr>, user.statuses_count <dbl>,
## # user.time_zone <int>, user.translator_type <chr>, user.url <chr>,
## # user.utc_offset <int>, coordinates.coordinates.0 <dbl>,
## # coordinates.coordinates.1 <dbl>, display_text_range.0 <dbl>,
## # display_text_range.1 <dbl>, entities.user_mentions.0.id <dbl>,
## # coordinates.type <chr>, entities.user_mentions.0.id_str <chr>,
## # entities.user_mentions.0.indices.0 <dbl>,
## # entities.user_mentions.0.indices.1 <dbl>,
## # entities.user_mentions.0.name <chr>,
## # entities.urls.0.display_url <chr>, entities.urls.0.expanded_url <chr>,
## # entities.urls.0.indices.0 <dbl>, entities.urls.0.indices.1 <dbl>,
## # entities.user_mentions.0.screen_name <chr>, entities.urls.0.url <chr>,
## # extended_tweet.display_text_range.0 <dbl>,
## # extended_tweet.display_text_range.1 <dbl>,
## # entities.user_mentions.1.id <dbl>,
## # extended_tweet.entities.hashtags <int>,
## # extended_tweet.entities.symbols <int>, …Creating your own token
Follow these steps to create your own token after your application has been approved:
- Go to https://developer.twitter.com/en/apps and sign in.
- If you don’t have a developer account, you will need to apply for one first. Fill in the application form and wait for a response.
- Once it’s approved, click on “Create New App”. You will need to have a phone number associated with your account in order to be able to create a token.
- Fill name, description, and website (it can be anything, even http://www.google.com). Make sure you leave ‘Callback URL’ empty.
- Agree to user conditions.
- From the “Keys and Access Tokens” tab, copy consumer key and consumer secret and paste below
- Click on “Create my access token”, then copy and paste your access token and access token secret below
library(ROAuth)
my_oauth <- list(consumer_key = "CONSUMER_KEY",
consumer_secret = "CONSUMER_SECRET",
access_token="ACCESS_TOKEN",
access_token_secret = "ACCESS_TOKEN_SECRET")
save(my_oauth, file="~/my_oauth")load("~/my_oauth")What can go wrong here? Make sure all the consumer and token keys are pasted here as is, without any additional space character. If you don’t see any output in the console after running the code above, that’s a good sign.
Note that I saved the list as a file in my hard drive. That will save us some time later on, but you could also just re-run the code in lines 22 to 27 before conecting to the API in the future.
To check that it worked, try running the line below:
library(tweetscores)
getUsers(screen_names="LSEnews", oauth = my_oauth)[[1]]$screen_name## [1] "LSEnews"If this displays LSEnews then we’re good to go!