Twitter is one of the most popular sources of data in this age of Artificial Intelligence. Today, data is key to almost everything. To extract data from this amazing platform, Twitter provides APIs. We can use the API endpoints provided by Twitter, but, in this blog, we will use the Tweepy library.
You can do so much with Twitter API/Tweepy using Python that it is hard to cover all in one blog. So, I will divide it into two parts. In this blog, we will cover 5 topics related to searching the tweets. We will learn:
How to search tweets with Keywords
How to search tweets with specific user mentioned or of specific user
How to find tweets with specific hashtags
How To combine all three options
How to do pagination/How to fetch N number of tweets while tacking limitation of API
Before we start with the actual part of our blog, just confirm that you have installed Tweepy on your system. If not try the following command in your terminal.
pip install Tweepy
Twitter API has some restrictions, so in this blog, I will show you how to get recent Tweets that contain specific keywords. For example, we want to get tweets that contain either bitcoin or python. To get tweets with any or both of these keywords, we have to form a query. And the query would be bitcoin OR python. One thing to note here is that OR must be in a capital case.
To fetch the tweets we can use the following code:
from tweepy import OAuth1UserHandler, API
auth = OAuth1UserHandler(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
api = API(auth)
KEYWORDS = "bitcoin OR python"
tweets = api.search_tweets(KEYWORDS)
I have written another blog that will show you how to form the search queries and how you can get your Twitter API keys.
So, how does this code work? To fetch data using Twitter API we need to authenticate first. In version 1.1, we need to use OAuth for authentication. Whereas, in version 2, we can do most of the tasks with Bearer Token only.
After the authentication, we need to form a query. In our case, we want tweets that contain either bitcoin or python. Afterwards, we need to use search_tweets() method to pass our query. And that's it!
To extract full tweets, we need to pass one more argument, tweet_mode="extended" in our search_tweets() method. So, our new code to extract full tweets is
from tweepy import OAuth1UserHandler, API
auth = OAuth1UserHandler(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
api = API(auth)
KEYWORDS = "bitcoin OR python"
tweets = api.search_tweets(KEYWORDS, tweet_mode="extended")
We just need to add one more argument in our serach_tweet() method.
tweets = api.search_tweets(KEYWORDS, lang="en", tweet_mode="extended")
What if you want recent tweets or most popular tweets? Using Tweepy we can get either recent tweets, popular tweets or mixed version. By default, Tweepy returns mixed Tweets.
tweets = api.search_tweets(KEYWORDS, result_type="recent", tweet_mode="extended")
tweets = api.search_tweets(KEYWORDS, result_type="popular", tweet_mode="extended")
tweets = api.search_tweets(KEYWORDS, result_type="mixed", tweet_mode="extended")
Good news! You have to just pass another argument count in search_tweets() method.
tweets = api.search_tweets(KEYWORDS, count=100, tweet_mode="extended")
print("Total:", len(tweets))
But, there is a catch here.
You see, even if I asked for 100 Tweets sometimes I got 90, sometimes 89 or even 85. So, I am not certain how many you can get using the count argument. So, for now, I will suggest if you use the count argument use the number below 85.
Don't Worry! Tweepy has provided a way to fetch N number of Tweets but sometimes it depends on the quota and subscription of your Twitter API.
To get tweets based on hashtags is the same as the keywords. Tweepy has not provided any specific function to retrieve Tweets based on Hashtags. So, to get these Tweets we have to again form the query as we did in the search based on keywords.
Let's extract Tweets based on #javascript and #backend. Here, we want tweets with both hashtags. So, the query will be #javascript AND #backend.
from tweepy import OAuth1UserHandler, API
auth = OAuth1UserHandler(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
api = API(auth)
print("-" * 15)
QUERY = "#javascript AND #backend -filter:retweets"
tweets = api.search_tweets(QUERY, tweet_mode="extended", count=5, result_type="recent")
for i in tweets:
print(i.full_text)
print("-" * 15)
Note: I am Showing only small 3 Tweets here on purpose. Because others were too big and it will just create more confusion.
As you can see in the output, Every tweet has javascript and backend both hashtags in them. Here, I have used -filter:retweets in our query to exclude annoying retweets from our result.
So far, we saw how to search tweets based on keywords and hashtags. But, now we will see how to extract Tweets that have specific user mentions.
It is really very simple. We have formed a query mentioning our desired user. For example, we want to fetch tweets that mention Elon Musk. So, the query will become @elonmusk. Here, elonmusk is the username of Elon Musk.
from tweepy import OAuth1UserHandler, API
auth = OAuth1UserHandler(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
api = API(auth)
print("-" * 15)
QUERY = "@elonmusk -filter:retweets"
tweets = api.search_tweets(QUERY, tweet_mode="extended", result_type="recent")
for i in tweets:
print(i.full_text)
print("-" * 15)
Again, I have shown only a few tweets here.
So far, we saw the three most used queries to fetch tweets. Just to give a bit of test to form a query now let's combine all of these three methods in a single go. So, we will try to fetch tweets that have the tesla keyword, cryptocurrency hashtag and elonmusk mentioned.
The query we need to form will be tesla AND #cryptocurrency AND @elonmusk
from tweepy import OAuth1UserHandler, API
auth = OAuth1UserHandler(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
api = API(auth)
print("-" * 15)
QUERY = "tesla AND #cryptocurrency AND @elonmusk -filter:retweets"
tweets = api.search_tweets(QUERY, tweet_mode="extended", count=7, result_type="recent", lang='en')
for i in tweets:
print(i.full_text)
print("-" * 15)
Earlier, we saw, how we can pass the count argument to the search_tweets() method and get a specific number of tweets. But using this method we can get only 100 tweets as per the documentation. So, how do we fetch more tweets?
To retrieve n number of tweets, we can use the Cursor class provided by Tweepy. The cursor operates or works like pagination which is what we need to retrieve N number of tweets.
So, how do we use it? 🤔
To fetch 1000 tweets using the Cursor class, we need to pass mainly two arguments:
Method that we want to paginate
Query
Now specify the number of tweets we want, we can use the items() method of Cursor class.
from tweepy import OAuth1UserHandler, API, Cursor
auth = OAuth1UserHandler(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
api = API(auth, wait_on_rate_limit=True)
all_tweets = []
for tweet in Cursor(api.search_tweets, "#python", count=1000).items():
all_tweets.append(tweet.text)
print(len(all_tweets))
Here, if you have noticed, I have passed one more argument in our API class. Twitter API has a rate limit/quota. If we don't want to handle the error manually then we can pass wait_on_rate_limit=True to API() class.
Because of this, whenever we hit a rate limit, we will get see the following message on our console.
Rate limit reached. Sleeping for: 802
The 802 seconds is not the fixed duration. In general, we have to wait between 13 to 15 minutes.
Conclusion
You might have observed that the most difficult task to search and fetch tweets is to form a proper query. To solve that problem, I have written a whole blog post explaining Twitter Search API. Once you go through that blog you will be able to write your own search queries.
Let me know if you need any help or want to discuss something. Reach out to me on Twitter or LinkedIn. Make sure to share any thoughts, questions, or concerns. I would love to see them.
Till the next time 👋
