I have been trying to use either Tweepy or Twython with the Twitter API to search for a specific hashtag, extract usernames of users tweeting with the hashtage, and then see how many of those users follow one another. My ultimate goal is to then visualize the connections with NetworkX.
So far, I have been able to search for the hashtag and get a list of users tweeting with it. However, I can't figure out how to see who is following whom on that list. I finally got a friendship lookup to work, but then realized that that parameter only searches friends of the authenticated user (me).
Here is the latest version of the code:
from twython import Twython
import tweepy
# fill these in from Twitter API Dev
CONSUMER_KEY = ''
CONSUMER_SECRET = ''
ACCESS_KEY = ''
ACCESS_SECRET = ''
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth, wait_on_rate_limit=True)
twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)
# Search for hashtag, limit number of users
try:
search_results = twitter.search(q='energy', count=5)
except TwythonError as e:
print e
test5 = []
for tweet in search_results['statuses']:
if tweet['user']['screen_name'] not in test5:
test5.append((tweet['user']['screen_name']).encode('utf-8'))
print test5
# Lookup friendships
relationships = api.lookup_friendships(screen_names=test5[0:5])
for relationship in relationships:
if relationship.is_following:
print("User is following", relationship.screen_name)
Thanks!
With Tweepy, you can check if user_a follows user_b using the API.exists_friendship method. The code would look something like:
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth, wait_on_rate_limit=True)
is_following = api.exists_friendship(user_a, user_b)
You can specify users by id or screenname.
Alternatively, you can fetch the entire list of followers using the API.followers_ids method:
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth, wait_on_rate_limit=True)
user_b_followers = api.followers_ids(user_b)
is_following = user_a in user_b_followers
This approach would make more sense for large networks of users.
Keep in mind that, for either approach, you will only be able to see friendships that are visible to the authenticated user. This is a restriction put in place by Twitter for privacy reasons.
Related
I'm trying to retrive Tweets that particular accounts has posted. I do use
user_timeline parameter from the tweepy library, but it includes also replies from the concrete Twitter user. Does anyone has a clue how to omit them?
Code:
import tweepy
consumer_key = key
consumer_secret = key
access_key = key
access_secret = key
def get_tweets(username):
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
#set count to however many tweets you want; twitter only allows 200 at once
number_of_tweets = 20
#get tweets
tweets = api.user_timeline(screen_name = username,count = number_of_tweets)
#create array of tweet information: username, tweet id, date/time, text
tweets_for_csv = [[username,tweet.id_str, tweet.created_at, tweet.text.encode("utf-8")] for tweet in tweets]
print(str(tweets_for_csv))
Pass exclude_replies as a kwarg.
tweets = api.user_timeline(screen_name=username, count=number_of_tweets, exclude_replies=True)
See Twitters API documentation for a full list of kwargs you can pass.
I am trying to gather the tweets of a user navalny, from 01.11.2017 to 31.01.2018 using tweepy. I have ids of the first and last tweets that I need, so I tried the following code:
import tweepy
consumer_key = ''
consumer_secret = ''
access_token = ''
access_token_secret = ''
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
t = api.user_timeline(screen_name='navalny', since_id = 933000445307518976, max_id = 936533580481814529)
However, the returned value is an empty list.
What is the problem here?
Are there any restrictions on the history of tweets that I can get?
What are possible solutions?
Quick answer:
Using Tweepy you can only retrieve the last 3200 tweets from the Twitter REST API for a given user.
Unfortunately the tweets you are trying to access are older than this.
Detailed answer:
I did a check using the code below:
import tweepy
from tweepy import OAuthHandler
def tweet_check(user):
"""
Scrapes a users most recent tweets
"""
# API keys and initial configuration
consumer_key = ""
consumer_secret = ""
access_token = ""
access_secret = ""
# Configure authentication
authorisation = OAuthHandler(consumer_key, consumer_secret)
authorisation.set_access_token(access_token, access_secret)
api = tweepy.API(authorisation)
# Requests most recent tweets from a users timeline
tweets = api.user_timeline(screen_name=user, count=2,
max_id=936533580481814529)
for tweet in tweets:
tid = tweet.id
print(tid)
twitter_users = ["#navalny"]
for twitter_user in twitter_users:
tweet_check(twitter_user)
This test returns nothing before 936533580481814529
Using a seperate script I scraped all 3200 tweets, the max Twitter will let you scrape and the youngest tweet id I can find is 943856915536326662
Seems like you have run into Twitter's tweet scraping limit for user timelines here.
I'm currently using the GET statuses/user_timeline twitter API in python to retrieve tweets, It says in the docs This method can only return up to 3,200 of a user’s most recent Tweets however when i run it, it only returns 100-150.
How do i get it to return more tweets than it is already?
Thanks in advance
You'll have to write code to work with timelines. Twitter's Working with timelines documentation has a discussion of how and why Twitter timelines work the way they do.
Essentially, set your count to the maximum amount (200) and use since_id and max_id to manage reading each page. Alternatively, you can use an existing library to make the task much easier. Here's an example, using the tweepy library.
consumer_key = "<your consumer_key>"
consumer_secret = "<your consumer_secret>"
access_token = "<your access_token>"
access_token_secret = "<your access_token_secret>"
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
for status in tweepy.Cursor(api.user_timeline, "JoeMayo").items():
print('status_id: {}, text: {}'.format(status.id, status.text.encode('utf-8')))
I am trying to get the tweets from my saved searches using tweepy with python. I am not super great with Python, but the documentation seems pretty good, however, it is just returning empty lists. I know there are results for the searches because I can load them up in a browser. However, tweepy doesn't seem to see them.
import tweepy
CONSUMER_KEY = ''
CONSUMER_SECRET = ''
ACCESS_TOKEN = ''
ACCESS_TOKEN_SECRET = ''
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
api = tweepy.API(auth, wait_on_rate_limit=True)
saved_searches = api.saved_searches()
for search in saved_searches:
saved_search = api.get_saved_search(search.id)
results = api.search(q=str(saved_search.query))
for result in results:
print(result.text)
Can anyone help me figure out what I am missing here?
So I'm trying to get a timeline of a specific user. Here is the code:
import tweepy
consumer_key = 'numbers'
consumer_secret = 'numbers'
access_token = 'numbers'
access_token_secret = 'numbers'
user_list = [list of users]
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
for user in user_list:
for page in api.user_timeline(screen_name =user, count = 200):
print page
I've tried using the old documentation. When I run it I get the "Sorry, that page doesn't exist. Code 34"
I found the answer reading through the code documentation change screen_name to user_name
for user in users:
for page in api.user_timeline(user_id =user, count = 200):
print page