I saw in some question on Stack Exchange that the limitation can be a function of the number of requests per 15 minutes and depends also on the complexity of the algorithm, except that this is not a complex one.
So I use this code:
import tweepy
import sqlite3
import time
db = sqlite3.connect('data/MyDB.db')
# Get a cursor object
cursor = db.cursor()
cursor.execute('''CREATE TABLE IF NOT EXISTS MyTable(id INTEGER PRIMARY KEY, name TEXT, geo TEXT, image TEXT, source TEXT, timestamp TEXT, text TEXT, rt INTEGER)''')
db.commit()
consumer_key = ""
consumer_secret = ""
key = ""
secret = ""
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(key, secret)
api = tweepy.API(auth)
search = "#MyHashtag"
for tweet in tweepy.Cursor(api.search,
q=search,
include_entities=True).items():
while True:
try:
cursor.execute('''INSERT INTO MyTable(name, geo, image, source, timestamp, text, rt) VALUES(?,?,?,?,?,?,?)''',(tweet.user.screen_name, str(tweet.geo), tweet.user.profile_image_url, tweet.source, tweet.created_at, tweet.text, tweet.retweet_count))
except tweepy.TweepError:
time.sleep(60 * 15)
continue
break
db.commit()
db.close()
I always get the Twitter limitation error:
Traceback (most recent call last):
File "stream.py", line 25, in <module>
include_entities=True).items():
File "/usr/local/lib/python2.7/dist-packages/tweepy/cursor.py", line 153, in next
self.current_page = self.page_iterator.next()
File "/usr/local/lib/python2.7/dist-packages/tweepy/cursor.py", line 98, in next
data = self.method(max_id = max_id, *self.args, **self.kargs)
File "/usr/local/lib/python2.7/dist-packages/tweepy/binder.py", line 200, in _call
return method.execute()
File "/usr/local/lib/python2.7/dist-packages/tweepy/binder.py", line 176, in execute
raise TweepError(error_msg, resp)
tweepy.error.TweepError: [{'message': 'Rate limit exceeded', 'code': 88}]
For anyone who stumbles upon this on Google, tweepy 3.2+ has additional parameters for the tweepy.api class, in particular:
wait_on_rate_limit – Whether or not to automatically wait for rate limits to replenish
wait_on_rate_limit_notify – Whether or not to print a notification when Tweepy is waiting for rate limits to replenish
Setting these flags to True will delegate the waiting to the API instance, which is good enough for most simple use cases.
The problem is that your try: except: block is in the wrong place. Inserting data into the database will never raise a TweepError - it's iterating over Cursor.items() that will. I would suggest refactoring your code to call the next method of Cursor.items() in an infinite loop. That call should be placed in the try: except: block, as it can raise an error.
Here's (roughly) what the code should look like:
# above omitted for brevity
c = tweepy.Cursor(api.search,
q=search,
include_entities=True).items()
while True:
try:
tweet = c.next()
# Insert into db
except tweepy.TweepError:
time.sleep(60 * 15)
continue
except StopIteration:
break
This works because when Tweepy raises a TweepError, it hasn't updated any of the cursor data. The next time it makes the request, it will use the same parameters as the request which triggered the rate limit, effectively repeating it until it goes though.
Just replace
api = tweepy.API(auth)
with
api = tweepy.API(auth, wait_on_rate_limit=True)
If you want to avoid errors and respect the rate limit you can use the following function which takes your api object as an argument. It retrieves the number of remaining requests of the same type as the last request and waits until the rate limit has been reset if desired.
def test_rate_limit(api, wait=True, buffer=.1):
"""
Tests whether the rate limit of the last request has been reached.
:param api: The `tweepy` api instance.
:param wait: A flag indicating whether to wait for the rate limit reset
if the rate limit has been reached.
:param buffer: A buffer time in seconds that is added on to the waiting
time as an extra safety margin.
:return: True if it is ok to proceed with the next request. False otherwise.
"""
#Get the number of remaining requests
remaining = int(api.last_response.getheader('x-rate-limit-remaining'))
#Check if we have reached the limit
if remaining == 0:
limit = int(api.last_response.getheader('x-rate-limit-limit'))
reset = int(api.last_response.getheader('x-rate-limit-reset'))
#Parse the UTC time
reset = datetime.fromtimestamp(reset)
#Let the user know we have reached the rate limit
print "0 of {} requests remaining until {}.".format(limit, reset)
if wait:
#Determine the delay and sleep
delay = (reset - datetime.now()).total_seconds() + buffer
print "Sleeping for {}s...".format(delay)
sleep(delay)
#We have waited for the rate limit reset. OK to proceed.
return True
else:
#We have reached the rate limit. The user needs to handle the rate limit manually.
return False
#We have not reached the rate limit
return True
import tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
# will notify user on ratelimit and will wait by it self no need of sleep.
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
I suggest you to use the new api v2 and use the Client obj with the flag wait_on_rate_limit=True the v1 will be deprecated asap
client = tweepy.Client(consumer_key=auth.consumer_key, consumer_secret=auth.consumer_secret, access_token_secret=auth.access_token_secret, access_token=auth.access_token,
bearer_token=twitter_bearer_token, wait_on_rate_limit=True)
It will be all automatic
Related
My flask API looks like this:
#app.route('/getCuentaReferencia', methods=['GET'])
def getFollowers():
x = set()
try:
for user in tweepy.Cursor(api.get_followers, screen_name=request.args.get('cuentaReferencia')).items(int(request.args.get('cantidadCuenta'))):
x.add(user._json['screen_name'])
except Exception as e:
print(e)
pass
with open(request.args.get("cuentaReferencia") + ".json", "w") as f:
json.dump(list(x), f)
return jsonify({
"cuentaReferencia": request.args.get("cuentaReferencia") + request.args.get("region")+".json",
})
From frontend I send the screen name of the account and the quantity of followers that I'm looking for:
const response = await fetch(
`http://localhost:5000/getCuentaReferencia?cuentaReferencia=${cuentaReferencia}&cantidadCuenta=${
cantidadCuenta ?? ""
}®ion=${region}`
);
But when the Twitter API reach the limit, after waiting the cooldown it didn't continue looking for followers and only returns a few ones (if I need to get 1000, it returns me 300 because in that number of followers reaches the limit) I need to get all the count of followers that I send from frontend, how can i fix this up?
You can set the count argument of get_followers() to 200, its maximum (while default is only 20). That will allow you to get 3000 followers without reaching the rate limit.
You can read more about this argument in the Twitter documentation here.
If you want to get even more followers, you should probably use the Twitter API V2 instead.
I recently started to learn python, i had 0 knowledge on pyhton, and in the last few weeks i've been studying python and the twitter api.
I decided to work on a simple twitter bot, that automatically posts whatever people send on my direct messages, and i maneged to do so.
Here's my code:
import tweepy
import json
import sys
import re
import time
print("acessing api...")
consumer_key = 'consumer_key'
consumer_secret = 'consumer_secret'
key = 'key'
secret = 'key_secret'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(key, secret)
api = tweepy.API(auth, wait_on_rate_limit = True)
print('api accessed')
print('colecting dms')
direct_message = api.list_direct_messages()
text = direct_message[0].message_create["message_data"]["text"]
def send_tweet():
print('sending tweet...')
api.update_status('/bot: ' + text)
while True:
send_tweet()
time.sleep(30)
The code works, but not 100%, i'm able to extract the text from the Dm and use it on the api.update_status("/bot: " + text) but when it loops i get the following error:
tweepy.error.TweepError: [{'code': 187, 'message': 'Status is a duplicate.'}]
i've looked it up, and tried many things, for example:
while True:
try:
send_tweet()
time.sleep(30)
except tweepy.TweepError as error:
if error.api_code == 187:
print('duplicate message')
else:
print('waiting for new message')
time.sleep(30)
All i want is a way to keep the code going and ignore messages and tweets that were already sent, also wait for new dm's
Another thing that kept happening was that when i tried testing the extraction of the text, i kept getting the same text from the same dm, instead of getting a nother newer one.
Direct Messages stay in the Inbox even after your read or reply. It is best to delete a DM after you process it (in your case tweet it) so you don't need to worry about it next time you fetch the messages.
I'm trying to see several friendships between two twitter users, and I'm using show_friendship(twitter_id_1, twitter_id_2) method from tweepy library inside a loop.
I wonder how can I handle RateLimit or using a cursor (http://docs.tweepy.org/en/v3.5.0/cursor_tutorial.html) with that method because after several loops it raises a RateLimitError.
This is the RateLimit Handle of search friends (people I'm following) but I can't figure out how to do the same with show_friendship()
def limit_handled(self, cursor):
while True:
try:
yield cursor.next()
except tweepy.RateLimitError:
time.sleep(15 * 60)
for friend in limit_handled(tweepy.Cursor(api.friends).items()):
#stuff
When you are calling the API then you can specify the rate limit,
Eg:
api = tweepy.API(author, wait_on_rate_limit=True, wait_on_rate_limit_notify=True, retry_count=5, retry_delay=5)
then you can call any api for Twitter and it would accommodate rate limit.
So you can call show friendship normally as you would do:
api.show_friendship(source_id = friend, target_id = neighbor)
I am coding a program to get the followers from a given user, then use the follower's list to get their followers and so on. The code I have so far is as I show below:
import tweepy
import time
#insert your Twitter keys here
consumer_key ='key'
consumer_secret='secret'
access_token='accesstoken'
access_secret='accesssecret'
auth = tweepy.auth.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)
list= open('/home/acrocephalus/GitHub/TweetViz/list.txt','w')
if(api.verify_credentials):
print '-------------------------\n*** You are logged in ***\n-------------------------'
#Set starting Twitter handle
username = ['moixera']
user = tweepy.Cursor(api.followers, screen_name=username).items()
#Set the number of levels to follow
depth=3
#Start extracting each level followers
while depth != 0:
for handle in username:
print '\n\n Getting followers from: #' + handle+'\n\n'
user = tweepy.Cursor(api.followers, screen_name=handle).items()
while True:
try:
u = next(user)
list.write(u.screen_name +'\n')
print u.screen_name
except:
time.sleep(15*60)
print 'We have exceeded the rate limit. Sleeping for 15 minutes'
u = next(user)
list.write(u.screen_name +'\n')
print u.screen_name
username = list.read().splitlines()
print 'Moving to next username'
depth = depth-1
list.close()
The problem is that it starts with the first user, gets her followers but doesn't continue with her followers list. I think that the problem is in the while loop. When it finishes getting the followers it jumps to the except part. The desired behaviour would be that when it has finished retrieving followers it jumps to the beginning of the for loop. The program should jumpt to the except part of the while loop when it reaches the Twitter's API hit limit and thus times out for 15 minutes. Can anyone help?
Cheers!
Dani
Use a for loop instead of the while loop:
user_list = open('/home/acrocephalus/GitHub/TweetViz/list.txt','w')
for user in tweepy.Cursor(api.followers, screen_name=handle).items():
user_list.write(user.screen_name +'\n')
print user.screen_name
N.B. don't use list as a variable name because it hides the list builtin.
I think that the API has some support for rate limiting, although I don't see it detailed in the documentation. You can enable it when initialising with tweepy.API(), see wait_on_rate_limit and wait_on_rate_limit_notify:
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
A very quick glance at the source code suggests that the API will figure out an appropriate waiting period based on headers returned from Twitter, e.g. x-rate-limit-reset, but I have not used this API so I can't be sure whether it works.
There are other problems with your code, however, these go beyond your question.
Question
I was writing a Twitter bot with Python an Tweepy. I successfully got the bot to work last night, however, after following 60 people, the bot started throwing an error that says [{u'message': u'Rate Limit Exceeded', u'code': 88}]. I understand that I am only allowed to do a certain amount of calls to the Twitter API and I have found this link that shows how many calls I can make on each of these functions. After reviewing my code, I have found that the error is being thrown where I am saying for follower in tweepy.Cursor(api.followers, me).items():. On the page that I found that says how many requests I get, it says that I get 15 requests for every 15 minutes for getting my followers. I have waited overnight and I retried the code this morning, however, it was still throwing the same error. I don't understand why Tweepy is throwing a rate limit exceeded error whenever I should still have requests left.
Code
Here is my code that's throwing the error.
#!/usr/bin/python
import tweepy, time, pprint
CONSUMER_KEY = ''
CONSUMER_SECRET = ''
ACCESS_KEY = ''
ACCESS_SECRET = ''
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth, wait_on_rate_limit=True)
me = api.me()
pprint.pprint(api.rate_limit_status())
while True:
try:
for follower in tweepy.Cursor(api.followers, me).items():
api.create_friendship(id=follower.id)
for follower in tweepy.Cursor(api.friends, me).items():
for friend in tweepy.Cursor(api.friends, follower.id).items():
if friend.name != me.name:
api.create_friendship(id=friend.id)
except tweepy.TweepError, e:
print "TweepError raised, ignoring and continuing."
print e
continue
I have found the line that throws the error by typing in the interactive prompt
for follower in tweepy.Cursor(api.followers, me).items():
print follower
where it gives me the error
**Traceback (most recent call last):
File "<pyshell#31>", line 1, in <module>
for follower in api.followers(id=me.id):
File "C:\Users\Lane\AppData\Local\Enthought\Canopy\User\lib\site-packages\tweepy\binder.py", line 239, in _call
return method.execute()
File "C:\Users\Lane\AppData\Local\Enthought\Canopy\User\lib\site-packages\tweepy\binder.py", line 223, in execute
raise TweepError(error_msg, resp)
TweepError: [{u'message': u'Rate limit exceeded', u'code': 88}]**
API().followers method is actually GET followers/list, which is limited to 15 requests per window (before next epoch). Each call returns 20 users list by default, and if you are reaching limit exceed error, I'm sure authenticated user has more than 300 followers.
The solution is to increase the length of user's list received during each call, such that within 15 calls it can fetch all the users. Modify your code as following
for follower in tweepy.Cursor(api.followers, id = me.id, count = 50).items():
print follower
count denotes the number of users to be fetched per request.
Maximum value of count can be 200, it means within 15 minute, you can only fetch 200 x 15 = 3000 followers, which is enough in general scenarios.