I am coding a Twitter bot which joins giveaways of users that I follow.
The problem is that when I use a for loop to iterate over a ItemIterator Cursor of 50 items it breaks before finishing. It usually does 20 or 39-40 iterations.
My main function is:
from funciones import *
from config import *
api = login(user)
i=0
while 1>i:
tweets = get_tweets(api, 50, True, None, None)
file = start_stats()
for tweet in tweets:
try:
i = i+1
tweet = is_RT(tweet)
show(tweet)
check(api,tweet,file)
print(f'{i}) 1.5 - 2m tweets cd')
sleep(random.randrange(40, 60,1))
except Exception as e:
print(str(e))
st.append(e)
print('15-20 min cooldown')
sleep(random.randrange(900, 1200,1))
So when the loop usually does 39 iterations, the code jumps into the 15 min. cooldown getting these of Tweets:
len(tweets.current_page) - 1
Out[251]: 19
tweets.page_index
Out[252]: 19
tweets.limit
Out[253]: 50
tweets.num_tweets
Out[254]: 20
I've seen this in the Tweepy cursor.py but I still don't know how to fix it.
def next(self):
if self.limit > 0:
if self.num_tweets == self.limit:
raise StopIteration
if self.current_page is None or self.page_index == len(self.current_page) - 1:
# Reached end of current page, get the next page...
self.current_page = self.page_iterator.next()
self.page_index = -1
self.page_index += 1
self.num_tweets += 1
return self.current_page[self.page_index]
The function I use in my main function to get the cursor is this:
def get_tweets(api,count=1,cursor = False, user = None, id = None):
if id is not None:
tweets = api.get_status(id=id, tweet_mode='extended')
return tweets
if cursor:
if user is not None:
if count>0:
tweets = tp.Cursor(api.user_timeline, screen_name=user, tweet_mode='extended').items(count)
else:
tweets = tp.Cursor(api.user_timeline, screen_name=user, tweet_mode='extended').items()
else:
if count>0:
tweets = tp.Cursor(api.home_timeline, tweet_mode='extended').items(count)
else:
tweets = tp.Cursor(api.home_timeline, tweet_mode='extended').items()
else:
if user is not None:
tweets = api.user_timeline(screen_name=user, count=count,tweet_mode='extended')
else:
tweets = api.home_timeline(count=count, tweet_mode='extended')
return tweets
When I've tried test codes like
j = 0
tweets = get_tweets(api,50,True)
for i in tweets:
j=j+1
print(j)
j and tweets.num_tweets are almost always 50, but I think when this is not 50 is because I don't wait between request, because I've reached j=300 with this, so maybe the problem is in the check function:
(It's a previous check function which also has the same problem, I've noticed it when I've started getting stats, the only difference is that I return values if the Tweets has been liked, rt, etc.)
def check(tweet):
if (bool(is_seen(tweet))
+ bool(age_check(tweet,3))
+ bool(ignore_check(tweet)) == 0):
rt_check(tweet)
like_check(tweet)
follow_check(tweet)
tag_n_drop_check(tweet)
quoted_check(tweet)
This is the first time I asked help so I don't know if I've posted all the info needed. This is driving me mad since last week and I don't know who to ask :(
Thanks in advance!
The IdIterator that Cursor returns when used with API.home_timeline stops when it receives a page with no results. This is most likely what's happening, since the default count for the endpoint is 20 and:
The value of count is best thought of as a limit to the number of tweets to return because suspended or deleted content is removed after the count has been applied.
https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-home_timeline
This is a limitation of this Twitter API endpoint, as there's not another good way to determine when to stop paginating.
However, you can pass a higher count (e.g. 100 if that works for you, up to 200) to the endpoint while using Cursor with it and you'll be less likely to receive a premature empty page.
Related
I continue to get the error, "'<=' not supported between instances of 'NoneType' and 'int'", but none of the fellow students are receiving this error. This code was provided to us, so I know I didn't do something wrong. My only guess is that it needs something different when it is run on a mac vs windows?
%%time
from pattern.web import Twitter, hashtags
engine = Twitter(language="en")
indexid = set()
tweets =[]
prev = None
for i in range(5):
print(i)
for tweet in engine.search("stock market bull or bear", start=prev, count= 20, cached=False):
print(f" hashtag = {hashtags(tweet.text)} text = {tweet.text}, author = {tweet.author}, date = {tweet.date} ")
if len(tweet.text) > 0 and tweet.id not in indexid:
tweets.append(tweet.text)
indexid.add(tweet.id)
prev = tweet.id
print(f"Found {len(tweets)} tweets!" )
print("")
The default to engine.search must be an integer or an id. The method is defined as follows:
def search(self, query, type=SEARCH, start=1, count=10, sort=RELEVANCY, size=None, cached=False, **kwargs):
""" Returns a list of results from Twitter for the given query.
- type : SEARCH,
- start: Result.id or int,
- count: maximum 100.
There is a limit of 150+ queries per 15 minutes.
"""
So in this case, all you need to do is initialize prev to start at 1, instead of None (which is what is causing your error).
prev = 1
for i in range(5):
...
Which resulted in:
Found 25 tweets!
Scenario:
I want to retrieve items from DynamoDB table which has 200k records, I am trying to get them in multiple requests
for first request I want 100 records only.
for second request I want next 100 records, here I don't want the records which are already in first request.
My implementation:
scan_kwargs=None
if scan_kwargs is None:
scan_kwargs = {}
complete = False
while not complete:
try:
response = table.scan(Limit=10000, **scan_kwargs,
FilterExpression=Key('timestamp').between(dateFrom, dateTo)
)
except botocore.exceptions.ClientError as error:
raise Exception('Error')
next_key = response.get('LastEvaluatedKey')
scan_kwargs['ExclusiveStartKey'] = next_key
complete = True if next_key is None else False
if response['Items']:
for record in response['Items']:
print(record)
totalRecords = totalRecords + 1
if totalRecords > 100:
break
if totalRecords > 100:
break
From the above code I am only able to get first 100 records for multiple requests.
But my requirement is to get from 101 to 200 records and ignore first 100 records
Can anyone help with working examples according to my requirement?
I am using the below code to fetch replies for a tweet with id 1504510003201265684 but I keep getting hit limit warning after a certain amount of replies has been fetched and also it seems some replies are ignored. How can I get around with that?
name = 'netflix'
tweet_id = '1504510003201265684'
count = 0
replies=[]
for tweet in tweepy.Cursor(api.search,q='to:'+name, result_type='mixed',tweet_mode='extended', timeout=999999).items():
count += 1
if hasattr(tweet, 'in_reply_to_status_id_str'):
if (tweet.in_reply_to_status_id_str==tweet_id):
replies.append(tweet)
Add a method that handles the tweepy.RateLimitError, in this case it sleeps for 15 minutes
def limit_handled(cursor):
while True:
try:
yield cursor.next()
except tweepy.RateLimitError:
time.sleep(15 * 60)
name = 'netflix'
tweet_id = '1504510003201265684'
count = 0
replies=[]
for tweet in limit_handled(tweepy.Cursor(api.search,q='to:'+name, result_type='mixed',tweet_mode='extended', timeout=999999).items()):
count += 1
if hasattr(tweet, 'in_reply_to_status_id_str'):
if (tweet.in_reply_to_status_id_str==tweet_id):
replies.append(tweet)
I've been using the example in this post
to create a system that searches and gets a large number of Tweets in a short time period. However, each time I switch to a new API key (make a new cursor) the search starts all over from the beginning and gets me repeated Tweets. How do I get each cursor to start where the other left off? What am I missing? Here's the code I am using:
currentAPI = 0
a = 0
currentCursor = tweepy.Cursor(apis[currentAPI].search, q = '%40deltaKshatriya')
c = currentCursor.items()
mentions = []
onlyMentions = []
while True:
try:
tweet = c.next()
if a > 100000:
break
else:
onlyMentions.append(tweet.text)
for t in tTweets:
if tweet.in_reply_to_status_id == t.id:
print str(a) + tweet.text
mentions.append(tweet.text)
a = a + 1
except tweepy.TweepError:
print "Rate limit hit"
if (currentAPI < 9):
print "Switching to next sat in constellation"
currentAPI = currentAPI + 1
#currentCursor = c.iterator.next_cursor
currentCursor = tweepy.Cursor(apis[currentAPI].search, q = '%40deltaKshatriya', cursor = currentCursor)
c = currentCursor.items()
else:
print "All sats maxed out, waiting and will try again"
currentAPI = 0
currentCursor = tweepy.Cursor(apis[currentAPI].search, q = '%40deltaKshatriya', cursor = currentCursor)
c = currentCursor.items()
time.sleep(60 * 15)
continue
except StopIteration:
break
I found a workaround that I think works, although I still encounter some issues. The idea is to add into
currentCursor = tweepy.Cursor(apis[currentAPI].search, q = '%40deltaKshatriya', cursor = currentCursor, max_id = max_id)
Where max_id is the id of the last tweet fetched before the rate limit was hit. The only issue I've encountered is with StopIteration being raised really early (before I get the full 100,000 Tweets) but that I think is a different SO question.
I know that Twitter search API has it's own limitations and returns much less search results rather than the actual results but I was searching through a popular hashtag and it only returns 60 result which is not acceptable at all!
here is my code in which I've used twython module.
results = {}
last_id = None
count = 0
while(len(results.keys()) != min_count):
if(last_id):
tmp_results = self.api.search(q="#mentionsomeoneimportantforyou", count=100, max_id=last_id)
else:
tmp_results = self.api.search(q=#mentionsomeoneimportantforyou, count=100)
count += len(tmp_results['statuses'])
print("new len: ", count)
last_id = get_max_id(tmp_results)
def get_max_id(results):
next_results_url_params = results['search_metadata']['next_results']
next_max_id = next_results_url_params.split('max_id=')[1].split('&')[0]
return next_max_id
Is there anything run with this code? It not, isn't 60 of many a joke?
The twython docs suggest not doing it that way, using the cursor approach instead:
twitter = Twython(APP_KEY, APP_SECRET,
OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
results = twitter.cursor(twitter.search, q='python')
count = 0
for result in results:
print result['id_str']
count += 1
print count
prints:
... many here ...
561918886380322816
561918859050229761
561919180480737282
561919151162130434
561919142450581504
561919113812246529
561919107134922753
561919103867559938
561919077481218049
561918994454556672
561918971755372546
561918962381127680
561918948288258048
561918911751655425
561918904126042112
561918886380322816
561918859050229761
645
I think I found the reason. According to this link Twitter doesn't return tweets older than a week through search api.