We Keep Coding

Trouble understanding tweepy pagination

I want to use search terms in the Twitter API between 2 dates which may return 1000s of results. Is this even possible in the free version? I’m using tweepy and unsure if I should be using pages or not. When I set ‘count’ to any number I always get the same number of results back. I have set count=100 and got over 900 results in my latest test. How can I count the number of results returned? I have done lots of googling but can’t find the answer to this simple question. It’s basic stuff I know, but I find the Twitter documentation as clear as mud and the tweepy documentation not simple enough for a newbie. I want to store the results in an SQLite database for doing analysis. I’m happy with the database but don’t understand the pagination in tweepy. The documentation uses api.friends. I want to use api.search. I want to retrieve the maximum number of results possible.
Any help hugely appreciated!
Basically the bit I'm not understanding is how to return the maximum number of tweets using code like:
tweets = tw.Cursor(api.search,
q=search_words,
lang='en',
since=date_since,
count=200).items()
db.add_tweets(tweets)
This returns more than 200 tweets. Is it returning all the tweets or will I hit a 3200 limit? Is it saying there are 200 tweets per page? Some of the examples in the tweeps docs use pages(). I don't understand pagination! How do I find the number of tweets in each page if there's not 200? How do I know how many tweets have been returned from the query without doing count on the database? How would I iterate over every page if I use pages()? I apologise for my lack of understanding but the documentation is definitely not for newbies! Please can someone help? Thanks

How to Cross the 1000 Results Limit of youtube data API?

I need all video's ID between X to Y dates of specific subject. I used YouTube data API v3 and every thing was OK, until it returned empty item after 1000 result. I searched about this issue and figured out it's API limitation!!! Does anyone have any solution?
If it helps I code in python.I did test in php too and same result. It doesn't depend on languages.
thanks

Just split your date range into multiple requests, so each request < result limit.

Cant get more than 600 tweets using tweepy search api

noofitem = 1000
tweets = tweepy.Cursor(api.search,q=['#iphone11, -filter:retweets'],since='2019-11-14',lang='en',tweet_mode='extended',retweeted=False).items(noofitem)
i = [tweet.full_text for tweet in tweets] #Tweet text
I am trying to get about 1000 tweets using tweepy. But the max tweets I get are around 600. Changing the date does not work. Any modification or other workarounds will be helpful. Thanks.

Please note that Twitter’s search service and, by extension, the
Search API is not meant to be an exhaustive source of Tweets. Not all
Tweets will be indexed or made available via the search interface.
Please refer to this link for more information: http://docs.tweepy.org/en/latest/api.html#help-methods
Probably you will need to set up a Stream to get the amount of data you need.

Tweepy not finding results that should be there

I am writing a script in Python, that uses tweepy to search for tweets with a given keyword. Here is the snippet:
for tweet in tweepy.Cursor(api.search, q=keyword, lang="en").items(10):
print tweet.id
I have everything authenticated properly and the code works most of the time. However, when I try to search for some keywords (examples below) it doesn't return anything.
The keywords that cause trouble are "digitalkidz" (a tech conference) and "newtrendbg" (a Bulgarian company). If you do a quick search on Twitter for either of those you will see that there are results. However, tweepy doesn't find anything. Again, it does work for pretty much any other keyword I use.
Do you have any ideas what might be the problem and how to fix it?
Thank you

I believe you're forgetting an important aspect of the twitter api, it's not exhaustive.
Taken from the api docs
Please note that Twitter’s search service and, by extension, the Search API is not meant to be an exhaustive source of Tweets. Not all Tweets will be indexed or made available via the search interface.
Regardless of whether you're using the streaming or rest api, you're going to have issues with this if you're looking for specific tweets.
Rest API
When looking for historical tweets, you unfortunately won't be able to obtain anything that is older than a week using api.search(). This is also shown in the docs.
Keep in mind that the search index has a 7-day limit. In other words, no tweets will be found for a date older than one week.
There are other ways of getting older tweets, this post details those options.
Streaming API
While it doesn't sound like you're using twitter's streaming API, it should be noted that this only gives a small sample of twitter's current tweet traffic (~1-2%).
Hopefully this is helpful. Let me know if you have any questions.

How to Collect Tweets More Quickly Using Twitter API in Python?

For a research project, I am collecting tweets using Python-Twitter. However, when running our program nonstop on a single computer for a week we manage to collect about only 20 MB of data per week. I am only running this program on one machine so that we do not collect the same tweets twice.
Our program runs a loop that calls getPublicTimeline() every 60 seconds. I tried to improve this by calling getUserTimeline() on some of the users that appeared in the public timeline. However, this consistently got me banned from collecting tweets at all for about half an hour each time. Even without the ban, it seemed that there was very little speed-up by adding this code.
I know about Twitter's "whitelisting" that allows a user to submit more requests per hour. I applied for this about three weeks ago, and have not hear back since, so I am looking for alternatives that will allow our program to collect tweets more efficiently without going over the standard rate limit. Does anyone know of a faster way to collect public tweets from Twitter? We'd like to get about 100 MB per week.
Thanks.

How about using the streaming API? This is exactly the use-case it was created to address. With the streaming API you will not have any problems gathering megabytes of tweets. You still won't be able to access all tweets or even a statistically significant sample without being granted access by Twitter though.

I did a similar project analyzing data from tweets. If you're just going at this from a pure data collection/analysis angle, you can just scrape any of the better sites that collect these tweets for various reasons. Many sites allow you to search by hashtag, so throw in a popular enough hashtag and you've got thousands of results. I just scraped a few of these sites for popular hashtags, collected these into a large list, queried that list against the site, and scraped all of the usable information from the results. Some sites also allow you to export the data directly, making this task even easier. You'll get a lot of garbage results that you'll probably need to filter (spam, foreign language, etc), but this was the quickest way that worked for our project. Twitter will probably not grant you whitelisted status, so I definitely wouldn't count on that.

There is pretty good tutorial from ars technica on using streaming API n Python that might be helpful here.
Otherwise you could try doing it via cURL.
.