noofitem = 1000
tweets = tweepy.Cursor(api.search,q=['#iphone11, -filter:retweets'],since='2019-11-14',lang='en',tweet_mode='extended',retweeted=False).items(noofitem)
i = [tweet.full_text for tweet in tweets] #Tweet text
I am trying to get about 1000 tweets using tweepy. But the max tweets I get are around 600. Changing the date does not work. Any modification or other workarounds will be helpful. Thanks.
Please note that Twitter’s search service and, by extension, the
Search API is not meant to be an exhaustive source of Tweets. Not all
Tweets will be indexed or made available via the search interface.
Please refer to this link for more information: http://docs.tweepy.org/en/latest/api.html#help-methods
Probably you will need to set up a Stream to get the amount of data you need.
How do I scrape historical tweet from twitter with their location inclusive
I was able to get tweets with their location using the tweeter API, but it doesn't go beyond a certain period. How do I get historical tweets with their location inclusive using a hashtag.
I saw this code from github which I could get historical tweets but I've been unable to modify it to get the location of the tweets.Also how do I get the parameters passed into the xpath and css methods. How do I modify the code to get the location of my tweets historical.
the tweet crawler I'm using is from : https://github.com/jonbakerfish/TweetScraper/blob/master/TweetScraper/spiders/TweetCrawler.py
Advice on any other way I have use to get it will be apprecaited
Im trying to get all of a user's tweets for a 2 month time period. In search I see results but this code returns an empty array. Why?
results = api.GetSearch(raw_query="q=&from=yikyakapp&since=2014-09-24&until=2014-11-24")
print(results)
This is because Twitter search API has a limit of 7 days. Check the API documentation
The Twitter Search API searches against a sampling of recent Tweets published in the past 7 days.
There is a detailed explanation here https://dev.twitter.com/rest/reference/get/search/tweets
Keep in mind that the search index has a 7-day limit. In other words,
no tweets will be found for a date older than one week.
In summary, you can't use Twitter API to search for tweets beyond 7 days. Of course, in the website they can show you whatever they want. They hold all the data.
I have been trying to figure this out but this is a really frustrating. I'm trying to get tweets with a certain hashtag (a great amount of tweets) using Tweepy. But this doesn't go back more than one week. I need to go back at least two years for a period of a couple of months. Is this even possible, if so how?
Just for the check here is my code
import tweepy
import csv
consumer_key = '####'
consumer_secret = '####'
access_token = '####'
access_token_secret = '####'
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
# Open/Create a file to append data
csvFile = open('tweets.csv', 'a')
#Use csv Writer
csvWriter = csv.writer(csvFile)
for tweet in tweepy.Cursor(api.search,q="#ps4",count=100,\
lang="en",\
since_id=2014-06-12).items():
print tweet.created_at, tweet.text
csvWriter.writerow([tweet.created_at, tweet.text.encode('utf-8')])
As you have noticed Twitter API has some limitations, I have implemented a code that do this using the same strategy as Twitter running over a browser. Take a look, you can get the oldest tweets: https://github.com/Jefferson-Henrique/GetOldTweets-python
You cannot use the twitter search API to collect tweets from two years ago. Per the docs:
Also note that the search results at twitter.com may return historical results while the Search API usually only serves tweets from the past week. - Twitter documentation.
If you need a way to get old tweets, you can get them from individual users because collecting tweets from them is limited by number rather than time (so in many cases you can go back months or years). A third-party service that collects tweets like Topsy may be useful in your case as well (shut down as of July 2016, but other services exist).
Found one code that would help retrieve older tweets.
https://github.com/Jefferson-Henrique/GetOldTweets-python
To get old tweets, run the following command in the directory where the code repository got extracted.
python Exporter.py --querysearch 'keyword' --since 2016-01-10 --until 2016-01-15 --maxtweets 1000
And it returned a file 'output_got.csv' with 1000 tweets during the above days with your keyword
You need to install a module 'pyquery' for this to work
PS: You can modify 'Exporter.py' python code file to get more tweet attributes as per your requirement.
2018 update:
Twitter has Premium search APIs that can return results from the beginning of time (2006):
https://developer.twitter.com/en/docs/tweets/search/overview/premium#ProductPackages
Search Tweets: 30-day endpoint → provides Tweets from the previous 30
days.
Search Tweets: Full-archive endpoint → provides complete and instant
access to Tweets dating all the way back to the first Tweet in March
2006.
With an example Python client:
https://github.com/twitterdev/search-tweets-python
Knowing this is a very old question but still, some folks might be facing the same issue.
After some digging, I found out Tweepy's search only returns data for the past 7 days and that some times lead to buy third party service.
I utilised python library, GetOldTweets3 and it worked fine for me. The utility of this library is really easy. The only limitation of this library that we can't search for more than one hashtag in one execution but it works fine to search for multiple accounts at the same time.
use the args "since" and "until" to adjust your timeframe. You are presently using since_id which is meant to correspond to twitter id values (not dates):
for tweet in tweepy.Cursor(api.search,
q="test",
since="2014-01-01",
until="2014-02-01",
lang="en").items():
As others have noted, the Twitter API has the date limitation, but not the actual advanced search as implemented on twitter.com. So so the solution is to use Python's wrapper for Selenium or PhantomJS to iterate through the twitter.com endpoint. Here's an implementation using Selenium that someone has posted on Github: https://github.com/bpb27/twitter_scraping/
I can't believe nobody said this but this git repository completely solved my problem. I haven't been able to utilize other solutions such as GOT or Twitter API Premium.
Try this, definitely useful:
https://betterprogramming.pub/how-to-scrape-tweets-with-snscrape-90124ed006af
https://github.com/MartinBeckUT/TwitterScraper/tree/master/snscrape/cli-with-python
Is there any better way to get tweets from twitter then crawling twitter.com and mutating URLs?
If there is, how can I get latest tweets with current hash-tag?
Thank you!
Did you try Twitter REST API? Particulary, you can use search tweets endpoint. There are some limitations though, enforced by Twitter.
You can use one of many available python libraries.
For example, some sample code for tweepy can be found here.