I have the following questions about tweepy python module
1.I am trying to retrieve all tweets for a specific location. I am able to do this by using tweepy python module (streaming API), but I get only those tweets whose geo locations are enabled, which means I would loose rest of the tweeter’s tweet who have not enabled their geo location. Is there a better way to retrieve all the tweets, given a location?
2.I use Stream.Sample method to retrieve all the tweets, Can someone tell me about the parameters used in sample method? I see count, and async as parameters. Now what should we specify here?
3.What does firehose method in tweepy.Stream do?
Any help is much appreciated
If tweepy doesn't have a feature you need, you can always access Twitter directly with an HTTP request. The full Twitter REST API is described here: https://dev.twitter.com/docs/api
The ones that seem relevant to your interest are:
GET trends/:woeid which looks up tweets by woeid, a Yahoo Identifier for collecting information about a given place/landmark/etc.
GET geo/id/:place_id which only mines geotagged tweets.
There is documentation of all the information available for a GET request but the IP address is not among the available fields: https://dev.twitter.com/docs/api/1/get/search .
Lastly, Twitter has a location search FAQ that may be of interest.
Related
For a project I've been using an API to get information from Instagram. However, I would like to get info from posts using keywords (words included inside the post description). This is a feature available in the app
see here, however I have been only able to make search by hashtag, which is not what I want.
I would like to know if any of you know an API/tool able to accomplish this task.
Here is a way to search instagram posts by words or phrases (not hashtags):
First, you would need to use tools for web scrapping Google search results. Check out this answers for guidance. The url that you will make the request to would be something like:
https://www.google.com/search?q=site%3Ahttps%3A%2%2Fwww.instagram.com%2Fp%2F+**put+the+phrase+here**
Once you get the urls from the posts that contain those words, you may want to use an API (e.g. from Rapidapi), build your own code for web scrapping, or use Python packages such as instagramy to get metadata from the instagram posts that you've got.
Usually, the information comes in JSON when using an API, so it is not very difficult to extract the data and put it in as a pandas dataframe if you want to.
I am writing a script in Python, that uses tweepy to search for tweets with a given keyword. Here is the snippet:
for tweet in tweepy.Cursor(api.search, q=keyword, lang="en").items(10):
print tweet.id
I have everything authenticated properly and the code works most of the time. However, when I try to search for some keywords (examples below) it doesn't return anything.
The keywords that cause trouble are "digitalkidz" (a tech conference) and "newtrendbg" (a Bulgarian company). If you do a quick search on Twitter for either of those you will see that there are results. However, tweepy doesn't find anything. Again, it does work for pretty much any other keyword I use.
Do you have any ideas what might be the problem and how to fix it?
Thank you
I believe you're forgetting an important aspect of the twitter api, it's not exhaustive.
Taken from the api docs
Please note that Twitter’s search service and, by extension, the Search API is not meant to be an exhaustive source of Tweets. Not all Tweets will be indexed or made available via the search interface.
Regardless of whether you're using the streaming or rest api, you're going to have issues with this if you're looking for specific tweets.
Rest API
When looking for historical tweets, you unfortunately won't be able to obtain anything that is older than a week using api.search(). This is also shown in the docs.
Keep in mind that the search index has a 7-day limit. In other words, no tweets will be found for a date older than one week.
There are other ways of getting older tweets, this post details those options.
Streaming API
While it doesn't sound like you're using twitter's streaming API, it should be noted that this only gives a small sample of twitter's current tweet traffic (~1-2%).
Hopefully this is helpful. Let me know if you have any questions.
I am trying to fetch all tweets using tweepy. I cannot find a document to show how to fetch all tweets rather than filter data using some keywords.
I tried "stream.filter()" without parameters, but it still does not work.
Could anyone give some help?
Thanks.
Is there any better way to get tweets from twitter then crawling twitter.com and mutating URLs?
If there is, how can I get latest tweets with current hash-tag?
Thank you!
Did you try Twitter REST API? Particulary, you can use search tweets endpoint. There are some limitations though, enforced by Twitter.
You can use one of many available python libraries.
For example, some sample code for tweepy can be found here.
I had an application running on GAE which accessed twitter Search API. However recently twitter has moved to the new version 1.1 and previous version of api is no longer functional. And I'm having trouble accessing the new API.
Basically I wanted to get tweets matching a search query and I simply accessed the url using python code and got all the data in json format. The url I accessed (which is not working now) is:
http://search.twitter.com/search.json?q=query
Instead I searched and found out that the equivalent url in new API is probably:
https://api.twitter.com/1.1/search/tweets.json?q=query
However if I just access it simply like I did before (just fetching the url and accessing the contents) then it gives an error and apparently I need to authenticate my request by obtaining some oauth credentials and using them with the request.
I tried searching of how to do that if there is another method beside just fetcihng the url contents, but in vain. I'd really appreciate if someone could help me out, and if possible give some sample python code. Thanks!
New Twitter API needs user authentication, meaning that each user visiting your site will need to go through auth procedure. This makes new Twitter API much less useable.
See this answer for details how to use OAuth on GAE/Python.