I would like to be able to call a function, in any language (preferably Python :)), where the parameters I pass in are simply a date, or date range, and I get returned a list of the topics that were trending on Twitter (or any other social media) on these dates.
Is it possible to do this? How can I do this if so?
Thank you very much.
It doesn't seem possible through the official 1.1 twitter API, which only provides currently trending topics, but http://twend.it archives trending topics on twitter. You may be able to scrape them or send an email asking if they have an API.
Related
I am rn trying to scrape twitter for an nlp research. I already used snscrape to get tweets with the required filters, the issue is that we need tweets from a specific age range. In my head I guess some profiles on twitter have their birthdate public, so maybe we can fetch that. Maybe webscrape that from the profile? Any ideas are welcomed.
Till now I have tried some methods of webscraping but can't find something concrete
Twitter has a pretty well documented API that works very well with Python.
Try to make a simple crawler and see one of the JSONs that you get for a Tweet/User.
You will need to sign up and get some Access Tokens/Keys to use in your script, but other than that you are ready to go:
https://developer.twitter.com/en/docs/twitter-api
The age of Twitter users is not made available via the API (and the website may show birthday but not year). There are also a number of other factors you should read about in relation to analysis of Twitter user data.
What I know:
I have been using Tweepy for a long period and I know how you can create a stream to get new Tweets with a given keyword.
I also know that you can get trending keywords in a specific region from the documentation.
What I am trying to achieve:
I was wondering how I can use the trending APIs to get trending topic but in a form of a stream, so if a topic starts trending in a place, I will like a similar event to the on_data event on the stream listener to get fired and return the new trending topic.
Does anyone know how I can achieve this?
Or any hint on how I can achieve this?
Twitter's API has no streaming endpoint for this, so you'll simply have to poll the GET trends/place endpoint with API.trends_place.
I am writing a script in Python, that uses tweepy to search for tweets with a given keyword. Here is the snippet:
for tweet in tweepy.Cursor(api.search, q=keyword, lang="en").items(10):
print tweet.id
I have everything authenticated properly and the code works most of the time. However, when I try to search for some keywords (examples below) it doesn't return anything.
The keywords that cause trouble are "digitalkidz" (a tech conference) and "newtrendbg" (a Bulgarian company). If you do a quick search on Twitter for either of those you will see that there are results. However, tweepy doesn't find anything. Again, it does work for pretty much any other keyword I use.
Do you have any ideas what might be the problem and how to fix it?
Thank you
I believe you're forgetting an important aspect of the twitter api, it's not exhaustive.
Taken from the api docs
Please note that Twitter’s search service and, by extension, the Search API is not meant to be an exhaustive source of Tweets. Not all Tweets will be indexed or made available via the search interface.
Regardless of whether you're using the streaming or rest api, you're going to have issues with this if you're looking for specific tweets.
Rest API
When looking for historical tweets, you unfortunately won't be able to obtain anything that is older than a week using api.search(). This is also shown in the docs.
Keep in mind that the search index has a 7-day limit. In other words, no tweets will be found for a date older than one week.
There are other ways of getting older tweets, this post details those options.
Streaming API
While it doesn't sound like you're using twitter's streaming API, it should be noted that this only gives a small sample of twitter's current tweet traffic (~1-2%).
Hopefully this is helpful. Let me know if you have any questions.
Is there any better way to get tweets from twitter then crawling twitter.com and mutating URLs?
If there is, how can I get latest tweets with current hash-tag?
Thank you!
Did you try Twitter REST API? Particulary, you can use search tweets endpoint. There are some limitations though, enforced by Twitter.
You can use one of many available python libraries.
For example, some sample code for tweepy can be found here.
I have the following questions about tweepy python module
1.I am trying to retrieve all tweets for a specific location. I am able to do this by using tweepy python module (streaming API), but I get only those tweets whose geo locations are enabled, which means I would loose rest of the tweeter’s tweet who have not enabled their geo location. Is there a better way to retrieve all the tweets, given a location?
2.I use Stream.Sample method to retrieve all the tweets, Can someone tell me about the parameters used in sample method? I see count, and async as parameters. Now what should we specify here?
3.What does firehose method in tweepy.Stream do?
Any help is much appreciated
If tweepy doesn't have a feature you need, you can always access Twitter directly with an HTTP request. The full Twitter REST API is described here: https://dev.twitter.com/docs/api
The ones that seem relevant to your interest are:
GET trends/:woeid which looks up tweets by woeid, a Yahoo Identifier for collecting information about a given place/landmark/etc.
GET geo/id/:place_id which only mines geotagged tweets.
There is documentation of all the information available for a GET request but the IP address is not among the available fields: https://dev.twitter.com/docs/api/1/get/search .
Lastly, Twitter has a location search FAQ that may be of interest.