So I have run into the issue of getting data from Google Finance. They have an html access system that you can use to access webpages that give stock data in simple text format (ideal for minimizing parsing). However, if you access this service too frequently, Google locks you out and you need to enter a captcha. I currently have a list of about 50 stocks and I want to update my price data every 15 seconds, but I soon get locked out (after about 3-4 minutes).
Does anyone have any solutions to this/understand the nature of how often is the max I could ping Google for this information?
Not sure why a feature like this would be on a service designed to give data like this... but similar alternative services with realtime data would also be accepted.
Probably because your usage is not what they intended. Every 40x every 15 seconds seems a bit excessive. They had an API that got discontinued some years ago, and there's another SO question with some available alternatives which is probably also a bit out of date.
From google, there is also it's Finance service with getStockInfo which allows to query its database but read their warnings.
Yahoo YQL works fairly well, but throws numerous HTTP 500 errors that need to be handled, they are all benign. TradeKing is an option, however, the oauth2 package is required and that is very difficult to install properly
Related
I just recently finished a nearly 6 week long conversation with multiple people on Twitter. Since several things that were said were quite interesting (particularly in hindsight), I'd like to be able to archive the entire conversation for reference later. From what I can tell, there are no existing solutions similar to threadreaderapp.com to recursively unroll an entire conversation. As such, I looked into doing it in Python with the Twitter API. In researching it, I found several people saying the free version of the API only lets you search replies from the last 7 days. However, then I found some places (e.g., here) that seemed to indicate the Twitter API v2 added access to a "conversation ID" that enabled this limitation to be avoided. However, when I tried to run that code to get the replies to my tweet, the response kept coming back empty. Specifically, as best I can tell, the request from line 19 of this code (link ... which is the code from step 7 of the previously mentioned article: direct link) is not returning data.
Am I missing something? Is it possible to recursively get all replies to a tweet from the past 6 weeks without needing to be considered an "Academic Researcher" to be able to access the full Twitter archive (reference)?
Ultimately, I can get all the tweets from the website in the browser, so I suppose if I knew what I was doing I could just use some sort of a HTML scraper or something, but I don't.
The Twitter API v2 allows you to use the conversation_id as a search parameter on both the recent search, and full archive search, endpoints. The difference is that the recent search API covers the past seven days (available in the Essential access tier / most users), and the full archive search API is limited to Academic access at this time.
So, to directly answer your question: no, the API does not allow you to recursively get all replies to a Tweet from the past 6 weeks, unless you are indeed a qualified academic researcher with access to the full archive search functionality.
Other retrieval methods are beyond the scope of the API and are not supported by Twitter.
As the title states I'm looking for a way to get backlinks for a given url / website using Google APIs, since I already have an api key and I'd rather use it instead of relying on other services.
I already tested services like ahrefs, majestic, moz, serpstat etc and actually they can give me the infomation I need, but I was wondering if there was a way to do it with Google.
For what I've read during my past researches I saw that Google offered a way to do it, but then it became deprecated, so no more usable. Do they really took away this feature for good?
I've also noticed that Google offers a similar service with his Google Search Console, but it can just be used for your own website, I'd like to get those kind of information for a random given url.
Actually I will be using Python in my project, but I don't think there's a package able to deliver me these kind of data, or at least I looked for it and didn't find anything.
Any help would be appreciated.
I realize that versions of this question have been asked and I spent several hours the other day trying a number of strategies.
What I would like to is use python to scrape all of the URLs from a google search that I can use in a separate script to do text analysis of a large corpus (news sites mainly). This seems relatively straightforward, but none of the attempts I've tried have worked properly.
This is as close as I got:
from google import search
for url in search('site:cbc.ca "kinder morgan" and "trans mountain" and protest*', stop=100):
print(url)
This returned about 300 URLs before I got kicked. An actual search using these parameters provides about 1000 results and I'd like all of them.
First: is this possible? Second: does anyone have any suggestions to do this? I basically just want a txt file of all the URLs that I can use in another script.
It seems that this package uses screen scraping to retrieve search results from google, so it doesn't play well with Google's Terms of Service which could be the reason why you've been blocked.
The relevant clause in Google's Terms of Service:
Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide. You may use our Services only as permitted by law, including applicable export and re-export control laws and regulations. We may suspend or stop providing our Services to you if you do not comply with our terms or policies or if we are investigating suspected misconduct.
I haven't been able to find a definite number, but it seems like their limit for the number of search queries a day is rather strict too - at 100 search queries / day on their JSON Custom Search API documentation here.
Nonetheless, there's no harm trying out other alternatives to see if they work better:
BeautifulSoup
Scrapy
ParseHub - this one is not in code, but is a useful piece of software with good documentation. Link to their tutorial on how to scrape a list of URLs.
I'm working with the github API in python.
Github limits most GET responses to a max of 100 items/per page. Each request takes a couple of seconds, so a 20 page request is not a great user experience.
What is the pythonic way of making the requests concurrently.
There are a number of ways to do it. The easiest is probably to use something like the concurrent.futures module (or the backported version for Python 2.x).
However, one very important thing to keep in mind is that GitHub apply rate limits to their API, and you can't just make as many requests as you like without hitting up against those rate limits. So make sure to check for HTTP/429 errors and back off accordingly.
For a research project, I am collecting tweets using Python-Twitter. However, when running our program nonstop on a single computer for a week we manage to collect about only 20 MB of data per week. I am only running this program on one machine so that we do not collect the same tweets twice.
Our program runs a loop that calls getPublicTimeline() every 60 seconds. I tried to improve this by calling getUserTimeline() on some of the users that appeared in the public timeline. However, this consistently got me banned from collecting tweets at all for about half an hour each time. Even without the ban, it seemed that there was very little speed-up by adding this code.
I know about Twitter's "whitelisting" that allows a user to submit more requests per hour. I applied for this about three weeks ago, and have not hear back since, so I am looking for alternatives that will allow our program to collect tweets more efficiently without going over the standard rate limit. Does anyone know of a faster way to collect public tweets from Twitter? We'd like to get about 100 MB per week.
Thanks.
How about using the streaming API? This is exactly the use-case it was created to address. With the streaming API you will not have any problems gathering megabytes of tweets. You still won't be able to access all tweets or even a statistically significant sample without being granted access by Twitter though.
I did a similar project analyzing data from tweets. If you're just going at this from a pure data collection/analysis angle, you can just scrape any of the better sites that collect these tweets for various reasons. Many sites allow you to search by hashtag, so throw in a popular enough hashtag and you've got thousands of results. I just scraped a few of these sites for popular hashtags, collected these into a large list, queried that list against the site, and scraped all of the usable information from the results. Some sites also allow you to export the data directly, making this task even easier. You'll get a lot of garbage results that you'll probably need to filter (spam, foreign language, etc), but this was the quickest way that worked for our project. Twitter will probably not grant you whitelisted status, so I definitely wouldn't count on that.
There is pretty good tutorial from ars technica on using streaming API n Python that might be helpful here.
Otherwise you could try doing it via cURL.
.