Tweepy Python suddenly becoming very slow, ProxyError - python

I've been using Tweepy for crawling tweets in certain area with geocode as parameter since 2 month ago. Everything was fine until last week, Tweepy suddenly becoming very slow and terminate itself.
Error message :
tweepy.error.TweepError:Failed to send request: HTTPSConnectionPool(host='api.twitter.com',port=443): Max retries exceeded with url: /1.1/search/tweet.json?count=200&geocode=-6.1750%2C106.8283%2C438.37km&since=2016-04-21&until=2016-04-22 (Caused by ProxyError('Cannot connect to proxy.', error(10054,'An existing connection was forcibly closed by the remote host')))
I've already set the proxy (as usual) using:
set http_proxy=http://152.118.24.10:8080
set https_proxy=https://152.118.24.10:8080
Anyone have same problem or know the solution?

Related

Python: request.get for github doesn't work

I am trying to run a GET request for a Github url. Unfortunately, I always get an error message.
I tried it for several different websites and it works, just not for github.
I am trying to do it with Jupyter Notebooks in Python, if that is important.
Here is the Error message:
ConnectionError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /jana-hoh/gdp/main/DP_LIVE_22102021141534889.csv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7a1c285d60>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
An error message that includes Temporary failure in name resolution indicates that the system's DNS server is unable to convert domain names into their corresponding IP address. Some of the causes are:
Your DNS configuration is correct, but the server is unable to respond to DNS requests at the moment
Firewall rules
No internet connectivity
Most of the times I've encountered this error stemmed from being disconnected to the internet. However, if your internet is working properly, you can try to add another DNS server in /etc/resolv.conf. For example, you can add cloudflare's:
nameserver 1.1.1.1

Complete list of causes of MaxRetryError

I am currently using from requests.packages.urllib3.util.retry import Retry to retry some API calls, but I seem to keep encountering different errors as the cause to ConnectionError / MaxRetryError. As I currently catch then generate custom errors, I'd like to go through a complete list of causes that lead to MaxRetryError. I thought this'd be easy to find, but I can't seem to find it anywhere.
Does anyone have any reference to a complete list of possible causes that can lead to ConnectionError / MaxRetryError? The only reference I seem to be able to find is this. Seems like this is an issue others are facing too.
An example of what this error looks like is this:
ConnectionError(MaxRetryError("HTTPSConnectionPool(host='localhost', port=8080): Max retries exceeded with url: ..... (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000026D26242688>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))"))

ebaysdk-python Connection Error

I have a django project which is currently being hosted on pythonanywhere that uses the Finding api of the open source project ebaysdk-python. Now on my local machine, the site worked perfectly, however, when I execute the api call I get this error message: HTTPConnectionPool(host='svcs.ebay.com', port=80): Max retries exceeded with url: /services/search/FindingService/v1 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6560105150>: Failed to establish a new connection: [Errno 111] Connection refused',)).
Now I have scoured the docs and other related questions, but could not figure out the issue. I have verified that my API keys are correct, and my code to execute the api call is straight from the docs. So that I may be pointed in the correct direction: What is the most likely cause for this error to be raised under these circumstances?
Thank you.

Socrata SODA API is rejecting with Max Retries Exceeded

I am using the REST API Modular Input within Splunk to GET data.SFGov.org data via SODA API. I have an APP TOKEN. I am getting the MAX RETRIES EXCEEDED repeatedly.
Background:
I am building a proto Splunk based stream cursor for SF opendata. I have been testing a GET using the REST API MODULAR INPUT from the configuration screen itself, have not written any python code yet. Here is the ERROR.
11-30-2016 16:24:57.432 -0800 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: HTTPSConnectionPool(host='data.sfgov.org', port=443): Max retries exceeded with url: [REDACTED] (Caused by : [Errno 8] nodename nor servname provided, or not known)
I found out that by mistake, the REST API module's polling interval was set to 60 seconds and it might have caused a problem? (I changed it to ONE DAY to avoid future issues). I then got a new APP TOKEN and tried a GET. I see the get going out in the log, but the same MAX RETRIES EXCEEDED error is coming. I am using the same IP address.
I will be testing for the next few weeks. How do I fix this and gracefully avoid this again?
#chrismetcalf - just flagging you.
Max Retries Exceeded is not an error message that I'd expect to see out of our API, especially if you were only making a call every 60 seconds. I think that may actually be Splunk giving up after trying and failing to make your HTTP call too many times.
The error message Caused by : [Errno 8] nodename nor servname provided, or not known makes me think that there's actually a DNS error on Splunk's side. That's the error message you usually see when a domain name can't be resolved.
Perhaps there's some DNS whitelisting you need to make in your Splunk environment?

socket Errno 10060

Im using requests to routinely download a webpage and check it for updates, but recently ive been getting these errors :
HTTPConnectionPool(host='somehost', port=someport): Max retries
exceeded with url: someurl (Caused by : [Errno
10060] A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond)
Now this script has been running for weeks with this issue never coming up. Could it be that the site administrator has started blocking my proxy's IP?
I should add that its not against the TOS of the site to scrape it.
Can anyone help me figure out whats the reason for this?
Thanks
The remote connection timed out.
The host you are trying to connect to is not answering; it is not refusing connections, it is just not responding at all to connection attempts.
Perhaps the host is overloaded or down? It could also be caused by the site blocking your IP address by dropping the packets (a firewall DROP rule instead of a REJECT rule).
You can try to connect to the site from a different IP address; if those connections work fine, but not from the original address, there is a higher likelihood that you are deliberately being blocked.

Categories