The script bellow is supposed to send the data to the url when the Google Compute Engine instance (using Container-Optimized OS image) is started and the dockerized app working.
Unfortunately, even if it fails to post the data, the data is received when the app is working.
The output is:
('Error',
ConnectionError(MaxRetryError("HTTPConnectionPool(host='34.7.8.8',
port=12345): Max retries exceeded with url: /didi.json (Caused by
NewConnectionError(': Failed to establish a new connection: [Errno 111]
Connection refused',))",),))
Does it come from GCE ?
Here is the python code:
for i in range(0,100):
while True:
try:
response = requests.post('http://%s:12345/didi.json' % ip_of_instance, data=data)
except requests.exceptions.RequestException as err:
print ("Error",err)
time.sleep(2)
continue
break
Edit - here are the parameters of the post request:
data = {
'url': 'www.website.com',
'project': 'webCrawl',
'spider': 'indexer',
'setting': 'ELASTICSEARCH_SERVERS=92.xx.xx.xx',
'protocol': 'https',
'scraper': 'light'
}
What I see is that you are using a while true loop, when it exceeds maximum retrys you get an error because you are being banned by the server but this status does not long forever, and when the banning is removed you start to get more data because the while still running.
If my theory is not right you can take a look at this other thread.
Max retries exceeded with URL
Related
I'm trying to run some code from this website but I don't understand why I get this error:
qbittorrentapi.exceptions.APIConnectionError: Failed to connect to qBittorrent. Connection Error: ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v2/auth/login (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001FA519F5840>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))"))
The code in question:
import qbittorrentapi
# instantiate a Client using the appropriate WebUI configuration
qbt_client = qbittorrentapi.Client(
host='localhost',
port=8080,
username='admin',
password='adminadmin',
)
# the Client will automatically acquire/maintain a logged-in state
# in line with any request. therefore, this is not strictly necessary;
# however, you may want to test the provided login credentials.
try:
qbt_client.auth_log_in()
except qbittorrentapi.LoginFailed as e:
print(e)
# display qBittorrent info
print(f'qBittorrent: {qbt_client.app.version}')
print(f'qBittorrent Web API: {qbt_client.app.web_api_version}')
for k,v in qbt_client.app.build_info.items(): print(f'{k}: {v}')
# retrieve and show all torrents
for torrent in qbt_client.torrents_info():
print(f'{torrent.hash[-6:]}: {torrent.name} ({torrent.state})')
# pause all torrents
qbt_client.torrents.pause.all()
I'd really appreciate some help with this, thanks ahead :)
I'm currently working with an Imgur Bot using Python and requests lib with a rotating proxy. I run multiple instances of this bot, but sometimes, some of them got this error:
HTTPSConnectionPool(host='api.imgur.com', port=443): Max retries exceeded with url: /3/credits (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response')))
Here is how the session is defined:
session = requests.Session()
session.proxies = {"http": cre["Proxy"]}
I have no idea why it crashes since only a part instances I run got this error. The others work well.
Thanks, mates!
Maybe your proxies are wrong configurated?
I think you run into some form of "requesting too much too fast".
Try to delay some requests (like 3 seconds).
I have a CherryPy server, which is processing requests using a thread pool of, say, 10 threads.
If I send many requests in parallel (~200 processes constantly sending), I start seeing in my logs (client side) 3 types of errors:
HTTPConnectionPool(host='localhost', port=8080):
Max retries exceeded with url: /my-url
(Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1095da780>:
Failed to establish a new connection: [Errno 61] Connection refused'))
BrokenPipeError(32, 'Broken pipe'))
and
('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
Why 3 different types, btw?
I suppose I see these errors because there were too many requests sent.
I adjusted server.socket_queue_size to match the number of parallel requests I send, and it started working fine. My server configuration looks like this:
cherrypy.config.update({
'server.socket_port': 8080,
'server.socket_host': 'localhost',
'server.thread_pool': 10,
'server.socket_queue_size': 200
})
However, I'm struggling to find the default value of socket_queue_size set in CherryPy. How much is it? cherrypy.config['server.socket_queue_size'] gives me nothing unsell I set this value myself.
So, what is the default value? How do I determine reasonable socket_queue_size?
I'm trying to make a simple request to a whatsmyip site while connected to tor but no matter what I try I continue to get this error:
requests.exceptions.ConnectionError: SOCKSHTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /get (Caused by NewConnectionError('<urllib3.contrib.socks.SOCKSHTTPSConnection object at 0x1018a7438>: Failed to establish a new connection: [Errno 61] Connection refused'))
I've looked at a lot of posts on here with similar issues but I can't seem to find a fix that works.
This is the current code but I've tried multiple ways and its the same error every time:
import requests
def main():
proxies = {
'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'
}
r = requests.get('https://httpbin.org/get', proxies=proxies)
print(r.text)
if __name__ == '__main__':
main()
Well the error says Max retries exceeded with url:, so possibly could be too many requests has been made from the tor exit nodes ip. Attempt to do it with a new Tor identity and see if that works.
If you wanted to you could catch the exception and put it in a loop to attempt every number of seconds, but this may lead to that ip address being refused by the server for longer.
As part of a project to scrape data from Craigslist, I include image scraping. I've noticed in testing that sometimes the connection is refused. Is there a way around this, or do I need to incorporate error catching for this in my code? I recall the twitter API limits queries, so a sleep timer is incorporated. Curious if I have the same situation with Craigslist. See code and error below.
import requests
from bs4 import BeautifulSoup
#loops through each image and stores it in a local folder
for img in soup_test.select('a.thumb'):
imgcount += 1
filename = (pathname + "/" + motoid + " - "+str(imgcount)+".jpg")
with open(filename, 'wb') as f:
response = requests.get(img['href'])
f.write(response.content)
ConnectionError: HTTPSConnectionPool(host='images.craigslist.org', port=443): Max retries exceeded with url: /00707_fbsCmug4hfR_600x450.jpg (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it',))
I have 2 questions about this behavior.
Do CL servers have any rules or protocols such as blocking nth request within a certain time frame?
Is there a way to pause the loop after a connection has been denied? Or do I just incorporate error catching so that it doesn't halt my program?