We Keep Coding

How to make it so a GET request using the Python request library never causes the program to fail?

I currently have the following method implemented that is meant to make a GET request (indefinitely until the call is successful in the event that the call is not successful):
def request_with_error_handling(self, url, filename):
current_datetime = str(datetime.now()).replace(" ", "_").replace(":",
".")
try:
rq = requests.get(url, headers=headers, proxies=random_proxy())
except Exception as e:
self.send_error_message(f"{url} - {filename}" + str(e))
retries = 0
while rq.status_code != 200:
retries += 1
try:
rq = requests.get(url, headers=headers, proxies=random_proxy())
except Exception as e:
self.send_error_message(f"{url} - {filename}" + str(e))
if retries == 4:
with open(filename + current_datetime, "w",
encoding="utf-8") as text_file:
text_file.write(rq.text)
self.send_error_message("Max retries exceeded error")
time.sleep(60)
retries = 0
return rq
My thought would be that this program would never fail as a result of a failed GET request, because I am wrapping every single GET request in a try catch block that should catch any exception, however, I am very occasionally getting some errors that cause the program to fail and I am not sure what I can do to stop the program from doing so.
Here is part of the the latest stack trace that caused one of my threads to fail. I could not post the whole thing, since it was well over the 30k character limit allowed by stack overflow. It over 100k in characters:
Exception in thread Thread-57720:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 488, in wrap_socket
cnx.do_handshake()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1663, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (10053, 'WSAECONNABORTED')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
self._prepare_proxy(conn)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 932, in _prepare_proxy
conn.connect()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 371, in connect
ssl_context=context,
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\ssl_.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 494, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: SysCallError(10053, 'WSAECONNABORTED')",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.random_website_test_blah_blah_blah43149676132846.co.uk', port=443): Max retries exceeded with url: /sch//blah3&_sop=10&_ipg=200&rt=nc (Caused by SSLError(SSLError("bad handshake: SysCallError(10053, 'WSAECONNABORTED')")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 93, in request_with_error_handling
rq = requests.get(url, headers=headers, proxies=random_proxy())
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.random_website_test_blah_blah_blah43149676132846.co.uk', port=443): Max retries exceeded with url: /sch//blah3&_sop=10&_ipg=200&rt=nc (Caused by SSLError(SSLError("bad handshake: SysCallError(10053, 'WSAECONNABORTED')")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11004] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
conn.connect()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 309, in connect
conn = self._new_conn()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x14C92E30>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='discord.com', port=443): Max retries exceeded with url: /api/webhooks/817872766140153857/QEOUFVJSlKdCuf4suHb-J6_LvPB6-0R0Taq9IONrUxj-jWODWhBlXh0WBLNPXtS0F5hk?wait=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x14C92E30>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 260, in consecutive_scrape
page)
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 95, in request_with_error_handling
self.send_discord_error(f"{url} - {filename}, {page}" + str(e))
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 87, in send_discord_error
response = webhook.execute()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\discord_webhook\webhook.py", line 142, in execute
response = requests.post(url, json=self.json, proxies=self.proxies, params={'wait': True})
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='discord.com', port=443): Max retries exceeded with url: /api/webhooks/817872766140153857/QEOUFVJSlKdCuf4suHb-J6_LvPB6-0R0Taq9IONrUxj-jWODWhBlXh0WBLNPXtS0F5hk?wait=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x14C92E30>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed'))
Exception in thread Thread-57723:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 488, in wrap_socket
cnx.do_handshake()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1663, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (10053, 'WSAECONNABORTED')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 667, in urlopen
self._prepare_proxy(conn)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 932, in _prepare_proxy
conn.connect()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 371, in connect
ssl_context=context,
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\ssl_.py", line 384, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 494, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: SysCallError(10053, 'WSAECONNABORTED')",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.random_website_test_blah_blah_blah43149676132846.co.uk', port=443): Max retries exceeded with url: /blah (Caused by SSLError(SSLError("bad handshake: SysCallError(10053, 'WSAECONNABORTED')")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 93, in request_with_error_handling
rq = requests.get(url, headers=headers, proxies=random_proxy())
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.random_website_test_blah_blah_blah43149676132846.co.uk', port=443): Max retries exceeded with url: /blah (Caused by SSLError(SSLError("bad handshake: SysCallError(10053, 'WSAECONNABORTED')")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11004] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
conn.connect()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 309, in connect
conn = self._new_conn()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x14C92890>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='discord.com', port=443): Max retries exceeded with url: /api/webhooks/817872766140153857/QEOUFVJSlKdCuf4suHb-J6_LvPB6-0R0Taq9IONrUxj-jWODWhBlXh0WBLNPXtS0F5hk?wait=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x14C92890>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 260, in consecutive_scrape
page)
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 95, in request_with_error_handling
self.send_discord_error(f"{url} - {filename}, {page}" + str(e))
File "C:/Users/user/PycharmProjects/random_website_test_blah_blah_blah43149676132846/restocks/restock_monitor.py", line 87, in send_discord_error
response = webhook.execute()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\discord_webhook\webhook.py", line 142, in execute
response = requests.post(url, json=self.json, proxies=self.proxies, params={'wait': True})
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='discord.com', port=443): Max retries exceeded with url: /api/webhooks/817872766140153857/QEOUFVJSlKdCuf4suHb-J6_LvPB6-0R0Taq9IONrUxj-jWODWhBlXh0WBLNPXtS0F5hk?wait=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x14C92890>: Failed to establish a new connection: [Errno 11004] getaddrinfo failed'))
Exception in thread Thread-57719:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\contrib\pyopenssl.py", line 488, in wrap_socket
cnx.do_handshake()
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "C:\Users\user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\OpenSSL\SSL.py", line 1663, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (10053, 'WSAECONNABORTED')
During handling of the above exception, another exception occurred:

You will need to catch the error as well:
try:
rq = requests.get(url, headers=headers, proxies=random_proxy())
except Exception as e:
self.send_error_message(f"{url} - {filename}" + str(e))
except requests.exceptions.SSLError as er:
self.send_error_message(f"{url} - {filename}" + str(er))
How to log error, you would need better but here are few functions that you can use to based your decision on:
er = SystemError("My error")
tr(er)
'My error'
er.args
('My error',)
UPDATE:
This is working snippet:
import requests
url = "abc"
filename = "xyz"
try:
# rq = requests.get(url, headers=headers, proxies=random_proxy())
raise requests.exceptions.SSLError("Handshake!")
except Exception as e:
print(f"{url} - {filename}" + str(e))
except requests.exceptions.SSLError as er:
print(f"{url} - {filename}" + str(er))
# Output: abc - xyzHandshake!
UPDATE 2:
Code on the GitHub

AWS Glue | API call | Python Shell | Connection | Failed to establish a new connection: [Errno 110] Connection timed out

We are trying to call An API using Python Shell in AWS glue. We are making an HTTPS call using python 3.6 using "requests" library. The issue is pertaining to AWS glue.
The Job config is
Python Shell
Python 3 ( Glue Version 1.0 )
This is the error that we get
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='xyz.com', port=443): Max retries exceeded with url: /api-v2/client/orders/?auth_token=#####&start_date=01-04-2021&end_date=02-04-2021&status_filter=all&page_number=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 123, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-fhh0x612/test.py", line 16, in <module>
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='xyz.com', port=443): Max retries exceeded with url: /api-v2/client/orders/?auth_token=#####0&start_date=01-04-2021&end_date=02-04-2021&status_filter=all&page_number=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 142, in <module>
raise e_type(e_value).with_traceback(new_stack)
File "/tmp/glue-python-scripts-fhh0x612/test.py", line 16, in <module>
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='xyz.com', port=443): Max retries exceeded with url: /api-v2/client/orders/?auth_token=####&start_date=01-04-2021&end_date=02-04-2021&status_filter=all&page_number=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out',))
Is there an issue with the security group that AWS Glue refers to or some issue with the API call. We are not able to figure out.

Is there more than one connection from glue? like calling S3 , redshift...etc. if more than one then u may have to add other connections in connection properties. Edit job -> add connection.

Unable to run script through selenium grid. Got timeout error

Here is the log for Hub creation
Here is the log for Node creation
Here is Grid console output
Driver folder
This is the Python Script I wrote
import time
import unittest
from selenium import webdriver
class SearchProducts(unittest.TestCase):
def setUp(self):
desired_caps = {}
desired_caps['platform'] = 'WINDOWS'
desired_caps['browserName'] = 'chrome'
self.driver = webdriver.Remote('http://192.168.43.250:4444/wd/hub', desired_caps)
self.driver.get('http://demo.magentocommerce.com/')
self.driver.implicitly_wait(30)
self.driver.maximize_window()
def testSearchByCategory(self):
time.sleep(5)
# get the search textbox
self.search_icon = self.driver.find_element_by_xpath("//span[contains(., 'Search')]/preceding::span/div")
self.search_icon.click()
time.sleep(5)
self.search_field = self.driver.find_element_by_xpath("//div[#class='inputWrapper']//input[#id='edit-keys']")
self.search_field.click()
self.search_field.clear()
# enter search keyword and submit
self.search_field.send_keys('phones')
self.search_field.submit()
time.sleep(5)
def tearDown(self):
# close the browser window
self.driver.quit()
When ran script I got below log at Console
Error
Traceback (most recent call last):
File "C:\Users\jyothi\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "C:\Users\jyothi\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\util\connection.py", line 84, in create_connection
raise err
File "C:\Users\jyothi\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\util\connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1004, in _send_output
self.send(msg)
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 944, in send
self.connect()
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connection.py", line 187, in connect
conn = self._new_conn()
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x000001A30BD477C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\unittest\case.py", line 60, in testPartExecutor
yield
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\unittest\case.py", line 672, in run
self._callSetUp()
File "C:\Users\AppData\Local\Programs\Python\Python38\lib\unittest\case.py", line 630, in _callSetUp
self.setUp()
File "C:\Users\PycharmProjects\apautomationpython\src\test\automation\web\test.py", line 12, in setUp
self.driver = webdriver.Remote('http://192.168.43.250:4444/wd/hub', desired_caps)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 319, in execute
response = self.command_executor.execute(driver_command, params)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 374, in execute
return self._request(command_info[0], url, body=data)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 402, in _request
resp = http.request(method, url, body=body, headers=headers)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\request.py", line 79, in request
return self.request_encode_body(
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\request.py", line 171, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\poolmanager.py", line 336, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connectionpool.py", line 752, in urlopen
return self.urlopen(
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connectionpool.py", line 752, in urlopen
return self.urlopen(
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connectionpool.py", line 752, in urlopen
return self.urlopen(
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "C:\Users\PycharmProjects\apautomationpython\venv\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.43.250', port=4444): Max retries exceeded with url: /wd/hub/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001A30BD477C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
Everything seems fine to me but Whats going wrong? How to resolve this error? Please help

How to prevent the `requests` module error when no DNS resolution is available?

While using the Python requests module to check for broken links (expected behavior being returned error codes for unavailable URLs such as 404, etc.), I encountered an older link on a page for a domain which no longer exists, and a DNS error was returned.
<snip>
http://listen.grooveshark.com/s/Folk+Song/2Np5i?src=5
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 56, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 196, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x1088ff748>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='listen.grooveshark.com', port=80): Max retries exceeded with url: /s/Folk+Song/2Np5i?src=5 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1088ff748>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./four_finder.py", line 64, in <module>
do_main()
File "./four_finder.py", line 48, in do_main
resp = requests.get(link)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 513, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='listen.grooveshark.com', port=80): Max retries exceeded with url: /s/Folk+Song/2Np5i?src=5 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1088ff748>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
[user#host:~/dev|22:13:13]
$ nslookup listen.grooveshark.com
(...)
** server can't find listen.grooveshark.com: SERVFAIL
[user#host:~/dev|22:28:05]
What is the best way to preempt this error from occurring? I can't be confident of future input URLs existing (as this example does not exist).

This is solved by using a try/except block and requests.ConnectionError. I hadn't searched SE for the right terms, and a useful answer is here.
try:
resp = requests.get(link)
except requests.ConnectionError as e:
### TODO: Properly handle this DNS error!
print('DNS ERROR: %s' % link)
continue
Also of interest is this answer.

I am using the bing news search api to accumulate news on given keywords, however, I am timing out quite frequently

This is the code in question :
#app.route('/getAcadNews')
def getAadNews():
headers = {
'accept': "application/json",
'cache-control': "no-cache",
'Ocp-Apim-Subscription-Key': "<API KEY HERE>"
}
news_items = {}
query = removeBadOrdinal(request.args.get('kws'))
url = "https://api.cognitive.microsoft.com/bing/v5.0/news/search?"
querystring = {"q":"\""+query+"\" and \"papers published\"","count":"10000","freshness":"Month"}
try:
papernews = json.loads(requests.request("GET", url, headers=headers, params=querystring).text)
except requests.exceptions.ConnectionError:
papernews = {}
if "totalEstimatedMatches" in papernews:
news_items['papers_published_news'] = papernews["totalEstimatedMatches"]
news_items['papers_published_news_item'] = getTopNews(papernews)
else:
news_items['papers_published_news'] = 0
news_items['papers_published_news_item'] = {}
querystring = {"q":"\""+query+"\" and \"patent\"","count":"10000","freshness":"Month"}
try:
patentnews = json.loads(requests.request("GET", url, headers=headers, params=querystring).text)
except requests.exceptions.ConnectionError:
patentnews = {}
if "totalEstimatedMatches" in patentnews:
news_items['patent_news'] = patentnews["totalEstimatedMatches"]
news_items['patent_news_item'] = getTopNews(patentnews)
else:
news_items['patent_news'] = 0
news_items['patent_news_item'] = {}
resp1 = (json.dumps(news_items, indent=4))
resp = Response(response=resp1,
status=200, \
mimetype="application/json")
return(resp)
This works sometimes, but at times, I am faced with the following error :
ERROR in app: Exception on /getAcadNews [GET]
Traceback (most recent call last):
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 142, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py", line 98, in create_connection
raise err
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/util/connection.py", line 88, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
chunked=chunked)
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 352, in _make_request
self._validate_conn(conn)
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 831, in _validate_conn
conn.connect()
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 254, in connect
conn = self._new_conn()
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 151, in _new_conn
self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d7a40b080>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/DataParser/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 640, in urlopen
_stacktrace=sys.exc_info()[2])
File "/root/DataParser/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 287, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.cognitive.microsoft.com', port=443): Max retries exceeded with url: /bing/v5.0/news/search?count=10000&q=%22+mobile+payments+platform%22+and+%22papers+published%22&freshness=Month (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d7a40b080>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/root/DataParser/lib/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/root/DataParser/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/root/DataParser/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "./webAppDisplay.py", line 978, in getAadNews
papernews = json.loads(requests.request("GET", url, headers=headers, params=querystring).text)
File "/root/DataParser/lib/python3.5/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/root/DataParser/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/root/DataParser/lib/python3.5/site-packages/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/root/DataParser/lib/python3.5/site-packages/requests/adapters.py", line 487, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.cognitive.microsoft.com', port=443): Max retries exceeded with url: /bing/v5.0/news/search?count=10000&q=%22+mobile+payments+platform%22+and+%22papers+published%22&freshness=Month (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d7a40b080>: Failed to establish a new connection: [Errno 110] Connection timed out',))
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /getAcadNews?kws=mobile%20payments%20platform (ip 14.102.84.2) !!!
uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during GET /getAcadNews?kws=mobile%20payments%20platform (14.102.84.2)
OSError: write error
It would be really helpful if somebody could please help me solve this problem. I am in the paid S2 tier. This problem occurs mostly when this function is called 3+ times back to back.

Unfortunately there is nothing you or I could do, it is Microsoft's fault. Although you may not like this answer, they are limiting requests and bandwidth. If you have slow internet that could be a factor.