I'm running many celery tasks (20,000) using gevent for the pool (also monkey patching all). Each of these tasks hit 3rd party services like adwords to pull data.
I keep having tasks fail because of underlying SSL errors. Below are the stack-traces from a few of the exceptions (in no particular order, these are failures from separate tasks). I also get WantWriteError and ZeroReturnError occasionally but the EOF error seems to come up the most.
These errors happen while using different client libraries like googleads (suds library for soap communication) as well as requests and elasticsearch. I'm guessing some of these libraries use urllib3 while others use urllib2 etc.
There has been a lot of info on the EOF issue and forcing TLSv1 but I can't seem to find a resolution that works.
I'm not sure if I'm running too many requests at once, if somethings blocking or what; any help would be greatly appreciated, I'm pulling my hair out over this one.
Traceback (most recent call last):
...
File "/srv/reporting/src/reporting/stats/adwords/client.py", line 58, in _awql_report
downloader = self._get_client(client_id).GetReportDownloader(version=self.REPORT_DOWNLOADER_VERSION)
File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 283, in GetReportDownloader
return ReportDownloader(self, version, server)
File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 400, in __init__
proxy=proxy_option, cache=self._adwords_client.cache).wsdl.schema
File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 115, in __init__
self.wsdl = reader.open(url)
File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 150, in open
d = self.fn(url, self.options)
File "/usr/local/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__
d = reader.open(url)
File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 74, in open
d = self.download(url)
File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 92, in download
fp = self.options.transport.open(Request(url))
File "/usr/local/lib/python2.7/dist-packages/suds/transport/https.py", line 62, in open
return HttpTransport.open(self, request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 67, in open
return self.u2open(u2request)
File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1216, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
Traceback (most recent call last):
...
File "/srv/reporting/src/reporting/stats/analytics/client.py", line 57, in get_access_token
response = requests.post('https://accounts.google.com/o/oauth2/token', data)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')
Traceback (most recent call last):
...
self.es.index(index=self.INDICE, doc_type=self.ROOT_CLASS.__name__, body=self.export(obj), id=obj.id)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 213, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 284, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 44, in perform_request
response = self.session.request(method, url, data=body, timeout=timeout or self.timeout)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
timeout=timeout
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
body=body, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
httplib_response = conn.getresponse(buffering=True)
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline()
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 273, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 995, in recv
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
raise ZeroReturnError()
ZeroReturnError
So let's break this down by each traceback block. The first ends with:
File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
This is coming from urllib2. The fact that this receives an EOF makes me think that the server closed the connection while you were waiting for that "thread" to read from the socket again. You might want to use more time.sleep(0) to yield to gevent.
The second traceback comes from requests:
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')
The [Errno bad handshake] would make me tend to think this is a problem establishing the connection which could be caused by an unexpected EOF. Is that caused by using gevent? I'm uncertain.
The final traceback is definitely from requests as well but it also is coming out of PyOpenSSL and isn't being caught by urllib3 or requests.
File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
raise ZeroReturnError()
ZeroReturnError
I did some searching and found that "According to the pyOpenSSL docs ZeroReturnError means that the SSL connection has been closed cleanly." This says to me that the server again closed the connection because you took to long to read anything from the socket.
In short, I think you need to explicitly yield more often just to ensure that these socket problems don't arise. That's just a guess though, so take it with a grain of salt.
Related
I have a python script in a docker that runs and collects data from an api on a loop.
`for idx, items in enumerate(SearchList):
#get access token
access_token = GetPricingAccTok()
#request id to get results
reqid = RequestIdCompSearch(items, access_token)
#api to get results by doing a while loop appending results together till api returns 100% as completed percentage
Result = GetReqResults(reqid, access_token)
Pricings = GetPricingDict(Result)
filenamepricing = "pricinghistory/CorrectFormat/Pricings_Data_{idn}_{ts}.json".format(
idn = idx,
ts= dt.datetime.now().isoformat()
)
GetAzContainer().upload_blob(
name=filenamepricing,
data=json.dumps(obj= Pricings, indent=4),
blob_type='BlockBlob'
)
`
When i run and test this code locally on my windows machine it works well without an issue and completes the requests
but in the docker on azure container instance it then returns an error showing that the connection has timed out. Not sure what the difference is between the running it locally and in the docker that i then recieve this error.
I have tried addinng this code at the start
import socket
from urllib3.connection import HTTPConnection
HTTPConnection.default_socket_options =(HTTPConnection.default_socket_options + [
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) #Enables the feature
,(socket.SOL_TCP, socket.TCP_KEEPIDLE, 45) #Overrides the time when the stack willl start sending KeppAlives after no data received on a Persistent Connection
,(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10) #Defines how often thoe KA will be sent between them
,(socket.SOL_TCP, socket.TCP_KEEPCNT, 60) #How many attemps will your code try if the server goes down before droping the connection.
]
)
but it still times out. showing the below error
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 353, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='service.xxx.com', port=443): Read timed out. (read timeout=None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/script.py", line 298, in <module>
reqid = RequestIdCompSearch(items, access_token)
File "src/script.py", line 111, in RequestIdCompSearch
InitResp =rq.post( ReqUrl, headers= ReqHead, data=Jsondata)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='service.xxxx.com', port=443): Read timed out. (read timeout=None)
I have two telegram bots made with PyTelegramBotApi (not the best library, I know) deployed on Heroku. One big difference that matters: one is running with Threading to periodically send notifications to "subscribers" (around 20 people). And it crashes approximately every month with following exception trace in Heroku console:
2022-01-14T08:00:11.068553+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/telebot/apihelper.py", line 139, in _make_request
2022-01-14T08:00:11.068739+00:00 app[worker.1]: result = _get_req_session().request(
2022-01-14T08:00:11.068748+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
2022-01-14T08:00:11.068983+00:00 app[worker.1]: resp = self.send(prep, **send_kwargs)
2022-01-14T08:00:11.068994+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
2022-01-14T08:00:11.069233+00:00 app[worker.1]: r = adapter.send(request, **kwargs)
2022-01-14T08:00:11.069242+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/requests/adapters.py", line 529, in send
2022-01-14T08:00:11.069448+00:00 app[worker.1]: raise ReadTimeout(e, request=request)
2022-01-14T08:00:11.069478+00:00 app[worker.1]: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.telegram.org', port=443): Read timed out. (read timeout=25)
I didn't record previous exception traces (my bad), but I can remember it looking pretty much the same (with the most recent exception being ReadTimeout). Moreover, any of my scripts were never seen in this trace (and this makes me feel it's incomplete due to some Heroku log console bug). I myself don't use requests module directly. Also it definitely can't be caused by dyno sleep as I use only worker dyno (it never sleeps). Both this bot and the one not crashing are using Heroku Postgres.
Any ideas on what could be causing this exception?
EDIT:
full exception trace I accidentally got in debug mode while testing somethin else on my machine:
Traceback (most recent call last):
File "D:\Deadliner0307\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "D:\Deadliner0307\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\vva07\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 1347, in getresponse
response.begin()
File "C:\Users\vva07\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\vva07\AppData\Local\Programs\Python\Python39-32\lib\http\client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\vva07\AppData\Local\Programs\Python\Python39-32\lib\socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "C:\Users\vva07\AppData\Local\Programs\Python\Python39-32\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "C:\Users\vva07\AppData\Local\Programs\Python\Python39-32\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Deadliner0307\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "D:\Deadliner0307\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "D:\Deadliner0307\lib\site-packages\urllib3\util\retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "D:\Deadliner0307\lib\site-packages\urllib3\packages\six.py", line 770, in reraise
raise value
File "D:\Deadliner0307\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "D:\Deadliner0307\lib\site-packages\urllib3\connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "D:\Deadliner0307\lib\site-packages\urllib3\connectionpool.py", line 336, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.telegram.org', port=443): Read timed out. (read timeout=25)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\vva07\OneDrive\Документы\проекты\Deadliner0307\main.py", line 450, in <module>
bot.polling(none_stop=True, interval=1)
File "C:\Users\vva07\OneDrive\Документы\проекты\Deadliner0307\main.py", line 446, in deadliner0307
else:
File "D:\Deadliner0307\lib\site-packages\telebot\__init__.py", line 633, in polling
self.__threaded_polling(non_stop, interval, timeout, long_polling_timeout, allowed_updates)
File "D:\Deadliner0307\lib\site-packages\telebot\__init__.py", line 692, in __threaded_polling
raise e
File "D:\Deadliner0307\lib\site-packages\telebot\__init__.py", line 654, in __threaded_polling
polling_thread.raise_exceptions()
File "D:\Deadliner0307\lib\site-packages\telebot\util.py", line 100, in raise_exceptions
raise self.exception_info
File "D:\Deadliner0307\lib\site-packages\telebot\util.py", line 82, in run
task(*args, **kwargs)
File "D:\Deadliner0307\lib\site-packages\telebot\__init__.py", line 391, in __retrieve_updates
updates = self.get_updates(offset=(self.last_update_id + 1),
File "D:\Deadliner0307\lib\site-packages\telebot\__init__.py", line 371, in get_updates
json_updates = apihelper.get_updates(self.token, offset, limit, timeout, allowed_updates, long_polling_timeout)
File "D:\Deadliner0307\lib\site-packages\telebot\apihelper.py", line 312, in get_updates
return _make_request(token, method_url, params=payload)
File "D:\Deadliner0307\lib\site-packages\telebot\apihelper.py", line 139, in _make_request
result = _get_req_session().request(
File "D:\Deadliner0307\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "D:\Deadliner0307\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "D:\Deadliner0307\lib\site-packages\requests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.telegram.org', port=443): Read timed out. (read timeout=25)
I have found a way to at least keep the bot working. Be careful: this may not fit your application (in your case restarting straight up after a specific exception may corrupt data or break something else). In main file, I made this construction, which basically restarts a bot after an unhandled exception, which would otherwise just stop the bot:
def exception_handler(count: int = 0):
"""Relaunching bot unless exceptions occur more than 2 times a day
(script is reset daily on Heroku)"""
if count < 3:
if count > 0:
print("An exception occurred, relaunching . . .")
time.sleep(5)
try:
deadliner0307() # bot main function with (bot.polling starts there)
except Exception as ex:
count += 1
# Notifying myself about exception via free Airbrake addon:
notifier = pybrake.Notifier(project_id=399289,
project_key='129d3450356965175fda762b69e1babf',
environment='production')
notifier.notify(ex)
exception_handler(count)
else:
print("Too much exceptions occurred, shutting down . . .")
if __name__ == '__main__':
exception_handler()
I have a kubernetes cluster (on GKE) which runs tornado.web.Application server, that calls heavy dask.distributed calculations on that cluster.
Now I want to test my server, and send it multiple requests at once, to see if it handles those requests as it should.
When I'm sending only one request, it does everything and behaves as expected, no matter how big the request is.
But when I send more than one request, after like 5 minutes (I also tried on server side just to sleep, without making any calculations and got the same results), I get the following:
Traceback (most recent call last):
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/home/michael/anaconda3/lib/python3.7/http/client.py", line 1354, in getresponse
response.begin()
File "/home/michael/anaconda3/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/home/michael/anaconda3/lib/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/michael/anaconda3/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/michael/anaconda3/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/home/michael/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/home/michael/anaconda3/lib/python3.7/http/client.py", line 1354, in getresponse
response.begin()
File "/home/michael/anaconda3/lib/python3.7/http/client.py", line 306, in begin
version, status, reason = self._read_status()
File "/home/michael/anaconda3/lib/python3.7/http/client.py", line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/home/michael/anaconda3/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/michael/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/michael/anaconda3/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<ipython-input-4-ce4ad22684d3>", line 24, in fire
response = requests.post(url, data=payload, headers=headers, timeout=3600)
File "/home/michael/anaconda3/lib/python3.7/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/home/michael/anaconda3/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/michael/anaconda3/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/michael/anaconda3/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/michael/anaconda3/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
And none of the requests returns.
How can I get the behaviour I need, without these errors?
I need a server that can handle many requests, and to test it from my computer.
UPDATE: I managed to do a request with urllib2, but I'm still wondering what is happening here.
I would like to do a HTTPS request with Python.
This works fine with the requests module, but I don't want to use external dependencies, so I'd like to use the standard library.
httplib
When I follow this example I don't get a response. I get a timeout instead. I'm out of ideas as to what would cause this.
Code:
import requests
print requests.get('https://python.org')
from httplib import HTTPSConnection
conn = HTTPSConnection('www.python.org')
conn.request('GET', '/index.html')
print conn.getresponse()
Output:
<Response [200]>
Traceback (most recent call last):
File "test.py", line 6, in <module>
conn.request('GET', '/index.html')
File "C:\Python27\lib\httplib.py", line 1069, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1109, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1282, in connect
HTTPConnection.connect(self)
File "C:\Python27\lib\httplib.py", line 831, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 575, in create_connection
raise err
socket.error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
urllib
This fails for a different (but possibly related) reason. Code:
import urllib
print urllib.urlopen("https://python.org")
Output:
Traceback (most recent call last):
File "test.py", line 10, in <module>
print urllib.urlopen("https://python.org")
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 215, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 445, in open_https
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1290, in connect
server_hostname=server_hostname)
File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
_context=self)
File "C:\Python27\lib\ssl.py", line 599, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:727)
What is requests doing that makes it succeed where both of these libraries fail?
requests.get without timeout parameter mean no timeout at all.
httplib.HTTPSConnection accept parameter timeout in Python 2.6 and newer according to httplib docs. If your problem was caused by timeout, setting high enough timeout should help. Please try replacing:
conn = HTTPSConnection('www.python.org')
with:
conn = HTTPSConnection('www.python.org', timeout=300)
which will give 300 seconds (5 minutes) for processing.
I'm trying to upload a file to Google via the Adwords API and I'm using the script provided by Google here:https://github.com/googleads/googleads-python-lib
I'm getting the following error after a few minutes. I've tried this on a smaller file and it works. Not sure why it is breaking here. Any ideas?
File "./g_leads.sh", line 71, in <module>
main(adwords_client, EMAILS)
File "./g_leads.sh", line 43, in main
response = user_list_service.mutateMembers([mutate_members_operation])
File "/opt/stateadm/.local/lib/python2.7/site-packages/googleads/common.py", line 296, in MakeSoapRequest
for arg in args])
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/client.py", line 581, in invoke
result = self.send(soapenv)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/client.py", line 613, in send
reply = self.options.transport.send(request)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/transport/https.py", line 66, in send
return HttpTransport.send(self, request)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/transport/http.py", line 82, in send
fp = self.u2open(u2request)
File "/opt/stateadm/.local/lib/python2.7/site-packages/suds/transport/http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "/opt/python/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/opt/python/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/opt/python/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/opt/python/lib/python2.7/urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/opt/python/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 32] Broken pipe>
Errno 32 Broken Pipe means the remote server closed the socket. I would guess the transfer is taking too long because the API documentation shows that they would return an error if the file itself is too big.
Conversely, this may be a intermittent issue at Google per this old thread.