TL;DR
My question is simple - where is the code responsible to raise ConnectionResetError on cpython3 following a call to self._sslobj.read(len, buffer) on ssl.py?
Background
I'm getting sometimes ConnectionResetError when trying to connect to S3 with ssl. this error occurs rarely so its tricky to reproduce it.
# trimmed stacktrace
File "/MYPROJECT/MY_FUNC.py", line 123, in <genexpr>
rows = (row for row in reader)
File "/XXX/lib/python3.6/csv.py", line 112, in _next_
row = next(self.reader)
File "/XXX/lib/python3.6/tarfile.py", line 706, in readinto
buf = self.read(len(b))
File "/XXX/lib/python3.6/tarfile.py", line 695, in read
b = self.fileobj.read(length)
File "/XXX/lib/python3.6/gzip.py", line 276, in read
return self._buffer.read(size)
File "/XXX/lib/python3.6/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/XXX/lib/python3.6/gzip.py", line 469, in read
buf = self._fp.read(io.DEFAULT_BUFFER_SIZE)
File "/XXX/lib/python3.6/gzip.py", line 91, in read
self.file.read(size-self._length+read)
File "/XXX/lib/python3.6/site-packages/s3fs/core.py", line 1311, in read
self._fetch(self.loc, self.loc + length)
File "/XXX/lib/python3.6/site-packages/s3fs/core.py", line 1292, in _fetch
req_kw=self.s3.req_kw)
File "/XXX/lib/python3.6/site-packages/s3fs/core.py", line 1496, in _fetch_range
return resp['Body'].read()
File "/XXX/lib/python3.6/site-packages/botocore/response.py", line 74, in read
chunk = self._raw_stream.read(amt)
File "/XXX/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 239, in read
data = self._fp.read()
File "/XXX/lib/python3.6/http/client.py", line 462, in read
s = self._safe_read(self.length)
File "/XXX/lib/python3.6/http/client.py", line 612, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "/XXX/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/XXX/lib/python3.6/ssl.py", line 1009, in recv_into
return self.read(nbytes, buffer)
File "/XXX/lib/python3.6/ssl.py", line 871, in read
return self._sslobj.read(len, buffer)
File "/XXX/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
What i've tried
looking at ssl.py:631 gives me no further clues - we have to go deeper!:
def read(self, len=1024, buffer=None):
"""Read up to 'len' bytes from the SSL object and return them.
If 'buffer' is provided, read into this buffer and return the number of
bytes read.
"""
if buffer is not None:
v = self._sslobj.read(len, buffer) # <--- exception here
else:
v = self._sslobj.read(len)
return v
i've tried searching it on CPython repo but AFAICS nothing seems to raise it, i suspect its hidden in SSL implementation or on some mapping between OSError to ConnectionError subclasses.
my final goal is to write py2 & py3 compatible code for handling this exceptions (ConnectionError is new on py3) by comparing the module's py2 & py3 versions that raises this error.
Update - py2 & py3 catch for ConnectionError subclasses
my question origins was to find a way to catch ConnectionError and its subclasses on python2 & python3, so here it is:
import errno
# ref: https://docs.python.org/3/library/exceptions.html#ConnectionError
_CONNECTION_ERRORS = frozenset({
errno.ECONNRESET, # ConnectionResetError
errno.EPIPE, errno.ESHUTDOWN, # BrokenPipeError
errno.ECONNABORTED, # ConnectionAbortedError
errno.ECONNREFUSED, # ConnectionRefusedError
})
try:
...
except OSError as e:
if e.errno not in _CONNECTION_ERRORS:
raise
print('got ConnectionError - %e' % e)
ConnectionResetError is raised when errno is ECONNRESET. errno is how libc indicates whether or not an error occurred in a system call.
You could search ConnectionResetError in Objects/exceptions.c to find out how this exception type get initialized and added to errnomap dict.
In the case of self._sslobj.read raised ConnectionResetError, _sslobj.read is implemented with _ssl__SSLSocket_read_impl, the actual ssl read is done with openssl's SSL_read:
count = SSL_read(self->ssl, mem, len);
_PySSL_UPDATE_ERRNO_IF(count <= 0, self, count);
as the error occurred, _PySSL_UPDATE_ERRNO_IF will set (sock)->ssl_errno = SSL_ERROR_SYSCALL and (sock)->c_errno = ECONNRESET.
later, in PySSL_SetError:
err = obj->ssl_errno;
switch (err) {
...
case SSL_ERROR_SYSCALL:
if (obj->c_errno) {
errno = obj->c_errno;
return PyErr_SetFromErrno(PyExc_OSError);
}
PyErr_SetFromErrno(PyExc_OSError) equals with:
OSError(errno.ECONNRESET, 'Connection reset by peer', ...)
when OSError constructs with an errno, it will lookup a more specified subclass, by lookup errno value in the aforementioned errnomap dict:
newtype = PyDict_GetItem(errnomap, myerrno);
if (newtype) {
assert(PyType_Check(newtype));
type = (PyTypeObject *) newtype;
}
it actually returns and raises out a ConnectionResetError exception.
Related
I'm having a problem handling socket timeouts with python35 ftplib. When a socket timeout error occurs, for some reason I am unable to catch the exception and the script raises the error anyway and exits. Here is the relevant code block:
try:
ftp = FTP(self.config.base_url, timeout=400)
ftp.login()
ftp.cwd(self.config.root_path)
ftp.retrbinary("RETR {0}".format(os.path.join(self.root_path, file_path)), fp.write, 1024)
ftp.quit()
except socket.timeout:
self.download_file(file_path)
For some reason this script will still error out with a socket timeout exception, how is that possible? A general catch-all doesn't work either. Here is the stack trace of the error:
File "ftp.py", line 82, in __init__
self.ftp = FTP(self.config.base_url, timeout=400)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 118, in __init__
self.connect(host)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 156, in connect
self.welcome = self.getresp()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 235, in getresp
resp = self.getmultiline()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 221, in getmultiline
line = self.getline()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ftplib.py", line 203, in getline
line = self.file.readline(self.maxline + 1)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 576, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
As you can see, this is the socket.timeout error being thrown, so how is that not caught? I was unable to find any useful info on how to resolve this issue after many hours of internet research, any insight into this issue would be much appreciated.
For reference, here is the relevant code block of socket.py:
def readinto(self, b):
"""Read up to len(b) bytes into the writable buffer *b* and return
the number of bytes read. If the socket is non-blocking and no bytes
are available, None is returned.
If *b* is non-empty, a 0 return value indicates that the connection
was shutdown at the other end.
"""
self._checkClosed()
self._checkReadable()
if self._timeout_occurred:
raise OSError("cannot read from timed out object")
while True:
try:
return self._sock.recv_into(b)
except timeout:
self._timeout_occurred = True
raise
except error as e:
if e.args[0] in _blocking_errnos:
return None
raise
UPDATE:
It seems the issue is that ftplib behaves unexpectedly if the timeout passed to the FTP constructor is greater than the default for a socket timeout. At the time of this edit there is an open python issue to address this behavior: http://bugs.python.org/issue30956
try to add
ftp.set_pasv(False)
disabling the passive mode it should work
So in my flask app right now I am using Celery to deploy servers on remote machines. Right now, I have an enum, status, which indicates the lifecycle of my deployment process:
#celery.task(bind=True)
def deploy_server(self, server_id):
server = Server.query.get(server_id)
if not server.can_launch():
return
try:
server.status = RemoteStatus.LAUNCHING
db.session.commit()
verify_DNS(server)
host = server.server.ssh_user + '#' + server.server.ip
execute(fabric_deploy_server, self, server, hosts=host)
server.status = RemoteStatus.LAUNCHED
db.session.commit()
except Exception as e:
server.status = RemoteStatus.ERROR
db.session.commit()
traceback.print_exc()
raise e
As you can see, when a server is being deployed, its status is changed to "Launching". If there is an exception, it will be changed to ERROR.
I found one exception which completely bypasses this bloc of code: when I overloaded my celery server with too many requests, I get this exception:
[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',):
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task
uuid, retval, task_request, publish_result,
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure
**retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set
pipe.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute
return execute(conn, stack, raise_on_error)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction
connection.send_packed_command(all_cmds)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect
self.on_connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response
raise error
ConnectionError: max number of clients reached
exc, exc_info.traceback)))
My biggest problem with this is that this error is raised somewhere outside of my Try/Catch bloc. Hence, when this exception occurs, all my servers remain in the "Launching" mode rather than "Error".
How can I catch this exception and modify Server.status?
In redis 2.4 there is a hard coded limit of max number of connections which is 10,000. In redis 2.6+ you can specify the max number of clients in redis.conf. also this is not a problem of celery your broker redis refused to accept connections that's the problem.
Set the max number of clients that can be handled by redis simultaneously using redis CLI. Check out redis clients
I keep getting the following error but i have no idea how to try..except it or handle it. Any ideas?
The error happens when i load test the server or if I send a request on port 443 that is not SSL, such as http://server:443/call
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
result = self._run(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/gevent/server.py", line 102, in wrap_socket_and_handle
ssl_socket = self.wrap_socket(client_socket, **self.ssl_args)
File "/usr/local/lib/python2.7/dist-packages/gevent/ssl.py", line 383, in wrap_socket
ciphers=ciphers)
File "/usr/local/lib/python2.7/dist-packages/gevent/ssl.py", line 94, in __init__
self.do_handshake()
File "/usr/local/lib/python2.7/dist-packages/gevent/ssl.py", line 305, in do_handshake
return self._sslobj.do_handshake()
SSLError: [Errno 8] _ssl.c:510: EOF occurred in violation of protocol
<Greenlet at 0x7f74b804eb90: <bound method WSGIServer.wrap_socket_and_handle of <WSGIServer at 0x7f74b9d16810 fileno=7 address=0.0.0.0:443>>(<socket at 0x7f74b2442590 fileno=198 sock=46.101.4, ('54.198.34.85', 26771))> failed with SSLError
The code on a basic level is as follows
from gevent.pywsgi import WSGIServer
try:
class HttpsHandler:
def application(self, env, start_response):
start_response("200 OK", "Content-Length: 0")
return []
WSGIServer(
('', 443),
application=HttpsHandler().application,
keyfile=cfg.ssl_keyfile,
certfile=cfg.ssl_crtfile,
ca_certs=cfg.ssl_cacerts,
log=open('logs/access.log', 'a')
).serve_forever()
except Exception, e:
print e
except KeyboardInterrupt, e:
pass
So my twisted mail receiver is working nicely. Right up until we try to handle a case where the config is fubarred, and a mismatched cert/key is passed to the certificate options object for the factory.
I have a module, custom_esmtp.py, which includes an overload of ext_STARTLS(self,rest) which I have modified as follows, to include a try/except:
elif self.ctx and self.canStartTLS:
try:
self.sendCode(220, 'Begin TLS negotiation now')
self.transport.startTLS(self.ctx)
self.startedTLS = True
except:
log.err()
self.sendCode(550, "Internal server error")
return
When I run the code, having passed a cert and key that do not match, I get the following call stack:
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
self.lineReceived(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 568, in lineReceived
return getattr(self, 'state_' + self.mode)(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 582, in state_COMMAND
method('')
--- <exception caught here> ---
File "custom_esmtp.py", line 286, in ext_STARTTLS
self.transport.startTLS(self.ctx)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 179, in startTLS
startTLS(self, ctx, normal, FileDescriptor)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 139, in startTLS
tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 769, in __init__
contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 648, in __init__
oldStyleContextFactory.getContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1429, in getContext
self._context = self._makeContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1439, in _makeContext
ctx.use_privatekey(self.privateKey)
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
Line 286 of custom_esmtp.py is the self.transport.startTLS(self.ctx). I've looked through all the twisted modules listed in the stack, at the quoted lines, and there are no other try/except blocks.... So my understanding is that the error should be passed back up the stack, unhandled, until it reaches my handler in custom_esmtp.py? So why is it not getting handled - especially since the only except I have is a "catch all"?
Thanks in advance!
If you want this error to be caught, you can do:
from OpenSSL import SSL
# ...
try:
# ...
except SSL.Error:
# ...
Perhaps the syntax changes a bit. I can't check because I don't use this precise package, but the idea is that you have to declare the import path of the exceptions you want to catch.
Encountered an error while building a web scraper to compile data and output into XLS format; when testing again a list of domains in which I wish to scrape from, the program faulters when it recieves a socket error. Hoping to find an 'if' statement that would null parsing a broken website and continue through my while-loop. Any ideas?
workingList = xlrd.open_workbook(listSelection)
workingSheet = workingList.sheet_by_index(0)
destinationList = xlwt.Workbook()
destinationSheet = destinationList.add_sheet('Gathered')
startX = 1
startY = 0
while startX != 21:
workingCell = workingSheet.cell(startX,startY).value
print ''
print ''
print ''
print workingCell
#Setup
preSite = 'http://www.'+workingCell
theSite = urlopen(preSite).read()
currentSite = BeautifulSoup(theSite)
destinationSheet.write(startX,0,workingCell)
And here's the error:
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
homeMenu()
File "C:\Python27\farming.py", line 31, in homeMenu
openList()
File "C:\Python27\farming.py", line 79, in openList
openList()
File "C:\Python27\farming.py", line 83, in openList
openList()
File "C:\Python27\farming.py", line 86, in openList
homeMenu()
File "C:\Python27\farming.py", line 34, in homeMenu
startScrape()
File "C:\Python27\farming.py", line 112, in startScrape
theSite = urlopen(preSite).read()
File "C:\Python27\lib\urllib.py", line 84, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 205, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 342, in open_http
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 951, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 811, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 773, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 754, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 11004] getaddrinfo failed
Ummm that looks like the error I get when my internet connection is down. HTTP 404 errors are what you get when you do have a connection but the URL that you specify can't be found.
There's no if statement to handle exceptions; you need to "catch" them using the try/except construct.
Update: Here's a demonstration:
import urllib
def getconn(url):
try:
conn = urllib.urlopen(url)
return conn, None
except IOError as e:
return None, e
urls = """
qwerty
http://www.foo.bar.net
http://www.google.com
http://www.google.com/nonesuch
"""
for url in urls.split():
print
print url
conn, exc = getconn(url)
if conn:
print "connected; HTTP response is", conn.getcode()
else:
print "failed"
print exc.__class__.__name__
print str(exc)
print exc.args
Output:
qwerty
failed
IOError
[Errno 2] The system cannot find the file specified: 'qwerty'
(2, 'The system cannot find the file specified')
http://www.foo.bar.net
failed
IOError
[Errno socket error] [Errno 11004] getaddrinfo failed
('socket error', gaierror(11004, 'getaddrinfo failed'))
http://www.google.com
connected; HTTP response is 200
http://www.google.com/nonesuch
connected; HTTP response is 404
Note that so far we have just opened the connection. Now what you need to do is check the HTTP response code and decide whether there is anything worth retrieving using conn.read()