We Keep Coding

Custom exception handling when sending message to remote syslog server in Python

At the moment i am writing a syslog client that will send messages to a remote syslog server. So far this is working pretty ok but i am running into the following problem.
When the syslog server goes down for some reason i need to catch this so the program will stop sending syslog messages and we can investigate the problem.
Unfortunately, the program continues running and doesn't see that the TCP socket is closed and raise an exception.
I only receive a traceback in my terminal:
--- Logging error --- Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\logging\handlers.py", line 941, in emit
self.socket.sendall(msg) ConnectionAbortedError: [WinError 10053] Call stack:
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\ptvsd_launcher.py", line 45, in <module>
main(ptvsdArgs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\__main__.py", line 265, in main
wait=args.wait)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\__main__.py", line 258, in handle_args
debug_main(addr, name, kind, *extra, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 45, in debug_main
run_file(address, name, *extra, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 79, in run_file
run(argv, addr, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 140, in _run
_pydevd.main()
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1925, in main
debugger.connect(host, port)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1283, in run
return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1290, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydev_imps\_pydev_execfile.py", line 25, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "c:\Users\Administrator\OneDrive\Documents\Python Scripts\testlogger.py", line 71, in <module>
my_logger.info(i) Message: 'test4' Arguments: ()
Relevant code:
my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.ERROR)
my_logger.setLevel(logging.INFO)
my_logger.setLevel(logging.DEBUG)
try:
handler = logging.handlers.SysLogHandler(('IP ADDRESS HOST', 514), socktype=socket.SOCK_STREAM)
my_logger.addHandler(handler)
except Exception as e:
print (e)
list1 = ['test','test2','test3','test4','test5','test6','test7','test8']
for i in list1:
try:
my_logger.info(i) #here i expected that an exception would be raised when the TCP socket is not alive anymore
except Exception as e:
print (e)
How can i make sure that the program stops and i can do the appropriate exception handling?
Thanks!

The default behavior of the SysLogHandler class (and all the ones who are using TCP) is to retry a connection, this is explained in the docs of the createSocket() method:
Tries to create a socket; on failure, uses an exponential back-off
algorithm. On initial failure, the handler will drop the message it
was trying to send. When subsequent messages are handled by the same
instance, it will not try connecting until some time has passed. The
default parameters are such that the initial delay is one second, and
if after that delay the connection still can’t be made, the handler
will double the delay each time up to a maximum of 30 seconds.
This behaviour is controlled by the following handler attributes:
retryStart (initial delay, defaulting to 1.0 seconds).
retryFactor (multiplier, defaulting to 2.0).
retryMax (maximum delay, defaulting to 30.0 seconds).
As it doesn't seem to have an option for the behavior of "not retrying" which you seem to want, so if you really want that you can create your own handler by subclassing it and overriding the createSocket() method with something like:
class MySysLogHandler(logging.handlers.SysLogHandler):
def createSocket(self):
try:
self.sock = self.makeSocket()
except OSError:
# do your own error handling here ...
You can dig a bit more by looking at the source code of createSocket() in CPython Github repo (beware, this is from master branch and might not be the exact version of Python you're using)

I switched to another solution and stopped using the SyslogHandler class.
I now use the following class were i wrote my own syslog sender through a socket.
class Syslog:
def __init__(self,host="localhost",port=514,facility=Facility.DAEMON):
self.host = host
self.port = port
self.facility = facility
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
try:
self.socket.connect((self.host, self.port))
except Exception as e:
print("failed setting up connection")
def send(self, message,level):
data = "<%d>%s" % (level + self.facility*8, message + "\n")
try:
self.socket.sendall(data.encode('utf-8'))
except Exception as e:
print("send failed")
#if __name__ == '__main__':
syslog1 = Syslog(host='HOSTIPADDRESS-NAME')
syslog1.connect()
messages = ["test1","test2","test3","test4","test5","test6","test7"]
for message in messages:
syslog1.send(message,Level.WARNING)
This is working quite well and runs into an exception when the syslog server goes down unexpectedly. The only problem now i discovered while debugging is the following:
When i shut down the syslog server it throws not immediatly an exception when i try to send a message.
Please see example below:
1.) the syslog server is started, i send the first message "test1" from the for loop, successfull.
2.) i shutdown the syslog server, now i send the second message "test2" from the for loop. Nothing happens, no exception!
3.) i send the third message "test3", now an exception is thrown.
How is this possible?

from where SSL ConnectionResetError comes from?

TL;DR
My question is simple - where is the code responsible to raise ConnectionResetError on cpython3 following a call to self._sslobj.read(len, buffer) on ssl.py?
Background
I'm getting sometimes ConnectionResetError when trying to connect to S3 with ssl. this error occurs rarely so its tricky to reproduce it.
# trimmed stacktrace
File "/MYPROJECT/MY_FUNC.py", line 123, in <genexpr>
rows = (row for row in reader)
File "/XXX/lib/python3.6/csv.py", line 112, in _next_
row = next(self.reader)
File "/XXX/lib/python3.6/tarfile.py", line 706, in readinto
buf = self.read(len(b))
File "/XXX/lib/python3.6/tarfile.py", line 695, in read
b = self.fileobj.read(length)
File "/XXX/lib/python3.6/gzip.py", line 276, in read
return self._buffer.read(size)
File "/XXX/lib/python3.6/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/XXX/lib/python3.6/gzip.py", line 469, in read
buf = self._fp.read(io.DEFAULT_BUFFER_SIZE)
File "/XXX/lib/python3.6/gzip.py", line 91, in read
self.file.read(size-self._length+read)
File "/XXX/lib/python3.6/site-packages/s3fs/core.py", line 1311, in read
self._fetch(self.loc, self.loc + length)
File "/XXX/lib/python3.6/site-packages/s3fs/core.py", line 1292, in _fetch
req_kw=self.s3.req_kw)
File "/XXX/lib/python3.6/site-packages/s3fs/core.py", line 1496, in _fetch_range
return resp['Body'].read()
File "/XXX/lib/python3.6/site-packages/botocore/response.py", line 74, in read
chunk = self._raw_stream.read(amt)
File "/XXX/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/response.py", line 239, in read
data = self._fp.read()
File "/XXX/lib/python3.6/http/client.py", line 462, in read
s = self._safe_read(self.length)
File "/XXX/lib/python3.6/http/client.py", line 612, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "/XXX/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/XXX/lib/python3.6/ssl.py", line 1009, in recv_into
return self.read(nbytes, buffer)
File "/XXX/lib/python3.6/ssl.py", line 871, in read
return self._sslobj.read(len, buffer)
File "/XXX/lib/python3.6/ssl.py", line 631, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
What i've tried
looking at ssl.py:631 gives me no further clues - we have to go deeper!:
def read(self, len=1024, buffer=None):
"""Read up to 'len' bytes from the SSL object and return them.
If 'buffer' is provided, read into this buffer and return the number of
bytes read.
"""
if buffer is not None:
v = self._sslobj.read(len, buffer) # <--- exception here
else:
v = self._sslobj.read(len)
return v
i've tried searching it on CPython repo but AFAICS nothing seems to raise it, i suspect its hidden in SSL implementation or on some mapping between OSError to ConnectionError subclasses.
my final goal is to write py2 & py3 compatible code for handling this exceptions (ConnectionError is new on py3) by comparing the module's py2 & py3 versions that raises this error.
Update - py2 & py3 catch for ConnectionError subclasses
my question origins was to find a way to catch ConnectionError and its subclasses on python2 & python3, so here it is:
import errno
# ref: https://docs.python.org/3/library/exceptions.html#ConnectionError
_CONNECTION_ERRORS = frozenset({
errno.ECONNRESET, # ConnectionResetError
errno.EPIPE, errno.ESHUTDOWN, # BrokenPipeError
errno.ECONNABORTED, # ConnectionAbortedError
errno.ECONNREFUSED, # ConnectionRefusedError
})
try:
...
except OSError as e:
if e.errno not in _CONNECTION_ERRORS:
raise
print('got ConnectionError - %e' % e)

ConnectionResetError is raised when errno is ECONNRESET. errno is how libc indicates whether or not an error occurred in a system call.
You could search ConnectionResetError in Objects/exceptions.c to find out how this exception type get initialized and added to errnomap dict.
In the case of self._sslobj.read raised ConnectionResetError, _sslobj.read is implemented with _ssl__SSLSocket_read_impl, the actual ssl read is done with openssl's SSL_read:
count = SSL_read(self->ssl, mem, len);
_PySSL_UPDATE_ERRNO_IF(count <= 0, self, count);
as the error occurred, _PySSL_UPDATE_ERRNO_IF will set (sock)->ssl_errno = SSL_ERROR_SYSCALL and (sock)->c_errno = ECONNRESET.
later, in PySSL_SetError:
err = obj->ssl_errno;
switch (err) {
...
case SSL_ERROR_SYSCALL:
if (obj->c_errno) {
errno = obj->c_errno;
return PyErr_SetFromErrno(PyExc_OSError);
}
PyErr_SetFromErrno(PyExc_OSError) equals with:
OSError(errno.ECONNRESET, 'Connection reset by peer', ...)
when OSError constructs with an errno, it will lookup a more specified subclass, by lookup errno value in the aforementioned errnomap dict:
newtype = PyDict_GetItem(errnomap, myerrno);
if (newtype) {
assert(PyType_Check(newtype));
type = (PyTypeObject *) newtype;
}
it actually returns and raises out a ConnectionResetError exception.

Tornado how to return error exception?

I want to run a method I know this method doesn't work and I want to get the error returned by the method.
This is my code :
def is_connect(s):
print("ok connection")
print(s)
ioloop.stop()
try:
current_job_ready = 0
print("ok1")
beanstalk = beanstalkt.Client(host='host', port=port)
print("ok1")
beanstalk.connect(callback=is_connect)
ioloop = tornado.ioloop.IOLoop.instance()
ioloop.start()
print("ok2")
except IOError as e:
print(e)
And this is the error I have when I run my program with wring port :
WARNING:tornado.general:Connect error on fd 7: ECONNREFUSED
ERROR:tornado.application:Exception in callback <functools.partial object at 0x7f5a0eac6f18>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 604, in _run_callback
ret = callback()
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 619, in <lambda>
self.add_future(ret, lambda f: f.result())
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 270, in wrapper
result = func(*args, **kwargs)
TypeError: connect() takes exactly 1 argument (2 given)
I want to have e when I enter a false port or host.
How can I do this?
I tired to add raise IOError("connection error") after beanstalk = beanstalkt.Client(host='host', port=port) But this force the error, and I just want to have error when it exist.

Here's where reading the code helps. In beanstalkt 0.6's connect, it creates an IOStream to connect to the server:
https://github.com/nephics/beanstalkt/blob/v0.6.0/beanstalkt/beanstalkt.py#L108
It registers your callback to be executed on success, but if the connection fails it'll just call Client._reconnect once per second forever. I think you should open a feature request in their GitHub project asking for an error-notification system for connect. With the current beanstalkt implementation, you just have to decide how long you're willing to wait for success:
import sys
from datetime import timedelta
from tornado.ioloop import IOLoop
def is_connect(s):
print("ok connection")
print(s)
loop.remove_timeout(timeout)
# Do something with Beanstalkd....
def connection_failed():
print(sys.stderr, "Connection failed!")
# Could call IOLoop.stop() or just quit.
sys.exit(1)
loop = IOLoop.current()
timeout = loop.add_timeout(timedelta(seconds=1), connection_failed)
beanstalk.connect(callback=is_connect)
loop.start()

Try/except not working with twisted starttls given cert/key mismatch

So my twisted mail receiver is working nicely. Right up until we try to handle a case where the config is fubarred, and a mismatched cert/key is passed to the certificate options object for the factory.
I have a module, custom_esmtp.py, which includes an overload of ext_STARTLS(self,rest) which I have modified as follows, to include a try/except:
elif self.ctx and self.canStartTLS:
try:
self.sendCode(220, 'Begin TLS negotiation now')
self.transport.startTLS(self.ctx)
self.startedTLS = True
except:
log.err()
self.sendCode(550, "Internal server error")
return
When I run the code, having passed a cert and key that do not match, I get the following call stack:
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
self.lineReceived(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 568, in lineReceived
return getattr(self, 'state_' + self.mode)(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 582, in state_COMMAND
method('')
--- <exception caught here> ---
File "custom_esmtp.py", line 286, in ext_STARTTLS
self.transport.startTLS(self.ctx)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 179, in startTLS
startTLS(self, ctx, normal, FileDescriptor)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 139, in startTLS
tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 769, in __init__
contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 648, in __init__
oldStyleContextFactory.getContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1429, in getContext
self._context = self._makeContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1439, in _makeContext
ctx.use_privatekey(self.privateKey)
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
Line 286 of custom_esmtp.py is the self.transport.startTLS(self.ctx). I've looked through all the twisted modules listed in the stack, at the quoted lines, and there are no other try/except blocks.... So my understanding is that the error should be passed back up the stack, unhandled, until it reaches my handler in custom_esmtp.py? So why is it not getting handled - especially since the only except I have is a "catch all"?
Thanks in advance!

If you want this error to be caught, you can do:
from OpenSSL import SSL
# ...
try:
# ...
except SSL.Error:
# ...
Perhaps the syntax changes a bit. I can't check because I don't use this precise package, but the idea is that you have to declare the import path of the exceptions you want to catch.

Python socket error resilience / workaround

I have a script running that is testing a series of urls for availability.
This is one of the functions.
def checkUrl(url): # Only downloads headers, returns status code.
p = urlparse(url)
conn = httplib.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status
Occasionally, the VPS will lose connectivity, the entire script crashes when that occurs.
File "/usr/lib/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib/python2.6/socket.py", line 561, in create_connection
raise error, msg
socket.error: [Errno 101] Network is unreachable
I'm not at all familiar with handling errors like this in python.
What is the appropriate way to keep the script from crashing when network connectivity is temporarily lost?
Edit:
I ended up with this - feedback?
def checkUrl(url): # Only downloads headers, returns status code.
try:
p = urlparse(url)
conn = httplib.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status
except IOError, e:
if e.errno == 101:
print "Network Error"
time.sleep(1)
checkUrl(url)
else:
raise
I'm not sure I fully understand what raise does though..

If you just want to handle this Network is unreachable 101, and let other exceptions throw an error, you can do following for example.
from errno import ENETUNREACH
try:
# tricky code goes here
except IOError as e:
# an IOError exception occurred (socket.error is a subclass)
if e.errno == ENETUNREACH:
# now we had the error code 101, network unreachable
do_some_recovery
else:
# other exceptions we reraise again
raise

Problem with your solution as it stands is you're going to run out of stack space if there are too many errors on a single URL (> 1000 by default) due to the recursion. Also, the extra stack frames could make tracebacks hard to read (500 calls to checkURL). I'd rewrite it to be iterative, like so:
def checkUrl(url): # Only downloads headers, returns status code.
while True:
try:
p = urlparse(url)
conn = httplib.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status
except IOError as e:
if e.errno == 101:
print "Network Error"
time.sleep(1)
except:
raise
Also, you want the last clause in your try to be a bare except not an else. Your else only gets executed if control falls through the try suite, which can never happen, since the last statement of the try suite is return.
This is very easy to change to allow a limited number of retries. Just change the while True: line to for _ in xrange(5) or however many retries you wish to accept. The function will then return None if it can't connect to the site after 5 attempts. You can have it return something else or raise an exception by adding return or raise SomeException at the very end of the function (indented the same as the for or while line).

put try...except: around your code to catch exceptions.
http://docs.python.org/tutorial/errors.html