We Keep Coding

Custom exception handling when sending message to remote syslog server in Python

At the moment i am writing a syslog client that will send messages to a remote syslog server. So far this is working pretty ok but i am running into the following problem.
When the syslog server goes down for some reason i need to catch this so the program will stop sending syslog messages and we can investigate the problem.
Unfortunately, the program continues running and doesn't see that the TCP socket is closed and raise an exception.
I only receive a traceback in my terminal:
--- Logging error --- Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\logging\handlers.py", line 941, in emit
self.socket.sendall(msg) ConnectionAbortedError: [WinError 10053] Call stack:
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\ptvsd_launcher.py", line 45, in <module>
main(ptvsdArgs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\__main__.py", line 265, in main
wait=args.wait)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\__main__.py", line 258, in handle_args
debug_main(addr, name, kind, *extra, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 45, in debug_main
run_file(address, name, *extra, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 79, in run_file
run(argv, addr, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 140, in _run
_pydevd.main()
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1925, in main
debugger.connect(host, port)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1283, in run
return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1290, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydev_imps\_pydev_execfile.py", line 25, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "c:\Users\Administrator\OneDrive\Documents\Python Scripts\testlogger.py", line 71, in <module>
my_logger.info(i) Message: 'test4' Arguments: ()
Relevant code:
my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.ERROR)
my_logger.setLevel(logging.INFO)
my_logger.setLevel(logging.DEBUG)
try:
handler = logging.handlers.SysLogHandler(('IP ADDRESS HOST', 514), socktype=socket.SOCK_STREAM)
my_logger.addHandler(handler)
except Exception as e:
print (e)
list1 = ['test','test2','test3','test4','test5','test6','test7','test8']
for i in list1:
try:
my_logger.info(i) #here i expected that an exception would be raised when the TCP socket is not alive anymore
except Exception as e:
print (e)
How can i make sure that the program stops and i can do the appropriate exception handling?
Thanks!

The default behavior of the SysLogHandler class (and all the ones who are using TCP) is to retry a connection, this is explained in the docs of the createSocket() method:
Tries to create a socket; on failure, uses an exponential back-off
algorithm. On initial failure, the handler will drop the message it
was trying to send. When subsequent messages are handled by the same
instance, it will not try connecting until some time has passed. The
default parameters are such that the initial delay is one second, and
if after that delay the connection still can’t be made, the handler
will double the delay each time up to a maximum of 30 seconds.
This behaviour is controlled by the following handler attributes:
retryStart (initial delay, defaulting to 1.0 seconds).
retryFactor (multiplier, defaulting to 2.0).
retryMax (maximum delay, defaulting to 30.0 seconds).
As it doesn't seem to have an option for the behavior of "not retrying" which you seem to want, so if you really want that you can create your own handler by subclassing it and overriding the createSocket() method with something like:
class MySysLogHandler(logging.handlers.SysLogHandler):
def createSocket(self):
try:
self.sock = self.makeSocket()
except OSError:
# do your own error handling here ...
You can dig a bit more by looking at the source code of createSocket() in CPython Github repo (beware, this is from master branch and might not be the exact version of Python you're using)

I switched to another solution and stopped using the SyslogHandler class.
I now use the following class were i wrote my own syslog sender through a socket.
class Syslog:
def __init__(self,host="localhost",port=514,facility=Facility.DAEMON):
self.host = host
self.port = port
self.facility = facility
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
try:
self.socket.connect((self.host, self.port))
except Exception as e:
print("failed setting up connection")
def send(self, message,level):
data = "<%d>%s" % (level + self.facility*8, message + "\n")
try:
self.socket.sendall(data.encode('utf-8'))
except Exception as e:
print("send failed")
#if __name__ == '__main__':
syslog1 = Syslog(host='HOSTIPADDRESS-NAME')
syslog1.connect()
messages = ["test1","test2","test3","test4","test5","test6","test7"]
for message in messages:
syslog1.send(message,Level.WARNING)
This is working quite well and runs into an exception when the syslog server goes down unexpectedly. The only problem now i discovered while debugging is the following:
When i shut down the syslog server it throws not immediatly an exception when i try to send a message.
Please see example below:
1.) the syslog server is started, i send the first message "test1" from the for loop, successfull.
2.) i shutdown the syslog server, now i send the second message "test2" from the for loop. Nothing happens, no exception!
3.) i send the third message "test3", now an exception is thrown.
How is this possible?

Google PubSub python client returning StatusCode.UNAVAILABLE

I am trying to establish a long running Pull subscription to a Google Cloud PubSub topic.
I am using a code very similar to the example given in the documentation here, i.e.:
def receive_messages(project, subscription_name):
"""Receives messages from a pull subscription."""
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
def callback(message):
print('Received message: {}'.format(message))
message.ack()
subscriber.subscribe(subscription_path, callback=callback)
# The subscriber is non-blocking, so we must keep the main thread from
# exiting to allow it to process messages in the background.
print('Listening for messages on {}'.format(subscription_path))
while True:
time.sleep(60)
The problem is that I'm receiving the following traceback sometimes:
Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/path/to/google/cloud/pubsub_v1/subscriber/_consumer.py", line 248, in _blocking_consume
self._policy.on_exception(exc)
File "/path/to/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 135, in on_exception
raise exception
File "/path/to/google/cloud/pubsub_v1/subscriber/_consumer.py", line 234, in _blocking_consume
for response in response_generator:
File "/path/to/grpc/_channel.py", line 348, in __next__
return self._next()
File "/path/to/grpc/_channel.py", line 342, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>
I saw that this was referenced in another question but here I am asking to how to handle it properly in Python. I have tried to wrap the request in an exception but it seems to run in the background and I am not able to retry in case of that error.

A somewhat hacky approach that is working for me is a custom policy_class. The default one has an on_exception function that ignores DEADLINE_EXCEEDED. You can make a class that inherits the default and also ignores UNAVAILABLE. Mine looks like this:
from google.cloud import pubsub
from google.cloud.pubsub_v1.subscriber.policy import thread
import grpc
class AvailablePolicy(thread.Policy):
def on_exception(self, exception):
"""The parent ignores DEADLINE_EXCEEDED. Let's also ignore UNAVAILABLE.
I'm not sure what triggers that error, but if you ignore it, your
subscriber seems to work just fine. It's probably an intermittent
thing and it reconnects later if you just give it a chance.
"""
# If this is UNAVAILABLE, then we want to retry.
# That entails just returning None.
unavailable = grpc.StatusCode.UNAVAILABLE
if getattr(exception, 'code', lambda: None)() == unavailable:
return
# For anything else, fallback on super.
super(AvailablePolicy, self).on_exception(exception)
subscriber = pubsub.SubscriberClient(policy_class=AvailablePolicy)
# Continue to set up as normal.
It looks a lot like the original on_exception just ignores a different error. If you want, you can add some logging whenever the exception is thrown and verify that everything still works. Future messages will still come through.

Handling Non-SSL Traffic in Python/Tornado

I have a webservice running in python 2.7.10 / Tornado that uses SSL. This service throws an error when a non-SSL call comes through (http://...).
I don't want my service to be accessible when SSL is not used, but I'd like to handle it in a cleaner fashion.
Here is my main code that works great over SSL:
if __name__ == "__main__":
tornado.options.parse_command_line()
#does not work on 2.7.6
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("...crt.pem","...key.pem")
ssl_ctx.load_verify_locations("...CA.crt.pem")
http_server = tornado.httpserver.HTTPServer(application, ssl_options=ssl_ctx, decompress_request=True)
http_server.listen(options.port)
mainloop = tornado.ioloop.IOLoop.instance()
print("Main Server started on port XXXX")
mainloop.start()
and here is the error when I hit that server with http://... instead of https://...:
[E 151027 20:45:57 http1connection:700] Uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 691, in _server_request_loop
ret = yield conn.read_response(request_delegate)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 807, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 209, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 810, in run
yielded = self.gen.throw(*sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 166, in _read_message
quiet_exceptions=iostream.StreamClosedError)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 807, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 209, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:590)
Any ideas how I should handle that exception?
And what the standard-conform return value would be when I catch a non-SSL call to an SSL-only API?
UPDATE
This API runs on a specific port e.g. https://example.com:1234/. I want to inform a user who is trying to connect without SSL, e.g. http://example.com:1234/ that what they are doing is incorrect by returning an error message or status code. As it is the uncaught exception returns a 500, which they could interpret as a programming error on my part. Any ideas?

There's an excelent discussion in this Tornado issue about that, where Tornado maintainer says:
If you have both HTTP and HTTPS in the same tornado process, you must be running two separate HTTPServers (of course such a feature should not be tied to whether SSL is handled at the tornado level, since you could be terminating SSL in a proxy, but since your question stipulated that SSL was enabled in tornado let's focus on this case first). You could simply give the HTTP server a different Application, one that just does this redirect.
So, the best solution it's to HTTPServer that listens on port 80 and doesn't has the ssl_options parameter setted.
UPDATE
A request to https://example.com/some/path will go to port 443, where you must have an HTTPServer configured to handle https traffic; while a request to http://example.com/some/path will go to port 80, where you must have another instance of HTTPServer without ssl options, and this is where you must return the custom response code you want. That shouldn't raise any error.

How to catch an Exception like this on Flask?

I run a simple flask app like this:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def welcome():
return "OK"
app.config.update(
DEBUG = True
)
if __name__ == '__main__':
app.run(use_reloader = False)
when I run it and visit it, sometimes(not always) it could't response the request and throw an except:
Exception happened during processing of request from ('127.0.0.1', 54481)
Traceback (most recent call last):
File "c:\python27\Lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "c:\python27\Lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "c:\python27\Lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "c:\python27\Lib\SocketServer.py", line 651, in __init__
self.finish()
File "c:\python27\Lib\SocketServer.py", line 710, in finish
self.wfile.close()
File "c:\python27\Lib\socket.py", line 279, in close
self.flush()
File "c:\python27\Lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053]
I can't understand what cause this fault? and how can I solve it?
and how can I use try except to catch it?

I recently ran into this error message while trying to use Flask to serve audio files. I get this error message whenever the client closes the stream before the end of the stream.
This error message doesn't originate from your Flask application, but rather from the underlying SocketServer used to dispatch request data. What is happening is the connection to the client is ending for some reason, but Flask continues to try to write data to the closed socket. You can't catch this exception from your Flask application, because Flask catches it for you. Flask prints it out as a service to you, notifying you that the stream was closed prematurely, i.e. before Flask finished writing data to the stream.
To sum it up, this error message is internal to Flask, Flask is printing it to tell you that it couldn't get all the data to the client before the connection closed. You can't catch it, and you shouldn't have any reason to catch it.

I've found this solution to be a good at least temporary fix.
if __name__ == '__main__':
while True:
try:
app.run(use_reloader = False)
except:
pass
You can add your own exit logic, or leave the program with CTRL + \ which sends SIGQUIT. (important if you're running threaded flask)
You however can't:
except KeyboardInterupt:
Because Flask already catches KeyboardInterupt exceptions and handles them.
error 10052 means you're using windows, so as far as I know, close the command window to exit the program

It is probably due to the port number being used which is 54481 by looking at your error message. It might be clashing with something else. I also suggest not to use the use_reloader parameter since your DEBUG is already set to False. So flask will not reload any code changes. Can you instead do this :
if __name__ == '__main__':
app.run(port=5000)

How can flask render a page before some task(like email sending) has not finished

How can flask render a page before some task(like email sending) has not finished?
Issue
In our case, while user submit register info to server, server will send out a certification email first and then render a success page. But it always take 3~5 seconds to finish the email sending, which mean it take at least 3s to get response after he submit the register information.
What we have done
1. Signal
We have try send a signal and trigger a mail-sending function to do this thing:
if approved:
# ...
send_activation_signal.send(email)
login_user(u)
return redirect(url_for('register_success'))
result
every thing is OK except it still take at least 3s before the register_success page is rendered.
2. Fork a thread
fork a thread to finish the email sending
t = threading.Thread(target = send_activation, args = (email, username, email_confirm_code, ))
t.start()
result
but it looks like request context is not processed in correct way.
> Exception in thread Thread-2: Traceback (most recent call last):
> File
> "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py",
> line 552, in __bootstrap_inner
> self.run() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py",
> line 505, in run
> self.__target(*self.__args, **self.__kwargs) File "/Users/arthasshih/flask/happyvs/main_app/views/user.py", line 267, in
> send_activation
> app.logger.debug("sending #############") File "/Users/arthasshih/virtual_evns/normal/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/local.py",
> line 336, in __getattr__
> return getattr(self._get_current_object(), name) File "/Users/arthasshih/virtual_evns/normal/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/local.py",
> line 295, in _get_current_object
> return self.__local() File "/Users/arthasshih/virtual_evns/normal/lib/python2.7/site-packages/Flask-0.8-py2.7.egg/flask/globals.py",
> line 19, in _lookup_object
> raise RuntimeError('working outside of request context') RuntimeError: working outside of request context

I also faced this issue when setting up email sending in flask but the issue is not flask. This process has to be run asynchronously if you want flask to return before finishing send mail task but it is running synchronously here so thats why the wait.
The solution is using Celery: Distributed Task Queue.This text is from their home page:
It is an asynchronous task queue/job queue based on distributed message passing.Tasks can execute asynchronously (in the background) or synchronously (wait until ready).
There is also a Flask extension called Flask-Celery which makes it easy to integrate celery with flask.
Simple add mail sending task to celery's asynchronous task queue and render page. You have to make sure that celery generates some error log if the task fails otherwise you would not if mail is sent or not.