We Keep Coding

Can I make Python output exceptions in one line / via logging?

I am using AWS and use AWS cloudwatch to view logs. While things should not break on AWS, they could. I just had such a case. Then I searched for Traceback and just got the lines
Traceback (most recent call last):
without the actual traceback. I have a working structured logging setup (see other question) and I would like to get tracebacks in a similar way.
So instead of:
Traceback (most recent call last):
File "/home/math/Desktop/test.py", line 32, in <module>
adf
NameError: name 'adf' is not defined
something like
{"message": "Traceback (most recent call last):\n File \"/home/math/Desktop/test.py\", line 32, in <module>\n adf\n NameError: name 'adf' is not defined", "lineno": 35, "pathname": "/home/math/Desktop/test.py"}
or even better also with the string in a JSON format.
The only way to achieve this I can think of is a giant try-except block. Pokemon-style. Is there a better solution?

You can use sys.excepthook. It is invoked whenever an exception occurs in your script.
import logging
import sys
import traceback
def exception_logging(exctype, value, tb):
"""
Log exception by using the root logger.
Parameters
----------
exctype : type
value : NameError
tb : traceback
"""
write_val = {'exception_type': str(exctype),
'message': str(traceback.format_tb(tb, 10))}
logging.exception(str(write_val))
Then in your script you have to override the value of sys.excepthook.
sys.excepthook = exception_logging
Now whenever an exception occurs it will be logged with your logger handler.
Note: Don't forget to setup logger before running this

In case somebody wants the exception logged in its default format, but in one line (for any reason), based on the accepted answer:
def exception_logging(exctype, value, tb):
"""
Log exception in one line by using the root logger.
Parameters
----------
exctype : exception type
value : seems to be the Exception object (with its message)
tb : traceback
"""
logging.error(''.join(traceback.format_exception(exctype, value, tb)))
Please also note, that it uses logging.error() instead of logging.exception() which also printed some extra "NoneType: None" line.
Also note that it only seems to work with uncaught exceptions.
For logging caught exceptions, visit How do I can format exception stacktraces in Python logging? and see also my answer.

A slight variation: If you run a Flask application, you can do this:
#app.errorhandler(Exception)
def exception_logger(error):
"""Log the exception."""
logger.exception(str(error))
return str(error)

Python logger handler config for exception logging

I have written this piece of code to get namespace from xml document.
I am trying to handle exception, and write full trace to the log. however trace is not getting written with custom message in log (though i can see it on screen).
I believe, i am missing somthing in Logger handler config. is there any specific configuration we need to deal with? below is my logger config so far.
Any help will be appreciated.!
logger = logging.getLogger(__name__)
hdlr = logging.FileHandler(r'C:\link.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
def get_ns(xmlroot):
"""Retrieve XML Document Namespace """
try:
logger.info("Trying to get XML namespace detail")
nsmap = xmlroot.nsmap.copy()
logger.info("Creating XML Namespace Object, {0}".format(nsmap))
nsmap['xmlns'] = nsmap.pop(None)
except (KeyError, SystemExit):
logging.exception("XML files does not contain namespace, Halting Program! ")
sys.exit()
else:
for ns in nsmap.values():
logger.info("Retrieved XML Namespace {0}".format(ns))
return ns
output on screen:
ERROR:root:XML files does not contain namespace, Halting Program!
Traceback (most recent call last):
File "C:\link.log", line 28, in get_ns
nsmap['xmlns'] = nsmap.pop(None)
KeyError: None

Change
logging.exception("XML files does not contain namespace, Halting Program! ")
to
logger.exception("XML files does not contain namespace, Halting Program! ")
Since it's logger that you have configured to write to file C:\link.log.
Using logging.exception uses the "root logger", which outputs to the console by default.

Catching and printing the Warning message on Python > Selenium Webdriver

I'm trying to catch a warning message, print the warning message, and then exit out of a test case with a passing state. Within the test case, I have the following code:
testcase.py:
try:
warnings.filterwarnings('error')
activeConfig(driver, url, None, None, True).confirmConfigSet()
except Warning as e:
print e.message
return
As I'm able to catch the warning without any issues, I'll only display the warning that's actually caught:
code-where-warning-is-caught.py:
.
.
.
except (Exception, NoSuchElementException, TimeoutException):
global_vars.attemptedToEnableDatabase = True
warnings.warn("\nWARNING : DATABASE PACKAGES HAVE NOT BEEN INSTALLED SO THE DATABASE CANNOT BE ENABLED.", UserWarning)
return
I can catch the warning just fine, but I keep getting the following error:
E DeprecationWarning: BaseException.message has been deprecated as of
Python 2.6
If I remove the 'e' parts of the test case code to look like...
try:
warnings.filterwarnings('error')
activeConfig(driver, url, None, None, True).confirmConfigSet()
except Warning:
return
...the test case runs EXACTLY the way I want it to, except it doesn't print the warning message. How do I catch the exception, print the warning message, and exit out of the test case with a passing state? I'm also open to any improvement ideas a more experienced coder may have.

Please take a read here. They have everything explained, you can experiment with them a bit.
https://docs.python.org/2/library/traceback.html
In my recent project, I managed to write critical only messages to a text file by doing this,
logging.basicConfig(level=logging.CRITICAL,
format = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt = '%m-%d %H:%M',
filename = self.testResultRecord,
filemode = 'w')
lines below are my personal formatting configurations, which have nothing to do with logging level.
format = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt = '%m-%d %H:%M',
Regards

python logging not printing datetime and other format values

I'm trying to use the python logging module to create a RotatingFileHandler for my program. My log handler logs output to the file: /var/log/pdmd.log and the basic functionality seems to work and log output as desired.
However, I'm trying to format my log string with this format:
"%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s"
But only the message portion of the exception is getting logged. Here is my code to setup the logger:
#class variable declared at the beginning of the class declaration
log = logging.getLogger("PdmImportDaemon")
def logSetup(self):
FORMAT = "%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s"
logging.basicConfig(format=FORMAT)
#logging.basicConfig(level=logging.DEBUG)
self.log.setLevel(logging.DEBUG) #by setting our logger to the DEBUG level (lowest level) we will include all other levels by default
#setup the rotating file handler to automatically increment the log file name when the max size is reached
self.log.addHandler( logging.handlers.RotatingFileHandler('/var/log/pdmd.log', mode='a', maxBytes=50000, backupCount=5) )
Now, when I run a method and make the program output to the log with the following code:
def dirIterate( self ):
try:
raise Exception( "this is my exception, trying some cool output stuff here!")
except Exception, e:
self.log.error( e )
raise e
And the output in the pdmd.log file is just the exception text and nothing else. For some reason, the formatting is not being respected; I expected:
ERROR 2013-09-03 06:53:18,416 dirIterate 89 this is my exception, trying some cool output stuff here!
Any ideas as to why the formatting that I setup in my logging.basicConfig is not being respected?

You have to add the format to the Handler too.
When you run basicConfig(), you are configuring a new Handler for the root Logger.
In this case your custom Handler is getting no format.
Replace
self.log.addHandler( logging.handlers.RotatingFileHandler('/var/log/pdmd.log', mode='a', maxBytes=50000, backupCount=5) )
with:
rothnd = logging.handlers.RotatingFileHandler('/var/log/pdmd.log', mode='a', maxBytes=50000, backupCount=5)
rothnd.setFormatter(logging.Formatter(FORMAT))
self.log.addHandler(rothnd)

Finding the source of format errors when using python logging

When I have lots of different modules using the standard python logging module, the following stack trace does little to help me find out where, exactly, I had a badly formed log statement:
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/__init__.py", line 768, in emit
msg = self.format(record)
File "/usr/lib/python2.6/logging/__init__.py", line 648, in format
return fmt.format(record)
File "/usr/lib/python2.6/logging/__init__.py", line 436, in format
record.message = record.getMessage()
File "/usr/lib/python2.6/logging/__init__.py", line 306, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
I'm only starting to use python's logging module, so maybe I am overlooking something obvious. I'm not sure if the stack-trace is useless because I am using greenlets, or if this is normal for the logging module, but any help would be appreciated. I'd be willing to modify the source, anything to make the logging library actually give a clue as to where the problem lies.

Rather than editing installed python code, you can also find the errors like this:
def handleError(record):
raise RuntimeError(record)
handler.handleError = handleError
where handler is one of the handlers that is giving the problem. Now when the format error occurs you'll see the location.

The logging module is designed to stop bad log messages from killing the rest of the code, so the emit method catches errors and passes them to a method handleError. The easiest thing for you to do would be to temporarily edit /usr/lib/python2.6/logging/__init__.py, and find handleError. It looks something like this:
def handleError(self, record):
"""
Handle errors which occur during an emit() call.
This method should be called from handlers when an exception is
encountered during an emit() call. If raiseExceptions is false,
exceptions get silently ignored. This is what is mostly wanted
for a logging system - most users will not care about errors in
the logging system, they are more interested in application errors.
You could, however, replace this with a custom handler if you wish.
The record which was being processed is passed in to this method.
"""
if raiseExceptions:
ei = sys.exc_info()
try:
traceback.print_exception(ei[0], ei[1], ei[2],
None, sys.stderr)
sys.stderr.write('Logged from file %s, line %s\n' % (
record.filename, record.lineno))
except IOError:
pass # see issue 5971
finally:
del ei
Now temporarily edit it. Inserting a simple raise at the start should ensure the error gets propogated up your code instead of being swallowed. Once you've fixed the problem just restore the logging code to what it was.

It's not really an answer to the question, but hopefully it will be other beginners with the logging module like me.
My problem was that I replaced all occurrences of print with logging.info ,
so a valid line like print('a',a) became logging.info('a',a) (but it should be logging.info('a %s'%a) instead.
This was also hinted in How to traceback logging errors? , but it doesn't come up in the research

Alternatively you can create a formatter of your own, but then you have to include it everywhere.
class DebugFormatter(logging.Formatter):
def format(self, record):
try:
return super(DebugFormatter, self).format(record)
except:
print "Unable to format record"
print "record.filename ", record.filename
print "record.lineno ", record.lineno
print "record.msg ", record.msg
print "record.args: ",record.args
raise
FORMAT = '%(levelname)s %(filename)s:%(lineno)d %(message)s'
formatter = DebugFormatter(FORMAT)
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

Had same problem
Such a Traceback arises due to the wrong format name. So while creating a format for a log file, check the format name once in python documentation: "https://docs.python.org/3/library/logging.html#formatter-objects"