We Keep Coding

How to capture the return status or exception raised from python logging module

I am trying to log some messages to a log file. Inputs of the logger happens to be some unicode strings as well.
Here is an example code:
import logging
logging.basicConfig(filename='ReviewReport.log',
filemode='w',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s')
myTestStr = 'My Str is DeltaΔ'
try:
logging.info(myTestStr )
except Exception:
print('In except block....')
#handle exception by encoding the string.
print('execute rest of the program')
When i do this i see below exception on console :
UnicodeEncodeError: 'charmap' codec can't encode character '\u0394' in position 49: character maps to <undefined>
I tried a lot to understand why my code is not reaching to except block. 'In except block....' does not gets printed but 'execute rest of the program' is printed which makes me feel that logger is not really generating any exception when 'myTestStr' is not logged to my log file.
My Question is:
Why the UnicodeEncodeError is not caught in except block ? Does logger module not raise any exception.
What is correct approach to know if logger was successful in logging the message to file.
Appreciate any support here. Thanks.

Python not logging error output

I'm using Jupyter Notebook to develop some Python. This is my first stab at logging errors and I'm having an issue where no errors are logged to my error file.
I'm using:
import logging
logger = logging.getLogger('error')
logger.propagate = False
hdlr = logging.FileHandler("error.log")
formatter = logging.Formatter('%(asctime)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
to create the logger.
I'm then using a try/Except block which is being called on purpose (for testing) by using a column that doesn't exist in my database:
try:
some_bad_call_to_the_database('start',group_id)
except Exception as e:
logger.exception("an error I would like to log")
print traceback.format_exc(limit=1)
and the exception is called as can be seen from my output in my notebook:
Traceback (most recent call last):
File "<ipython-input-10-ef8532b8e6e0>", line 19, in <module>
some_bad_call_to_the_database('start',group_id)
InternalError: (1054, u"Unknown column 'start_times' in 'field list'")
However, error.log is not being written to. Any thoughts would be appreciated.

try:
except Exception as e:
excep = logger.info("an error I would like to log")
for more information please look at the docs https://docs.python.org/2/library/logging.html
have a great day!

You're using logging.exception, which delegates to the root logger, instead of using logger.exception, which would use yours (and write to error.log).

Identify a specific exception in python

I have a problem with identifying an exception.
Im writing a scraper that scrapes a lot of different websites, and some errors I want to handle and some I only want to ignore.
I except my exceptions like this:
except Exception as e:
most of the exceptions I can identify like this:
type(e).__name__ == "IOError"
But I have one exception "[Errno 10054] An existing connection was forcibly closed by the remote host"
that has the name "error" which is too vague and Im guessing other errors also have that name. Im guessing I can somehow get the errno number from my exception and thus identify it. But I don't know how.

First, you should not rely on the exception's class name, but on the class itself - two classes from two different modules can have the same value for the __name__ attribute while being different exceptions. So what you want is:
try:
something_that_may_raise()
except IOError as e:
handle_io_error(e)
except SomeOtherError as e:
handle_some_other_error(e)
etc...
Then you have two kind of exceptions: the one that you can actually handle one way or another, and the other ones. If the program is only for your personal use, the best way to handle "the other ones" is usually to not handle them at all - the Python runtime will catch them, display a nice traceback with all relevant informations (so you know what happened and where and can eventually add some handling for this case).
If it's a "public" program and/or if you do have some things to clean up before the program crash, you can add a last "catch all" except clause at the program's top level that will log the error and traceback somewhere so it isn't lost (logging.exception is your friend), clean what has to be cleaned and terminate with a more friendly error message.
There are very few cases where one would really want to just ignore an exception (I mean pretending nothing wrong or unexpected happened and happily continue). At the very least you will want to notify the user one of the actions failed and why - in your case that might be a top-level loop iterating over a set of sites to scrap, with an inner try/except block catching "expected" error cases, ie:
# config:
config = [
# ('url', {params})
('some.site.tld', {"param1" : value1, "param2" : value2}),
('some.other.tld', {"param1" : value1, "answer" : 42}),
# etc
]
def run():
for url, params in config:
try:
results = scrap(url, **params)
except (SomeKnownError, SomeOtherExceptedException) as e:
# things that are to be expected and mostly harmless
#
# you configured your logger so that warnings only
# go to stderr
logger.warning("failed to scrap %s : %s - skipping", url, e)
except (MoreSeriousError, SomethingIWannaKnowAbout) as e:
# things that are more annoying and you want to know
# about but that shouldn't prevent from continuing
# with the remaining sites
#
# you configured your logger so that exceptions goes
# to both stderr and your email.
logger.exception("failed to scrap %s : %s - skipping", url, e)
else:
do_something_with(results)
Then have a top-level handler around the call to run() that takes care of unexpected errors :
def main(argv):
parse_args()
try:
set_up_everything()
run()
return 0
except Exception as e:
logger.exception("oops, something unexpected happened : %s", e)
return 1
finally:
do_some_cleanup()
if __name__ == "__main__":
sys.exit(main(sys.argv))
Note that the logging module has an SMTPHandler - but since mail can easily fail too you'd better still have a reliable log (stderr and tee to a file ?) locally. The logging module takes some time to learn but it really pays off in the long run.

python logging not printing datetime and other format values

I'm trying to use the python logging module to create a RotatingFileHandler for my program. My log handler logs output to the file: /var/log/pdmd.log and the basic functionality seems to work and log output as desired.
However, I'm trying to format my log string with this format:
"%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s"
But only the message portion of the exception is getting logged. Here is my code to setup the logger:
#class variable declared at the beginning of the class declaration
log = logging.getLogger("PdmImportDaemon")
def logSetup(self):
FORMAT = "%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s"
logging.basicConfig(format=FORMAT)
#logging.basicConfig(level=logging.DEBUG)
self.log.setLevel(logging.DEBUG) #by setting our logger to the DEBUG level (lowest level) we will include all other levels by default
#setup the rotating file handler to automatically increment the log file name when the max size is reached
self.log.addHandler( logging.handlers.RotatingFileHandler('/var/log/pdmd.log', mode='a', maxBytes=50000, backupCount=5) )
Now, when I run a method and make the program output to the log with the following code:
def dirIterate( self ):
try:
raise Exception( "this is my exception, trying some cool output stuff here!")
except Exception, e:
self.log.error( e )
raise e
And the output in the pdmd.log file is just the exception text and nothing else. For some reason, the formatting is not being respected; I expected:
ERROR 2013-09-03 06:53:18,416 dirIterate 89 this is my exception, trying some cool output stuff here!
Any ideas as to why the formatting that I setup in my logging.basicConfig is not being respected?

You have to add the format to the Handler too.
When you run basicConfig(), you are configuring a new Handler for the root Logger.
In this case your custom Handler is getting no format.
Replace
self.log.addHandler( logging.handlers.RotatingFileHandler('/var/log/pdmd.log', mode='a', maxBytes=50000, backupCount=5) )
with:
rothnd = logging.handlers.RotatingFileHandler('/var/log/pdmd.log', mode='a', maxBytes=50000, backupCount=5)
rothnd.setFormatter(logging.Formatter(FORMAT))
self.log.addHandler(rothnd)

Finding the source of format errors when using python logging

When I have lots of different modules using the standard python logging module, the following stack trace does little to help me find out where, exactly, I had a badly formed log statement:
Traceback (most recent call last):
File "/usr/lib/python2.6/logging/__init__.py", line 768, in emit
msg = self.format(record)
File "/usr/lib/python2.6/logging/__init__.py", line 648, in format
return fmt.format(record)
File "/usr/lib/python2.6/logging/__init__.py", line 436, in format
record.message = record.getMessage()
File "/usr/lib/python2.6/logging/__init__.py", line 306, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
I'm only starting to use python's logging module, so maybe I am overlooking something obvious. I'm not sure if the stack-trace is useless because I am using greenlets, or if this is normal for the logging module, but any help would be appreciated. I'd be willing to modify the source, anything to make the logging library actually give a clue as to where the problem lies.

Rather than editing installed python code, you can also find the errors like this:
def handleError(record):
raise RuntimeError(record)
handler.handleError = handleError
where handler is one of the handlers that is giving the problem. Now when the format error occurs you'll see the location.

The logging module is designed to stop bad log messages from killing the rest of the code, so the emit method catches errors and passes them to a method handleError. The easiest thing for you to do would be to temporarily edit /usr/lib/python2.6/logging/__init__.py, and find handleError. It looks something like this:
def handleError(self, record):
"""
Handle errors which occur during an emit() call.
This method should be called from handlers when an exception is
encountered during an emit() call. If raiseExceptions is false,
exceptions get silently ignored. This is what is mostly wanted
for a logging system - most users will not care about errors in
the logging system, they are more interested in application errors.
You could, however, replace this with a custom handler if you wish.
The record which was being processed is passed in to this method.
"""
if raiseExceptions:
ei = sys.exc_info()
try:
traceback.print_exception(ei[0], ei[1], ei[2],
None, sys.stderr)
sys.stderr.write('Logged from file %s, line %s\n' % (
record.filename, record.lineno))
except IOError:
pass # see issue 5971
finally:
del ei
Now temporarily edit it. Inserting a simple raise at the start should ensure the error gets propogated up your code instead of being swallowed. Once you've fixed the problem just restore the logging code to what it was.

It's not really an answer to the question, but hopefully it will be other beginners with the logging module like me.
My problem was that I replaced all occurrences of print with logging.info ,
so a valid line like print('a',a) became logging.info('a',a) (but it should be logging.info('a %s'%a) instead.
This was also hinted in How to traceback logging errors? , but it doesn't come up in the research

Alternatively you can create a formatter of your own, but then you have to include it everywhere.
class DebugFormatter(logging.Formatter):
def format(self, record):
try:
return super(DebugFormatter, self).format(record)
except:
print "Unable to format record"
print "record.filename ", record.filename
print "record.lineno ", record.lineno
print "record.msg ", record.msg
print "record.args: ",record.args
raise
FORMAT = '%(levelname)s %(filename)s:%(lineno)d %(message)s'
formatter = DebugFormatter(FORMAT)
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

Had same problem
Such a Traceback arises due to the wrong format name. So while creating a format for a log file, check the format name once in python documentation: "https://docs.python.org/3/library/logging.html#formatter-objects"