I'm trying to log various exceptions from libraries in python 2.7. I find that sometimes the exceptions contain a unicode string and sometimes a utf8 bytestring. I thought that logging.exception(e) was the right approach to log them, but the following doesn't seem to work:
# encoding: utf-8
import logging
try:
raise Exception('jörn')
except Exception as e:
logging.exception(e)
try:
raise Exception(u'jörn')
except Exception as e:
logging.exception(e)
saving this into a file and running it results in this:
$ python test.py
ERROR:root:jörn
Traceback (most recent call last):
File "test.py", line 4, in <module>
raise Exception('jörn')
Exception: jörn
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 859, in emit
msg = self.format(record)
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 732, in format
return fmt.format(record)
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 474, in format
s = self._fmt % record.__dict__
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in range(128)
Logged from file test.py, line 12
So as you see the utf8 exception worked fine, but the unicode exception broke logging, swallowing the real exception and hiding it behind a UnicodeEncodeError.
Is there some standard logging facility for exceptions that won't break my code? What am i missing?
Actually, i think i finally found the mistake and a proper way myself: I seem to have used logging.exception('msg') wrong the whole time. You're not meant to pass in the exception, but a message:
# encoding: utf-8
import logging
try:
raise Exception('jörn')
except Exception as e:
logging.exception('exception occurred')
try:
raise Exception(u'jörn')
except Exception as e:
logging.exception('exception occurred')
running the above correctly logs the exception:
$ python test.py
ERROR:root:exception occurred
Traceback (most recent call last):
File "test.py", line 4, in <module>
raise Exception('jörn')
Exception: jörn
ERROR:root:exception occurred
Traceback (most recent call last):
File "test.py", line 10, in <module>
raise Exception(u'jörn')
Exception: j\xf6rn
The reason why logging.exception(e) seems to fail is that it passes the exception e up into logging.Formatter.format() where the it arrives as record.message variable which still is an Exception object.
Then in Line 474 the following happens:
s = self._fmt % record.__dict__
which is equivalent to the following:
s = '%(levelname)s:%(name)s:%(message)s' % {
'levelname': 'ERROR',
'name': 'ROOT',
'message': Exception(u'jörn')
}
It turns out this is why if message is one of ['jörn', u'jörn', Exception('jörn')] it works and not if it is Exception(u'jörn'):
>>> 'foo %s' % 'jörn'
'foo j\xc3\xb6rn'
>>> 'foo %s' % u'jörn'
u'foo j\xf6rn'
>>> 'foo %s' % Exception('jörn')
'foo j\xc3\xb6rn'
>>> 'foo %s' % Exception(u'jörn')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in range(128)
As you can see there is an automatic upcasting that happens for unicode strings which is why the following works:
>>> logging.error('jörn')
ERROR:root:jörn
>>> logging.error(u'jörn')
ERROR:root:jörn
This transformation into unicode fails when attempting it with an Exception object that doesn't properly handle the encoding of its message (which sadly seems to be the case in a lot of libraries).
The logging.exception(msg) call seems to properly use the repr() for formatting the exception for logging and prefixes it with your msg. So if you don't make the mistake and pass the exception to logging.exception it will correctly log it.
So long story short:
Don't use logging.exception(e) but logging.exception('exception occurred'). It will automagically and correctly append the formatted exception to your log. If you really want to use the exception's message without assuming some encoding, the safest you can do is logging.exception(repr(e)).
It's not the logging that fails to deal with unicode, it's the Exception.__str__ method which does not support unicode strings as exception arguments. When you invoke logging.exception(e) it will do something like logging.exception(str(e)) which in turn does something like str(self.args) on the exception instance. That's where the error comes from, your self.args is a unicode string which cannot be encoded in ascii.
You have two options, either do logging.exception(unicode(e)) or implement your own exception class that provides a __str__ method that can deal with unicode objects in self.args.
The reason why your first test passes is that the editor encodes the string into UTF-8 and Python sees a string instance with encoded unicode characters.
There's a lot of ways of doing, but most simple is to "patch" logger in order for it to pre-convert any non unicode string so logging won't fail, regardless of source, which is quite useful since you can't control sources like tracebacks, when using something like logger.error(exc_info=True)
Patching sounds bad, but fortunately there is a mechanism called a contextfilter that does the job without modifying the code.
TL;DR: I have built the following code as a package Tested with all Python versions from 2.7 to 3.10. Use it with pip install ofunctions.logger_utils
Usage:
from ofunctions.logger_utils import logger_get_logger
logger = logger_get_logger(log_file='somepath')
logger.info('Ûnicode café")
TL;DR end
Consider the following code which sole purpose is to call function safe_string_convert() on a given logged message:
class FixPython2Logging(logging.Filter):
def __init__(self):
if sys.version_info[0] < 3:
# pylint: disable=E1003 (bad-super-call)
super(logging.Filter, self).__init__()
else:
super().__init__()
def filter(self, record):
# type: (str) -> bool
# Fix python2 unicodedecodeerrors when non unicode strings are sent to logger
if sys.version_info[0] < 3:
record.msg = safe_string_convert(record.msg)
return True
Of course we'd have to design a fool proof (or unicode decode error proof) function that makes sure we'll have a string that can be logged:
def safe_string_convert(string):
"""
Allows to encode strings for hacky UTF-8 logging in python 2.7
"""
try:
return string.decode("utf8")
except Exception: # noqa
try:
return string.decode("unicode-escape")
except Exception: # noqa
try:
return string.decode("latin1")
except Exception: # noqa
if sys.version_info[0] < 3:
# pylint: disable=E0602 (undefined-variable)
if isinstance(string, unicode): # noqa
return string
try:
return (
b"Cannot convert logged string. Passing it as binary blob: "
+ bytes(string)
)
except Exception: # noqa
return string
Now we can make sure FixPython2Logging filter will get executed by any logger call. Let's just add it as filter to our standard logger class:
log_filter = FixPython2Logging()
logger = logging.getLogger()
# Remove earlier handlers if exist
while _logger.handlers:
_logger.handlers.pop()
# Add context filter
logger.addFilter(log_filter)
# Try it
logger.info("Ûnicode café")
Here we go ;)
Of course the contextfilter works for your initial logger.exception() question ;)
I was expecting the following would work but PyDev is returning an error:
try fh = open(myFile):
logging.info("success")
except Exception as e:
logging.critical("failed because:")
logging.critical(e)
gives
Encountered "fh" at line 237, column 5. Was expecting: ":"...
I've looked around and cannot find a safe way to open a filehandle for reading in Python 3.4 and report errors properly. Can someone point me in the correct direction please?
You misplaced the :; it comes directly after try; it is better to put that on its own, separate line:
try:
fh = open(myFile)
logging.info("success")
except Exception as e:
logging.critical("failed because:")
logging.critical(e)
You placed the : after the open() call instead.
Instead of passing in e as a separate argument, you can tell logging to pick up the exception automatically:
try:
fh = open(myFile)
logging.info("success")
except Exception:
logging.critical("failed because:", exc_info=True)
and a full traceback will be included in the log. This is what the logging.exception() function does; it'll call logging.error() with exc_info set to true, producing a message at log level ERROR plus a traceback.
I'm using python to evaluate some measured data. Because of many possible results it is difficult to handle or possible combinations. Sometimes an error happens during the evaluation. It is usually an index error because I get out of range from measured data.
It is very difficult to find out on which place in code the problem happened. It would help a lot if I knew on which line the error was raised. If I use following code:
try:
result = evaluateData(data)
except Exception, err:
print ("Error: %s.\n" % str(err))
Unfortunately this only tells me that there is and index error. I would like to know more details about the exception (line in code, variable etc.) to find out what happened. Is it possible?
Thank you.
Solution, printing filename, linenumber, line itself and exception description:
import linecache
import sys
def PrintException():
exc_type, exc_obj, tb = sys.exc_info()
f = tb.tb_frame
lineno = tb.tb_lineno
filename = f.f_code.co_filename
linecache.checkcache(filename)
line = linecache.getline(filename, lineno, f.f_globals)
print 'EXCEPTION IN ({}, LINE {} "{}"): {}'.format(filename, lineno, line.strip(), exc_obj)
try:
print 1/0
except:
PrintException()
Output:
EXCEPTION IN (D:/Projects/delme3.py, LINE 15 "print 1/0"): integer division or modulo by zero
To simply get the line number you can use sys, if you would like to have more, try the traceback module.
import sys
try:
[][2]
except IndexError:
print("Error on line {}".format(sys.exc_info()[-1].tb_lineno))
prints:
Error on line 3
Example from the traceback module documentation:
import sys, traceback
def lumberjack():
bright_side_of_death()
def bright_side_of_death():
return tuple()[0]
try:
lumberjack()
except IndexError:
exc_type, exc_value, exc_traceback = sys.exc_info()
print "*** print_tb:"
traceback.print_tb(exc_traceback, limit=1, file=sys.stdout)
print "*** print_exception:"
traceback.print_exception(exc_type, exc_value, exc_traceback,
limit=2, file=sys.stdout)
print "*** print_exc:"
traceback.print_exc()
print "*** format_exc, first and last line:"
formatted_lines = traceback.format_exc().splitlines()
print formatted_lines[0]
print formatted_lines[-1]
print "*** format_exception:"
print repr(traceback.format_exception(exc_type, exc_value,
exc_traceback))
print "*** extract_tb:"
print repr(traceback.extract_tb(exc_traceback))
print "*** format_tb:"
print repr(traceback.format_tb(exc_traceback))
print "*** tb_lineno:", exc_traceback.tb_lineno
I use the traceback which is simple and robust:
import traceback
try:
raise ValueError()
except:
print(traceback.format_exc()) # or: traceback.print_exc()
Out:
Traceback (most recent call last):
File "catch.py", line 4, in <module>
raise ValueError()
ValueError
The simplest way is just to use:
import traceback
try:
<blah>
except IndexError:
traceback.print_exc()
or if using logging:
import logging
try:
<blah>
except IndexError as e:
logging.exception(e)
Gives you file, lineno, and exception for the last item in the call stack
from sys import exc_info
from traceback import format_exception
def print_exception():
etype, value, tb = exc_info()
info, error = format_exception(etype, value, tb)[-2:]
print(f'Exception in:\n{info}\n{error}')
try:
1 / 0
except:
print_exception()
prints
Exception in:
File "file.py", line 12, in <module>
1 / 0
ZeroDivisionError: division by zero
I would suggest using the python logging library, it has two useful methods that might help in this case.
logging.findCaller()
findCaller(stack_info=False) - Reports just the line number for the previous caller leading to the exception raised
findCaller(stack_info=True) - Reports the line number & stack for the previous caller leading to the exception raised
logging.logException()
Reports the line & stack within the try/except block that raised the exception
For more info checkout the api https://docs.python.org/3/library/logging.html
There are many answers already posted here that show how to get the line number, but it's worth noting that if you want variables containing the "raw data," so to speak, of the stack trace so that you can have more granular control of what you display or how you format it, using the traceback module you can step through the stack frame by frame and look at what's stored in the attributes of the frame summary objects. There are several simple and elegant ways to manipulate the frame summary objects directly. Let's say for example that you want the line number from the last frame in the stack (which tells you which line of code triggered the exception), here's how you could get it by accessing the relevant frame summary object:
Option 1:
import sys
import traceback
try:
# code that raises an exception
except Exception as exc:
exc_type, exc_value, exc_tb = sys.exc_info()
stack_summary = traceback.extract_tb(exc_tb)
end = stack_summary[-1] # or `stack_summary.pop(-1)` if you prefer
Option 2:
import sys
import traceback
try:
# code that raises an exception
except Exception as exc:
tbe = traceback.TracebackException(*sys.exc_info())
end = tbe.stack[-1] # or `tbe.stack.pop(-1)` if you prefer
In either of the above examples, end will be a frame summary object:
>>> type(end)
<class 'traceback.FrameSummary'>
which was in turn taken from a stack summary object:
>>> type(stack_summary) # from option 1
<class 'traceback.StackSummary'>
>>> type(tbe.stack) # from option 2
<class 'traceback.StackSummary'>
The stack summary object behaves like a list and you can iterate through all of the frame summary objects in it however you want in order to trace through the error. The frame summary object (end, in this example), contains the line number and everything else you need to locate where in the code the exception occurred:
>>> print(end.__doc__)
A single frame from a traceback.
- :attr:`filename` The filename for the frame.
- :attr:`lineno` The line within filename for the frame that was
active when the frame was captured.
- :attr:`name` The name of the function or method that was executing
when the frame was captured.
- :attr:`line` The text from the linecache module for the
of code that was running when the frame was captured.
- :attr:`locals` Either None if locals were not supplied, or a dict
mapping the name to the repr() of the variable.
And if you capture the exception object (either from except Exception as exc: syntax or from the second object returned by sys.exc_info()), you will then have everything you need to write your own highly customized error printing/logging function:
err_type = type(exc).__name__
err_msg = str(exc)
Putting it all together:
from datetime import datetime
import sys
import traceback
def print_custom_error_message():
exc_type, exc_value, exc_tb = sys.exc_info()
stack_summary = traceback.extract_tb(exc_tb)
end = stack_summary[-1]
err_type = type(exc_value).__name__
err_msg = str(exc_value)
date = datetime.strftime(datetime.now(), "%B %d, %Y at precisely %I:%M %p")
print(f"On {date}, a {err_type} occured in {end.filename} inside {end.name} on line {end.lineno} with the error message: {err_msg}.")
print(f"The following line of code is responsible: {end.line!r}")
print("Please make a note of it.")
def do_something_wrong():
try:
1/0
except Exception as exc:
print_custom_error_message()
if __name__ == "__main__":
do_something_wrong()
Let's run it!
user#some_machine:~$ python example.py
On August 25, 2022 at precisely 01:31 AM, a ZeroDivisionError occured in example.py inside do_something_wrong on line 21 with the error message: division by zero.
The following line of code is responsible: '1/0'
Please make a note of it.
At this point you can see how you could print this message for any place in the stack: end, beginning, anywhere in-between, or iterate through and print it for every frame in the stack.
Of course, the formatting functionality already provided by the traceback module covers most debugging use cases, but it's useful to know how to manipulate the traceback objects to extract the information you want.
I always use this snippet
import sys, os
try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
for different views and possible issues you can refer When I catch an exception, how do I get the type, file, and line number?
All the solutions answer the OPs problem, however, if one is holding onto a specific error instance and the last traceback stack won't do it they will not suffice —a corner scenario example given below.
In this case, the magic attribute __traceback__ of a raised exception instance may be used. This is a system traceback object (exposed as types.TracebackType, see Types) not the module. This traceback instance behaves just one like would expect (but it is always nice to check):
from collections import deque
from typing import (List, )
errors: List[Exception] = deque([], 5)
def get_raised(error_cls: type, msg: str) -> Exception:
# for debugging, an exception instance that is not raised
# ``__traceback__`` has ``None``
try:
raise error_cls(msg)
except Exception as error:
return error
error = get_raised(NameError, 'foo')
errors.append(error)
error = get_raised(ValueError, 'bar')
errors.append(error)
try:
raise get_raised(TypeError, 'bar')
except Exception as error:
errors.append(error)
Now, I can check out the first error's details:
import types
traceback = errors[0].__traceback__ # inadvisable name due to module
line_no: int = traceback.tb_lineno
frame: types.FrameType = traceback.tb_frame
previous: Union[type(None), types.TracebackType] = traceback.tb_next
filename: str = frame.f_code.co_filename
The traceback's previous is None for the second error despite a preceding error as expected, but for the third wherein an error is raised twice it is not.
This below is just a test which makes no sense contextually. A case were it is useful if when an exception is raised in a view of a webapp (a 500 status kind of incident) that gets caught and stored for the admit to inspect akin to Setry.io (but for free). Here is a minimal example where the home page / will raise an error, that gets caught and the route errors will list them. This is using Pyramid in a very concentrated way (multifile is way better) with no logging or authentication and the error logging could be better for the admin to inspect similar to Sentry.io.
from pyramid.config import Configurator
from waitress import serve
from collections import deque
# just for typehinting:
from pyramid.request import Request
from pyramid.traversal import DefaultRootFactory
from pyramid.router import Router
import types
from typing import (List, )
def home_view(context: DefaultRootFactory, request: Request) -> dict:
raise NotImplementedError('I forgot to fill this')
return {'status': 'ok'} # never reached.
def caught_view(error: Exception, request: Request) -> dict:
"""
Exception above is just type hinting.
This is controlled by the context argument in
either the ``add_exception_view`` method of config,
or the ``exception_view_config`` decorator factory (callable class)
"""
# this below is a simplification as URLDecodeError is an attack (418)
request.response.status = 500
config.registry.settings['error_buffer'].append(error)
#logging.exception(error) # were it set up.
#slack_admin(format_error(error)) # ditto
return {'status': 'error', 'message': 'The server crashed!!'}
def format_error(error: Exception) -> str:
traceback = error.__traceback__ # inadvisable name due to module
frame: types.FrameType = traceback.tb_frame
return f'{type(error).__name__}: {error}' +\
f'at line {traceback.tb_lineno} in file {frame.f_code.co_filename}'
def error_view(context: DefaultRootFactory, request: Request) -> dict:
print(request.registry.settings['error_buffer'])
return {'status': 'ok',
'errors':list(map(format_error, request.registry.settings['error_buffer']))
}
with Configurator(settings=dict()) as config:
config.add_route('home', '/')
config.add_route('errors', '/errors')
config.add_view(home_view, route_name='home', renderer='json')
config.add_view(error_view, route_name='errors', renderer='json')
config.add_exception_view(caught_view, context=Exception, renderer='json')
config.registry.settings['error_buffer']: List[Exception] = deque([], 5)
# not in config.registry.settings, not JSON serialisable
# config.add_request_method
app : Router = config.make_wsgi_app()
port = 6969
serve(app, port=port)
I want to catch an exception when user fails login due to wrong password .
So i make a function using imaplib .I enter a wrong password and get a traceback with error details.
Now my question is actually general.How do you identify the exception we have to mention in our "try and except" body from the error messages?
These is what I got->
>>> count("testarc31#gmail.com","Xbox#36")
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
count("testarc31#gmail.com","Xbox#36")
File "E:\Arindam\py_progs\Mail Notifier\0.0.19\Mail.py", line 24, in count
obj.login(m,p)
File "C:\Python27\lib\imaplib.py", line 500, in login
raise self.error(dat[-1])
error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
If i want to make a try and except,what will i mention in the exception part?
try:
login(mail,pass):
except ????:
something
Question :
1) What will be ???? here . Can it be deduced directly from the error report?
2) Is there a basic idea to identify what is the exception we have to use from each error we get ?
You want to use something like this:
try:
..code that might raise an exception...
except ExceptionType, e:
...do something...
In your case, that probably want this:
try:
login(mail,pass)
except imaplib.IMAP4.error, e:
print "Ouch -- an error from imaplib!"
To identify the type of an exception, you can look at its exception message. In this case it's just "error" -- unfortunately the module name is not included. You can get a better idea of exactly where it comes from by doing:
try:
login(mail,pass)
except Exception, e:
print type(e)
I am having trouble getting this to work correctly (obviously) - I am ALMOST there, and I have a good idea of WHY it is not working - just not sure how to make it work.
This is suppose to attempt to read a file into memory, if it fails it goes to the "except" clause of the block of code (that part is 'duh'). The error file prints: "<main.DebugOutput instance at 0x04021EB8>". What I want it to do is print the actual Error. Like a FileIOError or TraceBackError or whatever it is to that error file. This is just the beginning stages and I plan to add things like date stamps and to have it append, not write/create - I just need the actual error printed to the file. Advice?
import os, sys
try:
myPidFile = "Zeznadata.txt"
myOpenPID_File = open(myPidFile, "r") #Attempts to open the file
print "Sucessfully opened the file: \"" + myPidFile + "\"."
except:
print "This file, \"" + myPidFile + "\", does not exist. Please check the file name and try again. "
myFileErr = open("PIDErrorlog.txt", "w")
myStdError = str(sys.stderr)
myFileErr.write(myStdError)
myFileErr.close()
print "\nThis error was logged in the file (and stored in the directory): "
First, you should use a logging library. It will help you deal with different logging levels (info/warn/error), timestamps and more.
http://docs.python.org/library/logging.html
Second, you need to catch the error, and then you can log details about it. This example comes from the Python documentation.
import sys
try:
f = open('myfile.txt')
s = f.readline()
i = int(s.strip())
except IOError as (errno, strerror):
print "I/O error({0}): {1}".format(errno, strerror)
except ValueError:
print "Could not convert data to an integer."
except:
print "Unexpected error:", sys.exc_info()[0]
raise
See how it catches an IOError and assigns the error number and error message to variables? You can have an except block for each type of error you want to deal with. In the last (generic error) block, it uses sys.exc_info()[0] to get the error details.
http://docs.python.org/tutorial/errors.html
the problem is here:
myStdError = str(sys.stderr)
myFileErr.write(myStdError)
sys.stderr is a file-like interface defined in POSIX standards, called standard error, not "error text" that you can write to a file. So what you (probably) wanted to do is:
sys.stderr = myFileErr
This is the python equivalent to python your_python_sctipt.py 2> PIDErrorLog.txt in unix shell.
Use the logging module. The logging module have exception formatters that will help you print exceptions in a pretty way.
http://docs.python.org/library/logging.html
Like Kimvais says, your problem is:
myStdError = str(sys.stderr)
myFileErr.write(myStdError)
sys.stderr is a file handle to stderr (stdout is what print writes to).
You should do something like:
try:
...open file...
except IOError as (errno, strerror):
...open errfile...
errfile.write(strerror)
errfile.close()
Alison's answer is also very good and well written.