properly logging unicode & utf-8 exceptions in python 2 - python

I'm trying to log various exceptions from libraries in python 2.7. I find that sometimes the exceptions contain a unicode string and sometimes a utf8 bytestring. I thought that logging.exception(e) was the right approach to log them, but the following doesn't seem to work:
# encoding: utf-8
import logging
try:
raise Exception('jörn')
except Exception as e:
logging.exception(e)
try:
raise Exception(u'jörn')
except Exception as e:
logging.exception(e)
saving this into a file and running it results in this:
$ python test.py
ERROR:root:jörn
Traceback (most recent call last):
File "test.py", line 4, in <module>
raise Exception('jörn')
Exception: jörn
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 859, in emit
msg = self.format(record)
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 732, in format
return fmt.format(record)
File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 474, in format
s = self._fmt % record.__dict__
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in range(128)
Logged from file test.py, line 12
So as you see the utf8 exception worked fine, but the unicode exception broke logging, swallowing the real exception and hiding it behind a UnicodeEncodeError.
Is there some standard logging facility for exceptions that won't break my code? What am i missing?

Actually, i think i finally found the mistake and a proper way myself: I seem to have used logging.exception('msg') wrong the whole time. You're not meant to pass in the exception, but a message:
# encoding: utf-8
import logging
try:
raise Exception('jörn')
except Exception as e:
logging.exception('exception occurred')
try:
raise Exception(u'jörn')
except Exception as e:
logging.exception('exception occurred')
running the above correctly logs the exception:
$ python test.py
ERROR:root:exception occurred
Traceback (most recent call last):
File "test.py", line 4, in <module>
raise Exception('jörn')
Exception: jörn
ERROR:root:exception occurred
Traceback (most recent call last):
File "test.py", line 10, in <module>
raise Exception(u'jörn')
Exception: j\xf6rn
The reason why logging.exception(e) seems to fail is that it passes the exception e up into logging.Formatter.format() where the it arrives as record.message variable which still is an Exception object.
Then in Line 474 the following happens:
s = self._fmt % record.__dict__
which is equivalent to the following:
s = '%(levelname)s:%(name)s:%(message)s' % {
'levelname': 'ERROR',
'name': 'ROOT',
'message': Exception(u'jörn')
}
It turns out this is why if message is one of ['jörn', u'jörn', Exception('jörn')] it works and not if it is Exception(u'jörn'):
>>> 'foo %s' % 'jörn'
'foo j\xc3\xb6rn'
>>> 'foo %s' % u'jörn'
u'foo j\xf6rn'
>>> 'foo %s' % Exception('jörn')
'foo j\xc3\xb6rn'
>>> 'foo %s' % Exception(u'jörn')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in range(128)
As you can see there is an automatic upcasting that happens for unicode strings which is why the following works:
>>> logging.error('jörn')
ERROR:root:jörn
>>> logging.error(u'jörn')
ERROR:root:jörn
This transformation into unicode fails when attempting it with an Exception object that doesn't properly handle the encoding of its message (which sadly seems to be the case in a lot of libraries).
The logging.exception(msg) call seems to properly use the repr() for formatting the exception for logging and prefixes it with your msg. So if you don't make the mistake and pass the exception to logging.exception it will correctly log it.
So long story short:
Don't use logging.exception(e) but logging.exception('exception occurred'). It will automagically and correctly append the formatted exception to your log. If you really want to use the exception's message without assuming some encoding, the safest you can do is logging.exception(repr(e)).

It's not the logging that fails to deal with unicode, it's the Exception.__str__ method which does not support unicode strings as exception arguments. When you invoke logging.exception(e) it will do something like logging.exception(str(e)) which in turn does something like str(self.args) on the exception instance. That's where the error comes from, your self.args is a unicode string which cannot be encoded in ascii.
You have two options, either do logging.exception(unicode(e)) or implement your own exception class that provides a __str__ method that can deal with unicode objects in self.args.
The reason why your first test passes is that the editor encodes the string into UTF-8 and Python sees a string instance with encoded unicode characters.

There's a lot of ways of doing, but most simple is to "patch" logger in order for it to pre-convert any non unicode string so logging won't fail, regardless of source, which is quite useful since you can't control sources like tracebacks, when using something like logger.error(exc_info=True)
Patching sounds bad, but fortunately there is a mechanism called a contextfilter that does the job without modifying the code.
TL;DR: I have built the following code as a package Tested with all Python versions from 2.7 to 3.10. Use it with pip install ofunctions.logger_utils
Usage:
from ofunctions.logger_utils import logger_get_logger
logger = logger_get_logger(log_file='somepath')
logger.info('Ûnicode café")
TL;DR end
Consider the following code which sole purpose is to call function safe_string_convert() on a given logged message:
class FixPython2Logging(logging.Filter):
def __init__(self):
if sys.version_info[0] < 3:
# pylint: disable=E1003 (bad-super-call)
super(logging.Filter, self).__init__()
else:
super().__init__()
def filter(self, record):
# type: (str) -> bool
# Fix python2 unicodedecodeerrors when non unicode strings are sent to logger
if sys.version_info[0] < 3:
record.msg = safe_string_convert(record.msg)
return True
Of course we'd have to design a fool proof (or unicode decode error proof) function that makes sure we'll have a string that can be logged:
def safe_string_convert(string):
"""
Allows to encode strings for hacky UTF-8 logging in python 2.7
"""
try:
return string.decode("utf8")
except Exception: # noqa
try:
return string.decode("unicode-escape")
except Exception: # noqa
try:
return string.decode("latin1")
except Exception: # noqa
if sys.version_info[0] < 3:
# pylint: disable=E0602 (undefined-variable)
if isinstance(string, unicode): # noqa
return string
try:
return (
b"Cannot convert logged string. Passing it as binary blob: "
+ bytes(string)
)
except Exception: # noqa
return string
Now we can make sure FixPython2Logging filter will get executed by any logger call. Let's just add it as filter to our standard logger class:
log_filter = FixPython2Logging()
logger = logging.getLogger()
# Remove earlier handlers if exist
while _logger.handlers:
_logger.handlers.pop()
# Add context filter
logger.addFilter(log_filter)
# Try it
logger.info("Ûnicode café")
Here we go ;)
Of course the contextfilter works for your initial logger.exception() question ;)

Related

How to remove additional text while raising an error in python

So i made some random stuff:
def println(text: str) -> str:
print(text)
if not type(text) == str:
raise TypeError("Text not string type")
println("Hello, World!")
Now, when i put a string inside the println() function it works perfectly fine. But if i put an integer inside the println() function it raises a TypeError with the location where it's coming from like this:
Traceback (most recent call last):
File "c:\Users\???\Desktop\leahnn files\python projects (do not delete)\Main files\main.py", line 6, in <module>
println(1)
File "c:\Users\???\Desktop\leahnn files\python projects (do not delete)\Main files\main.py", line 4, in println
raise TypeError("Text not string type")
TypeError: Text not string type
It does raise the TypeError but it's saying where it's coming from. I was wondering if you can just remove the location it's coming from and just say:
TypeError: Text not string type
If i just do:
def println(text: str) -> str:
print(text)
if not type(text) == str:
print("TypeError: Text not string type")
It's gonna print out the integer that is inside the println() function and it's gonna print the TypeError after the integer inside the println() function is done executing. Is it possible?
You can handle exception with
try-except clause
and in except just print error message.
See https://docs.python.org/3/tutorial/errors.html#handling-exceptions
try:
println(1)
except TypeError as err:
print(err)

Why does my message for exception not change with my if statement

I currently have the following code which throws an error on exception just how i want:
try:
something ....
except Exception as e:
print(
'You have encountered the following in the main function \n ERROR: {}'.format(e))
However in some cases if I get a specific exception such as:
invalid literal for int() with base 10: ''
I want to change the message of e in the exception to what i want.. how would i go about this?
If e == "invalid literal for int() with base 10: ''":
e = 'my new message'
print(e)
but it doesnt seem to be working
Try catching the type of error instead of parsing the text of the error.
More info can be found at Handling Exceptions section of Python help but to be fully thorough (because I feel dumb for initially answering a Python question in C#) you can sort out what exception type you're looking for with something like this:
>>> # Create the error
>>> int('3.6')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '3.6'
Where ValueError is the error type you need to catch.
More realistically, you can incorporate figuring it your uncaught error types into your program and (hopefully) identify them during testing:
>>> try:
... # something ....
... int('3.6') # for the example, we'll generate error on purpose
... # Assume we've already figured out what to do with these 3 errors
... except (RuntimeError, TypeError, NameError):
... print("We know what to do with these errors")
... # Our generic except to catch unhandled errors.
... except:
... print("Unhandled error: {0}".format(err))
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: invalid literal for int() with base 10: '3.6'
Once you identify a new error type, add a specific handler for it:
>>> try:
... # something ....
... int('3.6')
... except (RuntimeError, TypeError, NameError):
... print("We know what to do with these errors")
... # The newly added handler for ValueError type
... except ValueError:
... print("And now we know what to do with a ValueError")
... print("My new message")
... except:
... print("Unhandled error: {0}".format(err))
And now we know what to do with a ValueError
My new message
Original (completely useless) answer kept here for posterity (and so the comments make sense)...
Try catching the type of error instead of parsing the text of the error.
e.g.
catch (FileNotFoundException e)
{
// FileNotFoundExceptions are handled here.
}
catch (IOException e)
{
// Extract some information from this exception, and then
// throw it to the parent method.
if (e.Source != null)
Console.WriteLine("IOException source: {0}", e.Source);
throw;
}
which is copied directly from here:
Microsoft try-catch (C# Reference)

Python - Traceback, how to show filename of imported

I've got the following:
try:
package_info = __import__('app') #app.py
except:
print traceback.extract_tb(sys.exc_info()[-1])
print traceback.tb_lineno(sys.exc_info()[-1])
And what i get from this is:
[('test.py', 18, '<module>', 'package_info = __import__(\'app\')')]
18
Now this is almost what i want, this is where the actual error begins but i need to follow this through and get the actual infection, that is app.py containing an ä on row 17 not 18 for instance.
Here's my actual error message if untreated:
Non-ASCII character '\xc3' in file C:\app.py on line 17, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details", ('C:\app.py', 17, 0, None)), )
I've found some examples but all of them show the point of impact and not the actual cause to the problem, how to go about this (pref Python2 and Python3 cross-support but Python2 is more important in this scenario) to get the filename, row and cause of the problem in a similar manner to the tuple above?
Catch the specific exception and see what information it has. The message is formatted from the exception object's parameters so its a good bet that its there. In this case, SyntaxError includes a filename attribute.
try:
package_info = __import__('app') #app.py
except SyntaxError, e:
print traceback.extract_tb(sys.exc_info()[-1])
print traceback.tb_lineno(sys.exc_info()[-1])
print e.filename
For me
except Exception as e:
print e.__unicode__()
works with returning exactly same message
To be python 2.5 and 3.x compatible at the same time (2.5 does not support except Exception as e), use
try:
package_info = __import__('app') #app.py
except SyntaxError:
exc_type, exc, tb = sys.exc_info()
print(exc)
print('\n\n'.join(traceback.format_tb(tb, limit=5)))
prints
Non-ASCII character '\xc3' in file /foo/app.py on line 2, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details (app.py, line 2)
File "foo.py", line 6, in <module>
package_info = __import__('app') #app.py

Python exception handling - line number

I'm using python to evaluate some measured data. Because of many possible results it is difficult to handle or possible combinations. Sometimes an error happens during the evaluation. It is usually an index error because I get out of range from measured data.
It is very difficult to find out on which place in code the problem happened. It would help a lot if I knew on which line the error was raised. If I use following code:
try:
result = evaluateData(data)
except Exception, err:
print ("Error: %s.\n" % str(err))
Unfortunately this only tells me that there is and index error. I would like to know more details about the exception (line in code, variable etc.) to find out what happened. Is it possible?
Thank you.
Solution, printing filename, linenumber, line itself and exception description:
import linecache
import sys
def PrintException():
exc_type, exc_obj, tb = sys.exc_info()
f = tb.tb_frame
lineno = tb.tb_lineno
filename = f.f_code.co_filename
linecache.checkcache(filename)
line = linecache.getline(filename, lineno, f.f_globals)
print 'EXCEPTION IN ({}, LINE {} "{}"): {}'.format(filename, lineno, line.strip(), exc_obj)
try:
print 1/0
except:
PrintException()
Output:
EXCEPTION IN (D:/Projects/delme3.py, LINE 15 "print 1/0"): integer division or modulo by zero
To simply get the line number you can use sys, if you would like to have more, try the traceback module.
import sys
try:
[][2]
except IndexError:
print("Error on line {}".format(sys.exc_info()[-1].tb_lineno))
prints:
Error on line 3
Example from the traceback module documentation:
import sys, traceback
def lumberjack():
bright_side_of_death()
def bright_side_of_death():
return tuple()[0]
try:
lumberjack()
except IndexError:
exc_type, exc_value, exc_traceback = sys.exc_info()
print "*** print_tb:"
traceback.print_tb(exc_traceback, limit=1, file=sys.stdout)
print "*** print_exception:"
traceback.print_exception(exc_type, exc_value, exc_traceback,
limit=2, file=sys.stdout)
print "*** print_exc:"
traceback.print_exc()
print "*** format_exc, first and last line:"
formatted_lines = traceback.format_exc().splitlines()
print formatted_lines[0]
print formatted_lines[-1]
print "*** format_exception:"
print repr(traceback.format_exception(exc_type, exc_value,
exc_traceback))
print "*** extract_tb:"
print repr(traceback.extract_tb(exc_traceback))
print "*** format_tb:"
print repr(traceback.format_tb(exc_traceback))
print "*** tb_lineno:", exc_traceback.tb_lineno
I use the traceback which is simple and robust:
import traceback
try:
raise ValueError()
except:
print(traceback.format_exc()) # or: traceback.print_exc()
Out:
Traceback (most recent call last):
File "catch.py", line 4, in <module>
raise ValueError()
ValueError
The simplest way is just to use:
import traceback
try:
<blah>
except IndexError:
traceback.print_exc()
or if using logging:
import logging
try:
<blah>
except IndexError as e:
logging.exception(e)
Gives you file, lineno, and exception for the last item in the call stack
from sys import exc_info
from traceback import format_exception
def print_exception():
etype, value, tb = exc_info()
info, error = format_exception(etype, value, tb)[-2:]
print(f'Exception in:\n{info}\n{error}')
try:
1 / 0
except:
print_exception()
prints
Exception in:
File "file.py", line 12, in <module>
1 / 0
ZeroDivisionError: division by zero
I would suggest using the python logging library, it has two useful methods that might help in this case.
logging.findCaller()
findCaller(stack_info=False) - Reports just the line number for the previous caller leading to the exception raised
findCaller(stack_info=True) - Reports the line number & stack for the previous caller leading to the exception raised
logging.logException()
Reports the line & stack within the try/except block that raised the exception
For more info checkout the api https://docs.python.org/3/library/logging.html
There are many answers already posted here that show how to get the line number, but it's worth noting that if you want variables containing the "raw data," so to speak, of the stack trace so that you can have more granular control of what you display or how you format it, using the traceback module you can step through the stack frame by frame and look at what's stored in the attributes of the frame summary objects. There are several simple and elegant ways to manipulate the frame summary objects directly. Let's say for example that you want the line number from the last frame in the stack (which tells you which line of code triggered the exception), here's how you could get it by accessing the relevant frame summary object:
Option 1:
import sys
import traceback
try:
# code that raises an exception
except Exception as exc:
exc_type, exc_value, exc_tb = sys.exc_info()
stack_summary = traceback.extract_tb(exc_tb)
end = stack_summary[-1] # or `stack_summary.pop(-1)` if you prefer
Option 2:
import sys
import traceback
try:
# code that raises an exception
except Exception as exc:
tbe = traceback.TracebackException(*sys.exc_info())
end = tbe.stack[-1] # or `tbe.stack.pop(-1)` if you prefer
In either of the above examples, end will be a frame summary object:
&gt&gt&gt type(end)
&ltclass &#39traceback.FrameSummary&#39&gt
which was in turn taken from a stack summary object:
&gt&gt&gt type(stack_summary) # from option 1
&ltclass &#39traceback.StackSummary&#39&gt
&gt&gt&gt type(tbe&#46stack) # from option 2
&ltclass &#39traceback.StackSummary&#39&gt
The stack summary object behaves like a list and you can iterate through all of the frame summary objects in it however you want in order to trace through the error. The frame summary object (end, in this example), contains the line number and everything else you need to locate where in the code the exception occurred:
>>> print(end.__doc__)
A single frame from a traceback.
- :attr:`filename` The filename for the frame.
- :attr:`lineno` The line within filename for the frame that was
active when the frame was captured.
- :attr:`name` The name of the function or method that was executing
when the frame was captured.
- :attr:`line` The text from the linecache module for the
of code that was running when the frame was captured.
- :attr:`locals` Either None if locals were not supplied, or a dict
mapping the name to the repr() of the variable.
And if you capture the exception object (either from except Exception as exc: syntax or from the second object returned by sys.exc_info()), you will then have everything you need to write your own highly customized error printing/logging function:
err_type = type(exc).__name__
err_msg = str(exc)
Putting it all together:
from datetime import datetime
import sys
import traceback
def print_custom_error_message():
exc_type, exc_value, exc_tb = sys.exc_info()
stack_summary = traceback.extract_tb(exc_tb)
end = stack_summary[-1]
err_type = type(exc_value).__name__
err_msg = str(exc_value)
date = datetime.strftime(datetime.now(), "%B %d, %Y at precisely %I:%M %p")
print(f"On {date}, a {err_type} occured in {end.filename} inside {end.name} on line {end.lineno} with the error message: {err_msg}.")
print(f"The following line of code is responsible: {end.line!r}")
print("Please make a note of it.")
def do_something_wrong():
try:
1/0
except Exception as exc:
print_custom_error_message()
if __name__ == "__main__":
do_something_wrong()
Let's run it!
user#some_machine:~$ python example.py
On August 25, 2022 at precisely 01:31 AM, a ZeroDivisionError occured in example.py inside do_something_wrong on line 21 with the error message: division by zero.
The following line of code is responsible: '1/0'
Please make a note of it.
At this point you can see how you could print this message for any place in the stack: end, beginning, anywhere in-between, or iterate through and print it for every frame in the stack.
Of course, the formatting functionality already provided by the traceback module covers most debugging use cases, but it's useful to know how to manipulate the traceback objects to extract the information you want.
I always use this snippet
import sys, os
try:
raise NotImplementedError("No error")
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print(exc_type, fname, exc_tb.tb_lineno)
for different views and possible issues you can refer When I catch an exception, how do I get the type, file, and line number?
All the solutions answer the OPs problem, however, if one is holding onto a specific error instance and the last traceback stack won't do it they will not suffice —a corner scenario example given below.
In this case, the magic attribute __traceback__ of a raised exception instance may be used. This is a system traceback object (exposed as types.TracebackType, see Types) not the module. This traceback instance behaves just one like would expect (but it is always nice to check):
from collections import deque
from typing import (List, )
errors: List[Exception] = deque([], 5)
def get_raised(error_cls: type, msg: str) -> Exception:
# for debugging, an exception instance that is not raised
# ``__traceback__`` has ``None``
try:
raise error_cls(msg)
except Exception as error:
return error
error = get_raised(NameError, 'foo')
errors.append(error)
error = get_raised(ValueError, 'bar')
errors.append(error)
try:
raise get_raised(TypeError, 'bar')
except Exception as error:
errors.append(error)
Now, I can check out the first error's details:
import types
traceback = errors[0].__traceback__ # inadvisable name due to module
line_no: int = traceback.tb_lineno
frame: types.FrameType = traceback.tb_frame
previous: Union[type(None), types.TracebackType] = traceback.tb_next
filename: str = frame.f_code.co_filename
The traceback's previous is None for the second error despite a preceding error as expected, but for the third wherein an error is raised twice it is not.
This below is just a test which makes no sense contextually. A case were it is useful if when an exception is raised in a view of a webapp (a 500 status kind of incident) that gets caught and stored for the admit to inspect akin to Setry.io (but for free). Here is a minimal example where the home page / will raise an error, that gets caught and the route errors will list them. This is using Pyramid in a very concentrated way (multifile is way better) with no logging or authentication and the error logging could be better for the admin to inspect similar to Sentry.io.
from pyramid.config import Configurator
from waitress import serve
from collections import deque
# just for typehinting:
from pyramid.request import Request
from pyramid.traversal import DefaultRootFactory
from pyramid.router import Router
import types
from typing import (List, )
def home_view(context: DefaultRootFactory, request: Request) -> dict:
raise NotImplementedError('I forgot to fill this')
return {'status': 'ok'} # never reached.
def caught_view(error: Exception, request: Request) -> dict:
"""
Exception above is just type hinting.
This is controlled by the context argument in
either the ``add_exception_view`` method of config,
or the ``exception_view_config`` decorator factory (callable class)
"""
# this below is a simplification as URLDecodeError is an attack (418)
request.response.status = 500
config.registry.settings['error_buffer'].append(error)
#logging.exception(error) # were it set up.
#slack_admin(format_error(error)) # ditto
return {'status': 'error', 'message': 'The server crashed!!'}
def format_error(error: Exception) -> str:
traceback = error.__traceback__ # inadvisable name due to module
frame: types.FrameType = traceback.tb_frame
return f'{type(error).__name__}: {error}' +\
f'at line {traceback.tb_lineno} in file {frame.f_code.co_filename}'
def error_view(context: DefaultRootFactory, request: Request) -> dict:
print(request.registry.settings['error_buffer'])
return {'status': 'ok',
'errors':list(map(format_error, request.registry.settings['error_buffer']))
}
with Configurator(settings=dict()) as config:
config.add_route('home', '/')
config.add_route('errors', '/errors')
config.add_view(home_view, route_name='home', renderer='json')
config.add_view(error_view, route_name='errors', renderer='json')
config.add_exception_view(caught_view, context=Exception, renderer='json')
config.registry.settings['error_buffer']: List[Exception] = deque([], 5)
# not in config.registry.settings, not JSON serialisable
# config.add_request_method
app : Router = config.make_wsgi_app()
port = 6969
serve(app, port=port)

Identify the exception to be used from the traceback

I want to catch an exception when user fails login due to wrong password .
So i make a function using imaplib .I enter a wrong password and get a traceback with error details.
Now my question is actually general.How do you identify the exception we have to mention in our "try and except" body from the error messages?
These is what I got->
>>> count("testarc31#gmail.com","Xbox#36")
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
count("testarc31#gmail.com","Xbox#36")
File "E:\Arindam\py_progs\Mail Notifier\0.0.19\Mail.py", line 24, in count
obj.login(m,p)
File "C:\Python27\lib\imaplib.py", line 500, in login
raise self.error(dat[-1])
error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
If i want to make a try and except,what will i mention in the exception part?
try:
login(mail,pass):
except ????:
something
Question :
1) What will be ???? here . Can it be deduced directly from the error report?
2) Is there a basic idea to identify what is the exception we have to use from each error we get ?
You want to use something like this:
try:
..code that might raise an exception...
except ExceptionType, e:
...do something...
In your case, that probably want this:
try:
login(mail,pass)
except imaplib.IMAP4.error, e:
print "Ouch -- an error from imaplib!"
To identify the type of an exception, you can look at its exception message. In this case it's just "error" -- unfortunately the module name is not included. You can get a better idea of exactly where it comes from by doing:
try:
login(mail,pass)
except Exception, e:
print type(e)

Categories