Let's say I want to be able to log to file every time any exception is raised, anywhere in my program. I don't want to modify any existing code.
Of course, this could be generalized to being able to insert a hook every time an exception is raised.
Would the following code be considered safe for doing such a thing?
class MyException(Exception):
def my_hook(self):
print('---> my_hook() was called');
def __init__(self, *args, **kwargs):
global BackupException;
self.my_hook();
return BackupException.__init__(self, *args, **kwargs);
def main():
global BackupException;
global Exception;
BackupException = Exception;
Exception = MyException;
raise Exception('Contrived Exception');
if __name__ == '__main__':
main();
If you want to log uncaught exceptions, just use sys.excepthook.
I'm not sure I see the value of logging all raised exceptions, since lots of libraries will raise/catch exceptions internally for things you probably won't care about.
Your code as far as I can tell would not work.
__init__ has to return None and you are trying to return an instance of backup exception. In general if you would like to change what instance is returned when instantiating a class you should override __new__.
Unfortunately you can't change any of the attributes on the Exception class. If that was an option you could have changed Exception.__new__ and placed your hook there.
the "global Exception" trick will only work for code in the current module. Exception is a builtin and if you really want to change it globally you need to import __builtin__; __builtin__.Exception = MyException
Even if you changed __builtin__.Exception it will only affect future uses of Exception, subclasses that have already been defined will use the original Exception class and will be unaffected by your changes. You could loop over Exception.__subclasses__ and change the __bases__ for each one of them to insert your Exception subclass there.
There are subclasses of Exception that are also built-in types that you also cannot modify, although I'm not sure you would want to hook any of them (think StopIterration).
I think that the only decent way to do what you want is to patch the Python sources.
This code will not affect any exception classes that were created before the start of main, and most of the exceptions that happen will be of such kinds (KeyError, AttributeError, and so forth). And you can't really affect those "built-in exceptions" in the most important sense -- if anywhere in your code is e.g. a 1/0, the real ZeroDivisionError will be raised (by Python's own internals), not whatever else you may have bound to that exceptions' name.
So, I don't think your code can do what you want (despite all the semicolons, it's still supposed to be Python, right?) -- it could be done by patching the C sources for the Python runtime, essentially (e.g. by providing a hook potentially caught on any exception even if it's later caught) -- such a hook currently does not exist because the use cases for it would be pretty rare (for example, a StopIteration is always raised at the normal end of every for loop -- and caught, too; why on Earth would one want to trace that, and the many other routine uses of caught exceptions in the Python internals and standard library?!).
Download pypy and instrument it.
Related
I have a bit of code that tries to parse an object as an integer:
long val = PyLong_AsLong(obj);
if(val == -1 && PyErr_Occurred()) {
return -1;
}
Here obj is a vanilla PyObject *, and PyLong_AsLong raises a very generic TypeError if obj is not an integer.
I would like to transform the error message into something a bit more informative, so I would like to either modify the existing error object, or to reraise it.
My current solution is to do this:
long val = PyLong_AsLong(obj);
if(val == -1 && PyErr_Occurred()) {
PyErr_Clear();
PyErr_Format(PyExc_TypeError, "Parameter must be an integer type, but got %s", Py_TYPE(obj)->tp_name);
return -1;
}
Is this the proper way to reraise an error? Specifically,
Do I need to call PyErr_Clear at all? I suspect that it properly decrefs the existing exception object, but I'm not sure.
Can I modify the message of the error that has already been thrown at that point without re-raising it?
Is there an option to do the equivalent of raise new_err from old_err?
I am not sure how to use PyErr_SetExcInfo for this situation, although my gut tells me it may be relevant somehow.
Your existing code is fine, but if you want to do the equivalent of exception chaining, you can. If you want to skip to how to do that, jump to point 3 near the end of the answer.
To explain how to do things like modify a propagating exception or perform the equivalent of raise Something() from existing_exception, first, we'll have to explain how exception state works at C level.
A propagating exception is represented by a per-thread error indicator consisting of a type, value, and traceback. That sounds a lot like sys.exc_info(), but it's not the same. sys.exc_info() is for exceptions that have been caught by Python-level code, not exceptions that are still propagating.
The error indicator may be unnormalized, which basically means that the work of constructing an exception object hasn't been performed, and the value in the error indicator isn't an instance of the exception type. This state exists for efficiency; if the error indicator is cleared by PyErr_Clear before normalization is needed, Python gets to skip much of the work of raising an exception. Exception normalization is performed by PyErr_NormalizeException, with a bit of extra work in PyException_SetTraceback to set the exception object's __traceback__ attribute.
PyErr_Clear is sort of like the C equivalent of an except block, but it just clears the error indicator, without letting you inspect much of the exception information. To catch an exception and inspect it, you'd want PyErr_Fetch. PyErr_Fetch is like catching an exception and examining sys.exc_info(), but it doesn't set sys.exc_info() or normalize the exception. It clears the error indicator and gives you the raw contents of the error indicator directly.
Explicit exception chaining (raise Something() from existing_exception) works by going through PyException_SetCause to set the new exception's __cause__ to the existing exception. This requires exception objects for both exceptions, so if you want to do the equivalent from C, you'll have to normalize the exceptions and call PyException_SetCause yourself.
Implicit exception chaining (raise Something() in an except block) works by going through PyException_SetContext to set the new exception's __context__ to the existing exception. Similar to PyException_SetCause, this requires exception objects and exception normalization. raise Something() from existing_exception inside an except block actually sets both __cause__ and __context__, and if you want to perform explicit exception chaining at C level, you should usually do the same.
Technically not necessary, as far as I can tell, but it's probably a good idea to do it anyway. It looks like PyErr_Format and other functions that set the error indicator will clear the error indicator first if it's already set, but this isn't documented for most of them.
Sort of, but it's probably a bad idea. You can normalize the error indicator and set the exception object's message attribute, but this won't affect args or anything else the exception class might do with its arguments, and that could lead to weird problems. Alternatively, you could fetch the error indicator with PyErr_Fetch and restore it with a new string for the value with PyErr_Restore, but that will throw away an existing exception object if there is one, and it makes assumptions about the exception class's signature.
Yeah, that's possible, but doing it through public C API functions is pretty awkward and manual. You'd have to manually do a lot of normalization, unraising, and raising exceptions.
There are efforts to make C-level exception chaining more convenient, but so far, the more convenient functions are all considered internal. For example, _PyErr_FormatFromCause is like PyErr_Format, but it chains the new exception off of an existing, propagating exception (through both __context__ and __cause__.
I wouldn't recommend calling it directly for now; it's very new (3.6+), and it's very likely to change (specifically, I would be unsurprised to see it lose its leading underscore in a new Python version). Instead, copying the implementation of _PyErr_FormatFromCause/_PyErr_FormatVFromCause (and respecting the license) is a good way to make sure you have the fiddly bits of normalization and chaining right.
It's also a useful reference to work from if you want to perform implicit (__context__-only) exception chaining at C level - just remove the part that handles __cause__.
I am stuck trying to write a test case that will check whether the code goes inside of an except block.
My method foo() in case of an exception doesn't throw/raise it, it just logs information.
I have tried to use assertRaises but later I realized that this is not working for me because I am not raising an exception.
In Python docs it is clearly said that:
Test that an exception is raised when callable is called with any positional or keyword arguments that are also passed to assertRaises(). The test passes if exception is raised, is an error if another exception is raised, or fails if no exception is raised.
So, if I have following method:
def foo():
try:
# Something that will cause an exception
except AttributeError:
log.msg("Shit happens")
is it possible to write a test case that will test whether execution goes inside of an except block?
You can't do this the way you want to. Python raises and handles exceptions all over the place—e.g., every single for loop exits by raising and handling a StopIteration. So, an assert that there was an exception somewhere, even if it was handled, would pretty much always pass.
What you can do is mock the logger, like this:
_logs = []
def mocklog(str):
_logs.append(str)
mymodule.log = mocklog
mymodule.foo()
assertEqual(_logs, ['Shit happens'])
Of course in a real-life project, you probably want to use a mocking library instead of hacking it in by hand like this, but that should demonstrate the idea.
You can use assertRaises (https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises) like this:
with self.assertRaises(Exception):
foo()
There is also a method assertLogs if you want to test logging.
I would like to handle a NameError exception by injecting the desired missing variable into the frame and then continue the execution from last attempted instruction.
The following pseudo-code should illustrate my needs.
def function():
return missing_var
try:
print function()
except NameError:
frame = inspect.trace()[-1][0]
# inject missing variable
frame.f_globals["missing_var"] = ...
# continue frame execution from last attempted instruction
exec frame.f_code from frame.f_lasti
Read the whole unittest on repl.it
Notes
As pointed out by ivan_pozdeev in his answer, this is known as resumption.
After more research, I found Veedrac's answer to the question Resuming program at line number in the context before an exception using a custom sys.excepthook posted by lc2817 very interesting. It relies on Richie Hindle's work.
Background
The code runs in a slave process, which is controlled by a parent. Tasks (functions really) are written in the parent and latter passed to the slave using dill. I expect some tasks (running in the slave process) to try to access variables from outer scopes in the parent and I'd like the slave to request those variables to the parent on the fly.
p.s.: I don't expect this magic to run in a production environment.
On the contrary to what various commenters are saying, "resume-on-error" exception handling is possible in Python. The library fuckit.py implements said strategy. It steamrollers errors by rewriting the source code of your module at import time, inserting try...except blocks around every statement and swallowing all exceptions. So perhaps you could try a similar sort of tactic?
It goes without saying: that library is intended as a joke. Don't ever use it in production code.
You mentioned that your use case is to trap references to missing names. Have you thought about using metaprogramming to run your code in the context of a "smart" namespace such as a defaultdict? (This is perhaps only marginally less of a bad idea than fuckit.py.)
from collections import defaultdict
class NoMissingNamesMeta(type):
#classmethod
def __prepare__(meta, name, bases):
return defaultdict(lambda: "foo")
class MyClass(metaclass=NoMissingNamesMeta):
x = y + "bar" # y doesn't exist
>>> MyClass.x
'foobar'
NoMissingNamesMeta is a metaclass - a language construct for customising the behaviour of the class statement. Here we're using the __prepare__ method to customise the dictionary which will be used as the class's namespace during creation of the class. Thus, because we're using a defaultdict instead of a regular dictionary, a class whose metaclass is NoMissingNamesMeta will never get a NameError. Any names referred to during the creation of the class will be auto-initialised to "foo".
This approach is similar to #AndréFratelli's idea of manually requesting the lazily-initialised data from a Scope object. In production I'd do that, not this. The metaclass version requires less typing to write the client code, but at the expense of a lot more magic. (Imagine yourself debugging this code in two years, trying to understand why non-existent variables are dynamically being brought into scope!)
The "resumption" exception handling technique has proven to be problematic, that's why it's missing from C++ and later languages.
Your best bet is to use a while loop to not resume where the exception was thrown but rather repeat from a predetermined place:
while True:
try:
do_something()
except NameError as e:
handle_error()
else:
break
You really can't unwind the stack after an exception is thrown, so you'd have to deal with the issue before hand. If your requirement is to generate these variables on the fly (which wouldn't be recommended, but you seem to understand that), then you'd have to actually request them. You can implement a mechanism for that (such as having a global custom Scope class instance and overriding __getitem__, or using something like the __dir__ function), but not as you are asking for it.
I would like to write a function in Python (2.6) that can determine if it is being called from exception handling code somewhere up the stack.
This is for a specialized logging use. In python's logging module, the caller has to explicitly specify that exception information should be logged (either by calling logger.exception() or by using the exc_info keyword). I would like my logger to do this automatically, based on whether it is being called from within exception handling code.
I thought that checking sys.exc_info() might be the answer, but it also returns exception information from an already-handled exception. (From the docs: "This function returns a tuple of three values that give information about the exception that is currently being handled... If the current stack frame is not handling an exception, the information is taken from the calling stack frame, or its caller, and so on until a stack frame is found that is handling an exception. Here, 'handling an exception' is defined as 'executing or having executed an except clause.'")
Also, since I want this to be transparent to the caller, I do not want to have to use exc_clear() or anything else in the except clause.
What's the right way to do this?
If you clear the exception using sys.exc_clear in your exception handlers, then sys.exc_info should work for you. For example: If you run the following script:
import sys
try:
1 / 0
except:
print sys.exc_info()
sys.exc_clear()
print sys.exc_info()
You should see this output:
(, ZeroDivisionError('integer division or modulo by zero',), )
(None, None, None)
Update: I don't believe there is a simple ("transparent") way of answering the question "Is an exception handler running?" without going to some trouble, and in my opinion it's not worth taking the trouble just for logging. It is of course easy to answer the question "Has an exception been raised (in this thread)?", even on a per-stack-frame basis (see the documentation for frame objects).
Like everything in Python, an exception is an object. Therefore, you could keep a (weak!) reference to the last exception handled and then use sys.exc_info().
Note: in case of multithreading code, you may have issues with this approach. And there could be other corner cases as well.
However, explicit is better than implicit; are you really sure that handling exception logging in the same way as normal one is a good feature to add to your system?
In my humble opinion, not.
Is there a way knowing (at coding time) which exceptions to expect when executing python code?
I end up catching the base Exception class 90% of the time since I don't know which exception type might be thrown (reading the documentation doesn't always help, since many times an exception can be propagated from the deep. And many times the documentation is not updated or correct).
Is there some kind of tool to check this (like by reading the Python code and libs)?
I guess a solution could be only imprecise because of lack of static typing rules.
I'm not aware of some tool that checks exceptions, but you could come up with your own tool matching your needs (a good chance to play a little with static analysis).
As a first attempt, you could write a function that builds an AST, finds all Raise nodes, and then tries to figure out common patterns of raising exceptions (e. g. calling a constructor directly)
Let x be the following program:
x = '''\
if f(x):
raise IOError(errno.ENOENT, 'not found')
else:
e = g(x)
raise e
'''
Build the AST using the compiler package:
tree = compiler.parse(x)
Then define a Raise visitor class:
class RaiseVisitor(object):
def __init__(self):
self.nodes = []
def visitRaise(self, n):
self.nodes.append(n)
And walk the AST collecting Raise nodes:
v = RaiseVisitor()
compiler.walk(tree, v)
>>> print v.nodes
[
Raise(
CallFunc(
Name('IOError'),
[Getattr(Name('errno'), 'ENOENT'), Const('not found')],
None, None),
None, None),
Raise(Name('e'), None, None),
]
You may continue by resolving symbols using compiler symbol tables, analyzing data dependencies, etc. Or you may just deduce, that CallFunc(Name('IOError'), ...) "should definitely mean raising IOError", which is quite OK for quick practical results :)
You should only catch exceptions that you will handle.
Catching all exceptions by their concrete types is nonsense. You should catch specific exceptions you can and will handle. For other exceptions, you may write a generic catch that catches "base Exception", logs it (use str() function) and terminates your program (or does something else that's appropriate in a crashy situation).
If you really gonna handle all exceptions and are sure none of them are fatal (for example, if you're running the code in some kind of a sandboxed environment), then your approach of catching generic BaseException fits your aims.
You might be also interested in language exception reference, not a reference for the library you're using.
If the library reference is really poor and it doesn't re-throw its own exceptions when catching system ones, the only useful approach is to run tests (maybe add it to test suite, because if something is undocumented, it may change!). Delete a file crucial for your code and check what exception is being thrown. Supply too much data and check what error it yields.
You will have to run tests anyway, since, even if the method of getting the exceptions by source code existed, it wouldn't give you any idea how you should handle any of those. Maybe you should be showing error message "File needful.txt is not found!" when you catch IndexError? Only test can tell.
The correct tool to solve this problem is unittests. If you are having exceptions raised by real code that the unittests do not raise, then you need more unittests.
Consider this
def f(duck):
try:
duck.quack()
except ??? could be anything
duck can be any object
Obviously you can have an AttributeError if duck has no quack, a TypeError if duck has a quack but it is not callable. You have no idea what duck.quack() might raise though, maybe even a DuckError or something
Now supposing you have code like this
arr[i] = get_something_from_database()
If it raises an IndexError you don't know whether it has come from arr[i] or from deep inside the database function. usually it doesn't matter so much where the exception occurred, rather that something went wrong and what you wanted to happen didn't happen.
A handy technique is to catch and maybe reraise the exception like this
except Exception as e
#inspect e, decide what to do
raise
Noone explained so far, why you can't have a full, 100% correct list of exceptions, so I thought it's worth commenting on. One of the reasons is a first-class function. Let's say that you have a function like this:
def apl(f,arg):
return f(arg)
Now apl can raise any exception that f raises. While there are not many functions like that in the core library, anything that uses list comprehension with custom filters, map, reduce, etc. are affected.
The documentation and the source analysers are the only "serious" sources of information here. Just keep in mind what they cannot do.
I ran into this when using socket, I wanted to find out all the error conditions I would run in to (so rather than trying to create errors and figure out what socket does I just wanted a concise list). Ultimately I ended up grep'ing "/usr/lib64/python2.4/test/test_socket.py" for "raise":
$ grep raise test_socket.py
Any exceptions raised by the clients during their tests
raise TypeError, "test_func must be a callable function"
raise NotImplementedError, "clientSetUp must be implemented."
def raise_error(*args, **kwargs):
raise socket.error
def raise_herror(*args, **kwargs):
raise socket.herror
def raise_gaierror(*args, **kwargs):
raise socket.gaierror
self.failUnlessRaises(socket.error, raise_error,
self.failUnlessRaises(socket.error, raise_herror,
self.failUnlessRaises(socket.error, raise_gaierror,
raise socket.error
# Check that setting it to an invalid value raises ValueError
# Check that setting it to an invalid type raises TypeError
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
Which is a pretty concise list of errors. Now of course this only works on a case by case basis and depends on the tests being accurate (which they usually are). Otherwise you need to pretty much catch all exceptions, log them and dissect them and figure out how to handle them (which with unit testing wouldn't be to difficult).
There are two ways that I found informative. The first one, run the code in iPython, which will display the exception type.
n = 2
str = 'me '
str + 2
TypeError: unsupported operand type(s) for +: 'int' and 'str'
In the second way we settle for catching too much and improve on it over time. Include a try expression in your code and catch except Exception as err. Print sufficient data to know what exception was thrown. As exceptions are thrown improve your code by adding a more precise except clause. When you feel that you have caught all relevant exceptions remove the all inclusive one. A good thing to do anyway because it swallows programming errors.
try:
so something
except Exception as err:
print "Some message"
print err.__class__
print err
exit(1)
normally, you'd need to catch exception only around a few lines of code. You wouldn't want to put your whole main function into the try except clause. for every few line you always should now (or be able easily to check) what kind of exception might be raised.
docs have an exhaustive list of built-in exceptions. don't try to except those exception that you're not expecting, they might be handled/expected in the calling code.
edit: what might be thrown depends on obviously on what you're doing! accessing random element of a sequence: IndexError, random element of a dict: KeyError, etc.
Just try to run those few lines in IDLE and cause an exception. But unittest would be a better solution, naturally.
This is a copy and pasted answer I wrote for How to list all exceptions a function could raise in Python 3?, I hope that is allowed.
I needed to do something similar and found this post. I decided I
would write a little library to help.
Say hello to Deep-AST. It's very early alpha but it is pip
installable. It has all of the limitations mentioned in this post
and some additional ones but its already off to a really good start.
For example when parsing HTTPConnection.getresponse() from
http.client it parses 24489 AST Nodes. It finds 181 total raised
Exceptions (this includes duplicates) and 8 unique Exceptions were
raised. A working code example.
The biggest flaw is this it currently does work with a bare raise:
def foo():
try:
bar()
except TypeError:
raise
But I think this will be easy to solve and I plan on fixing it.
The library can handle more than just figuring out exceptions, what
about listing all Parent classes? It can handle that too!