Related
I have a c++ class with several constructors:
MyClass(const std::string& configfilename);
MyClass(const MyClass& other);
I have python bindings for this class that were generated with cppyy - I don't do this myself, this is all part of the framework I'm using (CERN ROOT, in case you're wondering).
Now, I have a piece of python code that instantiates my class, with a nice try-except block:
try:
obj = MyClass(args.config)
except ConfigFileNotFoundError:
print("config file "+args.config+" was not found!")
exit(0)
Now, to test, I'm executing this with a wrong config file. But what I get is roughly this:
TypeError: none of the 2 overloaded methods succeeded. Full details:
MyClass(const std::string&) => ConfigFileNotFoundError
MyClass::MyClass(const MyClass&) => TypeError
So I'm wondering:
Since cppyy seems to handle function overloading with a try/except block, is there any reasonable way to do error handling for such applications?
I'd love to actually get the ConfigFileNotFoundError to handle it properly, rather than getting this TypeError. Also, what determines the actual error class I get in the end - does it depend on the order in which the overloads appear in the header file?
Any help, suggestions or pointers on where to find more information on this would be highly appreciated.
cppyy doesn't use try/except for overload resolution, hence there are also no __context__ and __cause__ set. To be more precise: the C++ exception is not an error that occurs during a handler. Rather, as-yet unresolved overloads are prioritized, then tried in order, with no distinction made between a Python failure (e.g. from an argument conversion) or a C++ failure (any exception that was automatically converted into a Python exception). This is a historic artifact predating run-time template instantiation and SFINAE: it allowed for more detailed run-time type matching in pre-instantiated templates.
If all overloads fail (Python or C++), the collected errors are summarized. Python still requires an exception type, however, and if the exception types across the collected types differ, a generic TypeError is raised, with a message string made up of all the collected exceptions. This is what happens here: there is ConfigFileNotFoundError raised by C++ in one overload and TypeError from argument conversion failure in the other.
There's an improvement now in the cppyy repo; to be released with 2.3.0, where in clear cases such as this one (a single overload succeeding in argument match but failing in the callee), you'll get the actual ConfigFileNotFoundError instance as long as its class is publicly derived from std::exception (but I think it already does, otherwise the error report you posted would have looked quite different).
(Note that CERN's ROOT contains an old fork of cppyy that has quite a bit diverged; you'll have to request them for a separate update there if that fork matters to you.)
I'm developing a Jinja extension that performs a few potentially harmful operations, which on failure will raise an Exception that holds some information on what went wrong.
When this occurs, of course the Exception will prevent Jinja from completing its render process and return None rather than a render result. This problem is easily mitigated using a try/catch statement and returning an empty string or whatever is suitable.
However, that effectively throws away the Exception and the debugging info, which I would rather pass on to my error log. The system responsible for setting up the Jinja environment has it's own logging service, which I would prefer to keep decoupled from the extension. I am aware though, that Jinja has an Undefined class which I have successfully used to intercept access of undefined variables in another project.
Is there any way I could raise a special type of exception (UndefinedException did not work), or tell the Jinja environment to log a warning, when an error occurs in my extension, while still allowing it to continue its execution?
What I have in mind is something along the lines of this example:
def _render(*args, **kwrags):
# ...
try:
self.potentially_harmful_operation()
except Exception as e:
self.environment.warning(e)
return Markup("")
# ...
return Markup('<img src"{}">'.format(img_path))
So, to clarify, as I wrote in a comment below:
Fundamentally I guess my question boils down to "how can I make Jinja produce a warning, as it does for e.g. missing variables".
For anyone interested, I managed to solve this by implementing my own Undefined class (largely inspired by make_logging_undefined and supplying that to the environment using the undefined keyword. Depending on your needs, the default make_logging_undefined might very well suffice.
Once that's in place, the trick is to emit the warning in the environment. This is done by simply instantiating the environment's Undefined and executing one of its magic methods (e.g. __str__). Hooking on to my example code from above, that could be accomplished like this:
def _render(*args, **kwrags):
# ...
try:
self.potentially_harmful_operation()
except Exception as e:
str(self.environment.undefined(e))
return Markup("")
# ...
return Markup('<img src"{}">'.format(img_path))
I'm currently trying to write code to catch exceptions, and depending upon which exception is thrown, a different module will be imported than when no exception was thrown.
try:
import sge_execution_engine as execution_engine
except ImportError:
print "Use local execution engine because SGE support is missing!"
print sys.exc_info() # Print out the exception message
import local_execution_engine as execution_engine
except RuntimeError:
print "Using local execution engine because SGE support is missing!"
print sys.exc_info()
import local_execution_engine as execution_engine
The first exception, ImportError that is caught, catches the exception thrown when python drmaa module cannot be found during the execution of import sge_execution_engine (inside sge_execution_engine, there is an import drmaa statement). The second exception, RuntimeError, is caught when the drmaa python library is found (likewsie during the execution of the import drmaa statement inside the sge_execution_engine), but the drmaa C library is not installed into the OS. We hope that these two except statements are sufficient to catch all possible exceptions that can be thrown when a user attempts to run this module on a machine that just does not have the python drmaa library, the drmaa C library, or does not have Sun Grid Engine installed. without any of these proceeds, the module proceeds to then import local_execution_engine and so the code can then execute on the user's machine locally. Right now the code works as expected in the sense that it goes to import local when it finds exceptions with sge, but we are still looking to improve the exception handling here to make it more robust.
In my opinion I think having the actual Exception message that was thrown be printed to stdout is good as it will allow the user to know why he was unable to import sge_execution_engine especially if he was not expecting it to fail being imported.
However, instead of using print sys.exc_info() to actually have the actual exception message be printed on screen, I realized that perhaps a better way would be to use the except EXCEPTION as some_variable_name format and then print out print some_variable_name and also call some of the attributes associated with the Exception that is thrown and assigned to some_variable_name.
I saw this being done in the Python tutorial on exceptions where there was this chunk of code:
import sys
try:
f = open('myfile.txt')
s = f.readline()
i = int(s.strip())
except IOError as e:
print "I/O error({0}): {1}".format(e.errno, e.strerror)
except ValueError:
print "Could not convert data to an integer."
except:
print "Unexpected error:", sys.exc_info()[0]
raise
It seems like the except IOError as e chunk is handling the exception message in a fine-grained way by specifically calling on the errno and strerror attributes of the IOError object. However, when I look at the IOError documentation , I do not see these specific attributes being listed out as part of the documentation for the exception. In fact, this is also the case for all the other exceptions under the Python documentation, so it seems there is no way we can figure out what attributes will be associated with a particular exception. If we don't know anything about this, then how will we be able to figure out what attributes to call on the some_variable_name object when we use the import EXCEPTION as some_variable_name syntax to handle our exceptions?
I would appreciate anyone's suggestion on this, and even if your answer is not directly answering my question, but if you have another entirely different suggestion on how I could better handle my exception here, please don't hesitate to make a post!
Thank you very much!
Using a bare except is rarely a good idea, but this is one of those rare times -- primarily because you have a backup that you want to use if, for any reason, you cannot import the system sge:
try:
import sge_execution_engine as execution_engine
except:
print "Use local execution engine because SGE support is missing!"
print sys.exc_info() # Print out the exception message
import local_execution_engine as execution_engine
Notice you now only need one except clause.
The better way to handle getting that information to the user is probably to go ahead and print it out, and then also use the logging module to make a permanent record:
import logging
logger = logging.getLogger()
logger.setLevel(logging.WARNING)
and then in your except clause add:
logger.exception('unable to import sge')
and your message, along with the actual exception, will be saved in a log file.
First you're right that it's better to catch the exception into a variable than to ignore it and then pull it back out with sys.exc_info(). There are a few good reasons to use exc_info (low-level code, code that has to work with both pre-2.6 and 3.x, etc.), but in general, when you get can do it your way, you should.
And this works even for your last bare except. In 2.7, a plain except: means the same thing as except Exception:, so you can write except Exception as e:, and then use the e value.
Also note that if you want to do exactly the same thing for multiple exception types, you can write except (RuntimeError, ImportError) as e:.
As for "making it more robust", that's not an absolute thing. For example, if there's some unexpected exception that's neither a RuntimeError nor an ImportError, do you want to log it and try the fallback code, or quite and dump the traceback? If it's, e.g., a SyntaxError caused by someone checking in a bad edit to your program, you may not want to treat that like a runtime error… or maybe you do; that really depends on your development practices and target userbase.
Meanwhile:
It seems like the except IOError as e chunk is handling the exception message in a fine-grained way by specifically calling on the errno and strerror attributes of the IOError object. However, when I look at the IOError documentation, I do not see these specific attributes being listed out as part of the documentation for the exception.
You need to look up the hierarchy. Notice that IOError is a subclass of EnvironmentError, which does document the errno and strerror attributes. (What these attributes actually mean is only documneted for OSError and its subclasses, but the fact that they exist is documented.)
If you think this is all a bit of a mess… well, it is. It's all cleaned up in Python 3.x, where IOError and EnvironmentError are merged into OSError, which clearly documents its attributes, and where you usually don't have to switch on errno in the first place because common errno values generate a specific subclass like FileNotFoundError, and so on. But as long as you're using 2.7, you don't get the benefits of the last 6 years of improvements to the language.
For example, looking at the hierarchy or ValueError=>StandardError=>Exception (from lowest to highest in the hierarchy), I can't find any attributes about it.
If you dir a ValueError, you'll see that it only has two attributes (besides the usual special stuff like __repr__ and __class_): args and message.
message isn't documented because it was deprecated in 2.5, and only exists in 2.7 to allow some pre-2.5 code to keep running.* But args is documented; you just need to go up one level further, to BaseException:
args
The tuple of arguments given to the exception constructor. Some built-in exceptions (like IOError) expect a certain number of arguments and assign a special meaning to the elements of this tuple, while others are usually called only with a single string giving an error message.
So, the reason you can't find the other attributes in ValueError is that there are no other attributes to find. And the same goes for those other classes. The handful of types that have special attributes (OSError, SyntaxError, maybe a few module-specific types elsewhere in the stdlib) document them explicitly.**
If we are using the except some_exception as e syntax, is doing a print e sufficient to get the exception printed out without calling its attributes
It's sufficient to get some useful form of the exception printed. Again, from the BaseException docs:
If str() or unicode() is called on an instance of this class, the representation of the argument(s) to the instance are returned or the empty string when there were no arguments.
In some cases, that's not what you want. In particular, notice that it doesn't include the type of the exception. You can get that with repr, which gives you something that looks like a constructor call (e.g., ValueError("invalid literal for int() with base 10: 'f'")).
If you want the same output you get in a traceback, you have to put the type and the str or unicode together yourself—e.g., '{}: {}'.format(type(e), e).
If you want to get the actual information out of the exception, like that base 10 or that string 'f'—well, you can't,*** because that information has already been thrown away. You'll have to write your own code to keep track of it, like:
try:
i = int(s, 16)
except ValueError as e:
print '{} is not a base-16 number'.format(s)
* It's as if BaseException.__init__(self, *args) were defined to do self.args = tuple(args); self.message = str(args[0]) if args else ''.
** I believe in 2.5 there were a few exception types that had undocumented attributes as an implementation detail of CPython, but by 2.7 they're all either gone or documented.
*** Well, you could parse the exception string, but needless to say, that's an implementation detail, not something guaranteed to be stable and portable. It could be different in Jython, or on a Spanish-language system, or it may not quote strings the way you expect, etc.
Sorry for the long title, but it seems most descriptive for my question.
Basically, I'm having a difficult time finding exception information in the official python documentation. For example, in one program I'm currently writing, I'm using the shutil libary's move function:
from shutil import move
move('somefile.txt', '/tmp/somefile.txt')
That works fine, as long as I have write access to /tmp/, there is enough diskspace, and if all other requirements are satisfied.
However, when writing generic code, it is often difficult to guarantee those factors, so one usually uses exceptions:
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except:
print 'Move failed for some reason.'
I'd like to actually catch the appropriate exceptions thrown instead of just catching everything, but I simply can't find a list of exceptions thrown for most python modules. Is there a way for me to see which exceptions a given function can throw, and why? This way I can make appropriate cases for each exception, eg:
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except PermissionDenied:
print 'No permission.'
except DestinationDoesNotExist:
print "/tmp/ doesn't exist"
except NoDiskSpace:
print 'No diskspace available.'
Answer points go to whoever can either link me to some relevant documentation that I've somehow overlooked in the official docs, or provide a sure-fire way to figure out exactly which exceptions are thrown by which functions, and why.
Thanks!
UPDATE: It seems from the answers given that there really isn't a 100% straight-forward way to figure out which errors are thrown by specific functions. With meta programming, it seems that I can figure out simple cases and list some exceptions, but this is not a particularly useful or convenient method.
I'd like to think that eventually there will be some standard for defining which exceptions are raised by each python function, and that this information will be included in the official documentation. Until then I think I will just allow those exceptions to pass through and error out for my users as it seems like the most sane thing to do.
To amplify Messa, catch what you expect are failure modes that you know how to recover from. Ian Bicking wrote an article that addresses some of the overarching principles as does Eli Bendersky's note.
The problem with the sample code is that it is not handling errors, just prettifying them and discarding them. Your code does not "know" what to do with a NameError and there isn't much it should do other than pass it up, look at Bicking's re-raise if you feel you must add detail.
IOError and OSError are reasonably "expectable" for a shutil.move but not necessarily handleable. And the caller of your function wanted it to move a file and may itself break if that "contract" that Eli writes of is broken.
Catch what you can fix, adorn and re-raise what you expect but can't fix, and let the caller deal with what you didn't expect, even if the code that "deals" is seven levels up the stack in main.
Python doesn't have a mechanism right now for declaring which exceptions are thrown, unlike (for example) Java. (In Java you have to define exactly which exceptions are thrown by what, and if one of your utility methods needs to throw another exception then you need to add it to all of the methods which call it which gets boring quickly!)
So if you want to discover exactly which exceptions are thrown by any given bit of python then you need to examine the documentation and the source.
However python has a really good exception hierarchy.
If you study the exception hierarchy below you'll see that the error superclass you want to catch is called StandardError - this should catch all the errors that might be generated in normal operations. Turning the error into into a string will give a reasonable idea to the user as to what went wrong, so I'd suggest your code above should look like
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except StandardError, e:
print 'Move failed: %s' % e
Exception hierarchy
BaseException
|---Exception
|---|---StandardError
|---|---|---ArithmeticError
|---|---|---|---FloatingPointError
|---|---|---|---OverflowError
|---|---|---|---ZeroDivisionError
|---|---|---AssertionError
|---|---|---AttributeError
|---|---|---BufferError
|---|---|---EOFError
|---|---|---EnvironmentError
|---|---|---|---IOError
|---|---|---|---OSError
|---|---|---ImportError
|---|---|---LookupError
|---|---|---|---IndexError
|---|---|---|---KeyError
|---|---|---MemoryError
|---|---|---NameError
|---|---|---|---UnboundLocalError
|---|---|---ReferenceError
|---|---|---RuntimeError
|---|---|---|---NotImplementedError
|---|---|---SyntaxError
|---|---|---|---IndentationError
|---|---|---|---|---TabError
|---|---|---SystemError
|---|---|---TypeError
|---|---|---ValueError
|---|---|---|---UnicodeError
|---|---|---|---|---UnicodeDecodeError
|---|---|---|---|---UnicodeEncodeError
|---|---|---|---|---UnicodeTranslateError
|---|---StopIteration
|---|---Warning
|---|---|---BytesWarning
|---|---|---DeprecationWarning
|---|---|---FutureWarning
|---|---|---ImportWarning
|---|---|---PendingDeprecationWarning
|---|---|---RuntimeWarning
|---|---|---SyntaxWarning
|---|---|---UnicodeWarning
|---|---|---UserWarning
|---GeneratorExit
|---KeyboardInterrupt
|---SystemExit
This also means that when defining your own exceptions you should base them off StandardError not Exception.
Base class for all standard Python exceptions that do not represent
interpreter exiting.
Yes, you can (for simple cases), but you need a bit of meta-programming. Like the other answers have said, a function does not declare that it throws a particular error type, so you need to look at the module and see what exception types it defines, or what exception types it raises. You can either try to grok the documentation or leverage the Python API to do this.
To first find which exception types a module defines, just write a simple script to go through each object in the module dictionary module.__dict__ and see if it ends in the word "Error" or if it is a subclass of Exception:
def listexns(mod):
"""Saved as: http://gist.github.com/402861
"""
module = __import__(mod)
exns = []
for name in module.__dict__:
if (issubclass(module.__dict__[name], Exception) or
name.endswith('Error')):
exns.append(name)
for name in exns:
print '%s.%s is an exception type' % (str(mod), name)
return
If I run this on your example of shutils I get this:
$ python listexn.py shutil
Looking for exception types in module: shutil
shutil.Error is an exception type
shutil.WindowsError is an exception type
$
That tells you which error types are defined, but not which ones are thrown. To achieve the latter, we need to walk over the abstract syntax tree generated when the Python interpreter parses the module, and look for every raise statement, then save a list of names which are raised. The code for this is a little long, so first I'll state the output:
$ python listexn-raised.py /usr/lib/python2.6/shutil.py
Looking for exception types in: /usr/lib/python2.6/shutil.py
/usr/lib/python2.6/shutil.py:OSError is an exception type
/usr/lib/python2.6/shutil.py:Error is an exception type
$
So, now we know that shutil.py defines the error types Error and WindowsError and raises the exception types OSError and Error. If we want to be a bit more complete, we could write another method to check every except clause to also see which exceptions shutil handles.
Here's the code to walk over the AST, it just uses the compiler.visitor interface to create a walker which implements the "visitor pattern" from the Gang of Four book:
class ExceptionFinder(visitor.ASTVisitor):
"""List all exceptions raised by a module.
Saved as: http://gist.github.com/402869
"""
def __init__(self, filename):
visitor.ASTVisitor.__init__(self)
self.filename = filename
self.exns = set()
return
def __visitName(self, node):
"""Should not be called by generic visit, otherwise every name
will be reported as an exception type.
"""
self.exns.add(node.name)
return
def __visitCallFunc(self, node):
"""Should not be called by generic visit, otherwise every name
will be reported as an exception type.
"""
self.__visitName(node.node)
return
def visitRaise(self, node):
"""Visit a raise statement.
Cheat the default dispatcher.
"""
if issubclass(node.expr1, compiler.ast.Name):
self.__visitName(node.expr1)
elif isinstance(node.expr1, compiler.ast.CallFunc):
self.__visitCallFunc(node.expr1)
return
As these operations usually use libc functions and operating system calls, mostly you get IOError or OSError with an errno number; these errors are listed in man pages of that libc/OS calls.
I know this is possibly not a complete answer, it would be good to have all exceptions listed in documentation...
Note: i edited my Q (in the title) so that it better reflects what i actually want to know. In the original title and in the text of my Q, i referred to the source of the thrown exception; what i meant, and what i should have referred to, as pointed out in one of the high-strung but otherwise helpful response below, is the module that the exception class is defined in. This is evidenced by the fact that, again, as pointed out in one of the answers below the answer to the original Q is that the exceptions were thrown from calls to cursor.execute and cursor.next, respectively--which of course, isn't the information you need to write the try/except block.
For instance (the Q has nothing specifically to do with SQLite or the PySQLite module):
from pysqlite2 import dbapi2 as SQ
try:
cursor.execute('CREATE TABLE pname (id INTEGER PRIMARY KEY, name VARCHARS(50)')
except SQ.OperationalError:
print("{0}, {1}".format("table already exists", "... 'CREATE' ignored"))
#
cursor.execute('SELECT * FROM pname')
while 1:
try:
print(cursor.next())
except StopIteration:
break
#
i let both snippets error out to see the exception thrown, then coded the try/finally blocks--but that didn't tell me anything about which module the exception class is defined. In my example, there's only a single imported module, but where there are many more, i am interested to know how an experienced pythonista identifies the exception source (search-the-docs-till-i-happen-to-find-it is my current method).
[And yes i am aware there's a nearly identical question on SO--but for C# rather than python, plus if you read the author's edited version, you'll see he's got a different problem in mind.]
the second [[exception was thrown]] from a python core module
False: it was thrown from a call to cursor.next, exactly like the first one was thrown from a call to cursor.execute -- it's hard to say why you're baldly asserting this counterfactual, but contrary to fact it nevertheless remains.
If you're speaking about, what module an exception class was defined in, as opposed to, as you say, where it was thrown from, that's a completely different thing of course:
try:
...whatever...
except Exception, e:
print "caught an exception defined in module", e.__class__.__module__
Built-in exceptions are actually defined in module exceptions, as this code will also tell you. Of course, once you have the module name (which this snippet gives you), you can further explore, if you wish, e.g. by obtaining the module object (just index sys.modules by module name), etc.
Usually, I just read the documentation. Part of the documentation for a Python module or function is a list of the exceptions it defines. If it doesn't list anything, I just assume I'm looking for a Python builtin exception.
By the way, the StopIteration exception is the standard way for any Python "iterable" object to signal that you have come to the end of the data it will provide to you. This loop:
cursor.execute('SELECT * FROM pname')
while 1:
try:
print(cursor.next())
except StopIteration:
break
Can be written as:
cursor.execute('SELECT * FROM pname')
for x in cursor:
print(x)
EDIT: I have to say that, despite my cheerful confidence that the exceptions would be documented, I cannot find such a list in the Python docstring help or web page help for dbapi2. In that case I guess I just do what you did: Google search for the name of the exception and see what pops out!
You could use python's inspect module and generic Exception handling like so:
from inspect import getmodule
try:
# attempt some task
except Exception, e:
print getmodule(e)
Which will spit out the module and its path like so:
<module 'some.module' from '/path/to/some/module.pyc'>
Usually I wrap the bare minimum that I can in a try..except block, e.g.:
try:
os.rename('foo', 'bar')
except EnvironmentError:
...
try:
open('bar')
except EnvironmentError:
...
This way I can handle each exception appropriately and separately.
Exception is somehow part of the function signature. The 1st thing to do is to read the documentation, the list of exception it can throw should be given. That's the case in most modules.
In your code, you should catch the minimum exception and just have one general catch at the top of your code (in your main for example) and print somewhere the traceback in order to know where unhandled exceptions comes from.
Then the right way to make sure that you didn't miss an exception is to write unit tests for your code and to cover every possible scenario.
I hope it helps