We Keep Coding

Modifying or Reraising Python error in C API

I have a bit of code that tries to parse an object as an integer:
long val = PyLong_AsLong(obj);
if(val == -1 && PyErr_Occurred()) {
return -1;
}
Here obj is a vanilla PyObject *, and PyLong_AsLong raises a very generic TypeError if obj is not an integer.
I would like to transform the error message into something a bit more informative, so I would like to either modify the existing error object, or to reraise it.
My current solution is to do this:
long val = PyLong_AsLong(obj);
if(val == -1 && PyErr_Occurred()) {
PyErr_Clear();
PyErr_Format(PyExc_TypeError, "Parameter must be an integer type, but got %s", Py_TYPE(obj)->tp_name);
return -1;
}
Is this the proper way to reraise an error? Specifically,
Do I need to call PyErr_Clear at all? I suspect that it properly decrefs the existing exception object, but I'm not sure.
Can I modify the message of the error that has already been thrown at that point without re-raising it?
Is there an option to do the equivalent of raise new_err from old_err?
I am not sure how to use PyErr_SetExcInfo for this situation, although my gut tells me it may be relevant somehow.

Your existing code is fine, but if you want to do the equivalent of exception chaining, you can. If you want to skip to how to do that, jump to point 3 near the end of the answer.
To explain how to do things like modify a propagating exception or perform the equivalent of raise Something() from existing_exception, first, we'll have to explain how exception state works at C level.
A propagating exception is represented by a per-thread error indicator consisting of a type, value, and traceback. That sounds a lot like sys.exc_info(), but it's not the same. sys.exc_info() is for exceptions that have been caught by Python-level code, not exceptions that are still propagating.
The error indicator may be unnormalized, which basically means that the work of constructing an exception object hasn't been performed, and the value in the error indicator isn't an instance of the exception type. This state exists for efficiency; if the error indicator is cleared by PyErr_Clear before normalization is needed, Python gets to skip much of the work of raising an exception. Exception normalization is performed by PyErr_NormalizeException, with a bit of extra work in PyException_SetTraceback to set the exception object's __traceback__ attribute.
PyErr_Clear is sort of like the C equivalent of an except block, but it just clears the error indicator, without letting you inspect much of the exception information. To catch an exception and inspect it, you'd want PyErr_Fetch. PyErr_Fetch is like catching an exception and examining sys.exc_info(), but it doesn't set sys.exc_info() or normalize the exception. It clears the error indicator and gives you the raw contents of the error indicator directly.
Explicit exception chaining (raise Something() from existing_exception) works by going through PyException_SetCause to set the new exception's __cause__ to the existing exception. This requires exception objects for both exceptions, so if you want to do the equivalent from C, you'll have to normalize the exceptions and call PyException_SetCause yourself.
Implicit exception chaining (raise Something() in an except block) works by going through PyException_SetContext to set the new exception's __context__ to the existing exception. Similar to PyException_SetCause, this requires exception objects and exception normalization. raise Something() from existing_exception inside an except block actually sets both __cause__ and __context__, and if you want to perform explicit exception chaining at C level, you should usually do the same.
Technically not necessary, as far as I can tell, but it's probably a good idea to do it anyway. It looks like PyErr_Format and other functions that set the error indicator will clear the error indicator first if it's already set, but this isn't documented for most of them.
Sort of, but it's probably a bad idea. You can normalize the error indicator and set the exception object's message attribute, but this won't affect args or anything else the exception class might do with its arguments, and that could lead to weird problems. Alternatively, you could fetch the error indicator with PyErr_Fetch and restore it with a new string for the value with PyErr_Restore, but that will throw away an existing exception object if there is one, and it makes assumptions about the exception class's signature.
Yeah, that's possible, but doing it through public C API functions is pretty awkward and manual. You'd have to manually do a lot of normalization, unraising, and raising exceptions.
There are efforts to make C-level exception chaining more convenient, but so far, the more convenient functions are all considered internal. For example, _PyErr_FormatFromCause is like PyErr_Format, but it chains the new exception off of an existing, propagating exception (through both __context__ and __cause__.
I wouldn't recommend calling it directly for now; it's very new (3.6+), and it's very likely to change (specifically, I would be unsurprised to see it lose its leading underscore in a new Python version). Instead, copying the implementation of _PyErr_FormatFromCause/_PyErr_FormatVFromCause (and respecting the license) is a good way to make sure you have the fiddly bits of normalization and chaining right.
It's also a useful reference to work from if you want to perform implicit (__context__-only) exception chaining at C level - just remove the part that handles __cause__.

Using except to mean "and if that doesn't work"

I fully get the general principle of not catching all exception, as explained in this question, for instance (Why is "except: pass" a bad programming practice?). Yet I have found myself writing this sequence to get the source of a file like object:
try:
self.doc.source = source.geturl()
except:
try:
self.doc.source = pathlib.Path(os.path.abspath(source.name)).as_uri()
except:
self.doc.source = None
Clearly with some spelunking I could figure out which specific errors to catch with a reasonable degree of confidence. But at is explained in this question (Python: How can I know which exceptions might be thrown from a method call) you can't ever be quite certain.
So is there a better way to do what I am doing here, which is essentially to say: try this and if it doesn't work try this and if that doesn't work do this. Since this is all about setting a single variable, and there is a fallback of setting it to None, it is not obvious to me wherein the peril lies in this construct.

Maybe it'd be better to put it into a dedicated function ?
def func(source):
try:
return source.geturl()
except Exception:
try:
return pathlib.Path(os.path.abspath(source.name)).as_uri()
except Exception:
pass
Notes on swallowing exceptions
Ignoring any kind of exception is not the best pattern : it'd be better to know what call can raise what kind of exception.
Even if except ...: pass is bad, that's already what you were doing in your example, and here it's clear that we try another way if it fails.
The bad programming practice is basing the error handling on the ability to catch any kind of error. In itself, except: pass is better than except: <lots_of_things_that_wont_raise_again>.
However, the first answer in the post you referred to says something very sensible : pass semantically indicates that you won't do ANYTHING with the exception, and as you do not store in a variable, it also shows that you'll never be able to access it again (well you could, but still..). As /u/Aprillion/, you'll want to at least log these errors, because otherwise, you may be swallowing very useful debugging information, which could make your life (or others) much more difficult at some point.
For example, what happens if there's a bug hidden in geturl, that makes it raises exception in normal cases ? It may stay hidden for long, because this exception will be swallowed and never shown anywhere. Good luck to the debugger to find it !
Ways to improve this
replace except Exception with the exceptions that could be raised
check before an attempt if it is going to fail, so that you don't have to catch an exception
Additionally, you probably want to use a logger to save/show these exceptions somewhere, at least in DEBUG mode.
Clearly with some spelunking I could figure out which specific errors to catch with a reasonable degree of confidence. But at is explained in this question (Python: How can I know which exceptions might be thrown from a method call) you can't ever be quite certain.
Why would that change anything ? If geturl returns specific exception types, you'd only catch these, and you'd want other unexpected errors to bubble up to the interpreter.
The big problem with the givn approach is that if geturl is undefined, or not callable, or take more arguments, this error would not make the interpreter crash, even though it is a clear programming error. That's because except: or except Exception: will catch a lof of Python errors, should it be AttributeError, NameError, ImportError or even SyntaxError.
To conclude, I really think you'd prefer to replace Exception with the list of exceptions you can catch (and at least send the exceptions to a logger). Note that you can write it as such :
try:
return source.geturl()
except (ExceptionA, ExceptionB, ExceptionC):
pass

Is it OK to raise a built-in exception, but with a different message, in Python?

Is it OK to raise a built-in exception with a custom text? or to raise a built-in warning also with custom text?
The documentation reads:
exception ValueError: Raised when a built-in operation or function receives an argument (…)
Is it implied that only built-in operations should raise a ValueError exception?
In practice, I understand that it is safe to create an exception class that inherits from ValueError or Exception. But is it OK not to do that, and directly raise a ValueError("custom text")?
Since ValueError is built-in, raising a ValueError (with a custom text) allows users to quickly see what kind of problem is involved, compared to a custom exception type (something like "ValueErrorSpecificModule", which is not standard).

There's nothing operationally wrong with doing something like:
raise ValueError("invalid input encoding")
In fact, I do that quite often when I'm writing the first pass of some code. The main problem with doing it that way is that clients of your code have a hard time being precise in their exception handling; in order to catch that specific exception, they would have to do string matching on the exception object they caught, which is obviously fragile and tedious. Thus, it would be better to introduce a ValueError subclass of your own; this could still be caught as ValueError, but also as the more specific exception class.
A general rule of thumb is that whenever you have code like:
raise ValueError('some problem: %s' % value)
You should probably replace it with something like:
class SomeProblem(ValueError):
"""
Raised to signal a problem with the specified value.
"""
# ...
raise SomeProblem(value)
You might say that the exception type specifies what went wrong, whereas the message / attributes specify how it went wrong.

It's perfectly ok.
However you may want to create your own subclass to help distinguish from the builtin exceptions
For example if you have something that works like a dict, you can raise a KeyError for the usual reasons, but what if the KeyError is really coming from an underlying dict you are using in the implementation.
Raising a subclass of KeyError makes it easier to see that there is a bug in the implementation, and not that the key just isn't in your object

It's OK and I do it all the time. I find it less surprising to see TypeError than MySpecialTypeError in many situations.
On the page you linked, I don't see the phrase "built-in":
exception TypeError: Raised when an operation or function is applied to an object of inappropriate type. The associated value is a string giving details about the type mismatch.
Perhaps someone saw your question and fixed the documentation already.
EDIT: It looks like you may have inserted the documentation for ValueError instead of TypeError

How can I know which exceptions might be thrown from a method call?

Is there a way knowing (at coding time) which exceptions to expect when executing python code?
I end up catching the base Exception class 90% of the time since I don't know which exception type might be thrown (reading the documentation doesn't always help, since many times an exception can be propagated from the deep. And many times the documentation is not updated or correct).
Is there some kind of tool to check this (like by reading the Python code and libs)?

I guess a solution could be only imprecise because of lack of static typing rules.
I'm not aware of some tool that checks exceptions, but you could come up with your own tool matching your needs (a good chance to play a little with static analysis).
As a first attempt, you could write a function that builds an AST, finds all Raise nodes, and then tries to figure out common patterns of raising exceptions (e. g. calling a constructor directly)
Let x be the following program:
x = '''\
if f(x):
raise IOError(errno.ENOENT, 'not found')
else:
e = g(x)
raise e
'''
Build the AST using the compiler package:
tree = compiler.parse(x)
Then define a Raise visitor class:
class RaiseVisitor(object):
def __init__(self):
self.nodes = []
def visitRaise(self, n):
self.nodes.append(n)
And walk the AST collecting Raise nodes:
v = RaiseVisitor()
compiler.walk(tree, v)
>>> print v.nodes
[
Raise(
CallFunc(
Name('IOError'),
[Getattr(Name('errno'), 'ENOENT'), Const('not found')],
None, None),
None, None),
Raise(Name('e'), None, None),
]
You may continue by resolving symbols using compiler symbol tables, analyzing data dependencies, etc. Or you may just deduce, that CallFunc(Name('IOError'), ...) "should definitely mean raising IOError", which is quite OK for quick practical results :)

You should only catch exceptions that you will handle.
Catching all exceptions by their concrete types is nonsense. You should catch specific exceptions you can and will handle. For other exceptions, you may write a generic catch that catches "base Exception", logs it (use str() function) and terminates your program (or does something else that's appropriate in a crashy situation).
If you really gonna handle all exceptions and are sure none of them are fatal (for example, if you're running the code in some kind of a sandboxed environment), then your approach of catching generic BaseException fits your aims.
You might be also interested in language exception reference, not a reference for the library you're using.
If the library reference is really poor and it doesn't re-throw its own exceptions when catching system ones, the only useful approach is to run tests (maybe add it to test suite, because if something is undocumented, it may change!). Delete a file crucial for your code and check what exception is being thrown. Supply too much data and check what error it yields.
You will have to run tests anyway, since, even if the method of getting the exceptions by source code existed, it wouldn't give you any idea how you should handle any of those. Maybe you should be showing error message "File needful.txt is not found!" when you catch IndexError? Only test can tell.

The correct tool to solve this problem is unittests. If you are having exceptions raised by real code that the unittests do not raise, then you need more unittests.
Consider this
def f(duck):
try:
duck.quack()
except ??? could be anything
duck can be any object
Obviously you can have an AttributeError if duck has no quack, a TypeError if duck has a quack but it is not callable. You have no idea what duck.quack() might raise though, maybe even a DuckError or something
Now supposing you have code like this
arr[i] = get_something_from_database()
If it raises an IndexError you don't know whether it has come from arr[i] or from deep inside the database function. usually it doesn't matter so much where the exception occurred, rather that something went wrong and what you wanted to happen didn't happen.
A handy technique is to catch and maybe reraise the exception like this
except Exception as e
#inspect e, decide what to do
raise

Noone explained so far, why you can't have a full, 100% correct list of exceptions, so I thought it's worth commenting on. One of the reasons is a first-class function. Let's say that you have a function like this:
def apl(f,arg):
return f(arg)
Now apl can raise any exception that f raises. While there are not many functions like that in the core library, anything that uses list comprehension with custom filters, map, reduce, etc. are affected.
The documentation and the source analysers are the only "serious" sources of information here. Just keep in mind what they cannot do.

I ran into this when using socket, I wanted to find out all the error conditions I would run in to (so rather than trying to create errors and figure out what socket does I just wanted a concise list). Ultimately I ended up grep'ing "/usr/lib64/python2.4/test/test_socket.py" for "raise":
$ grep raise test_socket.py
Any exceptions raised by the clients during their tests
raise TypeError, "test_func must be a callable function"
raise NotImplementedError, "clientSetUp must be implemented."
def raise_error(*args, **kwargs):
raise socket.error
def raise_herror(*args, **kwargs):
raise socket.herror
def raise_gaierror(*args, **kwargs):
raise socket.gaierror
self.failUnlessRaises(socket.error, raise_error,
self.failUnlessRaises(socket.error, raise_herror,
self.failUnlessRaises(socket.error, raise_gaierror,
raise socket.error
# Check that setting it to an invalid value raises ValueError
# Check that setting it to an invalid type raises TypeError
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
Which is a pretty concise list of errors. Now of course this only works on a case by case basis and depends on the tests being accurate (which they usually are). Otherwise you need to pretty much catch all exceptions, log them and dissect them and figure out how to handle them (which with unit testing wouldn't be to difficult).

There are two ways that I found informative. The first one, run the code in iPython, which will display the exception type.
n = 2
str = 'me '
str + 2
TypeError: unsupported operand type(s) for +: 'int' and 'str'
In the second way we settle for catching too much and improve on it over time. Include a try expression in your code and catch except Exception as err. Print sufficient data to know what exception was thrown. As exceptions are thrown improve your code by adding a more precise except clause. When you feel that you have caught all relevant exceptions remove the all inclusive one. A good thing to do anyway because it swallows programming errors.
try:
so something
except Exception as err:
print "Some message"
print err.__class__
print err
exit(1)

normally, you'd need to catch exception only around a few lines of code. You wouldn't want to put your whole main function into the try except clause. for every few line you always should now (or be able easily to check) what kind of exception might be raised.
docs have an exhaustive list of built-in exceptions. don't try to except those exception that you're not expecting, they might be handled/expected in the calling code.
edit: what might be thrown depends on obviously on what you're doing! accessing random element of a sequence: IndexError, random element of a dict: KeyError, etc.
Just try to run those few lines in IDLE and cause an exception. But unittest would be a better solution, naturally.

This is a copy and pasted answer I wrote for How to list all exceptions a function could raise in Python 3?, I hope that is allowed.
I needed to do something similar and found this post. I decided I
would write a little library to help.
Say hello to Deep-AST. It's very early alpha but it is pip
installable. It has all of the limitations mentioned in this post
and some additional ones but its already off to a really good start.
For example when parsing HTTPConnection.getresponse() from
http.client it parses 24489 AST Nodes. It finds 181 total raised
Exceptions (this includes duplicates) and 8 unique Exceptions were
raised. A working code example.
The biggest flaw is this it currently does work with a bare raise:
def foo():
try:
bar()
except TypeError:
raise
But I think this will be easy to solve and I plan on fixing it.
The library can handle more than just figuring out exceptions, what
about listing all Parent classes? It can handle that too!

Raise exception vs. return None in functions? [duplicate]

This question already has answers here:
Is it better to use an exception or a return code in Python?
(6 answers)
Python: Throw Exception or return None? [closed]
(3 answers)
Closed 1 year ago.
What's better practice in a user-defined function in Python: raise an exception or return None? For example, I have a function that finds the most recent file in a folder.
def latestpdf(folder):
# list the files and sort them
try:
latest = files[-1]
except IndexError:
# Folder is empty.
return None # One possibility
raise FileNotFoundError() # Alternative
else:
return somefunc(latest) # In my case, somefunc parses the filename
Another option is leave the exception and handle it in the caller code, but I figure it's more clear to deal with a FileNotFoundError than an IndexError. Or is it bad form to re-raise an exception with a different name?

It's really a matter of semantics. What does foo = latestpdf(d) mean?
Is it perfectly reasonable that there's no latest file? Then sure, just return None.
Are you expecting to always find a latest file? Raise an exception. And yes, re-raising a more appropriate exception is fine.
If this is just a general function that's supposed to apply to any directory, I'd do the former and return None. If the directory is, e.g., meant to be a specific data directory that contains an application's known set of files, I'd raise an exception.

I would make a couple suggestions before answering your question as it may answer the question for you.
Always name your functions descriptive. latestpdf means very little to anyone but looking over your function latestpdf() gets the latest pdf. I would suggest that you name it getLatestPdfFromFolder(folder).
As soon as I did this it became clear what it should return.. If there isn't a pdf raise an exception. But wait there more..
Keep the functions clearly defined. Since it's not apparent what somefuc is supposed to do and it's not (apparently) obvious how it relates to getting the latest pdf I would suggest you move it out. This makes the code much more readable.
for folder in folders:
try:
latest = getLatestPdfFromFolder(folder)
results = somefuc(latest)
except IOError: pass
Hope this helps!

I usually prefer to handle exceptions internally (i.e. try/except inside the called function, possibly returning a None) because python is dynamically typed. In general, I consider it a judgment call one way or the other, but in a dynamically typed language, there are small factors that tip the scales in favor of not passing the exception to the caller:
Anyone calling your function is not notified of the exceptions that can be thrown. It becomes a bit of an art form to know what kind of exception you are hunting for (and generic except blocks ought to be avoided).
if val is None is a little easier than except ComplicatedCustomExceptionThatHadToBeImportedFromSomeNameSpace. Seriously, I hate having to remember to type from django.core.exceptions import ObjectDoesNotExist at the top of all my django files just to handle a really common use case. In a statically typed world, let the editor do it for you.
Honestly, though, it's always a judgment call, and the situation you're describing, where the called function receives an error it can't help, is an excellent reason to re-raise an exception that is meaningful. You have the exact right idea, but unless you're exception is going to provide more meaningful information in a stack trace than
AttributeError: 'NoneType' object has no attribute 'foo'
which, nine times out of ten, is what the caller will see if you return an unhandled None, don't bother.
(All this kind of makes me wish that python exceptions had the cause attributes by default, as in java, which lets you pass exceptions into new exceptions so that you can rethrow all you want and never lose the original source of the problem.)

with python 3.5's typing:
example function when returning None will be:
def latestpdf(folder: str) -> Union[str, None]
and when raising an exception will be:
def latestpdf(folder: str) -> str
option 2 seem more readable and pythonic
(+option to add comment to exception as stated earlier.)

In general, I'd say an exception should be thrown if something catastrophic has occured that cannot be recovered from (i.e. your function deals with some internet resource that cannot be connected to), and you should return None if your function should really return something but nothing would be appropriate to return (i.e. "None" if your function tries to match a substring in a string for example).