We Keep Coding

Python: Best practice to report an error during closing a resource

What is the best practice in Python for reporting an error which occurs when closing a resource?
In particular, if I implement __enter__ and __exit__, I can use
with my_library.connect(endpoint) as connection:
connection.store_data("...")
But what should I do if closing my connection and persisting the changes fails, e.g. due to a network outage? I know that I could technically raise an error from within __exit__ but is this best practice/idiomatic in Python? Or should I, e.g., provide a separate persistChanges method, let the __exit__ swallow all errors, and then write in the documentation "if you don't call persistChanges you might lose your changes in error cases"?
My specific use case is: I am providing a Python API to other developers, and I wonder how to handle this case of "error on closing the resource" such that my API follows Python best practices/meets the expectations of Python devs using my library.

I would recommend making a custom error/warning class for your library. this can be very, very simple. There already exists a set of built-in exceptions that you can extend from. Based on your description above, I would recommend extending the RuntimeError like this:
class MyLibraryConnectionError(RuntimeError):
pass
or, if you want to only throw a warning, using the ResourceWarning like this:
class MyLibraryConnectionWarning(ResourceWarning):
pass
There is also the RuntimeWarning that could be extended for similar effect.
If you feel that ResourceWarning, RuntimeWarning, and RuntimeError don't accurately describe the exception, you can also just have them inherit directly from Exception or Warning, depending on whether you want them to only be flagged in Developer mode(Warning), or if you want the full exception functionality.
You can throw these like any other exception:
throw MyLibraryConnectionError("The underlying resource failed to close")
throw MyLibraryConnectionWarning("The underlying resource failed to close")
or even catch your dependency's thrown exception:
def __exit__(...):
try:
# potentially dangerous code with connections
underlyingConnection.close()
except TheErrorYouSeeThrown as e: # you can probably make this Exception instead of TheErrorYouSeeThrown. The more specific the better, so you don't accidentally catch random errors you didn't mean to.
throw MyLibraryConnectionError(e.message) # or whatever this needs to be, depending on the thrown exception type
Then your users could implement like so:
try:
with my_library.connect(endpoint) as connection:
connection.store_data("...")
except my_library.MyLibraryConnectionError as e:
# handle appropriately, or ignore
except Exception as e:
# Handle other errors that happen, that your library doesn't cause.

Is there a use case for an alternative "exception cause" in Python?

Background
In Python it is possible to suppress the context of an Exception if you raise your Exception from None. PEP 409 describes the rationale behind it. Sometimes you want to show only one meaningful (custom) Exception. With PEP 415 the implementation changes with the following argument:
The main problem with this scheme [from PEP 409] is it complicates the role of __cause__. __cause__ should indicate the cause of the exception not whether __context__ should be printed or not. This use of __cause__ is also not easily extended in the future. For example, we may someday want to allow the programmer to select which of __context__ and __cause__ will be printed.
Question
PEP 419 talks about future use cases. Are there any valid use cases right now (Python 3.3+) for using an alternative exception cause? For example, consider the following code:
class CustomError(BaseException):
pass
class StupidError(BaseException):
def __init__(self, message='This is just a stupid error.'):
super(StupidError, self).__init__(message)
self.message = message
try:
value = int('a')
except Exception:
raise CustomError('Custom error message') from StupidError
Output:
StupidError: This is just a stupid error.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): (...)
CustomError: Custom error message
Are there any real use cases in which you want to hide the ValueError but show the StupidError? I mean you could give some relevant information in the StupidError which are not present in a mere ValueError? Maybe I am just overthinking this whole thing.

Sure. Say you're writing a web app and need to hide sensitive/complicated data from the user.
Say a user tries a request, but the web app backend has trouble reading the necessary data from a MySQL database (for whatever reason). Instead of letting the MySQL module raise the error, which could either expose sensitive information about how the app works internally or just simply confuse the user, I would want to catch it and then throw my custom exception (let's call it serverError). That custom exception would show the user a HTTP 500 page as well as report the error internally so it can be analyzed by a developer to figure out what went wrong and how to prevent it.
This means I only have to write my general error handling code once, and then when I catch an error, I can raise serverError, which takes care of the error reporting for me.

Python: Exception raised even when caught in try/except clause [duplicate]

This question already has answers here:
How can I more easily suppress previous exceptions when I raise my own exception in response?
(3 answers)
Closed 8 years ago.
In my code I want to catch an exception when it occurs, print some information abut the exception to the screen, and then end the script once I have done so. I tried to use something equivalent to the following code, but I don't understand why I get the traceback I do.
When executing:
try:
1 / 0
except ZeroDivisionError:
print("Exception: ZeroDivisionError")
raise Exception
Console reads:
Exception: ZeroDivisionError
Traceback (most recent call last):
File "<pyshell#19>", line 2, in <module>
1 / 0
ZeroDivisionError: division by zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<pyshell#19>", line 5, in <module>
raise Exception
Exception
I thought that if I catch the ZeroDivisionError, it would no longer be raised, and the only thing that would show is the raise Exception I do at the end, but both show in the console.
Why do they both show, and how do I alter the code so only the second shows, or is there a better way to achieve what I want?

The console shows the context here; when an exception is raised from an exception handler, Python attaches the active exception as the __context__ attribute and Python shows that context later on if the new exception is not being handled. If you don't want the context to be shown, you need to supply a cause instead; you can supply an empty cause with with raise ... from None:
try:
1 / 0
except ZeroDivisionError:
print("Exception: ZeroDivisionError")
raise Exception from None
Quoting the raise statement documentation:
The from clause is used for exception chaining: if given, the second expression must be another exception class or instance, which will then be attached to the raised exception as the __cause__ attribute (which is writable). If the raised exception is not handled, both exceptions will be printed[...]
A similar mechanism works implicitly if an exception is raised inside an exception handler: the previous exception is then attached as the new exception’s __context__ attribute[...]
And from the Exceptions documentation:
When raising (or re-raising) an exception in an except clause __context__ is automatically set to the last exception caught; if the new exception is not handled the traceback that is eventually displayed will include the originating exception(s) and the final exception.
When raising a new exception (rather than using a bare raise to re-raise the exception currently being handled), the implicit exception context can be supplemented with an explicit cause by using from with raise:
raise new_exc from original_exc
The expression following from must be an exception or None. It will be set as __cause__ on the raised exception. Setting __cause__ also implicitly sets the __suppress_context__ attribute to True, so that using raise new_exc from None effectively replaces the old exception with the new one for display purposes (e.g. converting KeyError to AttributeError), while leaving the old exception available in __context__ for introspection when debugging.
The default traceback display code shows these chained exceptions in addition to the traceback for the exception itself. An explicitly chained exception in __cause__ is always shown when present. An implicitly chained exception in __context__ is shown only if __cause__ is None and __suppress_context__ is false.

How do I log an exception object with traceback in Python 3

How can I log an exception object with its traceback in Python 3, using the standard logging module?
Note that the exception in question isn't necessarily the one currently being handled.

Logger objects accept an exc_info argument to include exception information (including traceback), which is expected to be a tuple containing the exception's class, the exception itself and the exception's traceback. The trickiest part is to get hold of the traceback, but it turns out that since Python 3.0, exceptions have a __traceback__ attribute:
logger = logging.getLogger()
exc_info = (type(exc), exc, exc.__traceback__)
logger.error('Exception occurred', exc_info=exc_info)

Honestly I don't know if I'm contributing in the slightest by doing this, but I did find this resource that I thought pertinent to your question and I hope useful.
http://www.alexconrad.org/2013/02/loggingexception.html
The gist of it being, if you place logging.exception() inside of an except block, then you can log all of your errors.

Python: User-Defined Exception That Proves The Rule

Python documentations states:
Exceptions should typically be derived from the Exception class,
either directly or indirectly.
the word 'typically' leaves me in an ambiguous state.
consider the code:
class good(Exception): pass
class bad(object): pass
Heaven = good()
Hell = bad()
>>> raise Heaven
Traceback (most recent call last):
File "<pyshell#163>", line 1, in <module>
raise Heaven
good
>>> raise Hell
Traceback (most recent call last):
File "<pyshell#171>", line 1, in <module>
raise Hell
TypeError: exceptions must be classes or instances, not bad
so when reading the python docs, should i replace 'typically' with ''?
what if i have a class hierarchy that has nothing to do with the Exception class, and I want to 'raise' objects belonging to the hierarchy?
I can always raise an exception with an argument:
raise Exception, Hell
This seems slightly awkward to me
What's so special about the Exception (EDIT: or BaseException) class, that only its family members can be raised?

There are other valid classes you can inherit from apart from Exception, for example BaseException.
See the documentation for the exception hierarchy.
BaseException
+-- SystemExit
+-- KeyboardInterrupt
+-- GeneratorExit
+-- Exception
+-- StopIteration
+-- StandardError
etc..
In older versions of Python it was possible to throw things other than exceptions. For example in Python 2.5:
>>> raise "foo"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
foo
But you get this deprecation warning:
DeprecationWarning: raising a string exception is deprecated
In newer versions this is not allowed. Everything you raise must derive from BaseException.

"so when reading the python docs,
should i change 'typically' with ''?"
No.
Typically, you inherit from Exception. Period. That's what it says.
Sometimes, you might inherit from BaseException. That's what it doesn't say. You might extend BaseExcetion because you want to defeat except Exception handlers.
What's so special about ...
They're subclasses of BaseException. What more do you need to know? The source is readily available. You can read the source code for the raise statement to see exactly what it checks before it throws the TypeError.
http://svn.python.org/view/python/trunk/Python/ceval.c?annotate=80817
Lines 3456 to 3563.
However, all that matters from a practical stand-point is "subclasses of BaseException."

'Typically' is used because there are a few very rare types of exception that don't want to be caught by a generic Exception handler. If in doubt inherit from Exception, but there are exceptions to that rule.
Most exceptions are used to indicate some sort of error or exceptional condition that is a result of the code and data. These exceptions are all subclasses of Exception. If you want to raise your own exception it will probably fall into that category and should therefore also inherit Exception. If you want a generic exception handler, e.g. to log errors then it is perfectly reasonable to catch Exception and expect to catch any errors that way.
The other exceptions which inherit directly from BaseException are slightly different. SystemExit is raised when you call sys.exit() (or you can raise it directly). If you do have some top level code that logs errors then you probably don't want it to handle SystemExit in the same way. You used to have to include a separate handler for SystemExit just to stop the generic Exception handler catching that case.
KeyboardInterrupt does represent an unexpected condition, but it's one that is raised by external input from the user so it can happen anywhere in your code; it doesn't depend in any way on the code or data. That means even if you do want to handle it you probably want to handle it differently than the exceptions which inherit from Exception.

Things derived from the various error/exception classes can also be raised, but those are generally reserved for other things.

Sometimes you might want to raise things you do not consider an exception (but the rule). For these rare cases one can consider it nice to raise something else besides just exceptions.
In a recursive computation in which at some point an answer is found, raising it to the upwardly waiting catcher can be pretty neat (instead of returning it which means the recursion has to expect this, pass it upwards as well etc.).
But nowadays only old-style classes (for compatibility reasons I guess) and derivations from BaseException can be raised because too many people abused things like strings for exceptions.