I'm developing a Jinja extension that performs a few potentially harmful operations, which on failure will raise an Exception that holds some information on what went wrong.
When this occurs, of course the Exception will prevent Jinja from completing its render process and return None rather than a render result. This problem is easily mitigated using a try/catch statement and returning an empty string or whatever is suitable.
However, that effectively throws away the Exception and the debugging info, which I would rather pass on to my error log. The system responsible for setting up the Jinja environment has it's own logging service, which I would prefer to keep decoupled from the extension. I am aware though, that Jinja has an Undefined class which I have successfully used to intercept access of undefined variables in another project.
Is there any way I could raise a special type of exception (UndefinedException did not work), or tell the Jinja environment to log a warning, when an error occurs in my extension, while still allowing it to continue its execution?
What I have in mind is something along the lines of this example:
def _render(*args, **kwrags):
# ...
try:
self.potentially_harmful_operation()
except Exception as e:
self.environment.warning(e)
return Markup("")
# ...
return Markup('<img src"{}">'.format(img_path))
So, to clarify, as I wrote in a comment below:
Fundamentally I guess my question boils down to "how can I make Jinja produce a warning, as it does for e.g. missing variables".
For anyone interested, I managed to solve this by implementing my own Undefined class (largely inspired by make_logging_undefined and supplying that to the environment using the undefined keyword. Depending on your needs, the default make_logging_undefined might very well suffice.
Once that's in place, the trick is to emit the warning in the environment. This is done by simply instantiating the environment's Undefined and executing one of its magic methods (e.g. __str__). Hooking on to my example code from above, that could be accomplished like this:
def _render(*args, **kwrags):
# ...
try:
self.potentially_harmful_operation()
except Exception as e:
str(self.environment.undefined(e))
return Markup("")
# ...
return Markup('<img src"{}">'.format(img_path))
Related
What is the best practice in Python for reporting an error which occurs when closing a resource?
In particular, if I implement __enter__ and __exit__, I can use
with my_library.connect(endpoint) as connection:
connection.store_data("...")
But what should I do if closing my connection and persisting the changes fails, e.g. due to a network outage? I know that I could technically raise an error from within __exit__ but is this best practice/idiomatic in Python? Or should I, e.g., provide a separate persistChanges method, let the __exit__ swallow all errors, and then write in the documentation "if you don't call persistChanges you might lose your changes in error cases"?
My specific use case is: I am providing a Python API to other developers, and I wonder how to handle this case of "error on closing the resource" such that my API follows Python best practices/meets the expectations of Python devs using my library.
I would recommend making a custom error/warning class for your library. this can be very, very simple. There already exists a set of built-in exceptions that you can extend from. Based on your description above, I would recommend extending the RuntimeError like this:
class MyLibraryConnectionError(RuntimeError):
pass
or, if you want to only throw a warning, using the ResourceWarning like this:
class MyLibraryConnectionWarning(ResourceWarning):
pass
There is also the RuntimeWarning that could be extended for similar effect.
If you feel that ResourceWarning, RuntimeWarning, and RuntimeError don't accurately describe the exception, you can also just have them inherit directly from Exception or Warning, depending on whether you want them to only be flagged in Developer mode(Warning), or if you want the full exception functionality.
You can throw these like any other exception:
throw MyLibraryConnectionError("The underlying resource failed to close")
throw MyLibraryConnectionWarning("The underlying resource failed to close")
or even catch your dependency's thrown exception:
def __exit__(...):
try:
# potentially dangerous code with connections
underlyingConnection.close()
except TheErrorYouSeeThrown as e: # you can probably make this Exception instead of TheErrorYouSeeThrown. The more specific the better, so you don't accidentally catch random errors you didn't mean to.
throw MyLibraryConnectionError(e.message) # or whatever this needs to be, depending on the thrown exception type
Then your users could implement like so:
try:
with my_library.connect(endpoint) as connection:
connection.store_data("...")
except my_library.MyLibraryConnectionError as e:
# handle appropriately, or ignore
except Exception as e:
# Handle other errors that happen, that your library doesn't cause.
I'm relatively new to python, using VSCode for python development. As far as I can tell VSCode is using an extension called "pylance" to handle python support features, such as detecting errors in code as you write.
In the last language I spent time with (Scala), there was a great little expression ??? that could be used to mark a method as incomplete / unimplemented, so that it would not generate any errors in the compiler or IDE, and would just throw an exception if encountered at runtime.
Is there any equivalent in python, specifically that would be understood by pylance? The idea would be that there is an unimplemented function, or one with a return type hint that isn't satisfied because it is incomplete, but this wouldn't throw up any errors that would make it harder to find the problems with the part I'm actually working on.
Obviously this isn't critical, just a matter of preferences, but it would be nice to keep the signal-to-noise ratio down! Thank you
You can use use pass inside of a function definition to avoid having to give a function a body, this will not raise exceptions on its own. Alternatively you can use raise(NotImplementedError()) to raise an error when a function is called.
def foo():
pass
def baz():
raise NotImplementedError
EDIT With Update---
Similar to pass in Python 3+ an ellipsis can be used to indicate code that has not yet been written e.g.
def foo(): ...
If you want it to throw an exception at runtime, the standard thing is to just raise NotImplementedError:
def some_fn():
raise NotImplementedError
For example:
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.render('data.html', items = [])
It yields the following Pylint error:
warning (W0223, abstract-method, MainHandler) Method 'data_received' is abstract in class 'RequestHandler' but is not overridden
I understand that somehow it wants me to override this data_received method, but I do not understand why, and what it is for?
This is actually a problem with Pylint that's sort of unavoidable with the nature of Python.
The RequestHandler class has a lot of methods that act as hooks you can override in order to do different things, but only some of those hooks may actually be called, depending on your application's code. To make sure you're implementing everything you're supposed to when you're using certain functionality, the default data_received implementation throws a NotImplementedError that will get triggered when you do something that expects your class to have a custom implementation.
Normally this isn't any kind of issue because Python lets you have code paths that fail and doesn't throw any errors. Because Pylint tries to "help" make sure you've done everything you're supposed to, it's seeing that NotImplementedError throw and is warning you that you could trigger it depending on what you do.
The real problem is that because Python is an interpreted language, it's hard for a tool like Pylint to look at your code and make sure it's "safe". Python gives you a lot of flexibility and power, but in turn you bear the burden of keeping your program's logic straight in your head and knowing what possible problems are actually problems, and what aren't.
Luckily, Pylint is aware of its own limitations and gives you nice tools to disable extraneous warnings. Add the comment line
# pylint: disable=W0223
right before your class definition and the warning should stop popping up for this instance while leaving everything else alone.
I am running into the same issue as the OP, except my PyCharm (2018.3.4) seems not to be using Pylint, but its own inspection engine. I managed to solve a similar issue with the similar trick as R Phillip Castagna suggested:
# noinspection PyAbstractClass
class XyzRequestHandler(tornado.web.RequestHandler):
def prepare(self):
print('-' * 100)
self.set_header('Access-Control-Allow-Origin', '*')
def print_n_respond(self, result):
response = json.dumps(result)
print('responding with:\n', response)
print()
self.write(response)
Here is a list of PyCharm's inspections.
I have a package __init__.py that looks something like this:
import sys
__required_python_version = (2,6, 0)
if sys.version_info < __required_python_version:
this_version = '.'.join([str(x) for x in sys.version_info[0:3]])
required_version = '.'.join([str(x) for x in __required_python_version])
raise PythonVersionError(this_version, required_version)
class PythonVersionError(Exception):
def __init__(self, this_version, required_version):
self.this_version = this_version
self.required_version = required_version
def __str__(self):
return 'Python version %s is invalid. Must be at least %s' % (self.this_ver, self.required_ver)
While I'm certain there is a more elegant way to format those version strings and I could probably get by using a standard exception, my real question is how would I do something like this? Would the best approach be to move my custom exception into a separate file and import it? Or should I wrap the version check in a function that executes when the __init__ is run? I'm just looking for recommendations on the preferred approach.
Thanks
Since it looks like you won't have any user for that exception --
unless this module is to be used by other modules you are impleemnting as part of a larger system, I say you don't need a custom exception here.
There is very little to gain from it, apart from the error message given. Ay module trying to import yours would have to be aware of it, to catch the exception, or just let the program stop witha backtrace. Since be aware of it , it would need to import your module, it would just crash to a backtrace anyway -- wher ethe user can then read the error message.
For one to read the error message,a plain "Exception" stating it is the incorrect PythonVersin is as good as any custom exception.
On the technical side, Python would need to know about PythonVersionError before raising it: you need to put that code before you try to raise it inside the if block.
And finally, if you are building a larger system, and other parts of the system might try to catch PythonVersionError, the coorect thing to do is to put it in its own file/module, so that it becomes available to this module that will raise it, and any other modules that are importing this.
There seems to be something awkward here.
Is it really usefull to create a custom Exception class when it won't be reused anywhere else in other modules ? If everyone did this we would end up with every module defining it's own different (and probably incompatible) PythonVersionError class.
Why don't you use a standard existing exception ? For this one I would probably go for a standard RuntimeError exception.
OK, I know you don't want this answer, but anyway.
If I really wanted to do this at least I would define PythonVersionException class as a local instance of checking code to avoid polluting module namespace or any global namespace of other files of the module.
Sorry for the long title, but it seems most descriptive for my question.
Basically, I'm having a difficult time finding exception information in the official python documentation. For example, in one program I'm currently writing, I'm using the shutil libary's move function:
from shutil import move
move('somefile.txt', '/tmp/somefile.txt')
That works fine, as long as I have write access to /tmp/, there is enough diskspace, and if all other requirements are satisfied.
However, when writing generic code, it is often difficult to guarantee those factors, so one usually uses exceptions:
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except:
print 'Move failed for some reason.'
I'd like to actually catch the appropriate exceptions thrown instead of just catching everything, but I simply can't find a list of exceptions thrown for most python modules. Is there a way for me to see which exceptions a given function can throw, and why? This way I can make appropriate cases for each exception, eg:
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except PermissionDenied:
print 'No permission.'
except DestinationDoesNotExist:
print "/tmp/ doesn't exist"
except NoDiskSpace:
print 'No diskspace available.'
Answer points go to whoever can either link me to some relevant documentation that I've somehow overlooked in the official docs, or provide a sure-fire way to figure out exactly which exceptions are thrown by which functions, and why.
Thanks!
UPDATE: It seems from the answers given that there really isn't a 100% straight-forward way to figure out which errors are thrown by specific functions. With meta programming, it seems that I can figure out simple cases and list some exceptions, but this is not a particularly useful or convenient method.
I'd like to think that eventually there will be some standard for defining which exceptions are raised by each python function, and that this information will be included in the official documentation. Until then I think I will just allow those exceptions to pass through and error out for my users as it seems like the most sane thing to do.
To amplify Messa, catch what you expect are failure modes that you know how to recover from. Ian Bicking wrote an article that addresses some of the overarching principles as does Eli Bendersky's note.
The problem with the sample code is that it is not handling errors, just prettifying them and discarding them. Your code does not "know" what to do with a NameError and there isn't much it should do other than pass it up, look at Bicking's re-raise if you feel you must add detail.
IOError and OSError are reasonably "expectable" for a shutil.move but not necessarily handleable. And the caller of your function wanted it to move a file and may itself break if that "contract" that Eli writes of is broken.
Catch what you can fix, adorn and re-raise what you expect but can't fix, and let the caller deal with what you didn't expect, even if the code that "deals" is seven levels up the stack in main.
Python doesn't have a mechanism right now for declaring which exceptions are thrown, unlike (for example) Java. (In Java you have to define exactly which exceptions are thrown by what, and if one of your utility methods needs to throw another exception then you need to add it to all of the methods which call it which gets boring quickly!)
So if you want to discover exactly which exceptions are thrown by any given bit of python then you need to examine the documentation and the source.
However python has a really good exception hierarchy.
If you study the exception hierarchy below you'll see that the error superclass you want to catch is called StandardError - this should catch all the errors that might be generated in normal operations. Turning the error into into a string will give a reasonable idea to the user as to what went wrong, so I'd suggest your code above should look like
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except StandardError, e:
print 'Move failed: %s' % e
Exception hierarchy
BaseException
|---Exception
|---|---StandardError
|---|---|---ArithmeticError
|---|---|---|---FloatingPointError
|---|---|---|---OverflowError
|---|---|---|---ZeroDivisionError
|---|---|---AssertionError
|---|---|---AttributeError
|---|---|---BufferError
|---|---|---EOFError
|---|---|---EnvironmentError
|---|---|---|---IOError
|---|---|---|---OSError
|---|---|---ImportError
|---|---|---LookupError
|---|---|---|---IndexError
|---|---|---|---KeyError
|---|---|---MemoryError
|---|---|---NameError
|---|---|---|---UnboundLocalError
|---|---|---ReferenceError
|---|---|---RuntimeError
|---|---|---|---NotImplementedError
|---|---|---SyntaxError
|---|---|---|---IndentationError
|---|---|---|---|---TabError
|---|---|---SystemError
|---|---|---TypeError
|---|---|---ValueError
|---|---|---|---UnicodeError
|---|---|---|---|---UnicodeDecodeError
|---|---|---|---|---UnicodeEncodeError
|---|---|---|---|---UnicodeTranslateError
|---|---StopIteration
|---|---Warning
|---|---|---BytesWarning
|---|---|---DeprecationWarning
|---|---|---FutureWarning
|---|---|---ImportWarning
|---|---|---PendingDeprecationWarning
|---|---|---RuntimeWarning
|---|---|---SyntaxWarning
|---|---|---UnicodeWarning
|---|---|---UserWarning
|---GeneratorExit
|---KeyboardInterrupt
|---SystemExit
This also means that when defining your own exceptions you should base them off StandardError not Exception.
Base class for all standard Python exceptions that do not represent
interpreter exiting.
Yes, you can (for simple cases), but you need a bit of meta-programming. Like the other answers have said, a function does not declare that it throws a particular error type, so you need to look at the module and see what exception types it defines, or what exception types it raises. You can either try to grok the documentation or leverage the Python API to do this.
To first find which exception types a module defines, just write a simple script to go through each object in the module dictionary module.__dict__ and see if it ends in the word "Error" or if it is a subclass of Exception:
def listexns(mod):
"""Saved as: http://gist.github.com/402861
"""
module = __import__(mod)
exns = []
for name in module.__dict__:
if (issubclass(module.__dict__[name], Exception) or
name.endswith('Error')):
exns.append(name)
for name in exns:
print '%s.%s is an exception type' % (str(mod), name)
return
If I run this on your example of shutils I get this:
$ python listexn.py shutil
Looking for exception types in module: shutil
shutil.Error is an exception type
shutil.WindowsError is an exception type
$
That tells you which error types are defined, but not which ones are thrown. To achieve the latter, we need to walk over the abstract syntax tree generated when the Python interpreter parses the module, and look for every raise statement, then save a list of names which are raised. The code for this is a little long, so first I'll state the output:
$ python listexn-raised.py /usr/lib/python2.6/shutil.py
Looking for exception types in: /usr/lib/python2.6/shutil.py
/usr/lib/python2.6/shutil.py:OSError is an exception type
/usr/lib/python2.6/shutil.py:Error is an exception type
$
So, now we know that shutil.py defines the error types Error and WindowsError and raises the exception types OSError and Error. If we want to be a bit more complete, we could write another method to check every except clause to also see which exceptions shutil handles.
Here's the code to walk over the AST, it just uses the compiler.visitor interface to create a walker which implements the "visitor pattern" from the Gang of Four book:
class ExceptionFinder(visitor.ASTVisitor):
"""List all exceptions raised by a module.
Saved as: http://gist.github.com/402869
"""
def __init__(self, filename):
visitor.ASTVisitor.__init__(self)
self.filename = filename
self.exns = set()
return
def __visitName(self, node):
"""Should not be called by generic visit, otherwise every name
will be reported as an exception type.
"""
self.exns.add(node.name)
return
def __visitCallFunc(self, node):
"""Should not be called by generic visit, otherwise every name
will be reported as an exception type.
"""
self.__visitName(node.node)
return
def visitRaise(self, node):
"""Visit a raise statement.
Cheat the default dispatcher.
"""
if issubclass(node.expr1, compiler.ast.Name):
self.__visitName(node.expr1)
elif isinstance(node.expr1, compiler.ast.CallFunc):
self.__visitCallFunc(node.expr1)
return
As these operations usually use libc functions and operating system calls, mostly you get IOError or OSError with an errno number; these errors are listed in man pages of that libc/OS calls.
I know this is possibly not a complete answer, it would be good to have all exceptions listed in documentation...