We Keep Coding

Linting classes created at runtime in Python

For context, I am using the Python ctypes library to interface with a C library. It isn't necessary to be familiar with C or ctypes to answer this question however. All of this is taking place in the context of a python module I am creating.
In short, my question is: how can I allow Python linters (e.g. PyCharm or plugin for neovim) to lint objects that are created at runtime? "You can't" is not an answer ;). Of course there is always a way, with scripting and the like. I want to know what I would be looking at for the easiest way.
First I introduce my problem and the current approach I am taking. Second, I will describe what I want to do, and ask how.
Within this C library, a whole bunch of error codes are defined. I translated this information from the .h header file into a Python enum:
# CustomErrors.py
from enum import Enum
class CustomErrors(Enum):
ERROR_BROKEN = 1
ERROR_KAPUTT = 2
ERROR_BORKED = 3
Initially, my approach is to have a single exception class containing a type field which described the specific error:
# CustomException.py
from CustomErrors import CustomErrors
class CustomException(Exception):
def __init__(self, customErr):
assert type(customErr) is CustomError
self.type = customErr
super().__init__()
Then, as needed I can raise CustomException(CustomErrors.ERROR_KAPUTT).
Now, what I want to do is create a separate exception class corresponding to each of the enum items in CustomErrors. I believe it is possible to create types at runtime with MyException = type('MyException', (Exception,), {'__doc__' : 'Docstring for ABC class.'}).
I can create the exception classes at runtime like so:
#CustomException.py
from CustomErrors import CustomErrors
...
for ce in CustomErrors:
n = ce.name
vars()[n] = type(n, (Exception,), {'__doc__' : 'Docstring for {0:s} class.'.format(n)})
Note: the reason I want to create these at runtime is to avoid hard-coding of an Exception list that change in the future. I already have the problem of extracting the C enum automatically on the backburner.
This is all well and good, but I have a problem: static analysis cannot resolve the names of these exceptions defined in CustomException. This means PyCharm and other editors for Python will not be able to automatically resolve the names of the exceptions as a suggested autocomplete list when the user types CustomException.. This is not acceptable, as this is code for the end user, who will need to access the exception names for use in try-except constructs.
Here is the only solution I have been able to think of: writing a script which generates the .py files containing the exception names. I can do this using bash. Maybe people will tell me this is really the only option. But I would like to know what other approaches are suggested for solving this problem. Thanks for reading.

You can add a comment to tell mypy to ignore dynamically defined attribute errors. Perhaps the linters that you use share a similar way to silence such errors.
mypy docs on silencing errors based on error codes
This example shows how to ignore an error about an imported name mypy thinks is undefined:
# 'foo' is defined in 'foolib', even though mypy can't see the
# definition.
from foolib import foo # type: ignore[attr-defined]

Use isinstance with an undefined class

Assume that class MyClass is sometimes, but not always, defined. I have a function foo(a=None) in which argument a can be None, a string, or an object of MyClass.
My question is: If MyClass is not defined in my Python session, how can I check the type of argument a in a fashion similar to isinstance without getting a NameError?
Note on duck-typing: I am deliberately limiting the function.
I'm using Python 2.6.x and Updating is not an option. A forward-compatible solution (especially for 2.7.x) is highly appreciated.

I would suggest a different approach: polyfill the class so all code that wants to refer to it can simply do so:
try:
from foo import Bar # load the native class
except ImportError:
class Bar:
pass # implement necessary parts here
You can put this into your own module and then from mymodule import Bar everywhere it's needed. That allows all your code to use Bar regardless of whether it's defined natively or not.
Even if redefining the class isn't your preferred way to handle this, handling the ImportError is still the way to handle this situation, since you will have to import the class either way and that's where the error will occur. Instead of defining the class, you may instead want to set a class_exists = False flag or something.

If MyClass isn't defined then you have no way to reference its type.
Therefore you can have no way to verify that type(a) has the correct value.

I workarounded the problem by overriding a method in MyClass and doing nothing in it (pass). After that I no longer needed to check its type.
Different workarounds may exist for different cases. Catching the NameError could be another one.
t = 'asdfas'
print(isinstance(t, str))
try:
print(isinstance(t, MyClass))
except NameError:
print(False)
Seems to me, that such a construct may appear in future python. Like typed python, which is quite new. And in typed python we have a possibility to use future types, in apos.

What happens when an exception occurs in a module initialization

I have a module, and in this module I initialize some variables as soon as the module is imported.
my_mobule.py:
def _build_service():
# ...do some stuffs
_service = _build_service()
In this case, what will happen if the _build_service method raises an exception? And how my module can recover from an exception and try to invoke the _build_service again?
Thank you guys.

It's pretty similar to the behaviour should you call a function which raises an exception - if you don't handle the exception in the module itself, then it will be simply bump up the stack to whoever imported your module.
That would look like this:
>>> import my_mobule # sic
UhohError: something went wrong
If you have an opportunity to handle it in the module, you could do it like so:
try:
_service = _build_service()
except UhohError:
# your handling code here

How do I raise a custom exception from a package __init__?

I have a package __init__.py that looks something like this:
import sys
__required_python_version = (2,6, 0)
if sys.version_info < __required_python_version:
this_version = '.'.join([str(x) for x in sys.version_info[0:3]])
required_version = '.'.join([str(x) for x in __required_python_version])
raise PythonVersionError(this_version, required_version)
class PythonVersionError(Exception):
def __init__(self, this_version, required_version):
self.this_version = this_version
self.required_version = required_version
def __str__(self):
return 'Python version %s is invalid. Must be at least %s' % (self.this_ver, self.required_ver)
While I'm certain there is a more elegant way to format those version strings and I could probably get by using a standard exception, my real question is how would I do something like this? Would the best approach be to move my custom exception into a separate file and import it? Or should I wrap the version check in a function that executes when the __init__ is run? I'm just looking for recommendations on the preferred approach.
Thanks

Since it looks like you won't have any user for that exception --
unless this module is to be used by other modules you are impleemnting as part of a larger system, I say you don't need a custom exception here.
There is very little to gain from it, apart from the error message given. Ay module trying to import yours would have to be aware of it, to catch the exception, or just let the program stop witha backtrace. Since be aware of it , it would need to import your module, it would just crash to a backtrace anyway -- wher ethe user can then read the error message.
For one to read the error message,a plain "Exception" stating it is the incorrect PythonVersin is as good as any custom exception.
On the technical side, Python would need to know about PythonVersionError before raising it: you need to put that code before you try to raise it inside the if block.
And finally, if you are building a larger system, and other parts of the system might try to catch PythonVersionError, the coorect thing to do is to put it in its own file/module, so that it becomes available to this module that will raise it, and any other modules that are importing this.

There seems to be something awkward here.
Is it really usefull to create a custom Exception class when it won't be reused anywhere else in other modules ? If everyone did this we would end up with every module defining it's own different (and probably incompatible) PythonVersionError class.
Why don't you use a standard existing exception ? For this one I would probably go for a standard RuntimeError exception.
OK, I know you don't want this answer, but anyway.
If I really wanted to do this at least I would define PythonVersionException class as a local instance of checking code to avoid polluting module namespace or any global namespace of other files of the module.

How Can I Find a List of All Exceptions That a Given Library Function Throws in Python?

Sorry for the long title, but it seems most descriptive for my question.
Basically, I'm having a difficult time finding exception information in the official python documentation. For example, in one program I'm currently writing, I'm using the shutil libary's move function:
from shutil import move
move('somefile.txt', '/tmp/somefile.txt')
That works fine, as long as I have write access to /tmp/, there is enough diskspace, and if all other requirements are satisfied.
However, when writing generic code, it is often difficult to guarantee those factors, so one usually uses exceptions:
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except:
print 'Move failed for some reason.'
I'd like to actually catch the appropriate exceptions thrown instead of just catching everything, but I simply can't find a list of exceptions thrown for most python modules. Is there a way for me to see which exceptions a given function can throw, and why? This way I can make appropriate cases for each exception, eg:
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except PermissionDenied:
print 'No permission.'
except DestinationDoesNotExist:
print "/tmp/ doesn't exist"
except NoDiskSpace:
print 'No diskspace available.'
Answer points go to whoever can either link me to some relevant documentation that I've somehow overlooked in the official docs, or provide a sure-fire way to figure out exactly which exceptions are thrown by which functions, and why.
Thanks!
UPDATE: It seems from the answers given that there really isn't a 100% straight-forward way to figure out which errors are thrown by specific functions. With meta programming, it seems that I can figure out simple cases and list some exceptions, but this is not a particularly useful or convenient method.
I'd like to think that eventually there will be some standard for defining which exceptions are raised by each python function, and that this information will be included in the official documentation. Until then I think I will just allow those exceptions to pass through and error out for my users as it seems like the most sane thing to do.

To amplify Messa, catch what you expect are failure modes that you know how to recover from. Ian Bicking wrote an article that addresses some of the overarching principles as does Eli Bendersky's note.
The problem with the sample code is that it is not handling errors, just prettifying them and discarding them. Your code does not "know" what to do with a NameError and there isn't much it should do other than pass it up, look at Bicking's re-raise if you feel you must add detail.
IOError and OSError are reasonably "expectable" for a shutil.move but not necessarily handleable. And the caller of your function wanted it to move a file and may itself break if that "contract" that Eli writes of is broken.
Catch what you can fix, adorn and re-raise what you expect but can't fix, and let the caller deal with what you didn't expect, even if the code that "deals" is seven levels up the stack in main.

Python doesn't have a mechanism right now for declaring which exceptions are thrown, unlike (for example) Java. (In Java you have to define exactly which exceptions are thrown by what, and if one of your utility methods needs to throw another exception then you need to add it to all of the methods which call it which gets boring quickly!)
So if you want to discover exactly which exceptions are thrown by any given bit of python then you need to examine the documentation and the source.
However python has a really good exception hierarchy.
If you study the exception hierarchy below you'll see that the error superclass you want to catch is called StandardError - this should catch all the errors that might be generated in normal operations. Turning the error into into a string will give a reasonable idea to the user as to what went wrong, so I'd suggest your code above should look like
from shutil import move
try:
move('somefile.txt', '/tmp/somefile.txt')
except StandardError, e:
print 'Move failed: %s' % e
Exception hierarchy
BaseException
|---Exception
|---|---StandardError
|---|---|---ArithmeticError
|---|---|---|---FloatingPointError
|---|---|---|---OverflowError
|---|---|---|---ZeroDivisionError
|---|---|---AssertionError
|---|---|---AttributeError
|---|---|---BufferError
|---|---|---EOFError
|---|---|---EnvironmentError
|---|---|---|---IOError
|---|---|---|---OSError
|---|---|---ImportError
|---|---|---LookupError
|---|---|---|---IndexError
|---|---|---|---KeyError
|---|---|---MemoryError
|---|---|---NameError
|---|---|---|---UnboundLocalError
|---|---|---ReferenceError
|---|---|---RuntimeError
|---|---|---|---NotImplementedError
|---|---|---SyntaxError
|---|---|---|---IndentationError
|---|---|---|---|---TabError
|---|---|---SystemError
|---|---|---TypeError
|---|---|---ValueError
|---|---|---|---UnicodeError
|---|---|---|---|---UnicodeDecodeError
|---|---|---|---|---UnicodeEncodeError
|---|---|---|---|---UnicodeTranslateError
|---|---StopIteration
|---|---Warning
|---|---|---BytesWarning
|---|---|---DeprecationWarning
|---|---|---FutureWarning
|---|---|---ImportWarning
|---|---|---PendingDeprecationWarning
|---|---|---RuntimeWarning
|---|---|---SyntaxWarning
|---|---|---UnicodeWarning
|---|---|---UserWarning
|---GeneratorExit
|---KeyboardInterrupt
|---SystemExit
This also means that when defining your own exceptions you should base them off StandardError not Exception.
Base class for all standard Python exceptions that do not represent
interpreter exiting.

Yes, you can (for simple cases), but you need a bit of meta-programming. Like the other answers have said, a function does not declare that it throws a particular error type, so you need to look at the module and see what exception types it defines, or what exception types it raises. You can either try to grok the documentation or leverage the Python API to do this.
To first find which exception types a module defines, just write a simple script to go through each object in the module dictionary module.__dict__ and see if it ends in the word "Error" or if it is a subclass of Exception:
def listexns(mod):
"""Saved as: http://gist.github.com/402861
"""
module = __import__(mod)
exns = []
for name in module.__dict__:
if (issubclass(module.__dict__[name], Exception) or
name.endswith('Error')):
exns.append(name)
for name in exns:
print '%s.%s is an exception type' % (str(mod), name)
return
If I run this on your example of shutils I get this:
$ python listexn.py shutil
Looking for exception types in module: shutil
shutil.Error is an exception type
shutil.WindowsError is an exception type
$
That tells you which error types are defined, but not which ones are thrown. To achieve the latter, we need to walk over the abstract syntax tree generated when the Python interpreter parses the module, and look for every raise statement, then save a list of names which are raised. The code for this is a little long, so first I'll state the output:
$ python listexn-raised.py /usr/lib/python2.6/shutil.py
Looking for exception types in: /usr/lib/python2.6/shutil.py
/usr/lib/python2.6/shutil.py:OSError is an exception type
/usr/lib/python2.6/shutil.py:Error is an exception type
$
So, now we know that shutil.py defines the error types Error and WindowsError and raises the exception types OSError and Error. If we want to be a bit more complete, we could write another method to check every except clause to also see which exceptions shutil handles.
Here's the code to walk over the AST, it just uses the compiler.visitor interface to create a walker which implements the "visitor pattern" from the Gang of Four book:
class ExceptionFinder(visitor.ASTVisitor):
"""List all exceptions raised by a module.
Saved as: http://gist.github.com/402869
"""
def __init__(self, filename):
visitor.ASTVisitor.__init__(self)
self.filename = filename
self.exns = set()
return
def __visitName(self, node):
"""Should not be called by generic visit, otherwise every name
will be reported as an exception type.
"""
self.exns.add(node.name)
return
def __visitCallFunc(self, node):
"""Should not be called by generic visit, otherwise every name
will be reported as an exception type.
"""
self.__visitName(node.node)
return
def visitRaise(self, node):
"""Visit a raise statement.
Cheat the default dispatcher.
"""
if issubclass(node.expr1, compiler.ast.Name):
self.__visitName(node.expr1)
elif isinstance(node.expr1, compiler.ast.CallFunc):
self.__visitCallFunc(node.expr1)
return

As these operations usually use libc functions and operating system calls, mostly you get IOError or OSError with an errno number; these errors are listed in man pages of that libc/OS calls.
I know this is possibly not a complete answer, it would be good to have all exceptions listed in documentation...