For context, I am using the Python ctypes library to interface with a C library. It isn't necessary to be familiar with C or ctypes to answer this question however. All of this is taking place in the context of a python module I am creating.
In short, my question is: how can I allow Python linters (e.g. PyCharm or plugin for neovim) to lint objects that are created at runtime? "You can't" is not an answer ;). Of course there is always a way, with scripting and the like. I want to know what I would be looking at for the easiest way.
First I introduce my problem and the current approach I am taking. Second, I will describe what I want to do, and ask how.
Within this C library, a whole bunch of error codes are defined. I translated this information from the .h header file into a Python enum:
# CustomErrors.py
from enum import Enum
class CustomErrors(Enum):
ERROR_BROKEN = 1
ERROR_KAPUTT = 2
ERROR_BORKED = 3
Initially, my approach is to have a single exception class containing a type field which described the specific error:
# CustomException.py
from CustomErrors import CustomErrors
class CustomException(Exception):
def __init__(self, customErr):
assert type(customErr) is CustomError
self.type = customErr
super().__init__()
Then, as needed I can raise CustomException(CustomErrors.ERROR_KAPUTT).
Now, what I want to do is create a separate exception class corresponding to each of the enum items in CustomErrors. I believe it is possible to create types at runtime with MyException = type('MyException', (Exception,), {'__doc__' : 'Docstring for ABC class.'}).
I can create the exception classes at runtime like so:
#CustomException.py
from CustomErrors import CustomErrors
...
for ce in CustomErrors:
n = ce.name
vars()[n] = type(n, (Exception,), {'__doc__' : 'Docstring for {0:s} class.'.format(n)})
Note: the reason I want to create these at runtime is to avoid hard-coding of an Exception list that change in the future. I already have the problem of extracting the C enum automatically on the backburner.
This is all well and good, but I have a problem: static analysis cannot resolve the names of these exceptions defined in CustomException. This means PyCharm and other editors for Python will not be able to automatically resolve the names of the exceptions as a suggested autocomplete list when the user types CustomException.. This is not acceptable, as this is code for the end user, who will need to access the exception names for use in try-except constructs.
Here is the only solution I have been able to think of: writing a script which generates the .py files containing the exception names. I can do this using bash. Maybe people will tell me this is really the only option. But I would like to know what other approaches are suggested for solving this problem. Thanks for reading.
You can add a comment to tell mypy to ignore dynamically defined attribute errors. Perhaps the linters that you use share a similar way to silence such errors.
mypy docs on silencing errors based on error codes
This example shows how to ignore an error about an imported name mypy thinks is undefined:
# 'foo' is defined in 'foolib', even though mypy can't see the
# definition.
from foolib import foo # type: ignore[attr-defined]
Related
import util
class C():
save = util.save
setattr(C, 'load', util.load)
C.save is visible to the linter - but C.load isn't. There's thus some difference between assigning class methods from within the class itself, and from outside. Same deal for documentation builders; e.g. Sphinx won't acknowledge :meth:C.load - instead, need to do :func:util.load, which is misleading if load is meant to be C's method. An IDE (Spyder) also fails to "go to" method via self.load code.
The end-goal is to make linter (+docs & IDE) recognize load as C's method just like C.save is, but class method assignment needs to be dynamic (context). Can this be accomplished?
Note: the purpose of dynamic assignment is to automatically pull methods from modules (e.g. util) instead of having to manually update C upon method addition / removal.
Disclaimer: This solution does not work in all use cases, see comments. I leave it here, since it might still be useful under some circumstances.
I don't know about the linter and Sphinx, but an IDE like PyCharm will recognize the method if you declare it upfront using type hinting. In the code below, without the line load: Callable[[], None], I get the warning 'Unresolved attribute reference', but with the line there are no warnings in the file. Check the docs for more information about type hinting.
Notes:
Even with the more general load: Callable, the type checker is satisfied.
If you don't always declare a callable, a valid declaration is load: Optional[Callable] = None. This means that the default value is None. If you then call it without setting it, you will get an error, but you got that already anyway, that's unrelated to this typing.
p.s. I don't have your utils, so I defined some functions in the file itself.
from typing import Callable
def save():
pass
def load():
pass
class C:
load: Callable[[], None]
save = save
setattr(C, 'load', load)
C.load()
Short question: I have a module with objects. How can I do that if someone imports an object from my module, my specified exception is raised?
What I want to do: I write an architectural framework. A class for output depends on jinja2 external library. I want the framework to be usable without this dependency as well. In the package's __init__.py I write conditional import of my class RenderLaTeX (if jinja2 is available, the class is imported, otherwise not).
The problem with this approach is that I have some code which uses this class RenderLaTeX, but when I run it on a fresh setup, I receive an error like Import error: no class RenderLaTeX could be imported from output. This error is pretty unexpected and ununderstandable before I recall that jinja2 must be installed beforehand.
I thought about this solution: if the class is not available, __init__.py can create a string with this name. If a user tries to instantiate this object with the usual class constructor, they'll get a more meaningful error. Unfortunately, simple import
from output import RenderLaTeX
won't raise an error in this case.
What would you suggest, hopefully with the description of benefits and drawbacks?
Important UPD: I package my classes in modules and import them to the module via __init__.py, so that I import 'from lena.flow import ReadROOTFile', rather than 'from lena.flow.read_root_file import ReadROOTFile.'
When Python imports a module all of the code inside the file from which you are importing is executed.
If your RenderLaTeX class is therefore placed into a seperate file you can freely add logic which would prevent it from being imported (or run) if required dependencies are missing.
For example:
try:
import somethingidonthave
except ImportError:
raise Exception('You need this module!')
class RenderLaTeX(object):
pass
You can also add any custom message you want to the exception to better describe the error. This should work in both Python2 and Python3.
After a year of thinking, the solution appeared.
First of all, I think that this is pretty meaningless to overwrite an exception's type. The only good way would be to add a useful message for a missing import.
Second, I think that the syntax
from framework.renderers import MyRenderer
is really better than
from framework.renderers.my_renderer import MyRenderer
because it hides implementation details and requires less code from user (I updated my question to reflect that). For the former syntax to work, I have to import MyRenderer in __init__.py in the module.
Now, in my_renderer.py I would usually import third-party packages with
import huge_specific_library
in the header. This syntax is required by PEP 8. However, this would make the whole framework.renderers module depend on huge_specific_library.
The solution for that is to violate PEP 8 and import the library inside the class itself:
class MyRenderer():
def __init__(self):
import huge_specific_library
# ... use that...
Here you can catch the exception if that is important, change its message, etc. There is another benefit for that: there exist guides how to reduce import time, and they propose this solution (I read them a long time ago and forgot). Large modules require some time to be loaded. If you follow PEP 8 Style Guide (I still think that you usually should), this may lead to large delays just to make all imports to your program, not having done anything useful yet.
The only caveat is this: if you import the library in __init__, you should also import that to other class methods that use it, otherwise it won't be visible there.
For those who still doubt, I must add that since Python imports are cached, this doesn't affect performance if your method that uses import is not called too often.
Assume that class MyClass is sometimes, but not always, defined. I have a function foo(a=None) in which argument a can be None, a string, or an object of MyClass.
My question is: If MyClass is not defined in my Python session, how can I check the type of argument a in a fashion similar to isinstance without getting a NameError?
Note on duck-typing: I am deliberately limiting the function.
I'm using Python 2.6.x and Updating is not an option. A forward-compatible solution (especially for 2.7.x) is highly appreciated.
I would suggest a different approach: polyfill the class so all code that wants to refer to it can simply do so:
try:
from foo import Bar # load the native class
except ImportError:
class Bar:
pass # implement necessary parts here
You can put this into your own module and then from mymodule import Bar everywhere it's needed. That allows all your code to use Bar regardless of whether it's defined natively or not.
Even if redefining the class isn't your preferred way to handle this, handling the ImportError is still the way to handle this situation, since you will have to import the class either way and that's where the error will occur. Instead of defining the class, you may instead want to set a class_exists = False flag or something.
If MyClass isn't defined then you have no way to reference its type.
Therefore you can have no way to verify that type(a) has the correct value.
I workarounded the problem by overriding a method in MyClass and doing nothing in it (pass). After that I no longer needed to check its type.
Different workarounds may exist for different cases. Catching the NameError could be another one.
t = 'asdfas'
print(isinstance(t, str))
try:
print(isinstance(t, MyClass))
except NameError:
print(False)
Seems to me, that such a construct may appear in future python. Like typed python, which is quite new. And in typed python we have a possibility to use future types, in apos.
I have a module, errors.py in which several global constants are defined (note: I understand that Python doesn't have constants, but I've defined them by convention using UPPERCASE).
"""Indicates some unknown error."""
API_ERROR = 1
"""Indicates that the request was bad in some way."""
BAD_REQUEST = 2
"""Indicates that the request is missing required parameters."""
MISSING_PARAMS = 3
Using reStructuredText how can I document these constants? As you can see I've listed a docstring above them, but I haven't found any documentation that indicates to do that, I've just done it as a guess.
Unfortunately, variables (and constants) do not have docstrings. After all, the variable is just a name for an integer, and you wouldn't want to attach a docstring to the number 1 the way you would to a function or class object.
If you look at almost any module in the stdlib, like pickle, you will see that the only documentation they use is comments. And yes, that means that help(pickle) only shows this:
DATA
APPEND = b'a'
APPENDS = b'e'
…
… completely ignoring the comments. If you want your docs to show up in the built-in help, you have to add them to the module's docstring, which is not exactly ideal.
But Sphinx can do more than the built-in help can. You can configure it to extract the comments on the constants, or use autodata to do it semi-automatically. For example:
#: Indicates some unknown error.
API_ERROR = 1
Multiple #: lines before any assignment statement, or a single #: comment to the right of the statement, work effectively the same as docstrings on objects picked up by autodoc. Which includes handling inline rST, and auto-generating an rST header for the variable name; there's nothing extra you have to do to make that work.
As a side note, you may want to consider using an enum instead of separate constants like this. If you're not using Python 3.4 (which you probably aren't yet…), there's a backport.enum package for 3.2+, or flufl.enum (which is not identical, but it is similar, as it was the main inspiration for the stdlib module) for 2.6+.
Enum instances (not flufl.enum, but the stdlib/backport version) can even have docstrings:
class MyErrors(enum.Enum):
"""Indicates some unknown error."""
API_ERROR = 1
"""Indicates that the request was bad in some way."""
BAD_REQUEST = 2
"""Indicates that the request is missing required parameters."""
MISSING_PARAMS = 3
Although they unfortunately don't show up in help(MyErrors.MISSING_PARAMS), they are docstrings that Sphinx autodoc can pick up.
If you put a string after the variable, then sphinx will pick it up as the variable's documentation. I know it works because I do it all over the place. Like this:
FOO = 1
"""
Constant signifying foo.
Blah blah blah...
""" # pylint: disable=W0105
The pylint directive tells pylint to avoid flagging the documentation as being a statement with no effect.
This is an older question, but I noted that a relevant answer was missing.
Or you can just include a description of the constants in the docstring of the module via .. py:data::. That way the documentation is also made available via the interactive help. Sphinx will render this nicely.
"""
Docstring for my module.
.. data:: API_ERROR
Indicates some unknown error.
.. data:: BAD_REQUEST
Indicates that the request was bad in some way.
.. data:: MISSING_PARAMS
Indicates that the request is missing required parameters.
"""
You can use hash + colon to document attributes (class or module level).
#: Use this content as input for moo to do bar
MY_CONSTANT = "foo"
This will be picked up by some document generators.
An example here, could not find a better one: Sphinx document module properties
the following worked for me with Sphinx 2.4.4:
in foo.py :
API_ERROR = 1
"""int: Indicates some unknown error."""
then to document it:
.. automodule:: foo.py
:members:
I think you're out of luck here.
Python don't support directly docstrings on variables: there is no attribute that can be attached to variables and retrieved interactively like the __doc__ attribute on modules, classes and functions.
Source.
The Sphinx Napoleon Python documentation extension allows to document module-level variables in an Attributes section.
Per https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html :
Attributes
----------
module_level_variable1 : int
Module level variables may be documented in either the ``Attributes``
section of the module docstring, or in an inline docstring immediately
following the variable.
Either form is acceptable, but the two should not be mixed. Choose
one convention to document module level variables and be consistent
with it.
Writing only because I haven't seen this option in the answers so far:
You can also define your constants as functions that simply return the desired constant value when called, so for example:
def get_const_my_const() -> str:
"""Returns 'my_const'."""
return "my_const"
This way they'll be a bit "more constant" on one hand (less worrying about reassignment) and they'll also provide the opportunity for regular documentation, as with any other function.
I have a package __init__.py that looks something like this:
import sys
__required_python_version = (2,6, 0)
if sys.version_info < __required_python_version:
this_version = '.'.join([str(x) for x in sys.version_info[0:3]])
required_version = '.'.join([str(x) for x in __required_python_version])
raise PythonVersionError(this_version, required_version)
class PythonVersionError(Exception):
def __init__(self, this_version, required_version):
self.this_version = this_version
self.required_version = required_version
def __str__(self):
return 'Python version %s is invalid. Must be at least %s' % (self.this_ver, self.required_ver)
While I'm certain there is a more elegant way to format those version strings and I could probably get by using a standard exception, my real question is how would I do something like this? Would the best approach be to move my custom exception into a separate file and import it? Or should I wrap the version check in a function that executes when the __init__ is run? I'm just looking for recommendations on the preferred approach.
Thanks
Since it looks like you won't have any user for that exception --
unless this module is to be used by other modules you are impleemnting as part of a larger system, I say you don't need a custom exception here.
There is very little to gain from it, apart from the error message given. Ay module trying to import yours would have to be aware of it, to catch the exception, or just let the program stop witha backtrace. Since be aware of it , it would need to import your module, it would just crash to a backtrace anyway -- wher ethe user can then read the error message.
For one to read the error message,a plain "Exception" stating it is the incorrect PythonVersin is as good as any custom exception.
On the technical side, Python would need to know about PythonVersionError before raising it: you need to put that code before you try to raise it inside the if block.
And finally, if you are building a larger system, and other parts of the system might try to catch PythonVersionError, the coorect thing to do is to put it in its own file/module, so that it becomes available to this module that will raise it, and any other modules that are importing this.
There seems to be something awkward here.
Is it really usefull to create a custom Exception class when it won't be reused anywhere else in other modules ? If everyone did this we would end up with every module defining it's own different (and probably incompatible) PythonVersionError class.
Why don't you use a standard existing exception ? For this one I would probably go for a standard RuntimeError exception.
OK, I know you don't want this answer, but anyway.
If I really wanted to do this at least I would define PythonVersionException class as a local instance of checking code to avoid polluting module namespace or any global namespace of other files of the module.