Python's inspect module doesn't seem to be able to inspect the signatures of "built-in" functions, which include functions defined in C extension modules, like those defined by Cython. Is there any way to get the signature of a Python function you have defined in such a module, and specifically in Cython? I am looking to be able to find the available keyword arguments.
MWE:
# mwe.pyx
def example(a, b=None):
pass
and
import pyximport; pyximport.install()
import mwe
import inspect
inspect.signature(mwe.example)
yields:
Traceback (most recent call last):
File "mwe_py.py", line 5, in <module>
inspect.signature(mwe.example)
File "/nix/store/134l79vxb91w8mhxxkj6kb5llf7dmwpm-python3-3.4.5/lib/python3.4/inspect.py", line 2063, in signature
return _signature_internal(obj)
File "/nix/store/134l79vxb91w8mhxxkj6kb5llf7dmwpm-python3-3.4.5/lib/python3.4/inspect.py", line 1965, in _signature_internal
skip_bound_arg=skip_bound_arg)
File "/nix/store/134l79vxb91w8mhxxkj6kb5llf7dmwpm-python3-3.4.5/lib/python3.4/inspect.py", line 1890, in _signature_from_builtin
raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function example>
In Python 3.4.5 and Cython 0.24.1
I've retracted my duplicate suggestion (saying that it was impossible...) having investigated further. It seems to work fine with reasonably recent versions of Cython (v0.23.4) and Python 3.4.4.
import cython
import inspect
scope = cython.inline("""def f(a,*args,b=False): pass """)
print(inspect.getfullargspec(scope['f']))
gives the output
FullArgSpec(args=['a'], varargs='args', varkw=None, defaults=None, kwonlyargs=['b'], kwonlydefaults={'b': False}, annotations={})
Also mentioned in the documentation is the compilation option "binding" which apparently makes this detail more accessible (although I didn't need it).
I have a feeling that this may depend on improvements to inspect made relatively recently (possibly this fix) so if you're using Python 2 you're probably out of luck.
Edit: your example works if you using the binding compilation option:
import cython
#cython.binding(True)
def example(a, b=None):
pass
I suspect that inline adds it automatically (but the code to do inline is sufficiently convoluted that I can't find proof of that either way). You can also set it as a file-level option.
The answer above using the binding decorator works for me when running code that has been cythonized. But, when I was running that same code within a Django 2.2 app, the application would fail on start with an error that cython has no attribute 'binding'. To avoid this I have added this "special cython header" at the top of my file containing the cythonized function as documented here to achieve the same results.
# cython: binding=True
def example(a, b=None):
pass
Related
Is there any advantage to using the 'type hint' notation in python?
import sys
def parse(arg_line: int) -> str:
print (arg_line) # passing a string, returning None
if __name__ == '__main__':
parse(' '.join(sys.argv[1:]))
To me it seems like it complicates the syntax without providing any actual benefit (outside of perhaps within a development environment). Based on this:
Are there any plans for python to contain type constraints within the language itself?
What is the advantage of having a "type hint" ? Couldn't I just as easily throw that into the docstring or something?
I also don't see this much in the python codebase itself as far as I can tell -- most types are enforced manually, for example: argparse.py and any other files I've glanced at in https://github.com/python/cpython/blob/3.7/Lib/.
Are there any plans for python to contain type constraints within the language itself?
Almost certainly not, and definitely not before the next major version.
What is the advantage of having a "type hint" ? Couldn't I just as easily throw that into the docstring or something?
Off the top of my head, consider the following:
Type hints can be verified with tooling like mypy.
Type hints can be used by IDEs and other tooling to give hints and tips. E.g., when you're calling a function and you've just written foo(, the IDE can pick up on the type hints and display a box nearby that shows foo(x: int, y: List[int]). The advantage to you as a developer is that you have exactly the information you need exposed to you and don't have to munge an entire docstring.
Type hints can be used by modules like functools.singledispatch or external libraries like multipledispatch to add additional type-related features (in this case, dispatching function calls based on name and type, not just name).
One option to take advantage of type hints is the type_enforced module. Regarding official python support, it still seems unlikely that types hints will be enforced directly in the near future.
Going into type_enforced, the package allows you to take advantage of type hints. It supports both input and output typing. Only types that are specified are enforced. Multiple possible inputs are also supported so you can specify something like int or float.
Input types are first validated (lazily on the function call) and if valid, the function is processed where the return value is then validated.
There are some limitations such that nested type structures are not supported. For example you can not specify type as a list of integers, but only a list. You would need to validate the items in the list inside of your function.
pip install type_enforced
>>> import type_enforced
>>> #type_enforced.Enforcer
... def my_fn(a: int , b: [int, str] =2, c: int =3) -> None:
... pass
...
>>> my_fn(a=1, b=2, c=3)
>>> my_fn(a=1, b='2', c=3)
>>> my_fn(a='a', b=2, c=3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 47, in __call__
return self.__validate_types__(*args, **kwargs)
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 83, in __validate_types__
self.__check_type__(assigned_vars.get(key), value, key)
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 56, in __check_type__
self.__exception__(
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 37, in __exception__
raise TypeError(f"({self.__fn__.__qualname__}): {message}")
TypeError: (my_fn): Type mismatch for typed function (my_fn) with `a`. Expected one of the following `[<class 'int'>]` but got `<class 'str'>` instead.
Is there any advantage to using the 'type hint' notation in python?
import sys
def parse(arg_line: int) -> str:
print (arg_line) # passing a string, returning None
if __name__ == '__main__':
parse(' '.join(sys.argv[1:]))
To me it seems like it complicates the syntax without providing any actual benefit (outside of perhaps within a development environment). Based on this:
Are there any plans for python to contain type constraints within the language itself?
What is the advantage of having a "type hint" ? Couldn't I just as easily throw that into the docstring or something?
I also don't see this much in the python codebase itself as far as I can tell -- most types are enforced manually, for example: argparse.py and any other files I've glanced at in https://github.com/python/cpython/blob/3.7/Lib/.
Are there any plans for python to contain type constraints within the language itself?
Almost certainly not, and definitely not before the next major version.
What is the advantage of having a "type hint" ? Couldn't I just as easily throw that into the docstring or something?
Off the top of my head, consider the following:
Type hints can be verified with tooling like mypy.
Type hints can be used by IDEs and other tooling to give hints and tips. E.g., when you're calling a function and you've just written foo(, the IDE can pick up on the type hints and display a box nearby that shows foo(x: int, y: List[int]). The advantage to you as a developer is that you have exactly the information you need exposed to you and don't have to munge an entire docstring.
Type hints can be used by modules like functools.singledispatch or external libraries like multipledispatch to add additional type-related features (in this case, dispatching function calls based on name and type, not just name).
One option to take advantage of type hints is the type_enforced module. Regarding official python support, it still seems unlikely that types hints will be enforced directly in the near future.
Going into type_enforced, the package allows you to take advantage of type hints. It supports both input and output typing. Only types that are specified are enforced. Multiple possible inputs are also supported so you can specify something like int or float.
Input types are first validated (lazily on the function call) and if valid, the function is processed where the return value is then validated.
There are some limitations such that nested type structures are not supported. For example you can not specify type as a list of integers, but only a list. You would need to validate the items in the list inside of your function.
pip install type_enforced
>>> import type_enforced
>>> #type_enforced.Enforcer
... def my_fn(a: int , b: [int, str] =2, c: int =3) -> None:
... pass
...
>>> my_fn(a=1, b=2, c=3)
>>> my_fn(a=1, b='2', c=3)
>>> my_fn(a='a', b=2, c=3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 47, in __call__
return self.__validate_types__(*args, **kwargs)
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 83, in __validate_types__
self.__check_type__(assigned_vars.get(key), value, key)
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 56, in __check_type__
self.__exception__(
File "/home/conmak/development/personal/type_enforced/type_enforced/enforcer.py", line 37, in __exception__
raise TypeError(f"({self.__fn__.__qualname__}): {message}")
TypeError: (my_fn): Type mismatch for typed function (my_fn) with `a`. Expected one of the following `[<class 'int'>]` but got `<class 'str'>` instead.
The attr's package somehow ruins pytorch's parameter() method for a module. I am wondering if anyone has any work-arounds or solutions, so that the two packages can seamlessly integrate?
If not, any advice on which github to post the issue to? My instinct would be to post this onto attr's github, but the stack trace is almost entirely relevant to pytorch's codebase.
Python 3.7.3
attrs== 19.1.0
torch==1.1.0.post2
torchvision==0.3.0
import attr
import torch
class RegularModule(torch.nn.Module):
pass
#attr.s
class AttrsModule(torch.nn.Module):
pass
module = RegularModule()
print(list(module.parameters()))
module = AttrsModule()
print(list(module.parameters()))
The actual output is:
$python attrs_pytorch.py
[]
Traceback (most recent call last):
File "attrs_pytorch.py", line 18, in <module>
print(list(module.parameters()))
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 814, in parameters
for name, param in self.named_parameters(recurse=recurse):
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 840, in named_parameters
for elem in gen:
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 784, in _named_members
for module_prefix, module in modules:
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 975, in named_modules
if self not in memo:
TypeError: unhashable type: 'AttrsModule'
The expected output is:
$python attrs_pytorch.py
[]
[]
You may get it to work with one workaround and using dataclasses (which you should, as it's in standard Python library since 3.7 which you are apparently using). Though I think simple __init__ is more readable. One could do something similar using attrs library (disabling hashing), I just prefer the solution using standard libraries if possible.
The reason (if you manage to handle hashing related errors) is that you are calling torch.nn.Module.__init__() which generates _parameters attribute and other framework-specific data.
First solving hashing with dataclasses:
#dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
pass
This solves hashing issues as, as stated by the documentation, section about hash and eq:
By default, dataclass() will not implicitly add a hash() method
unless it is safe to do so.
which is needed by PyTorch so the model can be used in C++ backed (correct me if I'm wrong), furthermore:
If eq is false, hash() will be left untouched meaning the
hash() method of the superclass will be used (if the superclass is object, this means it will fall back to id-based hashing).
So you are fine using torch.nn.Module __hash__ function (refer to documentation of dataclasses if any further errors arise).
This leaves you with the error:
AttributeError: 'AttrsModule' object has no attribute '_parameters'
Because torch.nn.Module constructor is not called. Quick and dirty fix:
#dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
def __post_init__(self):
super().__init__()
__post_init__ is a function called after __init__ (who would of guessed), where you can initialize torch-specific parameters.
Still, I would advise against using those two modules together. For example, you are destroying PyTorch's __repr__ using your code, so repr=False should be passed to the dataclasses.dataclass constructor, which gives this final code (obvious collisions between libraries eliminated I hope):
import dataclasses
import torch
class RegularModule(torch.nn.Module):
pass
#dataclasses.dataclass(eq=False, repr=False)
class AttrsModule(torch.nn.Module):
def __post_init__(self):
super().__init__()
module = RegularModule()
print(list(module.parameters()))
module = AttrsModule()
print(list(module.parameters()))
For more on attrs please see hynek answer and his blog post.
attrs has a chapter on hashability that also explains the pitfalls of hashing in Python: https://www.attrs.org/en/stable/hashing.html
You’ll have to decide what behavior is adequate for your concrete problem. For more general information check out https://hynek.me/articles/hashes-and-equality/ — turns out hashing is surprisingly tricky in Python.
Most of the tutorials and books about Django or Flask import specific classes from files instead of importing the whole file.
For example, importing DataRequiered validator from wrtforms.validators is done via from wtforms import validators instead of importing it via import wtforms.validators as valids and then accessing DataRequiered with valids.DataRequiered.
My question is: Is there an reason for this ?
I thought to something like avoiding the loading a whole module for computation/memory optimization (is it really relevant?) ? Or is it simply to make the code more readable ?
My question is: Is there an reason for this ?
from module_or_package import something is the canonical pythonic idiom (when you only want to import something in your current namespace of course).
Also, import module_or_package.something only works if module_or_package is a package and something a submodule, it raises an ImportError(No module named something) if something is a function, class or whatever object defined in module_or_package, as can be seen in the stdlib with os.path (which is a submodule of the os.package) vs datetime.date (which is a class defined in the datetime module):
>>> import os.path as p
>>> p
<module 'posixpath' from '/home/bruno/.virtualenvs/blook/lib/python2.7/posixpath.pyc'>
vs
>>>import datetime.date as d
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named date
thought to something like avoiding the loading a whole module for computation/memory optimization (is it really relevant?)
Totally irrelevant - importing a given name from a module requires importing the whole module. Actually, this:
from module_or_package import something
is only syntactic sugar for
import module_or_package
something = module_or_package.something
del module_or_package
EDIT: You mention in a comment that
Right, but importing the whole module means loading it to the memory, which can be a reason for importing only a submodule/class
so it seems I failed to make the point clear: in Python, you can not "import only a submodule/class", period.
In Python, import, class and def are all executable statements (and actually just syntactic sugar for operation you can do 'manually' with functions and classes). Importing a module actually consists in executing all the code at the module's top-level (which will instanciate function and class objects) and create a module object (instance of module type) which attributes will be all names defined at the top-level via import, def and class statements or via explicit assignment. It's only when all this has been done that you can access any name defined in the module, and this is why, as I stated above,
from module import obj
is only syntactic sugar for
import module
obj = module.obj
del module
But (unless you do something stupid like defining a terabyte-huge dict or list in your module) this doesn't actually take that much time nor eat much ram, and a module is only effectively executed once per process the first time it's imported - then it's cached in sys.modules so subsequent imports only fetch it from cache.
Also, unless you actively prevents it, Python will cache the compiled version of the module (the .pyc files) and only recompile it if the .pyc is missing or older than the source .py file.
wrt/ packages and submodules, importing a submodule will also execute the package's __init__.py and build a module instance from it (IOW, at runtime, a package is also a module). Package initializer are canonically rather short, and actually quite often empty FWIW...
It depends, in the tutorial that was probably done for readability
Usually if you use most of the classes in a file, you import the file. If the files contains many classes but you only need a few, just import those.
It's both a matter of readability and optimization.
I have a problem similar to the first problem in this question, which as far as I can see went unanswered.
I have a file "config.py" which contains a lot of parameters to be used by a class (this config.py file will change), however I can't get these to propagate into the class via execfile.
In an example piece of code:
class Class():
def __init__(self):
execfile("config.py")
print x
# config.py
x = "foo"
>>> t = Class()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __init__
NameError: global name 'x' is not defined
Any help welcome, or any better methods of retrieving parameters from a file to be used in a class.
Many Thanks.
I don't get what you're trying to do (but i don't like it, and this is just me) but to fix your problem do (test in python2.6):
class Class():
def __init__(self):
execfile('config.py', locals()) # Not recommanded, maybe you want globals().
print x
But from the doc:
Note
The default locals act as described
for function locals() below:
modifications to the default locals
dictionary should not be attempted.
Pass an explicit locals dictionary if
you need to see effects of the code on
locals after function execfile()
returns. execfile() cannot be used
reliably to modify a function’s
locals.
and about :
Any help welcome, or any better
methods of retrieving parameters from
a file to be used in a class.
You can use import.
Even though it might be convenient to keep configuration settings in a Python file I would recommend against it. I think it opens up a whole set of problems that you don't really want to have do deal with. Anything could be placed in your configuration file, including malicious code.
I would use either the json module or the ConfigParser module to hold my configuration.
If you have trouble choosing between those two I would recommend the json module. Json is a simple yet flexible format for structured data.