I'm trying to generate some JavaScript based on the type annotations I have provided in on some Python functions by using the signature() function in the inspect module.
This part works as I expect when the type is a simple builtin class:
import inspect
def my_function() -> dict:
pass
signature = inspect.signature(my_function)
signature.return_annotation is dict # True
Though I'm not sure how to unwrap and inspect more complex annotations e.g:
from typing import List
import inspect
def my_function() -> List[int]:
pass
signature = inspect.signature(my_function)
signature.return_annotation is List[int] # False
Again similar problem with forward referencing a custom class:
def my_function() -> List['User']:
pass
...
signature.return_annotation # typing.List[_ForwardRef('User')]
What I'm looking to get out is something like this - so I can branch appropriately while generating the JavaScript:
type = signature.return_annotation... # list
member_type = signature.return_annotation... # int / 'User'
Thanks.
Python 3.8 provides typing.get_origin() and typing.get_args() for this!
assert get_origin(Dict[str, int]) is dict
assert get_args(Dict[int, str]) == (int, str)
assert get_origin(Union[int, str]) is Union
assert get_args(Union[int, str]) == (int, str)
See https://docs.python.org/3/library/typing.html#typing.get_origin
List is not a map of types to GenericMeta, despite the syntax. Each access to it generates a new instance:
>>> [ id(List[str]) for i in range(3) ]
[33105112, 33106872, 33046936]
This means that even List[int] is not List[int]. To compare two instances, you have multiple options:
Use ==, i.e., signature.return_annotation == List[int].
Store an instance of your type in a global variable and check against that, i.e.,
a = List[int]
def foo() -> a:
pass
inspect.signature(foo).return_annotation is a
Use issubclass. The typing module defines that. Note that this might do more than you'd like, make sure to read the _TypeAlias documentation if you use this.
Check against List only and read the contents yourself. Though the property is internal, it is unlikely that the implementation will change soon: List[int].__args__[0] contains the type argument starting from Python 3.5.2, and in earlier versions, its List[int].__parameters__[0].
If you'd like to write generic code for your exporter, then the last option is probably best. If you only need to cover a specific use case, I'd personally go with using ==.
Take note, this applies to Python 3.5.1
For Python 3.5.2 take a look at phillip's answer.
You shouldn't be checking with the identity operator as Phillip stated, use equality to get this right.
To check if a hint is a subclass of a list you could use issubclass checks (even though you should take note that this can be quirky in certain cases and is currently worked on):
issubclass(List[int], list) # True
To get the members of a type hint you generally have two watch out for the cases involved.
If it has a simple type, as in List[int] the value of the argument is located in the __parameters__ value:
signature.return_annotation.__parameters__[0] # int
Now, in more complex scenarios i.e a class supplied as an argument with List[User] you must again extract the __parameter__[0] and then get the __forward_arg__. This is because Python wraps the argument in a special ForwardRef class:
d = signature.return_annotation.__parameter__[0]
d.__forward_arg__ # 'User'
Take note, you don't need to actually use inspect here, typing has a helper function named get_type_hints that returns the type hints as a dictionary (it uses the function objects __annotations__ attribute).
Related
I'm trying to generate some JavaScript based on the type annotations I have provided in on some Python functions by using the signature() function in the inspect module.
This part works as I expect when the type is a simple builtin class:
import inspect
def my_function() -> dict:
pass
signature = inspect.signature(my_function)
signature.return_annotation is dict # True
Though I'm not sure how to unwrap and inspect more complex annotations e.g:
from typing import List
import inspect
def my_function() -> List[int]:
pass
signature = inspect.signature(my_function)
signature.return_annotation is List[int] # False
Again similar problem with forward referencing a custom class:
def my_function() -> List['User']:
pass
...
signature.return_annotation # typing.List[_ForwardRef('User')]
What I'm looking to get out is something like this - so I can branch appropriately while generating the JavaScript:
type = signature.return_annotation... # list
member_type = signature.return_annotation... # int / 'User'
Thanks.
Python 3.8 provides typing.get_origin() and typing.get_args() for this!
assert get_origin(Dict[str, int]) is dict
assert get_args(Dict[int, str]) == (int, str)
assert get_origin(Union[int, str]) is Union
assert get_args(Union[int, str]) == (int, str)
See https://docs.python.org/3/library/typing.html#typing.get_origin
List is not a map of types to GenericMeta, despite the syntax. Each access to it generates a new instance:
>>> [ id(List[str]) for i in range(3) ]
[33105112, 33106872, 33046936]
This means that even List[int] is not List[int]. To compare two instances, you have multiple options:
Use ==, i.e., signature.return_annotation == List[int].
Store an instance of your type in a global variable and check against that, i.e.,
a = List[int]
def foo() -> a:
pass
inspect.signature(foo).return_annotation is a
Use issubclass. The typing module defines that. Note that this might do more than you'd like, make sure to read the _TypeAlias documentation if you use this.
Check against List only and read the contents yourself. Though the property is internal, it is unlikely that the implementation will change soon: List[int].__args__[0] contains the type argument starting from Python 3.5.2, and in earlier versions, its List[int].__parameters__[0].
If you'd like to write generic code for your exporter, then the last option is probably best. If you only need to cover a specific use case, I'd personally go with using ==.
Take note, this applies to Python 3.5.1
For Python 3.5.2 take a look at phillip's answer.
You shouldn't be checking with the identity operator as Phillip stated, use equality to get this right.
To check if a hint is a subclass of a list you could use issubclass checks (even though you should take note that this can be quirky in certain cases and is currently worked on):
issubclass(List[int], list) # True
To get the members of a type hint you generally have two watch out for the cases involved.
If it has a simple type, as in List[int] the value of the argument is located in the __parameters__ value:
signature.return_annotation.__parameters__[0] # int
Now, in more complex scenarios i.e a class supplied as an argument with List[User] you must again extract the __parameter__[0] and then get the __forward_arg__. This is because Python wraps the argument in a special ForwardRef class:
d = signature.return_annotation.__parameter__[0]
d.__forward_arg__ # 'User'
Take note, you don't need to actually use inspect here, typing has a helper function named get_type_hints that returns the type hints as a dictionary (it uses the function objects __annotations__ attribute).
Say I have a function, that accepts a type object, and returns some value of this type. Like this:
>>> from typing import Any, List
>>> def read(T: type) -> Any: # can we somehow declare T instead of Any?
... # generating and returning value of type T
...
>>> read(str)
This is a string
>>> read(int)
13
>>> read(List[float])
[1.2, 3.4]
etc. (please note, that it is not limited to primitive types like int, str, list etc. but also works with objects of specific classes and more).
Is it possible to specify a concrete dynamic return type (instead of Any)? So that, for example, instead of s: str = read(str) I could write s = read(str), and my IDE would type-hint s as a string?
P.S. In my case, the IDE is JetBrains PyCharm, but would be nice to have a "general" solution. Any Python versions and modules are welcome.
Solution, inspired by #jonrsharpe's comment:
from typing import Type, TypeVar
T = TypeVar('T')
def read(t: Type[T]) -> T: ...
In my Python 3.7.4 code I have the following function.
def is_dict(klass: ???) -> bool:
return klass.__origin__ == dict
I'm struggling to get the type annotation right for the klass parameter. It's not type, for that mypy complains.
error: "type" has no attribute "__origin__"
I'm at a loss. What is the correct annotation and is there any good documentation about it?
Here's an example of how this function is used:
>>> is_dict(typing.Dict[str, int])
True
>>> is_dict(typing.List[str])
False
If you want to use something like GenericMeta as suggested in the comments, or try experimenting with creating a custom protocol that defines the appropriate __origin__ attribute, you'll first need to update the type hints for the typing module in typeshed to define these attributes.
However, for now, I recommend using just Type[Any] or Type[object].
from typing import Type, Any
# If you want to make minimal modifications to your code, make
# 'klass' dynamically typed.
def is_dict_1(klass: Type[Any]) -> bool:
return klass.__origin__ == dict
# If you want to make your code robust, assume nothing about 'klass'
# except that it's some sort of type and verify the attribute you're
# about to use exists.
def is_dict_2(klass: Type[object]) -> bool:
return getattr(klass, '__origin__', None) == dict
If you're specifically trying to directly manipulate type hint expressions because you're trying to create some kind of serialization/deserialization library, you could also try taking a look at the source code of libraries like pydantic for inspiration.
A little more broadly, I'd also recommend you explore the possibility of minimizing places in your code where you manipulate type hint expressions as runtime entities if possible. The Python typing ecosystem was largely designed to keep the static and runtime worlds separate, so mechanisms for intermingling the two worlds aren't really that convenient to use and aren't always backwards compatible. The typing library internals have changed several times since it was first released in Python 3.5, for example.
I have edited the answer:
import typing
def is_dict(klass: typing.GenericMeta) -> bool:
return klass.__origin__ == typing.Dict
if __name__ == "__main__":
print(is_dict(typing.Dict[str, int]))
print(is_dict(typing.List[str]))
Alternatively:
def is_dict(klass: typing.GenericMeta) -> bool:
return klass.__orig_bases__[0] == dict
My understanding is that you should use cls: type in this case, because it will accept types like str or int, but also typing.Dict, typing.List, etc. from a type level perspective.
You can easily get around mypy's complaint "type" has no attribute "__origin__" by using hasattr / getattr. The following should work on Python 3.6+ in terms of both type checking and runtime behavior.
from typing import Dict, List
def is_dict(cls: type) -> bool:
if cls is dict:
# Add this depending on whether you want to accept plain `dict`
return True
elif hasattr(cls, "__origin__"):
origin = getattr(cls, "__origin__")
# In Python 3.7+ origin will be `dict`, in Python 3.6 it is `Dict`
return origin == dict or origin == Dict
return False
assert not is_dict(int)
assert not is_dict(str)
assert not is_dict(List[int])
assert is_dict(dict)
assert is_dict(Dict[int, int])
In Python's typing module, it seems that anything that is not a generic type does not support subscripting:
>>> import typing
>>> from yarl import URL
>>> StrOrURL = typing.Union[URL, str]
>>> typing.List[StrOrURL] # OK; List is generic
typing.List[typing.Union[yarl.URL, str]]
>>> SetOrDict = typing.Union[set, dict]
>>> SetOrDict[StrOrURL] # Raises
TypeError: typing.Union[set, dict] is not a generic class
This will be the case whether SetOrDict is defined with typing.Union or typing.TypeVar. (In Python 3.7, at least.)
It looks like there is a way to subclass Generic, presumably like:
class SetOrDict(typing.Generic[StrOrURL]):
# ...
But this will immediately raise TypeError: Parameters to Generic[...] must all be type variables.
Is there a way to accommodate the above? I.e., what is the recommended practice for subscripting user-defined (non-generic) typedefs?
You can't use a Union[...], alias or otherwise, as a generic type, no. Your union contents are also not generic, you can't state what types a dict or set object can contain by using dict or set directly, you'd use Set or Dict.
You have a new union:
URLorStrSetOrDict = typing.Union[Set[StrOrUrl], Dict[StrOrUrl, Any]]
Note that a dictionary has keys and values, you need to provide information on both. If this used as the input for an API, then consider using the more generic and immutable AbstractSet and Mapping types; this makes it clear that your API only needs to read information.
Personally, I'd look hard at refactoring a codebase that is this muddy about what kinds of objects it can accept. Requiring that an API accepts only sets and existing code has to pass in set(dictionary) instead of dictionary is not an arduous requirement, or perhaps the API really would work with any iterable of StrOrUrl.
Does the typing module (or any other module) exhibit an API to typecheck a variable at runtime, similar to isinstance() but understanding the type classes defined in typing?
I'd like to be to run something akin to:
from typing import List
assert isinstance([1, 'bob'], List[int]), 'Wrong type'
I was looking for something similar and found the library typeguard. This can automatically do runtime type checks wherever you want. Checking types directly as in the question is also supported. From the docs,
from typeguard import check_type
# Raises TypeError if there's a problem
check_type('variablename', [1234], List[int])
There is no such function in the typing module, and most likely there won't ever be.
Checking whether an object is an instance of a class - which only means "this object was created by the class' constructor" - is a simple matter of testing some tagging.
However, checking whether an object is an "instance" of a type is not necessarily decidable:
assert isinstance(foo, Callable[[int], str]), 'Wrong type'
Although it is easy to inspect the typing annotations of foo (assuming it's not a lambda), checking whether it complies to them is generally undecidable, by Rice's theorem.
Even with simpler types, such as List[int] the test will easily become far too inefficient to be used for anything but the smallest toy examples.
xs = set(range(10000))
xs.add("a")
xs.pop()
assert isinstance(xs, Set[int]), 'Wrong type'
The trick that allows type checker to perform this operation in a relatively efficient way, is to be conservative: the type checker tries to prove that foo always return int. If it fails, it rejects the program, even though the program may be valid, i.e. this function is likely to be rejected, although it is perfectly safe:
def foo() -> int:
if "a".startswith("a"):
return 1
return "x"
This is what I have discovered recently, basically this decorator does type checking at runtime raising exception if some type definition did not match. It can also do type checking for nested types (dict of strings, etc)
https://github.com/FelixTheC/strongtyping
Example:
from strongtyping.strong_typing import match_typing
#match_typing
def func_a(a: str, b: int, c: list):
...
func_a('1', 2, [i for i in range(5)])
# >>> True
func_a(1, 2, [i for i in range(5)])
# >>> will raise a TypeMismatch Exception