I'm new to Python development, and trying to get a fix on things. I am using Pycharm for development. I am currently trying to annotate the types of variables for easier access with autocompletion and suggestions. I've tried iterations of the code with mixed results.
This is the code that has a problem:
path = os.path.dirname(os.path.realpath(__file__)) # type: str
components = path.split(os.sep) # type: list[str]
The first problem displayed is at the opening brace of the type annotation of the second line. It says:
Class 'type' does not define '__getitem__', so the '[]' operator cannot be used on its instances.
I've googled around, and although the problem seems clear, opening the code for the list class clearly shows a method __getitem__:
class list(object):
"""
list() -> new empty list
list(iterable) -> new list initialized from iterable's items
"""
....
def __getitem__(self, y): # real signature unknown; restored from __doc__
""" x.__getitem__(y) <==> x[y] """
pass
Okay, maybe that's not trivial to understand and there's some other loading mechanism at work. And besides, the "problem" seems to be that I used list[str] and not List[str]. So I altered the code:
path = os.path.dirname(os.path.realpath(__file__)) # type: str
components = path.split(os.sep) # type: List[str]
Now everything breaks: The second line now complains about this:
Expected type 'List[str]', got 'List[str]' instead`
The previous problem regarding __getitem__ yet persists.
Is there a way to annotate these variables without causing problems for the checker? I am not very happy with the Python documentation in that regard, not explicitly stating the return types of their built-in methods. I have to rely on the information Pycharm provides in the documentation pop-ups (Ctrl+q).
It was a right solution to use List[str] instead of list[str] since builtin types could not be used in type hints, the corresponding PEP has not been accepted yet.
From what module do you import List? I'm unable to reproduce an issue in 2019.3.
Related
I've been trying to tighten up a few classes of code with type hinting. I'm struggling with a nesting problem. The parameter of a function of interest is an iterable container of objects with known types. I'm having trouble getting my IDE to recognize the inner types (which is a nice convenience.)
I'm using python 3.9 on PyCharm, but I'm getting similar auto-complete results in Ipython.
When using the Iterable class out of typing module, the IDE cannot "see through" to the inner types. Similarly for Collection. But it can when using either list or tuple on the outer container.
Is this an IDE issue or is there another way to package this? It would be nice to be able to send this function any type of iterable, rather than hard code it...
from datetime import datetime
from typing import Iterable
Data_collection = Iterable[tuple[datetime, str]]
Data_list = list[tuple[datetime, str]]
def foo(bar: Data_collection):
bar[1][0]. # no type-hint on inner obj
def foo2(bar: Data_list):
bar[1][0]. # good type-hint on inner obj
Gets me (PyCharm) results like this:
You are using the wrong abstract base class. Iterable only promises/requires __iter__ to be implemented, not __getitem__. Statically speaking, bar[1] isn't guaranteed to be defined.
To specify any type that supports indexing, use Sequence instead.
from typing import Sequence
Data_collection = Sequence[tuple[datetime, str]]
Now, regardless of the runtime type of bar, your IDE can assume that bar.__getitem__ is defined and returns a tuple. (Whether your specific IDE does make that assumption depends on the IDE, not your code.)
I am trying to figure out how can I provide type hints for a dictionary argument being passed to a function without using Dict[str, str] as that doesnt provide what the keys will be.
I have tried two approaches so far, one with using typing_extensions so that I can have compatibility with 3.6, and also with pydantic, but I cant get the hits to show.
Consider this example code:
from typing_extensions import TypedDict
from pydantic import BaseModel
class Some(TypedDict):
"""keya is some key"""
keya: str
"""another_key is another key"""
another_key: str
def some(a: Some) -> None:
print(a.get('keya'))
return None
some({'keya': 'key', 'another_key': 'nonething'})
As expected, the type hints for the some function shows the type Some, but not its keys.
What I am trying to accomplish is 2 things.
Provide key hints when the function argument is a dict (most important)
Generate documentation with sphinx so that the keys reflect in the documentation.
Edit
As on of the comments pointed out, I can accomplish this with **kwargs to some extent, but that is not the intention. Setting **kwargs does not give me type hints either.
I think in this case, it might actually be better to file a feature request/submit a pull request to your editor improving the quality of its type hinting. Similarly, with sphinx, you could submit a pull request that ensures that the docs either properly link to the definition of Some or include a more detailed description within the function signature itself.
After all, the problem you're facing is a limitation of your editor/sphinx, not with Python or type hints, and you might get better long-term results if you tackle the problem at the source.
You may also get better results if you use the "constructor" of Some instead of passing in a dict literal. At least for me, doing this gets me full key hints for Some when using PyCharm. Not sure if that'll also be the case for your editor:
some(Some(keya='key', another_key='nonething'))
Note that doing Some(...) will actually just return a plain old regular dict at runtime, so this shouldn't lead to any difference in behavior.
It could also be worth trying to do:
x: Some = {
"keya": "key",
"another_key": "nonething",
}
some(x)
...to see if your editor might give better hints with that form.
I have somewhat narrowed down to a possible solution using the following code. It meets most of the requirements:
mypy type checks passes
shows keys
documentation also shows keys and types
The main caviate to this solution is that mypy thinks the value is optional because of a default value so the solution is not quite complete.
Using the validate_items function, I can validate that the values are there. Please see the comments on the code snippet and offer suggestions.
from typing_extensions import TypedDict
from typing import Union
class Some(TypedDict):
keya: str
another_key: str
def validate_items(v: dict) -> None:
for key, value in v.items():
assert isinstance(value,str), '{} is required'.format(key)
# Would love to pull the type of a key from the Some class here or
# would love to put the validation in the Some class itself
def some(a: Some={'keya': '', 'another_key': ''}) -> None:
"""[summary]
Args:
a (Some, optional): [description]. Defaults to {'keya': '', 'another_key': ''}.
Returns:
[type]: [description]
"""
validate_items(dict(a))
print(a.get('keya'))
return None
In the screenshot, I can see that mypy is complaining about the None value which is expected, and in the popup help, we can also see the keys that are required in the dictionary being passed along with the type that is being set to it.
The solution feels quite hacky, and would appreciate any corrections to make it more pythonic.
I recently started using type hints in my code, and have so far found them to be (mostly) very helpful.
However, one thing that I really do not like is the syntax to force the type checker to assume that a variable is of a certain type. Given this example:
import itertools
from typing import Iterable, Tuple
x: Iterable[Tuple[str, str]] = itertools.combinations('abc', 2)
# error: Incompatible types in assignment (expression has type "Iterable[Tuple[str, ...]]", variable has type "List[Tuple[str, str]]")
As far as I can tell, the recommended way to work around this is to explicitly cast the object to force the type checker to use the specified type, e.g.:
import itertools
from typing import Iterable, Tuple, cast
x = cast(Iterable[Tuple[str, str]], itertools.combinations('abc', 2))
I personally find this solution to be a bit gross. My primary concern is that, to the inexperienced reader, it is not clear that the cast is purely there to help the static analyzer. (If I didn't already know, I would assume based on the name and context that it is converting and doing a copy into an object of the specified type, when really there is no runtime cost.)
cast looks like any old function call. When I see that a function is being called on a value, I expect the value to be mutated and/or some other side-effects to occur, but in this case the only side effect is that mypy stops complaining. Type hints themselves have a distinct syntax, but I feel that this blurs the lines with a mixture of the new typing syntax and traditional python syntax. (It's already a bit blurry since you have to import the types and can compose them, but that's another discussion.)
Is there an alternative syntax for cast-like behavior? I haven't found anything, but I was hoping for something like:
x1 = itertools.combinations('abc', 2)) # cast: Iterable[Tuple[str, str]]
x2: Iterable[Tuple[str, str]] = itertools.combinations('abc', 2)) # type: cast
x3: Cast[Iterable[Tuple[str, str]]] = itertools.combinations('abc', 2))
Actually the latest version of Mypy does return the correct type Iterator[Tuple[str, str]].
This change was introduced to Typeshed in PR https://github.com/python/typeshed/pull/4309.
If you cannot update mypy to the latest version you can checkout the latest version from typeshed and use the config option custom_typeshed_dir.
See https://mypy.readthedocs.io/en/stable/config_file.html#confval-custom_typeshed_dir for more details.
I am trying to use a factory function to generate some type annotations – specifically for tuple types. I have one version of the factory that works fine (as in, it compiles, runs, and checks out satisfactorily in MyPy):
import typing as tx
HomogenousTypeVar = tx.TypeVar('HomogenousTypeVar')
TupleTypeReturnType = tx.Type[tx.Tuple[HomogenousTypeVar, ...]]
def TupleType(length: int,
tuptyp: tx.Type[HomogenousTypeVar] = str) -> TupleTypeReturnType:
""" Create a type annotation for a tuple of a given type and length """
assert length > 0
return tx.Tuple[tuple(tuptyp for idx in range(length))]
… for which the usage is like e.g.:
class Thing(object):
__slots__: TupleType(2) = ('yo', 'dogg')
other_fields: TupleType(4) = ('i', 'heard',
'you', 'like')
# etc, or what have you
… however, I was unsuccessful when I tried to add support for the typing.ClassVar annotation, which looked like this:
import typing as tx
HomogenousTypeVar = tx.TypeVar('HomogenousTypeVar')
TupleTypeReturnType = tx.Union[tx.Type[tx.Tuple[HomogenousTypeVar, ...]],
tx.Type[tx.ClassVar[tx.Tuple[HomogenousTypeVar, ...]]]]
def TupleType(length: int,
tuptyp: tx.Type[HomogenousTypeVar] = str,
clsvar: bool = False) -> TupleTypeReturnType:
""" Create a type annotation for a tuple of a given type and length,
specifying additionally whether or not it is a ClassVar """
assert length > 0
out = tx.Tuple[tuple(tuptyp for idx in range(length))]
return clsvar and tx.ClassVar[out] or out
… after this change, the code won’t even initially compile – it fails to do so with a TypeError from deep within the typing module:
TypeError: typing.ClassVar[typing.Tuple[~HomogenousTypeVar, ...]] is
not valid as type argument
… which, as errors go, strikes me as a little phoned-in; I mean, is not everything in typing supposed to be a valid type argument in some fashion, give-or-take?
In the typing source code related to ClassVar, there are a handful of restrictions to its use mentioned in the docstring – but this is not one of them. Is there something obvious I am missing? Is my attempt to use this annotation in this fashion a quixotic one? What else might I try?
Are you sure your original code snippet actually type-checks with mypy? When I try running it using either Mypy 0.620 or the latest version from github, I get the following errors:
test.py:13: error: invalid type comment or annotation
test.py:13: note: Suggestion: use TupleType[...] instead of TupleType(...)
test.py:14: error: invalid type comment or annotation
test.py:14: note: Suggestion: use TupleType[...] instead of TupleType(...)
I'm also not able to reproduce the error you get with your ClassVar code -- when I try running it, I get the following errors:
test.py:4: error: Invalid type: ClassVar nested inside other type
test.py:6: error: Incompatible default for argument "tuptyp" (default has type "Type[str]", argument has type "Type[HomogenousTypeVar]")
test.py:12: error: Invalid type alias
test.py:13: warning: Returning Any from function declared to return "Union[Type[Tuple[HomogenousTypeVar?, ...]], Type[Tuple[HomogenousTypeVar?, ...]]]"
test.py:15: error: Name 'Thing' is not defined
test.py:16: error: Revealed type is 'Any'
Are you sure you're actually running mypy, as opposed to just running the code? E.g. if you only run python3 test.py, you're basically skipping all type checks (apart from some bare minimum sanity checks built into the typing module).
If you want to type-check your code, you need to pip-install mypy and run python3 -m mypy test.py.
In any case, all of these error messages are expected behavior -- mypy (and any other PEP 484 compliant type checker) can only analyze your code statically and will not attempt to run or analyze any factory functions/any type-hint generation functions you may try and write.
So, this means that unfortunately you won't be able to use your generated type hint with ClassVars if you want PEP 484 compliant tools to be able to analyze your code -- they can't understand/interpret your original set of type hints, and adding ClassVars certainly won't help.
If you want to generate type hints, the only real option I can think of is to invent some sort of mini-language or macro system on top of Python that when run, will generate Python code. You would then run and typecheck that generated code instead of your macrofied Python language.
But I really don't recommend doing this -- it's a very fragile hack.
More broadly, whenever you start running into these sorts of type-related limitations, I think it's a sign that your code is too complicated. I would either look into simplifying your code or (if that's not possible) switch to a language like Haskell or Idris which would let you use a more expressive (albeit more complex) type system.
For example, in this case, you're trying to generalize the Tuple type -- that leads me to infer that your codebase contains many different kinds of tuples of different arities and types.
That strikes me as being a bit suspicious -- I would instead look into converting some of those tuples into either regular classes or (if you still need tuple-like functionality) a namedtuple. Dataclasses (which are new as of Python 3.7) could also be convenient here.
Those solutions would also help make your code a little more readable -- you can now give concrete names and meanings to each distinct kind of tuple.
Alternatively, if you have only a few distinct types of tuples but use those tuples all over the place, you could try using type aliases so you don't have to repeatedly re-type the same (long) type over and over. E.g. instead of doing:
def foo(x: Tuple[int, int, int, int]) -> None: ...
...you could do:
IpAddress = Tuple[int, int, int, int]
def foo(x: IpAddress) -> None: ...
I would like to have a type that references itself:
SelfReferenceType = Dict[str, Union[str, 'SelfReferenceType']]
With python 3.5 and latest mypy I get:
test.py:1: error: Invalid type "test.SelfReferenceType"
Is there an easy way to do this? My guess is that forward references are only supported for classes, not aliases?
Here is what I'm trying to do:
SelfReferenceType = Dict[str, Union[str, 'SelfReferenceType']]
class Foo(object):
def __init__(self):
self.data = {} # type: SelfReferenceType
# Functions that work on self.data
I think your example is correct and works in newer versions of Python:
SelfReferenceType = Dict[str, Union[str, 'SelfReferenceType']]
I tested with 3.10.6 and do not get any errors. It would seem that this has been fixed.
In general using quoted strings for circular or "forward" references is the correct approach as outlined at https://peps.python.org/pep-0484/#forward-references and this comment by Guido. I couldn't find any information one way or the other about following this pattern for type aliases, specifically, but I don't see any reason why it would be different.
It's also worth nothing that references like this might work without quoting with sufficiently new versions of mypy (that include this PR, and maybe have a flag enabled).