I would like to have a type that references itself:
SelfReferenceType = Dict[str, Union[str, 'SelfReferenceType']]
With python 3.5 and latest mypy I get:
test.py:1: error: Invalid type "test.SelfReferenceType"
Is there an easy way to do this? My guess is that forward references are only supported for classes, not aliases?
Here is what I'm trying to do:
SelfReferenceType = Dict[str, Union[str, 'SelfReferenceType']]
class Foo(object):
def __init__(self):
self.data = {} # type: SelfReferenceType
# Functions that work on self.data
I think your example is correct and works in newer versions of Python:
SelfReferenceType = Dict[str, Union[str, 'SelfReferenceType']]
I tested with 3.10.6 and do not get any errors. It would seem that this has been fixed.
In general using quoted strings for circular or "forward" references is the correct approach as outlined at https://peps.python.org/pep-0484/#forward-references and this comment by Guido. I couldn't find any information one way or the other about following this pattern for type aliases, specifically, but I don't see any reason why it would be different.
It's also worth nothing that references like this might work without quoting with sufficiently new versions of mypy (that include this PR, and maybe have a flag enabled).
Related
Short version
tuple[int, ...] is a valid type in python 3.9+ but mypy rejects a type alias for that. E.g. consider:
Thing = tuple[int, ...]
This results in "error: Unexpected '...'" (though python itself will load the module just fine).
Is this a bug in mypy?
What should/can I do? I thought maybe I could just use NewType instead but that fails in the same way.
Long version
Python 3.9 incorporates PEP 585, meaning we can use built-in types such as list, set and tuple as generics; as such the versions in typing are now deprecated (e.g. see docs for typing.Tuple).
mypy 0.800 introduced support for this, so now I can write the following and mypy will type-check it correctly:
def foo(a: tuple[int, str]):
...
Now, tuple (and typing.Tuple) of course have a special form for homogenously-typed tuples of unknown length: tuple[int, ...]; this works fine too:
def foo(a: tuple[int, ...]):
...
Handily, we can create aliases for types, so this works too:
from typing import Tuple
Thing = Tuple[int, ...]
def foo(a: Thing):
...
Unfortunately, the following (which puts together tuple, the ... form, and a type alias) does not work:
Thing = tuple[int, ...]
def foo(a: Thing):
...
Rather, this is rejected by mypy (v0.812) with the message "error: Unexpected '...'"
Python itself will load and run such a module just fine, however.
I haven't (yet) been able to find anything indicating that this shouldn't be possible. So:
Am I missing something? Is such a type alias in fact forbidden?
If not, is this a bug in mypy?
Are there any better ways for working around this than just using NewType instead of an alias?
I thought using NewType would work around this (with some extra work) but it seems to fail in exactly the same way. So right now there seems to be no way to write these types and keep mypy happy.
Aha, looks like it is indeed an open mypy bug.
So now the only question (until that's fixed/released) is how to work around it? Probably I have to bite the NewType bullet. :-/
Edit: hilariously, Thing = NewType('Thing', tuple[int, ...]) fails in exactly the same way.
I'm new to Python development, and trying to get a fix on things. I am using Pycharm for development. I am currently trying to annotate the types of variables for easier access with autocompletion and suggestions. I've tried iterations of the code with mixed results.
This is the code that has a problem:
path = os.path.dirname(os.path.realpath(__file__)) # type: str
components = path.split(os.sep) # type: list[str]
The first problem displayed is at the opening brace of the type annotation of the second line. It says:
Class 'type' does not define '__getitem__', so the '[]' operator cannot be used on its instances.
I've googled around, and although the problem seems clear, opening the code for the list class clearly shows a method __getitem__:
class list(object):
"""
list() -> new empty list
list(iterable) -> new list initialized from iterable's items
"""
....
def __getitem__(self, y): # real signature unknown; restored from __doc__
""" x.__getitem__(y) <==> x[y] """
pass
Okay, maybe that's not trivial to understand and there's some other loading mechanism at work. And besides, the "problem" seems to be that I used list[str] and not List[str]. So I altered the code:
path = os.path.dirname(os.path.realpath(__file__)) # type: str
components = path.split(os.sep) # type: List[str]
Now everything breaks: The second line now complains about this:
Expected type 'List[str]', got 'List[str]' instead`
The previous problem regarding __getitem__ yet persists.
Is there a way to annotate these variables without causing problems for the checker? I am not very happy with the Python documentation in that regard, not explicitly stating the return types of their built-in methods. I have to rely on the information Pycharm provides in the documentation pop-ups (Ctrl+q).
It was a right solution to use List[str] instead of list[str] since builtin types could not be used in type hints, the corresponding PEP has not been accepted yet.
From what module do you import List? I'm unable to reproduce an issue in 2019.3.
I recently started using type hints in my code, and have so far found them to be (mostly) very helpful.
However, one thing that I really do not like is the syntax to force the type checker to assume that a variable is of a certain type. Given this example:
import itertools
from typing import Iterable, Tuple
x: Iterable[Tuple[str, str]] = itertools.combinations('abc', 2)
# error: Incompatible types in assignment (expression has type "Iterable[Tuple[str, ...]]", variable has type "List[Tuple[str, str]]")
As far as I can tell, the recommended way to work around this is to explicitly cast the object to force the type checker to use the specified type, e.g.:
import itertools
from typing import Iterable, Tuple, cast
x = cast(Iterable[Tuple[str, str]], itertools.combinations('abc', 2))
I personally find this solution to be a bit gross. My primary concern is that, to the inexperienced reader, it is not clear that the cast is purely there to help the static analyzer. (If I didn't already know, I would assume based on the name and context that it is converting and doing a copy into an object of the specified type, when really there is no runtime cost.)
cast looks like any old function call. When I see that a function is being called on a value, I expect the value to be mutated and/or some other side-effects to occur, but in this case the only side effect is that mypy stops complaining. Type hints themselves have a distinct syntax, but I feel that this blurs the lines with a mixture of the new typing syntax and traditional python syntax. (It's already a bit blurry since you have to import the types and can compose them, but that's another discussion.)
Is there an alternative syntax for cast-like behavior? I haven't found anything, but I was hoping for something like:
x1 = itertools.combinations('abc', 2)) # cast: Iterable[Tuple[str, str]]
x2: Iterable[Tuple[str, str]] = itertools.combinations('abc', 2)) # type: cast
x3: Cast[Iterable[Tuple[str, str]]] = itertools.combinations('abc', 2))
Actually the latest version of Mypy does return the correct type Iterator[Tuple[str, str]].
This change was introduced to Typeshed in PR https://github.com/python/typeshed/pull/4309.
If you cannot update mypy to the latest version you can checkout the latest version from typeshed and use the config option custom_typeshed_dir.
See https://mypy.readthedocs.io/en/stable/config_file.html#confval-custom_typeshed_dir for more details.
Let's say I have a method like the following
def validate(self, item:dict, attrs:dict)-> list:
If I want to be more specific and tell that my return type is a list of ValidationMessages?
How should I / Can I achieve that?
(I would check off the mark as a duplicate, since this is not about extending a list or flattening) I'm asking about how to be more specific on identifying the return type of a method...
With Python 3.6, the built-in typing package will do the job.
from typing import List
def validate(self, item:dict, attrs:dict)-> List[str]:
...
The notation is a bit weird, since it uses brackets but works out pretty well.
Edit: With the new 3.9 version of Python, you can annotate types without importing from the typing module. The only difference is that you use real type names instead of defined types in the typing module.
def validate(self, item:dict, attrs:dict)-> list[str]:
...
NOTE: Type hints are just hints that help IDE. Those types are not enforced. You can add a hint for a variable as str and set an int to it like this:
a:str = 'variable hinted as str'
a = 5 # See, we can set an int
Your IDE will warn you but you will still be able to run the code. Because those are just hints. Python is not a type strict language. Instead, it employs dynamic typing.
I am trying to use a factory function to generate some type annotations – specifically for tuple types. I have one version of the factory that works fine (as in, it compiles, runs, and checks out satisfactorily in MyPy):
import typing as tx
HomogenousTypeVar = tx.TypeVar('HomogenousTypeVar')
TupleTypeReturnType = tx.Type[tx.Tuple[HomogenousTypeVar, ...]]
def TupleType(length: int,
tuptyp: tx.Type[HomogenousTypeVar] = str) -> TupleTypeReturnType:
""" Create a type annotation for a tuple of a given type and length """
assert length > 0
return tx.Tuple[tuple(tuptyp for idx in range(length))]
… for which the usage is like e.g.:
class Thing(object):
__slots__: TupleType(2) = ('yo', 'dogg')
other_fields: TupleType(4) = ('i', 'heard',
'you', 'like')
# etc, or what have you
… however, I was unsuccessful when I tried to add support for the typing.ClassVar annotation, which looked like this:
import typing as tx
HomogenousTypeVar = tx.TypeVar('HomogenousTypeVar')
TupleTypeReturnType = tx.Union[tx.Type[tx.Tuple[HomogenousTypeVar, ...]],
tx.Type[tx.ClassVar[tx.Tuple[HomogenousTypeVar, ...]]]]
def TupleType(length: int,
tuptyp: tx.Type[HomogenousTypeVar] = str,
clsvar: bool = False) -> TupleTypeReturnType:
""" Create a type annotation for a tuple of a given type and length,
specifying additionally whether or not it is a ClassVar """
assert length > 0
out = tx.Tuple[tuple(tuptyp for idx in range(length))]
return clsvar and tx.ClassVar[out] or out
… after this change, the code won’t even initially compile – it fails to do so with a TypeError from deep within the typing module:
TypeError: typing.ClassVar[typing.Tuple[~HomogenousTypeVar, ...]] is
not valid as type argument
… which, as errors go, strikes me as a little phoned-in; I mean, is not everything in typing supposed to be a valid type argument in some fashion, give-or-take?
In the typing source code related to ClassVar, there are a handful of restrictions to its use mentioned in the docstring – but this is not one of them. Is there something obvious I am missing? Is my attempt to use this annotation in this fashion a quixotic one? What else might I try?
Are you sure your original code snippet actually type-checks with mypy? When I try running it using either Mypy 0.620 or the latest version from github, I get the following errors:
test.py:13: error: invalid type comment or annotation
test.py:13: note: Suggestion: use TupleType[...] instead of TupleType(...)
test.py:14: error: invalid type comment or annotation
test.py:14: note: Suggestion: use TupleType[...] instead of TupleType(...)
I'm also not able to reproduce the error you get with your ClassVar code -- when I try running it, I get the following errors:
test.py:4: error: Invalid type: ClassVar nested inside other type
test.py:6: error: Incompatible default for argument "tuptyp" (default has type "Type[str]", argument has type "Type[HomogenousTypeVar]")
test.py:12: error: Invalid type alias
test.py:13: warning: Returning Any from function declared to return "Union[Type[Tuple[HomogenousTypeVar?, ...]], Type[Tuple[HomogenousTypeVar?, ...]]]"
test.py:15: error: Name 'Thing' is not defined
test.py:16: error: Revealed type is 'Any'
Are you sure you're actually running mypy, as opposed to just running the code? E.g. if you only run python3 test.py, you're basically skipping all type checks (apart from some bare minimum sanity checks built into the typing module).
If you want to type-check your code, you need to pip-install mypy and run python3 -m mypy test.py.
In any case, all of these error messages are expected behavior -- mypy (and any other PEP 484 compliant type checker) can only analyze your code statically and will not attempt to run or analyze any factory functions/any type-hint generation functions you may try and write.
So, this means that unfortunately you won't be able to use your generated type hint with ClassVars if you want PEP 484 compliant tools to be able to analyze your code -- they can't understand/interpret your original set of type hints, and adding ClassVars certainly won't help.
If you want to generate type hints, the only real option I can think of is to invent some sort of mini-language or macro system on top of Python that when run, will generate Python code. You would then run and typecheck that generated code instead of your macrofied Python language.
But I really don't recommend doing this -- it's a very fragile hack.
More broadly, whenever you start running into these sorts of type-related limitations, I think it's a sign that your code is too complicated. I would either look into simplifying your code or (if that's not possible) switch to a language like Haskell or Idris which would let you use a more expressive (albeit more complex) type system.
For example, in this case, you're trying to generalize the Tuple type -- that leads me to infer that your codebase contains many different kinds of tuples of different arities and types.
That strikes me as being a bit suspicious -- I would instead look into converting some of those tuples into either regular classes or (if you still need tuple-like functionality) a namedtuple. Dataclasses (which are new as of Python 3.7) could also be convenient here.
Those solutions would also help make your code a little more readable -- you can now give concrete names and meanings to each distinct kind of tuple.
Alternatively, if you have only a few distinct types of tuples but use those tuples all over the place, you could try using type aliases so you don't have to repeatedly re-type the same (long) type over and over. E.g. instead of doing:
def foo(x: Tuple[int, int, int, int]) -> None: ...
...you could do:
IpAddress = Tuple[int, int, int, int]
def foo(x: IpAddress) -> None: ...