I want to define a function example that takes an argument of type Widget or anything that extends Widget and returns the same type as the argument. So if Button extends Widget, calling example(Button()) returns type Button.
I tried the following:
T_co = TypeVar('T_co', Widget, covariant=True)
def example(widget: T_co) -> T_co:
...
However the type checker (Pyright) ignores the covariance. Upon further research I found a note in PEP 484:
Note: Covariance or contravariance is not a property of a type variable, but a property of a generic class defined using this variable. Variance is only applicable to generic types; generic functions do not have this property. The latter should be defined using only type variables without covariant or contravariant keyword arguments.
However if I try to define a generic function without the covariant argument as specified in the note:
T_co = TypeVar('T_co', Widget)
def example(widget: T_co) -> T_co:
...
I can only pass values of type Widget to the function (not Button).
How can I achieve this?
I was able to find the answer in the MyPy docs. Turns out I was looking for bound, not covariant. This can be done like so:
T = TypeVar('T', bound=Widget)
def example(widget: T) -> T:
...
Related
from typing import Generic, TypeVar
T_co = TypeVar('T_co', covariant=True)
class CovariantClass(Generic[T_co]):
def get_t(self, t: T_co) -> T_co: # <--- Mypy: cannot use a covariant type variable as a parameter
return t
see closed mypy issue
My reading of the PEPs 484 and 483 is that covariant functions are prohibited but covariant methods are allowed and mypy is "wrong" to flag it. That's to say that if the class is declared by the author as covariant then it is their responsibility to ensure that any covariant method in the class is well-behaved (e.g. by not appending to collection of covariant items).
from PEP 484:
Consider a class Employee with a subclass Manager. Now suppose we have a function with an argument annotated with List[Employee]. Should we be allowed to call this function with a variable of type List[Manager] as its argument? Many people would answer "yes, of course" without even considering the consequences. But unless we know more about the function, a type checker should reject such a call: the function might append an Employee instance to the list, which would violate the variable's type in the caller.
It turns out such an argument acts contravariantly, whereas the intuitive answer (which is correct in case the function doesn't mutate its argument!) requires the argument to act covariantly.
....
Covariance or contravariance is not a property of a type variable, but a property of a generic class defined using this variable. Variance is only applicable to generic types; generic functions do not have this property. The latter should be defined using only type variables without covariant or contravariant keyword arguments.
It then gives an example of a prohibited function with a covariant argument.
Also from PEP 483 as evidence against the argument put forward in the closed mypy issue that it is prohibited to keep python type-safe (emphasis as in the PEP):
It is possible to declare the variance for user defined generic types
PEP 484 contains a class with a covariant argument in the __init__ but there is no example I could find in the PEPs of a covariant argument in a method showing it to be explicitly either allowed or prohibited.
T_co = TypeVar('T_co', covariant=True)
class ImmutableList(Generic[T_co]):
def __init__(self, items: Iterable[T_co]) -> None: ...
def __iter__(self) -> Iterator[T_co]: ...
...
EDIT 1: simple example of where this could be useful as requested in the comments. An extra method in the above class from the PEP:
def is_member(self, item: T_co) -> bool:
return item in self._items
EDIT 2: another example in case the first one seems academic
def k_nearest_neighbours(self, target_item: T_co, k: int) -> list[T_co]:
nearest: list[T_co] = []
distances: list[float] = [item.distance_metric(target_item)
for item in self._items]
...
return nearest
class BaseClass:
p: int
class DerivedClass(BaseClass):
q: int
def p(q: Callable[[BaseClass], str]) -> None:
return None
def r(derived: DerivedClass) -> str:
return ""
p(r)
Expected behavior:
- No error from mypy -
Actual behavior:
Argument 1 to "p" has incompatible type "Callable[[DerivedClass], str]";
expected "Callable[[BaseClass], str]"
Let's talk about type variance. Under typical subtyping rules, if we have a type DerivedClass that is a subtype of a type BaseClass, then every instance of DerivedClass is an instance of BaseClass. Simple enough, right? But now the complexity arises when we have generic type arguments.
Let's suppose that we have a class that gets a value and returns it. I don't know how it gets it; maybe it queries a database, maybe it reads the file system, maybe it just makes one up. But it gets a value.
class Getter:
def get_value(self):
# Some deep magic ...
Now let's assume that, when we construct the Getter, we know what type it should be querying at compile-time. We can use a type variable to annotate this.
T = TypeVar("T")
class Getter(Generic[T]):
def get_value(self) -> T:
...
Now, Getter is a valid thing. We can have a Getter[int] which gets an integer and a Getter[str] which gets a string.
But here's a question. If I have a Getter[int], is that a valid Getter[object]? Surely, if I can get a value as an int, it's easy enough to upcast it, right?
my_getter_int: Getter[int] = ...
my_getter_obj: Getter[object] = my_getter_int
But Python won't allow this. See, Getter was declared to be invariant in its type argument. That's a fancy way of saying that, even though int is a subtype of object, Getter[int] and Getter[object] have no relationship.
But, like I said, surely they should have a relationship, right? Well, yes. If your type is only used in positive position (glossing over some details, that means roughly that it only appears as the return value of methods or as the type of read-only properties), then we can make it covariant.
T_co = TypeVar("T_co", covariant=True)
class Getter(Generic[T_co]):
def get_value(self) -> T_co:
...
By convention, in Python, we denote covariant type arguments using names that end in _co. But the thing that actually makes it covariant here is the covariant=True keyword argument.
Now, with this version of Getter, Getter[int] is actually a subtype of Getter[object]. In general, if A is a subtype of B, then Getter[A] is a subtype of Getter[B]. Covariance preserves subtyping.
Okay, that's covariance. Now consider the opposite. Let's say we have a setter which sets some value in a database.
class Setter:
def set_value(self, value):
...
Same assumptions as before. Suppose we know what the type is in advance. Nowe we write
T = TypeVar("T")
class Setter:
def set_value(self, value: T) -> None:
...
Okay, great. Now, if I have a value my_setter : Setter[int], is that a Setter[object]? Well, my_setter can always take an integer value, whereas a Setter[object] is guaranteed to be able to take any object. my_setter can't guarantee that, so it's actually not. If we try to make T covariant in this example, we'll get
error: Cannot use a covariant type variable as a parameter
Because it's actually not a valid relationship. In fact, in this case, we get the opposite relationship. If we have a my_setter : Setter[object], then that's a guarantee that we can pass it any object at all, so certainly we can pass it an integer, hence we have a Setter[int]. This is called contravariance.
T_contra = TypeVar("T_contra", contravariant=True)
class Setter:
def set_value(self, value: T_contra) -> None:
...
We can make our type contravariant if it only appears in negative position, which (again, oversimplifying a bit) generally means that it appears as arguments to functions, but not as a return value. Now, Setter[object] is a subtype of Setter[int]. It's backwards. In general, if A is a subtype of B, then Setter[B] is a subtype of Setter[A]. Contravariance reverses the subtyping relationship.
Now, back to your example. You have a Callable[[DerivedClass], str] and want to know if it's a valid Callable[[BaseClass], str]
Applying our principles from before, we have a type Callable[[T], S] (I'm assuming only one argument for simplicity's sake, but in reality this works in Python for any number of arguments) and want to ask if T and S are covariant, contravariant, or invariant.
Well, what is a Callable? It's a function. It has one thing we can do: call it with a T and get an S. So it's pretty clear that T is only used as an argument and S as a result. Things only used as arguments are contravariant, and those used as results are covariant, so in reality it's more correct to write
Callable[[T_contra], S_co]
Arguments to Callable are contravariant, which means that if DerivedClass is a subtype of BaseClass, then Callable[[BaseClass], str] is a subtype of Callable[[DerivedClass], str], the opposite relationship to the one you suggested. You need a function that can accept any BaseClass. A function with a BaseClass argument would suffice, and so would a function with an object argument, or any type which is a supertype of BaseClass, but subtypes are insufficient because they're too specific for your contract.
MyPy objects to your call of p with r as its argument because given only the type signatures, it can't be sure the function won't be called with a non-DerivedClass instance.
For instance, given the same type annotations, p could be implemented like this:
def p(q: Callable[[BaseClass], str]) -> None:
obj = BaseClass()
q(obj)
This will break p(r) if r has an implementation that depends on the derived attributes of its argument:
def r(derived: DerivedClass) -> str:
return str(derived.q)
I want to constrain a method parameter to be of the same type as the class it's called on (see the end for an example). While trying to do that, I've come across this behaviour that I'm struggling to get my head around.
The following doesn't type check
class A:
def foo(self) -> None:
pass
A.foo(1)
with
error: Argument 1 to "foo" of "A" has incompatible type "int"; expected "A"
as I'd expect, since I'd have thought A.foo should only take an A. If however I add a self type
from typing import TypeVar
Self = TypeVar("Self")
class A:
def foo(self: Self) -> None:
pass
A.foo(1)
it does type check. I would have expected it to fail, telling me I need to pass an A not an int. This suggests to me that the type checker usually infers the type A for self, and adding a Self type overrides that, I'm guessing to object. This fits with the error
from typing import TypeVar
Self = TypeVar("Self")
class A:
def bar(self) -> int:
return 0
def foo(self: Self) -> None:
self.bar()
error: "Self" has no attribute "bar"
which I can fix if I bound as Self = TypeVar("Self", bound='A')
Am I right that this means self is not constrained, in e.g. the same way I'd expect this to be constrained in Scala?
I guess this only has an impact if I specify the type of self to be anything but the class it's defined on, intentionally or otherwise. I'm also interested to know what the impact is of overriding self to be another type, and indeed whether it even makes sense with how Python resolves and calls methods.
Context
I want to do things like
class A:
def foo(self: Self, bar: List[Self]) -> Self:
...
but I was expecting Self to be constrained to be an A, and was surprised that it wasn't.
Two things:
self is only half-magic.
The self arg has the magical property that, if you call an attribute of an object as a function, and that function has self as its first arg, then the object itself will be prepended to the explicit args as the self.
I guess any good static analyzer would take as implicit that self has the class in question as its type, which is what you're seeing in your first example.
TypeVar is for polymorphism.
And I think that's what you're trying to do? In your third example, Self can be any type, depending on context. In the context of A.foo(1), Self is int, so self.bar() fails.
It may be possible to write an instance method that can be called as a static method against class non-members with parametric type restrictions, but it's probably not a good idea for any application in the wild. Just name the variable something else and declare the method to be static.
If you omit a type hint on self, the type checker will automatically assume it has whatever the type of the containing class is.
This means that:
class A:
def foo(self) -> None: pass
...is equivalent to doing:
class A:
def foo(self: A) -> None: pass
If you want self to be something else, you should set a custom type hint.
Regarding this code snippet:
from typing import TypeVar
Self = TypeVar("Self")
class A:
def foo(self: Self) -> None:
pass
A.foo(1)
Using a TypeVar only once in a function signature is either malformed or redundant, depending on your perspective.
But this is kind of unrelated to the main thrust of your question. We can repair your code snippet by instead doing:
from typing import TypeVar
Self = TypeVar("Self")
class A:
def foo(self: Self) -> Self:
return self
A.foo(1)
...which exhibits the same behaviors you noticed.
But regardless of which of the two code snippets we look at, I believe the type checker will indeed assume self has the same type as whatever the upper bound of Self is while type checking the body of foo. In this case, the upper bound is object, as you suspected.
We get this behavior whether or not we're doing anything fancy with self or not. For example, we'd get the exact same behavior by just doing:
def foo(x: Self) -> Self:
return x
...and so forth. From the perspective of the type checker, there's nothing special about the self parameter, except that we set a default type for it if it's missing a type hint instead of just using Any.
error: "Self" has no attribute "bar"
which I can fix if I bound as Self = TypeVar("Self", bound='A')
Am I right that this means self is not constrained, in e.g. the same way I'd expect this to be constrained in Scala?
I'm unfamiliar with how this is constrained in Scala, but it is indeed the case that if you chose to override the default type of self, you are responsible for setting your own constraints and bounds as appropriate.
To put it another way, once a TypeVar is defined, its meaning won't be changed when you try using it in a function definition. This is the rule for TypeVars/functions in general. And since mostly there's nothing special about self, the same rule also applies there.
(Though type checkers such as mypy will also try doing some basic sanity checks on whatever constraints you end up picking to ensure you don't end up with a method that's impossible to call or whatever. For example, it complain if you tried setting the bound of Self to int.)
Note that doing things like:
from typing import TypeVar, List
Self = TypeVar('Self', bound='A')
class A:
def foo(self: Self, bar: List[Self]) -> Self:
...
class B(A): pass
x = A().foo([A(), A()])
y = B().foo([B(), B()])
reveal_type(x) # Revealed type is 'A'
reveal_type(y) # Revealed type is 'B'
...is explicitly supported by PEP 484. The mypy docs also have a few examples.
I would like to to pass class itself as an argument in the constructor. I know it is possible in Python but I am having problems understanding how should I write a proper typing annotation. Use case is the following:
In the constructor of class A I want to pass a reference to some class X (not an object of class X) that inherits after BaseX. Both X and BaseX come from a library. Apart from the reference to X, constructor of A accepts arguments that help build X:
# Library
class BaseX:
def func():
print("Hey BaseX")
class X(BaseX):
def func():
print("X!")
# My client code
class A:
def __init__(arg x, arg y, layer: BaseX): # what should be the annotation of layer?
# construct BaseX object
self.layer = BaseX(x=x, y=y) # IDEs show x and y as unexpected arguments because they treat BaseX as an object and look into its __call__ func
A(5, 6, X)
I am unsure how can I express the annotation of layer so that it can be treated as a class and ensure its a derivative of BaseX. I would also like to ask about some comment about whether this is a Pythonic way to do this.
Cheers!
You can indicate that a variable is a reference to a type with the annotation Type[BaseX] (see Python docs on Type). A variable annotated with Type[T] holds any type that is a subtype of T.
For your specific use case of "constructing an object of the specified type, which is a subtype of BaseX", you can use more accurate annotations with the help of TypeVar. For example:
T = TypeVar('T', bound=BaseX)
def construct(cls: Type[T], *args, **kwargs) -> T:
return cls(*args, **kwargs)
Here:
TypeVar('T', bound=BaseX) defines a "type variable" that can be substituted with any type "bounded" by BaseX, i.e., is a subtype of BaseX.
The construct function takes an argument cls with annotation Type[T], indicating it's a reference to a subtype of BaseX.
The return type annotation is T, indicating the returned value is an instance of the subtype of BaseX.
All occurrences of a type variable within a function or class are bound to the same type. In this case, the type of the returned value is the type passed as argument.
Please consider this snippet of python 3.5 code:
class Foo:
pass
class Bar(Foo):
pass
class AbstractSomething:
def get_foobinator_type(self):
return Foo
I'd like to annotate (using PEP-0484 annotations) return value of get_foobinator_type method to say: "It returns a type, that is either a Foo or any subtype of it".
I din't find any sensible way to do it in Python. Here are approaches that are obviously wrong:
Following: def get_foobinator_type(self) -> Foo means that this method returns an instance of Foo.
Following: def get_foobinator_type(self) -> type means that this method returns a type, but sadly, there is no information about that this is needs to be a subtype of Foo.
In Java terms I'd like to have method with signature like: Class<Foo> getFoobinatorType().
As far as I understand, you really cannot. You're looking for a way to indicate the return type of a class; to check based on what the type of the class is, i.e its metaclass.
The problem with that is that a metaclass doesn't help a type checker evaluate what the inheritance of an object might be, if it's of type type it's alright.
Apart from that, and, not being sure what type-checker you use, mypy for example doesn't have support yet for custom metaclasses which you might use to group your objects in a more custom group.
The way I see it, you either don't annotate at all all, or, you change the implementation and annotate with Foo.
Lately (in Python 3.5.2) a generic Type was introduced, so solution is:
class AbstractSomething:
def get_foobinator_type(self) -> typing.Type[Foo]:
return Bar
See: python docs.
I think what you need is TypeVar from the typing module.
from typing import TypeVar
class Foo:
pass
class Bar(Foo):
pass
T = TypeVar('T', bound=Foo)
class AbstractSomething:
def get_foobinator_type(self) -> T:
return Foo
From the documentation of typing:
Alternatively, a type variable may specify an upper bound using
bound=. This means that an actual type substituted (explicitly
or implicitly) for the type variable must be a subclass of the
boundary type, see PEP 484