I have the following code:
from typing import Callable
MyCallable = Callable[[object], int]
MyCallableSubclass = Callable[['MyObject'], int]
def get_id(obj: object) -> int:
return id(obj)
def get_id_subclass(obj: 'MyObject') -> int:
return id(obj)
def run_mycallable_function_on_object(obj: object, func: MyCallable) -> int:
return func(obj)
class MyObject(object):
'''Object that is a direct subclass of `object`'''
pass
my_object = MyObject()
# works just fine
run_mycallable_function_on_object(my_object, get_id)
# Does not work (it runs, but Mypy raises the following error:)
# Argument 2 to "run_mycallable_function_on_object" has incompatible type "Callable[[MyObject], int]"; expected "Callable[[object], int]"
run_mycallable_function_on_object(my_object, get_id_subclass)
Since MyObject inherits from object, why doesn't MyCallableSubclass work in every place that MyCallable does?
I've read a bit about the Liskov substitution principle, and also consulted the Mypy docs about covariance and contravariance. However, even in the docs themselves, they give a very similar example where they say
Callable is an example of type that behaves contravariant in types of arguments, namely Callable[[Employee], int] is a subtype of Callable[[Manager], int].
So then why is using Callable[[MyObject], int] instead of Callable[[object], int] throwing an error in Mypy?
Overall I have two questions:
Why is this happening?
How do I fix it?
As I was writing this question, I realized the answer to my problem, so I figured I'd still ask the question and answer it to save people some time with similar questions later.
What's going on?
Notice that last example from the Mypy docs:
Callable is an example of type that behaves contravariant in types of arguments, namely Callable[[Employee], int] is a subtype of Callable[[Manager], int].
Here, Manager subclasses from Employee. That is, if something is expecting a function that can take in managers, it's alright if the function it gets overgeneralizes and can take in any employee, because it will definitely take in managers.
However, in our case, MyObject subclasses from object. So, if something is expecting a function that can take in objects, then it's not okay if the function it gets overspecifies and can only take in MyObjects.
Why? Imagine a class called NotMyObject that inherits from object, but doesn't inherit from MyObject. If a function should be able to take any object, it should be able to take in both NotMyObjects and MyObjects. However, the specific function can only take in MyObjects, so it won't work for this case.
How can I fix it?
Mypy is correct. You need to have the more specific function (MyCallableSubclass) as the type, otherwise either your code could have bugs, or you are typing incorrectly.
Related
class B:
pass
class InheritsB1(B):
pass
class InheritsB2(B):
pass
class A:
prop: list[B]
class InheritsA1(A):
prop: list[InheritsB1]
class InheritsA2(A):
prop: list[InheritsB2]
With this code mypy raises Incompatible types in assignment (expression has type "List[InheritsB2]", base class "A" defined the type as "List[B]").
How can I make this work?
InheritsB1 is a subclass of B, so list[InheritsB1] is always a list of B. How can I tell mypy that it's not incompatible? Or, how can I tell mypy that the prop in A is "list of B or any specific subclass of B"?
I understand the issue here: mypy trouble with inheritance of objects in lists. But in this case I want the prop object to be a list of a specific instances (B or any subclass of B). I know it will never be mixed, as in it will always be list[B] or list[SubclassOfB1] or list[SubclassOfB2], never list[SubclassOfB1 | SubclassOfB2]. How can I do this?
You linked a post that essentially addresses the issue, but you seem to still be confused:
InheritsB1 is a subclass of B, so list[InheritsB1] is always a list of B.
This is not true. To quote from this section of PEP 484:
By default generic types are considered invariant in all type variables
The theory section on variance in PEP 483 attempts to explain the concepts in much greater detail. I suggest you read through PEP 483 as well as PEP 484, if you want to get more serious about type safety.
The mypy complaint is justified and unless you want to follow the tip provided by mypy and change the type to something covariant like Sequence instead, you will have to work a bit more with generics yourself.
How best to solve your particular conundrum depends on other factors around your classes that are still ambiguous in your original post.
But one option might be to make A generic in terms of a type variable that has an upper bound of B. Then, if you want to define a subclass of A to hold elements of a specific subtype of B, there would be no problem:
from typing import Generic, TypeVar
_BType = TypeVar("_BType", bound="B")
class B:
pass
class SubB(B):
pass
class A(Generic[_BType]):
attribute: list[_BType]
class SubA(A[SubB]):
# attribute: list[SubB]
...
reveal_type(SubA().attribute) # Revealed type is "builtins.list[SubB]"
Note that in this setup, you don't even have to override/annotate SubA.attribute (which is why I commented it out) because you specify the type argument for A to be SubB during inheritance, which means that SubA is no longer generic and attribute will always be inferred as list[SubB].
The code is type safe (passes mypy --strict), but it still may or may not be practical for you, depending on what other requirements you have for the classes involved. If you provide more details, I can try to amend my answer to better suit your setup.
I always though that Callable is equivalent to having the dunder __call__ but apparently there is also __name__, because the following code is correct for mypy --strict:
def print_name(f: Callable[..., Any]) -> None:
print(f.__name__)
def foo() -> None:
pass
print_name(foo)
print_name(lambda x: x)
What is actual interface of python Callable?
I dug out what functools.wraps does. AFAIU it sets ('__module__', '__name__', '__qualname__', '__doc__', '__annotations__') - is that the same what the Callable is expected to have?
So the mypy position up until now seems to have been that most of the time, when a variable is annotated with Callable, the user expects it to stand for a user-defined function (i.e. def something(...): ...).
Even though user-defined functions are technically a subtype of the callable and even though they are the ones that define a number of those attributes you mentioned, some users are not aware of this distinction and would be surprised, if mypy raised an error with code like this:
from collections.abc import Callable
from typing import Any
def f(cal: Callable[..., Any]) -> None:
print(cal.__name__)
print(cal.__globals__)
print(cal.__kwdefaults__)
print(cal.foo)
Each of those print-lines should be an error, yet only the last actually triggers one.
Moreover, if we define a minimal callable class that doesn't have those attributes, it is treated as a subtype of Callable by both Python and mypy, creating a logical contradication:
class Bar:
def __call__(self) -> None:
print(f"hi mom")
f(Bar()) # this is valid
print(Bar().__name__) # this is an error
Their argument so far amounts to maintaining convenience for users that have so far failed to see the distinction between callable subtypes, and by extension avoiding confused issues being opened by those users, asking why callables shouldn't have __name__ or those other attributes. (I hope I am being charitable enough with my interpretation.)
I find this to be a very odd position (to put it mildly) and I expressed as much in the issue I opened for this. I'll keep this answer updated, if any new insights are reached in the discussion around the issue.
Bottom line is: You are right, callables must have the __call__ method and do not require anything else.
Consider I have a python class that has a attributes (i.e. a dataclass, pydantic, attrs, django model, ...) that consist of a union, i.e. None and and a state.
Now I have a complex checking function that checks some values.
If I use this checking function, I want to tell the type checker, that some of my class attributes are narrowed.
For instance see this simplified example:
import dataclasses
from typing import TypeGuard
#dataclasses.dataclass
class SomeDataClass:
state: tuple[int, int] | None
name: str
# Assume many more data attributes
class SomeDataClassWithSetState(SomeDataClass):
state: tuple[int, int]
def complex_check(data: SomeDataClass) -> TypeGuard[SomeDataClassWithSetState]:
# Assume some complex checks here, for simplicity it is only:
return data.state is not None and data.name.startswith("SPECIAL")
def get_sum(data: SomeDataClass) -> int:
if complex_check(data):
return data.state[0] + data.state[1]
return 0
Explore on mypy Playground
As seen it is possible to do this with subclasses, which for various reason is not an option for me:
it introduces a lot of duplication
some possible libraries used for dataclasses are not happy with being subclasses without side condition
there could be some Metaclass or __subclasses__ magic that handles all subclass specially, i.e. creating database for the dataclasses
So is there an option to type narrow a(n) attribute(s) of a class without introducing a solely new class, as proposed here?
TL;DR: You cannot narrow the type of an attribute. You can only narrow the type of an object.
As I already mentioned in my comment, for typing.TypeGuard to be useful it relies on two distinct types T and S. Then, depending on the returned bool, the type guard function tells the type checker to assume the object to be either T or S.
You say, you don't want to have another class/subclass alongside SomeDataClass for various (vaguely valid) reasons. But if you don't have another type, then TypeGuard is useless. So that is not the route to take here.
I understand that you want to reduce the type-safety checks like if obj.state is None because you may need to access the state attribute in multiple different places in your code. You must have some place in your code, where you create/mutate a SomeDataClass instance in a way that ensures its state attribute is not None. One solution then is to have a getter for that attribute that performs the type-safety check and only ever returns the narrower type or raises an error. I typically do this via #property for improved readability. Example:
from dataclasses import dataclass
#dataclass
class SomeDataClass:
name: str
optional_state: tuple[int, int] | None = None
#property
def state(self) -> tuple[int, int]:
if self.optional_state is None:
raise RuntimeError("or some other appropriate exception")
return self.optional_state
def set_state(obj: SomeDataClass, value: tuple[int, int]) -> None:
obj.optional_state = value
if __name__ == "__main__":
foo = SomeDataClass(optional_state=(1, 2), name="foo")
bar = SomeDataClass(name="bar")
baz = SomeDataClass(name="baz")
set_state(bar, (2, 3))
print(foo.state)
print(bar.state)
try:
print(baz.state)
except RuntimeError:
print("baz has no state")
I realize you mean there are many more checks happening in complex_check, but either that function doesn't change the type of data or it does. If the type remains the same, you need to introduce type-safety for attributes like state in some other place, which is why I suggest a getter method.
Another option is obviously to have a separate class, which is what is typically done with FastAPI/Pydantic/SQLModel for example and use clever inheritance to reduce code duplication. You mentioned this may cause problems because of subclassing magic. Well, if it does, use the other approach, but I can't think of an example that would cause the problems you mentioned. Maybe you can be more specific and show a case where subclassing would lead to problems.
class BaseClass:
p: int
class DerivedClass(BaseClass):
q: int
def p(q: Callable[[BaseClass], str]) -> None:
return None
def r(derived: DerivedClass) -> str:
return ""
p(r)
Expected behavior:
- No error from mypy -
Actual behavior:
Argument 1 to "p" has incompatible type "Callable[[DerivedClass], str]";
expected "Callable[[BaseClass], str]"
Let's talk about type variance. Under typical subtyping rules, if we have a type DerivedClass that is a subtype of a type BaseClass, then every instance of DerivedClass is an instance of BaseClass. Simple enough, right? But now the complexity arises when we have generic type arguments.
Let's suppose that we have a class that gets a value and returns it. I don't know how it gets it; maybe it queries a database, maybe it reads the file system, maybe it just makes one up. But it gets a value.
class Getter:
def get_value(self):
# Some deep magic ...
Now let's assume that, when we construct the Getter, we know what type it should be querying at compile-time. We can use a type variable to annotate this.
T = TypeVar("T")
class Getter(Generic[T]):
def get_value(self) -> T:
...
Now, Getter is a valid thing. We can have a Getter[int] which gets an integer and a Getter[str] which gets a string.
But here's a question. If I have a Getter[int], is that a valid Getter[object]? Surely, if I can get a value as an int, it's easy enough to upcast it, right?
my_getter_int: Getter[int] = ...
my_getter_obj: Getter[object] = my_getter_int
But Python won't allow this. See, Getter was declared to be invariant in its type argument. That's a fancy way of saying that, even though int is a subtype of object, Getter[int] and Getter[object] have no relationship.
But, like I said, surely they should have a relationship, right? Well, yes. If your type is only used in positive position (glossing over some details, that means roughly that it only appears as the return value of methods or as the type of read-only properties), then we can make it covariant.
T_co = TypeVar("T_co", covariant=True)
class Getter(Generic[T_co]):
def get_value(self) -> T_co:
...
By convention, in Python, we denote covariant type arguments using names that end in _co. But the thing that actually makes it covariant here is the covariant=True keyword argument.
Now, with this version of Getter, Getter[int] is actually a subtype of Getter[object]. In general, if A is a subtype of B, then Getter[A] is a subtype of Getter[B]. Covariance preserves subtyping.
Okay, that's covariance. Now consider the opposite. Let's say we have a setter which sets some value in a database.
class Setter:
def set_value(self, value):
...
Same assumptions as before. Suppose we know what the type is in advance. Nowe we write
T = TypeVar("T")
class Setter:
def set_value(self, value: T) -> None:
...
Okay, great. Now, if I have a value my_setter : Setter[int], is that a Setter[object]? Well, my_setter can always take an integer value, whereas a Setter[object] is guaranteed to be able to take any object. my_setter can't guarantee that, so it's actually not. If we try to make T covariant in this example, we'll get
error: Cannot use a covariant type variable as a parameter
Because it's actually not a valid relationship. In fact, in this case, we get the opposite relationship. If we have a my_setter : Setter[object], then that's a guarantee that we can pass it any object at all, so certainly we can pass it an integer, hence we have a Setter[int]. This is called contravariance.
T_contra = TypeVar("T_contra", contravariant=True)
class Setter:
def set_value(self, value: T_contra) -> None:
...
We can make our type contravariant if it only appears in negative position, which (again, oversimplifying a bit) generally means that it appears as arguments to functions, but not as a return value. Now, Setter[object] is a subtype of Setter[int]. It's backwards. In general, if A is a subtype of B, then Setter[B] is a subtype of Setter[A]. Contravariance reverses the subtyping relationship.
Now, back to your example. You have a Callable[[DerivedClass], str] and want to know if it's a valid Callable[[BaseClass], str]
Applying our principles from before, we have a type Callable[[T], S] (I'm assuming only one argument for simplicity's sake, but in reality this works in Python for any number of arguments) and want to ask if T and S are covariant, contravariant, or invariant.
Well, what is a Callable? It's a function. It has one thing we can do: call it with a T and get an S. So it's pretty clear that T is only used as an argument and S as a result. Things only used as arguments are contravariant, and those used as results are covariant, so in reality it's more correct to write
Callable[[T_contra], S_co]
Arguments to Callable are contravariant, which means that if DerivedClass is a subtype of BaseClass, then Callable[[BaseClass], str] is a subtype of Callable[[DerivedClass], str], the opposite relationship to the one you suggested. You need a function that can accept any BaseClass. A function with a BaseClass argument would suffice, and so would a function with an object argument, or any type which is a supertype of BaseClass, but subtypes are insufficient because they're too specific for your contract.
MyPy objects to your call of p with r as its argument because given only the type signatures, it can't be sure the function won't be called with a non-DerivedClass instance.
For instance, given the same type annotations, p could be implemented like this:
def p(q: Callable[[BaseClass], str]) -> None:
obj = BaseClass()
q(obj)
This will break p(r) if r has an implementation that depends on the derived attributes of its argument:
def r(derived: DerivedClass) -> str:
return str(derived.q)
I want to define a function. It has one argument whose value is a class. But not all classes are expected, only some subclass of a certain class can be passed.
I don't know if it is possible to use type hint to achieve this expectation.
If it does, how can I do that?
As far as I know, the type hint can only help when requiring value is a instance of certain type. And the relevant documentation is really poor.
class MyClass(object):
pass
def my_function(arg):
# arg should be the subclass of MyClass.
return
I just want the definition of my_function produce the same effect of what type hint do (the only difference is that normal type hint show value is a instance of certain class, here I want it show value is a subclass of certain class).
If you are trying to say that arg must be an instance of either MyClass or some subtype of MyClass, just do this:
class MyClass(object): pass
class Child(MyClass): pass
def my_function(arg: MyClass) -> None:
pass
my_function(MyClass()) # Type checks
my_function(Child()) # Type checks
my_function(3.14) # Does not type check
If you are trying to say that arg must be literally the MyClass class object or a subtype, use typing.Type:
from typing import Type
# ...snip...
def my_function_2(arg: Type[MyClass]) -> None:
pass
my_function_2(MyClass) # Type checks
my_function_2(Child) # Type checks
my_function_2(int) # Does not type check
my_function_2(MyClass()) # Does not type check
Of course, the type checker won't be able to tell exactly which subtype you happen to be using within the functions themselves.
the type hint can only help when requiring value is a instance of certain type.
This is actually not something you can do. It is impossible to construct a type hint that asserts some value must be exactly one type and never any subtypes of that type (though you can often simulate something close-ish by using generics).
A type hint always indicates that some value is either an instance of that type or some subtype.
And the relevant documentation is really poor.
You may find reading the mypy docs to be useful, if you haven't found them already.
Mypy is a PEP 484 compliant static type checker, and was developed in parallel with PEP 484 itself. (E.g. the core mypy devs were the ones who wrote PEP 484 and several other typing-related PEPs.)
The upshot is that the docs for mypy are generally more helpful than the official Python docs for typing.