I'm trying to write a small mixin class to somewhat bridge Set and MutableMapping types: I want the mapping types to have ability to receive some objects (bytes), hash them, and store them, so they are accessible by that hash.
Here's a working version of mixing this class with standard dict:
from hashlib import blake2b
class HashingMixin:
def add(self, content):
digest = blake2b(content).hexdigest()
self[digest] = content
class HashingDict(dict, HashingMixin):
pass
However I can't figure out how to add type annotations.
From https://github.com/python/mypy/issues/1996 it seems the mixin has to subclass abc.ABC and abc.abstractmethod-define all the methods it expects to call, so here's my shot:
import abc
from hashlib import blake2b
from typing import Dict
class HashingMixin(abc.ABC):
def add(self, content: bytes) -> None:
digest = blake2b(content).hexdigest()
self[digest] = content
#abc.abstractmethod
def __getitem__(self, key: str) -> bytes:
raise NotImplementedError
#abc.abstractmethod
def __setitem__(self, key: str, content: bytes) -> None:
raise NotImplementedError
class HashingDict(Dict[str, bytes], HashingMixin):
pass
Then Mypy complains about the HashingDict definition:
error: Definition of "__getitem__" in base class "dict" is incompatible with definition in base class "HashingMixin"
error: Definition of "__setitem__" in base class "dict" is incompatible with definition in base class "HashingMixin"
error: Definition of "__setitem__" in base class "MutableMapping" is incompatible with definition in base class "HashingMixin"
error: Definition of "__getitem__" in base class "Mapping" is incompatible with definition in base class "HashingMixin"
Revealing types with:
reveal_type(HashingMixin.__getitem__)
reveal_type(HashingDict.__getitem__)
yields:
error: Revealed type is 'def (coup.content.HashingMixin, builtins.str) -> builtins.bytes'
error: Revealed type is 'def (builtins.dict[_KT`1, _VT`2], _KT`1) -> _VT`2'
I don't know what is wrong :(
This appears to be a bug in mypy -- see this TODO in the code mypy uses to analyze the MRO of classes using multiple inheritance. In short, mypy is incorrectly completing ignoring that you've parameterized Dict with concrete values, and is instead analyzing the code as if you were using Dict.
I believe https://github.com/python/mypy/issues/5973 is probably the most relevant issue in the issue tracker: the root cause is the same.
Until that bug is fixed, you can suppress the errors mypy is generating on that line by adding a # type: ignore to whatever line has the errors. So in your case, you could do the following:
import abc
from hashlib import blake2b
from typing import Dict
class HashingMixin(abc.ABC):
def add(self, content: bytes) -> None:
digest = blake2b(content).hexdigest()
self[digest] = content
#abc.abstractmethod
def __getitem__(self, key: str) -> bytes:
raise NotImplementedError
#abc.abstractmethod
def __setitem__(self, key: str, content: bytes) -> None:
raise NotImplementedError
class HashingDict(Dict[str, bytes], HashingMixin): # type: ignore
pass
If you decide to take this approach, I recommend also leaving an additional comment documenting why you're suppressing those errors and running mypy with the --warn-unused-ignores flag.
The former is for the benefit of any future readers of your code; the latter will make mypy report a warning whenever it encounters a # type: ignore that is not actually suppressing any errors and so can safely be deleted.
(And of course, you can always take a stab at contributing a fix yourself!)
Related
Given a class with a helper method for initialization:
class TrivialClass:
def __init__(self, str_arg: str):
self.string_attribute = str_arg
#classmethod
def from_int(cls, int_arg: int) -> ?:
str_arg = str(int_arg)
return cls(str_arg)
Is it possible to annotate the return type of the from_int method?
I'v tried both cls and TrivialClass but PyCharm flags them as unresolved references which sounds reasonable at that point in time.
Starting with Python 3.11 you can use the new typing.Self object. For older Python versions you can get the same object by using the typing-extensions project:
try:
from typing import Self
except ImportError:
from typing_extensions import Self
class TrivialClass:
# ...
#classmethod
def from_int(cls, int_arg: int) -> Self:
# ...
return cls(...)
Note that you don't need to annotate cls in this case.
Warning: mypy support for the Self type has not yet been released; you'll need to wait for the next version after 0.991. Pyright already supports it.
If you can't wait for Mypy support, then you can use a generic type to indicate that you'll be returning an instance of cls:
from typing import Type, TypeVar
T = TypeVar('T', bound='TrivialClass')
class TrivialClass:
# ...
#classmethod
def from_int(cls: Type[T], int_arg: int) -> T:
# ...
return cls(...)
Any subclass overriding the class method but then returning an instance of a parent class (TrivialClass or a subclass that is still an ancestor) would be detected as an error, because the factory method is defined as returning an instance of the type of cls.
The bound argument specifies that T has to be a (subclass of) TrivialClass; because the class doesn't yet exist when you define the generic, you need to use a forward reference (a string with the name).
See the Annotating instance and class methods section of PEP 484.
Note: The first revision of this answer advocated using a forward reference
naming the class itself as the return value, but issue 1212 made it possible to use generics instead, a better solution.
As of Python 3.7, you can avoid having to use forward references in annotations when you start your module with from __future__ import annotations, but creating a TypeVar() object at module level is not an annotation. This is still true even in Python 3.10, which defers all type hint resolution in annotations.
From Python 3.7 you can use __future__.annotations:
from __future__ import annotations
class TrivialClass:
# ...
#classmethod
def from_int(cls, int_arg: int) -> TrivialClass:
# ...
return cls(...)
Edit: you can't subclass TrivialClass without overriding the classmethod, but if you don't require this then I think it's neater than a forward reference.
A simple way to annotate the return type is to use a string as the annotation for the return value of the class method:
# test.py
class TrivialClass:
def __init__(self, str_arg: str) -> None:
self.string_attribute = str_arg
#classmethod
def from_int(cls, int_arg: int) -> 'TrivialClass':
str_arg = str(int_arg)
return cls(str_arg)
This passes mypy 0.560 and no errors from python:
$ mypy test.py --disallow-untyped-defs --disallow-untyped-calls
$ python test.py
In Python 3.11 there is a nicer way to do this using the new Self type:
from typing import Self
class TrivialClass:
def __init__(self, str_arg: str):
self.string_attribute = str_arg
#classmethod
def from_int(cls, int_arg: int) -> Self:
str_arg = str(int_arg)
return cls(str_arg)
This also works correctly with sub classes as well.
class TrivialSubClass(TrivialClasss):
...
TrivialSubclass.from_int(42)
The IDE shows return type TrivialSubClass and not TrivialClass.
This is described in PEP 673.
I have a decorator that creates an abstractmethod from a simple method. It works as I'd expect, however if I run mypy, it tells me this:
mypy_try.py:20: error: Missing return statement [empty-body]
mypy_try.py:20: note: If the method is meant to be abstract, use #abc.abstractmethod
Found 1 error in 1 file (checked 1 source file)
My code:
import abc
from functools import wraps
import pytest
def make_it_abstract(method_to_decorate):
#wraps(method_to_decorate)
def decorated_method(*method_args, **method_kwargs):
return method_to_decorate(*method_args, **method_kwargs)
return abc.abstractmethod(decorated_method)
class MyInterfaceClass(abc.ABC):
#make_it_abstract
# #abc.abstractmethod
def my_method(self, value: int) -> int:
...
def test_abstract_method():
class MyImplementationClass(MyInterfaceClass):
pass
with pytest.raises(
TypeError,
match="Can't instantiate abstract class MyImplementationClass with abstract method my_method"
):
MyImplementationClass()
class MyImplementationClass(MyInterfaceClass):
def my_method(self, value: int) -> float:
return value +1
assert 43 == MyImplementationClass().my_method(42)
If I use the abc.abstractmethod decorator, it works fine.
What am I doing wrong?
You're doind everything fine, but mypy is not smart enough to figure out that your decorator calls abc.abstractmethod (and this is almost impossible, in fact, even if you've typed the decorator).
According to code in typeshed, abstractmethod is a no-op for type checkers. So mypy just detects the usage of abc.abstractmethod as decorator directly, as can be seen here. refers_to_fullname method expands aliases and basically checks if node name is equal to one of requested names.
So even the following raises the same error:
ab = abc.abstractmethod
class MyInterfaceClass(abc.ABC):
#ab
def my_method(self, value: int) -> int: # E: Missing return statement [empty-body]
...
In a class Foo I link a static method in a class variable to subclass that class later and just exchange that function with another. The class contains some methods which call this exchangable function. The code below does not produce any mypy issues.
def some_function(text: str) -> None:
print(text)
class Foo:
_some_func: ClassVar[staticmethod] = staticmethod(some_function)
def some_method(self, text: str) -> None:
self._some_func(text)
if __name__ == "__main__":
Foo().some_method("Hello World!")
Now, I am trying to improve my typing, so I want to use a callback protocol to actually add typing for Foo._some_func. I have created the following protocol class:
class SomeFuncProtocol(Protocol):
def __call__(self, __text: str) -> None:
...
It does work as long as I use _some_func: ClassVar[SomeFuncProtocol] = some_function, but I can't find a way to use staticmethod and the protocol class for typing. I wish for something like the following, but mypy tells me that staticmethod does not expect a type argument.
class Foo:
_some_func: ClassVar[staticmethod[SomeFuncProtocol]] = staticmethod(some_function)
...
Does anybody know how to do it?
I was stuck with similar thing for a while, here's what worked for me:
from typing import ClassVar, Protocol
def some_function(text: str) -> None:
print(text)
class SomeFuncProtocol(Protocol):
def __call__(self, __text: str) -> None:
return
class Foo:
_some_func: ClassVar[SomeFuncProtocol] = staticmethod(some_function)
Foo._some_func('a')
Foo()._some_func('a')
Foo._some_func = some_function
Foo()._some_func = some_function # E: Cannot assign to class variable "_some_func" via instance
The code above typechecks (except for last line that is intentionally incorrect).
You don't need staticmethod in type annotation: it's a function that (simplified) takes callable as argument and returns another callable with same signature, but with explicit sign that it doesn't accept self. So return type of staticmethod is the same callable, we can express it like this:
from typing import Any, Callable, TypeVar
_C = TypeVar('_C', bound=Callable[..., Any])
def staticmethod(func: _C) -> _C: ...
You can try it in playground.
I am trying to set up a class decorator in Python that acts like attr.frozen but adds an additional field before creation (as well as a few other things). While the code works fine, I'm having trouble getting mypy to realize that the new class has the new field. I've tried to do this through a combination of a custom mypy plugin (exactly as described in attr's documentation) and a Protocol that defines that the new class has the given field. In summary, the code breaks down as follows (all in a single file, although I've broken it up here).
It should be noted I'm running Python 3.7, so I'm using typing_extensions where needed, but I believe this problem persists regardless of version.
First define the Protocol that should inform mypy that the new class has the new field (called added here):
from typing_extensions import Protocol
class Proto(Protocol):
def __init__(self, added: float, *args, **kwargs):
...
#property
def added(self) -> float:
...
Now define the field_transformer function that adds the new field, as per attr's documentation:
from typing import Type, List
import attr
def _field_transformer(cls: type, fields: List[attr.Attribute]) -> List[attr.Attribute]:
return [
# For some reason mypy has trouble with attr.Attribute's signature
# Bonus points if someone can point out a fix that doesn't use type: ignore
attr.Attribute ( # type: ignore
"added", # name
attr.NOTHING, # default
None, # validator
True, # repr
None, # cmp
None, # hash
True, # init
False, # inherited
type=float,
order=float,
),
*fields,
]
Now, finally, set up a class decorator that does what we want:
from functools import wraps
from typing import Callable, TypeVar
_T = TypeVar("_T", bound=Proto)
_C = TypeVar("_C", bound=type)
def transform(_cls: _C = None, **kwargs):
def transform_decorator(cls: _C) -> Callable[[], Type[_T]]:
#wraps(cls)
def wrapper() -> Type[_T]:
if "field_transformer" not in kwargs:
kwargs["field_transformer"] = _field_transformer
return attr.frozen(cls, **kwargs)
return wrapper()
if _cls is None:
return transform_decorator
return transform_decorator(_cls)
And now for the (failing) mypy tests:
#transform
class Test:
other_field: str
# E: Too many arguments for "Test"
# E: Argument 1 to "Test" has incompatible type "float"; expected "str"
t = Test(0.0, "hello, world")
print(t.added) # E: "Test" has no attribute "added"
Ideally I'd like mypy to eliminate all three of these errors. I am frankly not sure whether this is possible; it could be that the dynamic addition of an attribute is just not typeable and we may have to force users of our library to write custom typing stubs when they use the decorator. However, since we always add the same attribute(s) to the generated class, it would be great if there is a solution, even if that means writing a custom mypy plugin that supports this decorator in particular (if that's even possible).
While trying to update my code to be PEP-484 compliant (I'm using mypy 0.610) I've ran into the following report:
$ mypy mymodule --strict-optional --ignore-missing-imports --disallow-untyped-calls --python-version 3.6
myfile.py:154: error: Signature of "deliver" incompatible with supertype "MyClass"
MyClass:
from abc import abstractmethod
from typing import Any
class MyClass(object):
#abstractmethod
def deliver(self, *args: Any, **kwargs: Any) -> bool:
raise NotImplementedError
myfile.py:
class MyImplementation(MyClass):
[...]
def deliver(self, source_path: str,
dest_branches: list,
commit_msg: str = None,
exclude_files: list = None) -> bool:
[...]
return True
I'm definitely doing something wrong here, but I can't quite understand what :)
Any pointers would be much appreciated.
#abstractmethod
def deliver(self, *args: Any, **kwargs: Any) -> bool:
raise NotImplementedError
This declaration doesn't mean subclasses can give deliver any signature they want. Subclass deliver methods must be ready to accept any arguments the superclass deliver method would accept, so your subclass deliver has to be ready to accept arbitrary positional or keyword arguments:
# omitting annotations
def deliver(self, *args, **kwargs):
...
Your subclass's deliver does not have that signature.
If all subclasses are supposed to have the same deliver signature you've written for MyImplementation, then you should give MyClass.deliver the same signature too. If your subclasses are going to have different deliver signatures, maybe this method shouldn't really be in the superclass, or maybe you need to rethink your class hierarchy, or give them the same signature.
You can solve the question by using Callable[..., Any] and type: ignore such like bellow.
from typing import Callable
class MyClass(object):
deliver: Callable[..., bool]
#abstractmethod
def deliver(self, *args, **kwargs): # type: ignore
raise NotImplementedError
Maybe you should work it around this way:
Define abstract method without arguments:
class MyClass:
#abstractmethod
def deliver(self) -> bool:
raise NotImplementedError
In implementations get all your data from self:
class MyImplementation(MyClass):
def __init__(
self,
source_path: str,
dest_branches: list,
commit_msg: str = None,
exclude_files: list = None
) -> None:
super().__init__()
self.source_path = source_path
self.dest_branches = dest_branches
self.commit_msg = commit_msg
self.exclude_files = exclude_files
def deliver(self) -> bool:
# some logic
if self.source_path and self.commit_msg:
return True
return False
This way you will have completely compatible method declarations and still can implement methods as you want.