I'm trying to design my code as follows - i.e., I'd like that each subclass which implements my functionlity will have as member a collection of fields, which can also inherit from a base dataclass.
from dataclasses import dataclass
from abc import ABC, abstractmethod
#dataclass
class BaseFields:
pass
#dataclass
class MoreFields(baseFields):
name: str = "john"
class A(ABC):
def __init__(self) -> None:
super().__init__()
self.fields: BaseFields = BaseFields()
#abstractmethod
def say_hi(self) -> None:
pass
class B(A):
def __init__(self) -> None:
super().__init__()
self.fields = MoreFields()
def say_hi(self) -> None:
print(f"Hi {self.fields.name}!")
if __name__ == "__main__":
b = B()
b.say_hi()
When I run it, I get Hi john! as output, as expected.
But mypy doesn't seem to recognize it:
❯ mypy dataclass_inheritence.py
dataclass_inheritence.py:25: error: "baseFields" has no attribute "name"
Found 1 error in 1 file (checked 1 source file)
I looked and found this github issue, and it links to another one, but doesn't seem like it offers a solution.
I should also note that if I remove the #dataclass decorators and implement the Fields classes as plain ol' classes, with __init__ - I still get the same mypy error.
My motivation (as you may tell) is to reference composite members within the implemented methods of the functional subclasses. Those members are constants, as in the example, so I might use some form of Enum inheritance, but looking at this question it's not a popular design choice (will have to use some 3rd party module which I'm not keen on doing).
Has anyone encountered something similar? Do you have suggestions for a design that could achieve my goal?
The type of self.fields is declared as baseFields in A.__init__, and is not narrowed implicitly by assigning a moreFields to it in B.__init__ -- after all, you might want to be able to re-assign it to another baseFields instance, and it is therefore never assumed to be anything more specific than baseFields.
If you explicitly annotate it as moreFields in B.__init__, the error goes away:
class B(A):
def __init__(self) -> None:
super().__init__()
self.fields: moreFields = moreFields()
def say_hi(self) -> None:
print(f"Hi {self.fields.name}!") # ok!
although this actually feels like a bug in mypy, because now you can do this, violating the LSP:
if __name__ == "__main__":
b: A = B()
b.fields = baseFields() # no mypy error, because b is an A, right?
b.say_hi() # runtime AttributeError because b is actually a B!
If I want a subclass to be able to narrow the type of an attribute, I make it a property backed by private attributes:
class A(ABC):
def __init__(self) -> None:
super().__init__()
self.__baseFields = baseFields()
#property
def fields(self) -> baseFields:
return self.__baseFields
#abstractmethod
def say_hi(self) -> None:
pass
class B(A):
def __init__(self) -> None:
super().__init__()
self.__moreFields = moreFields()
#property
def fields(self) -> moreFields:
return self.__moreFields
def say_hi(self) -> None:
print(f"Hi {self.fields.name}!") # ok!
You can use a generic base class to define the class. I would also have the fields attribute be passed to the base class constructor. There are some subtle tricks to get the signature on the init method working, but this should work.
Some imports you'll want:
from __future__ import annotations
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Generic, TypeVar, overload
Rename the classes with more pythonic names, and define a generic TypeVar to represent which fields we are using.
#dataclass
class BaseFields:
pass
#dataclass
class MoreFields(BaseFields):
name: str = "john"
Fields = TypeVar('Fields', bound=BaseFields)
For defining the base class, we want to allow the fields param to be anything satisfying the TypeVar. We also need to add some overloads to handle the case where a default is used or not.
class A(Generic[Fields], ABC):
fields: Fields
#overload
def __init__(self: A[BaseFields]) -> None:
...
#overload
def __init__(self: A[Fields], fields: Fields) -> None:
...
def __init__(self, fields=None):
self.fields = fields or BaseFields()
#abstractmethod
def say_hi(self) -> None:
pass
Now we can run our test:
class B(A[MoreFields]):
def __init__(self) -> None:
super().__init__(MoreFields())
def say_hi(self) -> None:
print(f"Hi {self.fields.name}!")
if __name__ == "__main__":
b = B()
b.say_hi()
$ mypy test.py
Success: no issues found in 1 source file
Related
hope i won't leave out anything important. My very much simplified situation is like this:
In my domain I have some defined data structures:
#dataclass
class Model:
var_1: str
var_2: str
class Book(Model):
...
class Page(Model):
...
I want to have 2 abstraction steps defining how the data is processed, like :
PARAMETERS = TypeVar("PARAMETERS", bound=Model)
RESULT = TypeVar("RESULT", bound=Model)
class Finder(Generic[PARAMETERS, RESULT], metaclass=abc.ABCMeta):
def run(self, parameters_dict: Dict[str, str]) -> RESULT:
parameters = self.parse_parameters(parameters_dict)
return self.do_stuff(parameters)
#abc.abstractmethod
def parse_parameters(self, parameters_dict: Dict[str, str]) -> PARAMETERS:
...
#abc.abstractmethod
def do_stuff(self, parameters: PARAMETERS) -> RESULT:
...
BookParameter = TypeVar("BookParameter", bound=Book)
BookResult = TypeVar("BookResult", bound=Page)
class BookFinder(Generic[BookParameter, BookResult], Finder[BookParameter, BookResult], abc.ABC):
def parse_parameters(self, parameters_dict: Dict[str, str]) -> BookParameter:
return Book(**parameters_dict)
#abc.abstractmethod
def do_stuff(self, parameters: BookParameter) -> BookResult:
...
and then use it like:
class ItalianBookFinder(BookFinder[Book, Page]):
def do_stuff(self, parameters: Book) -> Page:
# define this
class LatinBook(Book):
var_3: str
class LatinBookFinder(BookFinder[LatinBook, Page]):
def parse_parameters(self, parameters_dict: Dict[str, str]) -> LatinBook:
# define both this
def do_stuff(self, parameters: LatinBook) -> Page:
# and this too
class EnglishPageFinder(PageFinder[EnglishPage, PageFinderResult]):
....
(i'm omitting the RESULT part, but is similar to PARAMETERS, too long text already)
But when i run mypy check on my code i get the error:
Incompatible return value type (got "Book", expected "BookParameter") [return-value]
which leads me to think i am not doing this correctly. i might be totally missing some important part, or have design flaw in here, if anyone knows i'll be happy for any input.
Thank you.
I have a use-case where I have two classes: Foo and Bar.
I want to write a function that, given one of these classes, dynamically creates a new class that is a subclass of the given class. This is what I currently have:
from typing import Type, TypeVar
class Foo:
...
class Bar
...
T = TypeVar("T")
def generate_child_class(base_class: Type[T]) -> Type[T]:
class GeneratedClassPlaceholder(base_class):
def __init__(self, **kwargs) -> None:
super(GeneratedClassPlaceholder, self).__init__(**kwargs)
# some additional logic unique to the subclass
...
return GeneratedClassPlaceholder
I intend to use that function as following:
FooChildClass = generate_child_class(Foo)
BarChildClass = generate_child_class(Bar)
However, the type-checker complains about the function: "Incompatible return type [7]: Expected Type[Variable[T]] but got Type[GeneratedFixtureClass]."
How do I correctly type-annotate this code? Could a metaclass be useful here?
Thank you!
I want to create an abstract base class in Python where part of the contract is how instances can be created. The different concrete implementations represent various algorithms that can be used interchangeably. Below is a simplified example (usual disclaimer - the real use-case is more complex):
from abc import ABC, abstractmethod
from typing import Type
class AbstractAlgorithm(ABC):
#abstractmethod
def __init__(self, param: int):
pass
#abstractmethod
def get_result(self) -> int:
pass
class ConcreteAlgorithm(AbstractAlgorithm):
def __init__(self, param: int):
self._param = param
def get_result(self) -> int:
return self._param * 2
def use_algorithm(algorithm: Type[AbstractAlgorithm]) -> int:
a = algorithm(10)
return a.get_result()
The above works, but has the drawback that I can't call super().__init__(...) in ConcreteAlgorithm.__init__, which might break certain inheritance scenarios, I think (correct me if I'm wrong here, but calling super is important for multiple inheritance, right?). (Strictly speaking __init__ can be called, but with the same signature as the subclass __init__, which doesn't make sense).
Python classes are callables, so I could also express it like this:
from abc import ABC, abstractmethod
from typing import Callable
class AbstractAlgorithm(ABC):
#abstractmethod
def get_result(self) -> int:
pass
class ConcreteAlgorithm(AbstractAlgorithm):
def __init__(self, param: int):
self._param = param
def get_result(self) -> int:
return self._param * 2
def use_algorithm(algorithm: Callable[[int], AbstractAlgorithm]) -> int:
a = algorithm(10)
return a.get_result()
print(use_algorithm(ConcreteAlgorithm))
This works and doesn't have the drawback mentioned above, but I do like having the __init__-signature in the abstract base class for documentation purposes.
Finally, it is possible to have abstract classmethods, so this approach works as well:
from abc import ABC, abstractmethod
from typing import Type
class AbstractAlgorithm(ABC):
#classmethod
#abstractmethod
def initialize(cls, param: int) -> "AbstractAlgorithm":
pass
#abstractmethod
def get_result(self) -> int:
pass
class ConcreteAlgorithm(AbstractAlgorithm):
#classmethod
def initialize(cls, param: int) -> "ConcreteAlgorithm":
return cls(param)
def __init__(self, param: int):
self._param = param
def get_result(self) -> int:
return self._param * 2
def use_algorithm(algorithm: Type[AbstractAlgorithm]) -> int:
a = algorithm.initialize(10)
return a.get_result()
print(use_algorithm(ConcreteAlgorithm))
This works, but I lose the nice property of using algorithm like a callable (it's just more flexible, in case someone actually wants to drop in a function, for example to decide which algorithm to use based on certain parameter values).
So, is there an approach that satisfies all three requirements:
Full documentation of the interface in the abstract base class.
Concrete implementations usable as callables.
No unsafe behavior like not being able to call the base-class __init__.
Strictly speaking __init__ can be called, but with the same signature as the subclass __init__, which doesn't make sense.
No, it makes perfect sense.
You're prescribing the signature because you require each child class to implement it exactly. That means you need to call it exactly like that as well. Each child class needs to call its super().__init__ exactly according to the abstract definition, passing all defined parameters along.
I'm trying to implement two abstract classes in one class, but the two abstract classes contain abstract methods with the same name. In C#, I would be able to explicitly implement the abstract methods allowing them to be called on the context of the type. Is there a way to do something similar in python to allow for both abstract classes to be implemented?
from abc import ABC, abstractmethod
from builtins import str
class AbstractConfig1(ABC):
#property
#abstractmethod
def unique_prop(self) -> str:
pass
#property
#abstractmethod
def output_filepath(self) -> str: ## same name in AbstractConfig2
pass
class AbstractConfig2(ABC):
#property
#abstractmethod
def other_unique_prop(self) -> str:
pass
#property
#abstractmethod
def output_filepath(self) -> str: ## same name in AbstractConfig1
pass
class Config(AbstractConfig1, AbstractConfig2):
def __init__(self,
unique_prop:str,
other_unique_prop:str,
config1_output_filepath: str,
config2_output_filepath: str
):
self._unique_prop = unique_prop
self._other_unique_prop = other_unique_prop
self._config1_output_filepath = config1_output_filepath
self._config2_output_filepath = config2_output_filepath
#property
def unique_prop(self) -> str:
return self._unique_prop
#property
def other_unique_prop(self) -> str:
return self._other_unique_prop
#property
def AbstractConfig1.output_filepath(self) -> str: ## How I would explicitly implement this in C#
return self._config1_output_filepath
#property
def AbstractConfig2.output_filepath(self) -> str: ## How I would explicitly implement this in C#
return self._config2_output_filepath
Here is a link to what I'm attempting in terms of C#
https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/interfaces/explicit-interface-implementation
Edit to clear things up a little more:
I simplified this a little more than what my code is doing, instead of just passing through a string, the abstract methods I named output_filepath are returning objects built in the Config class. But I will continue using str in the example to simplify.
Essentially the Config class is acting as a facade to multiple AbstractConfig classes. This way, the facade Config can be configured and then passed to initialize other objects. This would look a bit like below:
class ClassUsingAbstractConfig1:
def __init__(self, config: AbstractConfig1):
self.config = config
def output_file(self):
path = self.config.output_filepath
# this object outputs to one filepath
class ClassUsingAbstractConfig2:
def __init__(self, config: AbstractConfig2):
self.config = config
def output_file(self):
path = self.config.output_filepath
# this object outputs to another filepath
config = Config("prop",
"prop2",
"filepath1",
"filepath2")
class1 = ClassUsingAbstractConfig1(config)
class2 = ClassUsingAbstractConfig2(config)
class1.output_file() # outputs to filepath1
class2.output_file() # outputs to filepath2
And it may just be that python won't allow this and I need to take a different approach.
You have to ask yourself: "What is the signature of string: Config::output_filepath(Config: self)"
What you're referring to is the Multiple inheritance - Diamond Problem. Basically, the class: Config can have only 1 implementation for the method (function) with the same signature. You have to imagine that each function uses its signature as the index for the function call table. This is how you call a function from one class to another, especially when they share the same name. But in your case it shares the same signature.
As a consequence, I think you'll have something like:
from abc import ABC, abstractmethod
from builtins import str
class AbstractConfig1(ABC):
#property
#abstractmethod
def unique_prop(self) -> str:
pass
#property
#abstractmethod
def output_filepath(self) -> str: ## same name in AbstractConfig2
pass
class AbstractConfig2(ABC):
#property
#abstractmethod
def other_unique_prop(self) -> str:
pass
#property
#abstractmethod
def output_filepath(self) -> str: ## same name in AbstractConfig1
pass
class Config(AbstractConfig1, AbstractConfig2):
def __init__(self,
unique_prop:str,
other_unique_prop:str,
config1_output_filepath: str,
config2_output_filepath: str
):
self._unique_prop = unique_prop
self._other_unique_prop = other_unique_prop
self._config1_output_filepath = config1_output_filepath
self._config2_output_filepath = config2_output_filepath
#property
def unique_prop(self) -> str:
return self._unique_prop
#property
def other_unique_prop(self) -> str:
return self._other_unique_prop
#property
def output_filepath(self) -> str:
# or whatever the implementation you want to be.
return self._config1_output_filepath + self._config2_output_filepath
This problem exists in virtually all high level languages and is related to the concept of Class and Function and how that ends up being translated on the Language Virtual Machine (if is a C#, Java, Python etc.) all the way to the kernel and CPU to be loaded and executed (other answer)
I want to use multiple generic protocols and ensure they're compatible:
from typing import TypeVar, Protocol, Generic
from dataclasses import dataclass
# checking fails as below and with contravariant=True or covariant=True:
A = TypeVar("A")
class C(Protocol[A]):
def f(self, a: A) -> None: pass
class D(Protocol[A]):
def g(self) -> A: pass
# Just demonstrates my use case; doesn't have errors:
#dataclass
class CompatibleThings(Generic[A]):
c: C[A]
d: D[A]
Mypy gives the following error:
Invariant type variable 'A' used in protocol where contravariant one is expected
Invariant type variable 'A' used in protocol where covariant one is expected
I know this can be done by making C and D generic ABC classes, but I want to use protocols.
The short explanation is that your approach breaks subtype transitivity; see this section of PEP 544 for more information. It gives a pretty clear explanation of why your D protocol (and, implicitly, your C protocol) run into this problem, and why it requires different types of variance for each to solve it. You can also look on Wikipedia for info on type variance.
Here's the workaround: use covariant and contravariant protocols, but make your generic dataclass invariant. The big hurdle here is inheritance, which you have to handle in order to use Protocols, but is kind of tangential to your goal. I'm going to switch naming here to highlight the inheritance at play, which is what this is all about:
A = TypeVar("A") # Invariant type
A_cov = TypeVar("A_cov", covariant=True) # Covariant type
A_contra = TypeVar("A_contra", contravariant=True) # Contravariant type
# Give Intake its contravariance
class Intake(Protocol[A_contra]):
def f(self, a: A_contra) -> None: pass
# Give Output its covariance
class Output(Protocol[A_cov]):
def g(self) -> A_cov: pass
# Just tell IntakeOutput that the type needs to be the same
# Since a is invariant, it doesn't care that
# Intake and Output require contra / covariance
#dataclass
class IntakeOutput(Generic[A]):
intake: Intake[A]
output: Output[A]
You can see that this works with the following tests:
class Animal:
...
class Cat(Animal):
...
class Dog(Animal):
...
class IntakeCat:
def f(self, a: Cat) -> None: pass
class IntakeDog:
def f(self, a: Dog) -> None: pass
class OutputCat:
def g(self) -> Cat: pass
class OutputDog:
def g(self) -> Dog: pass
compat_cat: IntakeOutput[Cat] = IntakeOutput(IntakeCat(), OutputCat())
compat_dog: IntakeOutput[Dog] = IntakeOutput(IntakeDog(), OutputDog())
# This is gonna error in mypy
compat_fail: IntakeOutput[Dog] = IntakeOutput(IntakeDog(), OutputCat())
which gives the following error:
main.py:48: error: Argument 2 to "IntakeOutput" has incompatible type "OutputCat"; expected "Output[Dog]"
main.py:48: note: Following member(s) of "OutputCat" have conflicts:
main.py:48: note: Expected:
main.py:48: note: def g(self) -> Dog
main.py:48: note: Got:
main.py:48: note: def g(self) -> Cat
So what's the catch? What are you giving up? Namely, inheritance in IntakeOutput. Here's what you can't do:
class IntakeAnimal:
def f(self, a: Animal) -> None: pass
class OutputAnimal:
def g(self) -> Animal: pass
# Ok, as expected
ok1: IntakeOutput[Animal] = IntakeOutput(IntakeAnimal(), OutputAnimal())
# Ok, because Output is covariant
ok2: IntakeOutput[Animal] = IntakeOutput(IntakeAnimal(), OutputDog())
# Both fail, because Intake is contravariant
fails1: IntakeOutput[Animal] = IntakeOutput(IntakeDog(), OutputDog())
fails2: IntakeOutput[Animal] = IntakeOutput(IntakeDog(), OutputAnimal())
# Ok, because Intake is contravariant
ok3: IntakeOutput[Dog] = IntakeOutput(IntakeAnimal(), OutputDog())
# This fails, because Output is covariant
fails3: IntakeOutput[Dog] = IntakeOutput(IntakeAnimal(), OutputAnimal())
fails4: IntakeOutput[Dog] = IntakeOutput(IntakeDog(), OutputAnimal())
So. There it is. You can play around with this more here.