As far as I understand var is a class variable here:
class MyClass:
var = 'hello'
def __init__(self):
print(self.var)
And thats an instance variable:
class MyClass:
def __init__(self, var):
self.var = var
print(self.var)
I had the problem, that I was looking for a method to make type hinting possible for instance variables. I can of course typehint the parameter with def __init__(self, var: str): but that would not effect the instance variable itself.
Then I noticed in some descriptions (like here) that they used the term instance variable for a var like this:
class MyClass:
var : str = 'hello'
def __init__(self, var : str = None):
self.var = var if var
print(self.var)
That would be the solution indeed, but is that still an instance variable? Because it is defined in the class body, it would be a class variable in my understanding. If you would use a list for var, all alterations to this list-var would be shared over the instances.
But in this case there would be no problem, because the string is replaced and would not be shared for other instances. However, it seems wrong to me if you call it an instance variable and I don't know if I should use it like this just to have the type hinting working.
That would be the solution indeed, but is that still an instance variable? Because it is defined in the class body, it would be a class variable in my understanding. [...snip...] However, it seems wrong to me if you call it an instance variable and I don't know if I should use it like this just to have the type hinting working.
For what it's worth, I also share the same discomfort. It seems like we're conceptually mixing two concepts there just for the sake of having cleaner type annotations.
However, I've asked Guido one or two times about this, and it seems like he does indeed prefers treating those class attributes as if they were instance attributes.
In any case, to answer your core question, if we do this:
class Test:
field1: int
field2: str = 'foo'
Then...
PEP 484 and 526 compliant type checkers will treat this class as if:
It has an instance attribute named field1
It has an instance attribute named field2 that has a default value of 'foo' (as per PEP 526).
At runtime, ignoring type hints, Python will:
Add a class annotation named field1 to Test, but not a class attribute. (Class annotations are not automatically turned into class attributes.)
Add both a class annotation named field2 to Test as well as a class attribute named field2 containing the value 'foo'.
So, it can get a bit muddled.
But regardless, this then begs the question: how do we indicate to a type checker that we want some field to genuinely be a class attribute?
Well, it turns out PEP 484 was amended semi-recently to contain the ClassVar type annotation, which does exactly that.
So, if we wanted to add a new class attribute, we could do this:
from typing import ClassVar
class Test:
field1: int
field2: str = 'foo'
field3: ClassVar[int] = 3
So now, field3 should be treated as a class attribute with a default value of '3'.
(Note: ClassVar was added to typing for Python 3.5.3 -- if you're using the older version of typing bundled with Python 3.5, you can get a "backport" of the type by installing the typing_extensions third part module via pip and importing ClassVar from there instead.)
I think whether you decide to embrace this approach or not use it is a personal preference.
On one hand, Guido's opinion, pretty much by definition, defines what's "Pythonic" or not, so from that stance, there's no issue adopting this new idiom. Furthermore, the language itself is slowly but surely shifting to adopt this new idiom -- see the very recently accepted PEP 557, for example, which ends up following this same idiom of treating class attributes/class annotations as instance attributes.
On the other hand, it's difficult to shake off the nagging worry that this subtle difference will lead to issues down the line. In that case, you could stick with the standard approach of just setting all your fields inside __init__. This approach also has the benefit of keeping your code compatible with Python 2 and 3.x - 3.5.
A middle ground might be to just simply never use class attributes, in any way, shape, or form, and just stick to using class annotations. This is slightly restrictive, since we can no longer give our instance variables default values, but we can now avoid conflating class attributes with instance attributes entirely. (As previously stated, and as pointed out in the comments, class annotations are not added as class attributes.)
Related
I'm trying to create a class which has an attribute that should be a constant. This attribute could have different types depending on were the class is used in the codebase. Moreover, the type of this attribute is used in various type hints throughout the class¹, so I decided to convert the class to a Generic, like so:
from typing import TypeVar, Generic, Final
T = TypeVar("T")
class Foo(Generic[T]):
bar: Final[T]
def __init__(self, bar: T) -> None:
self.bar = bar
However, MyPy complains saying that
Final name declared in class body cannot depend on type variables
while if I remove the Final annotation MyPy doesn't raise any errors.
I can't find any logical errors in my code: it simply says that, regardless of its type, the bar attribute should always be referencing the same object. Am I missing something or is this some limitation of Python and/or of MyPy?
¹In the example I'm showing only one such usage to keep things simple.
P.S. I'm using Python 3.10.8 and MyPy 0.991.
Your code is correct per PEP591, mypy applies the rules in a wrong order, annotate in __init__ to solve the issue.
Here are the links to the docs and PEP591.
mypy should've checked the presence of initializer and then decide whether the initializer is missing, but it doesn't in fact and thinks that you define a final class attribute despite missing initializer. Type variables make no sense in types of class attributes (because type variables are bound to instances, not classes), so mypy errors here.
Thus we need to help mypy resolve the kind of Final properly. To do so, we can annotate attribute in __init__ instead:
from typing import TypeVar, Generic, Final
_T = TypeVar("_T")
class Foo(Generic[_T]):
def __init__(self, bar: _T):
self.bar: Final[_T] = bar
This typechecks now (playground).
This seems to be a known restriction of mypy and a loosening of this restriction was mentioned in this feature request over two years ago. It seem to be a rather niche problem judging from the lack of discussion around it, but you can always participate there. (I would say after such a long time since the last activity in that thread, you might respectfully bump it, if you include a reasonable use case and explain why you would want this feature as well.)
This restriction around typing.Final is not documented by mypy, so at least that might be worth a mention.
The current workaround seems to be to omit the annotation in the class namespace and just annotate the instance attribute during assignment inside the __init__ method like this:
from typing import TypeVar, Generic, Final
T = TypeVar("T")
class Foo(Generic[T]):
def __init__(self, bar: T) -> None:
self.bar: Final[T] = bar
class SubA(Foo[T]):
def __init__(self, bar: T) -> None:
self.bar = bar
class SubB(Foo[int]):
bar = 42
This works as expected and mypy gives us an error for the attempt to re-assign bar:
error: Cannot assign to final attribute "bar" [misc]
error: Cannot assign to final name "bar" [misc]
I'm modifying an app, trying to use Pydantic for my application models and SQLAlchemy for my database models.
I have existing classes, where I defined attributes inside the __init__ method as I was taught to do:
class Measure:
def __init__(
self,
t_received: int,
mac_address: str,
data: pd.DataFrame,
battery_V: float = 0
):
self.t_received = t_received
self.mac_address = mac_address
self.data = data
self.battery_V = battery_V
In both Pydantic and SQLAlchemy, following the docs, I have to define those attributes outside the __init__ method, for example in Pydantic:
import pydantic
class Measure(pydantic.BaseModel):
t_received: int
mac_address: str
data: pd.DataFrame
battery_V: float
Why is it the case? Isn't this bad practice? Is there any impact on other methods (classmethods, staticmethods, properties ...) of that class?
Note that this is also very unhandy because when I instantiate an object of that class, I don't get suggestions on what parameters are expected by the constructor!
Defining attributes of a class in the class namespace directly is totally acceptable and is not special per se for the packages you mentioned. Since the class namespace is (among other things) essentially a blueprint for instances of that class, defining attributes there can actually be useful, when you want to e.g. provide all public attributes with type annotations in a single place in a consistent manner.
Consider also that a public attribute does not necessarily need to be reflected by a parameter in the constructor of the class. For example, this is entirely reasonable:
class Foo:
a: list[int]
b: str
def __init__(self, b: str) -> None:
self.a = []
self.b = b
In other words, just because something is a public attribute, that does not mean it should have to be provided by the user upon initialization. To say nothing of protected/private attributes.
What is special about Pydantic (to take your example), is that the metaclass of BaseModel as well as the class itself does a whole lot of magic with the attributes defined in the class namespace. Pydantic refers to a model's typical attributes as "fields" and one bit of magic allows special checks to be done during initialization based on those fields you defined in the class namespace. For example, the constructor must receive keyword arguments that correspond to the non-optional fields you defined.
from pydantic import BaseModel
class MyModel(BaseModel):
field_a: str
field_b: int = 1
obj = MyModel(
field_a="spam", # required
field_b=2, # optional
field_c=3.14, # unexpected/ignored
)
If I were to omit field_a during construction of a MyModel instance, an error would be raised. Likewise, if I had tried to pass field_b="eggs", an error would be raised.
So the fact that you don't write your own __init__ method is a feature Pydantic provides you. You only define the fields and an appropriate constructor is "magically" there for you already.
As for the drawback you mentioned, where you don't get any auto-suggestions, that is true by default for all IDEs. Static type checkers cannot understand that dynamic constructor and simply infer what arguments are expected. Currently this is solved via extensions, such as the mypy plugin and the PyCharm plugin. Maybe soon the #dataclass_transform decorator from PEP 681
will standardize this for similar packages and thus improve support by static type checkers.
It is also worth noting that even the standard library's dataclasses only work via special extensions in type checkers.
To your other question, there is obviously some impact on methods of such classes (by design), though the specifics are not always obvious. You should of course not simply write your own __init__ method without being careful to call the superclass' __init__ properly inside it. Also, #property-setters currently don't work as you would expect it (though it is debatable if it even makes sense to use properties on Pydantic models).
To wrap up, this approach is not only not bad practice, it is a great idea to reduce boilerplate code and it is extremely common these days, as evidenced by the fact that hugely popular and established packages (like the aforementioned Pydantic, as well as e.g. SQLAlchemy, Django and others) use this pattern to a certain extent.
Pydantic has its own (rewriting) magic, but SQLalchemy is a bit easier to explain.
A SA model looks like this :
>>> from sqlalchemy import Column, Integer, String
>>> class User(Base):
...
... id = Column(Integer, primary_key=True)
... name = Column(String)
Column, Integer and String are descriptors. A descriptor is a class that overrides the get and set methods. In practice, this means the class can control how data is accessed and stored.
For example this assignment would now use the __set__ method from Column:
class User(Base):
id = Column(Integer, primary_key=True)
name = Column(String)
user = User()
user.name = 'John'
This is the same as user.name.__set__('John') , however, because of the MRO, it finds a set method in Column, so uses that instead. In a simplified version the Column looks something like this:
class Column:
def __init__(self, field=""):
self.field= field
def __get__(self, obj, type):
return obj.__dict__.get(self.field)
def __set__(self, obj, val):
if validate_field(val)
obj.__dict__[self.field] = val
else:
print('not a valid value')
(This is similar to using #property. A Descriptor is a re-usable #property)
I don't like getting complaints from Mypy about function signatures, but I don't know how to resolve this one.
I am building a package, that is to be used by several programs. I have a subclass of IntEnum (FWIW, called _Event), which contains a number of relevant properties and methods. _Event is never used directly, since it contains no members, but several different programs use incompatible subclasses of _Event (AlphaEvent, BetaEvent, etc.), which define the actual members of the Enum (i.e, the actual events). Only one member of each subclass is common, and as it happens, it's called END. Since you can't subclass an enum with members, it is defined in each subclass individually (so _Event has no END member, but AlphaEvent.END and BetaEvent.END exist).
I have several functions which utilise subclasses of _Event. I have a couple which need to access properties of the END member, but are generic to all instances. So they contain a signature:
def generic_event_func(events: _Event):
...
events.END.action = <expr>
MyPy flags the last line of code above with "error: "_Event" has no attribute "END"
True enough, but the subclasses do. How do I annotate the function signature to remove this error?
I faced a kinda similar issue recently and ended refactored using ABC. Not sure if you have latitude to refactor much, but maybe it can help in some way:
from abc import ABC, abstractmethod
from enum import Enum
class Event(Enum):
a = EventA
b = EventB
c = EventC
class AbstractEvent(ABC):
#abstractmethod
def action(self):
pass
class EventA(AbstractEvent):
def action(self):
....
event_cls = Event["a"].value
event: Union[EventA, EventB, EventC] = event_cls()
event.action()
I'm not (yet) an optional static typing person, but something like the following might work:
from typing import Any
class _Event(IntEnum):
END: Any
End is now type hinted, but doesn't actually exist, so won't interfere with subclassing _Event.
Following this answer it seems that a class' metaclass may be changed after the class has been defined by using the following*:
class MyMetaClass(type):
# Metaclass magic...
class A(object):
pass
A = MyMetaClass(A.__name__, A.__bases__, dict(A.__dict__))
Defining a function
def metaclass_wrapper(cls):
return MyMetaClass(cls.__name__, cls.__bases__, dict(cls.__dict__))
allows me to apply a decorator to a class definition like so,
#metaclass_wrapper
class B(object):
pass
It seems that the metaclass magic is applied to B, however B has no __metaclass__ attribute. Is the above method a sensible way to apply metaclasses to class definitions, even though I am definiting and re-definiting a class, or would I be better off simply writing
class B(object):
__metaclass__ = MyMetaClass
pass
I presume there are some differences between the two methods.
*Note, the original answer in the linked question, MyMetaClass(A.__name__, A.__bases__, A.__dict__), returns a TypeError:
TypeError: type() argument 3 must be a dict, not dict_proxy
It seems that the __dict__ attribute of A (the class definition) has a type dict_proxy, whereas the type of the __dict__ attribute of an instance of A has a type dict. Why is this? Is this a Python 2.x vs. 3.x difference?
Admittedly, I am a bit late to the party. However, I fell this was worth adding.
This is completely doable. That being said, there are plenty of other ways to accomplish the same goal. However, the decoration solution, in particular, allows for delayed evaluation ( obj = dec(obj) ), which using __metaclass__ inside the class does not. In typical decorator style, my solution is below.
There is a tricky thing that you may run into if you just construct the class without changing the dictionary or copying its attributes. Any attributes that the class had previously (before decorating) will appear to be missing. So, it is absolutely essential to copy these over and then tweak them as I have in my solution.
Personally, I like to be able to keep track of how an object was wrapped. So, I added the __wrapped__ attribute, which is not strictly necessary. It also makes it more like functools.wraps in Python 3 for classes. However, it can be helpful with introspection. Also, __metaclass__ is added to act more like the normal metaclass use case.
def metaclass(meta):
def metaclass_wrapper(cls):
__name = str(cls.__name__)
__bases = tuple(cls.__bases__)
__dict = dict(cls.__dict__)
for each_slot in __dict.get("__slots__", tuple()):
__dict.pop(each_slot, None)
__dict["__metaclass__"] = meta
__dict["__wrapped__"] = cls
return(meta(__name, __bases, __dict))
return(metaclass_wrapper)
For a trivial example, take the following.
class MetaStaticVariablePassed(type):
def __new__(meta, name, bases, dct):
dct["passed"] = True
return(super(MetaStaticVariablePassed, meta).__new__(meta, name, bases, dct))
#metaclass(MetaStaticVariablePassed)
class Test(object):
pass
This yields the nice result...
|1> Test.passed
|.> True
Using the decorator in the less usual, but identical way...
class Test(object):
pass
Test = metaclass_wrapper(Test)
...yields, as expected, the same nice result.
|1> Test.passed
|.> True
The class has no __metaclass__ attribute set... because you never set it!
Which metaclass to use is normally determined by a name __metaclass__ set in a class block. The __metaclass__ attribute isn't set by the metaclass. So if you invoke a metaclass directly rather than setting __metaclass__ and letting Python figure it out, then no __metaclass__ attribute is set.
In fact, normal classes are all instances of the metaclass type, so if the metaclass always set the __metaclass__ attribute on its instances then every class would have a __metaclass__ attribute (most of them set to type).
I would not use your decorator approach. It obscures the fact that a metaclass is involved (and which one), is still one line of boilerplate, and it's just messy to create a class from the 3 defining features of (name, bases, attributes) only to pull those 3 bits back out from the resulting class, throw the class away, and make a new class from those same 3 bits!
When you do this in Python 2.x:
class A(object):
__metaclass__ = MyMeta
def __init__(self):
pass
You'd get roughly the same result if you'd written this:
attrs = {}
attrs['__metaclass__'] = MyMeta
def __init__(self):
pass
attrs['__init__'] = __init__
A = attrs.get('__metaclass__', type)('A', (object,), attrs)
In reality calculating the metaclass is more complicated, as there actually has to be a search through all the bases to determine whether there's a metaclass conflict, and if one of the bases doesn't have type as its metaclass and attrs doesn't contain __metaclass__ then the default metaclass is the ancestor's metaclass rather than type. This is one situation where I expect your decorator "solution" will differ from using __metaclass__ directly. I'm not sure exactly what would happen if you used your decorator in a situation where using __metaclass__ would give you a metaclass conflict error, but I wouldn't expect it to be pleasant.
Also, if there are any other metaclasses involved, your method would result in them running first (possibly modifying what the name, bases, and attributes are!) and then pulling those out of the class and using it to create a new class. This could potentially be quite different than what you'd get using __metaclass__.
As for the __dict__ not giving you a real dictionary, that's just an implementation detail; I would guess for performance reasons. I doubt there is any spec that says the __dict__ of a (non-class) instance has to be the same type as the __dict__ of a class (which is also an instance btw; just an instance of a metaclass). The __dict__ attribute of a class is a "dictproxy", which allows you to look up attribute keys as if it were a dict but still isn't a dict. type is picky about the type of its third argument; it wants a real dict, not just a "dict-like" object (shame on it for spoiling duck-typing). It's not a 2.x vs 3.x thing; Python 3 behaves the same way, although it gives you a nicer string representation of the dictproxy. Python 2.4 (which is the oldest 2.x I have readily available) also has dictproxy objects for class __dict__ objects.
My summary of your question: "I tried a new tricky way to do a thing, and it didn't quite work. Should I use the simple way instead?"
Yes, you should do it the simple way. You haven't said why you're interested in inventing a new way to do it.
I read What is a metaclass in Python?
and I tried to replicate the upper metaclass from the example and found that this doesn't work in all cases:
def upper(cls_name, cls_parents, cls_attr):
""" Make all class attributes uppper case """
attrs = ((name, value) for name, value in cls_attr.items()
if not name.startswith('__'))
upper_atts = dict((name.upper(), value) for name, value in attrs)
return type(cls_name, cls_parents, upper_atts)
__metaclass__ = upper #Module level
class Foo:
bar = 1
f = Foo()
print(f.BAR) #works in python2.6
The above fails (with an attribute error) in python3 which I think is natural because all classes in python3 already have object as their parent and metaclass resolution goes into the object class.
The question:
How do I make a module level metaclass in python3?
The module level metaclass isn't really "module level", it has to do with how class initialization worked. The class creation would look for the variable "__metaclass__" when creating the class, and if it wasn't in the local environment it would look in the global. Hence, if you had a "module level" __metaclass__ that would be used for every class afterwards, unless they had explicit metaclasses.
In Python 3, you instead specify the metaclass with a metaclass= in the class definition. Hence there is no module level metaclasses.
So what do you do? Easy: You specify it explicitly for each class.
It's really not much extra work, and you can even do it with a nice regexp search and replace if you really have hundreds of classes and don't want to do it manually.
If you want to change all the attributes to upper case, you should probably use the __init__ method to do so, than use a metaclass.
Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why).
-- Python Guru Tim Peters
If you need something deeper, you should also evaluate using Class Decorators.
Using MetaClasses and understanding how the classes are created is so unnecessary as long as you want to do something that you can do using class decorators or initialization.
That said, if you really want to use a Metaclass tho' pass that as a keyword argument to the class.
class Foo(object, metaclass=UpperCaseMetaClass)
where UpperCaseMetaClass is a class that extends type and not a method.
class UpperCaseMetaClass(type):
def __new__():
#Do your Magic here.