I'm trying to replicate mongoengine functionality that lets you define field objects that can be used like normal python objects in the code.
My idea is to create a FieldHolder class that contains the value and (de)serialization logic, and a Document object with overridden __setattr__ and __getattribute__ methods.
In my code draft, if I set x.h to some value, this value gets correctly assigned to x.h._value. When I get x.h, I correctly get x.h._value.
However, I would also like get h as a FieldHolder object and not as its value. I have tried using object.__getattribute__ (inside serialize method), but I'm still getting h._value (object.__getattribute__(self, 'h') returns abc). What am I doing wrong? Thanks
class FieldHolder:
_value = None
# Some serialization and deserialization methods
class Document(object):
h = FieldHolder()
def __setattr__(self, key, value):
attr = getattr(self, key, None)
if attr is not None and isinstance(attr, FieldHolder):
attr._value = value
else:
super().__setattr__(key, value)
def __getattribute__(self, key):
val = super().__getattribute__(key)
if isinstance(val, FieldHolder):
return val._value
else:
return val
def serialize(self):
res = {}
for name, value in vars(self).items():
obj = object.__getattribute__(self, name) # not working as expected
if isinstance(obj, FieldHolder):
res[name] = value
return res
x = Document()
x.h = "abc" # h._value is now "abc"
print(x.h) # prints "abc"
s = x.serialize() # should return {'h': 'abc'} but returns {}
print(s)
So I have this example code, where Foo is very dynamic data structure.
def fetch():
# simulate api request
return {'param1':1, 'param2':{'description':'param2', 'value':2}}
class Foo(object):
def __init__(self):
self._rawdata = fetch()
#set attributes accordingly to the downloaded data
for key, val in self._rawdata.items():
setattr(self, key, val)
def __str__(self):
# just an example
out = ''
for key, val in self._rawdata.items():
out += key + ': ' + str(val) + '\n'
return out
A user might want to try do this:
f = Foo()
print(f.param3)
The user doesn't know whether the 'param3' exists or not (API might not have any data avaliable, in which case it won't provide the key at all)
Naturally, this will result in AttributeError being raised.
print(f.param3)
AttributeError: 'Foo' object has no attribute 'param3'
My question is this: Is there some metaprogramming magic way to wrap the Foo() class into something that will make 'f.nonexistent_attribute' return 'None' instead of traceback? I would really like to avoid hardcoding expected properties (what if API changes?).
Implement the __getattr__ method; it'll be called for all nonexistent attributes:
def __getattr__(self, name):
# all non-existing attributes produce None
return None
From the documentation:
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Demo:
>>> def fetch():
... # simulate api request
... return {'param1':1, 'param2':{'description':'param2', 'value':2}}
...
>>> class Foo(object):
... def __init__(self):
... self._rawdata = fetch()
... #set attributes accordingly to the downloaded data
... for key, val in self._rawdata.items():
... setattr(self, key, val)
... def __getattr__(self, name):
... # all non-existing attributes produce None
... return None
...
>>> print(f.param1)
1
>>> f = Foo()
>>> print(f.param3)
None
I have this (Py2.7.2):
class MyClass(object):
def __init__(self, dict_values):
self.values = dict_values
self.changed_values = {} #this should track changes done to the values{}
....
I can use it like this:
var = MyClass()
var.values['age'] = 21
var.changed_values['age'] = 21
But I want to use it like this:
var.age = 21
print var.changed_values #prints {'age':21}
I suspect I can use properties to do that, but how?
UPDATE:
I don't know the dict contents at the design time. It will be known at run-time only. And it will likely to be not empty
You can create a class that inherits from a dict and override the needed functions
class D(dict):
def __init__(self):
self.changed_values = {}
self.__initialized = True
def __setitem__(self, key, value):
self.changed_values[key] = value
super(D, self).__setitem__(key, value)
def __getattr__(self, item):
"""Maps values to attributes.
Only called if there *isn't* an attribute with this name
"""
try:
return self.__getitem__(item)
except KeyError:
raise AttributeError(item)
def __setattr__(self, item, value):
"""Maps attributes to values.
Only if we are initialised
"""
if not self.__dict__.has_key('_D__initialized'): # this test allows attributes to be set in the __init__ method
return dict.__setattr__(self, item, value)
elif self.__dict__.has_key(item): # any normal attributes are handled normally
dict.__setattr__(self, item, value)
else:
self.__setitem__(item, value)
a = D()
a['hi'] = 'hello'
print a.hi
print a.changed_values
a.hi = 'wow'
print a.hi
print a.changed_values
a.test = 'test1'
print a.test
print a.changed_values
output
>>hello
>>{'hi': 'hello'}
>>wow
>>{'hi': 'wow'}
>>test1
>>{'hi': 'wow', 'test': 'test1'}
Properties (descriptors, really) will only help if the set of attributes to monitor is bounded. Simply file the new value away in the __set__() method of the descriptor.
If the set of attributes is arbitrary or unbounded then you will need to overrive MyClass.__setattr__() instead.
You can use the property() built-in function.
This is preferred to overriding __getattr__ and __setattr__, as explained here.
class MyClass:
def __init__(self):
self.values = {}
self.changed_values = {}
def set_age( nr ):
self.values['age'] = nr
self.changed_values['age'] = nr
def get_age():
return self.values['age']
age = property(get_age,set_age)
Sorry for the confusing title.
I would like to do the following: (Similar to defstruct in Lisp)
def mkstruct(structname, field_dict):
# create a function called "structname" and get/set functions
# called "structname_get(s, field_name)" and "structname_set(s, field_name, value)"
# Create a struct "lstnode"
mkstruct("lstnode", {ndkey : 0, nxt: None})
# Make a new struct
node = lstnode()
node_set(node, "ndkey", 5)
v = node_get(node, "ndkey") # v should be 5
This can be done in C with a macro define. The reason I am not using a class is because The "struct" I am creating will be "tied" to a database (just a text file in some format in this case). And I don't want to take up any memory associated with an object - I will represent the struct as a number (an object ID if you will)
This should be a step in the direction of what you want:
def mkstruct(name, attrs):
def init(self):
self.id = # not sure how you want to get the id
def getattr(self, attr):
if attr not in attrs:
raise AttributeError(attr)
# put your database lookup statement here
def setattr(self, attr, value):
if attr not in attrs:
raise AttributeError(attr)
# put your database update statement here
return type(
name,
(object,),
__init__=init,
__getattr__=getattr,
__setattr__=setattr)
lstnode = mkstruct("lstnode", ("ndkey", "nxt"))
Looks to me that what you're looking for is already provided by the type built-in:
def mkstruct(structname, field_dict):
return type(structname, (object,), field_dict)
lstnode = mkstruct("lstnode", {'ndkey' : 0, 'nxt': None})
node = lstnode()
node.ndkey = 5
v = node.ndkey
If you need just the keys in field_dict to be members of the structure, you can add '__slots__' to field_dict.
Note: This doesn't implement any setter or getter, but as pointed out already by the comments, this is not really needed when using classes.
It looks like that this isn't easy to do in python - after some research. The only way to add a inner function to the global namespace is to modify the globals() dict, which is rather awkward.
>>> def mkfunc(funcname):
... def func():
... print "my name is %s" % funcname
... func.__name__ = funcname
... return func
...
>>> mkfunc("abc")
<function abc at 0xb773ae64>
>>> globals()["abc"] = mkfunc("abc")
>>> abc()
my name is abc
As for my own problem, I am content to do the following:
def mkstruct(fields):
def maker(args):
# validate #args against #fields
oid = db_insert_row(fields)
return oid
def getter(oid, fieldname):
rec = db_retrieve(oid)
return rec[fieldname]
def setter(oid, fieldname, value):
db_update(oid, fieldname, value)
return (maker, getter, setter,)
lstnode, lstnode_get, lstnode_set = mkstruct({nodekey: 0, nxt: None})
n = lstnode(nodekey=5)
I want to be able to create a class (in Python) that once initialized with __init__, does not accept new attributes, but accepts modifications of existing attributes. There's several hack-ish ways I can see to do this, for example having a __setattr__ method such as
def __setattr__(self, attribute, value):
if not attribute in self.__dict__:
print "Cannot set %s" % attribute
else:
self.__dict__[attribute] = value
and then editing __dict__ directly inside __init__, but I was wondering if there is a 'proper' way to do this?
I wouldn't use __dict__ directly, but you can add a function to explicitly "freeze" a instance:
class FrozenClass(object):
__isfrozen = False
def __setattr__(self, key, value):
if self.__isfrozen and not hasattr(self, key):
raise TypeError( "%r is a frozen class" % self )
object.__setattr__(self, key, value)
def _freeze(self):
self.__isfrozen = True
class Test(FrozenClass):
def __init__(self):
self.x = 42#
self.y = 2**3
self._freeze() # no new attributes after this point.
a,b = Test(), Test()
a.x = 10
b.z = 10 # fails
Slots is the way to go:
The pythonic way is to use slots instead of playing around with the __setter__. While it may solve the problem, it does not give any performance improvement. The attributes of objects are stored in a dictionary "__dict__", this is the reason, why you can dynamically add attributes to objects of classes that we have created so far. Using a dictionary for attribute storage is very convenient, but it can mean a waste of space for objects, which have only a small amount of instance variables.
Slots are a nice way to work around this space consumption problem. Instead of having a dynamic dict that allows adding attributes to objects dynamically, slots provide a static structure which prohibits additions after the creation of an instance.
When we design a class, we can use slots to prevent the dynamic creation of attributes. To define slots, you have to define a list with the name __slots__. The list has to contain all the attributes, you want to use. We demonstrate this in the following class, in which the slots list contains only the name for an attribute "val".
class S(object):
__slots__ = ['val']
def __init__(self, v):
self.val = v
x = S(42)
print(x.val)
x.new = "not possible"
=> It fails to create an attribute "new":
42
Traceback (most recent call last):
File "slots_ex.py", line 12, in <module>
x.new = "not possible"
AttributeError: 'S' object has no attribute 'new'
Notes:
Since Python 3.3 the advantage optimizing the space consumption is not as impressive any more. With Python 3.3 Key-Sharing Dictionaries are used for the storage of objects. The attributes of the instances are capable of sharing part of their internal storage between each other, i.e. the part which stores the keys and their corresponding hashes. This helps to reduce the memory consumption of programs, which create many instances of non-builtin types. But still is the way to go to avoid dynamically created attributes.
Using slots come also with it's own cost. It will break serialization (e.g. pickle). It will also break multiple inheritance. A class can't inherit from more than one class that either defines slots or has an instance layout defined in C code (like list, tuple or int).
If someone is interested in doing that with a decorator, here is a working solution:
from functools import wraps
def froze_it(cls):
cls.__frozen = False
def frozensetattr(self, key, value):
if self.__frozen and not hasattr(self, key):
print("Class {} is frozen. Cannot set {} = {}"
.format(cls.__name__, key, value))
else:
object.__setattr__(self, key, value)
def init_decorator(func):
#wraps(func)
def wrapper(self, *args, **kwargs):
func(self, *args, **kwargs)
self.__frozen = True
return wrapper
cls.__setattr__ = frozensetattr
cls.__init__ = init_decorator(cls.__init__)
return cls
Pretty straightforward to use:
#froze_it
class Foo(object):
def __init__(self):
self.bar = 10
foo = Foo()
foo.bar = 42
foo.foobar = "no way"
Result:
>>> Class Foo is frozen. Cannot set foobar = no way
Actually, you don't want __setattr__, you want __slots__. Add __slots__ = ('foo', 'bar', 'baz') to the class body, and Python will make sure that there's only foo, bar and baz on any instance. But read the caveats the documentation lists!
The proper way is to override __setattr__. That's what it's there for.
I like very much the solution that uses a decorator, because it's easy to use it for many classes across a project, with minimum additions for each class. But it doesn't work well with inheritance.
So here is my version: It only overrides the __setattr__ function - if the attribute doesn't exist and the caller function is not __init__, it prints an error message.
import inspect
def froze_it(cls):
def frozensetattr(self, key, value):
if not hasattr(self, key) and inspect.stack()[1][3] != "__init__":
print("Class {} is frozen. Cannot set {} = {}"
.format(cls.__name__, key, value))
else:
self.__dict__[key] = value
cls.__setattr__ = frozensetattr
return cls
#froze_it
class A:
def __init__(self):
self._a = 0
a = A()
a._a = 1
a._b = 2 # error
What about this:
class A():
__allowed_attr=('_x', '_y')
def __init__(self,x=0,y=0):
self._x=x
self._y=y
def __setattr__(self,attribute,value):
if not attribute in self.__class__.__allowed_attr:
raise AttributeError
else:
super().__setattr__(attribute,value)
Here is approach i came up with that doesn't need a _frozen attribute or method to freeze() in init.
During init i just add all class attributes to the instance.
I like this because there is no _frozen, freeze(), and _frozen also does not show up in the vars(instance) output.
class MetaModel(type):
def __setattr__(self, name, value):
raise AttributeError("Model classes do not accept arbitrary attributes")
class Model(object):
__metaclass__ = MetaModel
# init will take all CLASS attributes, and add them as SELF/INSTANCE attributes
def __init__(self):
for k, v in self.__class__.__dict__.iteritems():
if not k.startswith("_"):
self.__setattr__(k, v)
# setattr, won't allow any attributes to be set on the SELF/INSTANCE that don't already exist
def __setattr__(self, name, value):
if not hasattr(self, name):
raise AttributeError("Model instances do not accept arbitrary attributes")
else:
object.__setattr__(self, name, value)
# Example using
class Dog(Model):
name = ''
kind = 'canine'
d, e = Dog(), Dog()
print vars(d)
print vars(e)
e.junk = 'stuff' # fails
I like the "Frozen" of Jochen Ritzel. The inconvenient is that the isfrozen variable then appears when printing a Class.__dict
I went around this problem this way by creating a list of authorized attributes (similar to slots):
class Frozen(object):
__List = []
def __setattr__(self, key, value):
setIsOK = False
for item in self.__List:
if key == item:
setIsOK = True
if setIsOK == True:
object.__setattr__(self, key, value)
else:
raise TypeError( "%r has no attributes %r" % (self, key) )
class Test(Frozen):
_Frozen__List = ["attr1","attr2"]
def __init__(self):
self.attr1 = 1
self.attr2 = 1
The FrozenClass by Jochen Ritzel is cool, but calling _frozen() when initialing a class every time is not so cool (and you need to take the risk of forgetting it). I added a __init_slots__ function:
class FrozenClass(object):
__isfrozen = False
def _freeze(self):
self.__isfrozen = True
def __init_slots__(self, slots):
for key in slots:
object.__setattr__(self, key, None)
self._freeze()
def __setattr__(self, key, value):
if self.__isfrozen and not hasattr(self, key):
raise TypeError( "%r is a frozen class" % self )
object.__setattr__(self, key, value)
class Test(FrozenClass):
def __init__(self):
self.__init_slots__(["x", "y"])
self.x = 42#
self.y = 2**3
a,b = Test(), Test()
a.x = 10
b.z = 10 # fails
None of the answers mention the performance impact of overriding __setattr__, which can be an issue when creating many small objects. (And __slots__ would be the performant solution but limits pickle/inheritance).
So I came up with this variant which installs our slower settatr after init:
class FrozenClass:
def freeze(self):
def frozen_setattr(self, key, value):
if not hasattr(self, key):
raise TypeError("Cannot set {}: {} is a frozen class".format(key, self))
object.__setattr__(self, key, value)
self.__setattr__ = frozen_setattr
class Foo(FrozenClass): ...
If you don't want to call freeze at the end of __init__, if inheritance is an issue, or if you don't want it in vars(), it can also be adapted: for example here is a decorator version based on the pystrict answer:
import functools
def strict(cls):
cls._x_setter = getattr(cls, "__setattr__", object.__setattr__)
cls._x_init = cls.__init__
#functools.wraps(cls.__init__)
def wrapper(self, *args, **kwargs):
cls._x_init(self, *args, **kwargs)
def frozen_setattr(self, key, value):
if not hasattr(self, key):
raise TypeError("Class %s is frozen. Cannot set '%s'." % (cls.__name__, key))
cls._x_setter(self, key, value)
cls.__setattr__ = frozen_setattr
cls.__init__ = wrapper
return cls
#strict
class Foo: ...
I wrote pystrict as a solution to this problem. It's too large to paste all of the code in stackoverflow.
pystrict is a pypi installable decorator that can be used with classes to freeze them. Many solutions here don't properly support inheritance.
If __slots__ doesn't work for you (because of inheritance issues), this is a good alternative.
There is an example to the README that shows why a decorator like this is needed even if you have mypy and pylint running on your project:
pip install pystrict
Then just use the #strict decorator:
from pystrict import strict
#strict
class Blah
def __init__(self):
self.attr = 1
#dataclass(slots=True) Nirvana (Python 3.10)
I'm in love with this #dataclass thing:
main.py
from dataclasses import dataclass
#dataclass(slots=True)
class C:
n: int
s: str
c = C(n=1, s='one')
assert c.n == 1
assert c.s == 'one'
c.n == 2
c.s == 'two'
c.asdf = 2
Outcome:
Traceback (most recent call last):
File "/home/ciro/main.py", line 15, in <module>
c.asdf = 2
AttributeError: 'C' object has no attribute 'asdf'
Note how #dataclass only requires use to define our attributes once with type annotations
n: int
s: str
and then, without any repetition we get for free:
def __init__(n, s):
self.n = n
self.s = s
__slots__ = ['n', 's']
Other free things not shown in this example:
__str__
__eq__: Compare object instances for equality by their attributes
__hash__ if you also use frozen=True: Object of custom type as dictionary key
Tested on Python 3.10.7, Ubuntu 22.10.