We Keep Coding

Python: Dynamically generating attributes from a list

I want to be able to dynamically generate attributes of a class from a list or dictionary. The idea is that I can define a list of attributes, and then be able to access those attributes using my_class.my_attribute
For example:
class Campaign(metaclass=MetaCampaign):
_LABELS = ['campaign_type', 'match_type', 'audience_type'] # <-- my list of attributes
for label in _LABELS:
setattr(cls, label, LabelDescriptor(label))
def __init__(self, campaign_protobuf, labels)
self._proto = campaign_protobuf
self._init_labels(labels_dict)
def _init_labels(self, labels_dict):
# magic...
This obviously won't work because cls doesn't exist, but I'd like:
my_campaign = Campaign(campaign, label_dict)
print(my_campaign.campaign_type)
to return the value campaign_type for the campaign. This is obviously a little complicated, as campaign_type is actually a Descriptor and does a bit of work to retrieve a value from a base Label object.
The Descriptor:
class DescriptorProperty(object):
def __init__(self):
self.data = WeakKeyDictionary()
def __set__(self, instance, value):
self.data[instance] = value
class LabelTypeDescriptor(DescriptorProperty):
"""A descriptor that returns the relevant metadata from the label"""
def __init__(self, pattern):
super(MetaTypeLabel, self).__init__()
self.cached_data = WeakKeyDictionary()
# Regex pattern to look in the label:
# r'label_type:ThingToReturn'
self.pattern = f"{pattern}:(.*)"
def __get__(self, instance, owner, refresh=False):
# In order to balance computational speed with memory usage, we cache label values
# when they are first accessed.
if self.cached_data.get(instance, None) is None or refresh:
ctype = re.search(self.pattern, self.data[instance].name) # <-- does a regex search on the label name (e.g. campaign_type:Primary)
if ctype is None:
ctype = False
else:
ctype = ctype.group(1)
self.cached_data[instance] = ctype
return self.cached_data[instance]
This enables me to easily access the value of a label, and if the label is of a type that I care about, it will return the relevant value, otherwise it will return False.
The Label Object:
class Label(Proto):
_FIELDS = ['id', 'name']
_PROTO_NAME = 'label'
# We define what labels can pull metadata directly through a property
campaign_type = LabelTypeDescriptor('campaign_type')
match_type = LabelTypeDescriptor('match_type')
audience_type = LabelTypeDescriptor('audience_type')
def __init__(self, proto, **kwargs):
self._proto = proto
self._set_default_property_values(self) # <-- the 'self' is intentional here, in the campaign object a label would be passed instead.
def _set_default_property_values(self, proto_wrapper):
props = [key for (key, obj) in self.__class__.__dict__.items() if isinstance(obj, DescriptorProperty)]
for prop in props:
setattr(self, prop, proto_wrapper)
So if I have a protobuf label object stored in my Label (which is basically just a wrapper) which looks like this:
resource_name: "customers/12345/labels/67890"
id {
value: 67890
}
name {
value: "campaign_type:Primary"
}
Then my_label.campaign_type would return Primary, and similarly my_label.match_type would return False
The reason being is that I'm creating a number of classes that are all labelled in the same way, and may have a lot of labels. Currently this all works as described, but I'd like to be able to define the attributes more dynamically as they all basically follow the same type of pattern. So instead of :
campaign_type = LabelTypeDescriptor('campaign_type')
match_type = LabelTypeDescriptor('match_type')
audience_type = LabelTypeDescriptor('audience_type')
... # (many more labels)
I simply have: _LABELS = ['campaign_type', 'match_type', 'audience_type', ... many more labels] and then have some loop that creates the attributes.
In turn I can cascade a similar approach through to my other classes, so that while a Campaign object may hold a Label object, I can access the value of the label simply by my_campaign.campaign_type. If the campaign does not have a label of the appropriate type, it will simply return False.

While cls does not exist when the class body is run, you can set the attributes by simply setting then in the dictionary returned by locals() inside the class body:
class Campaign(metaclass=MetaCampaign):
_LABELS = ['campaign_type', 'match_type', 'audience_type'] # <-- my list of attributes
for label in _LABELS:
locals()[label] = label, LabelDescriptor(label)
del label # so you don't get a spurious "label" attribute in your class
Other than that you can use a metaclass, yes, but also a __init_suclass__ on a base class. Less metaclasses mean less "moving parts" that can behave in strange ways in your system.
So, say your Proto class is the base for all others that need this feature:
class Proto:
def __init_subclass__(cls, **kwd):
super().__init_subclass__(**kwd)
for label in cls._LABELS:
setattr(cls, label, LabelDescriptor(label))
...
I had taken a look at your Descriptors and code there - if they ar already working, I'd say they are all right.
I can comment that it is more usual to store descriptor-related data in the instance's __dict__ itself, instead of creating the data and cached_datain the descriptor itself - so one don't need to care about weakrefs - but both approaches work (just this week, I had implemented a descriptor in this way, even though I usually go for the instance's __dict__)

You could define a classmethod that will initialize these attributes, and call this method after the class declaration:
class Campaign(metaclass=MetaCampaign):
_LABELS = ['campaign_type', 'match_type', 'audience_type'] # <-- my list of attributes
#classmethod
def _init_class(cls):
for label in cls._LABELS:
setattr(cls, label, LabelDescriptor(label))
# After the class has been declared, initialize the attributes
Campaign._init_class()

Python : How is sub class variable accessible by superclass functions when called by an object of sub-class

initialTreeCom=['ADD_CHILD None King_Shan Male','ADD_CHILD None Queen_Anga Female',
'AM King_Shan Queen_Anga','ADD_CHILD Queen_Anga Chit Male']
famTree=familyTree.FamiltyTree(initialTreeCom)
result=famTree.excommand('GET_RELATIONSHIP Chit Father')
print(result)
Q1. when i create an object of class FamilyTree which inherits from class Relationships . how is a function called excommand of class FamilyTree able to call a function of parent class as
getattr(self, argument[2].replace('-','_'))(argument[1])
but throws an error when i get the function reference by gettr function by using super (since i want to access function of parent class) instead of self?
Q2. How is the called function Father of super class Relationships able to access variable familyMembers of subclass FamilyTree using self.familyMembers ?
class Person():
def __init__(self,mother,name,gender):
self.name=name
self.gender=gender
self.mother=mother
self.marriedTo=None
def add_partner(self,marriedTo):
"""
Assigns passed marriedTo value to marriedTo attribute to calling objects
"""
if marriedTo != self.marriedTo:
self.marriedTo=marriedTo
class FamiltyTree(Relationships):
familyMembers=OrderedDict()
def __init__(self,initialTree):
for i,line in enumerate(initialTree):
# print('executing command',i,' : ',line)
self.excommand(line)
def excommand(self,command):
"""
Executes the passed command
Commands : ADD_CHILD, GET_RELATIONSHIP
"""
argument=command.split()
if argument[0]=='ADD_CHILD':
# print('adding child')
p=Person(argument[1], argument[2], argument[3])
return self.add_Person(p)
if argument[0]=='AM':
# print('adding married to')
return self.add_MarriedTo(argument[1], argument[2])
if argument[0]=='GET_RELATIONSHIP':
if argument[1] not in self.familyMembers:
return 'PERSON_NOT_FOUND'
# print('getting relationship',argument[2], 'for : ',argument[1])
**res= getattr(self,argument[2].replace('-','_'))(argument[1])**
if res:
return ' '.join(res)
return None
def add_Person(self,person):
"""
Adds passed Person to Family members, of calling object of FamiltyTree
"""
if person.mother!='None' and person.mother not in self.familyMembers:
return 'PERSON_NOT_FOUND'
if isinstance(person,Person) and person.name not in self.familyMembers and (person.mother=='None' or self.familyMembers[person.mother].gender=='Female'):
self.familyMembers[person.name]=person
return 'CHILD_ADDITION_SUCCEEDED'
else:
return 'CHILD_ADDITION_FAILED'
def add_MarriedTo(self,person1,person2):
"""
sets marriedTo Relationship among both passed person objects to each other.
"""
if person1 in self.familyMembers and person2 in self.familyMembers:
for key , value in self.familyMembers.items():
if key==person1:
value.add_partner(person2)
if key==person2:
value.add_partner(person1)
return True
else:
return False
class Relationships():
def Father(self,name):
"""
Finds Father of passed person name
"""
if name in self.familyMembers:
for key , value in self.familyMembers.items():
if self.familyMembers[self.familyMembers[name].mother].marriedTo==key:
return key
return None

A1. getattr() is searching for a member whose name corresponds to the string you pass it. In your case, the name is Father. This is a convenient way of mapping strings the user inputs (argument[2] in your case) so that you can see if an implementation of that relationship exists in the Relationships class.
This is clever. Now you can just add implementations for other relationships to the Relationship class without having to modify the implementation of GET_RELATIONSHIP.
Alternatively, you could have implemented Father as:
if argument[2] == 'Father':
res = self.Father(argument[1])
But then you'd have do that for every type of relationship you want to implement (eg, Brother, Cousin).
This Relationships class makes it easy for you to just implement individual relationships by walking the family tree. The Father() implementation is a good example of how to do that.
A2. Because when Python encounters self.x, it first searches the current class for a data attribute x, and if it finds none, it then proceeds to search for a class attribute x, and uses that for subsequent operations on self.x. In your case, familyMembers is class attribute, and it happens to be located in the superclass Relationships. super only looks at data members.
There is an excellent discussion of class and instance attributes here

Extract (not known beforehand) attributes from objects in a list

I have a class whose attributes are not known beforehand:
class Event():
def __init__(self, **kwargs):
for key, value in kwargs.items():
setattr(self, key, value)
and another one which is basically a list of objects Event:
class Collection(list):
def __init__(self):
self.members = []
def add(self,new):
try:
self.members.extend(new)
except TypeError:
self.members.append(new)
Let's say now that I define 3 objects Event:
a = Event(name="a",value=1)
b = Event(name="b",value=2)
c = Event(name="c",other=True)
And I create a Collection from them:
col = Collection()
col.add([a,b,c])
What I want is to be able to print out all the values of the objects in the list for a given attribute (if the attribute does not exist for an object, it should return None or any other pre-defined value). For example:
print col.name #should return ["a","b","c"]
print col.value #should return [1,2,None]
I have read the following answer: Extract list of attributes from list of objects in python
But that doesn't work here since the name of my attribute is not known by advance, and some might not even be defined. How should I define my class Collection(), or maybe even re-think everything to achieve my goal ?

This is a variation of "I want to create dynamic variable names". The solution here is the same: use a dictionary.
class Event(object):
def __init__(self, **kwargs):
self.attributes = dict(kwargs)
Your Collection class will need a custom __getattr__ method, so that it can look up values in its Event list instead.
class Collection(object):
# assume self.events is a list of Event objects
def __getattr__(self, name):
return [event.attributes.get(name) for event in self.events]
You could stick with your current implementation of Event, and have Collection look at event.__dict__ instead of event.attributes. I don't recall, though, if __dict__ might contain anything else besides the attributes you explicitly set. I'd err on the side of caution.

You can just override the __getattr__ method of the Collection class, which is called when an attribute is accessed. In order to access to unknown set of attributes you can use event.__dict__. So, a possible solution is like this:
def __getattr__(self, name):
return [m.__dict__.get(name) for m in self.members]

Python/Django - Dynamic property overloading for Django models

I have two related models:
class FirstModel(models.Model):
base_value = models.FloatField()
class SecondModel(models.Model):
parent = models.ForeignKey(FirstModel)
#property
def parent_value(self):
return self.parent.base_value
#property
def calculate(self):
return self.parent_value + 1
In general, SecondModel.calculate is mostly used in the context of its related FirstModel. However, I sometimes want to be able to call calculate with a temporary value as its parent_value. Something like this:
foo = SecondModel()
# would look in the database for the related FirstModel and add 1 to its base_value
foo.calculate
foo.parent_value = 10
foo.calculate # should return 11
Obviously you can't do this because the parent_value is a read-only property. I also have many different models similar to SecondModel that needs to have this kind of capability.
I've thought about and tried several things, but none have quite seemed to work:
1) Writing a Django proxy model - possible, but the number of objects is rather high, so I'd be writing a lot of similar code. Also, there appears to be a bug related to overriding properties: https://code.djangoproject.com/ticket/16176. But it'd look like this:
class ModelProxy(SecondModel):
class Meta:
proxy = True
def __init__(self, temp_value):
self.parent_value = temp_value
2) Overloading the parent_value property on the instance - like this:
foo = SecondModel()
setattr(foo, 'parent_value', 10)
but you can't do this because properties are members of the class, not the instance. And I only want the temporary value to be set for the instance
3) Metaclass or class generator? - Seems overly complicated. Also, I am uncertain what would happen if I used a metaclass to dynamically generate classes that are children of models.Model. Would I run into problems with the db tables not being in sync?
4) Rewriting the properties with proper getters and setters? - maybe the solution is to rewrite SecondModel so that the property can be set?
Any suggestions?

I believe a mixin would achieve what you want to do, and provide a simple and reusable way of supporting temporary values in your calculations. By mixing the below example into each model you want this behaviour on you can then:
Set a temporary parent value on each model
When calculate is called, it will check whether there is a property parent_value available, and if not it will use the temporary parent value in the calculation.
The code below should achieve what you are looking for - apologies I haven't been able to test it yet but it should be about right - please let me know if any problems that need editing.
class CalculateMixin(object):
#property
def temp_parent_value(self):
return self._temp_parent_value
#temp_parent_value.setter
def temp_parent_value(self, value):
self._temp_parent_value = value
#property
def calculate(self):
parent_value = self.parent_value if self.parent_value else self.temp_parent_value
return parent_value + 1
class SecondModel(models.Model, CalculateMixin):
parent = models.ForeignKey(FirstModel)
self.temp_parent_value = 'Whatever value you desire'
#property
def parent_value(self):
return self.parent.base_value

You can use the property setter:
class SecondModel(models.Model):
_base_value = None
parent = models.ForeignKey(FirstModel)
#property
def parent_value(self):
if self._base_value is None:
return self.parent.base_value
else:
return self._base_value
#parent_value.setter
def parent_value(self, value):
self._base_value = value
#property
def calculate(self):
return self.parent_value + 1

I think you can do what you need to using the mixin PropertyOverrideMixin shown below which, if some property value isn't available, then it will look for the same property prefixed with temp_. This will allow you to provide temporary values that can be used when the real property values can't be looked up.
Below is the mixin, some example models and a unit test to show how this can work. Hopefully this can be adapted for your problem! Finally it is worth mentioning that the properties here can be interchanged with normal object attributes and it should still all work.
from unittest import TestCase
class PropertyOverrideMixin(object):
def __getattribute__(self, name):
"""
Override that, if an attribute isn't found on the object, then it instead
looks for the same attribute prefixed with 'temp_' and tries to return
that value.
"""
try:
return object.__getattribute__(self, name)
except AttributeError:
temp_name = 'temp_{0}'.format(name)
return object.__getattribute__(self, temp_name)
class ParentModel(object):
attribute_1 = 'parent value 1'
class Model(PropertyOverrideMixin):
# Set our temporary property values
#property
def temp_attribute_1(self):
return 'temporary value 1'
#property
def temp_attribute_2(self):
return 'temporary value 2'
# Attribute 1 looks up value on its parent
#property
def attribute_1(self):
return self.parent.attribute_1
# Attribute 2 looks up a value on this object
#property
def attribute_2(self):
return self.some_other_attribute
class PropertyOverrideMixinTest(TestCase):
def test_attributes(self):
model = Model()
# Looking up attributes 1 and 2 returns the temp versions at first
self.assertEquals('temporary value 1', model.attribute_1)
self.assertEquals('temporary value 2', model.attribute_2)
# Now we set the parent, and lookup of attribute 1 works on the parent
model.parent = ParentModel()
self.assertEquals('parent value 1', model.attribute_1)
# now we set attribute_2, so this gets returned and the temporary ignored
model.some_other_attribute = 'value 2'
self.assertEquals('value 2', model.attribute_2)

How to implement property() with dynamic name (in python)

I am programming a simulations for single neurons. Therefore I have to handle a lot of Parameters. Now the Idea is that I have two classes, one for a SingleParameter and a Collection of parameters. I use property() to access the parameter value easy and to make the code more readable. This works perfect for a sinlge parameter but I don't know how to implement it for the collection as I want to name the property in Collection after the SingleParameter. Here an example:
class SingleParameter(object):
def __init__(self, name, default_value=0, unit='not specified'):
self.name = name
self.default_value = default_value
self.unit = unit
self.set(default_value)
def get(self):
return self._v
def set(self, value):
self._v = value
v = property(fget=get, fset=set, doc='value of parameter')
par1 = SingleParameter(name='par1', default_value=10, unit='mV')
par2 = SingleParameter(name='par2', default_value=20, unit='mA')
# par1 and par2 I can access perfectly via 'p1.v = ...'
# or get its value with 'p1.v'
class Collection(object):
def __init__(self):
self.dict = {}
def __getitem__(self, name):
return self.dict[name] # get the whole object
# to get the value instead:
# return self.dict[name].v
def add(self, parameter):
self.dict[parameter.name] = parameter
# now comes the part that I don't know how to implement with property():
# It shoule be something like
# self.__dict__[parameter.name] = property(...) ?
col = Collection()
col.add(par1)
col.add(par2)
col['par1'] # gives the whole object
# Now here is what I would like to get:
# col.par1 -> should result like col['par1'].v
# col.par1 = 5 -> should result like col['par1'].v = 5
Other questions that I put to understand property():
Why do managed attributes just work for class attributes and not for instance attributes in python?
How can I assign a new class attribute via __dict__ in python?

Look at built-in functions getattr and setattr. You'll probably be a lot happier.

Using the same get/set functions for both classes forces you into an ugly hack with the argument list. Very sketchy, this is how I would do it:
In class SingleParameter, define get and set as usual:
def get(self):
return self._s
def set(self, value):
self._s = value
In class Collection, you cannot know the information until you create the property, so you define the metaset/metaget function and particularize them only later with a lambda function:
def metaget(self, par):
return par.s
def metaset(self, value, par):
par.s = value
def add(self, par):
self[par.name] = par
setattr(Collection, par.name,
property(
fget=lambda x : Collection.metaget(x, par),
fset=lambda x, y : Collection.metaset(x,y, par))

Properties are meant to dynamically evaluate attributes or to make them read-only. What you need is customizing attribute access. __getattr__ and __setattr__ do that really fine, and there's also __getattribute__ if __getattr__ is not enough.
See Python docs on customizing attribute access for details.

Have you looked at the traits package? It seems that you are reinventing the wheel here with your parameter classes. Traits also have additional features that might be useful for your type of application (incidently I know a person that happily uses traits in neural simulations).

Now I implemented a solution with set-/getattr:
class Collection(object):
...
def __setattr__(self, name, value):
if 'dict' in self.__dict__:
if name in self.dict:
self[name].v = value
else:
self.__dict__[name] = value
def __getattr__(self, name):
return self[name].v
There is one thing I quite don't like that much: The attributes are not in the __dict__. And if I have them there as well I would have a copy of the value - which can be dangerous...

Finally I succeded to implement the classes with property(). Thanks a lot for the advice. It took me quite a bit to work it out - but I can promise you that this exercise helps you to understand better pythons OOP.
I implemented it also with __getattr__ and __setattr__ but still don't know the advantages and disadvantages to the property-solution. But this seems to be worth another question. The property-solutions seems to be quit clean.
So here is the code:
class SingleParameter(object):
def __init__(self, name, default_value=0, unit='not specified'):
self.name = name
self.default_value = default_value
self.unit = unit
self.set(default_value)
def get(*args):
self = args[0]
print "get(): "
print args
return self._v
def set(*args):
print "set(): "
print args
self = args[0]
value = args[-1]
self._v = value
v = property(fget=get, fset=set, doc='value of parameter')
class Collection(dict):
# inheriting from dict saves the methods: __getitem__ and __init__
def add(self, par):
self[par.name] = par
# Now here comes the tricky part.
# (Note: this property call the get() and set() methods with one
# more argument than the property of SingleParameter)
setattr(Collection, par.name,
property(fget=par.get, fset=par.set))
# Applying the classes:
par1 = SingleParameter(name='par1', default_value=10, unit='mV')
par2 = SingleParameter(name='par2', default_value=20, unit='mA')
col = Collection()
col.add(par1)
col.add(par2)
# Setting parameter values:
par1.v = 13
col.par1 = 14
# Getting parameter values:
par1.v
col.par1
# checking identity:
par1.v is col.par1
# to access the whole object:
col['par1']
As I am new I am not sure how to move on:
how to treat follow up questions (like this itself):
get() is seems to be called twice - why?
oop-design: property vs. "__getattr__ & __setattr__" - when should I use what?
is it rude to check the own answer to the own question as accepted?
is it recommended to rename the title in order to put correlated questions or questions elaborated with the same example into the same context?
Other questions that I put to understand property():
Why do managed attributes just work for class attributes and not for instance attributes in python?
How can I assign a new class attribute via __dict__ in python?

I have a class that does something similar, but I did the following in the collection object:
setattr(self, par.name, par.v)