We Keep Coding

How would I 'listen' to/decorate a setter from an imported class

I'm not sure whether this is a great approach to be using, but I'm not hugely experienced with Python so please accept my apologies. I've tried to do some research on this but other related questions have been given alternative problem-specific solutions - none of which apply to my specific case.
I have a class that handles the training/querying of my specific machine learning model. This algorithm is running on a remote sensor, various values are fed into the object which returns None if the algorithm isn't trained. Once trained, it returns either True or False depending on the classification assigned to new inputs. Occasionally, the class updates a couple of threshold parameters and I need to know when this occurs.
I am using sockets to pass messages from the remote sensor to my main server. I didn't want to complicate the ML algorithm class by filling it up with message passing code and so instead I've been handling this in a Main class that imports the "algorithm" class. I want the Main class to be able to determine when the threshold parameters are updated and report this back to the server.
class MyAlgorithmClass:
def feed_value(self):
....
class Main:
def __init__(self):
self._algorithm_data = MyAlgorithmClass()
self._sensor_data_queue = Queue()
def process_data(self):
while True:
sensor_value = self._sensor_data_queue.get()
result, value = self._algorithm_data.feed_value(sensor_value)
if result is None:
# value represents % training complete
self._socket.emit('training', value)
elif result is True:
# value represents % chance that input is categoryA
self._socket.emit('categoryA', value)
elif result is False:
...
My initial idea was to add a property to MyAlgorithmClass with a setter. I could then decorate this in my Main class so that every time the setter is called, I can use the value... for example:
class MyAlgorithmClass:
#property
def param1(self):
return self._param1
#param1.setter
def param1(self, value):
self._param1 = value
class Main:
def __init__(self):
self._algorithm_data = MyAlgorithmClass()
self._sensor_data_queue = Queue()
def watch_param1(func):
def inner(*args):
self._socket.emit('param1_updated', *args)
func(*args)
My problem now, is how do I decorate the self._algorithm_data.param1 setter with watch_param1? If I simply set self._algorithm_data.param1 = watch_param1 then I will just end up setting self._algorithm_data._param1 equal to my function which isn't what I want to do.
I could use getter/setter methods instead of a property, but this isn't very pythonic and as multiple people are modifying this code, I don't want the methods to be replaced/changed for properties by somebody else later on.
What is the best approach here? This is a small example but I will have slightly more complex examples of this later on and I don't want something that will cause overcomplication of the algorithm class. Obviously, another option is the Observer pattern but I'm not sure how appropriate it is here where I only have a single variable to monitor in some cases.
I'm really struggling to get a good solution put together so any advice would be much appreciated.
Thanks in advance,
Tom

Use descriptors. They let you customize attribute lookup, storage, and deletion in Python.
A simplified toy version of your code with descriptors looks something like:
class WatchedParam:
def __init__(self, name):
self.name = name
def __get__(self, instance, insttype=None):
print(f"{self.name} : value accessed")
return getattr(instance, '_' + self.name)
def __set__(self, instance, new_val):
print(f"{self.name} : value set")
setattr(instance, '_' + self.name, new_val)
class MyAlgorithmClass:
param1 = WatchedParam("param1")
param2 = WatchedParam("param2")
def __init__(self, param1, param2, param3):
self.param1 = param1
self.param2 = param2
self.param3 = param3
class Main:
def __init__(self):
self._data = MyAlgorithmClass(10, 20, 50)
m = Main()
m._data.param1 # calls WatchedParam.__get__
m._data.param2 = 100 # calls WatchedParam.__set__
The WatchedParam class is a descriptor and can be used in MyAlgorithmClass to specify the parameters that need to be monitored.

The solution I went for is as follows, using a 'Proxy' subclass which overrides the properties. Eventually, once I have a better understanding of the watched parameters, I won't need to watch them anymore. At this point I will be able to swap out the Proxy for the base class and continue using the code as normal.
class MyAlgorithmClassProxy(MyAlgorithmClass):
#property
def watch_param1(self):
return MyAlgorithmClass.watch_param1.fget(self)
#watch_param1.setter
def watch_param1(self, value):
self._socket.emit('param1_updated', *args)
MyAlgorithmClass.watch_param1.fset(self, value)

Python - how to add property accessors programmatically

class A:
def __init__(self, *args, **kwargs):
for item in ["itemA", "itemB"]:
setattr(self, item, property(lambda : self.__get_method(item)))
def __get_method(self, item):
# do some stuff and return result
# this is pretty complex method which requires db lookups etc.
return result
I am trying to come up with above example to create class properties during init. Items list will get bigger in the future and don't want to add #property every time new entry is added.
However can't get the result from property but object location.
a = A()
a.itemA # returns <property at 0x113a41590>
Initially it was like this and realized this could be better.
class A:
#property
def itemA(self):
return self.__get_method("itemA")
#property
def itemX(self):
...
# and so on
How could I add new properties just by adding new entries to the items list and the class itself will create accessor for it?

Additional to #juanpa.arrivillaga comment.
You also can implement __getattr__ method
For example:
class A:
def __getattr__(self, name):
#make everybody happy

Some doubts about #property in python 3

In order not to extend myself too much I will give a basic and hypothetical example of what I am trying to do.
Suppose the following class:
class foo():
def __init__(self):
self.keywords = []
## this method returns the entire list
def get_keywords(self):
return self.keywords
def set_keywords(self, value):
self.keywords.append(value)
But I want to code this in a pythonic way using the #property decorator.
My (wrong) attempt to do this:
class foo:
def __init__(self):
self.key = []
#property
def key(self):
return self.__key
#key.setter
def key(self, value):
self.__key.append(value)
So, whats is wrong in my attempt ?
ps: English is not my native language and I hope my doubt is understandable.

In your original code, self.set_keywords only appends to an existing list; it does not let you initialize the value of keywords to an arbitrary list. This restriction is preserved in your property-based code, which means you cannot assign directly to self.key; you have to initialize the underlying list in __init__ directly.
class foo:
def __init__(self):
# self.key = [] is equivalent to `self.__key.append([])`, but
# self.__key doesn't exist yet. (And would be wrong even if it did.)
self.__key = []
#property
def key(self):
return self.__key
#key.setter
def key(self, value):
self.__key.append(value)
However, this means an assignment like self.key = 3 doesn't actually perform what most people would expect of an assignment. It doesn't overwrite the old value, it adds to it instead. Use the setter to provide a fixed list, but a different method to add to an existing one.
class foo:
def __init__(self):
self.__keys = []
#property
def keys(self):
return self.__keys
#keys.setter
def keys(self, values):
self.__keys = values
def add_key(self, value):
self.__key.append(value)
And finally, it's not necessarily more Pythonic to use a property if you don't actually do any sort of extra work or validation in the getter or setter. If all you are doing is wrapping access to an underlying value, just let the value be used directly.
class foo:
def __init__(self):
self.keys = []
self.keys = [1,2,3]
print(self.keys)
self.keys.append(4)
# etc
The nice thing about properties is that if you start by allowing direct access to keys, then nothing about how you use keys changes if you later decide to replace it with a property.

You can give this a try:
class Foo:
def __init__(self):
self._key = []
#property
def key(self):
return self._key
#key.setter
def key(self, value):
self._key = value
Here are my two cents:
Rename the class foo to Foo
You can't initialize self.key, as this is the property, so initialize the correct variable in the constructor (i.e. __init__)
Private vars are prefixed with one _ scope and not two (two __ are Python internals)
I suppose you rather want my_instance.key = ['spam', 'eggs'] to replace the foo._key value than extend it. Because this is kind of a "setter" and that would result in a weird behaviour, or at least another developer won't expect that behaviour from that setter/function
However, and that's important: As long as you're only doing this, you won't need properties. You can simply initialize self.keys in the constructor and froget about the property and setter function. Later on, when you want to change the behaviour, you can still add the property and setter. That's one reason why we've properties in Python, so that you won't have to refactor your whole code in case "a bit more logic" comes into place.
Btw. if you're really depending everything on those dict functions, you might also want to inherit your class from the dict class. Depends what you're up to.

Python - extending properties like you'd extend a function

Question
How can you extend a python property?
A subclass can extend a super class's function by calling it in the overloaded version, and then operating on the result. Here's an example of what I mean when I say "extending a function":
# Extending a function (a tongue-in-cheek example)
class NormalMath(object):
def __init__(self, number):
self.number = number
def add_pi(self):
n = self.number
return n + 3.1415
class NewMath(object):
def add_pi(self):
# NewMath doesn't know how NormalMath added pi (and shouldn't need to).
# It just uses the result.
n = NormalMath.add_pi(self)
# In NewMath, fractions are considered too hard for our users.
# We therefore silently convert them to integers.
return int(n)
Is there an analogous operation to extending functions, but for functions that use the property decorator?
I want to do some additional calculations immediately after getting an expensive-to-compute attribute. I need to keep the attribute's access lazy. I don't want the user to have to invoke a special routine to make the calculations. basically, I don't want the user to ever know the calculations were made in the first place. However, the attribute must remain a property, since i've got legacy code I need to support.
Maybe this is a job for decorators? If I'm not mistaken, decorator is a function that wraps another function, and I'm looking to wrap a property with some more calculations, and then present it as a property again, which seems like a similar idea... but I can't quite figure it out.
My Specific Problem
I've got a base class LogFile with an expensive-to-construct attribute .dataframe. I've implemented it as a property (with the property decorator), so it won't actually parse the log file until I ask for the dataframe. So far, it works great. I can construct a bunch (100+) LogFile objects, and use cheaper methods to filter and select only the important ones to parse. And whenever I'm using the same LogFile over and over, i only have to parse it the first time I access the dataframe.
Now I need to write a LogFile subclass, SensorLog, that adds some extra columns to the base class's dataframe attribute, but I can't quite figure out the syntax to call the super class's dataframe construction routines (without knowing anything about their internal workings), then operate on the resulting dataframe, and then cache/return it.
# Base Class - rules for parsing/interacting with data.
class LogFile(object):
def __init__(self, file_name):
# file name to find the log file
self.file_name = file_name
# non-public variable to cache results of parse()
self._dataframe = None
def parse(self):
with open(self.file_name) as infile:
...
...
# Complex rules to interpret the file
...
...
self._dataframe = pandas.DataFrame(stuff)
#property
def dataframe(self):
"""
Returns the dataframe; parses file if necessary. This works great!
"""
if self._dataframe is None:
self.parse()
return self._dataframe
#dataframe.setter
def dataframe(self,value):
self._dataframe = value
# Sub class - adds more information to data, but does't parse
# must preserve established .dataframe interface
class SensorLog(LogFile):
def __init__(self, file_name):
# Call the super's constructor
LogFile.__init__(self, file_name)
# SensorLog doesn't actually know about (and doesn't rely on) the ._dataframe cache, so it overrides it just in case.
self._dataframe = None
# THIS IS THE PART I CAN'T FIGURE OUT
# Here's my best guess, but it doesn't quite work:
#property
def dataframe(self):
# use parent class's getter, invoking the hidden parse function and any other operations LogFile might do.
self._dataframe = LogFile.dataframe.getter()
# Add additional calculated columns
self._dataframe['extra_stuff'] = 'hello world!'
return self._dataframe
#dataframe.setter
def dataframe(self, value):
self._dataframe = value
Now, when these classes are used in an interactive session, the user should be able to interact with either in the same way.
>>> log = LogFile('data.csv')
>>> print log.dataframe
#### DataFrame with 10 columns goes here ####
>>> sensor = SensorLog('data.csv')
>>> print sensor.dataframe
#### DataFrame with 11 columns goes here ####
I have lots of existing code that takes a LogFile instance which provides a .dataframe attribute and dos something interesting (mostly plotting). I would LOVE to have SensorLog instances present the same interface so they can use the same code. Is it possible to extend the super-class's dataframe getter to take advantage of existing routines? How? Or am I better off doing this a different way?
Thanks for reading that huge wall of text. You are an internet super hero, dear reader. Got any ideas?

You should be calling the superclass properties, not bypassing them via self._dataframe. Here's a generic example:
class A(object):
def __init__(self):
self.__prop = None
#property
def prop(self):
return self.__prop
#prop.setter
def prop(self, value):
self.__prop = value
class B(A):
def __init__(self):
super(B, self).__init__()
#property
def prop(self):
value = A.prop.fget(self)
value['extra'] = 'stuff'
return value
#prop.setter
def prop(self, value):
A.prop.fset(self, value)
And using it:
b = B()
b.prop = dict((('a', 1), ('b', 2)))
print(b.prop)
Outputs:
{'a': 1, 'b': 2, 'extra': 'stuff'}
I would generally recommend placing side-effects in setters instead of getters, like this:
class A(object):
def __init__(self):
self.__prop = None
#property
def prop(self):
return self.__prop
#prop.setter
def prop(self, value):
self.__prop = value
class B(A):
def __init__(self):
super(B, self).__init__()
#property
def prop(self):
return A.prop.fget(self)
#prop.setter
def prop(self, value):
value['extra'] = 'stuff'
A.prop.fset(self, value)
Having costly operations within a getter is also generally to be avoided (such as your parse method).

If I understand correctly what you want to do is call the parent's method from the child instance. The usual way to do that is by using the super built-in.
I've taken your tongue-in-cheek example and modified it to use super in order to show you:
class NormalMath(object):
def __init__(self, number):
self.number = number
def add_pi(self):
n = self.number
return n + 3.1415
class NewMath(NormalMath):
def add_pi(self):
# this will call NormalMath's add_pi with
normal_maths_pi_plus_num = super(NewMath, self).add_pi()
return int(normal_maths_pi_plus_num)
In your Log example, instead of calling:
self._dataframe = LogFile.dataframe.getter()
you should call:
self._dataframe = super(SensorLog, self).dataframe
You can read more about super here
Edit: Even thought the example I gave you deals with methods, to do the same with #properties shouldn't be a problem.

You have some possibilities to consider:
1/ Inherit from logfile and override parse in your derived sensor class. It should be possible to modify your methods that work on dataframe to work regardless of the number of members that dataframe has - as you are using pandas a lot of it is done for you.
2/ Make sensor an instance of logfile then provide its own parse method.
3/ Generalise parse, and possibly some of your other methods, to use a list of data descriptors and possibly a dictionary of methods/rules either set in your class initialiser or set by a methods.
4/ Look at either making more use of the methods already in pandas, or possibly, extending pandas to provide the missing methods if you and others think that they would be accepted into pandas as useful extensions.
Personally I think that you would find the benefits of options 3 or 4 to be the most powerful.

The problem is that you're missing a self going into the parent class. If your parent is a singleton then a #staticmethod should work.
class X():
x=1
#staticmethod
def getx():
return X.x
class Y(X):
y=2
def getyx(self):
return X.getx()+self.y
wx = Y()
wx.getyx()
3

How to implement property() with dynamic name (in python)

I am programming a simulations for single neurons. Therefore I have to handle a lot of Parameters. Now the Idea is that I have two classes, one for a SingleParameter and a Collection of parameters. I use property() to access the parameter value easy and to make the code more readable. This works perfect for a sinlge parameter but I don't know how to implement it for the collection as I want to name the property in Collection after the SingleParameter. Here an example:
class SingleParameter(object):
def __init__(self, name, default_value=0, unit='not specified'):
self.name = name
self.default_value = default_value
self.unit = unit
self.set(default_value)
def get(self):
return self._v
def set(self, value):
self._v = value
v = property(fget=get, fset=set, doc='value of parameter')
par1 = SingleParameter(name='par1', default_value=10, unit='mV')
par2 = SingleParameter(name='par2', default_value=20, unit='mA')
# par1 and par2 I can access perfectly via 'p1.v = ...'
# or get its value with 'p1.v'
class Collection(object):
def __init__(self):
self.dict = {}
def __getitem__(self, name):
return self.dict[name] # get the whole object
# to get the value instead:
# return self.dict[name].v
def add(self, parameter):
self.dict[parameter.name] = parameter
# now comes the part that I don't know how to implement with property():
# It shoule be something like
# self.__dict__[parameter.name] = property(...) ?
col = Collection()
col.add(par1)
col.add(par2)
col['par1'] # gives the whole object
# Now here is what I would like to get:
# col.par1 -> should result like col['par1'].v
# col.par1 = 5 -> should result like col['par1'].v = 5
Other questions that I put to understand property():
Why do managed attributes just work for class attributes and not for instance attributes in python?
How can I assign a new class attribute via __dict__ in python?

Look at built-in functions getattr and setattr. You'll probably be a lot happier.

Using the same get/set functions for both classes forces you into an ugly hack with the argument list. Very sketchy, this is how I would do it:
In class SingleParameter, define get and set as usual:
def get(self):
return self._s
def set(self, value):
self._s = value
In class Collection, you cannot know the information until you create the property, so you define the metaset/metaget function and particularize them only later with a lambda function:
def metaget(self, par):
return par.s
def metaset(self, value, par):
par.s = value
def add(self, par):
self[par.name] = par
setattr(Collection, par.name,
property(
fget=lambda x : Collection.metaget(x, par),
fset=lambda x, y : Collection.metaset(x,y, par))

Properties are meant to dynamically evaluate attributes or to make them read-only. What you need is customizing attribute access. __getattr__ and __setattr__ do that really fine, and there's also __getattribute__ if __getattr__ is not enough.
See Python docs on customizing attribute access for details.

Have you looked at the traits package? It seems that you are reinventing the wheel here with your parameter classes. Traits also have additional features that might be useful for your type of application (incidently I know a person that happily uses traits in neural simulations).

Now I implemented a solution with set-/getattr:
class Collection(object):
...
def __setattr__(self, name, value):
if 'dict' in self.__dict__:
if name in self.dict:
self[name].v = value
else:
self.__dict__[name] = value
def __getattr__(self, name):
return self[name].v
There is one thing I quite don't like that much: The attributes are not in the __dict__. And if I have them there as well I would have a copy of the value - which can be dangerous...

Finally I succeded to implement the classes with property(). Thanks a lot for the advice. It took me quite a bit to work it out - but I can promise you that this exercise helps you to understand better pythons OOP.
I implemented it also with __getattr__ and __setattr__ but still don't know the advantages and disadvantages to the property-solution. But this seems to be worth another question. The property-solutions seems to be quit clean.
So here is the code:
class SingleParameter(object):
def __init__(self, name, default_value=0, unit='not specified'):
self.name = name
self.default_value = default_value
self.unit = unit
self.set(default_value)
def get(*args):
self = args[0]
print "get(): "
print args
return self._v
def set(*args):
print "set(): "
print args
self = args[0]
value = args[-1]
self._v = value
v = property(fget=get, fset=set, doc='value of parameter')
class Collection(dict):
# inheriting from dict saves the methods: __getitem__ and __init__
def add(self, par):
self[par.name] = par
# Now here comes the tricky part.
# (Note: this property call the get() and set() methods with one
# more argument than the property of SingleParameter)
setattr(Collection, par.name,
property(fget=par.get, fset=par.set))
# Applying the classes:
par1 = SingleParameter(name='par1', default_value=10, unit='mV')
par2 = SingleParameter(name='par2', default_value=20, unit='mA')
col = Collection()
col.add(par1)
col.add(par2)
# Setting parameter values:
par1.v = 13
col.par1 = 14
# Getting parameter values:
par1.v
col.par1
# checking identity:
par1.v is col.par1
# to access the whole object:
col['par1']
As I am new I am not sure how to move on:
how to treat follow up questions (like this itself):
get() is seems to be called twice - why?
oop-design: property vs. "__getattr__ & __setattr__" - when should I use what?
is it rude to check the own answer to the own question as accepted?
is it recommended to rename the title in order to put correlated questions or questions elaborated with the same example into the same context?
Other questions that I put to understand property():
Why do managed attributes just work for class attributes and not for instance attributes in python?
How can I assign a new class attribute via __dict__ in python?

I have a class that does something similar, but I did the following in the collection object:
setattr(self, par.name, par.v)