I'm following this link and trying to make a singleton class. But, taking arguments (passed while initiating a class) into account so that the same object is returned if the arguments are same.
So, instead of storing class name/class reference as a dict key, I want to store passed arguments as keys in dict. But, there could be unhashable arguments also (like dict, set itself).
What is the best way to store class arguments and class objects mapping? So that I can return an object corresponding to the arguments.
Thanks anyways.
EDIT-1 :
A little more explanation. Let's say there is class as follows
class A:
__metaclass__ == Singleton
def __init__(arg1, arg2):
pass
Now, A(1,2) should always return the same object. But, it should be different from A(3,4)
I think, the arguments very much define the functioning of a class. Let's say if the class is to make redis connections. I might want to create 2 singletons objects with diff redis hosts as parameters, but the underlying class/code could be common.
As theheadofabroom and me already mentioned in the comments, there are some odds when relying on non-hashable values for instance caching or memoization. Therefore, if you still want to do exactly that, the following example does not hide the memoization in the __new__ or __init__ method. (A self-memoizing class would be hazardous because the memoization criterion can be fooled by code that you don't control).
Instead, I provide the function memoize which returns a memoizing factory function for a class. Since there is no generic way to tell from non-hashable arguments, if they will result in an instance that is equivalent to an already existing isntance, the memoization semantics have to be provided explicitly. This is achieved by passing the keyfunc function to memoize. keyfunc takes the same arguments as the class' __init__ method and returns a hashable key, whose equality relation (__eq__) determines memoization.
The proper use of the memoization is in the responsibility of the using code (providing a sensible keyfunc and using the factory), since the class to be memoized is not modified and can still be instantiated normally.
def memoize(cls, keyfunc):
memoized_instances = {}
def factory(*args, **kwargs):
key = keyfunc(*args, **kwargs)
if key in memoized_instances:
return memoized_instances[key]
instance = cls(*args, **kwargs)
memoized_instances[key] = instance
return instance
return factory
class MemoTest1(object):
def __init__(self, value):
self.value = value
factory1 = memoize(MemoTest1, lambda value : value)
class MemoTest2(MemoTest1):
def __init__(self, value, foo):
MemoTest1.__init__(self, value)
self.foo = foo
factory2 = memoize(MemoTest2, lambda value, foo : (value, frozenset(foo)))
m11 = factory1('test')
m12 = factory1('test')
assert m11 is m12
m21 = factory2('test', [1, 2])
lst = [1, 2]
m22 = factory2('test', lst)
lst.append(3)
m23 = factory2('test', lst)
assert m21 is m22
assert m21 is not m23
I only included MemoTest2 as a sublclass of MemoTest1 to show that there is no magic involved in using regular class inheritance.
Related
I apologise if the title is cryptic, I could not think of a way to describe my problem in a sentence. I am building some code in python2.7 that I describe below.
Minimal working example
My code has a Parameter class that implements attributes such as name and value, which looks something like this.
class Parameter(object):
def __init__(self, name, value=None, error=None, dist=None, prior=None):
self.name = name
self._value = value # given value for parameter, this is going to be changed very often in an MCMC sampler
self.error = error # initial estimate of error for the parameter, will only be set once
self._dist = dist # a distribution for the parameter, will only be set once
self.prior = prior
#property
def value(self):
return self._value
#property
def dist(self):
return self._dist
The class also has several properties that returns the mean, median, etc. of Parameter.dist if a distribution is given.
I have another class, e.g. ParameterSample, that creates a population of different Parameter objects. Some of these Parameter objects have their attributes (e.g. value, error) set using the Parameter.set_parameter() function, but some other Parameter objects are not explicitly set, but their value and dist attributes depend on some of the other Parameter objects that are set:
class ParameterSample(object):
def __init__(self):
varied_parameters = ('a', 'b') # parameter names whose `value` attribute is varied
derived_parameters = ('c',) # parameter names whose `value` attribute is varied, but depends on `a.value` and `b.value`
parameter_names = varied_parameters + derived_parameters
# create `Parameter` objects for each parameter name
for name in parameter_names:
setattr(self, name, Parameter(name))
def set_parameter(self, name, **kwargs):
for key, val in kwargs.items():
if key == 'value':
key = '_'.join(['', key]) # add underscore to set `Parameter._value`
setattr(getattr(self, name), key, val) # basically does e.g. `self.a.value = 1`
I can now create a ParameterSample and use them like this:
parobj = ParameterSample()
parobj.set_parameter('a', value=1, error=0.1)
parobj.set_parameter('b', value=2, error=0.5)
parobj.a.value
>>> 1
parobj.b.error
>>> 0.5
parobj.set_parameter('b', value=3)
parobj.b.value
>>> 3
parobj.b.error
>>> 0.5
What I want
What I ultimately want, is to use Parameter.c the same way. For example:
parobj.c.value
>>> 4 # returns parobj.a.value + parobj.b.value
parobj.c.dist
>>> None # returns a.dist + b.dist, but since they are not currently set it is None
c therefore needs to be a Parameter object with all the same attributes as a and b, but where its value and dist are updated according to the current attributes of a and b.
However, I should also mention that I want to be able to set the allowed prior ranges for parameter c, e.g. parobj.set_parameter('c', prior=(0,10)) before making any calls to its value -- so c needs to be an already defined Parameter object upon the creation of the ParameterSample object.
How would I implement this into my ParameterSample class?
What I've tried
I have tried looking into making my own decorators, but I am not sure if that is the way to go since I don't fully understand how I would use those.
I've also considered adding a #property to c that creates a new Parameter object every time it is called, but I feel like that is not the way to go since it may slow down the code.
I should also note that the ParameterSample class above is going to be inherited in a different class, so whatever the solution is it should be able to be used in this setting:
class Companion(ParameterSample)
def __init__(self, name):
self.name = name
super(Companion, self).__init__()
comp = Companion(name='Earth')
comp.set_parameter('a', value=1)
comp.set_parameter('b', value=3)
comp.c.value
>>> 4
I could not get this to work in Python 2 - the setattr calls never seemed to propagate the attributes to the child classes (Companion would have no c attribute).
I was more successful with Python 3 though. Since you have two parameter types (varied vs. derived), it makes sense IMO to have two classes to implement the behavior, instead of treating them all as one.
I added a DerivedParameter class, inheriting from Parameter that takes a dependents argument (along with its parent class' args/kwargs), but redefining value and dist to give dependent behavior:
class DerivedParameter(Parameter):
def __init__(self, name, dependents, **kwargs):
self._dependents = dependents
super().__init__(name, **kwargs)
#property
def value(self):
try:
return sum(x._value for x in self._dependents if x is not None)
except TypeError:
return None
#property
def dist(self):
try:
return sum(x._dist for x in self._dependents if x is not None)
except TypeError:
return None
Then I adjusted how your parameter objects are added:
class ParameterSample:
def __init__(self):
# Store as instance attributes to reference later
self.varied_params = ('a', 'b') # parameter names whose `value` attribute is varied
self.derived_params = ('c',) # parameter names whose `value` attribute is varied, but depends on `a.value` and `b.value`
# No more combined names
# create `Parameter` objects for each varied parameter name
for name in self.varied_params:
setattr(self, name, Parameter(name))
# Create `DerivedParameter` objects for each derived parameter
# Derived parameters depend on all `Parameter` objects. It wasn't
# clear if this was the desired behavior though.
params = [v for _, v in self.__dict__.items() if isinstance(v, Parameter)]
for name in self.derived_params:
setattr(self, name, DerivedParameter(name, params))
def set_parameter(self, name, **kwargs):
for key, val in kwargs.items():
if key == 'value':
key = '_'.join(['', key]) # add underscore to set `Parameter._value`
setattr(getattr(self, name), key, val) # basically does e.g. `self.a.value = 1`
From this, I could then replicate your given example desired behavior:
>>> comp = Companion(name='Earth')
>>> comp.set_parameter('a', value=1)
>>> comp.set_parameter('b', value=3)
>>> print(comp.c.value)
>>> print(comp.c.dist)
4
None
>>> comp.set_parameter('c', prior=(0,10))
>>> print(comp.c.prior)
(0, 10)
As I pointed out in the comments, the design above ends up causing all derived parameters to use all varied parameters as their dependents - effectively making c and a potential d identical. You should be able to fix this fairly easily with some parameters/conditions.
Overall, I would have to agree with #Error - Syntactical Remorse though. This is a pretty complicated way to go about designing classes and would make maintenance confusing at best. I would strongly encourage you to reconsider your design and try to find an adaptable general solution that doesn't involve dynamic creation of attributes like this.
Edit:
This question has been marked duplicate but I don't think that it is. Implementing the suggested answer, that is to use the Mapping abc, does not have the behavior I would like:
from collections import Mapping
class data(Mapping):
def __init__(self,params):
self.params = params
def __getitem__(self,k):
print "getting",k
return self.params[k]
def __len__(self):
return len(self.params)
def __iter__(self):
return ( k for k in self.params.keys() )
def func(*args,**kwargs):
print "In func"
return None
ps = data({"p1":1.,"p2":2.})
print "\ncalling...."
func(ps)
print "\ncalling...."
func(**ps)
Output:
calling....
In func
calling....
in __getitem__ p2
in __getitem__ p1
In func
Which, as mentioned in the question, is not what I want.
The other solution, given in the comments, is to modify the routines that are causing problems. That will certainly work, however I was looking for a quick (lazy?) fix!
Question:
How can I implement the ** operator for a class, other than via __getitem__? For example I would like to be able to do this::
def func(**kwargs):
<do some clever stuff>
x = some_generic_class():
func( **x )
without an implicit call to some_generic_class.__getitem__(). In my application I have already implemented __getitem__ with some data logging which I do not want to perform when the class is referenced as above.
If it's not possible to overload the ** operator, is it possible to detect when __getitem__ is being called as a result of the class being passed to a function, rather than explicitly?
Background:
I am working on a physics model that is built out of a set of packages which are chosen according to user input at runtime. The flexible structure of the model means that I rarely know the required parameters and so i pass a dict of parameter names and values between the models. In order to make this more user friendly I am now trying to develop a class paramlist that overloads the dict functionality with a set of routines that do some consistency checking, set default values, etc. The idea is that I pass an instance of paramlist rather than a dict. One of the more important aims is to keep a log of which members of paramlist have been referenced by the physics packages and which ones have not. A stripped out version is below, which aims to maintain a second dict that logs whether a parameter has been referenced::
class paramlist(object):
def __init__( self, params ):
self.params = copy(params)
self.used = { k:False for k in self.params }
def __getitem__(self, k):
try:
v = self.params[k]
except KeyError:
raise KeyError("Parameter {} not in parameter list".format(k))
else:
self.used[k] = True
return v
def __setitem__(self,k,v):
self.params[k] = v
self.used[k] = False
Which does not have the behaviour I want:
ps = paramlist( {"p1":1.} )
def donothing( *args, **kwargs ):
return None
donothing(ps)
print paramlist.used["p1"]
donothing(**ps)
print paramlist.used["p1"]
Output:
False
True
I would like the use dict to remain False in both cases, so that I can tell the user that one of their parameters was not used (implying that they screwed up and a default value has been used instead). I presume that the ** case has the effect of calling __getitem__ on every entry in the paramlist.
In my code I have a class, where one method is responsible for filtering some data. To allow customization for descendants I would like to define filtering function as a class attribute as per below:
def my_filter_func(x):
return x % 2 == 0
class FilterClass(object):
filter_func = my_filter_func
def filter_data(self, data):
return filter(self.filter_func, data)
class FilterClassDescendant(FilterClass):
filter_func = my_filter_func2
However, such code leads to TypeError, as filter_func receives "self" as first argument.
What is a pythonic way to handle such use cases? Perhaps, I should define my "filter_func" as a regular class method?
You could just add it as a plain old attribute?
def my_filter_func(x):
return x % 2 == 0
class FilterClass(object):
def __init__(self):
self.filter_func = my_filter_func
def filter_data(self, data):
return filter(self.filter_func, data)
Alternatively, force it to be a staticmethod:
def my_filter_func(x):
return x % 2 == 0
class FilterClass(object):
filter_func = staticmethod(my_filter_func)
def filter_data(self, data):
return filter(self.filter_func, data)
Python has a lot of magic within. One of those magics has something to do with transforming functions into UnboundMethod objects (when assigned to the class, and not to an class' instance).
When you assign a function (And I'm not sure whether it applies to any callable or just functions), Python converts it to an UnboundMethod object (i.e. an object which can be called using an instance or not).
Under normal conditions, you can call your UnboundMethod as normal:
def myfunction(a, b):
return a + b
class A(object):
a = myfunction
A.a(1, 2)
#prints 3
This will not fail. However, there's a distinct case when you try to call it from an instance:
A().a(1, 2)
This will fail since when an instance gets (say, internal getattr) an attribute which is an UnboundMethod, it returns a copy of such method with the im_self member populated (im_self and im_func are members of UnboundMethod). The function you intended to call, is in the im_func member. When you call this method, you're actually calling im_func with, additionally, the value in im_self. So, the function needs an additional parameter (the first one, which will stand for self).
To avoid this magic, Python has two possible decorators:
If you want to pass the function as-is, you must use #staticmethod. In this case, you will have the function not converted to UnboundMethod. However, you will not be able to access the calling class, except as a global reference.
If you want to have the same, but be able to access the current class (disregarding whether the function it is called from an instance or from a class), then your function should have another first argument (INSTEAD of self: cls) which is a reference to the class, and the decorator to use is #classmethod.
Examples:
class A(object):
a = staticmethod(lambda a, b: a + b)
A.a(1, 2)
A().a(1, 2)
Both will work.
Another example:
def add_print(cls, a, b):
print cls.__name__
return a + b
class A(object):
ap = classmethod(add_print)
class B(A):
pass
A.ap(1, 2)
B.ap(1, 2)
A().ap(1, 2)
B().ap(1, 2)
Check this by yourseld and enjoy the magic.
Lets assume I have a class awhich has the function __eq__(self,other). Now I want to have a dictionary where the keys are instances of the class (and the values numbers, but that should not make a difference). Then I get the error:
unhashable type: 'a'
In the documenation it says that I should define __eq__ and __cmp__ in order to define __hash__, but that is not possible since my class is not comparable!!
How to solve this, folks!
edit: Ok I made it works with only a _eq_ and _hash_ method, but I am still not sure if python uses the hash method in the in operation or the _eq_ method (which should be the case I hope)
The documentation says that beside the __hash__() it needs an __eq__() or (not "and" as you suggest) __cmp__() method.
So in your case it is enough to define the __hash__() method.
A class can be a key for a dict, so long as the hashCode for the class is constant. If at any point in time that the key, hashCode, for the class can change, then you would not be able to use it as a key.
This is precisely why a list cannot be used as a key. An alternative for the list would be to use a tuple since tuples are immutable. Again, if you can guarantee that the hashCode won't change, you're good.
It works for me..
>>> class A:
... def __init__(self):
... self.a = 5
...
>>> a = A()
>>> d = { a:"hello"}
>>> d[a]
'hello'
You can use class instances as keys for a dict
This is not a direct answer, but may be it can help you.
You can override __cmp__ and raise NotImplementedError to avoid usage of it.
class SomeClass():
def __init__(self):
pass # your init here
def __cmp__(self, orher):
raise NotImplementedError('You can not compare this')
def __eq__(self, other):
pass # Your eq here
def __hash__(self):
pass # your hash function here
Demo:
>> s = SomeClass()
>> s == '12'
>> NotImplementedError: You can not compare this
I have a class (list of dicts) and I want it to sort itself:
class Table(list):
…
def sort (self, in_col_name):
self = Table(sorted(self, key=lambda x: x[in_col_name]))
but it doesn't work at all. Why? How to avoid it? Except for sorting it externally, like:
new_table = Table(sorted(old_table, key=lambda x: x['col_name'])
Isn't it possible to manipulate the object itself? It's more meaningful to have:
class Table(list):
pass
than:
class Table(object):
l = []
…
def sort (self, in_col_name):
self.l = sorted(self.l, key=lambda x: x[in_col_name])
which, I think, works.
And in general, isn't there any way in Python which an object is able to change itself (not only an instance variable)?
You can't re-assign to self from within a method and expect it to change external references to the object.
self is just an argument that is passed to your function. It's a name that points to the instance the method was called on. "Assigning to self" is equivalent to:
def fn(a):
a = 2
a = 1
fn(a)
# a is still equal to 1
Assigning to self changes what the self name points to (from one Table instance to a new Table instance here). But that's it. It just changes the name (in the scope of your method), and does affect not the underlying object, nor other names (references) that point to it.
Just sort in place using list.sort:
def sort(self, in_col_name):
super(Table, self).sort(key=lambda x: x[in_col_name])
Python is pass by value, always. This means that assigning to a parameter will never have an effect on the outside of the function. self is just the name you chose for one of the parameters.
I was intrigued by this question because I had never thought about this. I looked for the list.sort code, to see how it's done there, but apparently it's in C. I think I see where you're getting at; what if there is no super method to invoke? Then you can do something like this:
class Table(list):
def pop_n(self, n):
for _ in range(n):
self.pop()
>>> a = Table(range(10))
>>> a.pop_n(3)
>>> print a
[0, 1, 2, 3, 4, 5, 6]
You can call self's methods, do index assignments to self and whatever else is implemented in its class (or that you implement yourself).