We Keep Coding

Unclear descriptor caller reference evaluation

I am using Python descriptors to create complex interfaces on host objects.
I don't get the behaviour I would intuitively expect when I run code such as this:
class Accessor(object):
def __get__(self,inst,instype):
self._owner = inst
return self
def set(self,value):
self._owner._val = value
def get(self):
if hasattr(self._owner,'_val'):
return self._owner._val
else: return None
class TestClass(object):
acc = Accessor()
source = TestClass()
destination = TestClass()
source.acc.set('banana')
destination.acc.set('mango')
destination.acc.set(source.acc.get())
print destination.acc.get()
# Result: mango
I would expect in this case for destination.acc.get() to return 'banana', not 'mango'.
However, the intention (to copy _val from 'source' to 'destination') works if the code is refactored like this:
val = source.acc.get()
destination.acc.set(val)
print destination.acc.get()
# Result: banana
What is is that breaks down the 'client' reference passed through get if descriptors are used in a single line versus broken into separate lines? Is there a way to get the behaviour I would intuitively expect?
Many thanks in advance.
K

Your implementation ALMOST works. The problem with it comes up with destination.acc.set(source.acc.get()). What happens is that it first looks up destination.acc, which will set _owner to destination, but before it can call set(), it has to resolve the parameter, source.acc.get(), which will end up setting acc's _owner to source.
Since destination.acc and source.acc are the same object (descriptors are stored on the class, not the instance), you're calling set() on it after its _owner is set to source. That means you're setting source._val, not destination._val.
The way to get the behavior you would intuitively expect is to get rid or your get() and set() and replace them with __get__() and __set__(), so that your descriptor can be used for the reason a descriptor is used.
class Accessor(object):
def __get__(self, instance, owner): # you should use the conventional parameter names
if instance is None:
return self
else:
return instance._val
def __set__(self, instance, value):
instance._val = value
Then you could rewrite your client code as
source = TestClass()
destination = TestClass()
source.acc = 'banana'
destination.acc = 'mango'
destination.acc = source.acc
print destination.acc
The point of descriptors is to remove explicit getter and setter calls with implicit ones that look like simple attribute use. If you still want to be using your getters and setters on Accessor, then don't make it a descriptor. Do this instead:
class Accessor(object):
def get(self):
if hasattr(self, '_val'):
return self._val
else:
return None
def set(self, val):
self._val = val
Then rewrite TestClass to look more like this:
class TestClass(object):
def __init__(self):
self.acc = Accessor()
After that, your original client code would work.

I already said why it's not working in my other post. So, here's a way to use a descriptor while still retaining your get() and set() methods.
class Accessor(object):
def __get__(self, instance, owner):
if instance is None:
return self
elif not hasattr(instance, '_val'):
setattr(instance, '_val', Acc())
return getattr(instance, '_val')
class Acc(object):
def get(self):
if hasattr(self, '_val'):
return self._val
else:
return None
def set(self, val):
self._val = val
class TestClass(object):
acc = Accessor()
The trick is to move the get() and set() methods to a new class that is returned instead of returning self from the descriptor.

How to make python class support item assignment?

While looking over some code in Think Complexity, I noticed their Graph class assigning values to itself. I've copied a few important lines from that class and written an example class, ObjectChild, that fails at this behavior.
class Graph(dict):
def __init__(self, vs=[], es=[]):
for v in vs:
self.add_vertex(v)
for e in es:
self.add_edge(e)
def add_edge(self, e):
v, w = e
self[v][w] = e
self[w][v] = e
def add_vertex(self, v):
self[v] = {}
class ObjectChild(object):
def __init__(self, name):
self['name'] = name
I'm sure the different built in types all have their own way of using this, but I'm not sure whether this is something I should try to build into my classes. Is it possible, and how? Is this something I shouldn't bother with, relying instead on simple composition, e.g. self.l = [1, 2, 3]? Should it be avoided outside built in types?
I ask because I was told "You should almost never inherit from the builtin python collections"; advice I'm hesitant to restrict myself to.
To clarify, I know that ObjectChild won't "work", and I could easily make it "work", but I'm curious about the inner workings of these built in types that makes their interface different from a child of object.

In Python 3 and later, just add these simple functions to your class:
class some_class(object):
def __setitem__(self, key, value):
setattr(self, key, value)
def __getitem__(self, key):
return getattr(self, key)

They are accomplishing this magic by inheriting from dict. A better way of doing this is to inherit from UserDict or the newer collections.MutableMapping
You could accomplish a similar result by doing the same:
import collections
class ObjectChild(collections.MutableMapping):
def __init__(self, name):
self['name'] = name
You can also define two special functions to make your class dictionary-like: __getitem__(self, key) and __setitem__(self, key, value). You can see an example of this at Dive Into Python - Special Class Methods.

Disclaimer : I might be wrong.
the notation :
self[something]
is legit in the Graph class because it inherits fro dict. This notation is from the dictionnaries ssyntax not from the class attribute declaration syntax.
Although all namespaces associated with a class are dictionnaries, in your class ChildObject, self isn't a dictionnary. Therefore you can't use that syntax.
Otoh, in your class Graph, self IS a dictionnary, since it is a graph, and all graphs are dictionnaries because they inherit from dict.

Is using something like this ok?
def mk_opts_dict(d):
''' mk_options_dict(dict) -> an instance of OptionsDict '''
class OptionsDict(object):
def __init__(self, d):
self.__dict__ = d
def __setitem__(self, key, value):
self.__dict__[key] = value
def __getitem__(self, key):
return self.__dict__[key]
return OptionsDict(d)

I realize this is an old post, but I was looking for some details around item assignment and stumbled upon the answers here. Ted's post wasn't completely wrong. To avoid inheritance from dict, you can make a class inherit from MutableMapping, and then provide methods for __setitem__ and __getitem__.
Additionally, the class will need to support methods for __delitem__, __iter__, __len__, and (optionally) other inherited mixin methods, like pop. The documentation has more info on the details.
from collections.abc import MutableMapping
class ItemAssign(MutableMapping):
def __init__(self, a, b):
self.a = a
self.b = b
def __setitem__(self, k, v):
setattr(self, k, v)
def __getitem__(self, k):
getattr(self, k)
def __len__(self):
return 2
def __delitem__(self, k):
self[k] = None
def __iter__(self):
yield self.a
yield self.b
Example use:
>>> x = ItemAssign("banana","apple")
>>> x["a"] = "orange"
>>> x.a
'orange'
>>> del x["a"]
>>> print(x.a)
None
>>> x.pop("b")
'apple'
>>> print(x.b)
None
Hope this serves to clarify how to properly implement item assignment for others stumbling across this post :)

Your ObjectChild doesn't work because it's not a subclass of dict. Either of these would work:
class ObjectChild(dict):
def __init__(self, name):
self['name'] = name
or
class ObjectChild(object):
def __init__(self, name):
self.name = name

You don't need to inherit from dict. If you provide setitem and getitem methods, you also get the desired behavior I believe.
class a(object):
def __setitem__(self, k, v):
self._data[k] = v
def __getitem__(self, k):
return self._data[k]
_data = {}

Little memo about <dict> inheritance
For those who want to inherit dict.
In this case MyDict will have a shallow copy of original dict in it.
class MyDict(dict):
...
d = {'a': 1}
md = MyDict(d)
print(d['a']) # 1
print(md['a']) # 1
md['a'] = 'new'
print(d['a']) # 1
print(md['a']) # new
This could lead to problem when you have a tree of nested dicts and you want to covert part of it to an object. Changing this object will not affect its parent
root = {
'obj': {
'a': 1,
'd': {'x': True}
}
}
obj = MyDict(root['obj'])
obj['a'] = 2
print(root) # {'obj': {'a': 1, 'd': {'x': True}}} # 'a' is the same
obj['d']['x'] = False
print(root) # {'obj': {'a': 1, 'd': {'x': True}}} # 'x' chanded

Mapping obj.method({argument:value}) to obj.argument(value)

I don't know if this will make sense, but...
I'm trying to dynamically assign methods to an object.
#translate this
object.key(value)
#into this
object.method({key:value})
To be more specific in my example, I have an object (which I didn't write), lets call it motor, which has some generic methods set, status and a few others. Some take a dictionary as an argument and some take a list. To change the motor's speed, and see the result, I use:
motor.set({'move_at':10})
print motor.status('velocity')
The motor object, then formats this request into a JSON-RPC string, and sends it to an IO daemon. The python motor object doesn't care what the arguments are, it just handles JSON formatting and sockets. The strings move_at and velocity are just two of what might be hundreds of valid arguments.
What I'd like to do is the following instead:
motor.move_at(10)
print motor.velocity()
I'd like to do it in a generic way since I have so many different arguments I can pass. What I don't want to do is this:
# create a new function for every possible argument
def move_at(self,x)
return self.set({'move_at':x})
def velocity(self)
return self.status('velocity')
#and a hundred more...
I did some searching on this which suggested the solution lies with lambdas and meta programming, two subjects I haven't been able to get my head around.
UPDATE:
Based on the code from user470379 I've come up with the following...
# This is what I have now....
class Motor(object):
def set(self,a_dict):
print "Setting a value", a_dict
def status(self,a_list):
print "requesting the status of", a_list
return 10
# Now to extend it....
class MyMotor(Motor):
def __getattr__(self,name):
def special_fn(*value):
# What we return depends on how many arguments there are.
if len(value) == 0: return self.status((name))
if len(value) == 1: return self.set({name:value[0]})
return special_fn
def __setattr__(self,attr,value): # This is based on some other answers
self.set({attr:value})
x = MyMotor()
x.move_at = 20 # Uses __setattr__
x.move_at(10) # May remove this style from __getattr__ to simplify code.
print x.velocity()
output:
Setting a value {'move_at': 20}
Setting a value {'move_at': 10}
10
Thank you to everyone who helped!

What about creating your own __getattr__ for the class that returns a function created on the fly? IIRC, there's some tricky cases to watch out for between __getattr__ and __getattribute__ that I don't recall off the top of my head, I'm sure someone will post a comment to remind me:
def __getattr__(self, name):
def set_fn(self, value):
return self.set({name:value})
return set_fn
Then what should happen is that calling an attribute that doesn't exist (ie: move_at) will call the __getattr__ function and create a new function that will be returned (set_fn above). The name variable of that function will be bound to the name parameter passed into __getattr__ ("move_at" in this case). Then that new function will be called with the arguments you passed (10 in this case).
Edit
A more concise version using lambdas (untested):
def __getattr__(self, name):
return lambda value: self.set({name:value})

There are a lot of different potential answers to this, but many of them will probably involve subclassing the object and/or writing or overriding the __getattr__ function.
Essentially, the __getattr__ function is called whenever python can't find an attribute in the usual way.
Assuming you can subclass your object, here's a simple example of what you might do (it's a bit clumsy but it's a start):
class foo(object):
def __init__(self):
print "initting " + repr(self)
self.a = 5
def meth(self):
print self.a
class newfoo(foo):
def __init__(self):
super(newfoo, self).__init__()
def meth2(): # Or, use a lambda: ...
print "meth2: " + str(self.a) # but you don't have to
self.methdict = { "meth2":meth2 }
def __getattr__(self, name):
return self.methdict[name]
f = foo()
g = newfoo()
f.meth()
g.meth()
g.meth2()
Output:
initting <__main__.foo object at 0xb7701e4c>
initting <__main__.newfoo object at 0xb7701e8c>
5
5
meth2: 5

You seem to have certain "properties" of your object that can be set by
obj.set({"name": value})
and queried by
obj.status("name")
A common way to go in Python is to map this behaviour to what looks like simple attribute access. So we write
obj.name = value
to set the property, and we simply use
obj.name
to query it. This can easily be implemented using the __getattr__() and __setattr__() special methods:
class MyMotor(Motor):
def __init__(self, *args, **kw):
self._init_flag = True
Motor.__init__(self, *args, **kw)
self._init_flag = False
def __getattr__(self, name):
return self.status(name)
def __setattr__(self, name, value):
if self._init_flag or hasattr(self, name):
return Motor.__setattr__(self, name, value)
return self.set({name: value})
Note that this code disallows the dynamic creation of new "real" attributes of Motor instances after the initialisation. If this is needed, corresponding exceptions could be added to the __setattr__() implementation.

Instead of setting with function-call syntax, consider using assignment (with =). Similarly, just use attribute syntax to get a value, instead of function-call syntax. Then you can use __getattr__ and __setattr__:
class OtherType(object): # this is the one you didn't write
# dummy implementations for the example:
def set(self, D):
print "setting", D
def status(self, key):
return "<value of %s>" % key
class Blah(object):
def __init__(self, parent):
object.__setattr__(self, "_parent", parent)
def __getattr__(self, attr):
return self._parent.status(attr)
def __setattr__(self, attr, value):
self._parent.set({attr: value})
obj = Blah(OtherType())
obj.velocity = 42 # prints setting {'velocity': 42}
print obj.velocity # prints <value of velocity>

A python class that acts like dict

I want to write a custom class that behaves like dict - so, I am inheriting from dict.
My question, though, is: Do I need to create a private dict member in my __init__() method?. I don't see the point of this, since I already have the dict behavior if I simply inherit from dict.
Can anyone point out why most of the inheritance snippets look like the one below?
class CustomDictOne(dict):
def __init__(self):
self._mydict = {}
# other methods follow
Instead of the simpler...
class CustomDictTwo(dict):
def __init__(self):
# initialize my other stuff here ...
# other methods follow
Actually, I think I suspect the answer to the question is so that users cannot directly access your dictionary (i.e. they have to use the access methods that you have provided).
However, what about the array access operator []? How would one implement that? So far, I have not seen an example that shows how to override the [] operator.
So if a [] access function is not provided in the custom class, the inherited base methods will be operating on a different dictionary?
I tried the following snippet to test out my understanding of Python inheritance:
class myDict(dict):
def __init__(self):
self._dict = {}
def add(self, id, val):
self._dict[id] = val
md = myDict()
md.add('id', 123)
print md[id]
I got the following error:
KeyError: < built-in function id>
What is wrong with the code above?
How do I correct the class myDict so that I can write code like this?
md = myDict()
md['id'] = 123
[Edit]
I have edited the code sample above to get rid of the silly error I made before I dashed away from my desk. It was a typo (I should have spotted it from the error message).

class Mapping(dict):
def __setitem__(self, key, item):
self.__dict__[key] = item
def __getitem__(self, key):
return self.__dict__[key]
def __repr__(self):
return repr(self.__dict__)
def __len__(self):
return len(self.__dict__)
def __delitem__(self, key):
del self.__dict__[key]
def clear(self):
return self.__dict__.clear()
def copy(self):
return self.__dict__.copy()
def has_key(self, k):
return k in self.__dict__
def update(self, *args, **kwargs):
return self.__dict__.update(*args, **kwargs)
def keys(self):
return self.__dict__.keys()
def values(self):
return self.__dict__.values()
def items(self):
return self.__dict__.items()
def pop(self, *args):
return self.__dict__.pop(*args)
def __cmp__(self, dict_):
return self.__cmp__(self.__dict__, dict_)
def __contains__(self, item):
return item in self.__dict__
def __iter__(self):
return iter(self.__dict__)
def __unicode__(self):
return unicode(repr(self.__dict__))
o = Mapping()
o.foo = "bar"
o['lumberjack'] = 'foo'
o.update({'a': 'b'}, c=44)
print 'lumberjack' in o
print o
In [187]: run mapping.py
True
{'a': 'b', 'lumberjack': 'foo', 'foo': 'bar', 'c': 44}

Like this
class CustomDictOne(dict):
def __init__(self,*arg,**kw):
super(CustomDictOne, self).__init__(*arg, **kw)
Now you can use the built-in functions, like dict.get() as self.get().
You do not need to wrap a hidden self._dict. Your class already is a dict.

Check the documentation on emulating container types. In your case, the first parameter to add should be self.

UserDict from the Python standard library is designed for this purpose.

Here is an alternative solution:
class AttrDict(dict):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.__dict__ = self
a = AttrDict()
a.a = 1
a.b = 2

This is my best solution. I used this many times.
class DictLikeClass:
...
def __getitem__(self, key):
return getattr(self, key)
def __setitem__(self, key, value):
setattr(self, key, value)
...
You can use like:
>>> d = DictLikeClass()
>>> d["key"] = "value"
>>> print(d["key"])

A python class that acts like dict
What's wrong with this?
Can anyone point out why most of the inheritance snippets look like the one below?
class CustomDictOne(dict):
def __init__(self):
self._mydict = {}
Presumably there's a good reason to inherit from dict (maybe you're already passing one around and you want a more specific kind of dict) and you have a good reason to instantiate another dict to delegate to (because this will instantiate two dicts per instance of this class.) But doesn't that sound incorrect?
I never run into this use-case myself. I do like the idea of typing dicts where you are using dicts that are type-able. But in that case I like the idea of typed class attributes even moreso - and the whole point of a dict is you can give it keys of any hashable type, and values of any type.
So why do we see snippets like this? I personally think it's an easily made mistake that went uncorrected and thus perpetuated over time.
I would rather see, in these snippets, this, to demonstrate code reuse through inheritance:
class AlternativeOne(dict):
__slots__ = ()
def __init__(self):
super().__init__()
# other init code here
# new methods implemented here
or, to demonstrate re-implementing the behavior of dicts, this:
from collections.abc import MutableMapping
class AlternativeTwo(MutableMapping):
__slots__ = '_mydict'
def __init__(self):
self._mydict = {}
# other init code here
# dict methods reimplemented and new methods implemented here
By request - adding slots to a dict subclass.
Why add slots? A builtin dict instance doesn't have arbitrary attributes:
>>> d = dict()
>>> d.foo = 'bar'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'foo'
If we create a subclass the way most are doing it here on this answer, we see we don't get the same behavior, because we'll have a __dict__ attribute, causing our dicts to take up to potentially twice the space:
my_dict(dict):
"""my subclass of dict"""
md = my_dict()
md.foo = 'bar'
Since there's no error created by the above, the above class doesn't actually act, "like dict."
We can make it act like dict by giving it empty slots:
class my_dict(dict):
__slots__ = ()
md = my_dict()
So now attempting to use arbitrary attributes will fail:
>>> md.foo = 'bar'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'my_dict' object has no attribute 'foo'
And this Python class acts more like a dict.
For more on how and why to use slots, see this Q&A: Usage of __slots__?

I really don't see the right answer to this anywhere
class MyClass(dict):
def __init__(self, a_property):
self[a_property] = a_property
All you are really having to do is define your own __init__ - that really is all that there is too it.
Another example (little more complex):
class MyClass(dict):
def __init__(self, planet):
self[planet] = planet
info = self.do_something_that_returns_a_dict()
if info:
for k, v in info.items():
self[k] = v
def do_something_that_returns_a_dict(self):
return {"mercury": "venus", "mars": "jupiter"}
This last example is handy when you want to embed some kind of logic.
Anyway... in short class GiveYourClassAName(dict) is enough to make your class act like a dict. Any dict operation you do on self will be just like a regular dict.

The problem with this chunk of code:
class myDict(dict):
def __init__(self):
self._dict = {}
def add(id, val):
self._dict[id] = val
md = myDict()
md.add('id', 123)
...is that your 'add' method (...and any method you want to be a member of a class) needs to have an explicit 'self' declared as its first argument, like:
def add(self, 'id', 23):
To implement the operator overloading to access items by key, look in the docs for the magic methods __getitem__ and __setitem__.
Note that because Python uses Duck Typing, there may actually be no reason to derive your custom dict class from the language's dict class -- without knowing more about what you're trying to do (e.g, if you need to pass an instance of this class into some code someplace that will break unless isinstance(MyDict(), dict) == True), you may be better off just implementing the API that makes your class sufficiently dict-like and stopping there.

Don’t inherit from Python built-in dict, ever! for example update method woldn't use __setitem__, they do a lot for optimization. Use UserDict.
from collections import UserDict
class MyDict(UserDict):
def __delitem__(self, key):
pass
def __setitem__(self, key, value):
pass

Controlling getter and setter for a python's class

Consider the following class :
class Token:
def __init__(self):
self.d_dict = {}
def __setattr__(self, s_name, value):
self.d_dict[s_name] = value
def __getattr__(self, s_name):
if s_name in self.d_dict.keys():
return self.d_dict[s_name]
else:
raise AttributeError('No attribute {0} found !'.format(s_name))
In my code Token have some other function (like get_all() wich return d_dict, has(s_name) which tell me if my token has a particular attribute).
Anyway, I think their is a flaw in my plan since it don't work : when I create a new instance, python try to call __setattr__('d_dict', '{}').
How can I achieve a similar behaviour (maybe in a more pythonic way ?) without having to write something like Token.set(name, value) and get(name) each I want to set or get an attribute for a token.
Critics about design flaw and/or stupidity welcome :)
Thank !

You need to special-case d_dict.
Although of course, in the above code, all you do is replicate what any object does with __dict__ already, so it's pretty pointless. Do I guess correctly if you intended to special case some attributes and actally use methods for those?
In that case, you can use properties.
class C(object):
def __init__(self):
self._x = None
#property
def x(self):
"""I'm the 'x' property."""
return self._x
#x.setter
def x(self, value):
self._x = value
#x.deleter
def x(self):
del self._x

The special-casing of __dict__ works like this:
def __init__(self):
self.__dict__['d_dict'] = {}
There is no need to use a new-style class for that.

A solution, not very pythonic but works. As Lennart Regebro pointed, you have to use a special case for d_dict.
class Token(object):
def __init__(self):
super(Token,self).__setattr__('d_dict', {})
def __getattr__(self,name):
return self.a[name]
def __setattr__(self,name,value):
self.a[name] = value
You need to use new style classes.

the problem seems to be in time of evaluation of your code in __init__ method.
You could define __new__ method and initialize d_dict variable there instead of __init__.
Thats a bit hackish but it works, remember though to comment it as after few months it'll be total magic.
>>> class Foo(object):
... def __new__(cls, *args):
... my_cls = super(Foo, cls).__new__(cls, *args)
... my_cls.d_dict = {}
... return my_cls
>>> f = Foo()
>>> id(f.d_dict)
3077948796L
>>> d = Foo()
>>> id(d.d_dict)
3078142804L
Word of explanation why I consider that hackish: call to __new__ returns new instance of class so then d_dict initialised in there is kind of static, but it's initialised with new instance of dictionary each time class is "created" so everything works as you need.

It's worth remembering that __getattr__ is only called if the attribute doesn't exist in the object, whereas __setattr__ is always called.

I think we'll be able to say something about the overall design of your class if you explain its purpose. For example,
# This is a class that serves as a dictionary but also has user-defined methods
class mydict(dict): pass
# This is a class that allows setting x.attr = value or getting x.attr:
class mysetget: pass
# This is a class that allows setting x.attr = value or getting x.attr:
class mygetsethas:
def has(self, key):
return key in self.__dict__
x = mygetsethas()
x.a = 5
print(x.has('a'), x.a)
I think the last class is closest to what you meant, and I also like to play with syntax and get lots of joy from it, but unfortunately this is not a good thing. Reasons why it's not advisable to use object attributes to re-implement dictionary: you can't use x.3, you conflict with x.has(), you have to put quotes in has('a') and many more.