We Keep Coding

Python caching attributes in object with __slots__

I am trying to cache a computationally expensive property in a class defined with the __slots__ attribute.
Any idea, how to store the cache for later use? Of course the usual way to store a dictionary in instance._cache would not work without __dict__ being defined. For several reasons i do not want to add a '_cache' string to __slots__.
I was thinking whether this is one of the rare use cases for global. Any thoughts or examples on this matter?

There is no magic possible there - ou want to store a value, so you need a place to store your value.
You can't just decide "I won't have an extra entry on my __slots__ because it is not elegant" - you don't need to call it _cached:
give it whatever name you want, but these cached values are something you want to exist in each of the object's instances, and therefore you need an attribute.
You can cache in a global (module level) dictionary, in which the keys are id(self) - but that would be a major headache to keep synchronized when instances are deleted. (The same thing is true for a class-level dictionary, with the further downside of it still be visible on the instance).
TL;DR: the "one and obvious way to do it" is to have a shadow attribute, starting with "_" to keep the values you want cached, and declare these in __slots__. (If you use a _cached dictionary per instance, you loose the main advantage from __slots__, that is exactly not needing one dictionary per instance).

You don't quite need a global; you can store the cache as a class property and still define the expensive property as a property.
class Foo(object):
__slots__ = ('a', 'b', 'c')
expensive_cache = {}
#property
def expensive(self):
if self not in self.expensive_cache:
self.expensive_cache[self] = self._compute_expensive()
return self.expensive_cache[self]
def _compute_expensive(self):
print("Computing expensive property for {}".format(self))
return 3
f = Foo()
g = Foo()
print(f.expensive)
print("===")
print(f.expensive)
print("===")
print(g.expensive)
If you run this code, you can see that _compute_expensive is run only once, the first time you access expensive for each distinct object.
$ python3 tmp.py
Computing expensive property for <__main__.Foo object at 0x102861188>
3
===
3
===
Computing expensive property for <__main__.Foo object at 0x1028611c8>
3

Something like Borg pattern can help.
You can alterate the status of your instance in the __init__ or __new__ methods.

Weak reference to Python class method

Python 2.7 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
methods (both bound and unbound), ...
And Python 3.3 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
instance methods, ...
To me, these indicate that weakrefs to bound methods (in all versions Python 2.7 - 3.3) should be good, and that weakrefs to unbound methods should be good in Python 2.7.
Yet in Python 2.7, creating a weakref to a method (bound or unbound) results in a dead weakref:
>>> def isDead(wr): print 'dead!'
...
>>> class Foo:
... def bar(self): pass
...
>>> wr=weakref.ref(Foo.bar, isDead)
dead!
>>> wr() is None
True
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Not what I would have expected based on the docs.
Similarly, in Python 3.3, a weakref to a bound method dies on creation:
>>> wr=weakref.ref(Foo.bar, isDead)
>>> wr() is None
False
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Again not what I would have expected based on the docs.
Since this wording has been around since 2.7, it's surely not an oversight. Can anyone explain how the statements and the observed behavior are in fact not in contradiction?
Edit/Clarification: In other words, the statement for 3.3 says "instance methods can be weak referenced"; doesn't this mean that it is reasonable to expect that weakref.ref(an instance method)() is not None? and if it None, then "instance methods" should not be listed among the types of objects that can be weak referenced?

Foo.bar produces a new unbound method object every time you access it, due to some gory details about descriptors and how methods happen to be implemented in Python.
The class doesn't own unbound methods; it owns functions. (Check out Foo.__dict__['bar'].) Those functions just happen to have a __get__ which returns an unbound-method object. Since nothing else holds a reference, it vanishes as soon as you're done creating the weakref. (In Python 3, the rather unnecessary extra layer goes away, and an "unbound method" is just the underlying function.)
Bound methods work pretty much the same way: the function's __get__ returns a bound-method object, which is really just partial(function, self). You get a new one every time, so you see the same phenomenon.
You can store a method object and keep a reference to that, of course:
>>> def is_dead(wr): print "blech"
...
>>> class Foo(object):
... def bar(self): pass
...
>>> method = Foo.bar
>>> wr = weakref.ref(method, is_dead)
>>> 1 + 1
2
>>> method = None
blech
This all seems of dubious use, though :)
Note that if Python didn't spit out a new method instance on every attribute access, that'd mean that classes refer to their methods and methods refer to their classes. Having such cycles for every single method on every single instance in the entire program would make garbage collection way more expensive—and before 2.1, Python didn't even have cycle collection, so they would've stuck around forever.

#Eevee's answer is correct but there is a subtlety that is important.
The Python docs state that instance methods (py3k) and un/bound methods (py2.4+) can be weak referenced. You'd expect (naively, as I did) that weakref.ref(foo.bar)() would therefore be non-None, yet it is None, making the weak ref "dead on arrival" (DOA). This lead to my question, if the weakref to an instance method is DOA, why do the docs say you can weak ref a method?
So as #Eevee showed, you can create a non-dead weak reference to an instance method, by creating a strong reference to the method object which you give to weakref:
m = foo.bar # creates a *new* instance method "Foo.bar" and strong refs it
wr = weakref.ref(m)
assert wr() is not None # success
The subtlety (to me, anyways) is that a new instance method object is created every time you use Foo.bar, so even after the above code is run, the following will fail:
wr = weakref.ref(foo.bar)
assert wr() is not None # fails
because foo.bar is new instance of the "Foo instance" foo's "bar" method, different from m, and there is no strong ref to this new instance, so it is immediately gc'd, even if you have created a strong reference to it earlier (it is not the same strong ref). To be clear,
>>> d1 = foo.bla # assume bla is a data member
>>> d2 = foo.bla # assume bla is a data member
>>> d1 is d2
True # which is what you expect
>>> m1 = foo.bar # assume bar is an instance method
>>> m2 = foo.bar
>>> m1 is m2
False # !!! counter-intuitive
This takes many people by surprise since no one expects access to an instance member to be creating a new instance of anything. For example, if foo.bla is a data member of foo, then using foo.bla in your code does not create a new instance of the object referenced by foo.bla. Now if bla is a "function", foo.bla does create a new instance of type "instance method" representing the bound function.
Why the weakref docs (since python 2.4!) don't point that out is very strange, but that's a separate issue.

While I see that there's an accepted answer as to why this should be so, from a simple use-case situation wherein one would like an object that acts as a weakref to a bound method, I believe that one might be able to sneak by with an object as such. It's kind of a runt compared to some of the 'codier' things out there, but it works.
from weakref import proxy
class WeakMethod(object):
"""A callable object. Takes one argument to init: 'object.method'.
Once created, call this object -- MyWeakMethod() --
and pass args/kwargs as you normally would.
"""
def __init__(self, object_dot_method):
self.target = proxy(object_dot_method.__self__)
self.method = proxy(object_dot_method.__func__)
###Older versions of Python can use 'im_self' and 'im_func' in place of '__self__' and '__func__' respectively
def __call__(self, *args, **kwargs):
"""Call the method with args and kwargs as needed."""
return self.method(self.target, *args, **kwargs)
As an example of its ease of use:
class A(object):
def __init__(self, name):
self.name = name
def foo(self):
return "My name is {}".format(self.name)
>>> Stick = A("Stick")
>>> WeakFoo = WeakMethod(Stick.foo)
>>> WeakFoo()
'My name is Stick'
>>> Stick.name = "Dave"
>>> WeakFoo()
'My name is Dave'
Note that evil trickery will cause this to blow up, so depending on how you'd prefer it to work this may not be the best solution.
>>> A.foo = lambda self: "My eyes, aww my eyes! {}".format(self.name)
>>> Stick.foo()
'My eyes, aww my eyes! Dave'
>>> WeakFoo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __call__
ReferenceError: weakly-referenced object no longer exists
>>>
If you were going to be replacing methods on-the-fly you might need to use a getattr(weakref.proxy(object), 'name_of_attribute_as_string') approach instead. getattr is a fairly fast look-up so that isn't the literal worst thing in the world, but depending on what you're doing, YMMV.

How to get the object for a given class name in Python?

Is there any way to get the object name when the class name is known. If there are multiple objects for a class they also need to be printed.
Class A():
pass
Assume that some one have created objects for class A in some other files. So, I want to look all instances of 'Class A'

If you are the one creating the class you can simply store weak-references when instantiating the class:
import weakref
class A(object):
instances = []
def __init__(self):
A.instances.append(weakref.ref(self))
a, b, c = A(), A(), A()
instances = [ref() for ref in A.instances if ref() is not None]
Using weak-references allow the instances to be deallocated before the class.
See the weakref module for details on what it does.
Note that you may be able to use this technique even with classes that you didn't write. You simply have to monkey-patch the class.
For example:
def track_instances(cls):
def init(self, *args, **kwargs):
getattr(self, 'instances').append(weakref.ref(self))
getattr(self, '_old_init')(self, *args, **kwargs)
cls._old_init = cls.__init__
cls.__init__ = init
return cls
Then you can do:
track_instances(ExternalClass)
And all instances created after the execution of this statement will be found in ExternalClass.instances.
Depending on the class you may have to replace __new__ instead of __init__.
You can do this even without any special code in the class, simply using the garbage collector:
import gc
candidates = gc.get_referrers(cls_object)
instances = [candidate for candidate in candidates if isinstance(candidate, cls_object)]
And you can always obtain the class object since you can find it using object.__subclasses__ method:
cls_object = next(cls for cls in object.__subclasses__() if cls.__name__ == cls_name)
(assuming there is only a class with that name, otherwise you should try all of them)
However I cannot think of a situation where this is the right thing to do, so avoid this code in real applications.
I've done some testing and I believe that this solution may not work for built-in classes or classes defined in C extensions.
If you are in this case the last resort is to use gc.get_objects() to retrieve all tracked objects. However this will work only if the object support cyclic garbage collection, so there isn't a method that works in every possible situation.

Here the version getting the instances from memory, I wouldn't recommend using this in live code but it can be convenient for debugging:
import weakref
class SomeClass(object):
register = []
def __init__(self):
self.register.append(weakref.ref(self))
a = SomeClass()
b = SomeClass()
c = SomeClass()
# Now the magic :)
import gc
def get_instances(class_name):
# Get the objects from memory
for instance in gc.get_objects():
# Try and get the actual class
class_ = getattr(instance, '__class__', None)
# Only return if the class has the name we want
if class_ and getattr(class_, '__name__', None) == class_name:
yield instance
print list(get_instances('SomeClass'))

Python provides the types module that defined classes for built-in types and the locals() and globals() functions that return a list of local and global variables in the application.
One quick way to find objects by type is to do this.
import types
for varname, var_instance in locals().items():
if type(var_instance) == types.InstanceType and var_instance.__class__.__name__ == 'CLASS_NAME_YOU_ARE_LOOKING_FOR':
print "This instance was found:", varname, var_instance
It's worth going through the Python library documentation and read the docs for modules that work with the code directly. Some of which are inspect, gc, types, codeop, code, imp, ast. bdb, pdb. The IDLE source code is also very informative.

Instances are created within a namespace:
def some_function():
some_object = MyClass()
In this case, some_object is a name inside the "namespace" of the function that points at a MyClass instance. Once you leave the namespace (i.e., the function ends), Python's garbage collection cleans up the name and the instance.
If there would be some other location that also has a pointer to the object, the cleanup wouldn't happen.
So: no, there's no place where a list of instances is maintained.
It would be a different case where you to use a database with an ORM (object-relational mapper). In Django's ORM you can do MyClass.objects.all() if MyClass is a database object. Something to look into if you really need the functionality.
Update: See Bakuriu's answer. The garbage collector (which I mentioned) knows about all the instances :-) And he suggests the "weakref" module that prevents my won't-be-cleaned-up problem.

You cann get names for all the instances as they may not all have names, or the names they do have may be in scope. You may be able to get the instances.
If you are willing to keep track of the instances yourself, use a WeakSet:
import weakref
class SomeClass(object):
instances = weakref.WeakSet()
def __init__(self):
self.instances.add(self)
>>> instances = [SomeClass(), SomeClass(), SomeClass()]
>>> other = SomeClass()
>>> SomeClass.instances
<_weakrefset.WeakSet object at 0x0291F6F0>
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x028F0150>, <__main__.SomeClass object at 0x0291F210>]
Note that just deleting a name may not destroy the instance. other still exists until the garbage collected:
>>> del other
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x028F0150>, <__main__.SomeClass object at 0x0291F210>]
>>> import gc
>>> gc.collect()
0
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x0291F210>]
If you don't want to track them manually, then it is possible to use gc.get_objects() and filter out the instances you want, but that means you have to filter through all the objects in your program every time you do this. Even in the above example that means processing nearly 12,000 objects to find the 3 instances you want.
>>> [g for g in gc.get_objects() if isinstance(g, SomeClass)]
[<__main__.SomeClass object at 0x0291F210>, <__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>]

>>> class TestClass:
... pass
...
>>> foo = TestClass()
>>> for i in dir():
... if isinstance(eval(i), TestClass):
... print(i)
...
foo
>>>

Finally found a way to get through.
As I know the class name, I would search for the object created for that class in garbage collector(gc) like this...
for instance in gc.get_objects():
if str(type(instance)).find("dict") != -1:
for k in instance.keys():
if str(k).find("Sample") != -1:
return k
The above code returns an instance of the class which will be like this. Unfortunately,its in String format which doesn't suit the requirement. It should be of 'obj' type.
<mod_example.Sample object at 0x6f55250>
From the above value, parse the id(0x6f55250) and get the object reference based on the id.
obj_id = 0x6f55250
for obj in gc.get_objects():
# Converting decimal value to hex value
if id(obj) == ast.literal_eval(obj_id):
required_obj = obj
Hence required_obj will hold the object reference exactly in the 'obj' format.
:-)

Classes How I understand them. Correct me if Im wrong please

I really hope this is not a question posed by millions of newbies, but my search didn t really give me a satisfying answer.
So my question is fairly simple. Are classes basically a container for functions with its own namespace? What other functions do they have beside providing a separate namespace and holding functions while making them callable as class atributes? Im asking in a python context.
Oh and thanks for the great help most of you have been!

More importantly than functions, class instances hold data attributes, allowing you to define new data types beyond what is built into the language; and
they support inheritance and duck typing.
For example, here's a moderately useful class. Since Python files (created with open) don't remember their own name, let's make a file class that does.
class NamedFile(object):
def __init__(self, name):
self._f = f
self.name = name
def readline(self):
return self._f.readline()
Had Python not had classes, you'd probably be working with dicts instead:
def open_file(name):
return {"name": name, "f": open(name)}
Needless to say, calling myfile["f"].readline() all the time will cause your fingers to hurt at some point. You could of course introduce a function readline in a NamedFile module (namespace), but then you'd always have to use that exact function. By contrast, NamedFile instances can be used anywhere you need an object with a readline method, so it would be a plug-in replacement for file in many situation. That's called polymorphism, one of the biggest benefits of OO/class-based programming.
(Also, dict is a class, so using it violates the assumption that there are no classes :)

In most languages, classes are just pieces of code that describe how to produce an object. That's kinda true in Python too:
>>> class ObjectCreator(object):
... pass
...
>>> my_object = ObjectCreator()
>>> print my_object
<__main__.ObjectCreator object at 0x8974f2c>
But classes are more than that in Python. Classes are objects too.
Yes, objects.
As soon as you use the keyword class, Python executes it and creates an OBJECT. The instruction:
>>> class ObjectCreator(object):
... pass
...
creates in memory an object with the name ObjectCreator.
This object (the class) is itself capable of creating objects (the instances), and this is why it's a class.
But still, it's an object, and therefore:
you can assign it to a variable
you can copy it
you can add attributes to it
you can pass it as a function parameter
e.g.:
>>> print ObjectCreator # you can print a class because it's an object
<class '__main__.ObjectCreator'>
>>> def echo(o):
... print o
...
>>> echo(ObjectCreator) # you can pass a class as a parameter
<class '__main__.ObjectCreator'>
>>> print hasattr(ObjectCreator, 'new_attribute')
False
>>> ObjectCreator.new_attribute = 'foo' # you can add attributes to a class
>>> print hasattr(ObjectCreator, 'new_attribute')
True
>>> print ObjectCreator.new_attribute
foo
>>> ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable
>>> print ObjectCreatorMirror.new_attribute
foo
>>> print ObjectCreatorMirror()
<__main__.ObjectCreator object at 0x8997b4c>

Classes (or objects) are used to provide encapsulation of data and operations that can be performed on that data.
They don't provide namespacing in Python per se; module imports provide the same type of stuff and a module can be entirely functional rather than object oriented.
You might gain some benefit from looking at OOP With Python, Dive into Python, Chapter 5. Objects and Object Oriented Programming or even just the Wikipedia article on object oriented programming

A class is the definition of an object. In this sense, the class provides a namespace of sorts, but that is not the true purpose of a class. The true purpose is to define what the object will 'look like' - what the object is capable of doing (methods) and what it will know (properties).
Note that my answer is intended to provide a sense of understanding on a relatively non-technical level, which is what my initial trouble was with understanding classes. I'm sure there will be many other great answers to this question; I hope this one adds to your overall understanding.

Python: dereferencing weakproxy

Is there any way to get the original object from a weakproxy pointed to it? eg is there the inverse to weakref.proxy()?
A simplified example(python2.7):
import weakref
class C(object):
def __init__(self, other):
self.other = weakref.proxy(other)
class Other(object):
pass
others = [Other() for i in xrange(3)]
my_list = [C(others[i % len(others)]) for i in xrange(10)]
I need to get the list of unique other members from my_list. The way I prefer for such tasks
is to use set:
unique_others = {x.other for x in my_list}
Unfortunately this throws TypeError: unhashable type: 'weakproxy'
I have managed to solve the specific problem in an imperative way(slow and dirty):
unique_others = []
for x in my_list:
if x.other in unique_others:
continue
unique_others.append(x.other)
but the general problem noted in the caption is still active.
What if I have only my_list under control and others are burried in some lib and someone may delete them at any time, and I want to prevent the deletion by collecting nonweak refs in a list?
Or I may want to get the repr() of the object itself, not <weakproxy at xx to Other at xx>
I guess there should be something like weakref.unproxy I'm not aware about.

I know this is an old question but I was looking for an answer recently and came up with something. Like others said, there is no documented way to do it and looking at the implementation of weakproxy type confirms that there is no standard way to achieve this.
My solution uses the fact that all Python objects have a set of standard methods (like __repr__) and that bound method objects contain a reference to the instance (in __self__ attribute).
Therefore, by dereferencing the proxy to get the method object, we can get a strong reference to the proxied object from the method object.
Example:
>>> def func():
... pass
...
>>> weakfunc = weakref.proxy(func)
>>> f = weakfunc.__repr__.__self__
>>> f is func
True
Another nice thing is that it will work for strong references as well:
>>> func.__repr__.__self__ is func
True
So there's no need for type checks if either a proxy or a strong reference could be expected.
Edit:
I just noticed that this doesn't work for proxies of classes. This is not universal then.

Basically there is something like weakref.unproxy, but it's just named weakref.ref(x)().
The proxy object is only there for delegation and the implementation is rather shaky...
The == function doesn't work as you would expect it:
>>> weakref.proxy(object) == object
False
>>> weakref.proxy(object) == weakref.proxy(object)
True
>>> weakref.proxy(object).__eq__(object)
True
However, I see that you don't want to call weakref.ref objects all the time. A good working proxy with dereference support would be nice.
But at the moment, this is just not possible. If you look into python builtin source code you see, that you need something like PyWeakref_GetObject, but there is just no call to this method at all (And: it raises a PyErr_BadInternalCall if the argument is wrong, so it seems to be an internal function). PyWeakref_GET_OBJECT is used much more, but there is no method in weakref.py that could be able to do that.
So, sorry to disappoint you, but you weakref.proxy is just not what most people would want for their use cases. You can however make your own proxy implementation. It isn't to hard. Just use weakref.ref internally and override __getattr__, __repr__, etc.
On a little sidenote on how PyCharm is able to produce the normal repr output (Because you mentioned that in a comment):
>>> class A(): pass
>>> a = A()
>>> weakref.proxy(a)
<weakproxy at 0x7fcf7885d470 to A at 0x1410990>
>>> weakref.proxy(a).__repr__()
'<__main__.A object at 0x1410990>'
>>> type( weakref.proxy(a))
<type 'weakproxy'>
As you can see, calling the original __repr__ can really help!

weakref.ref is hashable whereas weakref.proxy is not. The API doesn't say anything about how you actually can get a handle on the object a proxy points to. with weakref, it's easy, you can just call it. As such, you can roll your own proxy-like class...Here's a very basic attemp:
import weakref
class C(object):
def __init__(self,obj):
self.object=weakref.ref(obj)
def __getattr__(self,key):
if(key == "object"): return object.__getattr__(self,"object")
elif(key == "__init__"): return object.__getattr__(self,"__init__")
else:
obj=object.__getattr__(self,"object")() #Dereference the weakref
return getattr(obj,key)
class Other(object):
pass
others = [Other() for i in range(3)]
my_list = [C(others[i % len(others)]) for i in range(10)]
unique_list = {x.object for x in my_list}
Of course, now unique_list contains refs, not proxys which is fundamentally different...

I know that this is an old question, but I've been bitten by it (so, there's no real 'unproxy' in the standard library) and wanted to share my solution...
The way I solved it to get the real instance was just creating a property which returned it (although I suggest using weakref.ref instead of a weakref.proxy as code should really check if it's still alive before accessing it instead of having to remember to catch an exception whenever any attribute is accessed).
Anyways, if you still must use a proxy, the code to get the real instance is:
import weakref
class MyClass(object):
#property
def real_self(self):
return self
instance = MyClass()
proxied = weakref.proxy(instance)
assert proxied.real_self is instance