I have a simple form in python + pylons that submits to a controller. However, each page load doesn't seem to be a fresh instantiation of the class. Rather, class variables specified on the previous page load are still accessible.
What's going on here? And what's the solution?
A common programmer oversight is that defining a list [] as a default argument or class initialiser is evaluated only once. If you have class variables such as lists, I recommend you initialise them in init. I'll give you an example.
>>> class Example(object):
... a = []
... def __init__(self):
... self.b = []
...
>>> foo = Example()
>>> bar = Example()
>>> foo.a
[]
>>> bar.a
[]
>>> foo.b
[]
>>> bar.b
[]
>>> foo.a.append(1)
>>> foo.b.append(2)
>>> foo.a
[1]
>>> foo.b
[2]
>>> bar.a
[1]
>>> bar.b
[]
Pylons uses a multi-threaded application server and variables are not cleared from request to request. This is a performance issue, as re-instantiating entire class trees would be expensive. Instead of storing the data returned by the user in a class, use a sessions system (Pylons comes with one or use something like Beaker) or back-end database like SQLAlchemy, SQLObject, or PyMongo.
Additionally, due to the multi-threaded nature of the framework, you should avoid shared objects (like globals) like the plague unless you are very careful to ensure you are using them in a thread-safe way (e.g. read-only). Certain Pylons-supplied objects (request/response) have been written to be thread-local, so don't worry about those.
Related
I am working on a small game server in Python using the class SimpleWebSocketServer found here. Everything works great, but the problem is each time I want to access a variable, I have to use self.variable_name. Let me give an example.
class SimpleEcho(WebSocket):
times_played_since_reset = 0
def handleMessage(self):
global times_played_since_reset
print times_played_since_reset
Whenever I try accessing times_played_since_reset using global it doesn't work and the server quits. Make it self.times_played_since_reset and everything works.
This variable needs affected by EVERY client connected. Unfortunately when I make it using self, only the client affects it's own instance. I need it to be where the client affects the class-wide variable instead of self.
You might want to consider using a mutable type for times_played_since_reset if you want it to be shared between all instances of the class.
Integers are not mutable, so they are not shared. As mentioned in the comments above, you could explicitly modify the class variable by doing something like SimpleEcho.times_played_since_reset += 1 however this only works as long as the instance attribute of the same name has not explicitly been set/modified.
For instance, take this example class:
class Foo(object):
bar = 1
If we create two instances:
>>> x = Foo()
>>> y = Foo()
Then:
>>> x.bar
1
>>> y.bar
1
And if we do:
>>> Foo.bar += 1
Then
>>> x.bar
2
>>> y.bar
2
But if we do:
>>> x.bar = 7
>>> Foo.bar +=1
Then:
>>> x.bar
7
>>> y.bar
3
If instead you were to use a mutable type like a list, For example like:
class Foo(object):
bar = [1]
whether you modify Foo.bar[0] or <instance>.bar.[0] all current and future instances would see the change. This is because they all reference the same list, and you have modified the contents of the list rather than changing the specific object the variable points to.
However, if you were to assign a new list via <instance>.bar = [78], only that instance would see the change and all other instances (current and future) would still reference the original list that was defined in the class definition.
To access global variables from python server, use:
(Class Name).variable
SimpleEcho.times_played_since_reset
Someone asked a similar one [question]:Printing all instances of a class.
While I am less concerned about printing them, I'd rather to know how many instances are currently "live".
The reason for this instance capture is more like a setting up a scheduled job, every hour check these "live" unprocessed instances and enrich the data. After that, either a flag in this instance is set or just delete this instance.
Torsten Marek 's answer in [question]:Printing all instances of a class using weakrefs need a call to the base class constructor for every class of this type, is it possible to automate this? Or we can get all instances with some other methods?
You can either track it on your own (see the other answers) or ask the garbage collector:
import gc
class Foo(object):
pass
foo1, foo2 = Foo(), Foo()
foocount = sum(1 for o in gc.get_referrers(Foo) if o.__class__ is Foo)
This can be kinda slow if you have a lot of objects, but it's generally not too bad, and it has the advantage of being something you can easily use with someone else's code.
Note: Used o.__class__ rather than type(o) so it works with old-style classes.
If you only want this to work for CPython, and your definition of "live" can be a little lax, there's another way to do this that may be useful for debugging/introspection purposes:
>>> import gc
>>> class Foo(object): pass
>>> spam, eggs = Foo(), Foo()
>>> foos = [obj for obj in gc.get_objects() if isinstance(obj, Foo)]
>>> foos
[<__main__.Foo at 0x1153f0190>, <__main__.Foo at 0x1153f0210>]
>>> del spam
>>> foos = [obj for obj in gc.get_objects() if isinstance(obj, Foo)]
>>> foos
[<__main__.Foo at 0x1153f0190>, <__main__.Foo at 0x1153f0210>]
>>> del foos
>>> foos = [obj for obj in gc.get_objects() if isinstance(obj, Foo)]
>>> foos
[<__main__.Foo at 0x1153f0190>]
Note that deleting spam didn't actually make it non-live, because we've still got a reference to the same object in foos. And reassigning foos didn't not help, because apparently the call to get_objects happened before the old version is released. But eventually it went away once we stopped referring to it.
And the only way around this problem is to use weakrefs.
Of course this will be horribly slow in a large system, with or without weakrefs.
Sure, store the count in a class attribute:
class CountedMixin(object):
count = 0
def __init__(self, *args, **kwargs):
type(self).count += 1
super().__init__(*args, **kwargs)
def __del__(self):
type(self).count -= 1
try:
super().__del__()
except AttributeError:
pass
You could make this slightly more magical with a decorator or a metaclass than with a base class, or simpler if it can be a bit less general (I've attempted to make this fit in anywhere in any reasonable multiple-inheritance hierarchy, which you usually don't need to worry about…), but basically, this is all there is to it.
If you want to have the instances themselves (or, better, weakrefs to them), rather than just a count of them, just replace count=0 with instances=set(), then do instances.add(self) instead of count += 1, etc. (Again, though, you probably want a weakref to self, rather than self.)
I cannot comment to the answer of kindall, thus I write my comment as answer:
The solution with gc.get_referrers(<ClassName>) does not work with inherited classes in python 3. The method gc.get_referrers(<ClassName>) does not return any instances of a class that was inherited from <ClassName>.
Instead you need to use gc.get_objects() which is much slower, since it returns a full list of objects. But in case of unit-tests, where you simply want to ensure your objects get deleted after the test (no circular references) it should be sufficient and fast enough.
Also do not forget to call gc.collect() before checking the number of your instances, to ensure all unreferenced instances are really deleted.
I also saw an issue with weak references which are also counted in this way. The problem with weak references is, that the object which is referenced might not exist any more, thus isinstance(Instance, Class) might fail with an error about non existing weak references.
Here is a simple code example:
import gc
def getInstances(Class):
gc.collect()
Number = 0
InstanceList = gc.get_objects()
for Instance in InstanceList:
if 'weakproxy' not in str(type(Instance)): # avoid weak references
if isinstance(Instance, Class):
Number += 1
return Number
Is there any way to get the object name when the class name is known. If there are multiple objects for a class they also need to be printed.
Class A():
pass
Assume that some one have created objects for class A in some other files. So, I want to look all instances of 'Class A'
If you are the one creating the class you can simply store weak-references when instantiating the class:
import weakref
class A(object):
instances = []
def __init__(self):
A.instances.append(weakref.ref(self))
a, b, c = A(), A(), A()
instances = [ref() for ref in A.instances if ref() is not None]
Using weak-references allow the instances to be deallocated before the class.
See the weakref module for details on what it does.
Note that you may be able to use this technique even with classes that you didn't write. You simply have to monkey-patch the class.
For example:
def track_instances(cls):
def init(self, *args, **kwargs):
getattr(self, 'instances').append(weakref.ref(self))
getattr(self, '_old_init')(self, *args, **kwargs)
cls._old_init = cls.__init__
cls.__init__ = init
return cls
Then you can do:
track_instances(ExternalClass)
And all instances created after the execution of this statement will be found in ExternalClass.instances.
Depending on the class you may have to replace __new__ instead of __init__.
You can do this even without any special code in the class, simply using the garbage collector:
import gc
candidates = gc.get_referrers(cls_object)
instances = [candidate for candidate in candidates if isinstance(candidate, cls_object)]
And you can always obtain the class object since you can find it using object.__subclasses__ method:
cls_object = next(cls for cls in object.__subclasses__() if cls.__name__ == cls_name)
(assuming there is only a class with that name, otherwise you should try all of them)
However I cannot think of a situation where this is the right thing to do, so avoid this code in real applications.
I've done some testing and I believe that this solution may not work for built-in classes or classes defined in C extensions.
If you are in this case the last resort is to use gc.get_objects() to retrieve all tracked objects. However this will work only if the object support cyclic garbage collection, so there isn't a method that works in every possible situation.
Here the version getting the instances from memory, I wouldn't recommend using this in live code but it can be convenient for debugging:
import weakref
class SomeClass(object):
register = []
def __init__(self):
self.register.append(weakref.ref(self))
a = SomeClass()
b = SomeClass()
c = SomeClass()
# Now the magic :)
import gc
def get_instances(class_name):
# Get the objects from memory
for instance in gc.get_objects():
# Try and get the actual class
class_ = getattr(instance, '__class__', None)
# Only return if the class has the name we want
if class_ and getattr(class_, '__name__', None) == class_name:
yield instance
print list(get_instances('SomeClass'))
Python provides the types module that defined classes for built-in types and the locals() and globals() functions that return a list of local and global variables in the application.
One quick way to find objects by type is to do this.
import types
for varname, var_instance in locals().items():
if type(var_instance) == types.InstanceType and var_instance.__class__.__name__ == 'CLASS_NAME_YOU_ARE_LOOKING_FOR':
print "This instance was found:", varname, var_instance
It's worth going through the Python library documentation and read the docs for modules that work with the code directly. Some of which are inspect, gc, types, codeop, code, imp, ast. bdb, pdb. The IDLE source code is also very informative.
Instances are created within a namespace:
def some_function():
some_object = MyClass()
In this case, some_object is a name inside the "namespace" of the function that points at a MyClass instance. Once you leave the namespace (i.e., the function ends), Python's garbage collection cleans up the name and the instance.
If there would be some other location that also has a pointer to the object, the cleanup wouldn't happen.
So: no, there's no place where a list of instances is maintained.
It would be a different case where you to use a database with an ORM (object-relational mapper). In Django's ORM you can do MyClass.objects.all() if MyClass is a database object. Something to look into if you really need the functionality.
Update: See Bakuriu's answer. The garbage collector (which I mentioned) knows about all the instances :-) And he suggests the "weakref" module that prevents my won't-be-cleaned-up problem.
You cann get names for all the instances as they may not all have names, or the names they do have may be in scope. You may be able to get the instances.
If you are willing to keep track of the instances yourself, use a WeakSet:
import weakref
class SomeClass(object):
instances = weakref.WeakSet()
def __init__(self):
self.instances.add(self)
>>> instances = [SomeClass(), SomeClass(), SomeClass()]
>>> other = SomeClass()
>>> SomeClass.instances
<_weakrefset.WeakSet object at 0x0291F6F0>
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x028F0150>, <__main__.SomeClass object at 0x0291F210>]
Note that just deleting a name may not destroy the instance. other still exists until the garbage collected:
>>> del other
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x028F0150>, <__main__.SomeClass object at 0x0291F210>]
>>> import gc
>>> gc.collect()
0
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x0291F210>]
If you don't want to track them manually, then it is possible to use gc.get_objects() and filter out the instances you want, but that means you have to filter through all the objects in your program every time you do this. Even in the above example that means processing nearly 12,000 objects to find the 3 instances you want.
>>> [g for g in gc.get_objects() if isinstance(g, SomeClass)]
[<__main__.SomeClass object at 0x0291F210>, <__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>]
>>> class TestClass:
... pass
...
>>> foo = TestClass()
>>> for i in dir():
... if isinstance(eval(i), TestClass):
... print(i)
...
foo
>>>
Finally found a way to get through.
As I know the class name, I would search for the object created for that class in garbage collector(gc) like this...
for instance in gc.get_objects():
if str(type(instance)).find("dict") != -1:
for k in instance.keys():
if str(k).find("Sample") != -1:
return k
The above code returns an instance of the class which will be like this. Unfortunately,its in String format which doesn't suit the requirement. It should be of 'obj' type.
<mod_example.Sample object at 0x6f55250>
From the above value, parse the id(0x6f55250) and get the object reference based on the id.
obj_id = 0x6f55250
for obj in gc.get_objects():
# Converting decimal value to hex value
if id(obj) == ast.literal_eval(obj_id):
required_obj = obj
Hence required_obj will hold the object reference exactly in the 'obj' format.
:-)
I'm trying to use pickle to save a custom class; something very much like the code below (though with a few methods defined on the class, and several more dicts and such for data). However, often when I run this, pickle and then unpickle, I lose whatever data was in the class, and its as if I created a new blank instance.
import pickle
class MyClass:
VERSION = 1
some_data = {}
more_data = set()
def save(self,filename):
with open(filename, 'wb') as f:
p = pickle.Pickler(f)
p.dump(self)
def load(filename):
with open(filename,'rb') as ifile:
u = pickle.Unpickler(ifile)
obj = u.load()
return obj
I was wondering if this had something to do with the memo of the pickle class, but I don't feel like it should. When it doesn't work, I look at my generated file and it looks something like this: (Obviously not meant to be readable, but it obviously contains no data)
€c__main__
MyClass
q
Anyways, I hope this is enough for someone to understand what might possibly be going on here, or what to look at.
The problem you're having is that you're using mutable class variables to hold your data, rather than putting the data into instance variables.
The pickle module only saves the data stored directly on the instance, not class variables that can also be accessed via self. When you're finding your unpickled instance have no data, what that probably means is that the class doesn't hold the data from the previous run, so the instances can't access it any more.
Using class variables that way will probably cause you other problems too, as the data will be shared by all instances of the class! Here's a Python console session code that illustrates the issue:
>>> class Foo(object):
class_var = []
def __init__(self, value):
self.class_var.append(value)
>>> f1 = Foo(1)
>>> f1.class_var
[1]
>>> f2 = Foo(2)
>>> f2.class_var
[1, 2]
That's probably not what you wanted. But it gets worse!
>>> f1.class_var
[1, 2]
The data you thought had belonged to f1 has been changed by the creation of f2. In fact, f1.class_var is the very same object as f2.class_var (it is also available via Foo.class_var directly, without going through any instances at all).
So, using a class variable is almost certainly not what you want. Instead, write an __init__ method for the class that creates a new value and saves it as an instance variable:
>>> class Bar(object):
def __init__(self, value):
self.instance_var = [] # creates a separate list for each instance!
self.instance_var.append(value)
>>> b1 = Bar(1)
>>> b1.instance_var
[1]
>>> b2 = Bar(2)
>>> b2.instance_var # doesn't include value from b1
[2]
>>> b1.instance_var # b1's data is unchanged
[1]
Pickle will handle this class as you expect. All of its data is in the instances, so you should never end up with an empty instance when you unpickle.
Is there any way to get the original object from a weakproxy pointed to it? eg is there the inverse to weakref.proxy()?
A simplified example(python2.7):
import weakref
class C(object):
def __init__(self, other):
self.other = weakref.proxy(other)
class Other(object):
pass
others = [Other() for i in xrange(3)]
my_list = [C(others[i % len(others)]) for i in xrange(10)]
I need to get the list of unique other members from my_list. The way I prefer for such tasks
is to use set:
unique_others = {x.other for x in my_list}
Unfortunately this throws TypeError: unhashable type: 'weakproxy'
I have managed to solve the specific problem in an imperative way(slow and dirty):
unique_others = []
for x in my_list:
if x.other in unique_others:
continue
unique_others.append(x.other)
but the general problem noted in the caption is still active.
What if I have only my_list under control and others are burried in some lib and someone may delete them at any time, and I want to prevent the deletion by collecting nonweak refs in a list?
Or I may want to get the repr() of the object itself, not <weakproxy at xx to Other at xx>
I guess there should be something like weakref.unproxy I'm not aware about.
I know this is an old question but I was looking for an answer recently and came up with something. Like others said, there is no documented way to do it and looking at the implementation of weakproxy type confirms that there is no standard way to achieve this.
My solution uses the fact that all Python objects have a set of standard methods (like __repr__) and that bound method objects contain a reference to the instance (in __self__ attribute).
Therefore, by dereferencing the proxy to get the method object, we can get a strong reference to the proxied object from the method object.
Example:
>>> def func():
... pass
...
>>> weakfunc = weakref.proxy(func)
>>> f = weakfunc.__repr__.__self__
>>> f is func
True
Another nice thing is that it will work for strong references as well:
>>> func.__repr__.__self__ is func
True
So there's no need for type checks if either a proxy or a strong reference could be expected.
Edit:
I just noticed that this doesn't work for proxies of classes. This is not universal then.
Basically there is something like weakref.unproxy, but it's just named weakref.ref(x)().
The proxy object is only there for delegation and the implementation is rather shaky...
The == function doesn't work as you would expect it:
>>> weakref.proxy(object) == object
False
>>> weakref.proxy(object) == weakref.proxy(object)
True
>>> weakref.proxy(object).__eq__(object)
True
However, I see that you don't want to call weakref.ref objects all the time. A good working proxy with dereference support would be nice.
But at the moment, this is just not possible. If you look into python builtin source code you see, that you need something like PyWeakref_GetObject, but there is just no call to this method at all (And: it raises a PyErr_BadInternalCall if the argument is wrong, so it seems to be an internal function). PyWeakref_GET_OBJECT is used much more, but there is no method in weakref.py that could be able to do that.
So, sorry to disappoint you, but you weakref.proxy is just not what most people would want for their use cases. You can however make your own proxy implementation. It isn't to hard. Just use weakref.ref internally and override __getattr__, __repr__, etc.
On a little sidenote on how PyCharm is able to produce the normal repr output (Because you mentioned that in a comment):
>>> class A(): pass
>>> a = A()
>>> weakref.proxy(a)
<weakproxy at 0x7fcf7885d470 to A at 0x1410990>
>>> weakref.proxy(a).__repr__()
'<__main__.A object at 0x1410990>'
>>> type( weakref.proxy(a))
<type 'weakproxy'>
As you can see, calling the original __repr__ can really help!
weakref.ref is hashable whereas weakref.proxy is not. The API doesn't say anything about how you actually can get a handle on the object a proxy points to. with weakref, it's easy, you can just call it. As such, you can roll your own proxy-like class...Here's a very basic attemp:
import weakref
class C(object):
def __init__(self,obj):
self.object=weakref.ref(obj)
def __getattr__(self,key):
if(key == "object"): return object.__getattr__(self,"object")
elif(key == "__init__"): return object.__getattr__(self,"__init__")
else:
obj=object.__getattr__(self,"object")() #Dereference the weakref
return getattr(obj,key)
class Other(object):
pass
others = [Other() for i in range(3)]
my_list = [C(others[i % len(others)]) for i in range(10)]
unique_list = {x.object for x in my_list}
Of course, now unique_list contains refs, not proxys which is fundamentally different...
I know that this is an old question, but I've been bitten by it (so, there's no real 'unproxy' in the standard library) and wanted to share my solution...
The way I solved it to get the real instance was just creating a property which returned it (although I suggest using weakref.ref instead of a weakref.proxy as code should really check if it's still alive before accessing it instead of having to remember to catch an exception whenever any attribute is accessed).
Anyways, if you still must use a proxy, the code to get the real instance is:
import weakref
class MyClass(object):
#property
def real_self(self):
return self
instance = MyClass()
proxied = weakref.proxy(instance)
assert proxied.real_self is instance