We Keep Coding

Multiprocessing proxy: let getters return proxies themselves

I have a complex unpickable object that has properties (defined via getters and setters) that are of complex and unpickable type as well. I want to create a multiprocessing proxy for the object to execute some tasks in parallel.
The problem: While I have succeeded to make the getter methods available for the proxy object, I fail to make the getters return proxies for the unpickable return objects.
My setup resembles the following:
from multiprocessing.managers import BaseManager, NamespaceProxy
class A():
#property
def a(self):
return B()
#property
def b(self):
return 2
# unpickable class
class B():
def __init__(self, *args):
self.f = lambda: 1
class ProxyBase(NamespaceProxy):
_exposed_ = ('__getattribute__', '__setattr__', '__delattr__')
class AProxy(ProxyBase): pass
class BProxy(ProxyBase): pass
class MyManager(BaseManager):pass
MyManager.register('A', A, AProxy)
if __name__ == '__main__':
with MyManager() as manager:
myA = manager.A()
print(myA.b) # works great
print(myA.a) # raises error, because the object B is not pickable
I know that I can specify the result type of a method when registering it with the manager. That is, I can do
MyManager.register('A', A, AProxy, method_to_typeid={'__getattribute__':'B'})
MyManager.register('B', B, BProxy)
if __name__ == '__main__':
with MyManager() as manager:
myA = manager.A()
print(myA.a) # works great!
print(myA.b) # returns the same as myA.a ?!
It is clear to me that my solution does not work since the __getattr__ method applies to all properties, whereas I only want it to return a proxy for B when property a is accessed. How could I achieve this?
As a side question: if I remove the *args argument from the __init__ method of B, I get an error that it is called with the wrong number of arguments. Why? How could I resolve this?

I don't this is possible without some hacks, since the choice to return a value or proxy is made based on the method name alone, and not the type of the return value (from Server.serve_client):
try:
res = function(*args, **kwds)
except Exception as e:
msg = ('#ERROR', e)
else:
typeid = gettypeid and gettypeid.get(methodname, None)
if typeid:
rident, rexposed = self.create(conn, typeid, res)
token = Token(typeid, self.address, rident)
msg = ('#PROXY', (rexposed, token))
else:
msg = ('#RETURN', res)
Also keep in mind exposing __getattribute__ in an unpickable class's proxy basically breaks the proxy functionality when calling methods.
But if you're willing to hack it and just need attribute access, here is a working solution (note calling myA.a.f() still won't work, the lambda is an attribute and is not proxied, only methods are, but that's a different problem).
import os
from multiprocessing.managers import BaseManager, NamespaceProxy, Server
class A():
#property
def a(self):
return B()
#property
def b(self):
return 2
# unpickable class
class B():
def __init__(self, *args):
self.f = lambda: 1
self.pid = os.getpid()
class HackedObj:
def __init__(self, obj, gettypeid):
self.obj = obj
self.gettypeid = gettypeid
def __getattribute__(self, attr):
if attr == '__getattribute__':
return object.__getattribute__(self, attr)
obj = object.__getattribute__(self, 'obj')
result = object.__getattribute__(obj, attr)
if isinstance(result, B):
gettypeid = object.__getattribute__(self, 'gettypeid')
# This tells the server that the return value of this method is
# B, for which we've registered a proxy.
gettypeid['__getattribute__'] = 'B'
return result
class HackedDict:
def __init__(self, data):
self.data = data
def __setitem__(self, key, value):
self.data[key] = value
def __getitem__(self, key):
obj, exposed, gettypeid = self.data[key]
if isinstance(obj, A):
gettypeid = gettypeid.copy() if gettypeid else {}
# Now we need getattr to update gettypeid based on the result
# luckily BaseManager queries the typeid info after the function
# has been invoked
obj = HackedObj(obj, gettypeid)
return (obj, exposed, gettypeid)
class HackedServer(Server):
def __init__(self, registry, address, authkey, serializer):
super().__init__(registry, address, authkey, serializer)
self.id_to_obj = HackedDict(self.id_to_obj)
class MyManager(BaseManager):
_Server = HackedServer
class ProxyBase(NamespaceProxy):
_exposed_ = ('__getattribute__', '__setattr__', '__delattr__')
class AProxy(ProxyBase): pass
class BProxy(ProxyBase): pass
MyManager.register('A', callable=A, proxytype=AProxy)
MyManager.register('B', callable=B, proxytype=BProxy)
if __name__ == '__main__':
print("This process: ", os.getpid())
with MyManager() as manager:
myB = manager.B()
print("Proxy process, using B directly: ", myB.pid)
myA = manager.A()
print('myA.b', myA.b)
print("Proxy process, via A: ", myA.a.pid)
The key to the solution is to replace the _Server in our manager, and then wrap the id_to_obj dict with the one that performs the hack for the specific method we need.
The hack consists on populating the gettypeid dict for the method, but only after it has been evaluated and we know the return type to be one that we would need a proxy for. And we're lucky in the order of evaluations, gettypeid is accessed after the method has been called.
Also luckily gettypeid is used as a local in the serve_client method, so we can return a copy of it and modify it and we don't introduce any concurrency issues.
While this was a fun exercise, I have to say I really advise against this solution, if you're dealing with external code that you cannot modify, you should simply create your own wrapper class that has explicit methods instead of #property accessors, proxy your own class instead, and use method_to_typeid.

Can't grasp : making a simple singleton in python (for a mongodb manager)

I have read the various main examples of singletons but I have trouble making what I want. My main inspiration here is this example :
class OnlyOne:
class __OnlyOne:
def __init__(self, arg):
self.val = arg
def __str__(self):
return repr(self) + self.val
instance = None
def __init__(self, arg):
if not OnlyOne.instance:
OnlyOne.instance = OnlyOne.__OnlyOne(arg)
else:
OnlyOne.instance.val = arg
def __getattr__(self, name):
return getattr(self.instance, name)
But I don't need anything like getattr, here is my code so far:
class MongoManager:
class __MongoManager:
def __init__(self):
# Initialise client
self.client = pymongo.MongoClient('localhost', 27017)
__instance = None
def __init__(self):
if not MongoManager.__instance:
MongoManager.__instance = MongoManager.__MongoManager()
All I want is to get the variable client when I type MongoManager.client, but I get the error AttributeError: 'MongoManager' object has no attribute 'client'.
I guess I could try with getattr and check something like if arg == "client" but it feels dirty, I'm sure there's a proper way to do that.

Try this:
class MongoManager:
__instance = None
#staticmethod
def getInstance():
if MongoManager.__instance == None:
MongoManager()
return MongoManager.__instance
def __init__(self):
if MongoManager.__instance != None:
raise Exception("This class is a singleton!")
else:
MongoManager.__instance = pymongo.MongoClient('localhost', 27017)
You can get instance of this class by calling MongoManager.getInstance().
What we are doing here? We have a class variable __instance and when object of the class is created, we are initializing it. Now, when we call getInstance() this instance which is stored in __instance will get passed instead of creating new instance.

I managed to do exactly what I wanted. I didn't completely understand how __getattr__ worked so a bit more reading was necessary.
class MongoManager:
class __MongoManager:
def __init__(self):
# Initialise mongo client
self.client = pymongo.MongoClient('localhost', 27017)
__instance = None
def __init__(self):
if not MongoManager.__instance:
MongoManager.__instance = MongoManager.__MongoManager()
def __getattr__(self, item):
return getattr(self.__instance, item)
Now whenever I call MongoManager().client, I get the desired client.

Inheritance issue when using super() (Python)

below is two classes showing inheritance for an API I am working on. So I want the base class, ServiceTemplateTest, to have a common set of properties for all services and to behave like an OrderedDict object. So the base class inherirts from OrderedDict. I then do a super() in the __init__ in order to clear up the MRO issue. Now when I actually use this base class, I am getting issues when I try to class __init() from the base class. According to my debugger it's saying I need to call: self._ServiceTemplateTest__init(), but shouldn't it just be __init() since I call super()? What is the proper way to allow me to inherit without have to do this call: self._ServiceTemplateTest__init()?
Do I need to create an __init__() on the non-base classes where I have multiple super() calls? If so, what super class should come first?
Thank you for any advice you can provide!
from collections import OrderedDict
import urllib2, json, urllib
class ServiceTemplateTest(OrderedDict):
_con = None
_url = None
def __init__(self, url, connection=None, initialize=False, **kwargs):
super(ServiceTemplateTest, self).__init__()
self._url = url
self._con = connection
if initialize:
self.__init(connection)
def __init(self, connection=None):
if connection is None:
connection = self._con
attributes = [attr for attr in dir(self)
if not attr.startswith('__') and \
not attr.startswith('_')]
params = {"f":"json"}
params = urllib.urlencode(params)
result = json.loads(
urllib2.urlopen(url="{url}?{params}".format(url=self._url,
params=params)).read())
for k,v in result.items():
if k in attributes:
setattr(self, "_"+ k, v)
self[k] = v
else:
self[k] = v
self.__dict__.update(result)
#----------------------------------------------------------------------
#property
def connection(self):
return self._con
#----------------------------------------------------------------------
#connection.setter
def connection(self, value):
self._con = value
self.refresh()
#----------------------------------------------------------------------
#property
def url(self):
return self._url
#----------------------------------------------------------------------
#url.setter
def url(self, value):
""""""
self._url = value
self.refresh()
#----------------------------------------------------------------------
def __str__(self):
return json.dumps(self)
#----------------------------------------------------------------------
def __repr__(self):
return self.__str__()
#----------------------------------------------------------------------
def refresh(self):
self.__init()
class SchematicService(ServiceTemplateTest):
"""schematic service"""
_con = None
_json_dict = None
_json = None
_url = None
_nbSchematicLayers = None
_nbTemplates = None
_type = None
_name = None
_nbEstimatedDiagrams = None
def __init__(self, url, connection=None, initialize=False, **kwargs):
super(SchematicService, self).__init__(url=url, connection=connection,
initialize=initialize, **kwargs)
self._url = url
self._con = connection
if initialize:
self.__init(connection)
#----------------------------------------------------------------------
#property
def nbSchematicLayers(self):
if self._nbSchematicLayers is None:
self.__init()
return self._nbSchematicLayers
#----------------------------------------------------------------------
#property
def nbTemplates (self):
if self._nbTemplates is None:
self.__init()
return self._nbTemplates
#----------------------------------------------------------------------
#property
def type(self):
if self._type is None:
self.__init()
return self._type
#----------------------------------------------------------------------
#property
def name(self):
if self._name is None:
self.__init()
return self._name
#----------------------------------------------------------------------
#property
def nbEstimatedDiagrams(self):
if self._nbEstimatedDiagrams is None:
self.__init()
return self._nbEstimatedDiagrams
#property
def somerandompropertytest(self):
return "hi"
if __name__ == "__main__":
url = "http://servicesbeta6.esri.com/arcgis/rest/services/S1_Schematics/MapServer"
s = SchematicService(url=url, initialize=True)
print s

The issue isn't inheritance or super(), but that you're trying to call a "private" method from outside of the class. Any method whose name begins with two underscores -- your __init() in this case -- are private to the class they're defined in.
Python doesn't really have "private" in the sense you might be familiar with from other OO languages, instead it does something called name mangling to make it inconvenient, rather than impossible. In essence, if you name a method like __init(), Python will convert that to a method named _NameOfClass__init(), and will do the same in calls (or access to attributes) with similar naming. The trick is, the "NameOfClass" part is always the name of the class you're accessing the method from -- the subclass, SchematicService, in your case. Since the names don't match, Python can't find the method.
Of course, in Python nothing is actually really private. You can access the private method by mangling its name yourself, if you want. The conventional wisdom, though, is generally not to use double-underscore private methods or attributes at all. By convention, if you want to have a method on a base class that is not meant to be called from outside of the base class (eg because it isn't part of the public API you want to support), name it with a single leading underscore. Pythonistas know that this means, "the signature, purpose, or even presence of this method or attribute might go away at a later date, and I should not rely on it".
My $0.02: If you intend the method to be callable from anywhere -- both in subclasses and other, unrelated code -- make it a regular public method (no leading underscores in the name); if you only want it accessible to subclasses, use a single leading underscore.

dcrosta already answered (mainly the same thing as I was about to post) on the problem with calling __init from a subclass. I just wanted to add that in your code example, the whole SchematicService.__init__() is just useless, as it will only redo what ServiceTemplateTest.__init__() already done.
Also, having both class attributes and instance attributes with the same names doesn't help wrt/ readability / maintainability. If these are intended to be defaults for not-yet-set instance attributes, it's better to set them as instance attributes in the __init__().

Python : use a class methods as static , when its implemented as instance methods

I have a big class which has a lot of functions and attributes.
the instances are created from data in a remote database.
the process of creating each instance is very long and heavy.
In performance sake ive created a bunch class from this heavy class.
so accessing the attributed is easy and works great .
the problem is how to use the methods from that class.
ex :
class clsA():
def __init__(self,obj):
self.attrA=obj.attrA
def someFunc(self):
print self
class bunchClsA(bunch):
def __getattr__(self, attr):
# this is the problem:
try:
#try and return a func
func = clsA.attr
return func
except:
# return simple attribute
return self.attr
Clearly this dosent work , Is there a way i could access the instance function staticly and override the "self" var ?

Found out a nice solution to the problem :
from bunch import Bunch
import types
#Original class:
class A():
y=6
def __init__(self,num):
self.x=num
def funcA(self):
print self.x
#class that wraps A using Bunch(thats what i needed .. u can use another):
class B(Bunch):
def __init__(self, data, cls):
self._cls = cls # notice, not an instance just the class it self
super(B, self).__init__(data)
def __getattr__(self, attr):
# Handles normal Bunch, dict attributes
if attr in self.keys():
return self[attr]
else:
res = getattr(self._cls, attr)
if isinstance(res, types.MethodType):
# returns the class func with self overriden
return types.MethodType(res.im_func, self, type(self))
else:
# returns class attributes like y
return res
data = {'x': 3}
ins_b = B(data, A)
print ins_b.funcA() # returns 3
print ins_b.y # returns 6
And this solves my issue, its a hack and if you have the privileges, redesign the code.

Overriding __getattr__ to support dynamic nested attributes

What is the best approach to take if you want to dynamically create and reference nested attributes?
I was writing a simple Flickr client, and wanted to match the documented API as closely as possible, without actually defining every method. For instance, to make a request to Flickr's flickr.people.getInfo API method:
flickr = Client()
data = flickr.people.getInfo(user_id='xxx')
In this case flickr.people.getInfo directly maps to the corresponding method in their API documentation. When called, people and getInfo are created as they are looked up, then the proper request to make is determined by the path to getInfo, which is people.getInfo. This is the approach I used:
class Attr(object):
def __init__(self, client, name, parent):
self._client = client
self._name = name
self._parent = parent
def __getattr__(self, name):
attr = Attr(self._client, name, self)
setattr(self, name, attr)
return attr
def _get_path(self, path=None):
if path:
path = '.'.join((self._name, path))
else:
path = self._name
if isinstance(self._parent, Attr):
return self._parent._get_path(path)
return path
def __call__(self, *args, **kwargs):
return self._client.execute_method(self._get_path(), *args, **kwargs)
class Client(object):
def __getattr__(self, name):
attr = Attr(self, name, None)
setattr(self, name, attr)
return attr
def execute_method(self, method, *args, **kwargs):
print method, args, kwargs
This works, but I'm curious if my approach to deal with nested attribute assignment/lookup can be improved, or if there are any errors lurking in wait, unbeknownst to me. In particular, I'm curious if there is a better way to figure out the "path" to a given attribute. For example, if I call Client().x.y.z(), x, y, z do not exist, and will be created one by one (as __getattr__ looks up a single attribute at a time). By the time z is called, I need to be able to discern that the path to z is x.y.z.

Thanks to Thomas K for pointing out that flipy already does this (and seems like a nice library for interacting with flickr). A cleaner approach:
class Method(object):
def __init__(self, client, method_name):
self.client = client
self.method_name = method_name
def __getattr__(self, key):
return Method(self.client, '.'.join((self.method_name, key)))
def __call__(self, **kwargs):
print self.method_name, kwargs
class Client(object):
def __getattr__(self, key):
return Method(self, key)
Et voilà:
>>> c = Client()
>>> c.some.method(x=1, y=2)
some.method {'y': 2, 'x': 1}