I have a problem with overriding the init method of the thread class.
I only need it to take a socket that i pass to the init method when I instantiate the thread object.
The code of the thread class is:
class client_handle(threading.Thread):
conne = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def __init__(self, conne=None):
threading.Thread.__init__(self, conne=conne)
When i create a client_handle object i write:
socket_conn, addr = s.accept()
client_thread = client_handle(socket_conn)
I also searched in other posts but i couldn't solve my problem.
You need to assign conne manually as attribute of the instance. Passing it as argument to Thread.__init__ won't work (would be set as the group-argument of the Thread-class).
from threading import Thread
class ClientHandle(Thread):
def __init__(self, conne=None):
super().__init__() # Python 3
self.conne = conne
Thread.__init__ doesn't have a parameter named conne, so you have to pass the value as a positional argument instead.
def __init__(self, conne=None):
threading.Thread.__init__(self, conne)
Incidentally, the class attribute client_handle.conne doesn't seem to serve any purpose; you can probably get rid of it.
Update: passing a socket to Thread.__init__ in particular is clearly wrong, but this applies to any attempt to "invent" a parameter name like conne for a method that doesn't have a parameter named conne.
I do not know why, but I get this strange error whenever I try to pass to the method of a shared object shared custom class object. Python version: 3.6.3
Code:
from multiprocessing.managers import SyncManager
class MyManager(SyncManager): pass
class MyClass: pass
class Wrapper:
def set(self, ent):
self.ent = ent
MyManager.register('MyClass', MyClass)
MyManager.register('Wrapper', Wrapper)
if __name__ == '__main__':
manager = MyManager()
manager.start()
try:
obj = manager.MyClass()
lst = manager.list([1,2,3])
collection = manager.Wrapper()
collection.set(lst) # executed fine
collection.set(obj) # raises error
except Exception as e:
raise
Error:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "D:\Program Files\Python363\lib\multiprocessing\managers.py", line 228, in serve_client
request = recv()
File "D:\Program Files\Python363\lib\multiprocessing\connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "D:\Program Files\Python363\lib\multiprocessing\managers.py", line 881, in RebuildProxy
return func(token, serializer, incref=incref, **kwds)
TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
---------------------------------------------------------------------------
What's the problem here?
I ran into this too, as noted, this is a bug in Python multiprocessing (see issue #30256) and the pull request that corrects this has not yet been merged. The pull request has since been superseded by another PR that makes the same change but adds a test as well.
Apart from manually patching your local installation, you have three other options:
you could use the MakeProxyType() callable to specify your proxytype, without relying on the AutoProxy proxy generator,
you could define a custom proxy class,
you can patch the bug with a monkeypatch
I'll describe those options below, after explaining what AutoProxy does:
What's the point of the AutoProxy class
The multiprocessing Manager pattern gives access to shared values by putting the values all in the same, dedicated 'canonical values server' process. All other processes (clients) talk to the server through proxies that then pass messages back and forth with the server.
The server does need to know what methods are acceptable for the type of object, however, so clients can produce a proxy object with the same methods. This is what the AutoProxy object is for. Whenever a client needs a new instance of your registered class, the default proxy the client creates is an AutoProxy, which then asks the server to tell it what methods it can use.
Once it has the method names, it calls MakeProxyType to construct a new class and then creates an instance for that class to return.
All this is deferred until you actually need an instance of the proxied type, so in principle AutoProxy saves a little bit of memory if you are not using certain classes you have registered. It's very little memory, however, and the downside is that this process has to take place in each client process.
These proxy objects use reference counting to track when the server can remove the canonical value. It is that part that is broken in the AutoProxy callable; a new argument is passed to the proxy type to disable reference counting when the proxy object is being created in the server process rather than in a client but the AutoProxy type wasn't updated to support this.
So, how can you fix this? Here are those 3 options:
Use the MakeProxyType() callable
As mentioned, AutoProxy is really just a call (via the server) to get the public methods of the type, and a call to MakeProxyType(). You can just make these calls yourself, when registering.
So, instead of
from multiprocessing.managers import SyncManager
SyncManager.register("YourType", YourType)
use
from multiprocessing.managers import SyncManager, MakeProxyType, public_methods
# arguments: classname, sequence of method names
YourTypeProxy = MakeProxyType("YourType", public_methods(YourType))
SyncManager.register("YourType", YourType, YourTypeProxy)
Feel free to inline the MakeProxyType() call there.
If you were using the exposed argument to SyncManager.register(), you should pass those names to MakeProxyType instead:
# SyncManager.register("YourType", YourType, exposed=("foo", "bar"))
# becomes
YourTypeProxy = MakeProxyType("YourType", ("foo", "bar"))
SyncManager.register("YourType", YourType, YourTypeProxy)
You'd have to do this for all the pre-registered types, too:
from multiprocessing.managers import SyncManager, AutoProxy, MakeProxyType, public_methods
registry = SyncManager._registry
for typeid, (callable, exposed, method_to_typeid, proxytype) in registry.items():
if proxytype is not AutoProxy:
continue
create_method = hasattr(managers.SyncManager, typeid)
if exposed is None:
exposed = public_methods(callable)
SyncManager.register(
typeid,
callable=callable,
exposed=exposed,
method_to_typeid=method_to_typeid,
proxytype=MakeProxyType(f"{typeid}Proxy", exposed),
create_method=create_method,
)
Create custom proxies
You could not rely on multiprocessing creating a proxy for you. You could just write your own. The proxy is used in all processes except for the special 'managed values' server process, and the proxy should pass messages back and forth. This is not an option for the already-registered types, of course, but I'm mentioning it here because for your own types this offers opportunities for optimisations.
Note that you should have methods for all interactions that need to go back to the 'canonical' value instance, so you'd need to use properties to handle normal attributes or add __getattr__, __setattr__ and __delattr__ methods as needed.
The advantage is that you can have very fine-grained control over what methods actually need to exchange data with the server process; in my specific example, my proxy class caches information that is immutable (the values would never change once the object was created), but were used often. That includes a flag value that controls if other methods would do something, so the proxy could just check the flag value and not talk to the server process if not set. Something like this:
class FooProxy(BaseProxy):
# what methods the proxy is allowed to access through calls
_exposed_ = ("__getattribute__", "expensive_method", "spam")
#property
def flag(self):
try:
v = self._flag
except AttributeError:
# ask for the value from the server, "realvalue.flag"
# use __getattribute__ because it's an attribute, not a property
v = self._flag = self._callmethod("__getattribute__", ("flag",))
return flag
def expensive_method(self, *args, **kwargs):
if self.flag: # cached locally!
return self._callmethod("expensive_method", args, kwargs)
def spam(self, *args, **kwargs):
return self._callmethod("spam", args, kwargs
SyncManager.register("Foo", Foo, FooProxy)
Because MakeProxyType() returns a BaseProxy subclass, you can combine that class with a custom subclass, saving yourself having to write any methods that just consist of return self._callmethod(...):
# a base class with the methods generated for us. The second argument
# doubles as the 'permitted' names, stored as _exposed_
FooProxyBase = MakeProxyType(
"FooProxyBase",
("__getattribute__", "expensive_method", "spam"),
)
class FooProxy(FooProxyBase):
#property
def flag(self):
try:
v = self._flag
except AttributeError:
# ask for the value from the server, "realvalue.flag"
# use __getattribute__ because it's an attribute, not a property
v = self._flag = self._callmethod("__getattribute__", ("flag",))
return flag
def expensive_method(self, *args, **kwargs):
if self.flag: # cached locally!
return self._callmethod("expensive_method", args, kwargs)
def spam(self, *args, **kwargs):
return self._callmethod("spam", args, kwargs
SyncManager.register("Foo", Foo, FooProxy)
Again, this won't solve the issue with standard types nested inside other proxied values.
Apply a monkeypatch
I use this to fix the AutoProxy callable, this should automatically avoid patching when you are running a Python version where the fix has already been applied to the source code:
# Backport of https://github.com/python/cpython/pull/4819
# Improvements to the Manager / proxied shared values code
# broke handling of proxied objects without a custom proxy type,
# as the AutoProxy function was not updated.
#
# This code adds a wrapper to AutoProxy if it is missing the
# new argument.
import logging
from inspect import signature
from functools import wraps
from multiprocessing import managers
logger = logging.getLogger(__name__)
orig_AutoProxy = managers.AutoProxy
#wraps(managers.AutoProxy)
def AutoProxy(*args, incref=True, manager_owned=False, **kwargs):
# Create the autoproxy without the manager_owned flag, then
# update the flag on the generated instance. If the manager_owned flag
# is set, `incref` is disabled, so set it to False here for the same
# result.
autoproxy_incref = False if manager_owned else incref
proxy = orig_AutoProxy(*args, incref=autoproxy_incref, **kwargs)
proxy._owned_by_manager = manager_owned
return proxy
def apply():
if "manager_owned" in signature(managers.AutoProxy).parameters:
return
logger.debug("Patching multiprocessing.managers.AutoProxy to add manager_owned")
managers.AutoProxy = AutoProxy
# re-register any types already registered to SyncManager without a custom
# proxy type, as otherwise these would all be using the old unpatched AutoProxy
SyncManager = managers.SyncManager
registry = managers.SyncManager._registry
for typeid, (callable, exposed, method_to_typeid, proxytype) in registry.items():
if proxytype is not orig_AutoProxy:
continue
create_method = hasattr(managers.SyncManager, typeid)
SyncManager.register(
typeid,
callable=callable,
exposed=exposed,
method_to_typeid=method_to_typeid,
create_method=create_method,
)
Import the above and call the apply() function to fix multiprocessing. Do so before you start the manager server!
Solution editing multiprocessing source code
The original answer by Sergey requires you to edit multiprocessing source code as follows:
Find your multiprocessing package (mine, installed via Anaconda, was in /anaconda3/lib/python3.6/multiprocessing).
Open managers.py
Add the key argument manager_owned=True to the AutoProxy function.
Original AutoProxy:
def AutoProxy(token, serializer, manager=None, authkey=None,
exposed=None, incref=True):
...
Edited AutoProxy:
def AutoProxy(token, serializer, manager=None, authkey=None,
exposed=None, incref=True, manager_owned=True):
...
Solution via code, at run time
I have managed to solve the unexpected keyword argument TypeError exception without editing directly the source code of multiprocessing by instead adding these few lines of code where I use multiprocessing's Managers:
import multiprocessing
# Backup original AutoProxy function
backup_autoproxy = multiprocessing.managers.AutoProxy
# Defining a new AutoProxy that handles unwanted key argument 'manager_owned'
def redefined_autoproxy(token, serializer, manager=None, authkey=None,
exposed=None, incref=True, manager_owned=True):
# Calling original AutoProxy without the unwanted key argument
return backup_autoproxy(token, serializer, manager, authkey,
exposed, incref)
# Updating AutoProxy definition in multiprocessing.managers package
multiprocessing.managers.AutoProxy = redefined_autoproxy
Found temporary solution here.
I've managed to fix it by adding needed keyword to initializer of AutoProxy in multiprocessing\managers.py Though, I don't know if this kwarg is responsible for anything.
I'm trying to override a python class (first time doing this), and I can't seem to override this method. When I run this, my recv method doesn't run. It runs the superclasses's method instead. What am I doing wrong here? (This is python 2.7 by the way.)
import socket
class PersistentSocket(socket.socket):
def recv(self, count):
print("test")
return super(self.__class__, self).recv(count)
if __name__ == '__main__':
s = PersistentSocket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 2300))
print(s.recv(1)
The socket type (officially socket.SocketType, though socket.socket happens to be the same object) makes the strange choice of implementing recv and a few other methods as instance attributes, rather than as normal methods in the class dict. In socket.SocketType.__init__, it sets a self.recv instance attribute that overrides the recv method you tried to define.
Picking on the explanation from #user2357112, one thing that seems to have helped is to do a delattr(self, 'recv') on the class constructor (inheriting from socket.SocketType) and then define you own recv method; for example:
class PersistentSocket(socket.SocketType):
def __init__(self):
"""As usual."""
delattr(self, 'recv')
def recv(self, buffersize=1024, flags=0):
"""Your own implementation here."""
return None
Here in given code it is passing class name i.e. MyRequestHandler to TCP and also after taking class name as argument what does it do with that.So my question is that can class name be used as argument and also class name doesn't refer to anything so how is it possible???i apologize for silly ques!!
from SocketServer import (TCPServer as TCP,
StreamRequestHandler as SRH)
from time import ctime
HOST = ''
PORT = 21567
ADDR = (HOST, PORT)
class MyRequestHandler(SRH):
def handle(self):
print '...connected from:',self.client_address
self.wfile.write('[%s] %s' % (ctime(),
self.rfile.readline()))
tcpServ = TCP(ADDR, MyRequestHandler)
print 'waiting for connection...'
tcpServ.serve_forever(
Absolutely you can pass a class name as an argument to a function:
>>> class A():
... def __init__(self):
... print "an object from class A is created"
...
>>> def hello(the_argument):
... x = the_argument()
...
>>> hello(A)
an object from class A is created
You aren't passing in the name of a class, you are passing in a reference to the class. A class is an object just like everything else. Think of it as a function that returns another object. You can pass a class as an argument just like you can pass a function, a string, or any other object.
As for what the called function can do with it -- it create create instances of that class. In this case, the called function doesn't really care what class it uses, as long as it implements a particular interface. So, the called function doesn't ever see the name of the class, it's just told how to create an instance of it (by virtue of being given a reference to the class)
In the case of a server, it needs to create a new instance of some object for every connection to the server. So, you give it the class you want it to use, and it creates instances for each connection. This lets the server create the objects when it needs them, rather than requiring you to create them ahead of time.
can class name be used as argument?
Yes. but in your code you are not passing a class name to the TCP constructor, you are passing a request handler to the constructor.
also class name doesn't refer to anything so how is it possible?
As mention above, you are passing a request handler to the Tcp constructor, your request handler refers to an action which TCP server will use to handle the incoming request. So it does refer to something.
yes you can pass to class args or function parameter
1 - using type(object)
2 - class Name
==>>passing ClassA to to ClassB and fname as parameters
class ClassA(object):
def __init__(self,):
pass
class ClassB(object):
def __init__(self, arg):
print(type(arg))
pass
def fname(arg):
print(type(arg))
pass
valueA: ClassA = ClassA()
ClassB(type(valueA))
ClassB(ClassA)
fname(type(valueA))
fname((ClassA))
I am using a simpleWebSocket server class and have a 1 second interval timer that I would like to call methods in a couple of different classes.
the wsscb() class is the handler for the SimpleWebSocketServer(), how can I call a method from the wss() object from another object such as the udt() timer ?
Calling wss.wsscb().myfunc() results in an error: "AttributeError: 'SimpleWebSocketServer' object has no attribute 'wsscb'"
calling wsscb.myfunc() results in: TypeError: unbound method myfunc() must be called with wsscb instance as first argument (got nothing instead)
class wsscb(WebSocket):
def __init__(self, server, sock, address):
WebSocket.__init__(self, server, sock, address)
def myfunc(self):
self.send('some data')
def handleMessage(self):
pass
def handleConnected(self):
pass
class udt(Thread):
def __init__(self, event):
Thread.__init__(self)
self.stopped = event
def run(self):
while not self.stopped.wait(1.00):
wss.wsscb().myfunc()
xxx.yyy()().anotherfunc()
## Main
wss = SimpleWebSocketServer('', 4545,wsscb)
## Start Timer
stopFlag = Event()
self.udt = udt(stopFlag)
self.udt.start()
wss.serveforever()
There are a couple problems.
wss.wsscb() isn't valid. Typing that means you're trying to call a function in wss called wsscb(). wss is a SimpleWebSocketServer, and there is no function called wsscb(). A function is not the same as calling an object.
wsscb() won't work either, because in your class, you're saying it's takes a WebSocket object, which I assume takes some parameters, so you need to pass it those.
I think it would be best to make a subclass of SimpleWebSocketServer (instead of WebSocket), and put your custom function in there. Your comment says "wsscb() is a subclass of SimpleSocketServer", but it is not. It's a subclass of WebSocket.
You also never created an object of type wsscb.
If you can explain what you're specifically trying to achieve, and what myfunc() is, we may be able to help more
Also, you really shouldn't subclass Thread. Scrap the udt class you made and instead
def myfunc(wsscb_object):
while True:
time.sleep(1)
wsscb_object.myfunc()
#whatever else you want
wsscb_object = wsscb(#pass the parameters)
thread = Thread(target=myfunc, args=(some_socket))
thread.start()
You may also want to read up more on inheritance:
python subclasses
http://www.jesshamrick.com/2011/05/18/an-introduction-to-classes-and-inheritance-in-python/
Using inheritance in python