I am intermittently receiving a httplib.CannotSendRequest exception when using a chain of SimpleXMLRPCServers that use the SocketServer.ThreadingMixin.
What I mean by 'chain' is the following:
I have a client script which uses xmlrpclib to call a function on a SimpleXMLRPCServer. That server, in turn, calls another SimpleXMLRPCServer. I realise how convoluted that sounds, but there are good reasons that this architecture has been selected, and I don't see a reason it shouldn't be possible.
(testclient)client_script ---calls-->
(middleserver)SimpleXMLRPCServer ---calls--->
(finalserver)SimpleXMLRPCServer --- does something
If I do not use SocketServer.ThreadingMixin then this issue doesn't occur (but I need the requests to be multi-threaded so this doesn't help.)
If I only have a single level of services (ie just client script calling final server directly) this doesn't happen.
I have been able to reproduce the issue in the simple test code below. There are three snippets:
finalserver:
import SocketServer
import time
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass
# Create server
server = AsyncXMLRPCServer(('', 9999), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()
def waste_time():
time.sleep(10)
return True
server.register_function(waste_time, 'waste_time')
server.serve_forever()
middleserver:
import SocketServer
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import xmlrpclib
class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass
# Create server
server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()
s = xmlrpclib.ServerProxy('http://localhost:9999')
def call_waste():
s.waste_time()
return True
server.register_function(call_waste, 'call_waste')
server.serve_forever()
testclient:
import xmlrpclib
s = xmlrpclib.ServerProxy('http://localhost:8888')
print s.call_waste()
To reproduce, the following steps should be used:
Run python finalserver.py
Run python middleserver.py
Run python testclient.py
While (3) is still running, run another instance of python testclient.py
Quite often (almost every time) you will get the error below the first time you try to run step 4. Interestingly, if you immediately try to run step (4) again the error will not occur.
Traceback (most recent call last):
File "testclient.py", line 6, in <module>
print s.call_waste()
File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
return u.close()
File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: "<class 'httplib.CannotSendRequest'>:">
The internet appears to say that this exception can be caused by multiple calls to httplib.HTTPConnection.request without intervening getresponse calls. However, the internet doesn't discuss this in the context of SimpleXMLRPCServer. Any pointers in the direction of resolving the httplib.CannotSendRequest issue would be appreciated.
===========================================================================================
ANSWER:
Okay, I'm a bit stupid. I think I was staring at the code for too protracted a period of time that I missed the obvious solution staring me in the face (quite literally, because the answer is actually in the actual question.)
Basically, the CannotSendRequest occurs when an httplib.HTTPConnection is interrupted by an intervening 'request' operation. Each httplib.HTTPConnection.request must be paired with a .getresponse() call. If that pairing is interrupted by another request operation, the second request will produce the CannotSendRequest error. so:
connection = httplib.HTTPConnection(...)
connection.request(...)
connection.request(...)
will fail because you have two requests on the same connection before any getresponse is called.
Linking that back to my question:
the only place in the three programs where such connections are being made are in the serverproxy calls.
the problem only occurs during threading, so it's likely a race condition.
the only place a serverproxy call is shared is in middleserver.py
The solution then, is obviously to have each thread create it's own serverproxy. The fixed version of middleserver is below, and it works:
import SocketServer
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import xmlrpclib
class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass
# Create server
server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()
def call_waste():
# Each call to this function creates its own serverproxy.
# If this function is called by concurrent threads, each thread
# will safely have its own serverproxy.
s = xmlrpclib.ServerProxy('http://localhost:9999')
s.waste_time()
return True
server.register_function(call_waste, 'call_waste')
server.serve_forever()
Since this version results in each thread having its own xmlrpclib.serverproxy, there is no risk of the same instance of serverproxy invoking HTTPConnection.request more than once in succession. The programs work as intended.
Sorry for the bother.
Okay, I'm a bit stupid. I think I was staring at the code for to protracted a period of time that I missed the obvious solution staring me in the face (quite literally, because the answer is actually in the actual question.)
Basically, the CannotSendRequest occurs when an httplib.HTTPConnection is interrupted by an intervening 'request' operation. Basically, each httplib.HTTPConnection.request must be paired with a .getresponse() call. If that pairing is interrupted by another request operation, the second request will produce the CannotSendRequest error. so:
connection = httplib.HTTPConnection(...)
connection.request(...)
connection.request(...)
will fail because you have two requests on the same connection before any getresponse is called.
Linking that back to my question:
the only place in the three programs where such connections are being made are in the serverproxy calls.
the problem only occurs during threading, so it's likely a race condition.
the only place a serverproxy call is shared is in middleserver.py
The solution then, is obviously to have each thread create it's own serverproxy. The fixed version of middleserver is below, and it works:
import SocketServer
from SimpleXMLRPCServer import SimpleXMLRPCServer
from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler
import xmlrpclib
class AsyncXMLRPCServer(SocketServer.ThreadingMixIn,SimpleXMLRPCServer): pass
# Create server
server = AsyncXMLRPCServer(('', 8888), SimpleXMLRPCRequestHandler)
server.register_introspection_functions()
def call_waste():
# Each call to this function creates its own serverproxy.
# If this function is called by concurrent threads, each thread
# will safely have its own serverproxy.
s = xmlrpclib.ServerProxy('http://localhost:9999')
s.waste_time()
return True
server.register_function(call_waste, 'call_waste')
server.serve_forever()
Since this version results in each thread having its own xmlrpclib.serverproxy, there is no risk of the serverproxy invoking HTTPConnection.request more than once in succession. The programs work as intended.
Sorry for the bother.
Related
In one of my Odoo installation I need to setup the socket_timeout variable of WorkerHTTP class directly from Python code, bypassing the usage of environment variable ODOO_HTTP_SOCKET_TIMEOUT.
If you never read about it, you can check here for more info: https://github.com/odoo/odoo/commit/49e3fd102f11408df00f2c3f6360f52143911d74#diff-b4207a4658979fdb11f2f2fa0277f483b4e81ba59ed67a5e84ee260d5837ef6d
In Odoo15, which i'm using, Worker classes are located at odoo/service/server.py
My idea was to inherit constructor for Worker class and simply setup self.sock_timeout = 10 or another value, but I can't make it work with inheritance.
EDIT: I almost managed it to work, but I have problems with static methods.
STEP 1:
Inherit WorkerHTTP constructor and add self.socket_timeout = 10
Then, I also have to inherit PreforkServer and override process_spawn() method so I can pass WorkerHttpExtend instead of WorkerHTTP, as argument for worker_spawn() method.
class WorkerHttpExtend(WorkerHTTP):
""" Setup sock_timeout class variable when WorkerHTTP object gets initialized"""
def __init__(self, multi):
super(WorkerHttpExtend, self).__init__(multi)
self.sock_timeout = 10
logging.info(f'SOCKET TIMEOUT: {self.sock_timeout}')
class PreforkServerExtend(PreforkServer):
""" I have to inherit PreforkServer and override process_spawn()
method so I can pass WorkerHttpExtend
instead of WorkerHTTP, as argument for worker_spawn() method.
"""
def process_spawn(self):
if config['http_enable']:
while len(self.workers_http) < self.population:
self.worker_spawn(WorkerHttpExtend, self.workers_http)
if not self.long_polling_pid:
self.long_polling_spawn()
while len(self.workers_cron) < config['max_cron_threads']:
self.worker_spawn(WorkerCron, self.workers_cron)
STEP 2:
static method start() should initialize PreforkServer with PreforkServerExtend, not with PreforkServer (last line in the code below). This is where I start to have problems.
def start(preload=None, stop=False):
"""Start the odoo http server and cron processor."""
global server
load_server_wide_modules()
if odoo.evented:
server = GeventServer(odoo.service.wsgi_server.application)
elif config['workers']:
if config['test_enable'] or config['test_file']:
_logger.warning("Unit testing in workers mode could fail; use --workers 0.")
server = PreforkServer(odoo.service.wsgi_server.application)
STEP 3:
At this point if I wanna go further (which I did) I should copy the whole start() method and import all package I need to make it work
import odoo
from odoo.service.server import WorkerHTTP, WorkerCron, PreforkServer, load_server_wide_modules, \
GeventServer, _logger, ThreadedServer, inotify, FSWatcherInotify, watchdog, FSWatcherWatchdog, _reexec
from odoo.tools import config
I did it and then in my custom start() method I wrote line
server = PreforkServerExtend(odoo.service.wsgi_server.application)
but even then, how do I tell to execute my start() method, instead of the original one??
I'm sure this would eventually work (mabe not safely, but would work) because at some point I wasn't 100% sure what I was doing, so I put my inherit classes WorkerHttpExtend and PreforkServerExtend in the original odoo/service/server.py and initialized server obj with PreforkServerExtend instead of PreforkServer.
server = PreforkServer(odoo.service.wsgi_server.application)
It works then: I get custom socket timeout value, print and logging info when Odoo service start, because PreforkServerExtend will call custom class on cascade at that point, otherwise my inherited class are there but they will never be called.
So I guess if I could tell the system to run my start() method I would have done it.
STEP 4 (not reached yet):
I'm pretty sure that start() method is called in odoo/cli/server.py, in main() method:
rc = odoo.service.server.start(preload=preload, stop=stop)
I could go deeper but I don't think the effort is worth for what I need.
So technically if I would be able to tell the system which start() method to choose, I would have done it. Still not sure it is safe procedure (probably not much actually, but at this point I was just experimenting), but I wonder if there is an easier method to set up socket timeout without using environment variable ODOO_HTTP_SOCKET_TIMEOUT.
I'm pretty sure there is an easier method than i'm doing, with low level python or maybe even with a class in odoo/service/server, but I can't figure out for now. If some one has an idea, let me know!
Working solution: I have been introduced to Monkeypatch in this post
Possible for a class to look down at subclass constructor?
This has solved my problem, now I'm able to patch process_request method of class WorkerHTTP :
import errno
import fcntl
import socket
import odoo
import odoo.service.server as srv
class WorkerHttpProcessRequestPatch(srv.WorkerHTTP):
def process_request(self, client, addr):
client.setblocking(1)
# client.settimeout(self.sock_timeout)
client.settimeout(10) # patching timeout setup to a needed value
client.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
flags = fcntl.fcntl(client, fcntl.F_GETFD) | fcntl.FD_CLOEXEC
fcntl.fcntl(client, fcntl.F_SETFD, flags)
self.server.socket = client
try:
self.server.process_request(client, addr)
except IOError as e:
if e.errno != errno.EPIPE:
raise
self.request_count += 1
# Switch process_request class attribute - this is what I needed to make it work
odoo.service.server.WorkerHTTP.process_request = WorkerHttpProcessRequestPatch.process_request
We're seeing exceptions in our log like the following:
ERROR Exception ignored in: <function Connection.__del__ at 0x7f9b70a5cc20>
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.7/site-packages/redis/connection.py", line 537, in __del__
File "/app/.heroku/python/lib/python3.7/site-packages/redis/connection.py", line 667, in disconnect
TypeError: catching classes that do not inherit from BaseException is not allowed
According to the Redis source code, the offending line is the except in the following snippet:
try:
if os.getpid() == self.pid:
shutdown(self._sock, socket.SHUT_RDWR)
self._sock.close()
except socket.error:
pass
Which would indicate that socket.exception doesn't inherit from BaseException. However, as far as I can tell (based on the docs and the mro class method), socket.exception does inherit from BaseException.
Why is this happening? What can I do to prevent it?
By the way, our code doesn't call Redis directly. We are using Redis Queue (rq), which is implemented using Redis.
This would happen if you didn't close the redis client explicitly. That's why you saw __del__ in the traceback.
I'm not using rq but I'll take celery as an example, yet the idea could also be applied to rq.
# tasks/__init__.py
from celeryapp import redis_client
#worker_shutdown.connect # this signal means it's about to shut down the worker
def cleanup(**kwargs):
redis_client.close() # without this you may see error
# celeryapp.py
from celery import Celery
import redis
app = Celery(config_source="celeryconfig")
redis_client = redis.StrictRedis()
I've two test classes (TrialTest1 and TrialTest2) written in two files (test_trial1.py and test_trial2.py) mostly identical (the only difference is the class name):
from twisted.internet import reactor
from twisted.trial import unittest
class TrialTest1(unittest.TestCase):
def setUp(self):
print("setUp()")
def test_main(self):
print("test_main")
reactor.callLater(1, self._called_by_deffered1)
reactor.run()
def _called_by_deffered1(self):
print("_called_by_deffered1")
reactor.callLater(1, self._called_by_deffered2)
def _called_by_deffered2(self):
print("_called_by_deffered2")
reactor.stop()
def tearDown(self):
print("tearDown()")
When I run each test idepently, everything is fine. But when I launch both I've the following output:
setUp()
test_main
_called_by_deffered1
_called_by_deffered2
tearDown()
setUp()
test_main
tearDown()
Error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 137, in maybeDeferred
result = f(*args, **kw)
File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 203, in runWithWarningsSuppressed
reraise(exc_info[1], exc_info[2])
File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 199, in runWithWarningsSuppressed
result = f(*a, **kw)
File "/home/kartoch/works/python/netkython/tests/test_twisted_trial2.py", line 13, in test_main
reactor.run()
File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1191, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1171, in startRunning
ReactorBase.startRunning(self)
File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 683, in startRunning
raise error.ReactorNotRestartable()
ReactorNotRestartable
Error
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x8d6482c [0.98535490036s] called=0 cancelled=0 TrialTest2._called_by_deffered1()>
Process finished with exit code 0
It seems the reactor is not switched off correctly after the first test. Does anyone know where is the problem ? It seems tearDown() is called to early (before _called_by_deffered1 in the second test), maybe a fix would be to use deferTearDown (not documented method of trial unittest).
EDIT
One of the solution proposed was to remove reactor.run() and reactor.stop() because a reactor is not restartable and you have only one reactor instance for all test by default:
class TrialTest1(unittest.TestCase):
def setUp(self):
print("setUp()")
def test_main(self):
print("test_main")
reactor.callLater(1, self._called_by_deffered1)
def _called_by_deffered1(self):
print("_called_by_deffered1")
reactor.callLater(1, self._called_by_deffered2)
def _called_by_deffered2(self):
print("_called_by_deffered2")
def tearDown(self):
print("tearDown()")
But when removing calls to such methods, my tests fail without executing _called_by_deffered methods:
setUp()
test_main
tearDown()
Error
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x94967ec [0.99936413765s] called=0 cancelled=0 TrialTest1._called_by_deffered1()>
setUp()
test_main
tearDown()
Error
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x94968cc [0.99958896637s] called=0 cancelled=0 TrialTest2._called_by_deffered1()>
If I want to use only one instance of reactor shared between tests, how _called_by_deffered methods could be part of the test (i.e. executed before tearDown) ?
The reactor is not restartable. There are two obvious options for your to pursue for writing your tests.
One is to use the global reactor. trial starts and stops it - your tests never have to call reactor.run or reactor.stop (and they never should). It is accessible in the usual way, from twisted.internet import reactor.
The other is to use a new reactor instance per test. There are some test-oriented reactor instances in twisted.test.proto_helpers (that is the only part of twisted.test that is a public, supported interface by the way). MemoryReactor and StringTransport get you most of the way towards being able to test network interactions. twisted.internet.task.Clock helps you out with testing time-based events.
With the help of Jean-Paul, this page and this question, I've been able to fix the problem using twisted.internet.task.deferLater(). To summarize the point i was looking for: if a test method returning a deferred, the 'tearDown()' method will be called only when all deferreds are fired.
This is the code:
from twisted.trial import unittest
from twisted.internet import reactor, task
class TrialTest1(unittest.TestCase):
def setUp(self):
print("setUp()")
def test_main(self):
print("test_main()")
return task.deferLater(reactor, 1, self._called_by_deffered1)
def _called_by_deffered1(self):
print("_called_by_deffered1()")
return task.deferLater(reactor, 1, self._called_by_deffered2)
def _called_by_deffered2(self):
print("_called_by_deffered2()")
def tearDown(self):
print("tearDown()")
Output:
setUp()
test_main()
// (1s wait)
_called_by_deffered1()
// (1s wait)
_called_by_deffered2()
tearDown()
I want to be able to at first call a simple script to enable or disable an external monitor from my netbook. I am running Fedora 17 with XFCE as my desktop. I see that I should be able to use python and python-dbus to flip toggle active on and off. My problem is that I can't figure out how to emit a signal to get the new setting to go active. Unfortunately Python is not a language that I use often. The code that I have in place is:
import dbus
item = 'org.xfce.Xfconf'
path = '/org/xfce/Xfconf'
channel = 'displays'
base = '/'
setting = '/Default/VGA1/Active'
bus = dbus.SessionBus()
remote_object = bus.get_object(item, path)
remote_interface = dbus.Interface(remote_object, "org.xfce.Xfconf")
if remote_interface.GetProperty(channel, setting):
remote_interface.SetProperty(channel, setting, '0')
remote_object.PropertyChanged(channel, setting, '0')
else:
remote_interface.SetProperty(channel, setting, '1')
remote_object.PropertyChanged(channel, setting, '0')
It is failing and kicking out:
Traceback (most recent call last): File "./vgaToggle", line 31, in <module>
remote_object.PropertyChanged(channel, setting, '0')
File "/usr/lib/python2.7/site-packages/dbus/proxies.py", line 140, in __call__
**keywords)
File "/usr/lib/python2.7/site-packages/dbus/connection.py", line 630, in call_blocking
message, timeout) dbus.exceptions.DBusException:
org.freedesktop.DBus.Error.UnknownMethod: Method "PropertyChanged"
with signature "sss" on interface "(null)" doesn't exist
I spent a bit of time searching and I am not finding many python examples doing anything close to this. Thanks in advance.
PropertyChanged is a signal, not a method. The services you are communicating with are responsible for emitting signals. In this case, the PropertyChanged should fire implicitly, whenever the value of the property on the respective objects or interfaces have changed.
This should happen implicitly when you call remote_interface.SetProperty(...), and you should not need to explicitly "call" the signal like a method.
If you are interested in receiving the signals, you will need to set up a glib main loop and call connect_to_signal on your proxy object, passing it a callback method to invoke.
I have a simple 'echo' PB client and server where the client sends an object to the server which echo the same object back to the client:
The client:
from twisted.spread import pb
from twisted.internet import reactor
from twisted.python import util
from amodule import aClass
factory = pb.PBClientFactory()
reactor.connectTCP("localhost", 8282, factory)
d = factory.getRootObject()
d.addCallback(lambda object: object.callRemote("echo", aClass()))
d.addCallback(lambda response: 'server echoed: '+response)
d.addErrback(lambda reason: 'error: '+str(reason.value))
d.addCallback(util.println)
d.addCallback(lambda _: reactor.stop())
reactor.run()
The server:
from twisted.application import internet, service
from twisted.internet import protocol
from twisted.spread import pb
from amodule import aClass
class RemoteClass(pb.RemoteCopy, aClass):
pass
pb.setUnjellyableForClass(aClass, RemoteClass)
class PBServer(pb.Root):
def remote_echo(self, a):
return a
application = service.Application("Test app")
# Prepare managers
clientManager = internet.TCPServer(8282, pb.PBServerFactory(PBServer()));
clientManager.setServiceParent(application)
if __name__ == '__main__':
print "Run with twistd"
import sys
sys.exit(1)
The aClass is a simple class implementing Copyable:
from twisted.spread import pb
class aClass(pb.Copyable):
pass
When i run the above code, i get this error:
twisted.spread.jelly.InsecureJelly: Module builtin not allowed (in type builtin.RemoteClass).
In fact, the object is sent to the server without any problem since it was secured with pb.setUnjellyableForClass(aClass, RemoteClass) on the server side, but once it gets returned to the client, that error is raised.
Am looking for a way to get an easy way to send/receive my objects between two peers.
Perspective broker identifies classes by name when talking about them over the network. A class gets its name in part from the module in which it is defined. A tricky problem with defining classes in a file that you run from the command line (ie, your "main script") is that they may end up with a surprising name. When you do this:
python foo.py
The module name Python gives to the code in foo.py is not "foo" as one might expect. Instead it is something like "__main__" (which is why the if __name__ == "__main__": trick works).
However, if some other part of your application later tries to import something from foo.py, then Python re-evaluates its contents to create a new module named "foo".
Additionally, the classes defined in the "__main__" module of one process may have nothing to do with the classes defined in the "__main__" module of another process. This is the case in your example, where __main__.RemoteClass is defined in your server process but there is no RemoteClass in the __main__ module of your client process.
So, PB gets mixed up and can't complete the object transfer.
The solution is to keep the amount of code in your main script to a minimum, and in particular to never define things with names there (no classes, no function definitions).
However, another problem is the expectation that a RemoteCopy can be sent over PB without additional preparation. A Copyable can be sent, creating a RemoteCopy on the peer, but this is not a symmetric relationship. Your client also needs to allow this by making a similar (or different) pb.setUnjellyableForClass call.