I have some issues with the requestLoop methode of the Pyro4.Daemon object.
What I want is to call remotely a "stop()" method for releasing the requestLoop function and shutdown my daemon.
This small exemple doesn't work
SERVER
#!/usr/bin/python
# -*- coding: utf-8 -*-
from daemon import Pyro4
class Audit(object):
def start_audit(self):
with Pyro4.Daemon() as daemon:
self_uri = daemon.register(self)
ns = Pyro4.locateNS()
ns.register("Audit", self_uri)
self.running = True
print("starting")
daemon.requestLoop(loopCondition=self.still_running)
print("stopped")
self.running = None
def hi(self, string):
print string
def stop(self):
self.running = False
def still_running(self):
return self.running
def main():
# lancement de l'auditor
auditor = Audit()
auditor.start_audit()
if __name__ == "__main__" :
main()
CLIENT
import Pyro4
def main():
with Pyro4.Proxy("PYRONAME:Audit") as au:
au.hi("hello")
au.hi("another hi")
au.stop()
What I expect is to see the server print "hello" and "another hi" and then shutdown.
But the shutdown doesn't happen, the server is still blocked in the requestloop method.
I can use my proxy as long as I want.
BUT, if I create another client, at the first remote call, the server will shutdown and the client will throw an error:
Pyro4.errors.ConnectionClosedError: receiving: not enough data
All my test are saying that I need to create a 2nd proxy and throw the exeption for pass the requestloop on my server.
Does any one have a idea of how to clean this issue ?
If you look at the examples/callback/client.py in the sources you'll see the following comment:
# We need to set either a socket communication timeout,
# or use the select based server. Otherwise the daemon requestLoop
# will block indefinitely and is never able to evaluate the loopCondition.
Pyro4.config.COMMTIMEOUT=0.5
Hence, you need to do is set the COMMTIMEOUT in your server file and it will work fine according to my tests.
Note: You can also add a print statement to the still_running method to check when it's being called. Without the configuration above, you'll see that it looks like the method is executed only when a new event is received, so the server doesn't shutdown after the next event to the one that set running to False is received. For example, if you execute the client program twice, the server will shutdown.
Related
I have an app similar to a chat-room writing in python that intends to do the following things:
A prompt for user to input websocket server address.
Then create a websocket client that connects to server and send/receive messages. Disable the ability to create a websocket client.
After receiving "close" from server (NOT a close frame), client should drop connecting and re-enable the app to create a client. Go back to 1.
If user exits the app, it exit the websocket client if there is one running.
My approach for this is using a main thread to deal with user input. When user hits enter, a thread is created for WebSocketClient using AutoBahn's twisted module and pass a Queue to it. Check if the reactor is running or not and start it if it's not.
Overwrite on message method to put a closing flag into the Queue when getting "close". The main thread will be busy checking the Queue until receiving the flag and go back to start. The code looks like following.
Main thread.
def main_thread():
while True:
text = raw_input("Input server url or exit")
if text == "exit":
if myreactor:
myreactor.stop()
break
msgq = Queue.Queue()
threading.Thread(target=wsthread, args=(text, msgq)).start()
is_close = False
while True:
if msgq.empty() is False:
msg = msgq.get()
if msg == "close":
is_close = True
else:
print msg
if is_close:
break
print 'Websocket client closed!'
Factory and Protocol.
class MyProtocol(WebSocketClientProtocol):
def onMessage(self, payload, isBinary):
msg = payload.decode('utf-8')
self.Factory.q.put(msg)
if msg == 'close':
self.dropConnection(abort=True)
class WebSocketClientFactoryWithQ(WebSocketClientFactory):
def __init__(self, *args, **kwargs):
self.queue = kwargs.pop('queue', None)
WebSocketClientFactory.__init__(self, *args, **kwargs)
Client thread.
def wsthread(url, q):
factory = WebSocketClientFactoryWithQ(url=url, queue=q)
factory.protocol = MyProtocol
connectWS(Factory)
if myreactor is None:
myreactor = reactor
reactor.run()
print 'Done'
Now I got a problem. It seems that my client thread never stops. Even if I receive "close", it seems still running and every time I try to recreate a new client, it creates a new thread. I understand the first thread won't stop since reactor.run() will run forever, but from the 2nd thread and on, it should be non-blocking since I'm not starting it anymore. How can I change that?
EDIT:
I end up solving it with
Adding stopFactory() after disconnect.
Make protocol functions with reactor.callFromThread().
Start the reactor in the first thread and put clients in other threads and use reactor.callInThread() to create them.
Your main_thread creates new threads running wsthread. wsthread uses Twisted APIs. The first wsthread becomes the reactor thread. All subsequent threads are different and it is undefined what happens if you use a Twisted API from them.
You should almost certainly remove the use of threads from your application. For dealing with console input in a Twisted-based application, take a look at twisted.conch.stdio (not the best documented part of Twisted, alas, but just what you want).
I have to pass a data from my test cases to a mock server.
What is the best way to do that ?
This is what I have so far
mock_server.py
class ThreadedUDPServer(SocketServer.ThreadingMixIn, SocketServer.UDPServer):
pass
class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server):
SocketServer.BaseRequestHandler.__init__(self,request,client_address,server)
def handle(self):
print server.data #this is where i need the data
class server_wrap:
def __init__(self):
self.server = ThreadedUDPServer( ("127.0.0.1",49555) , ThreadedUDPRequestHandler)
def set_data(self,data)
self.server.data = data
def start(self)
server_thread = threading.Thread(target=self.server.serve_forever())
def stop(self)
self.server.shutdown()
test_mock.py
server_inst = server_wrap()
server_inst.start()
#code which sets the data and expects the handle method to print the data set
server_inst.stop()
The problem which i have with this code is, the execution stops at server_inst.start(), where the server goes in to an infinite listening mode
Other Solutions that I have tried, but failed:
Using global variables
Using queues
starting mock_server.py
with its own main
Let me know about any other possible solutions. Thanks in advance
Update 1:
Using separate threads to send data to the socket:
Changes
test_mock.py
def test_set_data(data)
server_inst = server_wrap()
server_inst.set_data(data)
server_inst.start()
if __name__ == "__main__":
thread = Thread(target=test_set_data, args=("foo_data))
thread.setDaemon(True)
thread.start()
#test code which verifies if data set is same
#works so far, able to pass data
#problem starts now
thread = Thread(target=test_set_data, args=("bar_data))
thread.setDaemon(True)
thread.start()
#says address already in use error
#Tried calling server.shuddown() in handle , but error persists. Also there is no thread.shop in threading.Thread object
Thanks
The server should go to listening mode.
You don't need the server_inst.stop until all the data was sent, and the test finishes. Maybe in you test tear down, or when the the test suite is completed.
To send data to the server, and let the handle pick it, you should open a socket on anohter thread. Then send the data to the server via this socket.
This code should look something like this:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("127.0.0.1",49555))
sock.send(... the data ...)
received = sock.recv(1024) # the handle can send a response
sock.close()
Add a function in your django code, which does run on another thread. This function will open the socket, connect, send the data and get the response. You can call it from a view, a middleware etc.
Currently I am writing an application using the SimpleXMLRPCServer module in Python.
The basic aim of this application is to keep running on a server and keep checking a Queue for any task. If it encounters any new request in the Queue, serve the request.
Snapshot of what I am trying to do :
class MyClass():
"""
This class will have methods which will be exposed to the clients
"""
def __init__(self):
taskQ = Queue.Queue()
def do_some_task(self):
while True:
logging.info("Checking the Queue for any Tasks..")
task = taskQ.get()
# Do some processing based on the availability of some task
Main
if name == "main":
server = SimpleXMLRPCServer.SimpleXMLRPCServer((socket.gethostname(), Port)
classObj = MyClass()
rpcserver.register_function(classObj.do_some_task)
rpcserver.serve_forever()
Once the server is started it remains in the loop forever inside do_some_task method to keep checking the Queue for any task. This is what i wanted to achieve. But now i want to gracefully shutdown the server. In this case i am unable to shutdown the server.
Till now I have Tried using a global flag STOP_SERVER for 'True' and checking its status in the do_some_task while loop to get out of it and stop the server. But no help.
Tried using SHUTDOWN() method of the SimpleXMLRPCServer but it seems it is getting into a infinite loop of somekind.
Could you suggest some proper way to gracefully shutdown the server.
Thanks in advance
You should use handle_request() instead of serve_forever() if you want to close it manualy. Because SimpleXMLRPCServer is implemented as a single thread and the serve_forever() will make the server instance run into an infinite loop.
You can refer to this article. This is an example cited from there:
from SimpleXMLRPCServer import *
class MyServer(SimpleXMLRPCServer):
def serve_forever(self):
self.quit = 0
while not self.quit:
self.handle_request()
def kill():
server.quit = 1
return 1
server = MyServer(('127.0.0.1', 8000))
server.register_function(kill)
server.serve_forever()
By using handle_request(), this code use a state variable self.quit to indicate whether to quit the infinite loop.
The serve_forever function is inherited from a base class in the socketserver module called BaseServer. If you look at this fucntion you'll see it has an attribute called __shutdown_request, and this can be used to break the serving while loop. Because of the double underscore you'll have to access the variable with its mangled name: _BaseServer__shutdown_request.
Putting that all together you can make a very simple quit function as follows:
from xmlrpc.server import SimpleXMLRPCServer
class MyXMLRPCServer(SimpleXMLRPCServer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.register_function(self.quit)
def quit(self):
self._BaseServer__shutdown_request = True
return 0
I have a small python webserver script for hosting my own website, complete with request handling and error returning. This script worked perfectly on my PC, but when I tried it on my raspberry Pi, it would not restart every 3 minutes (Server would crash after 15, so restarting every 3 minutes seemed good).
So I rewrote my server script and it checks stuff like if it's booting up for the first time or restarting. I'll just show you the code.
#Handler class above here
...
...
class Server:
global server_class, server_adress, httpd
server_class = HTTPServer
server_adress = ('localhost', 8080)
httpd = server_class(server_adress, Handler)
def __init__(self):
self.status = False
self.process()
def process(self):
print(self.status)
process = threading.Timer(10, self.process)
process.start()
if self.status == True:
httpd.socket.close()
self.main()
if self.status == False:
self.main()
def main(self):
try:
if self.status == False:
print("Server online!")
self.status = True
httpd.serve_forever()
if self.status == True:
print("Server restarted!")
httpd.serve_forever()
except KeyboardInterrupt:
print("Server shutting down...")
httpd.socket.close()
if __name__ == "__main__":
instance = Server()
After the ten seconds of running (And it works, I can access my website on http://localhost:8080/index.html), it will continue giving the following error every ten seconds:
File "C:\Users\myname\Dropbox\Python\Webserver\html\server.py", line 187, in main httpd.serve_forever()
File "C:\Python33\lib\socketserver.py", line 237, in serve_forever poll_interval)
File "C:\Python33\lib\socketserver.py", line 155, in _eintr_retry return func(*args)
ValueError: file descriptor cannot be a negative integer (-1)
Basically, how do I fix this? I could just use a simple function with a threading timer to restart the function that is running the server, but somehow that doesn't work on my Raspberry Pi, but it does on my windows.
EDIT:
I should also note that the first time starting the script I can access the website and it's fast. After 10 seconds (after the server restarting), I can access it but it is very slow. After another 10 seconds I am not able to access my website.
The problem you get happens because you access the underlying socket of the server directly. Closing the socket is effectively like unplugging your network connection. The actual server that is sitting on top of the socket remains unaware of the fact that the socket was closed, and tries to continue to serve. As the socket was closed, there is no longer a file descriptor available (this is the error you get).
So instead of cutting the server off its connection, you should tell the server to actually shut down gracefully. This allows it to finish any ongoing connections and safely release everything it might do in the background. You can do that using the shutdown method. Executing that will internally tell the server to remember to shut down the next time the loop within serve_forever occurs.
If I remember correctly, serve_forever is a blocking method, meaning that it will not continue when it is executed. So the simplest way to make a server restart itself would be a single main thread doing this:
while True:
httpd.serve_forever()
So whenever the server stops—for whatever reason—it immediately starts again. Of course here you would now add some status variable (instead of True) which allows you to actually turn off the server. For example in the body of a KeyboardInterrupt catch, you would first set that variable to False and then shut down the server using httpd.shutdown().
I have a class that I wish to test via SimpleXMLRPCServer in python. The way I have my unit test set up is that I create a new thread, and start SimpleXMLRPCServer in that. Then I run all the test, and finally shut down.
This is my ServerThread:
class ServerThread(Thread):
running = True
def run(self):
self.server = #Creates and starts SimpleXMLRPCServer
while (self.running):
self.server.handle_request()
def stop(self):
self.running = False
self.server.server_close()
The problem is, that calling ServerThread.stop(), followed by Thread.stop() and Thread.join() will not cause the thread to stop properly if it's already waiting for a request in handle_request. And since there doesn't seem to be any interrupt or timeout mechanisms here that I can use, I am at a loss for how I can cleanly shut down the server thread.
I had the same problem and after hours of research i solved it by switching from using my own handle_request() loop to serve_forever() to start the server.
serve_forever() starts an internal loop like yours. This loop can be stopped by calling shutdown(). After stopping the loop it is possible to stop the server with server_close().
I don't know why this works and the handle_request() loop don't, but it does ;P
Here is my code:
from threading import Thread
from xmlrpc.server import SimpleXMLRPCServer
from pyWebService.server.service.WebServiceRequestHandler import WebServiceRquestHandler
class WebServiceServer(Thread):
def __init__(self, ip, port):
super(WebServiceServer, self).__init__()
self.running = True
self.server = SimpleXMLRPCServer((ip, port),requestHandler=WebServiceRquestHandler)
self.server.register_introspection_functions()
def register_function(self, function):
self.server.register_function(function)
def run(self):
self.server.serve_forever()
def stop_server(self):
self.server.shutdown()
self.server.server_close()
print("starting server")
webService = WebServiceServer("localhost", 8010)
webService.start()
print("stopping server")
webService.stop_server()
webService.join()
print("server stopped")
Two suggestions.
Suggestion One is to use a separate process instead of a separate thread.
Create a stand-alone XMLRPC server program.
Start it with subprocess.Popen().
Kill it when the test is done. In standard OS's (not Windows) the kill works nicely. In Windows, however, there's no trivial kill function, but there are recipes for this.
The other suggestion is to have a function in your XMLRPC server which causes server self-destruction. You define a function that calls sys.exit() or os.abort() or raises a similar exception that will stop the process.
This is my way. send SIGTERM to self. (Works for me)
Server code
import os
import signal
import xmlrpc.server
server = xmlrpc.server.SimpleXMLRPCServer(("0.0.0.0", 8000))
server.register_function(lambda: os.kill(os.getpid(), signal.SIGTERM), 'quit')
server.serve_forever()
Client code
import xmlrpc.client
c = xmlrpc.client.ServerProxy("http://localhost:8000")
try:
c.quit()
except ConnectionRefusedError:
pass