I have an application in python with a custom main loop (I don't believe the details are important). I'd like to integrate a simple non-blocking web server into the application which can introspect the application objects and possibly provide an interface to manipulate them. What's the best way to do this?
I'd like to avoid anything that uses threading. The ideal solution would be a server with a "stepping" function that can be called from my main loop, do its thing, then return program control until the next go-round.
The higher-level the solution, the better (though something as monolithic as Django might be overkill).
Ideally, a solution will look like this:
def main():
"""My main loop."""
http_server = SomeCoolHttpServer(port=8888)
while True:
# Do my stuff here...
# ...
http_server.next() # Server gets it's turn.
# Do more of my stuff here...
# ...
Twisted is designed to make stuff like that fairly simple
import time
from twisted.web import server, resource
from twisted.internet import reactor
class Simple(resource.Resource):
isLeaf = True
def render_GET(self, request):
return "<html>%s Iterations!</html>"%n
def main():
global n
site = server.Site(Simple())
reactor.listenTCP(8080, site)
reactor.startRunning(False)
n=0
while True:
n+=1
if n%1000==0:
print n
time.sleep(0.001)
reactor.iterate()
if __name__=="__main__":
main()
I'd suggest creating a new thread and running a web server (such as Python's built-in SimpleHTTPServer or BaseHTTPServer). Threads really aren't that scary when it comes down to it.
from threading import Event, Thread
import BaseHTTPServer
shut_down = Event()
def http_server():
server_address = ('', 8000)
httpd = BaseHTTPServer.HTTPServer(server_address, BaseHTTPServer.BaseHTTPRequestHandler)
while not shut_down.is_set():
httpd.handle_request()
thread = Thread(target=http_server)
thread.start()
Related
I'm writing a small web server in Python, using BaseHTTPServer and a custom subclass of BaseHTTPServer.BaseHTTPRequestHandler. Is it possible to make this listen on more than one port?
What I'm doing now:
class MyRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def doGET
[...]
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
pass
server = ThreadingHTTPServer(('localhost', 80), MyRequestHandler)
server.serve_forever()
Sure; just start two different servers on two different ports in two different threads that each use the same handler. Here's a complete, working example that I just wrote and tested. If you run this code then you'll be able to get a Hello World webpage at both http://localhost:1111/ and http://localhost:2222/
from threading import Thread
from SocketServer import ThreadingMixIn
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-type", "text/plain")
self.end_headers()
self.wfile.write("Hello World!")
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
daemon_threads = True
def serve_on_port(port):
server = ThreadingHTTPServer(("localhost",port), Handler)
server.serve_forever()
Thread(target=serve_on_port, args=[1111]).start()
serve_on_port(2222)
update:
This also works with Python 3 but three lines need to be slightly changed:
from socketserver import ThreadingMixIn
from http.server import HTTPServer, BaseHTTPRequestHandler
and
self.wfile.write(bytes("Hello World!", "utf-8"))
Not easily. You could have two ThreadingHTTPServer instances, write your own serve_forever() function (don't worry it's not a complicated function).
The existing function:
def serve_forever(self, poll_interval=0.5):
"""Handle one request at a time until shutdown.
Polls for shutdown every poll_interval seconds. Ignores
self.timeout. If you need to do periodic tasks, do them in
another thread.
"""
self.__serving = True
self.__is_shut_down.clear()
while self.__serving:
# XXX: Consider using another file descriptor or
# connecting to the socket to wake this up instead of
# polling. Polling reduces our responsiveness to a
# shutdown request and wastes cpu at all other times.
r, w, e = select.select([self], [], [], poll_interval)
if r:
self._handle_request_noblock()
self.__is_shut_down.set()
So our replacement would be something like:
def serve_forever(server1,server2):
while True:
r,w,e = select.select([server1,server2],[],[],0)
if server1 in r:
server1.handle_request()
if server2 in r:
server2.handle_request()
I would say that threading for something this simple is overkill. You're better off using some form of asynchronous programming.
Here is an example using Twisted:
from twisted.internet import reactor
from twisted.web import resource, server
class MyResource(resource.Resource):
isLeaf = True
def render_GET(self, request):
return 'gotten'
site = server.Site(MyResource())
reactor.listenTCP(8000, site)
reactor.listenTCP(8001, site)
reactor.run()
I also thinks it looks a lot cleaner to have each port be handled in the same way, instead of having the main thread handle one port and an additional thread handle the other. Arguably that can be fixed in the thread example, but then you're using three threads.
Currently I am writing an application using the SimpleXMLRPCServer module in Python.
The basic aim of this application is to keep running on a server and keep checking a Queue for any task. If it encounters any new request in the Queue, serve the request.
Snapshot of what I am trying to do :
class MyClass():
"""
This class will have methods which will be exposed to the clients
"""
def __init__(self):
taskQ = Queue.Queue()
def do_some_task(self):
while True:
logging.info("Checking the Queue for any Tasks..")
task = taskQ.get()
# Do some processing based on the availability of some task
Main
if name == "main":
server = SimpleXMLRPCServer.SimpleXMLRPCServer((socket.gethostname(), Port)
classObj = MyClass()
rpcserver.register_function(classObj.do_some_task)
rpcserver.serve_forever()
Once the server is started it remains in the loop forever inside do_some_task method to keep checking the Queue for any task. This is what i wanted to achieve. But now i want to gracefully shutdown the server. In this case i am unable to shutdown the server.
Till now I have Tried using a global flag STOP_SERVER for 'True' and checking its status in the do_some_task while loop to get out of it and stop the server. But no help.
Tried using SHUTDOWN() method of the SimpleXMLRPCServer but it seems it is getting into a infinite loop of somekind.
Could you suggest some proper way to gracefully shutdown the server.
Thanks in advance
You should use handle_request() instead of serve_forever() if you want to close it manualy. Because SimpleXMLRPCServer is implemented as a single thread and the serve_forever() will make the server instance run into an infinite loop.
You can refer to this article. This is an example cited from there:
from SimpleXMLRPCServer import *
class MyServer(SimpleXMLRPCServer):
def serve_forever(self):
self.quit = 0
while not self.quit:
self.handle_request()
def kill():
server.quit = 1
return 1
server = MyServer(('127.0.0.1', 8000))
server.register_function(kill)
server.serve_forever()
By using handle_request(), this code use a state variable self.quit to indicate whether to quit the infinite loop.
The serve_forever function is inherited from a base class in the socketserver module called BaseServer. If you look at this fucntion you'll see it has an attribute called __shutdown_request, and this can be used to break the serving while loop. Because of the double underscore you'll have to access the variable with its mangled name: _BaseServer__shutdown_request.
Putting that all together you can make a very simple quit function as follows:
from xmlrpc.server import SimpleXMLRPCServer
class MyXMLRPCServer(SimpleXMLRPCServer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.register_function(self.quit)
def quit(self):
self._BaseServer__shutdown_request = True
return 0
I am using Tornado and I want to monitor a socket continuously for notifications from a DB server. My application so far looks like this:
import functools
import tornado
import tornado.httpserver
from tornado.ioloop import IOLoop
class Application(tornado.web.Application):
def __init__(self):
handlers = [(r"/", MyHandler),]
super(Application, self).__init__(handlers)
fd = get_socket_file_descriptor()
callback = functools.partial(self.my_callback)
io_loop = IOLoop.current()
io_loop.add_handler(fd, callback, io_loop.READ)
def my_callback(self, fd, events):
# do something
pass
if __name__ == '__main__':
http_server = tornado.httpserver.HTTPServer(Application())
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()
My problem is this as soon as there is activity on the socket, the callback is called infinitely. I want the IOLoop to handle the call back and go back to listening on to the file descriptor.
The IOLoop will call your handler repeatedly as long as there is data to be read. Your callback must consume all the data in the socket to allow it to become idle again.
Your callback must call io_loop.remove_handler(fd). But consider connecting an IOStream to the file descriptor for a much more convenient and higher-level interface. You'd attach a callback to IOStream.read_bytes:
http://tornado.readthedocs.org/en/latest/iostream.html#tornado.iostream.BaseIOStream.read_bytes
Consider passing partial=True or a streaming_callback if you don't know what message length to expect, or use read_bytes with a length argument, or read_until_regex if you have some knowledge of when the end of the message will be.
I want to start a simple web server locally, then launch a browser with an url just served. This is something that I'd like to write,
from wsgiref.simple_server import make_server
import webbrowser
srv = make_server(...)
srv.blocking = False
srv.serve_forever()
webbrowser.open_new_tab(...)
try:
srv.blocking = True
except KeyboardInterrupt:
pass
print 'Bye'
The problem is, I couldn't find a way to set a blocking option for the wsgiref simple server. By default, it's blocking, so the browser would be launched only after I stopped it. If I launch the browser first, the request is not handled yet. I'd prefer to use a http server from the standard library, not an external one, like tornado.
You either have to spawn a thread with the server, so you can continue with your control flow, or you have to use 2 python processes.
untested code, you should get the idea
class ServerThread(threading.Thread):
def __init__(self, port):
threading.Thread.__init__(self)
def run(self):
srv = make_server(...)
srv.serve_forever()
if '__main__'==__name__:
ServerThread().start()
webbrowser.open_new_tab(...)
I have a class that I wish to test via SimpleXMLRPCServer in python. The way I have my unit test set up is that I create a new thread, and start SimpleXMLRPCServer in that. Then I run all the test, and finally shut down.
This is my ServerThread:
class ServerThread(Thread):
running = True
def run(self):
self.server = #Creates and starts SimpleXMLRPCServer
while (self.running):
self.server.handle_request()
def stop(self):
self.running = False
self.server.server_close()
The problem is, that calling ServerThread.stop(), followed by Thread.stop() and Thread.join() will not cause the thread to stop properly if it's already waiting for a request in handle_request. And since there doesn't seem to be any interrupt or timeout mechanisms here that I can use, I am at a loss for how I can cleanly shut down the server thread.
I had the same problem and after hours of research i solved it by switching from using my own handle_request() loop to serve_forever() to start the server.
serve_forever() starts an internal loop like yours. This loop can be stopped by calling shutdown(). After stopping the loop it is possible to stop the server with server_close().
I don't know why this works and the handle_request() loop don't, but it does ;P
Here is my code:
from threading import Thread
from xmlrpc.server import SimpleXMLRPCServer
from pyWebService.server.service.WebServiceRequestHandler import WebServiceRquestHandler
class WebServiceServer(Thread):
def __init__(self, ip, port):
super(WebServiceServer, self).__init__()
self.running = True
self.server = SimpleXMLRPCServer((ip, port),requestHandler=WebServiceRquestHandler)
self.server.register_introspection_functions()
def register_function(self, function):
self.server.register_function(function)
def run(self):
self.server.serve_forever()
def stop_server(self):
self.server.shutdown()
self.server.server_close()
print("starting server")
webService = WebServiceServer("localhost", 8010)
webService.start()
print("stopping server")
webService.stop_server()
webService.join()
print("server stopped")
Two suggestions.
Suggestion One is to use a separate process instead of a separate thread.
Create a stand-alone XMLRPC server program.
Start it with subprocess.Popen().
Kill it when the test is done. In standard OS's (not Windows) the kill works nicely. In Windows, however, there's no trivial kill function, but there are recipes for this.
The other suggestion is to have a function in your XMLRPC server which causes server self-destruction. You define a function that calls sys.exit() or os.abort() or raises a similar exception that will stop the process.
This is my way. send SIGTERM to self. (Works for me)
Server code
import os
import signal
import xmlrpc.server
server = xmlrpc.server.SimpleXMLRPCServer(("0.0.0.0", 8000))
server.register_function(lambda: os.kill(os.getpid(), signal.SIGTERM), 'quit')
server.serve_forever()
Client code
import xmlrpc.client
c = xmlrpc.client.ServerProxy("http://localhost:8000")
try:
c.quit()
except ConnectionRefusedError:
pass