I have a small python webserver script for hosting my own website, complete with request handling and error returning. This script worked perfectly on my PC, but when I tried it on my raspberry Pi, it would not restart every 3 minutes (Server would crash after 15, so restarting every 3 minutes seemed good).
So I rewrote my server script and it checks stuff like if it's booting up for the first time or restarting. I'll just show you the code.
#Handler class above here
...
...
class Server:
global server_class, server_adress, httpd
server_class = HTTPServer
server_adress = ('localhost', 8080)
httpd = server_class(server_adress, Handler)
def __init__(self):
self.status = False
self.process()
def process(self):
print(self.status)
process = threading.Timer(10, self.process)
process.start()
if self.status == True:
httpd.socket.close()
self.main()
if self.status == False:
self.main()
def main(self):
try:
if self.status == False:
print("Server online!")
self.status = True
httpd.serve_forever()
if self.status == True:
print("Server restarted!")
httpd.serve_forever()
except KeyboardInterrupt:
print("Server shutting down...")
httpd.socket.close()
if __name__ == "__main__":
instance = Server()
After the ten seconds of running (And it works, I can access my website on http://localhost:8080/index.html), it will continue giving the following error every ten seconds:
File "C:\Users\myname\Dropbox\Python\Webserver\html\server.py", line 187, in main httpd.serve_forever()
File "C:\Python33\lib\socketserver.py", line 237, in serve_forever poll_interval)
File "C:\Python33\lib\socketserver.py", line 155, in _eintr_retry return func(*args)
ValueError: file descriptor cannot be a negative integer (-1)
Basically, how do I fix this? I could just use a simple function with a threading timer to restart the function that is running the server, but somehow that doesn't work on my Raspberry Pi, but it does on my windows.
EDIT:
I should also note that the first time starting the script I can access the website and it's fast. After 10 seconds (after the server restarting), I can access it but it is very slow. After another 10 seconds I am not able to access my website.
The problem you get happens because you access the underlying socket of the server directly. Closing the socket is effectively like unplugging your network connection. The actual server that is sitting on top of the socket remains unaware of the fact that the socket was closed, and tries to continue to serve. As the socket was closed, there is no longer a file descriptor available (this is the error you get).
So instead of cutting the server off its connection, you should tell the server to actually shut down gracefully. This allows it to finish any ongoing connections and safely release everything it might do in the background. You can do that using the shutdown method. Executing that will internally tell the server to remember to shut down the next time the loop within serve_forever occurs.
If I remember correctly, serve_forever is a blocking method, meaning that it will not continue when it is executed. So the simplest way to make a server restart itself would be a single main thread doing this:
while True:
httpd.serve_forever()
So whenever the server stops—for whatever reason—it immediately starts again. Of course here you would now add some status variable (instead of True) which allows you to actually turn off the server. For example in the body of a KeyboardInterrupt catch, you would first set that variable to False and then shut down the server using httpd.shutdown().
Related
I am using pyGTK and a python gRPC server. Both in a very basic setup.
I just create a gtk.Window(), show() it and run a gtk.main() loop.
My server starts like that:
def startServing():
global server
print("Starting server...")
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
laserClient_pb2_grpc.add_LComServicer_to_server(LCom(), server)
server.add_insecure_port('[::]:50053') # [::] is the same as 0.0.0.0 server.start()
print("Server is running...")
So I call:
try:
startServing()
gtk.main()
except KeyboardInterrupt:
server.stop(0)
This creates the window correctly but I never receive a request from my Java client. (The java client is not the problem.)
I read a lot on the internet and I do not understand all of the pyGTK thread handling but I tried gtk.gdk.threads_init() right before startServing() and I received the requests. However I receive just 1 request per second whereas my client sends a request every 50ms. If I delete gtk.main() and add a while loop:
while True:
time.sleep(60)
...I receive requests nearly every 50 to 100ms. This is the expected behaviour!
However my window won't get updated since there is no gtk.main() loop anymore. I even tried adding:
while True:
while gtk.events_pending():
gtk.main_iteration()
time.sleep(0.05)
But this gives, again, just 1 request per second.
I have no idea what I should do now. I really want to use gRPC and pyGTK together in the same program.
I have an app similar to a chat-room writing in python that intends to do the following things:
A prompt for user to input websocket server address.
Then create a websocket client that connects to server and send/receive messages. Disable the ability to create a websocket client.
After receiving "close" from server (NOT a close frame), client should drop connecting and re-enable the app to create a client. Go back to 1.
If user exits the app, it exit the websocket client if there is one running.
My approach for this is using a main thread to deal with user input. When user hits enter, a thread is created for WebSocketClient using AutoBahn's twisted module and pass a Queue to it. Check if the reactor is running or not and start it if it's not.
Overwrite on message method to put a closing flag into the Queue when getting "close". The main thread will be busy checking the Queue until receiving the flag and go back to start. The code looks like following.
Main thread.
def main_thread():
while True:
text = raw_input("Input server url or exit")
if text == "exit":
if myreactor:
myreactor.stop()
break
msgq = Queue.Queue()
threading.Thread(target=wsthread, args=(text, msgq)).start()
is_close = False
while True:
if msgq.empty() is False:
msg = msgq.get()
if msg == "close":
is_close = True
else:
print msg
if is_close:
break
print 'Websocket client closed!'
Factory and Protocol.
class MyProtocol(WebSocketClientProtocol):
def onMessage(self, payload, isBinary):
msg = payload.decode('utf-8')
self.Factory.q.put(msg)
if msg == 'close':
self.dropConnection(abort=True)
class WebSocketClientFactoryWithQ(WebSocketClientFactory):
def __init__(self, *args, **kwargs):
self.queue = kwargs.pop('queue', None)
WebSocketClientFactory.__init__(self, *args, **kwargs)
Client thread.
def wsthread(url, q):
factory = WebSocketClientFactoryWithQ(url=url, queue=q)
factory.protocol = MyProtocol
connectWS(Factory)
if myreactor is None:
myreactor = reactor
reactor.run()
print 'Done'
Now I got a problem. It seems that my client thread never stops. Even if I receive "close", it seems still running and every time I try to recreate a new client, it creates a new thread. I understand the first thread won't stop since reactor.run() will run forever, but from the 2nd thread and on, it should be non-blocking since I'm not starting it anymore. How can I change that?
EDIT:
I end up solving it with
Adding stopFactory() after disconnect.
Make protocol functions with reactor.callFromThread().
Start the reactor in the first thread and put clients in other threads and use reactor.callInThread() to create them.
Your main_thread creates new threads running wsthread. wsthread uses Twisted APIs. The first wsthread becomes the reactor thread. All subsequent threads are different and it is undefined what happens if you use a Twisted API from them.
You should almost certainly remove the use of threads from your application. For dealing with console input in a Twisted-based application, take a look at twisted.conch.stdio (not the best documented part of Twisted, alas, but just what you want).
I'm trying to run a django development server from within a Kivy application. This did work out quite well so far.
Now i want to allow the user to continue working with the program while the server is running. My idea was to create a multiprocessing.Process for the httpd.serve_forever() to avoid a complete lock of the main program. Did work well. This is the code in my internal_django module:
import multiprocessing
import os
import time
from wsgiref.simple_server import make_server
def django_wsgi_application():
PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
settings_module = "djangosettings"#%s.djangosettings" % PROJECT_ROOT.split(os.sep)[-1]
os.environ.update({"DJANGO_SETTINGS_MODULE":settings_module})
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
return application
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
class DjangoServer():
__metaclass__ = Singleton
def start(self):
self.httpd = make_server('', 8000, django_wsgi_application())
self.server = multiprocessing.Process(target=self.httpd.serve_forever)
self.server.start()
print "Now serving on port 8000..."
print "Server Process PID = %s" %self.server.pid
def stop(self):
print("shutdown initiated")
print "Server Process PID = %s" %self.server.pid
while self.server.is_alive():
self.server.terminate()
print("Server should have shut down")
time.sleep(1)
print("Server is_alive: %s" %self.server.is_alive())
self.server.join()
print("server process joined")
if __name__ == "__main__":
server = DjangoServer()
server.start()
time.sleep(3)
server.stop()
When i run this code, everything works as expected. This is what is being put out in the console:
Now serving on port 8000...
Server Process PID = 1406
shutdown initiated
Server Process PID = 1406
Server should have shut down
Server is_alive: False
server process joined
Next step was to provide a way to stop the server from within the Kivy application. For that i just wanted to use my DjangoServer class as i did before:
from internal_django import DjangoServer
class StartScreen(Screen):
def start_server(self):
server = DjangoServer()
server.start()
class StopScreen(Screen):
def stop_server(self):
server = DjangoServer()
server.stop()
But when doing so, the process once started never quits. My first idea was that the Singleton did not work as expected, and that i try to quit the wrong process. but as you can see in the output, the PID's are identical. The server receives the terminate command, but just continues to work. This is what the console looks like:
Now serving on port 8000...
Server Process PID = 1406
shutdown initiated
Server Process PID = 1406
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
(and so on, until i manually kill the server process)
Am i using multiprocessing in a completely wrong way? Is Kivy somehow interfering with the process?
I think the problems here might be two:
A signal handler is intercepting the TERM request sent by Process.terminate() and ignores it. To verify that simply use the signal.getsignal(signal.SIGTERM) from within the new process and print the results. To circumvent such issue you can reset the default behavior with signal.signal(signal.SIGTERM, signal.SIG_DFL), nevertheless keep in mind that there might be a reason why SIGTERM is silenced by the frameworks (I'm not familiar neither with Django nor with Kivy).
If you're using Python 2 you must consider that the interpreter does not process signals if it's blocked on a synchronization primitive from threading library (Locks, Semaphores..) or on a native C call. The serve_forever() function might fall in these cases (use the force of the source!). Quick check could be trying to run the code on Python 3 and see whether it works or not.
A quick and dirty solution consists in waiting a small amount of time and send a SIGKILL if the process is still alive.
import os
import signal
process.terminate()
process.join(1)
if process.is_alive() and os.name != 'nt':
try:
os.kill(process.pid, signal.SIGKILL)
process.join()
except OSError:
return # process might have died while checking it
On windows you cannot kill a process in such simple way that's why I test the os.name.
It's a pretty raw approach so I'd rather recommend to find the cause of the issue.
What happens if you call terminate(), then join() and skip the while loop? Also, I shuffle the code a little and factor some code into _create_server(). Please let me know if this works out for you.
class DjangoServer():
__metaclass__ = Singleton
def _create_server(self):
httpd = make_server('', 8000, django_wsgi_application())
print "Now serving on port {}...".format(httpd.server_port)
httpd.serve_forever()
def start(self):
self.server = multiprocessing.Process(target=self._create_server)
self.server.start()
print "Server Process PID = %s" %self.server.pid
def stop(self):
print("shutdown initiated")
print "Server Process PID = %s" %self.server.pid
self.server.terminate()
self.server.join()
print("server process terminated")
I am writing a simple client-server program in python. In the client program, I am creating two threads (using Python's threading module), one for receiving, one for sending. The receiving thread continuously receives strings from the server side; while the sending thread continuously listens to the user input (using raw_input()) and send it to the server side. The two threads communicate using a Queue (which is natively synchronized, LIKE!).
The basic logic is like following:
Receiving thread:
global queue = Queue.Queue(0)
def run(self):
while 1:
receive a string from the server side
if the string is QUIT signal:
sys.exit()
else:
put it into the global queue
Sending thread:
def run(self):
while 1:
str = raw_input()
send str to the server side
fetch an element from the global queue
deal with the element
As you can see, in the receiving thread, I have a if condition to test whether the server has sent a "QUIT signal" to the client. If it has, then I want the whole program to stop.
The problem here is that for most of its time, the sending thread is blocked by "raw_input()" and waiting for the user input. When it is blocked, calling "sys.exit()" from the other thread (receiving thread) will not terminate the sending thread immediately. The sending thread has to wait for the user to type something and hit the enter button.
Could anybody inspire me how to get around with this? I do not mind using alternatives of "raw_input()". Actually I do not even mind changing the whole structure.
-------------EDIT-------------
I am running this on a linux machine, and my Python version is 2.7.5
You could just make the sending thread daemonic:
send_thread = SendThread() # Assuming this inherits from threading.Thread
send_thread.daemon = True # This must be called before you call start()
The Python interpreter won't be blocked from exiting if the only threads left running are daemons. So, if the only thread left is send_thread, your program will exit, even if you're blocked on raw_input.
Note that this will terminate the sending thread abruptly, no matter what its doing. This could be dangerous if it accesses external resources that need to be cleaned up properly or shouldn't be interrupted (like writing to a file, for example). If you're doing anything like that, protect it with a threading.Lock, and only call sys.exit() from the receiving thread if you can acquire that same Lock.
The short answer is you can't. input() like a lot of such input commands is blocking and it's blocking whether everything about the thread has been killed. You can sometimes call sys.exit() and get it to work depending on the OS, but it's not going to be consistent. Sometimes you can kill the program by deferring out to the local OS. But, then you're not going to be widely cross platform.
What you might want to consider if you have this is to funnel the functionality through the sockets. Because unlike input() we can do timeouts, and threads and kill things rather easily. It also gives you the ability to do multiple connections and maybe accept connections more broadly.
import socket
import time
from threading import Thread
def process(command, connection):
print("Command Entered: %s" % command)
# Any responses are written to connection.
connection.send(bytes('>', 'utf-8'))
class ConsoleSocket:
def __init__(self):
self.keep_running_the_listening_thread = True
self.data_buffer = ''
Thread(target=self.tcp_listen_handle).start()
def stop(self):
self.keep_running_the_listening_thread = False
def handle_tcp_connection_in_another_thread(self, connection, addr):
def handle():
while self.keep_running_the_listening_thread:
try:
data_from_socket = connection.recv(1024)
if len(data_from_socket) != 0:
self.data_buffer += data_from_socket.decode('utf-8')
else:
break
while '\n' in self.data_buffer:
pos = self.data_buffer.find('\n')
command = self.data_buffer[0:pos].strip('\r')
self.data_buffer = self.data_buffer[pos + 1:]
process(command, connection)
except socket.timeout:
continue
except socket.error:
if connection is not None:
connection.close()
break
Thread(target=handle).start()
connection.send(bytes('>', 'utf-8'))
def tcp_listen_handle(self, port=23, connects=5, timeout=2):
"""This is running in its own thread."""
sock = socket.socket()
sock.settimeout(timeout)
sock.bind(('', port))
sock.listen(connects) # We accept more than one connection.
while self.keep_running_the_listening_thread:
connection = None
try:
connection, addr = sock.accept()
address, port = addr
if address != '127.0.0.1': # Only permit localhost.
connection.close()
continue
# makes a thread deals with that stuff. We only do listening.
connection.settimeout(timeout)
self.handle_tcp_connection_in_another_thread(connection, addr)
except socket.timeout:
pass
except OSError:
# Some other error.
if connection is not None:
connection.close()
sock.close()
c = ConsoleSocket()
def killsocket():
time.sleep(20)
c.stop()
Thread(target=killsocket).start()
This launches a listener thread for the connections set on port 23 (telnet), and you connect and it passes that connection off to another thread. And it starts a killsocket thread that disables the various threads and lets them die peacefully (for demonstration purposes). You cannot however connect localhost within this code, because you'd need input() to know what to send to the server, which recreates the problem.
I have a class that I wish to test via SimpleXMLRPCServer in python. The way I have my unit test set up is that I create a new thread, and start SimpleXMLRPCServer in that. Then I run all the test, and finally shut down.
This is my ServerThread:
class ServerThread(Thread):
running = True
def run(self):
self.server = #Creates and starts SimpleXMLRPCServer
while (self.running):
self.server.handle_request()
def stop(self):
self.running = False
self.server.server_close()
The problem is, that calling ServerThread.stop(), followed by Thread.stop() and Thread.join() will not cause the thread to stop properly if it's already waiting for a request in handle_request. And since there doesn't seem to be any interrupt or timeout mechanisms here that I can use, I am at a loss for how I can cleanly shut down the server thread.
I had the same problem and after hours of research i solved it by switching from using my own handle_request() loop to serve_forever() to start the server.
serve_forever() starts an internal loop like yours. This loop can be stopped by calling shutdown(). After stopping the loop it is possible to stop the server with server_close().
I don't know why this works and the handle_request() loop don't, but it does ;P
Here is my code:
from threading import Thread
from xmlrpc.server import SimpleXMLRPCServer
from pyWebService.server.service.WebServiceRequestHandler import WebServiceRquestHandler
class WebServiceServer(Thread):
def __init__(self, ip, port):
super(WebServiceServer, self).__init__()
self.running = True
self.server = SimpleXMLRPCServer((ip, port),requestHandler=WebServiceRquestHandler)
self.server.register_introspection_functions()
def register_function(self, function):
self.server.register_function(function)
def run(self):
self.server.serve_forever()
def stop_server(self):
self.server.shutdown()
self.server.server_close()
print("starting server")
webService = WebServiceServer("localhost", 8010)
webService.start()
print("stopping server")
webService.stop_server()
webService.join()
print("server stopped")
Two suggestions.
Suggestion One is to use a separate process instead of a separate thread.
Create a stand-alone XMLRPC server program.
Start it with subprocess.Popen().
Kill it when the test is done. In standard OS's (not Windows) the kill works nicely. In Windows, however, there's no trivial kill function, but there are recipes for this.
The other suggestion is to have a function in your XMLRPC server which causes server self-destruction. You define a function that calls sys.exit() or os.abort() or raises a similar exception that will stop the process.
This is my way. send SIGTERM to self. (Works for me)
Server code
import os
import signal
import xmlrpc.server
server = xmlrpc.server.SimpleXMLRPCServer(("0.0.0.0", 8000))
server.register_function(lambda: os.kill(os.getpid(), signal.SIGTERM), 'quit')
server.serve_forever()
Client code
import xmlrpc.client
c = xmlrpc.client.ServerProxy("http://localhost:8000")
try:
c.quit()
except ConnectionRefusedError:
pass