We Keep Coding

Python Process won't stop by calling kill method

So I'm trying having a self-made led-controller (raspberry pi).
The Controller should be able to play different scenes which were pre-defined by myself.
Now to the main problem...
The controller runs as TCP server and gets his scene-changes by tcp messages.
i coded a lot of scenes which need to run in an endless while loop.
So I decided to use multiprocessing to be able killing the running "scene process" if a new tcp message arrives.
So the awkward result of my two scripts are that if I'm running the server script on windows its perfectly working, but if im changing to run the server script on raspberry pi the running process isn't getting killed like it should.
so as my server test script I used the following:
import multiprocessing
import time
from time import sleep
try:
from analyse import *
from handler import *
from history import writeState
from led import *
except: pass
import socket
from subprocess import check_output #um die IP-Adresse unter Raspian auszulesen
from platform import system
class SYSINFO():
os=system() #Fragt das System nach dem laufenden OS-Typ ab
if os=="Linux":
IP_ADDRESS=check_output(["hostname",'-I']).decode().split(" ")[0]
elif os=="Windows" or os=="Darwin":
IP_ADDRESS= socket.gethostbyname(socket.gethostname())#"192.168.168.97"
class MyProcess(multiprocessing.Process):
def __init__(self, ):
multiprocessing.Process.__init__(self)
self.exit = multiprocessing.Event()
def run(self):
while not self.exit.is_set():
print(round(time.perf_counter()), self.pid)
time.sleep(1)
print("You exited!")
def shutdown(self):
print("Shutdown initiated")
self.exit.set()
class TCPController(multiprocessing.Process):
def __init__(self, ):
multiprocessing.Process.__init__(self)
self.exit = multiprocessing.Event()
def run(self):
counter=0
def shutdown(self):
print("Shutdown initiated")
self.exit.set()
if __name__ == "__main__":
HOST = SYSINFO.IP_ADDRESS # Standard loopback interface address (localhost
PORT = 6060 # Port to listen on (non-privileged ports are > 1023)
while True:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
print(f"server listens under {HOST!r} , {PORT!r} now")
s.listen()
while True:
try:
conn, addr = s.accept()
print("waiting for connection")
with conn:
print(f"Connected by {addr}")
data = conn.recv(1024).decode()
print(data)
if data=="on":
process = MyProcess()
process.daemon=True
process.start()
time.sleep(3)
elif data=="off":
#process.shutdown()
process.kill()
time.sleep(3)
print("Child process state: %d" % process.is_alive())
except: pass
sleep(.5)
my client test cycle script looks like that
# echo-client.py
import socket
from time import sleep
class heimkinoSteuereinheit:
HOST = "192.168.168.97" #"192.168.168.97" # The server's hostname or IP address
PORT = 6060 # The port used by the server
def cinemaclient(msg):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((heimkinoSteuereinheit.HOST, heimkinoSteuereinheit.PORT))
s.sendall(msg.encode())
data = s.recv(1024).decode()
print(f"Received {data!r}")
return data
while True:
sleep(1)
cinemaclient("on")
sleep(5)
cinemaclient("off")
Hope you guys could help.
Thanks for your help,
Luca

The variable process is only defined in the
if data=="on"
While you use the variable process in the
if data=="off
It has not been defined. Is that done intentionally?
Furthermore what do you mean by the code isn't working. Do you get any errors?

Are Python socket objects thread safe?

I've planning on designing a UDP server that works as follows: I will be communicating with clients that are behind a firewall and need to be able to send data at any time. Therefore, the client will first initiate a connection to me and periodically keep the connection alive by regularly sending keep alive packets. When I receive one, I need to acknowledge it. At the same time, if I have any data to send, I need to immediately send it. I've put together the following test code:
import threading
import queue
import socket
import time
class SharedAddress():
def __init__(self):
self.lock = threading.Lock()
self.addr = ()
def set_addr(self, addr):
self.lock.acquire()
self.addr = addr
self.lock.release()
def get_addr(self):
self.lock.acquire()
addr = self.addr
self.lock.release()
return addr
class Reader(threading.Thread):
def __init__(self, socket, shared_address):
super().__init__(name='Reader Thread')
self.socket = socket
self.shared_address = shared_address
def run(self):
while True:
# Wait for data from the client
data, addr = self.socket.recvfrom(4096)
#print("Received data from {}".format(addr))
# Echo it back
self.socket.sendto(data, addr)
self.shared_address.set_addr(addr)
class Writer(threading.Thread):
def __init__(self, socket, shared_address):
super().__init__(name='Writer Thread')
self.socket = socket
self.tx_queue = queue.Queue()
self.shared_address = shared_address
def run(self):
while True:
# Wait for data to be received
data = self.tx_queue.get()
# Send it to the client
addr = self.shared_address.get_addr()
if addr:
self.socket.sendto(data, addr)
### Main loop
# Create the socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('', 2000))
sa = SharedAddress()
r = Reader(s, sa)
w = Writer(s, sa)
r.start()
w.start()
while True:
w.tx_queue.put(b'>>Hi this is a message<<\n')
time.sleep(0.1)
r.join()
w.join()
print("Program ended")
Although the code appears to work, I'm concerned about the fact that I'm using the same socket object from two different threads without any sort of lock. I then modified the Writer class to create its own socket object:
class Writer(threading.Thread):
def __init__(self, shared_address):
super().__init__(name='Writer Thread')
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.tx_queue = queue.Queue()
self.shared_address = shared_address
This also seems to work just fine. My questions are as follows:
Are socket objects thread safe?
If you create multiple UDP socket objects in Python and use them to send data to same address, do they actually end up referencing the same underlying connection object?
What happens if I call the close command on one of the socket objects? Presumably it will close the underlying OS socket and prevent the other socket objects from then receiving and transmitting.

sockets with threadpool server python

I have a simple multithreading server, But it creates a new thread for each socket, I don't want to create a lot of threads. My idea is to receive the messages in other way: when the user send a message, it will add the message to a queue of messages and with a threadpool the server will handle these requests.
The simple multithreaded server:
import socket
import threading
class ThreadedServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((self.host, self.port))
def listen(self):
self.sock.listen(5)
while True:
client, address = self.sock.accept()
client.settimeout(60)
threading.Thread(target = self.listenToClient,args = (client,address)).start()
def listenToClient(self, client, address):
size = 1024
while True:
try:
data = client.recv(size)
if data:
# Set the response to echo back the recieved data
response = data
client.send(response)
else:
raise error('Client disconnected')
except:
client.close()
return False
if __name__ == "__main__":
port_num = input("Port? ")
ThreadedServer('',port_num).listen()
How can I implement my idea or is there better way to do it?

The question seems to be pretty old but i also stumble upon the same issue while working on the socket server, so here is the below code which you can use to make threaded socket server which doesnt spawn new threads on arrival.
Just to give gist ThreadingMixIn classes is overided with threaded pool.
class ThreadPoolMixIn(socketserver.ThreadingMixIn):
'''
use a thread pool instead of a new thread on every request
'''
# numThreads = 50
allow_reuse_address = True # seems to fix socket.error on server restart
def serve_forever(self):
'''
Handle one request at a time until doomsday.
'''
print('[X] Server is Running with No of Threads :- {}'.format(self.numThreads))
# set up the threadpool
self.requests = Queue(self.numThreads)
for x in range(self.numThreads):
t = threading.Thread(target = self.process_request_thread)
t.setDaemon(1)
t.start()
# server main loop
while True:
self.handle_request()
self.server_close()
def process_request_thread(self):
'''
obtain request from queue instead of directly from server socket
'''
while True:
socketserver.ThreadingMixIn.process_request_thread(self, *self.requests.get())
def handle_request(self):
'''
simply collect requests and put them on the queue for the workers.
'''
try:
request, client_address = self.get_request()
except socket.error:
return
if self.verify_request(request, client_address):
self.requests.put((request, client_address))
And then it is called in ThreadedTCPRequest Handler and override the numThreads parameter :
class ThreadedTCPServer(ThreadPoolMixIn, socketserver.TCPServer):
#Extend base class and overide the thread paramter to control the number of threads.
def __init__(self, no_of_threads, server_address, ThreadedTCPRequestHandler):
self.numThreads = no_of_threads
super().__init__(server_address, ThreadedTCPRequestHandler)
Ultimately creating the server which serves forever :
def create_multi_threaded_socket(CONFIG, HandlerClass = ThreadedTCPRequestHandler,
ServerClass = ThreadedTCPServer,
protocol="HTTP/1.0"):
server_address = ('', CONFIG.port)
HandlerClass.protocol_version = protocol
# httpd = ServerClass(server_address, HandlerClass)
server = ThreadedTCPServer(CONFIG.no_of_threads, server_address, ThreadedTCPRequestHandler)
sa = server.socket.getsockname()
print("Serving HTTP on {} port : {}".format(sa[0], sa[1]))
server.serve_forever()
I got the sample code from :
http://code.activestate.com/recipes/574454-thread-pool-mixin-class-for-use-with-socketservert/
Modified bit according to my need.
Hope this helps :) .

How to exit a multithreaded program?

I was just messing around with threading in python, wrote this basic IM thingy [code at bottom]
I noticed that when I kill the program with C-c it doesn't exit, it just hangs forever.
I'm just guessing it's waiting for each thread to finish what they are doing, but since it's an endless loop that will never happen.
So I guess I need to kill each thread manually, or end the loop when the killsignal comes in.
How would I do that?
#!/usr/bin/env python
import threading
import socket
class Listen(threading.Thread):
def run(self):
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.bind(('', 2727))
conn.listen(1)
while True:
channel, details = conn.accept()
print str(details)+": "+channel.recv(250)
channel.send("got it")
channel.close()
class Shout(threading.Thread):
def run(self):
while True:
try:
address = raw_input("who u talking to? ")
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.connect((address, 2727))
break
except:
print "can't connect to "+ str(address)
while True:
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.connect((address, 2727))
conn.send(raw_input())
conn.close()
listen = Listen().start()
shout = Shout().start()

I see several causes of the misbehavior in your code.
Ctrl+C causes a "KeyboardInterrupt" exception in the main thread. So you should handle it there.
Your socket is in blocking mode. This causes several socket functions to block the calling thread until the function returns. During this state the thread cannot react to any termination event.
As you already said: your endless loop in the thread's run() function is ... really endless. So the thread execution is never ending (at least not without an unexpected exception). You should use some kind of synchronization object, like an threading.Event object, to be able to tell a thread externally that it should terminate itself.
I would discourage the use of raw_input() out of the main thread. Imagine what happens when you have more than one Shout thread.
Why are you always closing and reconnecting the socket when a message has been transmitted in your Shout class? Network connections should be re-established only in special cases because of the setup costs.
Without a frame protocol for the communication you can never expect to have received all data that was sent by the other host when the recv() function returns.
The start() function of the thread object does not return a value or object. So saving the returned value (=None) doesn't make much sense.
You can never expect the send() function to transmit all passed data. Therefore you must check the result of the function and appropriately handle the situation when not all bytes were really transmitted.
To learn threading there are surely better problems to solve than network communication, since that topic is in itself really complex.
Beside all these things, here is my try for a solution. Still there is much that can be improved. You should consider the answer from Mark Tolonen too, since the SocketServer class is surely provided to ease several things in handling this kind of stuff. But you should keep on studying the basics too.
#!/usr/bin/env python
import threading
import socket
import time
import errno
class StoppableThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.stop_event = threading.Event()
def stop(self):
if self.isAlive() == True:
# set event to signal thread to terminate
self.stop_event.set()
# block calling thread until thread really has terminated
self.join()
class Accept(StoppableThread):
def __init__(self, port):
StoppableThread.__init__(self)
self.port = port
self.threads = []
def run(self):
# handle connection acception
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.bind(('', self.port ))
conn.listen(5)
# set socket timeout to ~10ms
conn.settimeout(0.01)
while self.stop_event.is_set() == False:
try:
csock, caddr = conn.accept()
# spawn a new thread to handle the client connection
listen_thread = Listen(csock, caddr)
self.threads.append(listen_thread)
listen_thread.start()
except socket.timeout:
# socket operation timeout
# clear all terminated threads from thread list
for thread in self.threads:
if thread.isAlive() == False:
self.threads.remove(thread)
self.stop_threads()
def stop_threads(self):
# stop all running threads
for listen_thread in self.threads:
if listen_thread.isAlive() == True:
listen_thread.stop()
self.threads = []
class Listen(StoppableThread):
def __init__(self, csock, caddr):
StoppableThread.__init__(self)
self.csock = csock
self.caddr = caddr
self.csock.setblocking(False)
def run(self):
while self.stop_event.is_set() == False:
try:
recv_data = self.csock.recv(250)
if len(recv_data) > 0:
print str(self.caddr)+": " + recv_data
self.csock.send("got it")
else:
# connection was closed by foreign host
self.stop_event.set()
except socket.error as (sock_errno, sock_errstr):
if (sock_errno == errno.EWOULDBLOCK):
# socket would block - sleep sometime
time.sleep(0.1)
else:
# unexpected / unhandled error - terminate thread
self.stop_event.set()
channel.close()
class Shout(StoppableThread):
def __init__(self, sport):
StoppableThread.__init__(self)
self.sport = sport
def run(self):
while self.stop_event.is_set() == False:
try:
address = raw_input("who u talking to? ")
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.connect((address, self.sport))
break
except socket.error:
# handle connection problems
print "can't connect to "+ str(address)
except:
# exit thread in case of an unexpected error
self.stop_event.set()
while self.stop_event.is_set() == False:
try:
# chat loop: send messages to remote host
print "what to send? :",
msg = raw_input()
# beware: send() function may block indefinitly here and it might not send all bytes as expected !!
conn.send(msg)
except:
# exit thread in case of an unexpected error
self.stop_event.set()
# close socket before thread terminates
conn.close()
def main():
do_exit = False
server_port = 2727
# start server socket thread
accept = Accept(server_port)
accept.start()
# start transmitting client socket thread
shout = Shout(server_port)
shout.start()
while do_exit == False:
try:
# sleep some time
time.sleep(0.1)
except KeyboardInterrupt:
# Ctrl+C was hit - exit program
do_exit = True
# stop all running threads
shout.stop()
accept.stop()
# exit main program after all threads were terminated gracefully
if __name__ == "__main__":
main()

Look at the Python library source for SocketServer.py, in particular the implementation of server_forever() to see how a server implements a quit. It uses select() to poll the server socket for new connections and tests a quit flag. Here's a hack on your source to use SocketServer, and I added a quit flag to Shout(). It will run the Shout and Listen threads for 5 seconds and then stop them.
import socket
import SocketServer
import threading
import time
class Handler(SocketServer.StreamRequestHandler):
def handle(self):
print str(self.client_address) + ": " + self.request.recv(250)
self.request.send("got it\n")
class Listen(threading.Thread):
def run(self):
self.server = SocketServer.TCPServer(('',2727),Handler)
self.server.serve_forever()
def stop(self):
self.server.shutdown()
class Shout(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.quit = False
def run(self):
while not self.quit:
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.connect(('localhost', 2727))
conn.send('sending\n')
print conn.recv(100)
conn.close()
def stop(self):
self.quit = True
listen = Listen()
listen.start()
shout = Shout()
shout.start()
time.sleep(5)
shout.stop()
listen.stop()

Python socket accept blocks - prevents app from quitting

I've written a very simple python class which waits for connections on a socket. The intention is to stick this class into an existing app and asyncronously send data to connecting clients.
The problem is that when waiting on an socket.accept(), I cannot end my application by pressing ctrl-c. Neither can I detect when my class goes out of scope and notify it to end.
Ideally the application below should quit after the time.sleep(4) expires. As you can see below, I tried using select, but this also prevents the app from responding to ctrl-c. If I could detect that the variable 'a' has gone out of scope in the main method, I could set the quitting flag (and reduce the timeout on select to make it responsive).
Any ideas?
thanks
import sys
import socket
import threading
import time
import select
class Server( threading.Thread ):
def __init__( self, i_port ):
threading.Thread.__init__( self )
self.quitting = False
self.serversocket = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
self.serversocket.bind( (socket.gethostname(), i_port ) )
self.serversocket.listen(5)
self.start()
def run( self ):
# Wait for connection
while not self.quitting:
rr,rw,err = select.select( [self.serversocket],[],[], 20 )
if rr:
(clientsocket, address) = self.serversocket.accept()
clientsocket.close()
def main():
a = Server( 6543 )
time.sleep(4)
if __name__=='__main__':
main()

Add self.setDaemon(True) to the __init__ before self.start().
(In Python 2.6 and later, self.daemon = True is preferred).
The key idea is explained here:
The entire Python program exits when
no alive non-daemon threads are left.
So, you need to make "daemons" of those threads who should not keep the whole process alive just by being alive themselves. The main thread is always non-daemon, by the way.

I don't recommend the setDaemon feature for normal shutdown. It's sloppy; instead of having a clean shutdown path for threads, it simply kills the thread with no chance for cleanup. It's good to set it, so your program doesn't get stuck if the main thread exits unexpectedly, but it's not a good normal shutdown path except for quick hacks.
import sys, os, socket, threading, time, select
class Server(threading.Thread):
def __init__(self, i_port):
threading.Thread.__init__(self)
self.setDaemon(True)
self.quitting = False
self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.serversocket.bind((socket.gethostname(), i_port))
self.serversocket.listen(5)
self.start()
def shutdown(self):
if self.quitting:
return
self.quitting = True
self.join()
def run(self):
# Wait for connection
while not self.quitting:
rr,rw,err = select.select([self.serversocket],[],[], 1)
print rr
if rr:
(clientsocket, address) = self.serversocket.accept()
clientsocket.close()
print "shutting down"
self.serversocket.close()
def main():
a = Server(6543)
try:
time.sleep(4)
finally:
a.shutdown()
if __name__=='__main__':
main()
Note that this will delay for up to a second after calling shutdown(), which is poor behavior. This is normally easy to fix: create a wakeup pipe() that you can write to, and include it in the select; but although this is very basic, I couldn't find any way to do this in Python. (os.pipe() returns file descriptors, not file objects that we can write to.) I havn't dig deeper, since it's tangental to the question.