Socket.close() does not stop any blocking socket.accept() calls that are already running on that socket.
I have several threads in my python program that only run a blocking socket.accept() call on a unix domain socket that has been closed already.
I want to kill these threads by making the socket.accept() calls stop or
raise an exception.
I am trying to do this by loading new code in the program, without stopping the program.
Therefore, changing the code that spawned these threads or that closed the sockets is not an option.
Is there any way to do this?
This is similar to https://stackoverflow.com/a/10090348/3084431, but these solutions wont work for my code:
This point is not true, closing won't raise an exception on the accept. shutdown does, but that can not be called anymore when the thread is closed.
I can not connect to this socket anymore. The socket is closed.
The threads with the accept calls are already running, I can't change them.
Same as 3
For clarification, I have written some example code that has this problem.
This code works in both python 2 and python 3.
import socket
import threading
import time
address = "./socket.sock"
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind(address)
sock.listen(5)
def accept():
print(sock.accept())
t = threading.Thread(target=accept, name="acceptorthread")
t.start()
sock.close()
time.sleep(0.5) # give the thread some time to register the closing
print(threading.enumerate()) # the acceptorthread will still be running
What I need is something that I can run after this code has finished that can stop the acceptor thread somehow.
There is no mechanism in kernel to notify every listener that a socket is closed. You have to write something yourself. A simple solution is to use timeout on socket:
sock.settimeout(1)
def accept():
while True:
try:
print(sock.accept())
except socket.timeout:
continue
break
Now when you close the socket the next call (after a timeout) to .accept() will throw a "bad descriptor" exception.
Also remember that sockets api in Python is not thread safe. Wrapping every socket call with a lock (or other synchronization methods) is advised in multi-threaded environment.
More advanced (and efficient) would be to use wrap your socket with a select call. Note that the socket does not have to be in non-blocking mode in order to use it.
Therefore, changing the code that spawned these threads or that closed the sockets is not an option.
If that's the case, then you are doomed. Without changing the code running in threads it is impossible to achieve. It's like asking "how can I fix my broken car without modifying the car". Won't happen, mate.
You should only call .accept() on a socket that has given the "readable" result from some selectors. Then, accept doesn't need to be interrupted.
But in case of spurious wakeup, you should have the listening socket in O_NONBLOCK mode anyway.
Related
I'm trying to write a server program and I have a thread for listening for new clients:
class ClientFinder(Thread):
def __init__(self, port):
Thread.__init__(self)
self._continue = True
self._port = port
# try to create socket
def run(self):
# listen for new clients
while self._continue:
# add new clients
def stop(self):
# stop client
self._continue = False
client_finder = ClientFinder(8000)
client_finder.start()
client_finder.stop()
client_finder.join()
I can't join client_finder because it never ends. Calling stop() lets the thread stop after the next client is accepted, so the program just hangs forever.
1) Is it okay for my program to just end even if I haven't joined all my threads (such as by removing the join)? Or is this lazy/bad practice?
2) If it is a problem, what's the solution/best practice to avoid this? From what I've found so far, there's no way to force a thread to stop.
Whether waiting for the current clients to finish is a problem is really your choice. It may be a good idea, or you may prefer to kill connections.
Waiting for a new client is probably a worse thing, since it may never happen. An easy solution would be to have some reasonable timeout for the listening - let's say if nobody connects in 5s, you go back to the loop to check the flag. This is short enough for a typical shutdown solution, but long enough that rechecking shouldn't affect your CPU usage.
If you don't want to wait for a short timeout, you can add a pipe/socket between the thread doing shutdown and your ClientFinder and send a notification to shutdown. Instead of only waiting for a new client, you'd need to wait on both fds (I'm assuming ClientFinder uses sockets) and check which of them got a message.
I have client and server module, each one can be started by a function. I just need to find a way to run booth in parallel which:
in case of an exception in the client/server would stop the other so the test runner would not stay stuck
in case of an exception in client/server would print the exception or propagate it to the runner so I could see it and debug the client/server using the test suite
would preferably use threads for performance reasons
The first tentative with simple threads ended with an ugly os._exit(1) when catching a exception in the run method of the thread (which kills the test runner...) Edit: with the threading package
The second tentative (to try to avoid os._exit()) was with concurrent.futures.ThreadPoolExecutor. It allows to get the exception out of the thread but I still can't find a way to abort the other thread.
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
server_future = executor.submit(server)
client_future = executor.submit(client)
concurrent.futures.wait([server_future, client_future],
return_when=concurrent.futures.FIRST_EXCEPTION)
if client_future.done() && client_future.exception():
# we can handle the client exception here
# but how to stop the server from waiting the client?
# also, raise is blocking
if server_future.done() && server_future.exception():
# same here
Is there a way to achieve this with threads?
If not with threads, is there a simple way to test a client server app at all? (I think the two first requirements are enough to have a usable solution)
Edit: The client or the server would be blocked on an accept() or a receive() call so I can't periodically pool a flag a decide to exit.(one of classic method to stop a thread)
You can use the threading package. Be aware though that force killing thread is not a good idea, as discussed here. It seems there is no official way to kill Thread in Python, but you can follow one of the example given on the linked post.
Now you need to wait for one thread to exit before stopping the other one, avoiding your test runner to be stuck. You can use Threads wrapping your server/client launch, and have your main Thread waiting for either client/server Thread to exit before killing the other one.
You can define your client/server Thread like this:
# Server thread (replace
class testServerThread (threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
# Do stuff if required
def run(self):
try:
startServer() # Or startClient() for your client thread
except: Exception
# Print your exception here, so you can debug
Then, start both client and server thread, and wait for one of them to exit. Once one of them is not alive anymore, you can kill the other and continue on testing.
# Create and start client/server
serverThread = testServerThread ()
clientThread = testClientThread ()
serverThread.start()
clientThread.start()
# Wait at most 5 seconds for them to exit, and loop if they're still both alive
while(serverThread.is_alive() and clientThread.is_alive()):
serverThread.join(5)
clientThread.join(5)
# Either client or server exited. Kill the other one.
# Note: the kill function you'll have to define yourself, as said above
if(serverThread.is_alive()):
serverThread.kill()
if(clientThread.islive()):
clientThread.kill()
# Done! Your Test runner can continue its work
The central piece of code is the join() function:
Wait until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception –, or until the optional timeout occurs.
So in our case, it will wait 5 seconds for the client and 5 seconds for the server, and if both of them are still alive afterward it will loop again. Whenever one of them exit, the loop will stop, and the remaining thread will be killed.
I'm using SocketServer.ThreadingMixIn, pretty much as in the docs.
Other than having extracted the clients to run on their own script, I've also redefined the handle method as I want the connection to the client to keep alive and receive more messages:
def handle(self):
try:
while True:
data = self.request.recv(1024)
if not data:
break # Quits the thread if the client was disconnected
else:
print(cur_thread.name)
self.request.send(data)
except:
pass
The problem is that even when I try to terminate the server with server.shutdown() or by KeyboardInterrupt, it will still be blocked on the handle as long as the client maintains an open socket.
So how I can effectively stop the server even if there are still connected clients?
The best solution I found was to use SocketServer.ForkingMixIn instead of SocketServer.ThreadingMixIn.
This way the daemon actually works, even though using processes instead of threads was not exactly what I wanted.
I have a python tcp server, there is thread for every connection to listen.
When I call close on connection object exception "bad file descriptor" is thrown.
By googling I've found some solutions, each using loop in order to receive client data and breaking that loop, when they decide to disconnect client. My client is written in C# and does not "get", that it's "disconnected" from server, python simply ignores incomming data from C# client.
What's the legit, best practice way to disconnect tcp connection from server side in python ?
Thanks in advance
A bad file descriptor, most likely, means that the socket was already closed by another thread.
Here are some thoughts on general practices. For the client, one way to know that it is disconnected is to check if the recv() value is 0. If it is, then that means the remote side has closed the connection. Basically, you should use select (or poll) and pass fds of all the clients and teh server to select. If you get a read event on any of the fds, then depending upon the fd type, here is what happens. If the fd is server type, then a read event means that there is a pending connection and you should issue an accept() to get the new connection. On the other hand, if hte fd is a non-server type (meaning a regular tcp connection), then a read event means that there is some data and you should issue a recv() event to read data.
You would use a loop for the select. Basically, start the loop using a select() call and once you get an event, do something with that event, and then reenter the loop and issue the next select().
You might find these links helpful: http://ilab.cs.byu.edu/python/select/echoserver.html and http://docs.python.org/2/library/socket.html
From the docs:
Note: close() releases the resource associated with a connection but
does not necessarily close the connection immediately. If you want to
close the connection in a timely fashion, call shutdown() before
close().
So you should call shutdown() before calling close(). Also you should pass SHUT_RDWR flag to completely shutdown the connection:
from socket import SHUT_RDWR
...
try:
s.shutdown(SHUT_RDWR)
s.close()
except Exception:
pass
The "bad file description" error means (most likely) that the socket is already closed (at least from Python side).
I'd like to create a python socket (or SocketServer) that, once connected to a single device, maintains an open connection in order for regular checks to be made to see if any data has been sent. The socket will only listen for one connection.
E.g.:
def get_data(conn):
response='back atcha'
data = conn.recv(1024)
print 'get_data:',data
if data:
conn.send(response)
s = open_socket()
conn, addr = s.accept()
while True:
print 'running'
time.sleep(1)
get_data(conn)
#do other stuff
Once the server socket is bound and the connection has been accepted, the socket blocks when running a .recv until either the connecting client sends some data or closes its socket. As I am waiting for irregular data (could be seconds, could be a day), and the program needs to perform other tasks in the meantime, this blocking is a problem.
I don't want the client to close its socket, as it may need to send (or receive) data at any time to (from) the server. Is the only solution to run this in a separate thread, or is there a simple way to setup the client/server sockets to maintain the connection forever (and is this safe? It'll be running on a VLAN) while not blocking when no data has been received?
You're looking for non-blocking I/O, also called asynchronous I/O. Using a separate thread which blocks on this is very inefficient but it's pretty straightforward.
For a Python asynchronous I/O framework I highly recommend Twisted. Also check out asyncore which comes with the standard library.