We Keep Coding

Twisted and a command line interface

I have a program where I want to do two things:
Interact with a server and respond to events from the server. I am doing this using twisted.
Have a command line prompt for the user where he can issue additional commands. I am using the python cmd module for this so far.
There seems to be no other choice than having two threads, as readline only has a blocking interface and needs to handle stuff like auto completion. Twisted on the other hand has to continuously run the reactor.
Now the problem is that it seems very hard to handle Ctrl-C for this. The easy solution would seem to have the command line run in the main thread and just use reactor.callFromThread for every interaction with the rest of the program. This is very easy, as overwriting Cmd.onecmd can do this in a generic way. However when I try to spawn the reactor in a thread with
t = Thread(target=reactor.run)
t.start()
I immediately get an exception
File "/usr/lib/python3.6/signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
builtins.ValueError: signal only works in main thread
Everyone using twisted insists that the twisted reactor should run in the main thread, as that would be a better design.
When trying to do it that way and running twisted in the main thread, it will catch the Ctrl-C, exit the reactor and I am stuck with a thread that does not exit, as the call to input() inside cmdloop does not return. I tried searching for a solution to this and how to get out of the input() call, but everyone also insists that a command line interface should run in the main thread.
One potential option I found was to run twisted as the main thread and make the input thread a daemon, so it should exit when the reactor exits, however the daemon flag did not change anything (the thread did not exit when the main thread did). Furthermore this is likely dangerous, as the thread might be doing something important when it is killed.
Is there any way out of this?

Take a look at how invective does this with Twisted and without threads (one way to read the code might be to start at the mainpoint and work your way in).

How to use threads for functional tests of client server application?

I have client and server module, each one can be started by a function. I just need to find a way to run booth in parallel which:
in case of an exception in the client/server would stop the other so the test runner would not stay stuck
in case of an exception in client/server would print the exception or propagate it to the runner so I could see it and debug the client/server using the test suite
would preferably use threads for performance reasons
The first tentative with simple threads ended with an ugly os._exit(1) when catching a exception in the run method of the thread (which kills the test runner...) Edit: with the threading package
The second tentative (to try to avoid os._exit()) was with concurrent.futures.ThreadPoolExecutor. It allows to get the exception out of the thread but I still can't find a way to abort the other thread.
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
server_future = executor.submit(server)
client_future = executor.submit(client)
concurrent.futures.wait([server_future, client_future],
return_when=concurrent.futures.FIRST_EXCEPTION)
if client_future.done() && client_future.exception():
# we can handle the client exception here
# but how to stop the server from waiting the client?
# also, raise is blocking
if server_future.done() && server_future.exception():
# same here
Is there a way to achieve this with threads?
If not with threads, is there a simple way to test a client server app at all? (I think the two first requirements are enough to have a usable solution)
Edit: The client or the server would be blocked on an accept() or a receive() call so I can't periodically pool a flag a decide to exit.(one of classic method to stop a thread)

You can use the threading package. Be aware though that force killing thread is not a good idea, as discussed here. It seems there is no official way to kill Thread in Python, but you can follow one of the example given on the linked post.
Now you need to wait for one thread to exit before stopping the other one, avoiding your test runner to be stuck. You can use Threads wrapping your server/client launch, and have your main Thread waiting for either client/server Thread to exit before killing the other one.
You can define your client/server Thread like this:
# Server thread (replace
class testServerThread (threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
# Do stuff if required
def run(self):
try:
startServer() # Or startClient() for your client thread
except: Exception
# Print your exception here, so you can debug
Then, start both client and server thread, and wait for one of them to exit. Once one of them is not alive anymore, you can kill the other and continue on testing.
# Create and start client/server
serverThread = testServerThread ()
clientThread = testClientThread ()
serverThread.start()
clientThread.start()
# Wait at most 5 seconds for them to exit, and loop if they're still both alive
while(serverThread.is_alive() and clientThread.is_alive()):
serverThread.join(5)
clientThread.join(5)
# Either client or server exited. Kill the other one.
# Note: the kill function you'll have to define yourself, as said above
if(serverThread.is_alive()):
serverThread.kill()
if(clientThread.islive()):
clientThread.kill()
# Done! Your Test runner can continue its work
The central piece of code is the join() function:
Wait until the thread terminates. This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception –, or until the optional timeout occurs.
So in our case, it will wait 5 seconds for the client and 5 seconds for the server, and if both of them are still alive afterward it will loop again. Whenever one of them exit, the loop will stop, and the remaining thread will be killed.

Gracefully stop multi-process application in Python

I have an application which is stuck in a file.read call. Before it goes into a loop where this call is made it forks a child which starts a gevent WSGI server. The purpose of this setup is that I want to wait for a keystroke, send this keystroke to the child websocket server which spreads the message among other connected websocket-clients. My problem is that I don't know how to stop this thing.
If I Ctrl+C the child server process gets the sigint and stops. But my parent only responds if it can read something out of his file. Isn't there something like an asynchronous handler? I also tried registering for SIGINT via signal.signal and manually sending the signal, but the signal handler was only called if something was written to the file.
BTW: I'm runnning Linux.

Process Doesn't End When Closed

I built a web-scraper application with Python. It consists of three main parts:
The GUI (built on tkinter)
A Client (controls interface between front- and back-end)
Back-end code (various threaded processes).
The problem I have is that when the user hits X to exit the program instead of quitting through the interface, it seems like root.destroy() never gets called and the application runs forever, even though the window does disappear. This ends up consuming vast amounts of system resources.
I have tried setting all threads to Daemon without much success. Is there any other reason the program would keep eating up CPU after exit?

You don't want to set all threads to daemon. You want to set the client thread and the back-end thread to daemon. That way, when the GUI thread dies, the threads with daemon set to True end as well.
From the documentation:
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left.

twisted reactor invocations on separate thread

I am working on an application that involves fetching data over tcp using twisted api.
Our process is listener application that keeps listening to events and does the following..
Process event notification and build dictionary to sent it to third party application
In order to complete dictionary..it calls a process using twisted api to get some additional data and complete the dictionary.
I can not execute twisted api executions on main thread else after single execution..reactor stops and main thread no further progress happens..
What I want is:
for each event notification--- spool a new thread to make a twisted call over tcp to get data.
join mainThread with newThread to wait for its completion
get results, merge with half built dictionary and send it to third party app.
Let say I listen to eventObj1 in main thread ---
processing involves steps for ex step1---step2---step3---step4--send to party.
assume step4 involves fetching data over tcp and we should wait until results are available before we can finish dictionary and send it to third party.
so as eventObj1 comes I queue it for fetching over tcp.. while doing so I say reactor.start() (reactor is started in the main thread) and after sometime I get the data and callback is invoked dict is built for event1 and send to third party.
and but I cant queue up any more events for data fetching, until I do reactor.stop() ..because until reactor.stop() is not called main thread can't go back to process eventObj2..
So I think, what I need is reactor start in separate thread..and keep queuing up events from main thread stop reactor before main program exits

You don't need any threads. You just want the reactor to do multiple things, which is actually the whole point of having a reactor. See this question for an explanation: Twisted reactor starting multiple times in a single program?