I have a jabber client that is reading from its stdin and posting PubSub messages. If I get EOF on stdin, I want to terminate the client.
I first tried sys.exit(), but this causes an exception and the client does not exit. I then did some searching and found out that I should call reactor.stop(), but I am unable to make this work. The following code in my client:
from twisted.internet import reactor
reactor.stop()
Results in exceptions.AttributeError: 'module' object has no attribute 'stop'
What do I need to do to cause twistd to shut my application down and exit?
EDIT 2
The original problem was caused by some symlinks messing up the module import. After fixing that problem, I get a new exception:
twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.
After the exception, twistd shuts down. I think this may be caused by the call to MyClient.loop in MyClient.connectionInitialized. Perhaps I need to defer the call until later?
EDIT
Here's the .tac file for my client
import sys
from twisted.application import service
from twisted.words.protocols.jabber.jid import JID
from myApp.clients import MyClient
clientJID = JID('client#example.com')
serverJID = JID('pubsub.example.com')
password = 'secret'
application = service.Application('XMPP client')
xmppClient = client.XMPPClient(clientJID, password)
xmppClient.logTraffic = True
xmppClient.setServiceParent(application)
handler = MyClient(clientJID, serverJID, sys.stdin)
handler.setHandlerParent(xmppClient)
Which I'm invoking with
twistd -noy sentry/myclient.tac < input.txt
Here's the code for MyClient:
import os
import sys
import time
from datetime import datetime
from wokkel.pubsub import PubSubClient
class MyClient(PubSubClient):
def __init__(self, entity, server, file, sender=None):
self.entity = entity
self.server = server
self.sender = sender
self.file = file
def loop(self):
while True:
line = self.file.readline()
if line:
print line
else:
from twisted.internet import reactor
reactor.stop()
def connectionInitialized(self):
self.loop()
from twisted.internet import reactor
reactor.stop()
that should work. The fact that it doesn't means something else is wrong on your application. I can't figure out what's wrong from the information you provided.
Can you provide more (all) of the code?
EDIT:
Ok, now the problem is that you don't stop your own while True loop, so it will keep looping and eventually stop the reactor again.
Try this:
from twisted.internet import reactor
reactor.stop()
return
Now, I suspect your loop isn't very good thing for a event-driven framework. While you're just printing lines, it is fine, but depending on what you want to really do (I suspect you'll do more than just print lines) you'll have to refactor that loop to work with events.
Use reactor.callFromThread(reactor.stop) instead of reactor.stop. This should solve the issue.
I used to do this way (in sigint handler of a non-twistd called application):
reactor.removeAll()
reactor.iterate()
reactor.stop()
I'm not 100% sure it is the right way, but twisted is happy
the same application started in a tac is handled directly by twistd signal handler, I've found this question because I have some rpc client requests that I would to wait for and handle result before exiting and looks like twistd is just killing the reactor without letting the call finish
Related
How do you kill a websocket server programmatically? I'll be deploying this server to production along side other things. And I like to build a single python script that sends a kill signal to everything. I cannot figure out how to kill this thing without a user keyboard interrupt or a kill -9.
sys.exit() didn't work.
psutil and terminate() didn't work either
import os
import psutil
current_system_pid = os.getpid()
ThisSystem = psutil.Process(current_system_pid)
ThisSystem.terminate()
I'm out of ideas. For now I'm killing it on the command line with kill -9.
When I kill it varous ways, it tend to see this message below, but the scrip is still running
2020-12-12 12:24:54-0500 [autobahn.twisted.websocket.WebSocketServerFactory] (TCP Port 8080 Closed)
2020-12-12 12:24:54-0500 [-] Stopping factory <autobahn.twisted.websocket.WebSocketServerFactory object at 0x110680f28>
autobahn install:
pip install autobahn[twisted]
Code:
from autobahn.twisted.websocket import WebSocketServerProtocol, WebSocketServerFactory
import sys
from twisted.python import log
from twisted.internet import reactor
class MyServerProtocol(WebSocketServerProtocol):
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
def onMessage(self, payload, isBinary):
print("Text message received: {0}".format(payload.decode('utf8')))
# echo back message verbatim
# self.sendMessage(payload, isBinary)
def onClose(self, wasClean, code, reason):
print("WebSocket connection closed: {0}".format(reason))
def StopWebsocketServer():
PrintAndLog_FuncNameHeader("Begin")
reactor.stop()
PrintAndLog_FuncNameHeader("End")
if __name__ == '__main__':
# TODO remove the logging that came in the example
log.startLogging(sys.stdout)
factory = WebSocketServerFactory("ws://127.0.0.1:8080")
factory.protocol = MyServerProtocol
# note to self: if using putChild, the child must be bytes...
reactor.listenTCP(Port_ws, factory)
reactor.run()
Solution using #Jean-Paul Calderone's answer:
import os
import signal
os.kill(os.getpid(), signal.SIGKILL)
I have an external python script that sends a kill signal to each of my python scripts. The kill signal is simply the existence of a file that every script knows to look for. Once that kill signal appears, each script knows it has x seconds before it will be killed. This way they have a few seconds to gracefully finish something.
twisted.internet.reactor.stop() is how you cause the reactor to shut down. This is usually what results in a Twisted-based program exiting (though of course it doesn't necessarily have to, if the program does more things after the reactor shuts down - but this is uncommon).
However, it sounds like you don't want to know what Python code to run inside the process to end it. You want to know what some other process can do to your Twisted-based process to make it exit. You gave two solutions - KeyboardInterrupt and SIGKILL. You didn't mention why either of these two solutions is inappropriate. They seem fine to me.
If you're uncomfortable with SIGKILL (which you shouldn't be, after all, your program might meet an untimely demise for many reasons and you should be prepared to deal with this) then what you might have overlooked about KeyboardInterrupt is that it is merely the exception that is raised inside a Python program by the default SIGINT handler.
If you send SIGINT to a Twisted-based process then, under normal usage, this will stop the reactor and allow an orderly shutdown.
I am trying to figure out how to run my overloaded customized BaseHTTPServer instance in the background after running the "".serve_forever() method.
Normally when you run the method execution will hang until you execute a keyboard interrupt, but I would like it to serve requests in the background while continuing script execution. Please help!
You can start the server in a different thread: https://docs.python.org/3/library/_thread.html#thread.start_new_thread
So something like:
def start_server():
# Setup stuff here...
server.serve_forever()
# start the server in a background thread
thread.start_new_thread(start_server)
print('The server is running but my script is still executing!')
I was trying to do some long-term animation using async and thought I'd have to rewrite server to use aiohttp (https://docs.aiohttp.org/en/v0.12.0/web.html), but Olivers technique of using seperate thread saved me all that pain. My code looks like this, where MyHTTPServer is simply my custom sublass of HTTPServer
import threading
import asyncio
from http.server import BaseHTTPRequestHandler, HTTPServer
import socketserver
import io
import threading
async def tick_async(server):
while True:
server.animate_something()
await asyncio.sleep(1.0)
def start_server():
httpd.serve_forever()
try:
print('Server listening on port 8082...')
httpd = MyHTTPServer(('', 8082), MyHttpHandler)
asyncio.ensure_future(tick_async(httpd))
loop = asyncio.get_event_loop()
t = threading.Thread(target=start_server)
t.start()
loop.run_forever()
So I'm writing a small script to use with Deluge. Deluge uses Twisted, and I really don't have a firm grasp on how it works. Normally I'd just look up more info on it, but getting started with Twisted would take a long time and is beyond the scope of this little project. So I figured I would just ask here.
Now, I have this code. I'll try to explain the specifig parts I need help with
import base64
import processargs
from deluge.ui.client import client
from twisted.internet import reactor
from deluge.log import setupLogger
setupLogger()
options = processargs.readConfig(os.path.expanduser("~/.deluge-automator"))
d = client.connect(
host=options['host'],
port=int(options['port']),
username=options['username'],
password=options['password']
)
def start():
#other code
t = client.core.add_torrent_file(tfile,
base64.encodestring(data), None)
t.addCallback(on_torrent_added_success, tfile)
t.addErrback(on_torrent_added_fail)
def handle_stop_signal(SIGNAL, stack):
client.disconnect()
reactor.stop()
def on_torrent_added_success(result, tfile):
#other code
start()
def on_torrent_added_fail(result):
print "Add torrent failed!"
print "result: ", result
def on_connect_success(result):
#other code
start()
d.addCallback(on_connect_success)
def on_connect_fail(result):
print "Connection failed!"
print "result: ", result
d.addErrback(on_connect_fail)
signal.signal(signal.SIGTERM, handle_stop_signal)
signal.signal(signal.SIGINT, handle_stop_signal)
reactor.run()
When a torrent is successfully added, it should go back to start(), and it does, but I think it loses the reactor or something. Because now whenever it recieves a SIGTERM or SIGINT, the reactor closes, but doesn't quit the program:
± % python2 main.py
Connection was successful!
result: 10
^C^CConnection failed!
result: [Failure instance: Traceback: <class 'twisted.internet.error.ReactorNotRunning'>: Can't stop reactor that isn't running.
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:551:_runCallbacks
/usr/lib/python2.7/site-packages/deluge/ui/client.py:412:__on_login
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:368:callback
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:464:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/internet/defer.py:551:_runCallbacks
main.py:70:on_connect_success
main.py:32:start
main.py:49:handle_stop_signal
/usr/lib/python2.7/site-packages/twisted/internet/base.py:577:stop
]
So the reactor gets stopped, but it doesn't quit the program. I have to keyboard interrupt twice. Once to stop the reactor, and a second time to throw the error. Is there a certain way to set up a loop like this?
reactor handles sigint, sigterm itself (there might be a parameter of reactor.run() that disables that). Install reactor.addSystemEventTrigger('before', 'shutdown', client.disconnect) instead.
See twisted: catch keyboardinterrupt and shutdown properly.
let's consider this code in python:
import socket
import threading
import sys
import select
class UDPServer:
def __init__(self):
self.s=None
self.t=None
def start(self,port=8888):
if not self.s:
self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.bind(("",port))
self.t=threading.Thread(target=self.run)
self.t.start()
def stop(self):
if self.s:
self.s.close()
self.t.join()
self.t=None
def run(self):
while True:
try:
#receive data
data,addr=self.s.recvfrom(1024)
self.onPacket(addr,data)
except:
break
self.s=None
def onPacket(self,addr,data):
print addr,data
us=UDPServer()
while True:
sys.stdout.write("UDP server> ")
cmd=sys.stdin.readline()
if cmd=="start\n":
print "starting server..."
us.start(8888)
print "done"
elif cmd=="stop\n":
print "stopping server..."
us.stop()
print "done"
elif cmd=="quit\n":
print "Quitting ..."
us.stop()
break;
print "bye bye"
It runs an interactive shell with which I can start and stop an UDP server.
The server is implemented through a class which launches a thread in which there's a infinite loop of recv/onPacket callback inside a try/except block which should detect the error and the exits from the loop.
What I expect is that when I type "stop" on the shell the socket is closed and an exception is raised by the recvfrom function because of the invalidation of the file descriptor.
Instead, it seems that recvfrom still to block the thread waiting for data even after the close call.
Why this strange behavior ?
I've always used this patter to implements an UDP server in C++ and JAVA and it always worked.
I've tried also with a "select" passing a list with the socket to the xread argument, in order to get an event of file descriptor disruption from select instead that from recvfrom, but select seems to be "insensible" to the close too.
I need to have a unique code which maintain the same behavior on Linux and Windows with python 2.5 - 2.6.
Thanks.
The usual solution is to have a pipe tell the worker thread when to die.
Create a pipe using os.pipe. This gives you a socket with both the reading and writing ends in the same program. It returns raw file descriptors, which you can use as-is (os.read and os.write) or turn into Python file objects using os.fdopen.
The worker thread waits on both the network socket and the read end of the pipe using select.select. When the pipe becomes readable, the worker thread cleans up and exits. Don't read the data, ignore it: its arrival is the message.
When the master thread wants to kill the worker, it writes a byte (any value) to the write end of the pipe. The master thread then joins the worker thread, then closes the pipe (remember to close both ends).
P.S. Closing an in-use socket is a bad idea in a multi-threaded program. The Linux close(2) manpage says:
It is probably unwise to close file descriptors while they may be in use by system calls in other threads in the same process. Since a file descriptor may be re-used, there are some obscure race conditions that may cause unintended side effects.
So it's lucky your first approach did not work!
This is not java. Good hints:
Don't use threads. Use asynchronous IO.
Use a higher level networking framework
Here's an example using twisted:
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor, stdio
from twisted.protocols.basic import LineReceiver
class UDPLogger(DatagramProtocol):
def datagramReceived(self, data, (host, port)):
print "received %r from %s:%d" % (data, host, port)
class ConsoleCommands(LineReceiver):
delimiter = '\n'
prompt_string = 'myserver> '
def connectionMade(self):
self.sendLine('My Server Admin Console!')
self.transport.write(self.prompt_string)
def lineReceived(self, line):
line = line.strip()
if line:
if line == 'quit':
reactor.stop()
elif line == 'start':
reactor.listenUDP(8888, UDPLogger())
self.sendLine('listening on udp 8888')
else:
self.sendLine('Unknown command: %r' % (line,))
self.transport.write(self.prompt_string)
stdio.StandardIO(ConsoleCommands())
reactor.run()
Example session:
My Server Admin Console!
myserver> foo
Unknown command: 'foo'
myserver> start
listening on udp 8888
myserver> quit
I have a class that I wish to test via SimpleXMLRPCServer in python. The way I have my unit test set up is that I create a new thread, and start SimpleXMLRPCServer in that. Then I run all the test, and finally shut down.
This is my ServerThread:
class ServerThread(Thread):
running = True
def run(self):
self.server = #Creates and starts SimpleXMLRPCServer
while (self.running):
self.server.handle_request()
def stop(self):
self.running = False
self.server.server_close()
The problem is, that calling ServerThread.stop(), followed by Thread.stop() and Thread.join() will not cause the thread to stop properly if it's already waiting for a request in handle_request. And since there doesn't seem to be any interrupt or timeout mechanisms here that I can use, I am at a loss for how I can cleanly shut down the server thread.
I had the same problem and after hours of research i solved it by switching from using my own handle_request() loop to serve_forever() to start the server.
serve_forever() starts an internal loop like yours. This loop can be stopped by calling shutdown(). After stopping the loop it is possible to stop the server with server_close().
I don't know why this works and the handle_request() loop don't, but it does ;P
Here is my code:
from threading import Thread
from xmlrpc.server import SimpleXMLRPCServer
from pyWebService.server.service.WebServiceRequestHandler import WebServiceRquestHandler
class WebServiceServer(Thread):
def __init__(self, ip, port):
super(WebServiceServer, self).__init__()
self.running = True
self.server = SimpleXMLRPCServer((ip, port),requestHandler=WebServiceRquestHandler)
self.server.register_introspection_functions()
def register_function(self, function):
self.server.register_function(function)
def run(self):
self.server.serve_forever()
def stop_server(self):
self.server.shutdown()
self.server.server_close()
print("starting server")
webService = WebServiceServer("localhost", 8010)
webService.start()
print("stopping server")
webService.stop_server()
webService.join()
print("server stopped")
Two suggestions.
Suggestion One is to use a separate process instead of a separate thread.
Create a stand-alone XMLRPC server program.
Start it with subprocess.Popen().
Kill it when the test is done. In standard OS's (not Windows) the kill works nicely. In Windows, however, there's no trivial kill function, but there are recipes for this.
The other suggestion is to have a function in your XMLRPC server which causes server self-destruction. You define a function that calls sys.exit() or os.abort() or raises a similar exception that will stop the process.
This is my way. send SIGTERM to self. (Works for me)
Server code
import os
import signal
import xmlrpc.server
server = xmlrpc.server.SimpleXMLRPCServer(("0.0.0.0", 8000))
server.register_function(lambda: os.kill(os.getpid(), signal.SIGTERM), 'quit')
server.serve_forever()
Client code
import xmlrpc.client
c = xmlrpc.client.ServerProxy("http://localhost:8000")
try:
c.quit()
except ConnectionRefusedError:
pass