I am using subprocess.Popen in a Django application. I then store the pid in a database. The user can then send a rest request to /api/task/task.id/stop/ in order to kill the task.
This all works fine, except that when I kill the subprocess the Django development server is also terminated. How do I go about killing the subprocess without killing the dev server?
I start my process like this:
process = subprocess.Popen(["nohup {} {} {}".format(settings.PYTHON, "task.py", task.id) ], shell=True )
task.pid = process.pid
task.save()
And I am terminating it like this:
os.killpg( os.getpgid(task.pid), signal.SIGTERM )
Related
My python daemon is run through systemd, with stderr logging on a file.
A simplified version of the systemd service file is this:
[Unit]
Description=Test
[Service]
ExecStart=/path/to/test.py
StandardError=file:/path/to/error.log
[Install]
WantedBy=multi-user.target
The python daemon instantiates a threaded TCP server and runs serve_forever on it.
The problem is any error happening after the thread start is not logged on the error.log file. Any error happening before is logged.
A simplified version of the python daemon is this:
import threading, socketserver
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
class TCPHandler(socketserver.BaseRequestHandler):
def handle(self):
pass
#barfoo
TCPServer = ThreadedTCPServer(("127.0.0.1", 12345), TCPHandler)
TCPThread = threading.Thread(target=TCPServer.serve_forever)
TCPThread.start()
#foobar
If I uncomment "barfoo", a "name not defined" is logged in the error.log file.
If I uncomment "foobar", nothing is logged.
Also, since the TCP server thread is still running, the systemd daemon is shown as running, so I can't even detect the error from there.
If I run the daemon from command line, both errors are correctly logged on stderr (and redirected if I redirect stderr on a file). Trying the same redirect in the systemd script, instead of the StandardError directive, has the same effect as the latter.
EDIT:
I tested the same thing with a generic infinite looping thread, for the same effect:
import threading
def run():
while True:
pass
#aaaaaa # logs an error on systemd's stderr
test_thread = threading.Thread(target=run)
test_thread.start()
#foobar # doesn't log an error on systemd's stderr
I found the solution. Leaving this for future reference.
It was actually not a systemd problem at all, but rather a buffering problem.
Disabling the default Python (for versions < 3.7) buffering, by running "python -u" or using the PYTHONUNBUFFERED environment variable, causes all the output to be correctly logged.
I have an app writen by python with Flask, and deploy it use uwsgi +ngix, here is my config of uwsgi:
[uwsgi]
master=true
socket = :8223
chdir= /SWS/swdzweb
wsgi-file = manage.py
callable = app
processes = 4
threads = 2
My app will response a request which want to start or stop a daemon process writen by pytho too. as below
in the request function do
os.system("python /SWS/webservice.py %s" % cmd)
where cmd is start|stop. in my daemon process, it is single process and single thread, and i capture SIGTEM then exit,like this
signal(SIGTERM, lambda signo,handler:sys.exit(0))
But. when I start this daemon process by uwsgi in my request function, i can't stop it, for example
kill -15 pid or python /SWS/web service.py stop
just like the SIGTERM signal does not send to my daemon process.
however, when i config uwsgi with 4 processes and 1 thread, this works fine. config like this
[uwsgi]
master=true
socket = :8223
chdir= /SWS/swdzweb
wsgi-file = manage.py
callable = app
processes = 4
threads = 1
I can not figure out the reason, so I have to ask for help.
Thanks!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I have a Python script that is running as a service. It writes to disk. If a user calls systemctl stop on the service, I would like to handle the command in my own way to reduce the risk of file corruption.
How can I catch the systemctl stop command?
My file in /usr/lib/systemd/system is:
[Unit]
Description=foo
[Service]
Type=simple
ExecStart=/usr/bin/python /srv/go.py
User=jon
Restart=always
[Install]
WantedBy=graphical.target
My Python script is:
#!/usr/bin/python
import time
import datetime
import threading
def worker():
while True:
with open('/tmp/go.txt', 'a') as f:
s = str(datetime.datetime.utcnow()) + '\n'
f.write(s)
print(s)
time.sleep(1)
if __name__=='__main__':
t = threading.Thread(target=worker)
t.start()
As described in systemd documentation, processes will receive SIGTERM and then, immediately, SIGHUP. After some time, SIGKILL will be send.
All signals, as well as strategy of sending them, can be changed in relevant service file.
See another stackoverflow question to see how handle these signals in your Python program.
I am using rabbitMQ to launch processes in remote hosts located in other parts of the world. Eg, RabbitMQ is running in an Oregon host, and it receives a client message to launch processes in Ireland and California.
Most of the time, the processes are launched, and, when they finish, rabbitMQ returns the output to the client. But, sometimes, the jobs finish successfully but rabbitMQ hasn't return the output to the client, and the client keeps hanging waiting for the response. These processes can take 10 minutes to execute, so the client is 10 minutes hanged waiting for the response.
I am using celery to connect to the rabbitMQ, and the client calls are blocking using task.get(). In other words, the client hangs until it receives the response for its call. I would like to understand why the client did not get the response if the jobs have finished. How can I debug this problem?
Here is my celeryconfig.py
import os
import sys
# add hadoop python to the env, just for the running
sys.path.append(os.path.dirname(os.path.basename(__file__)))
# broker configuration
# medusa-rabbitmq is the name of the hosts where rabbitmq is running
BROKER_URL = "amqp://celeryuser:celery#medusa-rabbitmq/celeryvhost"
CELERY_RESULT_BACKEND = "amqp"
TEST_RUNNER = 'celery.contrib.test_runner.run_tests'
# for debug
# CELERY_ALWAYS_EAGER = True
# module loaded
CELERY_IMPORTS = ("medusa.mergedirs", "medusa.medusasystem",
"medusa.utility", "medusa.pingdaemon", "medusa.hdfs", "medusa.vote.voting")
I have a flask application running with gevent-socketio that I create this way:
server = SocketIOServer(('localhost', 2345), app, resource='socket.io')
gevent.spawn(send_queued_messages_loop, server)
server.serve_forever()
I launch send_queued_messages_loop in a gevent thread that keeps on polling on a gevent.Queue where my program stores data to send it to the socket.io connected clients
I tried different approaches to stop the server (such as using sys.exit) either from the socket.io handler (when the client sends a socket.io message) or from a normal route (when the client makes a request to /shutdown) but in any case, sys.exit seems to fail because of the presence of greenlets.
I tried to call gevent.shutdown() first, but this does not seem to change anything
What would be the proper way to shutdown the server?
Instead of using serve_forever() create a gevent.event.Event and wait for it. To actually initiate shutdown, trigger the event using its set() method:
from gevent.event import Event
stopper = Event()
server = SocketIOServer(('localhost', 2345), app, resource='socket.io')
server.start()
gevent.spawn(send_queued_messages_loop)
try:
stopper.wait()
except KeyboardInterrupt:
print
No matter from where you now want to terminate your process - all you need to do is calling stopper.set().
The try..except is not really necessary but I prefer not getting a stacktrace on a clean CTRL-C exit.