I need to spawn a background process in django, the view returns immediately, the background process continues make some changes, then update the db. This is done by os.spawnl() function to call a separate .py file.
The problem is after the background process is done, it becames a zombie function [python] <defunct>.
How do I avoid that? I followed this and this example but I still got the child process as zombie after the django render process.
I want to take this chance to practice my *nix process management skills so please do me a favor, don't give me Celery or other mq/async task solutions, and I hate dependencies.
This got to long for a comment-
The wait syscall (which os.wait is a wrapper for) reaps exit codes/pids from dead processes. You will want to os.wait in the process that is a generation above your zombie processes; the parent of the zombies processes. The parent processes will receive a SIGCHLD signal when one of its child processes die. If you insist on doing all of this yourself, you will need to install a signal handler to trap for SIGCHLD and in the signal handler call os.wait. Read some documentation on unix process handling and the Python documentation on the os module as there are variations of the os.wait function that will be non-blocking which maybe helpful.
import signal
signal.signal(signal.SIGCHLD, lambda _x,_y: os.wait())
I had a similar problem. I used active_children() from multiprocessing module.
import multiprocessing
# somewhere in middleware or where appropriate call
active_children()
Related
I used multiprocessing.Pool to imporove the performance of my Python server, when a task failed, I want to terminate it's child processes immediately.
I found that, if I create a process using Process, the terminate method can meets my needs, but if I create a process with Pool.apply_async, the return type is 'ApplyResult', and it can't terminate the corresponding process.
Is there any other way to do it?
I am using Python's multiprocessing Process class for a project to handle a function in a separate process. My question is, what happens when does the function in a separate process do its job? Is it that the process remains idle, or is process killed by the end of the function? Also, will there be any issues with giving the process/function a heavy load?
The code block is like this:
p = Process(target=function, args=[status.json])
if not p.is_alive():
p.start()
p.join()
If you're using the Process class, then the process starts when you call start(), and terminates when the target function completes, or you call the Process object's terminate() method. The process does not consume execution resources after terminating, but it will remain in the process table as a zombie process until you call join().
Regarding your concerns about putting a "heavy load" on the new process, it shouldn't have any effect. The multiprocessing library launches full-blown independent processes, so they operate largely the same as your main process does. Now whether or not the Process class is really the appropriate solution to your problem is hard to say since "heavy load" can mean many different things. If your tasks are IO intensive, the threading module may be a better choice. If you are performing the same tasks on many different objects, process pools may be a more appropriate choice. But a "heavy load" in and of itself shouldn't cause problems.
In one of my Django views, I am calling a python script and getting its pid with:
from subprocess import Popen
p = Popen(['python', 'script.py'])
mypid = p.pid
When trying to find out if the process still is running from another page, I use the following function on mypid (thanks to this question):
def doesProcessExist(pid):
if pid < 0:
return False
try:
os.kill(pid, 0)
except OSError, e:
return e.errno == errno.EPERM
else:
return True
No matter how long I wait, the process still shows up as running. The only thing that stops it, is if I spawn a new python script process with Popen. Is there anyway I can fix this? I am not sure if this is caused by Django not closing python properly after the script is finished or something else. In Ubuntu's process status manager, the process shows up as [python] <defunct>.
--
The problem is true for all script.py I have tried. I am currently using one as simple as:
from time import sleep
sleep(5)
Really, what you're doing is wrong. When you use a high-level wrapper like a subprocess.Popen, you need to manage the process through that object. Just having the PID elsewhere isn't enough to manage it.
If you insist on dealing in PIDs instead of Popen objects, then you should use the low-level APIs in os.
Fortunately, you're not doing anything complicated, like creating pipes to talk to the child process. So, you can just launch it with your favorite spawn variant, then wait for it with waitpid or one of its variants.
I'm assuming you're doing this all in a single-process web server. If you're using a forking web server, where the other page could be in a different process, even using PIDs won't work. The parent process has to reap the child, not some other arbitrary process. If you want to make that work, you'll have to make things more complicated, and you're really going to have to learn about the Unix process model before anyone can explain it to you.
What you see is a zombie process. It doesn't keep running. It can't. It is dead. The only thing that is left is some info that allows for related processes to retrieve its status.
To find out whether a subprocess is alive without blocking, call p.poll(). If it returns None then the process is still alive, otherwise you can safely forget about it (it is already reaped by .poll()).
subprocess module calls _cleanup() function that reaps zombie processes inside Popen() constructor. So normally your script won't create many zombie processes anyway.
To see a list of zombie processes:
import os
#NOTE: don't use Popen() here
print os.popen(r"ps aux | grep Z | grep -v grep").read(),
Processes in Unix stick around until the parent waits for them. calling wait on the object returned by thepopen will wait for the process to be done and will wait for it so it goes away. Until you do that it will exist as a zombie process See this message for info on getting the process to go away in the background while your web server runs without waiting for it in a foreground thread/view.
So, let's say that you do
p = subprocess.Popen(...)
At some point you need to call
p.wait()
I've been reading python's threadpool module's code.
It manipulates threads in this way : All workerThreads are created as daemonic thread. And it also have a dismiss mechanism that you can safely quit the worker thread by setting event, after all the job's done the dismissed threads will be joined in the main thread.
The python doc says that if worker threads were set daemonic, they will quit when main thread terminates. But it might be an ugly implementation, a better way is to make them non-daemonic and stop them with event.
Here is my question: Is it a good design to use both of the quit strategies? Is it better to set the threads non-daemonic and join them all before the main thread terminates?
In looking at this particular threadpool module, it appears to be designed to work either by allowing you to quit summarily, or waiting for the threads to complete. You would choose one or the other depending on how you want to handle requests currently in process:
If you don't care about whether threads die in the middle of processing requests, just let the program exit, and the daemon threads will be taken care of.
On the other hand, if you want to make sure a thread exits only between fully processing requests, either use dismissWorkers with do_join=True, or use dismissWorkers followed by joinAllDismissedWorkers.
That choice would vary depending on what you're processing and how. Note that the sample code that comes in the main routine does some of one and some of the other, which is probably not what you'd want to do in a real situation – the sample code is just designed to demonstrate capabilities.
You could argue that it's bad form to create daemon threads when you do care about how/when they exit, and it wouldn't be hard to fix the library so that daemon is an option for your worker threads when they are created, not a necessity. Currently, however, the module picks a default that favors ease of use over consistency.
I have a multiprocessing application where the parent process creates a queue and passes it to worker processes. All processes use this queue for creating a queuehandler for the purpose of logging. There is a worker process reading from this queue and doing logging.
The worker processes continuously check if parent is alive or not. The problem is that when I kill the parent process from command line, all workers are killed except for one. The logger process also terminates. I don't know why one process keeps executing. Is it because of any locks etc in queue? How to properly exit in this scenario? I am using
sys.exit(0)
for exiting.
I would use sys.exit(0) only if there is no other chance. It's always better to cleanly finish each thread / process. You will have some while loop in your Process. So just do break there, so that it can come to an end.
Tidy up before you leave, i.e., release all handles of external resources, e.g., files, sockets, pipes.
Somewhere in these handles might be the reason for the behavior you see.