We Keep Coding

Python : Set the name of a Process from multiprocessing.Process

Is it possible to set the name of the Processes spawned by multiprocessing.Process or billiard.Process. SOmething like:
import billiard
for d in list:
processes.append(billiard.Process(target=evaluate))
for p in processes:
p.name = 'someID'
p.start()
I want to find those specific processes afterwards while they are running with:
import psutil
for proc in psutil.process_iter():
if proc.name() == 'someID':
print(proc)

If I may suggest, use process id instead of process name - thats way better, and would save you a whole lot of trouble.. since you just need a reference to the process later, use their ids instead of name.
Python 3.6 Official Docs have illustrated a very good way of playing around with process ids. Here is a snippet from the documentation
from multiprocessing import Process
import os
def info(title):
print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())
def f(name):
info('function f')
print('hello', name)
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
Also I see, Celery in your tags, if you're going to play with concurrent processes, I would recommend Supervisord instead. Gives you a very good control over the processes, if you're having a small scale project.

python how to run process in detached mode

here is a example:
from multiprocessing import Process
import time
def func():
print('sub process is running')
time.sleep(5)
print('sub process finished')
if __name__ == '__main__':
p = Process(target=func)
p.start()
print('done')
what I expect is that the main process will terminate right after it start a subprocess. But after printing out 'done', the terminal is still waiting....Is there any way to do this so that the main process will exit right after printing out 'done', instead of waiting for subprocess? I'm confused here because I'm not calling p.join()

Python will not end if there exists a non-daemon process.
By setting, daemon attribute before start() call, you can make the process daemonic.
p = Process(target=func)
p.daemon = True # <-----
p.start()
print('done')
NOTE: There will be no sub process finished message printed; because the main process will terminate sub-process at exit. This may not be what you want.
You should do double-fork:
import os
import time
from multiprocessing import Process
def func():
if os.fork() != 0: # <--
return # <--
print('sub process is running')
time.sleep(5)
print('sub process finished')
if __name__ == '__main__':
p = Process(target=func)
p.start()
p.join()
print('done')

Following the excellent answer from #falsetru, I wrote out a quick generalization in the form of a decorator.
import os
from multiprocessing import Process
def detachify(func):
"""Decorate a function so that its calls are async in a detached process.
Usage
-----
.. code::
import time
#detachify
def f(message):
time.sleep(5)
print(message)
f('Async and detached!!!')
"""
# create a process fork and run the function
def forkify(*args, **kwargs):
if os.fork() != 0:
return
func(*args, **kwargs)
# wrapper to run the forkified function
def wrapper(*args, **kwargs):
proc = Process(target=lambda: forkify(*args, **kwargs))
proc.start()
proc.join()
return
return wrapper
Usage (copied from docstring):
import time
#detachify
def f(message):
time.sleep(5)
print(message)
f('Async and detached!!!')
Or if you like,
def f(message):
time.sleep(5)
print(message)
detachify(f)('Async and detached!!!')

python subprocess: find out in the signal handler which child terminated

I have a python script which starts multiple commands using subprocess.Popen. I added a signal handler which is called if a child exits. I want to check which child terminated. I can do this by iterating over all children:
#!/usr/bin/env python
import subprocess
import signal
procs = []
def signal_handler(signum, frame):
for proc in procs:
proc.poll()
if proc.returncode is not None:
print "%s returned %s" % (proc.pid, proc.returncode)
procs.remove(proc)
def main():
signal.signal(signal.SIGCHLD, signal_handler)
procs.append(subprocess.Popen(["/bin/sleep", "2"]))
procs.append(subprocess.Popen(["/bin/sleep","5"]))
# wait so the main process does not terminate immediately
procs[1].wait()
if __name__ == "__main__":
main()
I would like to avoid querying all subprocesses. Is there a way to determine in the signal handler which child terminated?

You could achieve a similar result using multiprocessing. You could use the threading package instead if you didn't want to spawn the extra processes. It has pretty much the exact same interface. Basically, each subprocess call happens in a new process, which then launches your sleep processes.
import subprocess
import multiprocessing
def callback(result):
# do something with result
pid, returncode = result
print pid, returncode
def call_process(cmd):
p = subprocess.Popen(cmd)
p.wait()
return p.pid, p.returncode
def main():
pool = multiprocessing.Pool()
pool.apply_async(call_process, [["/bin/sleep", "2"]], callback=callback)
pool.apply_async(call_process, [["/bin/sleep", "5"]], callback=callback)
pool.close()
pool.join()
main()

Multiprocessing in Python on Windows

I am trying out the examples listed in the python docs http://docs.python.org/library/multiprocessing.html
particularly these two on Windows:
1)
from multiprocessing import Process
def f(name):
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
2)
from multiprocessing import Process
import os
def info(title):
print title
print 'module name:', __name__
print 'parent process:', os.getppid()
print 'process id:', os.getpid()
def f(name):
info('function f')
print 'hello', name
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()
Here is the problem: I don't get any output from the child process. It works on Linux though. What is going on?

example 1 works well.( I hope you saved the program in a file and then executed it else it will not recognise the function f at all).
example 2 won't work if u want the parent process's id. There is no getppid in windows.
Just take the print os.getppid and execute, its brilliant as ever !
Please refer this for more by Doug. (UPDATE: The original link isn't working, here is something similar.)

In Python how to call subprocesses under a different user?

For a Linux system, I am writing a program in Python, who spawns child processes. I am using the "multiprocessing" library and I am wondering if there is a method to call sub-processes with a different user than the current one. I'd like to be able to run each subprocess with a different user (like Postfix, for example.)
Any idea or pointers ?

modified python documentation example, I've added setuid in the function to be called, I'm not sure it fits to your needs and you may also need setgid, sedeuid setegid
from multiprocessing import Process
import os
def info(title):
print title
print 'module name:', __name__
print 'parent process:', os.getppid()
print 'process id:', os.getpid()
def f(name):
os.setuid(1000)
info('function f')
print 'hello', name
os.system('id')
if __name__ == '__main__':
info('main line')
p = Process(target=f, args=('bob',))
p.start()
p.join()

You could look in os.setpgid(pid, pgrp) direction.