I have a python script (unix-like, based on RHEL), called MyScript, that has two functions, called A and B. I'd like them to run in different, independent processes (detach B and A):
Start script MyScript
Execute function A
Spawn a new process, passing data from function A to B
While function B runs, continue with function A
When function A completes, exit MyScript even if B is still running
I thought I should use multiprocessing to create a daemon process, but the documentation suggests that's not the right usecase. So, I decided to spawn a child process and child^2 process (the child's child), and then force the child to terminate. While this workaround appears to work, it seems really ugly.
Can you help me make it more pythonic? Does the subprocess module have a method that will operate on a function? Sample code below.
import multiprocessing
import time
import sys
import os
def parent_child():
p = multiprocessing.current_process()
print 'Starting parent child:', p.name, p.pid
sys.stdout.flush()
cc = multiprocessing.Process(name='childchild', target=child_child)
cc.daemon = False
cc.start()
print 'Exiting parent child:', p.name, p.pid
sys.stdout.flush()
def child_child():
p = multiprocessing.current_process()
print 'Starting child child:', p.name, p.pid
sys.stdout.flush()
time.sleep(30)
print 'Exiting child child:', p.name, p.pid
sys.stdout.flush()
def main():
print 'starting main', os.getpid()
d = multiprocessing.Process(name='parentchild', target=parent_child)
d.daemon = False
d.start()
time.sleep(5)
d.terminate()
print 'exiting main', os.getpid()
main()
Here is just a random version of your original code that moves the functionality into a single call spawn_detached(callable). It keeps the detached process running even after the program exits:
import time
import os
from multiprocessing import Process, current_process
def spawn_detached(callable):
p = _spawn_detached(0, callable)
# give the process a moment to set up
# and then kill the first child to detach
# the second.
time.sleep(.001)
p.terminate()
def _spawn_detached(count, callable):
count += 1
p = current_process()
print 'Process #%d: %s (%d)' % (count, p.name, p.pid)
if count < 2:
name = 'child'
elif count == 2:
name = callable.func_name
else:
# we should now be inside of our detached process
# so just call the function
return callable()
# otherwise, spawn another process, passing the counter as well
p = Process(name=name, target=_spawn_detached, args=(count, callable))
p.daemon = False
p.start()
return p
def operation():
""" Just some arbitrary function """
print "Entered detached process"
time.sleep(15)
print "Exiting detached process"
if __name__ == "__main__":
print 'starting main', os.getpid()
p = spawn_detached(operation)
print 'exiting main', os.getpid()
Related
I've searched StackOverflow and although I've found many questions on this, I haven't found an answer that fits for my situation/not a strong python programmer to adapt their answer to fit my need.
I've looked here to no avail:
kill a function after a certain time in windows
Python: kill or terminate subprocess when timeout
signal.alarm replacement in Windows [Python]
I am using multiprocessing to run multiple SAP windows at once to pull reports. The is set up to run on a schedule every 5 minutes. Every once in a while, one of the reports gets stalled due to the GUI interface and never ends. I don't get an error or exception, it just stalls forever. What I would like is to have a timeout function that during this part of the code that is executed in SAP, if it takes longer than 4 minutes, it times out, closes SAP, skips the rest of the code, and waits for next scheduled report time.
I am using Windows Python 2.7
import multiprocessing
from multiprocessing import Manager, Process
import time
import datetime
### OPEN SAP ###
def start_SAP():
print 'opening SAP program'
### REPORTS IN SAP ###
def report_1(q, lock):
while True: # logic to get shared queue
if not q.empty():
lock.acquire()
k = q.get()
time.sleep(1)
lock.release()
break
else:
time.sleep(1)
print 'running report 1'
def report_2(q, lock):
while True: # logic to get shared queue
if not q.empty():
lock.acquire()
k = q.get()
time.sleep(1)
lock.release()
break
else:
time.sleep(1)
print 'running report 2'
def report_3(q, lock):
while True: # logic to get shared queue
if not q.empty():
lock.acquire()
k = q.get()
time.sleep(1)
lock.release()
break
else:
time.sleep(1)
time.sleep(60000) #mimicking the stall for report 3 that takes longer than allotted time
print 'running report 3'
def report_N(q, lock):
while True: # logic to get shared queue
if not q.empty():
lock.acquire()
k = q.get()
time.sleep(1)
lock.release()
break
else:
time.sleep(1)
print 'running report N'
### CLOSES SAP ###
def close_SAP():
print 'closes SAP'
def format_file():
print 'formatting files'
def multi_daily_pull():
lock = multiprocessing.Lock() # creating a lock in multiprocessing
shared_list = range(6) # creating a shared list for all functions to use
q = multiprocessing.Queue() # creating an empty queue in mulitprocessing
for n in shared_list: # putting list into the queue
q.put(n)
print 'Starting process at ', time.strftime('%m/%d/%Y %H:%M:%S')
print 'Starting SAP Pulls at ', time.strftime('%m/%d/%Y %H:%M:%S')
StartSAP = Process(target=start_SAP)
StartSAP.start()
StartSAP.join()
report1= Process(target=report_1, args=(q, lock))
report2= Process(target=report_2, args=(q, lock))
report3= Process(target=report_3, args=(q, lock))
reportN= Process(target=report_N, args=(q, lock))
report1.start()
report2.start()
report3.start()
reportN.start()
report1.join()
report2.join()
report3.join()
reportN.join()
EndSAP = Process(target=close_SAP)
EndSAP.start()
EndSAP.join()
formatfile = Process(target=format_file)
formatfile .start()
formatfile .join()
if __name__ == '__main__':
multi_daily_pull()
One way to do what you want would be to use the optional timeout argument that the Process.join() method accepts. This will make it only block the calling thread at most that length of time.
I also set the daemon attribute of each Process instance so your main thread will be able to terminate even if one of the processes it started is still "running" (or has hung up).
One final point, you don't need a multiprocessing.Lock to control access a multiprocessing.Queue, because they handle that aspect of things automatically, so I removed it. You may still want to have one for some other reason, such as controlling access to stdout so printing to it from the various processes doesn't overlap and mess up what is output to the screen.
import multiprocessing
from multiprocessing import Process
import time
import datetime
def start_SAP():
print 'opening SAP program'
### REPORTS IN SAP ###
def report_1(q):
while True: # logic to get shared queue
if q.empty():
time.sleep(1)
else:
k = q.get()
time.sleep(1)
break
print 'report 1 finished'
def report_2(q):
while True: # logic to get shared queue
if q.empty():
time.sleep(1)
else:
k = q.get()
time.sleep(1)
break
print 'report 2 finished'
def report_3(q):
while True: # logic to get shared queue
if q.empty():
time.sleep(1)
else:
k = q.get()
time.sleep(60000) # Take longer than allotted time
break
print 'report 3 finished'
def report_N(q):
while True: # logic to get shared queue
if q.empty():
time.sleep(1)
else:
k = q.get()
time.sleep(1)
break
print 'report N finished'
def close_SAP():
print 'closing SAP'
def format_file():
print 'formatting files'
def multi_daily_pull():
shared_list = range(6) # creating a shared list for all functions to use
q = multiprocessing.Queue() # creating an empty queue in mulitprocessing
for n in shared_list: # putting list into the queue
q.put(n)
print 'Starting process at ', time.strftime('%m/%d/%Y %H:%M:%S')
print 'Starting SAP Pulls at ', time.strftime('%m/%d/%Y %H:%M:%S')
StartSAP = Process(target=start_SAP)
StartSAP.start()
StartSAP.join()
report1 = Process(target=report_1, args=(q,))
report1.daemon = True
report2 = Process(target=report_2, args=(q,))
report2.daemon = True
report3 = Process(target=report_3, args=(q,))
report3.daemon = True
reportN = Process(target=report_N, args=(q,))
reportN.daemon = True
report1.start()
report2.start()
report3.start()
reportN.start()
report1.join(30)
report2.join(30)
report3.join(30)
reportN.join(30)
EndSAP = Process(target=close_SAP)
EndSAP.start()
EndSAP.join()
formatfile = Process(target=format_file)
formatfile .start()
formatfile .join()
if __name__ == '__main__':
multi_daily_pull()
I am new to python and threading. I am trying to run multiple threads at a time. Here is my basic code :
import threading
import time
threads = []
print "hello"
class myThread(threading.Thread):
def __init__(self,i):
threading.Thread.__init__(self)
print "i = ",i
for j in range(0,i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = myThread(i)
thread.start()
While 1 thread is waiting for time.sleep(5) i want another thread to start. In short, all the threads should run parallel.
You might have some misunderstandings on how to subclass threading.Thread, first of all __init__() method is roughly what represents a constructor in Python, basically it'll get executed every time you create an instance, so in your case when thread = myThread(i) executes, it'll block till the end of __init__().
Then you should move your activity into run(), so that when start() is called, the thread will start to run. For example:
import threading
import time
threads = []
print "hello"
class myThread(threading.Thread):
def __init__(self, i):
threading.Thread.__init__(self)
self.i = i
def run(self):
print "i = ", self.i
for j in range(0, self.i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = myThread(i)
thread.start()
P.S. Because of the existence of GIL in CPython, you might not be able to fully take advantages of all your processors if the task is CPU-bound.
Here is an example on how you could use threading based on your code:
import threading
import time
threads = []
print "hello"
def doWork(i):
print "i = ",i
for j in range(0,i):
print "j = ",j
time.sleep(5)
for i in range(1,4):
thread = threading.Thread(target=doWork, args=(i,))
threads.append(thread)
thread.start()
# you need to wait for the threads to finish
for thread in threads:
thread.join()
print "Finished"
import threading
import subprocess
def obj_func(simid):
simid = simid
workingdir = './' +str (simid) # the working directory for the simulation
cmd = './run_delwaq.sh' # cmd is a bash commend to launch the external execution
subprocess.Popen(cmd, cwd=workingdir).wait()
def example_subprocess_files():
num_threads = 4
jobs = []
# Launch the threads and give them access to the objective function
for i in range(num_threads):
workertask = threading.Thread(target=obj_func(i))
jobs.append(workertask)
for j in jobs:
j.start()
for j in jobs:
j.join()
print('All the work finished!')
if __name__ == '__main__':
example_subprocess_files()
This one not works for my case that the task is not print but CPU-Intensive task. The thread are excluded in serial.
I have faces a very strange behavior of Python. It looks like when I start parallel program which uses multiprocessing and in the main process spawn 2 more(producer, consumer) I see 4 processes running. I think there should be only 3: the main, Producer, Consumer. But after some time the 4th process appears.
I have made a minimal example of the code to reproduce the problem. It create two processes in which calculate Fibonacci numbers using recursion:
from multiprocessing import Process, Queue
import os, sys
import time
import signal
def fib(n):
if n == 1 or n == 2:
return 1
result = fib(n-1) + fib(n-2)
return result
def worker(queue, amount):
pid = os.getpid()
def workerProcess(a, b):
print a, b
print 'This is Writer(', pid, ')'
signal.signal(signal.SIGUSR1, workerProcess)
print 'Worker', os.getpid()
for i in range(0, amount):
queue.put(fib(35 - i % 4))
queue.put('end')
print 'Worker finished'
def writer(queue):
pid = os.getpid()
def writerProcess(a, b):
print a, b
print 'This is Writer(', pid, ')'
signal.signal(signal.SIGUSR1, writerProcess)
print 'Writer', os.getpid()
working = True
while working:
if not queue.empty():
value = queue.get()
if value != 'end':
fib(32 + value % 4)
else:
working = False
else:
time.sleep(1)
print 'Writer finished'
def daemon():
print 'Daemon', os.getpid()
while True:
time.sleep(1)
def useProcesses(amount):
q = Queue()
writer_process = Process(target=writer, args=(q,))
worker_process = Process(target=worker, args=(q, amount))
writer_process.daemon = True
worker_process.daemon = True
worker_process.start()
writer_process.start()
def run(amount):
print 'Main', os.getpid()
pid = os.getpid()
def killThisProcess(a, b):
print a, b
print 'Main killed by signal(', pid, ')'
sys.exit(0)
signal.signal(signal.SIGTERM, killThisProcess)
useProcesses(amount)
print 'Ready to exit main'
while True:
time.sleep(1)
def main():
run(1000)
if __name__=='__main__':
main()
What I see in the output is:
$ python python_daemon.py
Main 13257
Ready to exit main
Worker 13258
Writer 13259
but in htop I see the following:
And it looks like the process with PID 13322 is actually a thread. The question is what is it? Who spawn it? Why?
If I send SIGUSR1 to this PID I see in the output appears:
10 <frame object at 0x7f05c14ed5d8>
This is Writer( 13258 )
This question is slightly related with: Python multiprocessing: more processes than requested
The threads belongs to the Queue object.
It uses internally a thread to dispatch the data over a Pipe.
From the docs:
class multiprocessing.Queue([maxsize])
Returns a process shared queue implemented using a pipe and a few locks/semaphores. When a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe.
I am trying to emulate a scenario where the child spawned by python multiprocessing pool gets killed. The subprocess never returns, but I would like the parent to get notified in such a scenario.The test code I am using is:
import multiprocessing as mp
import time
import os
result_map = {}
def foo_pool(x):
print x,' : ',os.getpid()
pid = os.getpid()
if x == 1:
os.kill(pid,9)
return x
result_list = []
def log_result(result):
print 'callback',result
def apply_async_with_callback():
print os.getpid()
pool = mp.Pool()
for i in range(2):
result_map[i] = pool.apply_async(foo_pool, args = (i, ), callback = log_result)
pool.close()
pool.join()
for k,v in result_map.iteritems():
print k,' : ',v.successful()
print(result_list)
if __name__ == '__main__':
apply_async_with_callback()
The Pool of processes does not expose any mechanism to notify the parent process about a child process termination.
Projects like billiard or pebble might do what you're looking for.
Keep in mind that there's no way to intercept a SIGKILL signal so using signal handlers is pointless.
I'm trying to use os.fork, os.waitpid, and threading to dispatch some time-consuming work to a child process, wait at most a certain amount of time for it to complete, and then continue on, leaving it running in the background. There is also a secondary timer in the child that prevents it from running for too long in the background, like so:
Fork execution
In the fork's child:
Start a thread to execute a task
If that thread runs for longer than X milliseconds:
thread.stop(), a custom method that softly stops the thread.
In the fork's parent:
If the child's PID stays running for longer than Y milliseconds:
return -1
else:
return 0
After returning a value, I want the script to terminate. The child can and should keep running if it isn't done.
The code I have tried (abbreviated) is:
class MyCustomThread(threading.Thread):
abort = False
def run(self):
counter = 0
while True:
if self.abort: break
counter += 1
if counter == 30: self.stop()
sleep(1)
return 0
def stop(self):
self.abort = True
def test():
X = 2000
Y = 500
pid = os.fork()
if pid == 0:
thread1 = MyCustomThread() #Sleeps for 30 seconds and ends.
thread1.start()
print "Started 1!"
timeout = X # say, 1000ms
while timeout > 0:
if not thread1.is_alive(): return "custom thread finished before the deadline!"
timeout -= 1
sleep(0.001)
if thread1.is_alive():
return "custom thread didn't finish before the deadline!"
thread1.stop()
exit()
else:
print pid
thread2 = Thread(target = os.waitpid, args = (pid, 0))
thread2.start()
print "Started 2!"
timeout2 = Y # say, 500ms
while timeout2 > 0:
if not thread2.is_alive(): return "child PID finished!"
timeout2 -= 1
sleep(0.001)
if thread2.is_alive():
return "child PID didn't finish yet!"
print test()
print "all done!"
The output is correct, in that I get
1171
Started 2!
Started 1!
child PID didn't finish yet!
all done!
custom thread didn't finish before the deadline!
all done!
But then the script doesn't exit! It sleeps for the remaining 28 seconds before
How do I make the main execution of this script complete after the forked parent returns a value? As I said before, the child can and should keep running in the background, it just shouldn't block execution of the next task on the parent.
I really don't even care if the child can print output to standard out--in the actual implementation, all it's doing is talking to a database, so it doesn't need to report anything interactively. The parent, however, needs to dispatch the child, wait at most Y seconds for the child to finish, and then (as far as whoever invoked the script is concerned) end the script's execution so that the next thing can be done. The other timer (X) isn't relevant, I think; it only exists to keep the child from running too long in the background.
Any ideas? I'm probably approaching this totally wrong, so "start over and do it _ way" ideas are welcome.
Try this one, it doesn't use threading, just pure fork/waitpid/alarm/sighandler:
child_exited = False
def sigh(signum, frame):
global child_exited
if signum == signal.SIGALRM:
print "custom thread didn't finish before the deadline!"
#forced exit:
exit(1)
if signum == signal.SIGCHLD:
(pid, status) = os.waitpid(-1, 0)
print "child exited with status: " + str(os.WEXITSTATUS(status))
child_exited = True
def test():
global child_exited
pid = os.fork()
if pid == 0:
signal.signal(signal.SIGALRM, sigh)
signal.alarm(30)
#do work:
print "Started 1"
time.sleep(60)
#clean exit:
exit(0)
elif (pid > 0):
signal.signal(signal.SIGCHLD, sigh)
print "Started 2"
#this sleep will return prematurely if the child exits
time.sleep(10)
if not child_exited:
print "child PID didn't finish yet!"
else:
print "fork() failed"
print test()
print "all done!"
This is not an exact answer to your question, but rather a "start over and do it _ way" idea.
You could use the multiprocessing module. The function Pool.apply_async() allows to execute a function in the background, and the returned AsyncResult object features wait() and get() methods with a timeout parameter.
Your code would essentially become (untested)
import multiprocessing
def computationally_intensive():
# whatever
p = multiprocessing.Pool(1)
deferred = p.apply_async(computationally_intensive)
try:
result = deferred.get(10) # 10 being the timeout
except multiprocessing.TimeoutError:
# Handle time out
# ...
p.terminate()
Figured it out, thanks for all the pointers!
Edit: fixed is_pid_alive function to work if called from within a child as well.
The issue was that the parent's "watcher" thread was never completing, since os.waitpid isn't a pollable/loopable function. The solution was to remove the "watcher" thread, and instead implement a polling loop that checks a pid_is_alive() function every millisecond, like so:
def pid_is_alive(pid):
try:
os.kill(pid, 0)
os.waitpid(pid, os.WNOHANG)
os.kill(pid, 0)
except OSError:
return False
return True
def test():
X = 1000 * 1000
Y = 5000
pid = os.fork()
if pid == 0:
thread1 = MyCustomThread() #Sleeps for 30 seconds and ends.
thread1.start()
print "Started 1!"
timeout = X # say, 1000ms
while timeout > 0:
if not thread1.is_alive(): return "custom thread finished before the deadline!"
timeout -= 1
sleep(0.001)
if thread1.is_alive():
return "custom thread didn't finish before the deadline!"
thread1.stop()
exit()
else:
timeout2 = Y # say, 500ms
while timeout2 > 0:
if not pid_is_alive(pid): return "child PID finished!"
timeout2 -= 1
sleep(0.001)
if pid_is_alive(pid):
print "child PID didn't finish yet!"
exit()
print test()
print "all done!"
you may use join method of Thread
thread2.join(timeout=Y)
if not thread2.is_alive():
# TODO: Post Processing