Take a look at this simple python code with Process:
from multiprocessing import Process
import time
def f(name):
time.sleep(100)
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()#Has to be terminated in 5 seconds
#p.join()
print "This Needs to be Printed Immediately"
I guess I am looking for a function like p.start(timeout).
I want to terminate the p process if it has not self-finished in like 5 seconds. How can I do that? There seems to be no such function.
If p.join() is uncommented, the following print line will have to wait 100 seconds and can not be 'Printed Immediately'.But I want it be done immediately so the p.join() has to be commented out.
Use a separate thread to start the process, wait 5 seconds, then terminate the process. Meanwhile the main thread can do the work you want to happen immediately:
from multiprocessing import Process
import time
import threading
def f(name):
time.sleep(100)
print 'hello', name
def run_process_with_timeout(timeout, target, args):
p = Process(target=target, args=args)
p.start()
time.sleep(timeout)
p.terminate()
if __name__ == '__main__':
t = threading.Thread(target=run_process_with_timeout, args=(5,f,('bob',)))
t.start()
print "This Needs to be Printed Immediately"
You might want to take a look at that SO thread.
basically their solution is to use the timeout capability of the threading module by running the process in a separate thread.
You are right, there is no such function in Python 2.x in the subprocess library.
However, with Python 3.3 you can use:
p = subprocess.Popen(...)
try:
p.wait(timeout=5)
except TimeoutError:
p.kill()
With older Python versions, you would have to write a loop that calls p.poll() and checks the returncode, e.g. once per second.
This is (like polling in general) not optimal from performance point-of-view, but it always depends on what you expect.
Try something like this:
def run_process_with_timeout(timeout, target, args):
p = Process(target=target, args=args)
running = False
second = int(time.strftime("%S"))
if second+timeout > 59:
second = (second+timeout)-60
else:
second = second+timeout
print second
while second > int(time.strftime("%S")):
if running == False:
p.start()
running = True
p.terminate()
basically just using the time module to allow a loop to run for five seconds and then moving on, this assumes timeout is given in seconds.
Though I'd point out that if this was used with the code the OP originally posted, this would work, as print was in a second function separate from the loop and would be carried out immediately after calling this function.
Why not use the timeout option of Process.join(), as in:
import sys
...
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()#Has to be terminated in 5 seconds
# print immediately and flush output
print "This Needs to be Printed Immediately"
sys.stdout.flush()
p.join(5)
if p.is_alive():
p.terminate()
Related
I am trying to run, pause and terminate the child processes in Python from the parent process. I have tried to use multiprocessing.Value but for some reason the parent process never finishes completely although I terminate and join all the processes. My use case is something like:
def child(flow_flag):
while True:
with flow_flag.get_lock():
flag_value = flow_flag.value
if flag_value == 0:
print("This is doing some work")
elif flag_value == 1:
print("This is waiting for some time to check back later")
time.sleep(5)
else:
print("Time to exit")
break
def main():
flow_flag = Value('i', 0)
processes = [Process(target=child, args=(flow_flag,)) for i in range(10)]
[p.start() for p in processes]
print("Waiting for some work")
with flow_flag.get_lock():
flow_flag.value = 1
print("Do something else")
with flow_flag.get_lock():
flow_flag.value = 0
print("Waiting for more work")
with flow_flag.get_lock():
flow_flag.value = 2
print("Exiting")
for p in processes:
p.terminate()
p.join()
This never finishes properly and I have to Ctrl+C eventually. Then I see this message:
Traceback (most recent call last):
File "/home/abcde/anaconda3/lib/python3.7/threading.py", line 1308, in _shutdown
lock.acquire()
KeyboardInterrupt
What is a better way? FYI, while waiting for something else, I am spawning some other processes. I also had them not terminating properly, and I was using Value with them too. It got fixed when I switched to using Queue for them. However, Queue does not seem to be appropriate for the case above.
P.S. : I am ssh'ing into Ubuntu 18.04.
EDIT: After a lot of debugging, not exiting turned out to be because of a library I am using that I did not suspect to cause this. My apologies for false alarm. Thanks for the suggestions on the better way of controlling the child processes.
Your program works for me, but let me chime in on "is there another way". Instead of polling at 5 second intervals you could create a shared event object that lets the child processes know when they can do their work. Instead of polling for Value 1, wait for the event.
from multiprocessing import *
import time
import os
def child(event, times_up):
while True:
event.wait()
if times_up.value:
print(os.getpid(), "time to exit")
return
print(os.getpid(), "doing work")
time.sleep(.5)
def main():
manager = Manager()
event = manager.Event()
times_up = manager.Value(bool, False)
processes = [Process(target=child, args=(event, times_up)) for i in range(10)]
[p.start() for p in processes]
print("Let processes work")
event.set()
time.sleep(2)
print("Make them stop")
event.clear()
time.sleep(4)
print("Make them go away")
times_up.value = True
event.set()
print("Exiting")
for p in processes:
p.join()
if __name__ == "__main__":
main()
With Python 3.7.7 running on FreeBSD 12.1 (64-bit) I cannot reproduce your problem.
After fixing the indentation and adding the necessary imports The changed program runs fine AFAICT.
BTW, you might want to import sys and add
sys.stdout.reconfigure(line_buffering=True)
to the beginning of your main().
I have the codes like this:
It is clear the 'finished' has been printed out. but join still blocks.
Why should this happend?
from multiprocessing import Process
class MyProcess(Process):
def run(self):
## do someting
print 'finished'
processes = []
for i in range(3):
p = MyProcess()
p.start()
processes.append(p)
for p in processes:
p.join()
you should add this line if __name__ == '__main__': for things to work properly
Explanation:
your main script will be imported by process.py module, then it will execute your script lines 2 times, one during importing and one from your script execution,
here is the runtime error if we didn't include if __name__ == '__main__':
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
your working code in python 3.6 is:
from multiprocessing import Process
class MyProcess(Process):
def run(self):
## do someting
print ('finished')
processes = []
if __name__ == '__main__':
for i in range(3):
p = MyProcess()
p.start()
processes.append(p)
for p in processes:
p.join()
print('we are done here .......')
output:
finished
finished
finished
we are done here .......
join would not block if the task is finished, also your program is invalid.
for i in 3: # X integer is not iterable,
for i in range(3): # should be like this.
I've been reading up and trying to implement multithreading into my program, but no matter how I do it, it will not run my functions in parallel. I'm using sensors for a raspberry pi 3, trying to have them print out statuses in parallel rather than wait for one to finish and then move to the next function.
What happens now is that it waits for those 20 seconds before the program checks the seconds sensor and prints out that status message. I have no idea why!
Code:
import RPi.GPIO as GPIO
import time
from multiprocessing import Process
''' Define pins and setup the sensors '''
def runInParallel(*fns):
proc = []
for fn in fns:
p = Process(target=fn)
p.start()
proc.append(p)
for p in proc:
p.join()
def sensor1():
#Sleep timer long so I can check that I can see prints from 2nd sensor while this thread is sleeping
time.sleep(20)
#Get status from sensor---
if status == 1:
print "Ouch!"
else:
print "Good!"
def sensor2():
time.sleep(0.2)
#Get status from 2nd sensor---
if status == 1:
print "Ouch2!"
else:
print "Good2!"
runInParallel(sensor1, sensor2)
I don't know why your example isn't working, but I tried this:
import time
from threading import Thread
''' Define pins and setup the sensors '''
status = 0
def runInParallel(*fns):
proc = []
for fn in fns:
p = Thread(target=fn)
proc.append(p)
for p in proc:
p.start()
def sensor1():
#Sleep timer long so I can check that I can see prints from 2nd sensor while this thread is sleeping
time.sleep(.2)
#Get status from sensor---
if status == 1:
print("Ouch!")
else:
print("Good!")
def sensor2():
time.sleep(0.2)
#Get status from 2nd sensor---
if status == 1:
print("Ouch2!")
else:
print("Good2!")
runInParallel(sensor1, sensor2)
and it outputted good2 and good almost simultaneously. If you really need the outputs to be exact then try to debug you example, but if closer than you can notice with the naked eye is OK then I think that the threading module will work well enough.
EDIT:
Ok I think that you problem is that you think that the Process.join() counts the waits in the function. Process.join() only makes sure that the functions start at the same time. If you have a wait in one function, then runInParallel won't care about that.
I'm struggling to get my head around multiprocessing and passing a global True/False variable into my function.
After get_data() finishes I want the analysis() function to start and process the data, while fetch() continues running. How can I make this work? TIA
import multiprocessing
ready = False
def fetch():
global ready
get_data()
ready = True
return
def analysis():
analyse_data()
if __name__ == '__main__':
p1 = multiprocessing.Process(target=fetch)
p2 = multiprocessing.Process(target=analysis)
p1.start()
if ready:
p2.start()
You should run the two processes and use a shared queue to exchange information between them, such as signaling the completion of an action in one of the processes.
Also, you need to have a join() statement to properly wait for completion of the processes you spawn.
from multiprocessing import Process, Queue
import time
def get_data(q):
#Do something to get data
time.sleep(2)
#Put an event in the queue to signal that get_data has finished
q.put('message from get_data to analyse_data')
def analyse_data(q):
#waiting for get_data to finish...
msg = q.get()
print msg #Will print 'message from get_data to analyse_data'
#get_data has finished
if __name__ == '__main__':
#Create queue for exchanging messages between processes
q = Queue()
#Create processes, and send the shared queue to them
processes = [Process(target=get_data,args(q,)),Process(target=analyse_data,args=(q,))]
#Start processes
for p in processes:
p.start()
#Wait until all processes complete
for p in processes:
p.join()
You example won't work for a few reasons :
Process cannot share a piece of memory with each other (you can't change the global in one process and see the change in the other)
Even if you could change the global value, you are checking it too fast and most likely it won't change in time
Read https://docs.python.org/3/library/ipc.html for more possibilities for inter-process-communications
I am trying to insert and update a few million rows using psycopg and multiprocessing. Going by the documentation found in http://initd.org/psycopg/docs/usage.html#thread-and-process-safety, each child has its own connection to the DB.
But during the course of execution, only one child runs while the others become zombies. The script in itself is pretty simple and here is a trimmed version of the same,
import os
import psycopg2
from multiprocessing import Process
def _target(args):
# Each forked process will have its own connection
# http://initd.org/psycopg/docs/usage.html#thread-and-process-safety
conn = get_db_connection()
# Stuff seems to execute till this point in all the children
print os.getpid(), os.getppid()
# Do some updates here. After this only one child is active and running
# Others become Zombies after a while.
if __name__ == '__main__':
args = "Foo"
for i in xrange(3):
p = Process(target=_target, args=(args,))
p.start()
I also checked if the tables have an escalated lock by peeking into pg_locks, but it looks like its not the case. Am I missing something obvious?
your processes become zombies because there jobs are finished but the processes are not joined.
I reproduced your problem with this single test (I added sleep to simulate long jobs) :
import os
import time
from multiprocessing import Process
def _target(args):
print os.getpid(), os.getppid()
time.sleep(2)
print os.getpid(), "will stop"
if __name__ == '__main__':
args = "Foo"
for i in xrange(3):
p = Process(target=_target, args=(args,))
p.start()
import time
time.sleep(10)
when executing this, after the 3 processes print that they will stop, they become in the ps view (they don't move anymore, but are not really dead because the father still hold them).
If I replace the main part with this, i have no more zombies :
if __name__ == '__main__':
args = "Foo"
processes = []
for i in xrange(3):
p = Process(target=_target, args=(args,))
processes.append(p)
p.start()
for p in processes:
p.join()
import time
time.sleep(10)