I have a black box legacy FORTRAN package for solving PDEs (partial differential equations) with a python wrapper. Sometimes the FORTRAN PDE solver takes a long time to converge. If the FORTRAN is taking too long to converge, I would like my python script to be able to force the FORTRAN to stop so the python can finish running, perhaps with an error message.
Any thoughts on how to automate this watchdog timer sort of behavior in python when working with a wrapped FORTRAN black box? Would it be possible with multiprocessing?
I found a solution that worked using python's built-in multiprocessing module. Here is a simplified version of my solution. The FORTRAN blackbox has been replaced in the code below with the do_calculation() method. I would be interested in better ways to answer this question, but for now, this is the way I am doing it. Here is the most helpful youtube video I found: https://www.youtube.com/watch?v=iwFGC_3sVio&t=622s
import multiprocessing as mp
import time
import numpy as np
def timer_func(seconds, queue):
print(f" {seconds}-second timer started")
time.sleep(seconds)
print("Time up!")
queue.put("Time up!")
pass
def do_calculation(listy_list, queue):
print("Starting Calculation!")
time.sleep(10)
output = 0
for i in listy_list:
output = np.sqrt((output+i)*i)
print("Done Calculating!")
queue.put(output)
pass
def main():
in_list = np.arange(0,10000)
queue = mp.Queue()
p1 = mp.Process(target=do_calculation, args=[in_list, queue])
p2 = mp.Process(target=timer_func, args=[3, queue])
p1.start()
p2.start()
result = queue.get()
print(f"Queue returned {result}")
if result == "Time up!":
p1.terminate()
print("p1 terminated")
else:
p2.terminate()
print(f"Queue resturned {result}")
p1.join()
p2.join()
# print(f"result of p1 {result}")
pass
if __name__ == '__main__':
main()
Related
I'm creating two instances of a process over here but when I'm running this program I'm getting only main function output.
import multiprocessing
import time
def sleepy_man():
print("Starting to Sleep")
time.sleep(1)
print("Done Sleeping")
tic = time.time()
p1 = multiprocessing.Process(target=sleepy_man)
p2 = multiprocessing.Process(target=sleepy_man)
p1.start()
p2.start()
toc = time.time()
print("Done in {:.4f} seconds".format(toc-tic))
Output
Done in 0.0554 seconds
I was doing it for practice from this blog only.
Source: https://www.analyticsvidhya.com/blog/2021/04/a-beginners-guide-to-multi-processing-in-python/
It is worth noting you would see the same behavior if you had somehow set p1.daemon = p2.daemon = True.
It is also possibly due to output buffering, rather than logic errors.
Two questions:
If you add a sys.stdout.flush() or flush=True to your print, do you see different behavior?
If you run this with time python foobar.py does it take .02s or 1s to run?
Obviously, continuing your tutorial and correctly adding .join() below will resolve the issue in a way that would be expected for normal usage.
import multiprocessing as mp
import time
def sleepy_man():
print("Starting to Sleep")
time.sleep(1)
print("Done Sleeping")
# if you are on Windows, which use spawning to create child processes, use __name__ == '__main__'
if __name__ == '__main__':
tic = time.time()
processes = [
mp.Process(target=sleepy_man),
mp.Process(target=sleepy_man)
]
[p.start() for p in processes]
# if you want to see results of process work, join them
# otherwise if main process finish its work before their children
# you'll get no results since parent process will finish children
# you can also declare Process as daemon=False - as another choice
# in that case you can use no join()
# on the other hand join() makes parent process to wait for children join()
# and only then it prints time in your case
[p.join() for p in processes]
toc = time.time()
print("Done in {:.4f} seconds".format(toc-tic))
I just want to see a simple code implementation of multiprocessing on windows, but it doesn't enter/run functions neither in jupyternotebook or running saved .py
import time
import multiprocessing
s=[1,4]
def subu(remo):
s[remo-1]=remo*9
print(f'here{remo}')
return
if __name__=="__main__":
p1=multiprocessing.Process(target=subu , args=[1])
p2=multiprocessing.Process(target=subu , args=[2])
p1.start()
p2.start()
p1.join()
p2.join()
# print("2222here")
print(s)
input()
the output by .py is:
[1, 4]
[1, 4]
and the output by jupyternotebook is:
[1,4]
which I hoped to be:
here1
here2
[9,18]
what's wrong with code above? and what about this code:
import concurrent
thread_num=2
s=[1,4]
def subu(remo):
s[remo-1]=remo*9
print(f'here{remo}')
return
with concurrent.futures.ProcessPoolExecutor() as executor:
## or if __name__=="__main__":
##... with concurrent.futures.ProcessPoolExecutor() as executor:
results=[executor.submit(subu,i) for i in range(thread_num)]
for f in concurrent.futures.as_completed(results):
print(f.result())
input()
doesnot run at all in jupyter pulling error
BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
I kinda know I can't expect jupyter to run multiprocessing. but saved.py also can't run it. and it exits without waiting for input()
There are a couple of potential problems. The worker function needs to be importable (at least on Windows) so that it can be found by the subprocess. And since subprocess memory isn't visible to the parent, the results need to be returned. So, putting the worker in a separate module
subumodule.py
def subu(remo):
remo = remo*9
print(f'here{remo}')
return remo
And using a process pool's existing infrastructure to return a worker return value to the parent. You could
import time
import multiprocessing
if __name__=="__main__":
with multiprocessing.Pool(2) as pool:
s = list(pool.map(subu, (1,2))) #here
print(s)
input()
On a machine running Windows Server 2012 R2, in the Spyder IDE from Anaconda and running Python 3.7 with the following code:
import time
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
And I get the output
name is main
Program completed in 0.13s
Goodbye!
My expectation was that I would see the two print statements
func start
func end
and also (because .join was envoked) that the program would take >1s to complete.
I suspect that the .start() call did not successfully call the do_something function.
FYI, I am following this tutorial, which I know needs to be modified to include the if statement on windows. I am also seeing similar code on other sites, but doesn't seem to work on my end.
Any suggestions on troubleshooting would be much appreciated.
**EDIT: Per the comment below from Azy_Crw4282, the code seems to work on his end and, per his suggestion, it seems to work from the cmd prompt. So this seems to be a bug specifically with the Spyder IDE.
FYI, I wanted to understand whether the issue was that the process was getting kicked off but the IDE wasn't capturing the output OR the process wasn't getting kicked off. I tried two things, 1) the code below writes a dataframe to csv. When doing this in the multiprocessing function, it does NOT write the file. 2) I created a global variable and changed variable value in the function. Spyder keeps the variable values after the code runs, and when I printed the variable it was unchanged.
So, in summary - it seems that the Spyder IDE does not work with the multiprocessing module.**
import time
import multiprocessing
start=time.perf_counter()
df=pd.DataFrame(data={'Col1':[1.1,2.1,3.1],
'Col2':[1.2,2.2,3.2],
'Col3':[1.3,2.3,3.3]}, columns=['Col1','Col2','Col3'])
def do_something():
print('func start')
df.to_csv('C:/testMp.csv')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
When I ran your code, I get the following output. Can you try running your code in another ide/ cmd line/ terminal?
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
Outputs:
name is main
Program completed in 0.0s
Goodbye!
func start
func end
Program completed in 1.27s
Goodbye!
The above result is not probably what you expect. In order to achieve, what you want, you need to indent the outer print section so that it appears within the main call.
import time
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
Outputs:
name is main
func start
func end
Program completed in 1.33s
Goodbye!
I am sure many similar questions have been asked before, but after reading many of them I am still not very sure what I should do. So, I have a Python script to control some external instruments (a camera and a power meter). I have written class for both instruments by calling the C functions in the .dll files using ctypes. Right now it looks something like this:
for i in range(10):
power_reading = newport.get_reading(N=100,interval=1) # take power meter reading
img = camera.capture(N=10)
value = image_processing(img) # analyze the img (ndarray) to get some values
results.append([power_reading,value]) # add both results to a list
I want to start executing the first two lines at the same time. Both newport.get_reading and camera.capture take about 100ms-1s to run (they will run for the same time if I choose the correct arguments). I don't need them to start at EXACTLY the same time, but ideally the delay should be smaller than about 10-20% of the total run time (so less than 0.2s delay when take each take about 1s to run). From what I have read, I can use the multiprocessing module. So I try something like this based on this post:
def p_get_reading(newport,N,interval,return_dict):
reading = newport.get_reading(N,interval,return_dict)
return_dict['power_meter'] = reading
def p_capture(camera,N,return_dict):
img = camera.capture(N)
return_dict['image'] = img
for i in range(10):
manager = multiprocessing.Manager()
return_dict = manager.dict()
p = multiprocessing.Process(target=p_capture, args=(camera,10))
p.start()
p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1))
p2.start()
p.join()
p2.join()
print(return_dict)
I have a few problems/questions:
I need to get the return values from both function calls. Using my current method, return_dict is only showing the entry for capture_img but not the power meter reading, why is that? It also read that I can use Queue, what is the best method for my current purpose?
How can I know whether both functions indeed start running at the same time? I am thinking of using the time module to record both the start and end time of both functions, maybe using some wrapper function to do the time logging, but will the use of multiprocessing pose any restrictions?
I usually run my code on an IDE (spyder), and from what I have read, I need to run in command prompt to see the output (I have some print statements inside the functions for debugging purposes). Can I still run in IDE for having both functions run at the same time?
Using a Lock may help with synchronisation:
import multiprocessing
def p_get_reading(newport, N, interval, lock, return_dict):
lock.acquire()
lock.release()
reading = newport.get_reading(N, interval)
return_dict['power_meter'] = reading
def p_capture(camera, N, lock, return_dict):
lock.acquire()
lock.release()
img = camera.capture(N)
return_dict['image'] = img
if __name__ == "__main__":
for i in range(10):
manager = multiprocessing.Manager()
return_dict = manager.dict()
lock = multiprocessing.Lock()
lock.acquire()
p = multiprocessing.Process(target=p_capture, args=(camera,10,lock,return_dict))
p.start()
p2 = multiprocessing.Process(target=p_get_reading, args=(newport,100,1,lock,return_dict))
p2.start()
lock.release()
p.join()
p2.join()
print(return_dict)
The two Process objects can now be created and start()ed in any order as the main routine has already acquired the lock. Once released, the two processes will fight between themselves to acquire and release the lock, and be ready almost at the same time.
Also, note the use of if __name__ == "__main__" as this helps when multiprocessing makes new processes.
I must say this seems like an abuse of a Lock
An answer to your first question is simply no if you are doing in normal way, but yes if you want it to be. No because the target function cannot communicate back to spawning thread using a return. One way to do it is to use a queue and wrapper functions as following:
from threading import Thread
from Queue import Queue
def p_get_reading(newport,N,interval,return_dict):
reading = newport.get_reading(N,interval,return_dict)
return_dict.update({'power_meter': reading})
return return_dict
def p_capture(camera,N,return_dict):
img = camera.capture(N)
return_dict.update({'image': img})
return return_dict
def wrapper1(func, arg1, arg2, queue):
queue.put(func(arg1, arg2))
def wrapper2(func, arg1, arg2, arg3, queue):
queue.put(func(arg1, arg2, arg3))
q = Queue()
Thread(target=wrapper1, args=(p_capture, camera, 10 , q)).start()
Thread(target=wrapper2, args=(p_get_reading, newport, 100, 1, q)).start()
Now q holds the updated and returned dict from p_capture() and p_get_reading().
I trying to play around with multi-threading so I can better at it, but for some weird reason, my code doesn't want to follow the commands. It's suppose to go into a while loop and print, but it doesn't, and it's also not raising any errors, so which lines is the mistake on?
#!/usr/bin/env python
#
#
#
import random
import thread
import time
import sys
import os
def DisplayA(name,wait):
while True:
print 'Display: 1';time.sleep(wait)
def DisplayB(name,wait):
while True:
print 'Display: 2';time.sleep(wait)
def DisplayC(name,wait):
while True:
print 'Display: 3';time.sleep(wait)
thread.start_new_thread(DisplayA,('Display1',2))
thread.start_new_thread(DisplayB,('Display2',3))
thread.start_new_thread(DisplayC,('Display3',5))
Add this to the bottom:
while True:
pass
The problem is that you're running off the bottom of your main program. This terminates the entire execution session.
Quick and short solution:
while True:
time.sleep(1)
Do not use pass in the while loop, because it eats CPU.
Expensive way of doing nothing.
If you want a more general solution, then you can import Tread from threading, then you can use join:
from threading import Thread
...
p1 = Thread(name="A", target=DisplayA, args=('Display1',2))
p2 = Thread(name="B", target=DisplayB, args=('Display2',3))
p3 = Thread(name="C", target=DisplayC, args=('Display3',5))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
This solution works also if the threads do not run endless, and your program can continue after the threads have finished.
You can either do what Prune here suggested, or you can suspend the main thread after initiating DisplayA, DisplayB and DisplayC.