So, I've been playing with Python's multiprocessing module for a few days now, and there's something that I can't understand. Maybe someone can give me a bit of help.
I'm trying to run two methods from the same class in parallel, but apparently there's something that I'm missing:
from multiprocessing import Process
import time
class SomeClass:
def __init__(self):
pass
def meth1(self):
print(1)
time.sleep(10)
def meth2(self):
print(2)
time.sleep(5 * 60)
def main():
while True:
s = SomeClass()
p1 = Process(target=s.meth1) # I want this to run from 10 to 10 seconds
p1.start()
p2 = Process(target=s.meth2) # I want this to run from 5 to 5 minutes
# while the first one still does its own
# job from 10s to 10s
p2.start()
p1.join()
p2.join()
if __name__ == '__main__':
main()
What I would expect to happen is:
the first method should print 1;
then the second one should print 2 (without waiting 10s - which does happen and seems to work as expected);
then I should see only 1s for the next 4 minutes and 50s (this isn't happening and the program just waits for that time to pass.
What am I missing? Why does the second step work as expected, but the 3rd one doesn't? How can I make it work?
Its difficult to really know what you want, but this code below does what you describe, but has no convenient way of exiting:
from multiprocessing import Process
import time
class SomeClass:
def __init__(self):
pass
def meth1(self):
while True:
print(1)
time.sleep(10)
def meth2(self):
while True:
print(2)
time.sleep(5 * 60)
def main():
s = SomeClass()
p1 = Process(target=s.meth1) # I want this to run from 10 to 10 seconds
p1.start()
p2 = Process(target=s.meth2) # I want this to run from 5 to 5 minutes while the first one still does its own job from 10s to 10s
p2.start()
p1.join()
p2.join()
if __name__ == '__main__':
main()
I have moved the while into each of the methods of SomeClass
This code will never exit, hanging at p1.join()
Think of processes as friends / clones of yours that you can call on the phone, have come in and do something, and then when they are all done they go home, leaving you a note.
The lines:
p1 = Process(target=s.meth1)
p1.start()
call one clone up and have him run s.meth1. He comes over and prints 1 on your screen, waits 10 seconds, then leaves you a note: "all done, no exceptions occurred, I've gone home".
Meanwhile (while your first clone is coming over), the lines:
p2 = Process(target=s.meth2)
p2.start()
call up another clone and have him run s.meth2. He comes over, prints 2 on your screen, waits around for 5 minutes, then leaves you a note: "all done, no exceptions occurred, I've gone home".
While clones #1 and #2 are doing their work, the line:
p1.join()
waits for clone #1 to leave you his note. That happens after 10 seconds. You then go on to:
p2.join()
which waits for clone #2 to leave you his note. That happens after another 4 minutes and 50 seconds. Then you go back around to your while True and start everything over again.
If you want not to wait for clone #2 to finish and leave you his note, don't call p2.join() yet. Eventually, though, you should call p2.join() to make sure that everything went well and that he went home successfully, and isn't lying dead in your living room in a pool of blood, shot by some exception. :-)
Related
I'm creating two instances of a process over here but when I'm running this program I'm getting only main function output.
import multiprocessing
import time
def sleepy_man():
print("Starting to Sleep")
time.sleep(1)
print("Done Sleeping")
tic = time.time()
p1 = multiprocessing.Process(target=sleepy_man)
p2 = multiprocessing.Process(target=sleepy_man)
p1.start()
p2.start()
toc = time.time()
print("Done in {:.4f} seconds".format(toc-tic))
Output
Done in 0.0554 seconds
I was doing it for practice from this blog only.
Source: https://www.analyticsvidhya.com/blog/2021/04/a-beginners-guide-to-multi-processing-in-python/
It is worth noting you would see the same behavior if you had somehow set p1.daemon = p2.daemon = True.
It is also possibly due to output buffering, rather than logic errors.
Two questions:
If you add a sys.stdout.flush() or flush=True to your print, do you see different behavior?
If you run this with time python foobar.py does it take .02s or 1s to run?
Obviously, continuing your tutorial and correctly adding .join() below will resolve the issue in a way that would be expected for normal usage.
import multiprocessing as mp
import time
def sleepy_man():
print("Starting to Sleep")
time.sleep(1)
print("Done Sleeping")
# if you are on Windows, which use spawning to create child processes, use __name__ == '__main__'
if __name__ == '__main__':
tic = time.time()
processes = [
mp.Process(target=sleepy_man),
mp.Process(target=sleepy_man)
]
[p.start() for p in processes]
# if you want to see results of process work, join them
# otherwise if main process finish its work before their children
# you'll get no results since parent process will finish children
# you can also declare Process as daemon=False - as another choice
# in that case you can use no join()
# on the other hand join() makes parent process to wait for children join()
# and only then it prints time in your case
[p.join() for p in processes]
toc = time.time()
print("Done in {:.4f} seconds".format(toc-tic))
What I'd like to do is the following program to print out:
Running Main
Running Second
Running Main
Running Second
[...]
Code:
from multiprocessing import Process
import time
def main():
while True:
print('Running Main')
time.sleep(1)
def second():
while True:
print('Running Second')
time.sleep(1)
p1 = Process(main())
p2 = Process(second())
p1.start()
p2.start()
But it doesn't have the desired behavior. Instead it just prints out:
Running Main
Running Main
[...]
I suspect my program doesn't work because of the while statement?
Is there any way I can overcome this problem and have my program print out what I mentioned no matter what I execute in my function?
The issue here seems to be when you make the process vars. I suspect the reason for why the process inclusively runs the first function is because of syntax. My interpretation is that instead of creating a process out of a function you are making a process that executes a function exclusively.
When you want to create Process object you want to avoid using this
p1 = Process(target=main())
and rather write
p1 = Process(target=main)
That also means if you want to include any input for the function you will have to
p1 = Process(target=main, args=('hi',))
On a machine running Windows Server 2012 R2, in the Spyder IDE from Anaconda and running Python 3.7 with the following code:
import time
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
And I get the output
name is main
Program completed in 0.13s
Goodbye!
My expectation was that I would see the two print statements
func start
func end
and also (because .join was envoked) that the program would take >1s to complete.
I suspect that the .start() call did not successfully call the do_something function.
FYI, I am following this tutorial, which I know needs to be modified to include the if statement on windows. I am also seeing similar code on other sites, but doesn't seem to work on my end.
Any suggestions on troubleshooting would be much appreciated.
**EDIT: Per the comment below from Azy_Crw4282, the code seems to work on his end and, per his suggestion, it seems to work from the cmd prompt. So this seems to be a bug specifically with the Spyder IDE.
FYI, I wanted to understand whether the issue was that the process was getting kicked off but the IDE wasn't capturing the output OR the process wasn't getting kicked off. I tried two things, 1) the code below writes a dataframe to csv. When doing this in the multiprocessing function, it does NOT write the file. 2) I created a global variable and changed variable value in the function. Spyder keeps the variable values after the code runs, and when I printed the variable it was unchanged.
So, in summary - it seems that the Spyder IDE does not work with the multiprocessing module.**
import time
import multiprocessing
start=time.perf_counter()
df=pd.DataFrame(data={'Col1':[1.1,2.1,3.1],
'Col2':[1.2,2.2,3.2],
'Col3':[1.3,2.3,3.3]}, columns=['Col1','Col2','Col3'])
def do_something():
print('func start')
df.to_csv('C:/testMp.csv')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
When I ran your code, I get the following output. Can you try running your code in another ide/ cmd line/ terminal?
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
Outputs:
name is main
Program completed in 0.0s
Goodbye!
func start
func end
Program completed in 1.27s
Goodbye!
The above result is not probably what you expect. In order to achieve, what you want, you need to indent the outer print section so that it appears within the main call.
import time
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
Outputs:
name is main
func start
func end
Program completed in 1.33s
Goodbye!
Use Python programming language to accomplish the following task:
Create two processes (let’s call them P1 and P2). P1 should print “I am P1”, P2 should print “I am P2”. The main process (the process that creates P1 and P2) should wait for them. Then, after P1 and P2 are done, the main process should print “I am the main process, the two processes are done”.
In windows, we don't have fork system call, so we can use a python module called multiprocessing as:-
from multiprocessing import Process, Lock
import time
import os
def f(lock,id,sleepTime):
lock.acquire()
print "I'm P"+str(id)+" Process ID: "+str(os.getpid())
lock.release()
time.sleep(sleepTime) #sleeps for some time
if __name__ == '__main__':
print "Main Process ID: "+str(os.getpid())
lock=Lock()
p1=Process(target=f, args=(lock,1,3,)) #P1 sleeps for 3 seconds
p2=Process(target=f, args=(lock,2,5,)) #P2 sleeps for 5 seconds
start=time.time()
p1.start()
p2.start()
p1.join()
p2.join()
end=time.time()
print "I am the main process, the two processes are done"
print "Time taken:- "+str(end-start)+"secs" #MainProcess terminates at approx ~ 5 secs.
The processes as captured in task manager:-
The code output was:-
Main Process ID: 9804
I'm P1 Process ID: 6088
I'm P2 Process ID: 4656
I am the main process, the two processes are done
Time taken:- 5.15300011635secs
Hope that helps!!
I didn't notice the Windows tag first. So I wrote according to UNIX. I kept the answer instead of deleting in hoping that it will aid UNIX users too.The proper code demonstrating the same is:-
import os
import time
def child(id, sleepTime):
print "I'm P"+str(id)
time.sleep(sleepTime)
os._exit(0)
p1=os.fork()
if (p1==0):
child(1,3) #P1 sleeps for 3 seconds
p2=os.fork()
if (p2==0):
child(2,5) #P2 sleeps for 5 seconds
if (p1>0 and p2>0):
os.waitpid(p1,0) #Waiting for child 1
os.waitpid(p2,0) #Waiting for child2
print "I am the main process, the two processes are done" #Printed after approx 5 seconds
I executed
time python fork.py
The output was as expected:-
I'm P1
I'm P2
I am the main process, the two processes are done
real 0m5.020s
user 0m0.004s
sys 0m0.008s
I trying to play around with multi-threading so I can better at it, but for some weird reason, my code doesn't want to follow the commands. It's suppose to go into a while loop and print, but it doesn't, and it's also not raising any errors, so which lines is the mistake on?
#!/usr/bin/env python
#
#
#
import random
import thread
import time
import sys
import os
def DisplayA(name,wait):
while True:
print 'Display: 1';time.sleep(wait)
def DisplayB(name,wait):
while True:
print 'Display: 2';time.sleep(wait)
def DisplayC(name,wait):
while True:
print 'Display: 3';time.sleep(wait)
thread.start_new_thread(DisplayA,('Display1',2))
thread.start_new_thread(DisplayB,('Display2',3))
thread.start_new_thread(DisplayC,('Display3',5))
Add this to the bottom:
while True:
pass
The problem is that you're running off the bottom of your main program. This terminates the entire execution session.
Quick and short solution:
while True:
time.sleep(1)
Do not use pass in the while loop, because it eats CPU.
Expensive way of doing nothing.
If you want a more general solution, then you can import Tread from threading, then you can use join:
from threading import Thread
...
p1 = Thread(name="A", target=DisplayA, args=('Display1',2))
p2 = Thread(name="B", target=DisplayB, args=('Display2',3))
p3 = Thread(name="C", target=DisplayC, args=('Display3',5))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
This solution works also if the threads do not run endless, and your program can continue after the threads have finished.
You can either do what Prune here suggested, or you can suspend the main thread after initiating DisplayA, DisplayB and DisplayC.