I'm trying to implement very simple multiprocessing code in python 2.7, but it looks like the code run serially and not parallel.
The following code prints *****1***** while I expect it to print *****2***** immediately after *****1*****.
import os
import multiprocessing
from time import sleep
def main():
func1_proc = multiprocessing.Process(target=func1())
func2_proc = multiprocessing.Process(target=func2())
func1_proc.start()
func2_proc.start()
pass
def func1():
print "*****1*****"
sleep(100)
def func2():
print "*****2*****"
sleep(100)
if __name__ == "__main__":
main()
You're calling func1 and func2 before passing their returning values to Process, so func1 is going to sleep 100 seconds before returning None, for which Process will raise an error.
You should pass function objects to Process instead so that it will run them in separate processes:
func1_proc = multiprocessing.Process(target=func1)
func2_proc = multiprocessing.Process(target=func2)
Related
I have two python function which I want to run in parallel. I don't want sub_task function to wait for main_task function.
from threading import Thread
from multiprocessing import Process
from time import sleep,time
def main_task():
while True:
sleep(2)
print('main task running')
def sub_task():
while True:
sleep(5)
print('sub task running')
When I used thread this way, I can see output
q=Thread(target = main_task).start()
s=Thread(target = sub_task).start()
But when I used Process this way, I cannot see ouput
q=Process(target = main_task).start()
s=Process(target = sub_task).start()
So what is wrong with the implementation.
I have two python script, script1.py and script2.py. One is a counter which increment int x independently, and script2.py is to fetch the value of int x every 5 seconds, inputted into script2.py. I have tried doing this with multiprocessing verbatim from the following post,
Passing data between separately running Python scripts
and i applied While True function for script1. Here is my attempt, but i dont think i understand the general thought and i am getting various errors, since i am new to python and i miss some details.
script1.py:
from multiprocessing import Process, Pipe
x = 0
def function(child_conn):
global x
while True:
x += 1
print(x)
child_conn.send(x)
child_conn.close()
script2.py:
from multiprocessing import Proces,Queue,Pipe
from script1 import function
from time import sleep
if __name__=='__main__':
parent_conn,child_conn = Pipe()
p = Process(target=function, args=(child_conn,))
p.start()
print(parent_conn.recv())
time.sleep(5)
thanks in Advance!
You have a loop in the child process but no loop in the parent process. With no loop, the child can only send a single message then throws an error.
Try this code. Run script2.py to start the process.
script1.py
from multiprocessing import Process, Pipe
from time import sleep
x = 0
def function(child_conn):
global x
while True:
x += 1
print(x)
child_conn.send(x)
#child_conn.close()
sleep(1)
script2.py
from multiprocessing import Process,Queue,Pipe
from script1 import function
from time import sleep
if __name__=='__main__':
parent_conn,child_conn = Pipe()
p = Process(target=function, args=(child_conn,))
p.start()
while True:
print(parent_conn.recv())
sleep(1)
I am trying to use Threading in Python, and struggle to kick off two functions at the same time, then wait for both to finish and load returned data into variables in the main code. How can this be achieved?
import threading
from threading import Thread
func1():
#<do something>
return(x,y,z)
func2():
#<do something>
return(a,b,c)
Thread(target=func1).start()
Thread(target=func2).start()
#<hold until both threads are done, load returned values>
More clarity is definitely required from the question asked. Perhaps you're after something like the below?
import threading
from threading import Thread
def func1():
print("inside func1")
return 5
def func2():
print("inside func2")
return 6
if __name__ == "__main__":
t1 = Thread(target=func1)
t2 = Thread(target=func2)
threads = [t1, t2]
for t in threads:
t.start()
I believe you were missing the start() method to actually launch your threads?
I have two scripts, new.py and test.py.
Test.py
import time
while True:
x = "hello"
time.sleep(1)
x = "world"
time.sleep(1)
new.py
import time
while True:
import test
x = test.x
print(x)
time.sleep(1)
Now from my understanding this should print "hello" and a second later "world" all the time when executing new.py.
It does not print anything, how can i fix that?
Thanks
I think the code below captures what you are asking. Here I simulate two scripts running independently (by using threads), then show how you can use shelve to communicate between them. Note, there are likely much better ways to get to what you are after -- but if you absolutely must run the scripts independently, this will work for you.
Incidentally, any persistent source would do (such as a database).
import shelve
import time
import threading
def script1():
while True:
with shelve.open('my_store') as holder3:
if holder3['flag'] is not None: break
print('waiting')
time.sleep(1)
print("Done")
def script2():
print("writing")
with shelve.open('my_store') as holder2:
holder2['flag'] = 1
if __name__ == "__main__":
with shelve.open('my_store') as holder1:
holder1['flag'] = None
t = threading.Thread(target=script1)
t.start()
time.sleep(5)
script2()
t.join()
Yields:
waiting
waiting
waiting
waiting
waiting
writing
Done
Test.py
import time
def hello():
callList = ['hello', 'world']
for item in callList:
print item
time.sleep(1)
hello()
new.py
from parent import hello
while True:
hello()
I am writing a module such that in one function I want to use the Pool function from the multiprocessing library in Python 3.6. I have done some research on the problem and the it seems that you cannot use if __name__=="__main__" as the code is not being run from main. I have also noticed that the python pool processes get initialized in my task manager but essentially are stuck.
So for example:
class myClass()
...
lots of different functions here
...
def multiprocessFunc()
do stuff in here
def funcThatCallsMultiprocessFunc()
array=[array of filenames to be called]
if __name__=="__main__":
p = Pool(processes=20)
p.map_async(multiprocessFunc,array)
I tried to remove the if __name__=="__main__" part but still no dice. any help would appreciated.
It seems to me that your have just missed out a self. from your code. I should think this will work:
class myClass():
...
# lots of different functions here
...
def multiprocessFunc(self, file):
# do stuff in here
def funcThatCallsMultiprocessFunc(self):
array = [array of filenames to be called]
p = Pool(processes=20)
p.map_async(self.multiprocessFunc, array) #added self. here
Now having done some experiments, I see that map_async could take quite some time to start up (I think because multiprocessing creates processes) and any test code might call funcThatCallsMultiprocessFunc and then quit before the Pool has got started.
In my tests I had to wait for over 10 seconds after funcThatCallsMultiprocessFunc before calls to multiprocessFunc started. But once started, they seemed to run just fine.
This is the actual code I've used:
MyClass.py
from multiprocessing import Pool
import time
import string
class myClass():
def __init__(self):
self.result = None
def multiprocessFunc(self, f):
time.sleep(1)
print(f)
return f
def funcThatCallsMultiprocessFunc(self):
array = [c for c in string.ascii_lowercase]
print(array)
p = Pool(processes=20)
p.map_async(self.multiprocessFunc, array, callback=self.done)
p.close()
def done(self, arg):
self.result = 'Done'
print('done', arg)
Run.py
from MyClass import myClass
import time
def main():
c = myClass()
c.funcThatCallsMultiprocessFunc()
for i in range(30):
print(i, c.result)
time.sleep(1)
if __name__=="__main__":
main()
The if __name__=='__main__' construct is an import protection. You want to use it, to stop multiprocessing from running your setup on import.
In your case, you can leave out this protection in the class setup. Be sure to protect the execution points of the class in the calling file like this:
def apply_async_with_callback():
pool = mp.Pool(processes=30)
for i in range(z):
pool.apply_async(parallel_function, args = (i,x,y, ), callback = callback_function)
pool.close()
pool.join()
print "Multiprocessing done!"
if __name__ == '__main__':
apply_async_with_callback()