I trying to play around with multi-threading so I can better at it, but for some weird reason, my code doesn't want to follow the commands. It's suppose to go into a while loop and print, but it doesn't, and it's also not raising any errors, so which lines is the mistake on?
#!/usr/bin/env python
#
#
#
import random
import thread
import time
import sys
import os
def DisplayA(name,wait):
while True:
print 'Display: 1';time.sleep(wait)
def DisplayB(name,wait):
while True:
print 'Display: 2';time.sleep(wait)
def DisplayC(name,wait):
while True:
print 'Display: 3';time.sleep(wait)
thread.start_new_thread(DisplayA,('Display1',2))
thread.start_new_thread(DisplayB,('Display2',3))
thread.start_new_thread(DisplayC,('Display3',5))
Add this to the bottom:
while True:
pass
The problem is that you're running off the bottom of your main program. This terminates the entire execution session.
Quick and short solution:
while True:
time.sleep(1)
Do not use pass in the while loop, because it eats CPU.
Expensive way of doing nothing.
If you want a more general solution, then you can import Tread from threading, then you can use join:
from threading import Thread
...
p1 = Thread(name="A", target=DisplayA, args=('Display1',2))
p2 = Thread(name="B", target=DisplayB, args=('Display2',3))
p3 = Thread(name="C", target=DisplayC, args=('Display3',5))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
This solution works also if the threads do not run endless, and your program can continue after the threads have finished.
You can either do what Prune here suggested, or you can suspend the main thread after initiating DisplayA, DisplayB and DisplayC.
Related
Desired goal
pass information to other sub processes or main process
Concerns
-Is it a problem that the processes never finish?
-Is it a problem the class methods never return/finish?
will this cause stack overflow? ( I expect to run for hours).
Alternative
Would it be appropriate to have several different scripts and save to, load from a database/pickle?
What works
If I swap the multiprocessing for threading the desired output is achieved.
Have tried
-Tried switching the multiprocessing for threading which seems to give me the desired outcome, but would like to understand why multiprocessing is not giving the same output.
-Tried using Queue() instead or LifoQueue()
-Tried running and printing Queue.get() from another child subprocess instead of the main script
-Tried .join() all tasks
Desired output
I expect the color to be changed within a subprocess and passed to the Queue, I expect to retrieve the new color with Queue.get().
Example
import multiprocessing
import threading
import time
class MyClass:
def __init__(self):
self.color = 'gray'
def task_blue(self,q):
print('sleeping 5..')
time.sleep(5)
while True:
time.sleep(1)
self.color = 'blue'
q.put([self.color])
def task_red(self,q):
print('sleeping 5..')
time.sleep(5)
while True:
time.sleep(3)
self.color = 'red'
q.put([self.color])
def printer(self,q):
while True:
time.sleep(.1)
if q.empty():
print('<empty>')
else:
print(q.get())
from queue import LifoQueue, Queue
q = LifoQueue()
my_class = MyClass()
p1 = multiprocessing.Process(target=my_class.task_blue,args=(q,),name='BLUE')
p2 = multiprocessing.Process(target=my_class.task_red,args=(q,),name='RED')
# p3 = multiprocessing.Process(target=my_class.printer,args=(q,),name='PRINTER')
tasks = []
tasks.append(p1)
tasks.append(p2)
# tasks.append(p3)
for task in tasks:
task.start()
while True:
time.sleep(.2)
if q.empty():
print(['empty'])
else:
print(q.get())
output
Is it a problem that the processes never finish?
It's OK for a process to run forever.
Is it a problem the class methods never return/finish?
No. If you want your program to run forever, there is going to be some section of code that doesn't return. It's OK if this area of code is inside a method.
I like the idea of the entire program being one script/class
You can do it many ways. There's no correct answer here, especially absent other requirements.
Tried using Queue() instead or LifoQueue() -Tried running and printing Queue.get() from another child subprocess instead of the main
script -Tried .join() all the tasks
when using the multiprocessing package, you should use the Queue class from the multiprocessing package. E.g. queue = multiprocessing.Queue(). Docs here: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue
What I'd like to do is the following program to print out:
Running Main
Running Second
Running Main
Running Second
[...]
Code:
from multiprocessing import Process
import time
def main():
while True:
print('Running Main')
time.sleep(1)
def second():
while True:
print('Running Second')
time.sleep(1)
p1 = Process(main())
p2 = Process(second())
p1.start()
p2.start()
But it doesn't have the desired behavior. Instead it just prints out:
Running Main
Running Main
[...]
I suspect my program doesn't work because of the while statement?
Is there any way I can overcome this problem and have my program print out what I mentioned no matter what I execute in my function?
The issue here seems to be when you make the process vars. I suspect the reason for why the process inclusively runs the first function is because of syntax. My interpretation is that instead of creating a process out of a function you are making a process that executes a function exclusively.
When you want to create Process object you want to avoid using this
p1 = Process(target=main())
and rather write
p1 = Process(target=main)
That also means if you want to include any input for the function you will have to
p1 = Process(target=main, args=('hi',))
I'm currently coding a chatbot for my streaming. Since it needs to do multiple things at once I'm using the multiprocessing module, that way it can still respond to commands and do functions at the same time. My problem now is that I have one process dedicated to some web scraping and another one to look at chat and respond if a command is being typed. My thoughts were, if I append the information from one process to a global list, and then when the command is being typed in chat, the other process can use the information in the appended list. Well, this didn't work and I learned that this is because the 2 processes don't have shared memory, although both having access to the same list, they are both copies of the list, so even if one appends, in the other process' case, it will still be empty. I've come across a few questions regarding this here on stack overflow, but the examples are very specific and since I'm fairly new to coding still, I had a hard time figuring out how to apply it to my own code. For this exact reason, I've simplified the problem so it can help others who are in a similar situation, by having my example broad enough and simple enough for anyone to understand it once they read the solution. Thus this is not the code I'm actually using for my chatbot, but one that mimics the problem.
import multiprocessing as mp
import time
globalList = []
def readList():
while True:
time.sleep(2)
if globalList:
print(globalList)
else:
print("List is Empty")
print(globalList)
def writeList():
while True:
time.sleep(3)
globalList.append("Item")
print(globalList)
if __name__ == '__main__':
p1 = mp.Process(target=readList)
p2 = mp.Process(target=writeList)
p1.start()
p2.start()
When running this code you can see that the writeList function will keep adding another item to the list, but the readList function will keep showing an empty list.
I hope some master wiz out there can help me with this problem.
In Python processes cannot straightforwardly access global mutable objects created by other processes. For this, you can use, for example, a multiprocessing.Manager and its proxy objects. Your adapted example:
import multiprocessing as mp
import time
def readList(shared_list):
while True:
time.sleep(2)
if shared_list:
print(shared_list)
else:
print("List is Empty")
print(shared_list)
def writeList(shared_list):
while True:
time.sleep(3)
shared_list.append("Item")
print(shared_list)
if __name__ == '__main__':
manager = mp.Manager()
shared_list = manager.list()
p1 = mp.Process(target=readList, args=(shared_list,))
p2 = mp.Process(target=writeList, args=(shared_list,))
p1.start()
p2.start()
p1.join()
p2.join()
You can not have that by normal means. Processes have their own memory space. Threads, on the other hand, have same memory space and are ran within one process.
For more, please, reffer to this answer Multiprocessing vs Threading Python
On a machine running Windows Server 2012 R2, in the Spyder IDE from Anaconda and running Python 3.7 with the following code:
import time
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
And I get the output
name is main
Program completed in 0.13s
Goodbye!
My expectation was that I would see the two print statements
func start
func end
and also (because .join was envoked) that the program would take >1s to complete.
I suspect that the .start() call did not successfully call the do_something function.
FYI, I am following this tutorial, which I know needs to be modified to include the if statement on windows. I am also seeing similar code on other sites, but doesn't seem to work on my end.
Any suggestions on troubleshooting would be much appreciated.
**EDIT: Per the comment below from Azy_Crw4282, the code seems to work on his end and, per his suggestion, it seems to work from the cmd prompt. So this seems to be a bug specifically with the Spyder IDE.
FYI, I wanted to understand whether the issue was that the process was getting kicked off but the IDE wasn't capturing the output OR the process wasn't getting kicked off. I tried two things, 1) the code below writes a dataframe to csv. When doing this in the multiprocessing function, it does NOT write the file. 2) I created a global variable and changed variable value in the function. Spyder keeps the variable values after the code runs, and when I printed the variable it was unchanged.
So, in summary - it seems that the Spyder IDE does not work with the multiprocessing module.**
import time
import multiprocessing
start=time.perf_counter()
df=pd.DataFrame(data={'Col1':[1.1,2.1,3.1],
'Col2':[1.2,2.2,3.2],
'Col3':[1.3,2.3,3.3]}, columns=['Col1','Col2','Col3'])
def do_something():
print('func start')
df.to_csv('C:/testMp.csv')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
When I ran your code, I get the following output. Can you try running your code in another ide/ cmd line/ terminal?
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
Outputs:
name is main
Program completed in 0.0s
Goodbye!
func start
func end
Program completed in 1.27s
Goodbye!
The above result is not probably what you expect. In order to achieve, what you want, you need to indent the outer print section so that it appears within the main call.
import time
import multiprocessing
start=time.perf_counter()
def do_something():
print('func start')
time.sleep(1)
print('func end')
if __name__=='__main__':
print('name is main')
p1=multiprocessing.Process(target=do_something)
p1.start()
p1.join()
finish=time.perf_counter()
print('\n\nProgram completed in '+str(round((finish-start),2))+'s')
print('Goodbye!')
Outputs:
name is main
func start
func end
Program completed in 1.33s
Goodbye!
So, I've been playing with Python's multiprocessing module for a few days now, and there's something that I can't understand. Maybe someone can give me a bit of help.
I'm trying to run two methods from the same class in parallel, but apparently there's something that I'm missing:
from multiprocessing import Process
import time
class SomeClass:
def __init__(self):
pass
def meth1(self):
print(1)
time.sleep(10)
def meth2(self):
print(2)
time.sleep(5 * 60)
def main():
while True:
s = SomeClass()
p1 = Process(target=s.meth1) # I want this to run from 10 to 10 seconds
p1.start()
p2 = Process(target=s.meth2) # I want this to run from 5 to 5 minutes
# while the first one still does its own
# job from 10s to 10s
p2.start()
p1.join()
p2.join()
if __name__ == '__main__':
main()
What I would expect to happen is:
the first method should print 1;
then the second one should print 2 (without waiting 10s - which does happen and seems to work as expected);
then I should see only 1s for the next 4 minutes and 50s (this isn't happening and the program just waits for that time to pass.
What am I missing? Why does the second step work as expected, but the 3rd one doesn't? How can I make it work?
Its difficult to really know what you want, but this code below does what you describe, but has no convenient way of exiting:
from multiprocessing import Process
import time
class SomeClass:
def __init__(self):
pass
def meth1(self):
while True:
print(1)
time.sleep(10)
def meth2(self):
while True:
print(2)
time.sleep(5 * 60)
def main():
s = SomeClass()
p1 = Process(target=s.meth1) # I want this to run from 10 to 10 seconds
p1.start()
p2 = Process(target=s.meth2) # I want this to run from 5 to 5 minutes while the first one still does its own job from 10s to 10s
p2.start()
p1.join()
p2.join()
if __name__ == '__main__':
main()
I have moved the while into each of the methods of SomeClass
This code will never exit, hanging at p1.join()
Think of processes as friends / clones of yours that you can call on the phone, have come in and do something, and then when they are all done they go home, leaving you a note.
The lines:
p1 = Process(target=s.meth1)
p1.start()
call one clone up and have him run s.meth1. He comes over and prints 1 on your screen, waits 10 seconds, then leaves you a note: "all done, no exceptions occurred, I've gone home".
Meanwhile (while your first clone is coming over), the lines:
p2 = Process(target=s.meth2)
p2.start()
call up another clone and have him run s.meth2. He comes over, prints 2 on your screen, waits around for 5 minutes, then leaves you a note: "all done, no exceptions occurred, I've gone home".
While clones #1 and #2 are doing their work, the line:
p1.join()
waits for clone #1 to leave you his note. That happens after 10 seconds. You then go on to:
p2.join()
which waits for clone #2 to leave you his note. That happens after another 4 minutes and 50 seconds. Then you go back around to your while True and start everything over again.
If you want not to wait for clone #2 to finish and leave you his note, don't call p2.join() yet. Eventually, though, you should call p2.join() to make sure that everything went well and that he went home successfully, and isn't lying dead in your living room in a pool of blood, shot by some exception. :-)