I have a worker process that goes like this:
class worker(Process):
def __init__(self):
# init stuff
def run(self):
# do stuff
logging.info("done") # to confirm that the process is done running
And I start 3 processes like this:
processes = 3
aproc = [None for _ in processes]
bproc = [None for _ in processes]
for i in range(processes):
aproc[i] = worker(foo, bar)
bproc[i] = worker2(foo, bar) # different worker class
aproc[i].start()
bproc[i].start()
However, at the end of my code, I .join each of the processes, but they just hang and the script never ends.
for i in range(processes):
aproc[i].join()
bproc[i].join()
Hitting CTRL+C gives me this traceback:
Traceback (most recent call last):
File "[REDACTED]", line 571, in <module>
sproc[0].join()
File "/usr/lib/python3.9/multiprocessing/process.py", line 149, in join
res = self._popen.wait(timeout)
File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 43, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 27, in poll
pid, sts = os.waitpid(self.pid, flag)
I've heard of the typical deadlock, but this shouldn't be the case since all the processes print the logging statement that they are done running. Why is .join() still waiting on them? Any ideas? Thank you!
Edit: Unfortunately I can't get a minimal example working to share. Also, they do communicate with each other through multiprocessing.Queue()s, if that is relevant.
Edit 2:
Traceback of another test:
Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.9/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.9/multiprocessing/queues.py", line 201, in _finalize_join
thread.join()
File "/usr/lib/python3.9/threading.py", line 1033, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.9/threading.py", line 1049, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
I'm far from being adapt in python and since 3 days I'm trying to figure out how to properly work with multiprocessing, but now I hit a dead and and need some assistance.
Basically what the program is supposed to do, is controlling different segments of an LED strip from multiple (seni-random) inputs at the same time. Therefore I came to the conclusion that I probably need to use multiprocessing.
I've written a module for it using an existing module from Adafruit. (I stripped it down for demonstration)
import time
import RPi.GPIO as GPIO
from multiprocessing import Lock
import Adafruit_WS2801
import Adafruit_GPIO.SPI as SPI
class Pixels(object):
def __init__(self, pixelCount, spiPort, spiDevice):
self.l = Lock()
self.pixels = Adafruit_WS2801.WS2801Pixels(pixelCount, spi=SPI.SpiDev(spiPort, spiDevice), gpio=GPIO)
# Clear all the pixels to turn them off.
self.pixels.clear()
self.pixels.show()
def set_color(self, target_pixel, color=(255,0,0)):
for k in target_pixel:
self.l.acquire()
self.pixels.set_pixel(k, Adafruit_WS2801.RGB_to_color( color[0], color[1], color[2] ))
self.l.release()
self.l.acquire()
self.pixels.show()
self.l.release()
def blink_color_blank(self, target_pixel, blink_times=1, wait=0.5, color=(255,0,0)):
for i in range(blink_times):
self.set_color(target_pixel, color)
time.sleep(wait)
self.set_color(target_pixel, (0,0,0))
time.sleep(wait)
Inside of self.pixels all the information about which LED should have which color is stored.
self.pixels.set_pixel() writes the new values to storage.
self.pixels.show() actually sends these values to the SPI-Bus.
Now my attempt at multiprocessing starts like this.
from multiprocessing import Process, Manager
from multiprocessing.managers import BaseManager
import LED_WS2801
if __name__ == '__main__':
BaseManager.register('LedClass', LED_WS2801.Pixels)
manager = BaseManager()
manager.start()
inst = manager.LedClass(10,0,0)
Now my problem arises when I start a process while another is still active.
p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p.start()
p = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p.start()
p.join()
This gives me following error:
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in blink_color_blank
File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
kind, result = conn.recv()
EOFError
But when I do something like this, everything is fine.
p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p.start()
b = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
b.start()
p.join()
b.join()
But I don't know my final number of processes as they get spawned by external inputs, so I need some way to control a variable number of processes. My idea was to use a list like this:
jobs = []
jobs.append(Process(target=inst.set_color, args=([0,1,2],(255,0,255),)))
jobs[0].start()
But much to my disappointment this returns with another error:
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in set_color
File "/usr/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
self._connect()
File "/usr/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
c = SocketClient(address)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
s.connect(address)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
I hope I made my problem as understandable and clear as possible. As I haven't found anything like this I guess I'm doing something fundamentally wrong. So would you help me out, please?
Thank you.
you have to wait all child processes to finish its job, with re-asign p:
p = Process(...)
p.start()
p = Process(...)
p.start()
p.join()
you are just waiting for the later one in p to finish, the error comes when master wants to terminate but the first child process is still running. try this to wait for all child to finish:
p1 = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p1.start()
p2 = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p2.start()
childs = [p1, p2]
while any(p.is_alive() for p in childs):
for p in childs:
p.join(1)
besides, there is an multiprocessing.active_children() api to get all the children of the current process, in case you really cant gather the list from the beginning.
I'm reading and applying code from the python book and I can't use multiprocessing in simple example that you can see below:
import multiprocessing
def myProcess():
print("Currently Executing Child Process")
print("This process has it's own instance of the GIL")
print("Executing Main Process")
print("Creating Child Process")
myProcess = multiprocessing.Process(target=myProcess)
myProcess.start()
myProcess.join()
print("Child Process has terminated, terminating main process")
My platform is Windows 10 64 bit and using if __name_ == "__main_" : doesn't work in this case. What's wrong here? This code should work in python version 3.5 and above. Python version I use is 3.7. Full error message below:
C:\Users\Xian\AppData\Local\Programs\Python\Python37-32\python.exe "C:/OneDrive/Utilizing sub-process.py"
Traceback (most recent call last):
File "C:/OneDrive/Utilizing sub-process.py", line 25, in <module>
myProcess.start()
File "C:\Users\Xian\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Users\Xian\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Xian\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\Xian\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\Xian\AppData\Local\Programs\Python\Python37-32\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function myProcess at 0x02B2D420>: it's not the same object as __main__.myProcess
try this
def test()
import multiprocessing
multiprocessing.set_start_method("fork")
p = multiprocessing.Process(target=xxx)
p.start()
python multiprocessing Contexts and start methods
This question already has answers here:
Using python multiprocessing Pool in the terminal and in code modules for Django or Flask
(3 answers)
Closed 5 years ago.
I want to put the generated result of the func in pool.apply_async() method into a queue, everything seems very well but the error confused me a lot.
My purpose is try to make multiple asynchronized producers(maybe not correct here) and multiple consumers.
Here is my toy example:
from multiprocessing import Pool
import multiprocessing
from threading import Thread
from six.moves import xrange
pool = Pool(processes=2, maxtasksperchild=1000)
# resp_queue = multiprocessing.Queue(1000)
manager = multiprocessing.Manager()
resp_queue = manager.Queue()
rang = 10000
def fetch_page(url):
resp_queue.put(url)
def parse_response():
url = resp_queue.get()
print(url)
r_threads = []
def start_processing():
for i in range(2):
r_threads.append(Thread(target=parse_response))
print("start %s thread.." % i)
r_threads[-1].start()
urls = map(lambda x: "this is url %s" % x, xrange(rang))
for i in xrange(rang):
pool.apply_async(fetch_page, (urls[i],))
start_processing()
pool.close()
pool.join()
The error reads that:
> Process PoolWorker-1: Process PoolWorker-2: Traceback (most recent
> call last): Traceback (most recent call last): File
> "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
> self.run()
> self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run File
> "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
> File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
> task = get()
> task = get() File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
> File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
> return recv()
> return recv() AttributeError: 'module' object has no attribute 'fetch_page' AttributeError: 'module' object has no attribute
> 'fetch_page' start 0 thread.. start 1 thread..
I have read this answer but found it very strange, and this answer doesn't work on my Ubuntu machine.
Any suggestions are highly appreciated. Thanks very much.
Have a look at the code below. Changes I made to your version:
I'm using map instead of apply as it gets an iterable and splits work between workers nicely.
I've added a while loop to your parse_resp function (now get_url) so each thread will get values from queue to exhaustion.
Pool instantiation & calling is after __name__ == '__main__' which is a windows hack needed for Python multiprocessing (as much as I know, might be wrong I'm on Ubuntu).
from multiprocessing import Pool
import multiprocessing
from threading import Thread
manager = multiprocessing.Manager()
url_queue = manager.Queue()
rang = 10000
def put_url(url):
url_queue.put(url)
def get_url(thread_id):
while not url_queue.empty():
print('Thread {0} got url {1}'.format(str(thread_id), url_queue.get()))
r_threads = []
def start_threading():
for i in range(2):
r_threads.append(Thread(target=get_url, args=(i,)))
print("start %s thread.." % i)
r_threads[-1].start()
for i in r_threads:
i.join()
urls = ["url %s" % x for x in range(rang)]
if __name__ == '__main__':
pool = Pool(processes=2, maxtasksperchild=1000)
pool.map_async(put_url, urls)
start_threading()
pool.close()
pool.join()
Prints:
start 0 thread..
start 1 thread..
Thread 0 got url 0
Thread 0 got url 1
Thread 1 got url 2
Thread 0 got url 3
Thread 0 got url 4
Thread 1 got url 5
Thread 0 got url 6
Recently in a project, I had a multiprocessing Process that crashed. A child process was supposed to do a calculation, then send it to the parent process in a Pipe. If the child crashed, the parent would freeze while reading from the pipe. Is there a 'correct' way to send data that would avoid blocking the parent forever if a child dies?
This is an example that reproduces the problem I'm having:
import multiprocessing as mp
def f(pipe):
a = 1/0
pipe.send('hola')
parent, child = mp.Pipe()
proc = mp.Process(target=f, args=(child,))
proc.start()
print "Grabbing result"
print "Result: {0}".format(parent.recv())
proc.join()
The parent process could use the connection's poll(...) method to determine if any result was forthcoming with a reasonable time limit:
import multiprocessing as mp
timelimit = 3
def f(pipe):
a = 1/0
pipe.send('hola')
parent, child = mp.Pipe()
proc = mp.Process(target=f, args=(child,))
proc.start()
print "Grabbing result"
if parent.poll(timelimit):
print "Result: {0}".format(parent.recv())
else:
print "No data available after {0} seconds...".format(timelimit)
proc.join()
When I run this code I get the following results:
Grabbing result
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "s.py", line 6, in f
a = 1/0
ZeroDivisionError: integer division or modulo by zero
No data available after 3 seconds...