Recently in a project, I had a multiprocessing Process that crashed. A child process was supposed to do a calculation, then send it to the parent process in a Pipe. If the child crashed, the parent would freeze while reading from the pipe. Is there a 'correct' way to send data that would avoid blocking the parent forever if a child dies?
This is an example that reproduces the problem I'm having:
import multiprocessing as mp
def f(pipe):
a = 1/0
pipe.send('hola')
parent, child = mp.Pipe()
proc = mp.Process(target=f, args=(child,))
proc.start()
print "Grabbing result"
print "Result: {0}".format(parent.recv())
proc.join()
The parent process could use the connection's poll(...) method to determine if any result was forthcoming with a reasonable time limit:
import multiprocessing as mp
timelimit = 3
def f(pipe):
a = 1/0
pipe.send('hola')
parent, child = mp.Pipe()
proc = mp.Process(target=f, args=(child,))
proc.start()
print "Grabbing result"
if parent.poll(timelimit):
print "Result: {0}".format(parent.recv())
else:
print "No data available after {0} seconds...".format(timelimit)
proc.join()
When I run this code I get the following results:
Grabbing result
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "s.py", line 6, in f
a = 1/0
ZeroDivisionError: integer division or modulo by zero
No data available after 3 seconds...
Related
I'm far from being adapt in python and since 3 days I'm trying to figure out how to properly work with multiprocessing, but now I hit a dead and and need some assistance.
Basically what the program is supposed to do, is controlling different segments of an LED strip from multiple (seni-random) inputs at the same time. Therefore I came to the conclusion that I probably need to use multiprocessing.
I've written a module for it using an existing module from Adafruit. (I stripped it down for demonstration)
import time
import RPi.GPIO as GPIO
from multiprocessing import Lock
import Adafruit_WS2801
import Adafruit_GPIO.SPI as SPI
class Pixels(object):
def __init__(self, pixelCount, spiPort, spiDevice):
self.l = Lock()
self.pixels = Adafruit_WS2801.WS2801Pixels(pixelCount, spi=SPI.SpiDev(spiPort, spiDevice), gpio=GPIO)
# Clear all the pixels to turn them off.
self.pixels.clear()
self.pixels.show()
def set_color(self, target_pixel, color=(255,0,0)):
for k in target_pixel:
self.l.acquire()
self.pixels.set_pixel(k, Adafruit_WS2801.RGB_to_color( color[0], color[1], color[2] ))
self.l.release()
self.l.acquire()
self.pixels.show()
self.l.release()
def blink_color_blank(self, target_pixel, blink_times=1, wait=0.5, color=(255,0,0)):
for i in range(blink_times):
self.set_color(target_pixel, color)
time.sleep(wait)
self.set_color(target_pixel, (0,0,0))
time.sleep(wait)
Inside of self.pixels all the information about which LED should have which color is stored.
self.pixels.set_pixel() writes the new values to storage.
self.pixels.show() actually sends these values to the SPI-Bus.
Now my attempt at multiprocessing starts like this.
from multiprocessing import Process, Manager
from multiprocessing.managers import BaseManager
import LED_WS2801
if __name__ == '__main__':
BaseManager.register('LedClass', LED_WS2801.Pixels)
manager = BaseManager()
manager.start()
inst = manager.LedClass(10,0,0)
Now my problem arises when I start a process while another is still active.
p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p.start()
p = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p.start()
p.join()
This gives me following error:
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in blink_color_blank
File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
kind, result = conn.recv()
EOFError
But when I do something like this, everything is fine.
p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p.start()
b = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
b.start()
p.join()
b.join()
But I don't know my final number of processes as they get spawned by external inputs, so I need some way to control a variable number of processes. My idea was to use a list like this:
jobs = []
jobs.append(Process(target=inst.set_color, args=([0,1,2],(255,0,255),)))
jobs[0].start()
But much to my disappointment this returns with another error:
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in set_color
File "/usr/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
self._connect()
File "/usr/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
c = SocketClient(address)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
s.connect(address)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
I hope I made my problem as understandable and clear as possible. As I haven't found anything like this I guess I'm doing something fundamentally wrong. So would you help me out, please?
Thank you.
you have to wait all child processes to finish its job, with re-asign p:
p = Process(...)
p.start()
p = Process(...)
p.start()
p.join()
you are just waiting for the later one in p to finish, the error comes when master wants to terminate but the first child process is still running. try this to wait for all child to finish:
p1 = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p1.start()
p2 = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p2.start()
childs = [p1, p2]
while any(p.is_alive() for p in childs):
for p in childs:
p.join(1)
besides, there is an multiprocessing.active_children() api to get all the children of the current process, in case you really cant gather the list from the beginning.
This question already has answers here:
Using python multiprocessing Pool in the terminal and in code modules for Django or Flask
(3 answers)
Closed 5 years ago.
I want to put the generated result of the func in pool.apply_async() method into a queue, everything seems very well but the error confused me a lot.
My purpose is try to make multiple asynchronized producers(maybe not correct here) and multiple consumers.
Here is my toy example:
from multiprocessing import Pool
import multiprocessing
from threading import Thread
from six.moves import xrange
pool = Pool(processes=2, maxtasksperchild=1000)
# resp_queue = multiprocessing.Queue(1000)
manager = multiprocessing.Manager()
resp_queue = manager.Queue()
rang = 10000
def fetch_page(url):
resp_queue.put(url)
def parse_response():
url = resp_queue.get()
print(url)
r_threads = []
def start_processing():
for i in range(2):
r_threads.append(Thread(target=parse_response))
print("start %s thread.." % i)
r_threads[-1].start()
urls = map(lambda x: "this is url %s" % x, xrange(rang))
for i in xrange(rang):
pool.apply_async(fetch_page, (urls[i],))
start_processing()
pool.close()
pool.join()
The error reads that:
> Process PoolWorker-1: Process PoolWorker-2: Traceback (most recent
> call last): Traceback (most recent call last): File
> "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
> self.run()
> self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run File
> "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> self._target(*self._args, **self._kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
> File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
> task = get()
> task = get() File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
> File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
> return recv()
> return recv() AttributeError: 'module' object has no attribute 'fetch_page' AttributeError: 'module' object has no attribute
> 'fetch_page' start 0 thread.. start 1 thread..
I have read this answer but found it very strange, and this answer doesn't work on my Ubuntu machine.
Any suggestions are highly appreciated. Thanks very much.
Have a look at the code below. Changes I made to your version:
I'm using map instead of apply as it gets an iterable and splits work between workers nicely.
I've added a while loop to your parse_resp function (now get_url) so each thread will get values from queue to exhaustion.
Pool instantiation & calling is after __name__ == '__main__' which is a windows hack needed for Python multiprocessing (as much as I know, might be wrong I'm on Ubuntu).
from multiprocessing import Pool
import multiprocessing
from threading import Thread
manager = multiprocessing.Manager()
url_queue = manager.Queue()
rang = 10000
def put_url(url):
url_queue.put(url)
def get_url(thread_id):
while not url_queue.empty():
print('Thread {0} got url {1}'.format(str(thread_id), url_queue.get()))
r_threads = []
def start_threading():
for i in range(2):
r_threads.append(Thread(target=get_url, args=(i,)))
print("start %s thread.." % i)
r_threads[-1].start()
for i in r_threads:
i.join()
urls = ["url %s" % x for x in range(rang)]
if __name__ == '__main__':
pool = Pool(processes=2, maxtasksperchild=1000)
pool.map_async(put_url, urls)
start_threading()
pool.close()
pool.join()
Prints:
start 0 thread..
start 1 thread..
Thread 0 got url 0
Thread 0 got url 1
Thread 1 got url 2
Thread 0 got url 3
Thread 0 got url 4
Thread 1 got url 5
Thread 0 got url 6
I am trying to get multiprocessing and multi-threading to work together nicely. I have the following code which I have derived from the multiprocessing documentation found at https://docs.python.org/2/library/multiprocessing.html
from multiprocessing import Process, Manager
from threading import Thread
def foo():
print ("hello world")
def simple_process(threads_manager):
"""Simple process that starts Threads
"""
threads = []
for i in range(10):
t = Thread(target=foo, args=())
t.start()
t.join()
threads.append(t)
threads_manager['threads'] = threads
manager = Manager()
threads_manager = manager.dict()
p = Process(target=simple_process, args=(threads_manager, ), kwargs={})
p.start()
p.join()
threads = threads_manager.get('threads')
print (threads)
but when I run the code I get the following error.
hello world
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "multiprocessing_multithreading.py", line 17, in simple_process
threads_manager['threads'] = threads
File "<string>", line 2, in __setitem__
File "/usr/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod
conn.send((self._id, methodname, args, kwds))
TypeError: can't pickle thread.lock objects
None
I'm trying to get the threads spawned by the simple_process function as a threads list.
Can someone please help?
I am currently running a simulation program in Python that requires me to run a large number of processes. Each of these processes has a loop that executes a certain block of code.
The problem I am facing is that I need to processes to run for an unknown amount of time (until my experiment is complete). There is no way to know the value of this time beforehand.
I am looking for a way to terminate the processes or simply have them stop executing when the experiment is complete.
Currently, I have created a flag variable using the Manager in Python's multiprocessing module. Each of the child processes contain a loop that only executes while this flag is set to true. This solves the problem but generates many errors, one for each process, when the toggle flag is set to false.
Traceback (most recent call last):
File "C:\Python33\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Python33\lib\multiprocessing\process.py", line 95, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Michael\PycharmProjects\untitled1\pmonitor.py", line 33, in backgroundTaskLauncher
while flag[0]:
File "<string>", line 2, in __getitem__
File "C:\Python33\lib\multiprocessing\managers.py", line 726, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "C:\Python33\lib\multiprocessing\connection.py", line 207, in send
self._send_bytes(buf.getbuffer())
File "C:\Python33\lib\multiprocessing\connection.py", line 281, in _send_bytes
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
BrokenPipeError: [WinError 232] The pipe is being closed
Process Process-20:
I am wondering if there is a proper way to do what I am trying to go for here. I'm not even sure if I'm using the correct terminology.
The code from the main process is as follows:
if __name__ == '__main__':
manager = Manager()
flag = manager.list([True])
for taskSize in taskSizes:
flag[0] = True
for i in range(1,TASK_LAUNCHERS):
Process(target=backgroundTaskLauncher, args=(taskSize,flag)).start()
#Experiment goes here
flag[0] = False
The code from the process launched is:
def backgroundTaskLauncher(taskSize,flag):
while flag[0]:
for i in range(0,CHUNK_AMOUNT):
Thread(target=task, args=(taskSize,)).start()
sleep(MICROCHUNK_GAP)
sleep(INTERCHUNK_GAP*random.random()*2)
Essentially the main method calls a number of backgroundTaskLauncher processes who, in turn, launch many threads while the toggle flag is enabled and stop and complete when the flag is disabled.
I am looking for the proper way to go about getting this behavior.
I think you are missing to join your child processes before the program ends. So the child processes are left without their daddy ;-)
Try this:
if __name__ == '__main__':
manager = Manager()
flag = manager.list([True])
for taskSize in taskSizes:
flag[0] = True
processes = [] # a list to store the process handles
for i in range(1,TASK_LAUNCHERS):
p = Process(target=backgroundTaskLauncher, args=(taskSize,flag))
p.start()
processes.append(p) # save process handle
# Experiment goes here (I think it goes here (unindented(?)))
flag[0] = False
# after you are done with your experiment, join all child processes
for p in processes:
p.join()
Im trying to execute a program in a python subprocess:
class MiThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
try:
from Queue import Queue, Empty
except ImportError:
#from queue import Queue, Empty # python 3.x
print "error"
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p= Popen(["java -Xmx256m -jar bin/HelloWorld.jar"],cwd=r'/home/karen/sphinx4-1.0beta5-src/sphinx4-1.0beta5/',stdout=PIPE, shell=True, bufsize= 4024)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
print "estoy en el hilo"
t.daemon = True # thread dies with the program
t.start()
print l
But when i execute the thread it fails with the following error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/site-packages/GNS3/Workspace.py", line 65, in run
t = Thread(target=enqueue_output, args=(p.stdout, q))
NameError: global name 'Thread' is not defined
QObject::connect: Cannot queue arguments of type 'QTextCursor'
(Make sure 'QTextCursor' is registered using qRegisterMetaType().)
i dont have any idea! What is happening?
Try changing:
t = Thread(target=enqueue_output, args=(p.stdout, q))
to:
t = threading.Thread(target=enqueue_output, args=(p.stdout, q))
In your current namespace, Thread exists as threading.Thread (a member of the threading module), so when you say Thread alone, Python can't find a match and throws that error.