We Keep Coding

Multiprocessing with undetermined number of processes

I'm far from being adapt in python and since 3 days I'm trying to figure out how to properly work with multiprocessing, but now I hit a dead and and need some assistance.
Basically what the program is supposed to do, is controlling different segments of an LED strip from multiple (seni-random) inputs at the same time. Therefore I came to the conclusion that I probably need to use multiprocessing.
I've written a module for it using an existing module from Adafruit. (I stripped it down for demonstration)
import time
import RPi.GPIO as GPIO
from multiprocessing import Lock
import Adafruit_WS2801
import Adafruit_GPIO.SPI as SPI
class Pixels(object):
def __init__(self, pixelCount, spiPort, spiDevice):
self.l = Lock()
self.pixels = Adafruit_WS2801.WS2801Pixels(pixelCount, spi=SPI.SpiDev(spiPort, spiDevice), gpio=GPIO)
# Clear all the pixels to turn them off.
self.pixels.clear()
self.pixels.show()
def set_color(self, target_pixel, color=(255,0,0)):
for k in target_pixel:
self.l.acquire()
self.pixels.set_pixel(k, Adafruit_WS2801.RGB_to_color( color[0], color[1], color[2] ))
self.l.release()
self.l.acquire()
self.pixels.show()
self.l.release()
def blink_color_blank(self, target_pixel, blink_times=1, wait=0.5, color=(255,0,0)):
for i in range(blink_times):
self.set_color(target_pixel, color)
time.sleep(wait)
self.set_color(target_pixel, (0,0,0))
time.sleep(wait)
Inside of self.pixels all the information about which LED should have which color is stored.
self.pixels.set_pixel() writes the new values to storage.
self.pixels.show() actually sends these values to the SPI-Bus.
Now my attempt at multiprocessing starts like this.
from multiprocessing import Process, Manager
from multiprocessing.managers import BaseManager
import LED_WS2801
if __name__ == '__main__':
BaseManager.register('LedClass', LED_WS2801.Pixels)
manager = BaseManager()
manager.start()
inst = manager.LedClass(10,0,0)
Now my problem arises when I start a process while another is still active.
p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p.start()
p = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p.start()
p.join()
This gives me following error:
Process Process-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in blink_color_blank
File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
kind, result = conn.recv()
EOFError
But when I do something like this, everything is fine.
p = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p.start()
b = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
b.start()
p.join()
b.join()
But I don't know my final number of processes as they get spawned by external inputs, so I need some way to control a variable number of processes. My idea was to use a list like this:
jobs = []
jobs.append(Process(target=inst.set_color, args=([0,1,2],(255,0,255),)))
jobs[0].start()
But much to my disappointment this returns with another error:
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in set_color
File "/usr/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
self._connect()
File "/usr/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
c = SocketClient(address)
File "/usr/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
s.connect(address)
File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
I hope I made my problem as understandable and clear as possible. As I haven't found anything like this I guess I'm doing something fundamentally wrong. So would you help me out, please?
Thank you.

you have to wait all child processes to finish its job, with re-asign p:
p = Process(...)
p.start()
p = Process(...)
p.start()
p.join()
you are just waiting for the later one in p to finish, the error comes when master wants to terminate but the first child process is still running. try this to wait for all child to finish:
p1 = Process(target=inst.blink_color_blank, args=([6,7,8], 10, 0.25, (255,0,0),))
p1.start()
p2 = Process(target=inst.set_color, args=([3,4,5,6],(0,255,0),))
p2.start()
childs = [p1, p2]
while any(p.is_alive() for p in childs):
for p in childs:
p.join(1)
besides, there is an multiprocessing.active_children() api to get all the children of the current process, in case you really cant gather the list from the beginning.

ThreadPoolExecutor hangs when call futur result in a futur callback

I am using "requests-futures" package and call asynchronous get/post in asynchronous get/post result callback (add_done_callback on futur result). Sometimes, my code hangs. After many investigation hours, I can reproduce the lock with a minimal code:
from concurrent.futures import ThreadPoolExecutor
import time
pool = ThreadPoolExecutor(max_workers=10)
def f(_):
time.sleep(0.1) # Try to force context switch
x = pool.submit(lambda: None)
print "1"
x.result()
print "2"
def main():
x = pool.submit(lambda : None)
x.add_done_callback(f)
print "3"
x.result()
print "4"
print "==="
main()
If I run this peace of code in a bash loop:
$> while true; do python code.py; done;
The program hangs every times with the "trace":
(...)
===
1
2
3
4
===
3
4
1
If I break it with ctrl^c, I have the following stack trace:
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/yienyien/Angus/test/futur/env/local/lib/python2.7/site-
packages/concurrent/futures/thread.py", line 46, in _python_exit
t.join(sys.maxint)
File "/usr/lib/python2.7/threading.py", line 951, in join
self.__block.wait(delay)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/yienyien/Angus/test/futur/env/local/lib/python2.7/site-
packages/concurrent/futures/thread.py", line 46, in _python_exit
t.join(sys.maxint)
File "/usr/lib/python2.7/threading.py", line 951, in join
self.__block.wait(delay)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
KeyboardInterrupt
Somebody could explain me what is happening ? I check the possible deadlocks in the concurrent.futures module, but I do not think it matches.
Thank you.

tasks submitted to a fixed-sized thread pool may not call blocking operations like Future.result(). This leads to a specific kind of deadlock, called "thread starvation". Using time.sleep() also switches a thread off the service and increases probability of thread starvation.

I answer to my own question.
After investigation, it's simple. I do not shutdown the TheadPoolExecutor and do not use with, then sometimes the main function completes and the finalize the main thread, the ThreadPoolExecutor state becomes "shutdown" whereas callback is not completed.

multiprocessing multithreading manager python

I am trying to get multiprocessing and multi-threading to work together nicely. I have the following code which I have derived from the multiprocessing documentation found at https://docs.python.org/2/library/multiprocessing.html
from multiprocessing import Process, Manager
from threading import Thread
def foo():
print ("hello world")
def simple_process(threads_manager):
"""Simple process that starts Threads
"""
threads = []
for i in range(10):
t = Thread(target=foo, args=())
t.start()
t.join()
threads.append(t)
threads_manager['threads'] = threads
manager = Manager()
threads_manager = manager.dict()
p = Process(target=simple_process, args=(threads_manager, ), kwargs={})
p.start()
p.join()
threads = threads_manager.get('threads')
print (threads)
but when I run the code I get the following error.
hello world
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "multiprocessing_multithreading.py", line 17, in simple_process
threads_manager['threads'] = threads
File "<string>", line 2, in __setitem__
File "/usr/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod
conn.send((self._id, methodname, args, kwds))
TypeError: can't pickle thread.lock objects
None
I'm trying to get the threads spawned by the simple_process function as a threads list.
Can someone please help?

Receive return value from different thread in another module

I am trying to replicate C# code in python which executes a thread, waits for it to finish and returns a value. Essentially the method RunAndWait is in a helper class because a call to that method is being made multiple times.
C# code is as follows:
public static bool RunAndWait(Action _action, long _timeout)
{
Task t = Task.Run(() =>
{
Log.Message(Severity.MESSAGE, "Executing " + _action.Method.Name);
_action();
});
if (!t.Wait(Convert.ToInt32(_timeout)))
{
Log.Message(Severity.ERROR, "Executing " + _action.Method.Name + " timedout. Could not execute MCS command.");
throw new AssertFailedException();
}
t.Dispose();
t = null;
return true;
}
In python I have been struggling with a few things. Firstly, there seem to be different types of Queue's where I simply picked the import that seemed to be working import Queue. Secondly, I receive a TypeError as below.
Traceback (most recent call last):
File "C:/Users/JSC/Documents/Git/EnterprisePlatform/Enterprise/AI.App.Tool.AutomatedMachineTest/Scripts/monkey.py",
line 9, in
File "C:\Users\JSC\Documents\Git\EnterprisePlatform\Enterprise\AI.App.Tool.AutomatedMachineTest\Scripts\Libs\MonkeyHelper.py",
line 4, in RunCmdAndWait
TypeError: module is not callable
Here is the python code for monkey:
from Libs.CreateConnection import CreateMcsConnection
import Libs.MonkeyHelper as mh
import Queue
q = Queue.Queue()
to = 5000 #timeout
mh.RunCmdAndWait(CreateMcsConnection, to, q)
serv, con = q.get()
and MonkeyHelper.py:
import threading
def RunCmdAndWait(CmdToRun, timeout, q):
t = threading(group=None, target=CmdToRun, arg=q)
t.start()
t.join(timeout=timeout)
I am not sure what I am doing wrong. I am fairly new to python. Could someone please help me out?
Edit
t = threading.Thread(group=None, target=CmdToRun, args=q)
correcting the line above brought up another error:
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Program Files (x86)\IronPython 2.7\Lib\threading.py", line 552, in _Thread__bootstrap_inner
self.run()
File "C:\Program Files (x86)\IronPython 2.7\Lib\threading.py", line 505, in run
self.target(*self.__args, **self.__kwargs)
AttributeError: Queue instance has no attribute '__len'
Is that because Thread expects multiple args or because the queue is still empty at this point? From what I've seen is that the queue is just being passed as an argument to receive the return value. Is that the right way to go?
Edit2
Changed t = threading.Thread(group=None, target=CmdToRun, args=q) to t = threading.Thread(group=None, target=CmdToRun, args=(q,))
The change yields in a TypeError below, seems weird to me since Thread is expecting a tuple.
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Program Files (x86)\IronPython 2.7\Lib\threading.py", line 552, in _Thread__bootstrap_inner
self.run()
File "C:\Program Files (x86)\IronPython 2.7\Lib\threading.py", line 505, in run
self.__target(*self.__args, **self.__kwargs)
TypeError: tuple is not callable

threading is a module. You likely mean to replace
t = threading(group=None, target=CmdToRun, arg=q)
with
t = threading.Thread(group=None, target=CmdToRun, args=(q,))
args is an argument tuple.

Convert a multi-threaded Python to a multi-process one using concurrent futures

I have the following working code (Python 3.5) which uses concurrent futures to parse files in a threaded manner, and then do some post-processing on the results when they come back (in any order).
from concurrent import futures
with futures.ThreadPoolExecutor(max_workers=4) as executor:
# A dictionary which will contain a list the future info in the key, and the filename in the value
jobs = {}
# Loop through the files, and run the parse function for each file, sending the file-name to it, along with the kwargs of parser_variables.
# The results of the functions can come back in any order.
for this_file in files_list:
job = executor.submit(parse_log_file.parse, this_file, **parser_variables)
jobs[job] = this_file
# Get the completed jobs whenever they are done
for job in futures.as_completed(jobs):
debug.checkpointer("Multi-threaded Parsing File finishing")
# Send the result of the file the job is based on (jobs[job]) and the job (job.result)
result_content = job.result()
this_file = jobs[job]
I want to convert this to use processes instead of threads because threads don't offer any speedup. In theory I just need to change ThreadPoolExecutor into ProcessPoolExecutor.
The problem is, if I do that I get this exception:
Process Process-2:
Traceback (most recent call last):
File "C:\Python35\lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "C:\Python35\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Python35\lib\concurrent\futures\process.py", line 169, in _process_worker
call_item = call_queue.get(block=True)
File "C:\Python35\lib\multiprocessing\queues.py", line 113, in get
return ForkingPickler.loads(res)
TypeError: Required argument 'fileno' (pos 1) not found
Traceback (most recent call last):
File "c:/myscript/main.py", line 89, in <module>
main()
File "c:/myscript/main.py", line 59, in main
system_counters = process_system(system, filename)
File "c:\myscript\per_system.py", line 208, in process_system
system_counters = process_filelist(**file_handling_variables)
File "c:\myscript\per_logfile.py", line 31, in process_filelist
results_list = job.result()
File "C:\Python35\lib\concurrent\futures\_base.py", line 398, in result
return self.__get_result()
File "C:\Python35\lib\concurrent\futures\_base.py", line 357, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
I think that this might have something to do with pickling, but googling for the error hasn't found anything.
How do I convert the above to use multiple processes?

It turns out this is because one of the things I'm passing inside parser_variables is a class (a reader from a third-party module). If I remove the class, the above works fine.
For whatever reason, pickle doesn't seem to be able to handle this particular object.