We Keep Coding

"Dictionary size changed during iteration" from Pebble ProcessPool

We've some parallel processing code which is built around Pebble, it's been working robustly for quite some time but we seem to have run into some odd edge-case.
Based on the exception trace (and the rock-simple code feeding it) I suspect that it's actually a bug in Pebble but who knows.
The code feeding the process pool is pretty trivial:
pool = ProcessPool(max_workers=10, max_tasks=10)
for path in filepaths:
try:
future = pool.schedule(function=self.analyse_file, args(path), timeout=30)
future.add_done_callback(self.process_result)
exception Exception as e:
print("Exception fired:" + e) # NOT where the exception is firing
pool.close()
pool.join()
So in essence, we schedule a bunch of stuff to run, close out the pool then wait for the pool to complete the scheduled tasks. NOTE: the exception is not being thrown in the schedule loop, it gets fired AFTER we call join().
This is the exception stack trace:
Traceback (most recent call last):
File "/home/user/.pyenv/versions/3.6.0/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/user/.pyenv/versions/3.6.0/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/user/.pyenv/versions/scrapeapp/lib/python3.6/site-packages/pebble/pool/process.py", line 150, in task_scheduler_loop
pool_manager.schedule(task)
File "/home/user/.pyenv/versions/scrapeapp/lib/python3.6/site-packages/pebble/pool/process.py", line 198, in schedule
self.worker_manager.dispatch(task)
File "/home/user/.pyenv/versions/scrapeapp/lib/python3.6/site-packages/pebble/pool/process.py", line 327, in dispatch
self.pool_channel.send(WorkerTask(task.id, task.payload))
File "/home/user/.pyenv/versions/scrapeapp/lib/python3.6/site-packages/pebble/pool/channel.py", line 66, in send
return self.writer.send(obj)
File "/home/user/.pyenv/versions/3.6.0/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/home/user/.pyenv/versions/3.6.0/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
RuntimeError: dictionary changed size during iteration
I think it's got to be some weird race condition, as the code will work flawlessly on some data sets but fail at what appears to be a random point on another dataset.
We were using pebble 4.3.1 when we first ran into the issue (same version we'd had since the beginning), tried upgrading to 4.5.0, no change.
Has anybody run into similar issues with Pebble in the past? If so what was your fix?

Python Server Error With Stdin Read

I'm making a python server to keep connected 3 clients. I have finished all and now, when I'm doing the stdin commands for the server (like quit, see (client), etc) I realised I have an apparently big problem for me:
This is the structure:
import sys, os, traceback, optparse
import time, datetime
import random
import socket
import thread
import select
while True:
data,addr = dad_sock.recvfrom(UDP)
if(data != ""):
print "First_Data:",sys.stdin.readline()
thread.start_new_thread(MAIN_son,(UDP,data,addr,number_sons))
number_threads += 1
print "Second_Data:",sys.stdin.readline()
What MAIN_son does is to create a new thread wich look for pkg recived and keep the clients connected to the server.
And here is my problem:
Data: ERROR, UNEXPECTED EXCEPTION
I/O operation on closed file
Traceback (most recent call last):
File "srv.py", line 588, in
start()
File "srv.py", line 136, in start
main()
File "srv.py", line 481, in main
setup()
File "srv.py", line 164, in setup
dad_wait(srv.UDP,number_sons)
File "srv.py", line 172, in dad_wait
print "Data:",sys.stdin.readline()
ValueError: I/O operation on closed file
Line 172 is my first read from stdin() but the program lets me to do it so I think its at the second round of the while when it fails
I need to say that when MAIN_son thread is called starts to print all the pkg info from the client pakages. I don't know if this prints can be the cause.
If someone needs more information about the code I'll refresh the question, Thanks

Error in Python multiprocessing process

I am trying a write a python code having multiple processes whose structure and flow is something like this:
import multiprocessing
import ctypes
import time
import errno
m=multiprocessing.Manager()
mylist=m.list()
var1=m.Value('i',0)
var2=m.Value('i',1)
var3=m.Value('i',2)
var4=m.Value(ctypes.c_char_p,"a")
var5=m.Value(ctypes.c_char_p,"b")
var6=3
var7=4
var8=5
var9=6
var10=7
def func(var1,var2,var4,var5,mylist):
i=0
try:
if var1.value==0:
print var2.value,var4.value,var5.value
mylist.append(time.time())
elif var1.value==1:
i=i+2
print var2.value+2,var4.value,var5.value
mylist.append(time.time())
except IOError as e:
if e.errno==errno.EPIPE:
var3.value=var3.value+1
print "Error"
def work():
for i in range(var3.value):
print i,var6,var7,va8,var9,var10
p=multiprocessing.Process(target=func,args=(var1,var2,var4,var5,mylist))
p.start()
work()
When I run this code, sometimes it works perfectly, sometimes it does not run for exact amount of loop counts and sometimes I get following error:
0
1
Process Process-2:
Traceback (most recent call last):
File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "dummy.py", line 19, in func
if var1.value==0:
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 1005, in get
return self._callmethod('get')
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
self._connect()
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/lib64/python2.6/multiprocessing/connection.py", line 149, in Client
answer_challenge(c, authkey)
File "/usr/lib64/python2.6/multiprocessing/connection.py", line 383, in answer_challenge
message = connection.recv_bytes(256) # reject large message
EOFError
What does this error mean? What wrong am I doing here? What this error indicates? Kindly guide me to the correct path. I am using CentOS 6.5

Working with shared variables in multiprocessing is tricky. Because of the python Global Interpreter Lock (GIL), multiprocessing is not directly possible in Python. When you use the multiprocessing module, you can launch several task on different process, BUT you can't share the memory.
In you case, you need this so you try to use shared memory. But what happens here is that you have several processes trying to read the same memory at the same time. To avoid memory corruption, a process lock the memory address it is currently reading, forbidding other processes to access it until it finishes reading.
Here you have 3 processes trying to evaluate var1.value in the first if loop of your func : the first process read the value, and the other are blocked, raising an error.
To avoid this mechanism, you should always manage the Lock of your shared variables yourself.
You can try with syntax:
var1=multiprocessing.Value('i',0) # create shared variable
var1.acquire() # get the lock : it will wait until lock is available
var1.value # read the value
var1.release() # release the lock
External documentation :
Locks : https://docs.python.org/2/librar/multiprocessing.html#synchronization-between-processes
GIL : https://docs.python.org/2/glossary.html#term-global-interpreter-lock

subprocess.popen pid change

I have a script in which I use subprocess.Popen to start an external program and process.kill() to kill it pretty much as soon as it's started. I've been getting Windows Error [5] (Access Denied) every time the script tries to kill it. I've realized that the pid of the program is actually changing after it's opened. Is there a way, in Python, to monitor the process for the change, or to just retrieve the new pid?
Here is the code:
import subprocess
import time
proc = subprocess.Popen(Path/to/WinSCP.exe)
time.sleep(2)
proc.kill()
The error:
Traceback (most recent call last):
File "C:\Python27\lib\lib-tk\Tkinter.py", line 1410, in __call__
return self.func(*args)
File "C:\Path", line 231, in __call__
self.scpsetup()
File "C:\Path", line 329, in scpsetup
proc.kill()
File "C:\Python27\lib\subprocess.py", line 1019, in terminate
_subprocess.TerminateProcess(self._handle, 1)
WindowsError: [Error 5] Access is denied

This is what I ended up doing;
import tempfile
import subprocess
import time
# Create a temp file to receive output
tf = tempfile.NamedTemporaryFile(delete=False)
output = open(tf.name, "w")
# Open and close WinSCP
subprocess.Popen(Path/To/WinSCP.exe)
time.sleep(2)
subprocess.call("TASKKILL /IM WinSCP.exe", stdout=output)
tf.close()
The issue I had with methods like this before was that I couldn't hide the output of the command. This may not be the prettiest way to accomplish this but it works.
Also note that I am using Windows 8. I understand that the command itself may vary slightly in different versions of Windows.

multiprocessing Pool hangs when there is a exception in any of the thread

I am new to Python and trying a multiprocessing.pool program to process files, it works fine as long as there are no exceptions. If any of the thread/process gets an exception the whole program waits for the thread
snippet of the code:
cp = ConfigParser.ConfigParser()
cp.read(gdbini)
for table in cp.sections():
jobs.append(table)
#print jobs
poolreturn = pool.map(worker, jobs)
pool.close()
pool.join()
Failure Message:
Traceback (most recent call last):
File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 525, in __bootstrap_inner
self.run()
File "/opt/cnet-python/default-2.6/lib/python2.6/threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "/opt/cnet-python/default-2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
task = get()
TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'ConfigParser.NoOptionError'>, ("No option 'inputfilename' in section: 'section-1'",))
I went ahead added a exception handler to terminate the process
try:
ifile=cp.get(table,'inputfilename')
except ConfigParser.NoSectionError,ConfigParser.NoOptionError:
usage("One of Parameter not found for"+ table)
terminate()
but still it waits, not sure whats missing.

In Python 3.2+ this works as expected. For Python 2, this bug was fixed in r74545 and will be available in Python 2.7.3. In the mean time, you can use the configparser library which is a backport of the configparser from 3.2+. Check it out.

I had the same issue. It happens when a worker process raises a user exception which has a custom constructor. Make sure your exception (ConfigParser.NoOptionError in that case) initializes the base exception with exactly two arguments:
class NoOptionError(ValueError):
def __init__(self, message, *args):
super(NoOptionError, self).__init__(message, args)