Related
I'm having trouble trying run a few loops in parallel when employing Pari via cypari2. I'll including a couple of small working examples along with the Tracebacks in case anyone has some insight on this.
Example 1 -- using joblib:
from cypari2 import Pari
from joblib import Parallel, delayed
def AddOne(v):
return v + pari.one()
pari = Pari()
vec = [pari('x_1'), pari('x_2')]
print(vec)
#works
newVec = Parallel(n_jobs=1)(delayed(AddOne)(i) for i in vec)
print(newVec)
#doesn't work
newVec2 = Parallel(n_jobs=2)(delayed(AddOne)(i) for i in vec)
print(newVec2)
The output:
[x_1, x_2]
[x_1 + 1, x_2 + 1]
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/joblib/externals/loky/backend/queues.py", line 150, in _feed
obj_ = dumps(obj, reducers=reducers)
File "/usr/lib/python3/dist-packages/joblib/externals/loky/backend/reduction.py", line 247, in dumps
dump(obj, buf, reducers=reducers, protocol=protocol)
File "/usr/lib/python3/dist-packages/joblib/externals/loky/backend/reduction.py", line 240, in dump
_LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
File "/usr/lib/python3/dist-packages/joblib/externals/cloudpickle/cloudpickle_fast.py", line 538, in dump
return Pickler.dump(self, obj)
File "stringsource", line 2, in cypari2.pari_instance.Pari.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "min_jake_joblib.py", line 16, in <module>
newVec2 = Parallel(n_jobs=2)(delayed(AddOne)(i) for i in vec)
File "/usr/lib/python3/dist-packages/joblib/parallel.py", line 1016, in __call__
self.retrieve()
File "/usr/lib/python3/dist-packages/joblib/parallel.py", line 908, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/usr/lib/python3/dist-packages/joblib/_parallel_backends.py", line 554, in wrap_future_result
return future.result(timeout=timeout)
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.
Seems to be a problem with pickling the Pari objects, but is there any way around it?
Example 2 -- using multiprocessing:
from cypari2 import Pari
import multiprocessing
def AddOne(v):
return v + pari.one()
pari = Pari()
vec = [pari('x_1'), pari('x_2')]
print(vec)
#doesn't work
if __name__ == '__main__':
pool = multiprocessing.Pool(processes = 2) ## doesn't matter how many I use
newVec = pool.map(AddOne, (i for i in vec))
print(newVec)
It seg faults, but doesn't completely exit automatically, so I have to use Ctrl^C to kill it. The output:
[x_1, x_2]
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 576, in _handle_results
task = get()
File "/usr/lib/python3.8/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "cypari2/gen.pyx", line 4705, in cypari2.gen.objtogen
File "cypari2/gen.pyx", line 4812, in cypari2.gen.objtogen
File "cypari2/convert.pyx", line 557, in cypari2.convert.PyObject_AsGEN
cysignals.signals.SignalError: Segmentation fault
^CProcess ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
File "min_jake_multiprocessing.py", line 14, in <module>
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/queues.py", line 356, in get
res = self._reader.recv_bytes()
File "/usr/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
File "src/cysignals/signals.pyx", line 320, in cysignals.signals.python_check_interrupt
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File "/usr/lib/python3.8/multiprocessing/queues.py", line 355, in get
with self._rlock:
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
File "src/cysignals/signals.pyx", line 320, in cysignals.signals.python_check_interrupt
KeyboardInterrupt
KeyboardInterrupt
newVec = pool.map(AddOne, (i for i in vec))
File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.8/multiprocessing/pool.py", line 765, in get
self.wait(timeout)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 762, in wait
self._event.wait(timeout)
File "/usr/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
File "src/cysignals/signals.pyx", line 320, in cysignals.signals.python_check_interrupt
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
finalizer()
File "/usr/lib/python3.8/multiprocessing/util.py", line 224, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 695, in _terminate_pool
raise AssertionError(
AssertionError: Cannot have cache with result_hander not alive
I suppose someone will tell me to use sympy or some other symbolic algebra package instead, but the symbolic algebra I need to do is quite complex and Pari can handle it extremely well. However, in the end I'd like to be able to process a queue of class objects that contain Pari objects in parallel. Any thoughts/suggestions are appreciated.
Well, this isn't a full answer, but it works for me so I wanted to share in case anyone else runs into this issue.
The first issue appears to be that the versions of libpari-dev and pari-gp on the apt repository were too old. The apt repository contains version 2.11 whereas the version on Pari's git repository is version 2.14. Uninstalling and following the instructions from here to install from source fixed most of my problems.
Interestingly, I still needed to install libpari-gmp-tls6 from the apt repository to get things to work. But, after that I was able to get the test examples above to run. The example using multiprocessing ran successfully without modification, but the example using joblib required the use of the "threading" backend in order to run.
So I tried making an simple example using the python multiprocessing module.
import time
import multiprocessing
start = time.perf_counter()
def do_something():
print('Sleeping 2 sec ...')
time.sleep(2)
print('Done Sleeping')
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')
But when I run it, a very long error traceback is printed:
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Goal\VSB\PDS\Project> & C:/Users/Goal/AppData/Local/Programs/Python/Python39/python.exe
c:/Goal/VSB/PDS/Project/MultiprocessingPython.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
self._popen = self._Popen(self)
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
ne 134, in _check_not_importing_main
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Finished in 0.2 second(s)
PS C:\Goal\VSB\PDS\Project> & C:/Users/Goal/AppData/Local/Programs/Python/Python39/python.exe
c:/Goal/VSB/PDS/Project/MultiprocessingPython.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
Traceback (most recent call last):
File "<string>", line 1, in <module>
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
exitcode = _main(fd, parent_sentinel)
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable. p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Finished in 0.16 second(s)
PS C:\Goal\VSB\PDS\Project>
And there are so many things here I don't know if they can be called error? They are just so many that I dont even know what to search on the internet to solve my problem.
When Python throws these massive error statements, make sure to know what you are looking for- the important part to notice is that the error is stemming from a raise call, meaning that you need to look after the raise keywords in the error text.
As you can see, the actual text error thrown (The text found after the line with the raise keyword) is the following:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
After that, you can easily search this up and find a very helpful and detailed answer here (TLDR: You need to use if __name__ == '__main__': so that the code does not recursively import).
I tried to run a multiprocessing function on spyder3. After it wouldnt print anything inside a loop and be stuck infinitly i read that I could run it on an external terminal in spyder like this.
Run > Configuration per file > Execute in an external system terminal
now it finally shows me something. Sadly its this:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 161, in <module>
cpus,closeIdx=valueSeries.index,t1=df['t1'])
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 27, in mpPandasObj
out=processJobs(jobs,numThreads=numThreads)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 75, in processJobs
pool=mp.Pool(processes=numThreads)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 119, in Pool
context=self.get_context())
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 176, in __init__
self._repopulate_pool()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
w.start()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
and the this occours in a (probably infinite) loop:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
this is the multiprocessing function:
def processJobs(jobs,task=None,numThreads=24):
# Run in parallel.
# jobs must contain a ’func’ callback, for expandCall
if task is None:
task=jobs[0]['func'].__name__
pool=mp.Pool(processes=numThreads)
outputs,out,time0=pool.imap_unordered(expandCall,jobs),[],time.time()
# Process asynchronous output, report progress
for i,out_ in enumerate(outputs,1): # <---- Here.
out.append(out_)
reportProgress(i,len(jobs),time0,task)
pool.close(); pool.join() # this is needed to prevent memory leaks return out
return out
How can I solve this? (I really need multiprocessing)
EDIT: tried it on Pycharm -> it reports the same errors
I just ran into this. You are probably directly calling processJobs() directly in the Python script.
You just need to add the main idiom like so:
if __name__ == '__main__':
processJobs()
I suspect multiprocessing will fork/spawn multiple Python processes in the OS and load the module, and not having this guard causes the Pool instantiation to be invoked recursively.
Having the main guard ensures that the Pool instantiation runs only once.
I have written a custom environment so I can play around with reinforcement learning (PPO) and tf-agents.
This works fine if I wrap my env ( which inherits from py_environment.PyEnvironment) in a TfPyEnvironment, but fails if I try to wrap it into a ParallelPyEnvironment. I have tried playing around with all the keyword arguments of ParallelPyEnvironment but the code just runs up to the line and then nothing happens - no Exception, the program does not terminate etc.
Here is my code initialising the environment and showing off the working variant for the eval_env:
train_env = tf_py_environment.TFPyEnvironment(
ParallelPyEnvironment(
[CardGameEnv()] * hparams['parallel_environments']
)
)
# this works perfectly:
eval_env = tf_py_environment.TFPyEnvironment(CardGameEnv(debug=True))
If I terminate the script via CTRL+C, this is what is being output:
Traceback (most recent call last):
Traceback (most recent call last):
File "E:\Users\tmp\Documents\Programming\Neural Nets\Poker_AI\poker_logic\train.py", line 229, in <module>
File "<string>", line 1, in <module>
train(model_num=3)
File "C:\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
File "E:\Users\tmp\Documents\Programming\Neural Nets\Poker_AI\poker_logic\train.py", line 64, in train
[CardGameEnv()] * hparams['parallel_environments']
exitcode = _main(fd)
File "E:\Users\tmp\AppData\Roaming\Python\Python37\site-packages\gin\config.py", line 1009, in wrapper
File "C:\Python37\lib\multiprocessing\spawn.py", line 113, in _main
preparation_data = reduction.pickle.load(from_parent)
KeyboardInterrupt
return fn(*new_args, **new_kwargs)
File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 70, in __init__
self.start()
File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 83, in start
env.start(wait_to_start=self._start_serially)
File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 223, in start
self._process.start()
File "C:\Python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Python37\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Python37\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Python37\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 264, in __getattr__
return self._receive()
File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 333, in _receive
message, payload = self._conn.recv()
File "C:\Python37\lib\multiprocessing\connection.py", line 250, in recv
buf = self._recv_bytes()
File "C:\Python37\lib\multiprocessing\connection.py", line 306, in _recv_bytes
[ov.event], False, INFINITE)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\tf_agents\environments\parallel_py_environment.py", line 289, in close
self._process.join(5)
File "C:\Python37\lib\multiprocessing\process.py", line 139, in join
assert self._popen is not None, 'can only join a started process'
AssertionError: can only join a started process
From that I conclude that the thread ParallelPyEnvironment is trying to start does not do that, but since I'm not very experienced with threading in Python, I have no idea where to go from here, especially how to fix this.
Current training takes a long time and does not use my PC's capabilities at all (3GB of 32GB RAM used, processor at 3%, GPU barely working at all but VRAM full), so this should speed up training time significantly.
The solution is to pass in callables, not environments, so the ParallelPyEnvironment can construct them itself:
train_env = tf_py_environment.TFPyEnvironment(
ParallelPyEnvironment(
[CardGameEnv] * hparams['parallel_environments']
)
)
This is the multithreaded code I'm trying to import as a module. It works fine when I run it as a stand alone file. It simply just prints out a list of numbers. All I change is which main() is commented out. I think that the problem might be that the imported code is calling itself as a module after the reader file has already called it as a module.
threadtest.py
from multiprocessing import Process, Value, Array
import time
import sys
def f(n, a):
n.value = 3.1415927
for i in range(len(a)):
a[i] = -a[i]
def is_prime(x, top):
for j in range(2,int(top**.5 + 1)):
if x%j == 0:
return False
return True
def f1(top, c):
print('test f1')
for i in range(2,top):
if is_prime(i,top):
c.value = c.value + 1
def f3(c1,c2):
for k in range(0,20):
time.sleep(1) # 1 second
sys.stdout.write(str(c1.value) + '|' + str(c2.value) + '\n')
sys.stdout.flush()
def main():
count1 = Value('d', 0)
count2 = Value('d', 0)
#arr = Array('i', range(10))
p1 = Process(target=f1, args=(1000000, count1))
p2 = Process(target=f1, args=(1000000, count2))
p3 = Process(target=f3, args=(count1, count2))
p1.start()
p2.start()
p3.start()
p1.join()
print('p1.join()')
p2.join()
print('p2.join()')
p3.join()
print('p3.join()')
print(count1.value)
print(count2.value)
if __name__ == '__main__':
print('grovetest is being run as main')
#main()
else:
print('grovetest is being run as module')
main()
This is the code that imports the multithreaded module and attempts to read the output.
readertest.py
import threadtest
def main(fname):
try:
p = subprocess.Popen(fname, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
print('success')
return p.communicate() # this gets you pipe values
except Exception as e:
return 'error'+str(e)
else:
return "else"
if __name__ == '__main__':
main(threadtest)
Here is the error that is produced when I run the readertest.py
grovetest is being run as module
grovetest is being run as module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
grovetest is being run as module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
grovetest is being run as module
p1.join()
p2.join()
p3.join()
0.0
0.0
I'm new to this. Thank you for your feedback.
You're violating a significant Windows-specific constraint called out here:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
One flaw in the paragraph I just quoted is that it says "the main module" when it should say "every module", but most people don't try to run complex code from imported modules and thus don't run across this issue. In your case, what happens is that readertest imports grovetest, which invokes the multiprocessing code, which fires up a new Python interpreter on grovetest using the special Windows multiprocess bootstrap system, which fires up the multiprocessing code, which detects that it's being fired up under the special bootstrap case and aborts:
(start)
readertest:
import grovetest
=> grovetest
import multiprocessing
=> multiprocessing: define some stuff
=> grovetest
if __name__ == '__main__': # it's not equal
... # so we skip this
else:
print('grovetest is being run as module') # prints
main() # calls main
...
p1 = Process(target=f1, args=(1000000, count1)) # calls multiprocessing module
=> new process
import grovetest # multiprocessing does this using its magic bootstrap
...
if __name__ == '__main__': # it's not
...
else:
print(...)
...
p1 = Process(...)
At this point the inner Process code detects the constraint violation and raises the first error. This terminates the inner process entirely, and we're back to the outer process that does p2 = Process(...), which kicks off the same error again, and so on.
To fix this, you must not call your main in grovetest when grovetest is imported as a module. Instead, call grovetest.main later, from readertest. That will allow the special bootstrap magic (in the multiprocessing module, run up when you invoke multiprocessing.Process) to import grovetest in the new process. Once this new process has finished importing grovetest, it will wait for instructions from its parent process (the other Python interpreter, where you called multiprocessing.Process, from your readertest code calling grovetest.main). Those instructions will be to invoke function f1 (from target=f1) with the supplied arguments.