So I tried making an simple example using the python multiprocessing module.
import time
import multiprocessing
start = time.perf_counter()
def do_something():
print('Sleeping 2 sec ...')
time.sleep(2)
print('Done Sleeping')
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')
But when I run it, a very long error traceback is printed:
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Goal\VSB\PDS\Project> & C:/Users/Goal/AppData/Local/Programs/Python/Python39/python.exe
c:/Goal/VSB/PDS/Project/MultiprocessingPython.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
self._popen = self._Popen(self)
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
ne 134, in _check_not_importing_main
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Finished in 0.2 second(s)
PS C:\Goal\VSB\PDS\Project> & C:/Users/Goal/AppData/Local/Programs/Python/Python39/python.exe
c:/Goal/VSB/PDS/Project/MultiprocessingPython.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
Traceback (most recent call last):
File "<string>", line 1, in <module>
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
exitcode = _main(fd, parent_sentinel)
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable. p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Finished in 0.16 second(s)
PS C:\Goal\VSB\PDS\Project>
And there are so many things here I don't know if they can be called error? They are just so many that I dont even know what to search on the internet to solve my problem.
When Python throws these massive error statements, make sure to know what you are looking for- the important part to notice is that the error is stemming from a raise call, meaning that you need to look after the raise keywords in the error text.
As you can see, the actual text error thrown (The text found after the line with the raise keyword) is the following:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
After that, you can easily search this up and find a very helpful and detailed answer here (TLDR: You need to use if __name__ == '__main__': so that the code does not recursively import).
Related
When I try to run spider test suite like described here:
https://github.com/taoyds/test-suite-sql-eval
I get this error:
Traceback (most recent call last): File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\user\PycharmProjects\test-suite-sql-eval\evaluate_classical.py", line 20, in <module> m = Manager() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 57, in Manager m.start() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\managers.py", line 562, in start self._process.start() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen return Popen(process_obj) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__ prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data _check_not_importing_main() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. python-BaseException
Can someome please help?
Thanks!
I tried to run this example command:
python3 evaluate_classical.py --pred=evaluation_examples/classical_test_gold.txt --out_file=all_eval_results.json
I expected to get some kind of evaluation, but got the above error.
It seems for some websites I need to use undetected chromedriver for Selenium automation. I already got Selenium and chromedriver installed in a venv environment, and also ran the below to install undetected chromedriver in the venv: python -m pip install undetected-chromedriver. I used the below code to start undetected chromedriver:
from selenium import webdriver
import undetected_chromedriver as uc
browser = uc.Chrome()
browser.get('any url here')
When I run the script in the venv I got error message. Could anyone help what was wrong?
PS C:\Users\user\my_venv> & c:/Users/user/my_venv/Scripts/Activate.ps1
(my_venv) PS C:\Users\user\my_venv> & c:/Users/user/my_venv/Scripts/python.exe c:/Users/user/my_venv/toughsite_test.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 269, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\Users\user\my_venv\toughsite_test.py", line 15, in <module>
browser = uc.Chrome()
File "C:\Users\user\my_venv\lib\site-packages\undetected_chromedriver\__init__.py", line 356, in __init__
self.browser_pid = start_detached(
File "C:\Users\user\my_venv\lib\site-packages\undetected_chromedriver\dprocess.py", line 35, in start_detached
).start()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
The simple fix is that you just have to write:
uc.Chrome(use_subprocess=True)
And thank me later ;)
This is because you do not have your main method wrapped in
if __name__ == "__main__":
foo()
I am using executor.map and storing results in 'results'.
Following code works without any problem on Jupiter notebook. However, it crashes when executing the python script in Windows command prompt.
rop_test.py
print ('start of process')
import concurrent.futures
from funcfile import func
if __name__ == '__main__':
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(func, varlist1,varlist2)
global master_list
master_list=list(results)
print(master_list)
#do something with master_list
funcfile.py
def func(var1,var2):
var3=var1+var2
return var3
Error
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Administrator\notebooks\rop_test.py", line 640, in <module>
print (master_list)
NameError: name 'master_list' is not defined
Finished in 0.0 second(s)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Administrator\notebooks\rop_test.py", line 640, in <module>
print (master_list)
NameError: name 'master_list' is not defined
Traceback (most recent call last):
File "rop_test.py", line 634, in <module>
master_list=list(results)
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists
for element in iterable:
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 598, in result_iterator
yield fs.pop().result()
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
I have tried putting this in a main command, adding wait, adding [var for var in results], list(results) etc. but to no avail.
N.B. I am on python 3.6
I tried to run a multiprocessing function on spyder3. After it wouldnt print anything inside a loop and be stuck infinitly i read that I could run it on an external terminal in spyder like this.
Run > Configuration per file > Execute in an external system terminal
now it finally shows me something. Sadly its this:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 161, in <module>
cpus,closeIdx=valueSeries.index,t1=df['t1'])
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 27, in mpPandasObj
out=processJobs(jobs,numThreads=numThreads)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 75, in processJobs
pool=mp.Pool(processes=numThreads)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 119, in Pool
context=self.get_context())
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 176, in __init__
self._repopulate_pool()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
w.start()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
and the this occours in a (probably infinite) loop:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
this is the multiprocessing function:
def processJobs(jobs,task=None,numThreads=24):
# Run in parallel.
# jobs must contain a ’func’ callback, for expandCall
if task is None:
task=jobs[0]['func'].__name__
pool=mp.Pool(processes=numThreads)
outputs,out,time0=pool.imap_unordered(expandCall,jobs),[],time.time()
# Process asynchronous output, report progress
for i,out_ in enumerate(outputs,1): # <---- Here.
out.append(out_)
reportProgress(i,len(jobs),time0,task)
pool.close(); pool.join() # this is needed to prevent memory leaks return out
return out
How can I solve this? (I really need multiprocessing)
EDIT: tried it on Pycharm -> it reports the same errors
I just ran into this. You are probably directly calling processJobs() directly in the Python script.
You just need to add the main idiom like so:
if __name__ == '__main__':
processJobs()
I suspect multiprocessing will fork/spawn multiple Python processes in the OS and load the module, and not having this guard causes the Pool instantiation to be invoked recursively.
Having the main guard ensures that the Pool instantiation runs only once.
This is the multithreaded code I'm trying to import as a module. It works fine when I run it as a stand alone file. It simply just prints out a list of numbers. All I change is which main() is commented out. I think that the problem might be that the imported code is calling itself as a module after the reader file has already called it as a module.
threadtest.py
from multiprocessing import Process, Value, Array
import time
import sys
def f(n, a):
n.value = 3.1415927
for i in range(len(a)):
a[i] = -a[i]
def is_prime(x, top):
for j in range(2,int(top**.5 + 1)):
if x%j == 0:
return False
return True
def f1(top, c):
print('test f1')
for i in range(2,top):
if is_prime(i,top):
c.value = c.value + 1
def f3(c1,c2):
for k in range(0,20):
time.sleep(1) # 1 second
sys.stdout.write(str(c1.value) + '|' + str(c2.value) + '\n')
sys.stdout.flush()
def main():
count1 = Value('d', 0)
count2 = Value('d', 0)
#arr = Array('i', range(10))
p1 = Process(target=f1, args=(1000000, count1))
p2 = Process(target=f1, args=(1000000, count2))
p3 = Process(target=f3, args=(count1, count2))
p1.start()
p2.start()
p3.start()
p1.join()
print('p1.join()')
p2.join()
print('p2.join()')
p3.join()
print('p3.join()')
print(count1.value)
print(count2.value)
if __name__ == '__main__':
print('grovetest is being run as main')
#main()
else:
print('grovetest is being run as module')
main()
This is the code that imports the multithreaded module and attempts to read the output.
readertest.py
import threadtest
def main(fname):
try:
p = subprocess.Popen(fname, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
print('success')
return p.communicate() # this gets you pipe values
except Exception as e:
return 'error'+str(e)
else:
return "else"
if __name__ == '__main__':
main(threadtest)
Here is the error that is produced when I run the readertest.py
grovetest is being run as module
grovetest is being run as module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
grovetest is being run as module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
grovetest is being run as module
p1.join()
p2.join()
p3.join()
0.0
0.0
I'm new to this. Thank you for your feedback.
You're violating a significant Windows-specific constraint called out here:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
One flaw in the paragraph I just quoted is that it says "the main module" when it should say "every module", but most people don't try to run complex code from imported modules and thus don't run across this issue. In your case, what happens is that readertest imports grovetest, which invokes the multiprocessing code, which fires up a new Python interpreter on grovetest using the special Windows multiprocess bootstrap system, which fires up the multiprocessing code, which detects that it's being fired up under the special bootstrap case and aborts:
(start)
readertest:
import grovetest
=> grovetest
import multiprocessing
=> multiprocessing: define some stuff
=> grovetest
if __name__ == '__main__': # it's not equal
... # so we skip this
else:
print('grovetest is being run as module') # prints
main() # calls main
...
p1 = Process(target=f1, args=(1000000, count1)) # calls multiprocessing module
=> new process
import grovetest # multiprocessing does this using its magic bootstrap
...
if __name__ == '__main__': # it's not
...
else:
print(...)
...
p1 = Process(...)
At this point the inner Process code detects the constraint violation and raises the first error. This terminates the inner process entirely, and we're back to the outer process that does p2 = Process(...), which kicks off the same error again, and so on.
To fix this, you must not call your main in grovetest when grovetest is imported as a module. Instead, call grovetest.main later, from readertest. That will allow the special bootstrap magic (in the multiprocessing module, run up when you invoke multiprocessing.Process) to import grovetest in the new process. Once this new process has finished importing grovetest, it will wait for instructions from its parent process (the other Python interpreter, where you called multiprocessing.Process, from your readertest code calling grovetest.main). Those instructions will be to invoke function f1 (from target=f1) with the supplied arguments.