Related
I am developing a code where I need to create a Manager to synchronize a Queue. In the project I am using hydra and everything works fine until I create a multiprocessing Manager Object. This is the raised error with HYDRA_FULL_ERROR=1:
Primary config module 'config' not found.
Check that it's correct and contains an __init__.py file
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
File "mre.py", line 8, in <module>
main()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\main.py", line 32, in decorated_main
_run_hydra(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 355, in _run_hydra
run_and_report(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 207, in run_and_report
return func()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 356, in <lambda>
lambda: hydra.run(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\hydra.py", line 107, in run
return run_job(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\core\utils.py", line 125, in run_job
ret.return_value = task_function(task_cfg)
File "mre.py", line 7, in main
m = Manager()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 57, in Manager
m.start()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\managers.py", line 583, in start
self._address = reader.recv()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\connection.py", line 250, in recv
buf = self._recv_bytes()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\connection.py", line 305, in _recv_bytes
waitres = _winapi.WaitForMultipleObjects(
KeyboardInterrupt
^C
C:\Users\UX331U\Desktop>SET HYDRA_FULL_ERROR=1
C:\Users\UX331U\Desktop>python mre.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\UX331U\Desktop\mre.py", line 8, in <module>
main()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\main.py", line 32, in decorated_main
_run_hydra(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 355, in _run_hydra
run_and_report(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 210, in run_and_report
raise ex
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 207, in run_and_report
return func()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 356, in <lambda>
lambda: hydra.run(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\hydra.py", line 100, in run
cfg = self.compose_config(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\hydra.py", line 505, in compose_config
self.config_loader.ensure_main_config_source_available()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\config_loader_impl.py", line 135, in ensure_main_config_source_available
self.missing_config_error(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\config_loader_impl.py", line 109, in missing_config_error
raise MissingConfigException(
hydra.errors.MissingConfigException: Primary config module 'config' not found.
Check that it's correct and contains an __init__.py file
I also attach a minimal code snippet to reproduce the error.
import hydra
from multiprocessing import Manager
#hydra.main(config_path='config', config_name='config')
def main(cfg):
m = Manager() # The error is in this line
m.Queue()
main()
The config.yaml is correctly created and located since when I don't use the Manager, the entire code works with no errors:
- PROJECT ----- config ---- config.yaml
|
|
----- main.py
As far as I have understood, the problem comes from the fact that hydra changes the directory when initialized and afterwards, when the Manager is created, hydra is called again to be initialized but as the directory has been changed, it can not find the config folder. If this is true, is it a bug or can it be solved?
Thank you so much for your help.
I'm not able to reproduce the error on my machine.
You can stop hydra from changing directories at runtime by passing hydra.job.chdir=False at the command line -- see the docs on how to disable changing current working dir to job's output dir.
python main.py hydra.job.chdir=False
It seems for some websites I need to use undetected chromedriver for Selenium automation. I already got Selenium and chromedriver installed in a venv environment, and also ran the below to install undetected chromedriver in the venv: python -m pip install undetected-chromedriver. I used the below code to start undetected chromedriver:
from selenium import webdriver
import undetected_chromedriver as uc
browser = uc.Chrome()
browser.get('any url here')
When I run the script in the venv I got error message. Could anyone help what was wrong?
PS C:\Users\user\my_venv> & c:/Users/user/my_venv/Scripts/Activate.ps1
(my_venv) PS C:\Users\user\my_venv> & c:/Users/user/my_venv/Scripts/python.exe c:/Users/user/my_venv/toughsite_test.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 269, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\Users\user\my_venv\toughsite_test.py", line 15, in <module>
browser = uc.Chrome()
File "C:\Users\user\my_venv\lib\site-packages\undetected_chromedriver\__init__.py", line 356, in __init__
self.browser_pid = start_detached(
File "C:\Users\user\my_venv\lib\site-packages\undetected_chromedriver\dprocess.py", line 35, in start_detached
).start()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
The simple fix is that you just have to write:
uc.Chrome(use_subprocess=True)
And thank me later ;)
This is because you do not have your main method wrapped in
if __name__ == "__main__":
foo()
I tried to run a multiprocessing function on spyder3. After it wouldnt print anything inside a loop and be stuck infinitly i read that I could run it on an external terminal in spyder like this.
Run > Configuration per file > Execute in an external system terminal
now it finally shows me something. Sadly its this:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 161, in <module>
cpus,closeIdx=valueSeries.index,t1=df['t1'])
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 27, in mpPandasObj
out=processJobs(jobs,numThreads=numThreads)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 75, in processJobs
pool=mp.Pool(processes=numThreads)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 119, in Pool
context=self.get_context())
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 176, in __init__
self._repopulate_pool()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
w.start()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
and the this occours in a (probably infinite) loop:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
this is the multiprocessing function:
def processJobs(jobs,task=None,numThreads=24):
# Run in parallel.
# jobs must contain a ’func’ callback, for expandCall
if task is None:
task=jobs[0]['func'].__name__
pool=mp.Pool(processes=numThreads)
outputs,out,time0=pool.imap_unordered(expandCall,jobs),[],time.time()
# Process asynchronous output, report progress
for i,out_ in enumerate(outputs,1): # <---- Here.
out.append(out_)
reportProgress(i,len(jobs),time0,task)
pool.close(); pool.join() # this is needed to prevent memory leaks return out
return out
How can I solve this? (I really need multiprocessing)
EDIT: tried it on Pycharm -> it reports the same errors
I just ran into this. You are probably directly calling processJobs() directly in the Python script.
You just need to add the main idiom like so:
if __name__ == '__main__':
processJobs()
I suspect multiprocessing will fork/spawn multiple Python processes in the OS and load the module, and not having this guard causes the Pool instantiation to be invoked recursively.
Having the main guard ensures that the Pool instantiation runs only once.
Colleague provided me set of py files for testing, which are running fine on his machine. However when I run the py files, I am getting following error:
Traceback (most recent call last):
File "D:\workspaces\pyworkspace\Analytics Python\python_server\server1.py", line 84, in <module>
start_server()
File "D:\workspaces\pyworkspace\Analytics Python\python_server\server1.py", line 57, in start_server
socketObj.bind(server_address)
LookupError: unknown encoding: idna
Here, it says to add import encodings.idna. So I added that import (even though eclipse says "unused import"), but still getting the same error. So I put some breakpoints and started up debugging, but debugger didnt hit the breakpoints. The program termniates without hitting any breakpoints giving following error:
pydev debugger: starting (pid: 8316)
Traceback (most recent call last):
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 741, in _pydevd_bundle.pydevd_cython_win32_34_64.PyDBFrame.trace_dispatch (_pydevd_bundle/pydevd_cython_win32_34_64.c:15515)
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 254, in _pydevd_bundle.pydevd_cython_win32_34_64.PyDBFrame.do_wait_suspend (_pydevd_bundle/pydevd_cython_win32_34_64.c:5631)
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 764, in do_wait_suspend
self._activate_mpl_if_needed()
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 416, in _activate_mpl_if_needed
activate_function()
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 407, in <lambda>
self.mpl_modules_for_patching = {"matplotlib": lambda: activate_matplotlib(do_enable_gui),
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydev_ipython\matplotlibtools.py", line 93, in activate_matplotlib
is_interactive = is_interactive_backend(backend)
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydev_ipython\matplotlibtools.py", line 56, in is_interactive_backend
from matplotlib.rcsetup import interactive_bk, non_interactive_bk # #UnresolvedImport
ImportError: cannot import name 'interactive_bk'
Traceback (most recent call last):
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 1621, in <module>
main()
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 1615, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 1022, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\_pydev_imps\_pydev_execfile.py", line 25, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "D:\Mahesh\workspaces\pyworkspace\Analytics Python\python_server\server.py", line 84, in <module>
start_server()
File "D:\Mahesh\workspaces\pyworkspace\Analytics Python\python_server\server.py", line 84, in <module>
start_server()
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 982, in _pydevd_bundle.pydevd_cython_win32_34_64.SafeCallWrapper.__call__ (_pydevd_bundle/pydevd_cython_win32_34_64.c:19346)
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 498, in _pydevd_bundle.pydevd_cython_win32_34_64.PyDBFrame.trace_dispatch (_pydevd_bundle/pydevd_cython_win32_34_64.c:18639)
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 750, in _pydevd_bundle.pydevd_cython_win32_34_64.PyDBFrame.trace_dispatch (_pydevd_bundle/pydevd_cython_win32_34_64.c:15669)
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 741, in _pydevd_bundle.pydevd_cython_win32_34_64.PyDBFrame.trace_dispatch (_pydevd_bundle/pydevd_cython_win32_34_64.c:15515)
File "_pydevd_bundle\pydevd_cython_win32_34_64.pyx", line 254, in _pydevd_bundle.pydevd_cython_win32_34_64.PyDBFrame.do_wait_suspend (_pydevd_bundle/pydevd_cython_win32_34_64.c:5631)
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 764, in do_wait_suspend
self._activate_mpl_if_needed()
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 416, in _activate_mpl_if_needed
activate_function()
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydevd.py", line 407, in <lambda>
self.mpl_modules_for_patching = {"matplotlib": lambda: activate_matplotlib(do_enable_gui),
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydev_ipython\matplotlibtools.py", line 93, in activate_matplotlib
is_interactive = is_interactive_backend(backend)
File "D:\Programs\Eclipse Oxygen\plugins\org.python.pydev_6.2.0.201711281614\pysrc\pydev_ipython\matplotlibtools.py", line 56, in is_interactive_backend
from matplotlib.rcsetup import interactive_bk, non_interactive_bk # #UnresolvedImport
ImportError: cannot import name 'interactive_bk'
Why is this happening? (I am using python 3.4, eclipse oxygen 4.7 and PyDev 6.2.0.201711281614)
Apparently the problem is because pydev is expecting matplotlib to have some modules which aren't available in your installation... so, the import:
from matplotlib.rcsetup import interactive_bk, non_interactive_bk # #UnresolvedImport
is failing (and that's expected in PyDev if you have matplotlib installed).
Not sure why... I need to investigate whether you have a broken matplotlib or if matplotlib actually changed (in which case the pydev debugger would need to be fixed).
Can you add details on the matplotlib version you have?
This is the multithreaded code I'm trying to import as a module. It works fine when I run it as a stand alone file. It simply just prints out a list of numbers. All I change is which main() is commented out. I think that the problem might be that the imported code is calling itself as a module after the reader file has already called it as a module.
threadtest.py
from multiprocessing import Process, Value, Array
import time
import sys
def f(n, a):
n.value = 3.1415927
for i in range(len(a)):
a[i] = -a[i]
def is_prime(x, top):
for j in range(2,int(top**.5 + 1)):
if x%j == 0:
return False
return True
def f1(top, c):
print('test f1')
for i in range(2,top):
if is_prime(i,top):
c.value = c.value + 1
def f3(c1,c2):
for k in range(0,20):
time.sleep(1) # 1 second
sys.stdout.write(str(c1.value) + '|' + str(c2.value) + '\n')
sys.stdout.flush()
def main():
count1 = Value('d', 0)
count2 = Value('d', 0)
#arr = Array('i', range(10))
p1 = Process(target=f1, args=(1000000, count1))
p2 = Process(target=f1, args=(1000000, count2))
p3 = Process(target=f3, args=(count1, count2))
p1.start()
p2.start()
p3.start()
p1.join()
print('p1.join()')
p2.join()
print('p2.join()')
p3.join()
print('p3.join()')
print(count1.value)
print(count2.value)
if __name__ == '__main__':
print('grovetest is being run as main')
#main()
else:
print('grovetest is being run as module')
main()
This is the code that imports the multithreaded module and attempts to read the output.
readertest.py
import threadtest
def main(fname):
try:
p = subprocess.Popen(fname, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
print('success')
return p.communicate() # this gets you pipe values
except Exception as e:
return 'error'+str(e)
else:
return "else"
if __name__ == '__main__':
main(threadtest)
Here is the error that is produced when I run the readertest.py
grovetest is being run as module
grovetest is being run as module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
grovetest is being run as module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 240, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Joseph\documents\python projects\multitest.py", line 1, in <module>
import grovetest
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 56, in <module>
main()
File "C:\Users\Joseph\documents\python projects\grovetest.py", line 37, in main
p1.start()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Joseph\AppData\Local\Programs\Python\Python35-32\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
grovetest is being run as module
p1.join()
p2.join()
p3.join()
0.0
0.0
I'm new to this. Thank you for your feedback.
You're violating a significant Windows-specific constraint called out here:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
One flaw in the paragraph I just quoted is that it says "the main module" when it should say "every module", but most people don't try to run complex code from imported modules and thus don't run across this issue. In your case, what happens is that readertest imports grovetest, which invokes the multiprocessing code, which fires up a new Python interpreter on grovetest using the special Windows multiprocess bootstrap system, which fires up the multiprocessing code, which detects that it's being fired up under the special bootstrap case and aborts:
(start)
readertest:
import grovetest
=> grovetest
import multiprocessing
=> multiprocessing: define some stuff
=> grovetest
if __name__ == '__main__': # it's not equal
... # so we skip this
else:
print('grovetest is being run as module') # prints
main() # calls main
...
p1 = Process(target=f1, args=(1000000, count1)) # calls multiprocessing module
=> new process
import grovetest # multiprocessing does this using its magic bootstrap
...
if __name__ == '__main__': # it's not
...
else:
print(...)
...
p1 = Process(...)
At this point the inner Process code detects the constraint violation and raises the first error. This terminates the inner process entirely, and we're back to the outer process that does p2 = Process(...), which kicks off the same error again, and so on.
To fix this, you must not call your main in grovetest when grovetest is imported as a module. Instead, call grovetest.main later, from readertest. That will allow the special bootstrap magic (in the multiprocessing module, run up when you invoke multiprocessing.Process) to import grovetest in the new process. Once this new process has finished importing grovetest, it will wait for instructions from its parent process (the other Python interpreter, where you called multiprocessing.Process, from your readertest code calling grovetest.main). Those instructions will be to invoke function f1 (from target=f1) with the supplied arguments.