It seems for some websites I need to use undetected chromedriver for Selenium automation. I already got Selenium and chromedriver installed in a venv environment, and also ran the below to install undetected chromedriver in the venv: python -m pip install undetected-chromedriver. I used the below code to start undetected chromedriver:
from selenium import webdriver
import undetected_chromedriver as uc
browser = uc.Chrome()
browser.get('any url here')
When I run the script in the venv I got error message. Could anyone help what was wrong?
PS C:\Users\user\my_venv> & c:/Users/user/my_venv/Scripts/Activate.ps1
(my_venv) PS C:\Users\user\my_venv> & c:/Users/user/my_venv/Scripts/python.exe c:/Users/user/my_venv/toughsite_test.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 269, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "c:\Users\user\my_venv\toughsite_test.py", line 15, in <module>
browser = uc.Chrome()
File "C:\Users\user\my_venv\lib\site-packages\undetected_chromedriver\__init__.py", line 356, in __init__
self.browser_pid = start_detached(
File "C:\Users\user\my_venv\lib\site-packages\undetected_chromedriver\dprocess.py", line 35, in start_detached
).start()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
The simple fix is that you just have to write:
uc.Chrome(use_subprocess=True)
And thank me later ;)
This is because you do not have your main method wrapped in
if __name__ == "__main__":
foo()
Related
When I try to run spider test suite like described here:
https://github.com/taoyds/test-suite-sql-eval
I get this error:
Traceback (most recent call last): File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\user\PycharmProjects\test-suite-sql-eval\evaluate_classical.py", line 20, in <module> m = Manager() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 57, in Manager m.start() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\managers.py", line 562, in start self._process.start() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 336, in _Popen return Popen(process_obj) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__ prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data _check_not_importing_main() File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. python-BaseException
Can someome please help?
Thanks!
I tried to run this example command:
python3 evaluate_classical.py --pred=evaluation_examples/classical_test_gold.txt --out_file=all_eval_results.json
I expected to get some kind of evaluation, but got the above error.
I am developing a code where I need to create a Manager to synchronize a Queue. In the project I am using hydra and everything works fine until I create a multiprocessing Manager Object. This is the raised error with HYDRA_FULL_ERROR=1:
Primary config module 'config' not found.
Check that it's correct and contains an __init__.py file
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
File "mre.py", line 8, in <module>
main()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\main.py", line 32, in decorated_main
_run_hydra(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 355, in _run_hydra
run_and_report(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 207, in run_and_report
return func()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 356, in <lambda>
lambda: hydra.run(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\hydra.py", line 107, in run
return run_job(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\core\utils.py", line 125, in run_job
ret.return_value = task_function(task_cfg)
File "mre.py", line 7, in main
m = Manager()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 57, in Manager
m.start()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\managers.py", line 583, in start
self._address = reader.recv()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\connection.py", line 250, in recv
buf = self._recv_bytes()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\connection.py", line 305, in _recv_bytes
waitres = _winapi.WaitForMultipleObjects(
KeyboardInterrupt
^C
C:\Users\UX331U\Desktop>SET HYDRA_FULL_ERROR=1
C:\Users\UX331U\Desktop>python mre.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\UX331U\Desktop\mre.py", line 8, in <module>
main()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\main.py", line 32, in decorated_main
_run_hydra(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 355, in _run_hydra
run_and_report(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 210, in run_and_report
raise ex
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 207, in run_and_report
return func()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\utils.py", line 356, in <lambda>
lambda: hydra.run(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\hydra.py", line 100, in run
cfg = self.compose_config(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\hydra.py", line 505, in compose_config
self.config_loader.ensure_main_config_source_available()
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\config_loader_impl.py", line 135, in ensure_main_config_source_available
self.missing_config_error(
File "C:\Users\UX331U\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\hydra\_internal\config_loader_impl.py", line 109, in missing_config_error
raise MissingConfigException(
hydra.errors.MissingConfigException: Primary config module 'config' not found.
Check that it's correct and contains an __init__.py file
I also attach a minimal code snippet to reproduce the error.
import hydra
from multiprocessing import Manager
#hydra.main(config_path='config', config_name='config')
def main(cfg):
m = Manager() # The error is in this line
m.Queue()
main()
The config.yaml is correctly created and located since when I don't use the Manager, the entire code works with no errors:
- PROJECT ----- config ---- config.yaml
|
|
----- main.py
As far as I have understood, the problem comes from the fact that hydra changes the directory when initialized and afterwards, when the Manager is created, hydra is called again to be initialized but as the directory has been changed, it can not find the config folder. If this is true, is it a bug or can it be solved?
Thank you so much for your help.
I'm not able to reproduce the error on my machine.
You can stop hydra from changing directories at runtime by passing hydra.job.chdir=False at the command line -- see the docs on how to disable changing current working dir to job's output dir.
python main.py hydra.job.chdir=False
I am using executor.map and storing results in 'results'.
Following code works without any problem on Jupiter notebook. However, it crashes when executing the python script in Windows command prompt.
rop_test.py
print ('start of process')
import concurrent.futures
from funcfile import func
if __name__ == '__main__':
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(func, varlist1,varlist2)
global master_list
master_list=list(results)
print(master_list)
#do something with master_list
funcfile.py
def func(var1,var2):
var3=var1+var2
return var3
Error
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Administrator\notebooks\rop_test.py", line 640, in <module>
print (master_list)
NameError: name 'master_list' is not defined
Finished in 0.0 second(s)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Administrator\notebooks\rop_test.py", line 640, in <module>
print (master_list)
NameError: name 'master_list' is not defined
Traceback (most recent call last):
File "rop_test.py", line 634, in <module>
master_list=list(results)
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists
for element in iterable:
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 598, in result_iterator
yield fs.pop().result()
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
I have tried putting this in a main command, adding wait, adding [var for var in results], list(results) etc. but to no avail.
N.B. I am on python 3.6
So I tried making an simple example using the python multiprocessing module.
import time
import multiprocessing
start = time.perf_counter()
def do_something():
print('Sleeping 2 sec ...')
time.sleep(2)
print('Done Sleeping')
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')
But when I run it, a very long error traceback is printed:
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Goal\VSB\PDS\Project> & C:/Users/Goal/AppData/Local/Programs/Python/Python39/python.exe
c:/Goal/VSB/PDS/Project/MultiprocessingPython.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
self._popen = self._Popen(self)
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
ne 134, in _check_not_importing_main
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Finished in 0.2 second(s)
PS C:\Goal\VSB\PDS\Project> & C:/Users/Goal/AppData/Local/Programs/Python/Python39/python.exe
c:/Goal/VSB/PDS/Project/MultiprocessingPython.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
Traceback (most recent call last):
File "<string>", line 1, in <module>
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
exitcode = _main(fd, parent_sentinel)
p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
prepare(preparation_data)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
main_content = runpy.run_path(main_path,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return _run_module_code(code, init_globals, run_name,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_run_code(code, mod_globals, init_globals,
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
exec(code, run_globals)
File "c:\Goal\VSB\PDS\Project\MultiprocessingPython.py", line 14, in <module>
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable. p1.start()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py",
line 121, in start
self._popen = self._Popen(self)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py",
line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Goal\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Finished in 0.16 second(s)
PS C:\Goal\VSB\PDS\Project>
And there are so many things here I don't know if they can be called error? They are just so many that I dont even know what to search on the internet to solve my problem.
When Python throws these massive error statements, make sure to know what you are looking for- the important part to notice is that the error is stemming from a raise call, meaning that you need to look after the raise keywords in the error text.
As you can see, the actual text error thrown (The text found after the line with the raise keyword) is the following:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
After that, you can easily search this up and find a very helpful and detailed answer here (TLDR: You need to use if __name__ == '__main__': so that the code does not recursively import).
I tried to run a multiprocessing function on spyder3. After it wouldnt print anything inside a loop and be stuck infinitly i read that I could run it on an external terminal in spyder like this.
Run > Configuration per file > Execute in an external system terminal
now it finally shows me something. Sadly its this:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 161, in <module>
cpus,closeIdx=valueSeries.index,t1=df['t1'])
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 27, in mpPandasObj
out=processJobs(jobs,numThreads=numThreads)
File "C:\Users\ben\Arc_Project\löschbarer_test_der_mp_func.py", line 75, in processJobs
pool=mp.Pool(processes=numThreads)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 119, in Pool
context=self.get_context())
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 176, in __init__
self._repopulate_pool()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
w.start()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "c:\users\ben\appdata\local\programs\python\python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
and the this occours in a (probably infinite) loop:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
this is the multiprocessing function:
def processJobs(jobs,task=None,numThreads=24):
# Run in parallel.
# jobs must contain a ’func’ callback, for expandCall
if task is None:
task=jobs[0]['func'].__name__
pool=mp.Pool(processes=numThreads)
outputs,out,time0=pool.imap_unordered(expandCall,jobs),[],time.time()
# Process asynchronous output, report progress
for i,out_ in enumerate(outputs,1): # <---- Here.
out.append(out_)
reportProgress(i,len(jobs),time0,task)
pool.close(); pool.join() # this is needed to prevent memory leaks return out
return out
How can I solve this? (I really need multiprocessing)
EDIT: tried it on Pycharm -> it reports the same errors
I just ran into this. You are probably directly calling processJobs() directly in the Python script.
You just need to add the main idiom like so:
if __name__ == '__main__':
processJobs()
I suspect multiprocessing will fork/spawn multiple Python processes in the OS and load the module, and not having this guard causes the Pool instantiation to be invoked recursively.
Having the main guard ensures that the Pool instantiation runs only once.