This is the first time I am using pathos library in python under Visual Studio 2019 on windows 10. When the debugger encounters the line
solver.SetMapper(Pool(self.Config.NumberOfProcessors).map)
I get error
ModuleNotFoundError: No module named 'multiprocess'
I have the following statement at the beginning of my code
from pathos.pools import ProcessPool as Pool
I have C++ compilers(multiple version of Visual studios) and I have used latest version of pip to install the packages. I also see that the multiprocess package has been installed under pathos.
I do see multiple question on the same topic on the web, but unable to resolve the issue
Actually I can reproduce the same situation with a simple example like:
def foo(x):
return x
def bar(x):
return foo(x)
x=Pool(4).map(bar, [0, 1])
print (x)
A new installation of python 3.8.1 seems to make this issue go away
Related
I am trying to implement multiprocessing in my code, and so, I thought that I would start my learning with some examples. I used the first example found in this documentation.
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
When I run the above code I get an AttributeError: can't get attribute 'f' on <module '__main__' (built-in)>. I do not know why I am getting this error. I am also using Python 3.5 if that helps.
This problem seems to be a design feature of multiprocessing.Pool. See https://bugs.python.org/issue25053. For some reason Pool does not always work with objects not defined in an imported module. So you have to write your function into a different file and import the module.
File: defs.py
def f(x):
return x*x
File: run.py
from multiprocessing import Pool
import defs
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(defs.f, [1, 2, 3]))
If you use print or a different built-in function, the example should work. If this is not a bug (according to the link), the given example is chosen badly.
The multiprocessing module has a major limitation when it comes to IPython use:
Functionality within this package requires that the __main__ module be
importable by the children. [...] This means that some examples, such
as the multiprocessing.pool.Pool examples will not work in the
interactive interpreter. [from the documentation]
Fortunately, there is a fork of the multiprocessing module called multiprocess which uses dill instead of pickle to serialization and overcomes this issue conveniently.
Just install multiprocess and replace multiprocessing with multiprocess in your imports:
import multiprocess as mp
def f(x):
return x*x
with mp.Pool(5) as pool:
print(pool.map(f, [1, 2, 3, 4, 5]))
Of course, externalizing the code as suggested in this answer works as well, but I find it very inconvenient: That is not why (and how) I use IPython environments.
<tl;dr> multiprocessing does not work in IPython environments right away, use its fork multiprocess instead.
This answer is for those who get this error on Windows 10 in 2021.
I've researched this error a bit since I got it myself. I get this error when running any examples from the official Python 3 documentation on multiprocessing.
Test environment:
x86 Windows 10.0.19043.1165 + Python 3.9.2 - there is an error
x86 Windows 10.0.19043.1165 + Python 3.9.6 - there is an error
x86 Windows 10.0.19043.1110 + Python 3.9.6 - there is an error
ARM Windows 10.0.21354.1 + Python 3.9.6 - no error (version from DEV branch)
ARM macOS 11.5.2 + Python 3.9.6 - no errors
I have no way to test this situation in other conditions. But my guess is that the problem is on Windows as there is no such bug in the developer version "10.0.21354.1", but this ARM version probably has x86 emulation.
Also note that there was no such bug at the time Python 3.9.2 was released (February). Since all this time I was working on the same computer, I was surprised by the situation when the previously working code stopped working, and only the version for Windows changed.
I was unable to find a bug request with a similar situation in the Python bug tracker (I probably did a poor search). And the message marked "Correct answer" refers to a different situation. The problem is easy to reproduce, you can try to follow any example from the multiprocessing documentation on a freshly installed Windows 10 + Python 3.
Later, I will have the opportunity to check out Python 3.10 and the latest version of Windows 10.
I am also interested in this situation in the context of Windows 11.
If you have information about this error (link to the bug tracker or something similar), be sure to share it.
At the moment I switched to Linux to continue working.
Why not use joblib? Your code is equivalent to:
# pip install joblib
from joblib import Parallel, delayed
def f(x):
return x*x
res = Parallel(
n_jobs=5
)(
delayed(f)(x) for x in [1, 2, 3]
)
If you're using Jupyter notebook (like the OP), then defining the function in a separate cell and executing that cell first fixes the problem. The accepted answer works too, but it's more work. Defining the function before, i.e. above the pool, isn't adequate. It has to be in a completely different notebook cell which is executed first.
Python version: 3.8.1
Spyder version: 3.3.6
Qt version: 5.12.9
Wrapper: develop using PyBind11
I am wrapping a dll develop in C++ which use Qt dlls to be used with Python. I wrote the wrapper with Visual Studio 2019 using the compiler MSVC (as my dll is compiled with MSVC). After generating the solution in VS2019 I obtain a .pyd file which can be import with python.
It works good when I use python on line command:
Start cmd.exe
$python
import MyLibName
I can use the functions/classes ...
But if I try with Spyder, I get the following error:
ImportError: DLL load failed while importing PyStack: The specified module could not be found..
So here are my questions :
Is there a way to get more information about ImportError like the name of the missing dll or something?
I don't understand why the issue only happen with spyder. I tried with IPython Qt Console and it work. Does spyder use a embeded python version or something ?
I don't fully understand how dll shall be managed, I mean shall I provide dll like libGLESV2.dll with the .pyd or just give a path where to find it ?
Thank you in advance.
My guess
I think I find out which part of Qt/python is producing this issue, but I still don't know how to solve it.
My dll use signals/slots which need an event loop to be performed. If an event loop is already running the dll will try to use it, if the loop version (ex : PyQt5==5.14.1) isn’t the same as mine (ex Qt==5.15.1) import will be impossible.
Note that the reverse is true, if I run my dll an then try to start a loop with %gui qt the command will throw an error.
How to reproduce the issue :
Compile a Qt project available here.
Copy the output dll in the folder PyMyStack/dependencies of the VS Project (available here)
Compile the VS project.
Open an IPython console (without using qt has event loop)
Import the module created with VS (Import PyMyStack)
Run the magic command %gui qt
Last command shall print : ERROR:root:DLL load failed while importing QtSvg: The specified procedure could not be found.
How to hide/solve the problem:
Disclaimer : The solutions presented here are surely not the best, if you know a better one please share it ☺
If you just want to import your lib in Spyder, you can use another event loop. Here are the steps to change this:
In Spyder menus go to Tools→Preferences
Select “IPython console”
Go to “Graphics” tab and change the backend combo box to any other values than Qt or Automatic
If you want to use Qt event loop you will have to update it. You can do this with pip command, but remember than Spyder is not compatible with some version. Here is the pip command:
Pip install PyQt5==X.Y.Z
Where X and Y are the same version use to compile your Qt project. The last digit version seems to not be important.
I am trying to implement multiprocessing in my code, and so, I thought that I would start my learning with some examples. I used the first example found in this documentation.
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
When I run the above code I get an AttributeError: can't get attribute 'f' on <module '__main__' (built-in)>. I do not know why I am getting this error. I am also using Python 3.5 if that helps.
This problem seems to be a design feature of multiprocessing.Pool. See https://bugs.python.org/issue25053. For some reason Pool does not always work with objects not defined in an imported module. So you have to write your function into a different file and import the module.
File: defs.py
def f(x):
return x*x
File: run.py
from multiprocessing import Pool
import defs
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(defs.f, [1, 2, 3]))
If you use print or a different built-in function, the example should work. If this is not a bug (according to the link), the given example is chosen badly.
The multiprocessing module has a major limitation when it comes to IPython use:
Functionality within this package requires that the __main__ module be
importable by the children. [...] This means that some examples, such
as the multiprocessing.pool.Pool examples will not work in the
interactive interpreter. [from the documentation]
Fortunately, there is a fork of the multiprocessing module called multiprocess which uses dill instead of pickle to serialization and overcomes this issue conveniently.
Just install multiprocess and replace multiprocessing with multiprocess in your imports:
import multiprocess as mp
def f(x):
return x*x
with mp.Pool(5) as pool:
print(pool.map(f, [1, 2, 3, 4, 5]))
Of course, externalizing the code as suggested in this answer works as well, but I find it very inconvenient: That is not why (and how) I use IPython environments.
<tl;dr> multiprocessing does not work in IPython environments right away, use its fork multiprocess instead.
This answer is for those who get this error on Windows 10 in 2021.
I've researched this error a bit since I got it myself. I get this error when running any examples from the official Python 3 documentation on multiprocessing.
Test environment:
x86 Windows 10.0.19043.1165 + Python 3.9.2 - there is an error
x86 Windows 10.0.19043.1165 + Python 3.9.6 - there is an error
x86 Windows 10.0.19043.1110 + Python 3.9.6 - there is an error
ARM Windows 10.0.21354.1 + Python 3.9.6 - no error (version from DEV branch)
ARM macOS 11.5.2 + Python 3.9.6 - no errors
I have no way to test this situation in other conditions. But my guess is that the problem is on Windows as there is no such bug in the developer version "10.0.21354.1", but this ARM version probably has x86 emulation.
Also note that there was no such bug at the time Python 3.9.2 was released (February). Since all this time I was working on the same computer, I was surprised by the situation when the previously working code stopped working, and only the version for Windows changed.
I was unable to find a bug request with a similar situation in the Python bug tracker (I probably did a poor search). And the message marked "Correct answer" refers to a different situation. The problem is easy to reproduce, you can try to follow any example from the multiprocessing documentation on a freshly installed Windows 10 + Python 3.
Later, I will have the opportunity to check out Python 3.10 and the latest version of Windows 10.
I am also interested in this situation in the context of Windows 11.
If you have information about this error (link to the bug tracker or something similar), be sure to share it.
At the moment I switched to Linux to continue working.
Why not use joblib? Your code is equivalent to:
# pip install joblib
from joblib import Parallel, delayed
def f(x):
return x*x
res = Parallel(
n_jobs=5
)(
delayed(f)(x) for x in [1, 2, 3]
)
If you're using Jupyter notebook (like the OP), then defining the function in a separate cell and executing that cell first fixes the problem. The accepted answer works too, but it's more work. Defining the function before, i.e. above the pool, isn't adequate. It has to be in a completely different notebook cell which is executed first.
I’m having trouble using python’s multiprocessing module. This is the first time I’ve tried using the module. I’ve tried simplifying my processing to the bare bones, but keep getting the same error. I’m using python 2.7.2, and Windows 7.
The script I’m trying to run is called learnmp.py, and the error message says that the problem is that it can't find module learnmp.
import multiprocessing
def doSomething():
"""worker function"""
print 'something'
return
if __name__ == '__main__':
jobs = []
for i in range(2):
p = multiprocessing.Process(target=doSomething)
jobs.append(p)
p.start()
The error is :
File “<string>”, line 1, in <module> File “C:\Python27\ArcGISx6410.1\lib\multiprocessing\forking.py”, line 373,
in main prepare(preparation_data) File “C:\Python27\ArcGISx6410.1\lib\multiprocessing\forking.py”, line 482,
in prepare file, path_name, etc = imp.find_module (main_name, dirs)
ImportError: No module named learnmp
What’s causing the error, and how can I solve it?
EDIT: I still don't know what was causing the error, but changing the file name eliminated it.
I know it's been a while, but I ran into this same error, also using the version of Python distributed with ArcGIS, and I've found a solution which at least worked in my case.
The problem that I had was that I was calling my program name, Test.py, as test.py. Note the difference in case.
c:\python27\arcgisx6410.2\python.exe c:\temp\test.py
c:\python27\arcgisx6410.2\python.exe c:\temp\Test.py
This isn't normally an issue if you're not using the multiprocessing library. However, when you write:
if __name__ == '__main__':
what appears to be happening is that the part of the program in main is being bound to the name of the python file. In my case that was test. However, there is no test, just Test. So although Windows will allow case-incorrect filenames in cmd, PowerShell, and in batch files, Python's multiprocessing library balks at this and throws a nasty series of errors.
Hopefully this helps someone.
Looks like you might be going down a rabbit-hole looking into multiprocessing. As the traceback shows, your python install is trying to look in the ArcGIS version of python before actually looking at your system install.
My guess is that the version of python that ships with ArcGIS is slightly customized for some reason or another and can't find your python script. The question then becomes:
Why is your Windows machine looking in ArcGIS for python?
Without looking at your machine at a slightly lower level I can't quite be sure, but if I had to guess, you probably added the ArcGIS directory to your PATH variable in front of the standard python directory, so it looks in ArcGIS first. If you move the ArcGIS path to the end of your PATH variable it should resolve the problem.
Changing your PATH variable: http://www.computerhope.com/issues/ch000549.htm
Microsoft Visual C++ 9.0 is required for some python modules to work in windows,so download below package it will work.
http://aka.ms/vcpython27
This package contains the compiler and set of system headers necessary for producing binary wheels for Python 2.7 packages.
I'm working on a program that needs some threading. I wanted to get myself into this and tried several tutorials. Because of some special hardware I need to stick to Python 2.5.2 (r25:51908)
When I want to run the following code
import threading
class PrintThread(threading.Thread):
def run(self):
print "foobar"
thread = PrintThread()
thread.start()
I get the famous error "pythonw.exe has encountered a problem ...". This only happens on my PC. When I tried the very same code on a differenct PC with the same installation of Python (also with the same additional libraries like visa), it just works!
I uninstalled Python and reinstalled it again, but the error is still occuring. Any suggestions?