I want to run an MXNet module in GPU.
I have a system which has Ubuntu 18.04 along Cuda 10.0 installed. Apparently this is not covered yet by MXNet binary files so I was focusing on installing 2 cuda versions in my pc (see also here).
Anyway I now have 2 cuda toolkits in my pc in different folders. I need a way to direct my system to use Cuda 9.2 when run from PyCharm. The funny thing is that from a typical console I can run it just fine (at least the MXNet loading part that is of course).
In the module I want to run the program is stuck in:
import mxnet as mx
which leads to base.py in MXNet:
def _load_lib():
"""Load library by searching possible path."""
lib_path = libinfo.find_lib_path()
lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL) # <- This is where is throws the error.
# DMatrix functions
lib.MXGetLastError.restype = ctypes.c_char_p
return lib
the strange thing is that lib_path[0] just points to the location of libmxnet.so (which is correct by the way) and suddenly it throws an error:
OSError: libcudart.so.9.2: cannot open shared object file: No such
file or directory
Even if I follow the error trace the last command is this:
self._handle = _dlopen(self._name, mode)
with self._name being the same location of libmxnet.so.
I have tried to make it work by changing the system variable with
os.environ["LD_LIBRARY_PATH"] = "/usr/local/cuda-9.2/lib64"
as the second line of the module (the 1st is of course import os!) but this does not seem to work. Apparently it's taken into consideration.
So, how can I bypass this?
Any solution would be acceptable being on the MXNet side or pyCharm side.
Well, to make this available to anyone facing the same problem I will post my solution.
I managed to make it work by defining the environmental variable inside pycharm from the run configuration menu (the one that it's available from Run->Run... or Alt+Shift+F10) and defining it there as environmental variable.
LD_LIBRARY_PATH: /usr/local/cuda-9.2/lib64
I am not sure why for that case pycharm is working fine while when the same variable is defined inside the code it does not though (any explanation welcome).
Related
I am trying to run a C function in Python. I followed examples online, and compiled the C source file into a .so shared library, and tried to pass it into the ctypes CDLL() initializer function.
import ctypes
cFile = ctypes.CDLL("libchess.so")
At this point python crashes with the message:
Could not find module 'C:\Users\user\PycharmProjects\project\libchess.so' (or one of its dependencies). Try using the full path with constructor syntax.
libchess.so is in the same directory as this Python file, so I don't see why there would be an issue finding it.
I read some stuff about how shared libraries might be hidden from later versions of python, but the suggested solutions I tried did not work. Most solutions were also referring to fixes involving linux system environment variables, but I'm on Windows.
Things I've tried that have not worked:
changing "libchess.so" to "./libchess.so" or the full path
using cdll.LoadLibrary() instead of CDLL() (apparently both do the same thing)
adding the parent directory to system PATH variable
putting os.add_dll_directory(os.getcwd()) in the code before trying to load the file
Any more suggestions are appreciated.
Solved:
Detailed explanation here: https://stackoverflow.com/a/64472088/16044321
The issue is specific to how Python performs a DLL/SO search on Windows. While the ctypes docs do not specify this, the CDLL() function requires the optional argument winmode=0 to work correctly on Windows when loading a .dll or .so. This issue is also specific to Python versions greater than 3.8.
Thus, simply changing the 2nd line to cFile = ctypes.CDLL("libchess.so", winmode=0) works as expected.
I am new with programming, so it is maybe harder for me to understand but I have the following issue:
I have a script that imports "eurostag.dll", which according to its manual, should work until Python 3.6. (but the manual is not updated, so it may work also with later updates, I assume).\ The issue is that I have Python 3.8. and when running the script I receive the following message:
"Failed to load EUROSTAG library (Could not find module 'D:\Eurostag\eustag_esg.dll' (or one of its dependencies). Try using the full path with constructor syntax.)"
I have tried to move the .dll library where the script is, but nothing changed. I tried also changing the directory with os.chdir, but the same message appears (with the same 'D:\Eurostag\eustag_esg.dll', so the directory was not changed.
Does anybody know if there is any workaround for this?
Thank you!
I'm currently having interop problems between a proprietry library and pyembree.
Problem
I have two mostly identical conda enviroments. Both contain this in-house library. The only difference is that I installed pyembree via these instructions in one environment. Now in this environment, the in-house library cannot load a DLL which it has no problem loading in the other. The absolute (correct) path is even supplied to the ctypes.WinDLL object:
On module level (in a module of the proprietatry library):
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
kernel32.AddDllDirectory(os.path.abspath(os.path.dirname(__file__)))
On import this is called (where name is the correct absolute path and mode is 4096):
handle = kernel32.LoadLibraryExW(name, None, mode)
Which raises a ctypes.WinError with the message 'OSError: [WinError 127] The specified procedure could not be found.'.
Tracking overwritten DLLs
I tried working with that error message based on https://stackoverflow.com/a/2603386/5409315 . That answer says that the error message means that a DLL is loaded, but in the wrong version. Initially, that made a lot of sense - the install of pyembree could have overwritten a dependency of the proprietry lib. (Although I don't understand why that message would be triggered on load as compared to when the procedure is supposed to be used.)
However, what I found with Process Monitor was that in both environments two dozen DLLs are searched but not found (e.g. OPENGL32.dll) and that the only DLL that is successfully loaded in both environments (vcomp140.dll) does not differ between the two environments.
I cross-checked this with Dependency Walker, where I found a lot more depended DLLs but had the same ultimate result: In both environments, the depended libraries are either literally identical or bytewise identical.
The PATH environment variable is also equivalent (that is, where a path contains the environment directory, it differs in exactly that regard between the environments).
Minimal reproducible example
I can't provide a minimally reproducible example because the proprietary library is not publicly available.
Another approach
I'd like to dump all global variables just before the attempt to load the DLL in both environments, diff the output and thus find the crucial difference. Is this possible in VS Code? Is this possible in some other context (pdb, ...)?
I created a python extension using Boost::Python. To make it easier to use the extension on different target machines, I have included the libboost_python36.so.1.75.0 library in the same directory as the generated extension (pyshmringbuffer.so).
I checked out pyshmringbuffer.so and libboost_python36.so.1.75.0 onto a machine other than it was compiled in the directory : /path/to/pyshmringbuffer
After setting LD_LIBRARY_PATH to: /path/to/pyshmringbuffer and changing to this directory, I am able to run python3.6 and import the shared object just fine.
The problem comes when I try to run python from an alternate directory. From any other directory, I append the python path as follows:
import sys
sys.path.append("/path/to/pyshmringbuffer")
Then, when I try to import pyshmringbuffer, I get the following undefined symbol:
ImportError: /path/to/pyshmringbuffer/pyshmringbuffer.so: undefined symbol: _ZNK5boost6python7objects21py_function_impl_base9max_arityEv
I was under the impression that all symbols are self contained within the shared object. Why does it matter where I import the shared library from?
The symbol in your error message is an internal one, generated by one of the build tools. Having one undefined suggests that one of your components was built with an incompatible tool version, or that a *.so file (shared object) is out of date in some other fashion.
The simplest way to fix this is usually to rebuild your product components from scratch, in the proper order.
I was able to resolve my issue by prepending /path/to/pyshmringbuffer to my python path using:
sys.path.insert(0,"/path/to/pyshmringbuffer")
I can't say for sure, but as #PRUNE pointed out, there is something in my python path that python is seeing before it sees the intended library.
Coincidentally, I DO have a build of libboost_python36.so.1.75.0 located elsewhere on the target machine. The path for this doesn't appear on my PYTHONPATH or LD_LIBRARY_PATH, so I wouldn't EXPECT it to have been interfering, but I can't be positive.
I’m having trouble using python’s multiprocessing module. This is the first time I’ve tried using the module. I’ve tried simplifying my processing to the bare bones, but keep getting the same error. I’m using python 2.7.2, and Windows 7.
The script I’m trying to run is called learnmp.py, and the error message says that the problem is that it can't find module learnmp.
import multiprocessing
def doSomething():
"""worker function"""
print 'something'
return
if __name__ == '__main__':
jobs = []
for i in range(2):
p = multiprocessing.Process(target=doSomething)
jobs.append(p)
p.start()
The error is :
File “<string>”, line 1, in <module> File “C:\Python27\ArcGISx6410.1\lib\multiprocessing\forking.py”, line 373,
in main prepare(preparation_data) File “C:\Python27\ArcGISx6410.1\lib\multiprocessing\forking.py”, line 482,
in prepare file, path_name, etc = imp.find_module (main_name, dirs)
ImportError: No module named learnmp
What’s causing the error, and how can I solve it?
EDIT: I still don't know what was causing the error, but changing the file name eliminated it.
I know it's been a while, but I ran into this same error, also using the version of Python distributed with ArcGIS, and I've found a solution which at least worked in my case.
The problem that I had was that I was calling my program name, Test.py, as test.py. Note the difference in case.
c:\python27\arcgisx6410.2\python.exe c:\temp\test.py
c:\python27\arcgisx6410.2\python.exe c:\temp\Test.py
This isn't normally an issue if you're not using the multiprocessing library. However, when you write:
if __name__ == '__main__':
what appears to be happening is that the part of the program in main is being bound to the name of the python file. In my case that was test. However, there is no test, just Test. So although Windows will allow case-incorrect filenames in cmd, PowerShell, and in batch files, Python's multiprocessing library balks at this and throws a nasty series of errors.
Hopefully this helps someone.
Looks like you might be going down a rabbit-hole looking into multiprocessing. As the traceback shows, your python install is trying to look in the ArcGIS version of python before actually looking at your system install.
My guess is that the version of python that ships with ArcGIS is slightly customized for some reason or another and can't find your python script. The question then becomes:
Why is your Windows machine looking in ArcGIS for python?
Without looking at your machine at a slightly lower level I can't quite be sure, but if I had to guess, you probably added the ArcGIS directory to your PATH variable in front of the standard python directory, so it looks in ArcGIS first. If you move the ArcGIS path to the end of your PATH variable it should resolve the problem.
Changing your PATH variable: http://www.computerhope.com/issues/ch000549.htm
Microsoft Visual C++ 9.0 is required for some python modules to work in windows,so download below package it will work.
http://aka.ms/vcpython27
This package contains the compiler and set of system headers necessary for producing binary wheels for Python 2.7 packages.