I'm attempting to switch from a GNU + mingwpy to an MSVC toolchain to build my wheel, which includes a Cython wrapper around a binary.
I've built the external DLL with an MSVC toolchain in Appveyor, and, also in Appveyor, I've got the wheel build step working. After the build step completes, I'm testing by doing the following:
installing and activating a clean virtualenv
installing dependencies
installing the wheel I just built
running nosetests wheelname
However, this is now failing:
======================================================================
ERROR: Failure: ImportError (No module named cutil)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Miniconda\envs\test_env\lib\site-packages\nose\loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "C:\Miniconda\envs\test_env\lib\site-packages\nose\importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "C:\Miniconda\envs\test_env\lib\site-packages\nose\importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "C:\projects\simplification\simplification\test\test.py", line 5, in <module>
from simplification.cutil import simplify_coords as csimplify_coords
ImportError: No module named cutil
----------------------------------------------------------------------
see full traceback
Running dumpbin /dependents on cutil.pyd seems to show what I expected:
Dump of file simplification/cutil.pyd
File Type: DLL
Image has the following dependencies:
rdp.dll
python27.dll
MSVCR90.dll
KERNEL32.dll
Summary
4000 .data
2000 .rdata
3000 .reloc
12000 .text
Where rdp.dll is the external binary I'm wrapping with Cython.
What's going wrong here?
Update: My module can also import the binary using Ctypes, and when I disable the Cython-specific imports and tests in my test script, everything works. So I know my external binary is OK. I'm still confused as to why the import is failing, especially as the Cython build step of cutil.pyd completes, and dumpbin says it's looking for its rdp.dll dependency in the right place (i.e. in the same directory).
Update 2: Running dumpbin on cutil.pyd that's been built with a GNU toolchain gives the following:
Dump of file simplification/cutil.pyd
File Type: DLL
Image has the following dependencies:
msvcr90.dll
python27.dll
KERNEL32.dll
USER32.dll
rdp.dll
Summary
1000 .CRT
1000 .bss
3000 .data
1000 .edata
2000 .idata
1000 .pdata
4000 .rdata
1000 .reloc
1000 .rsrc
1E000 .text
1000 .tls
1000 .xdata
But I have no idea whether the dependency order is significant or whether the metadata in the Summary section matters
This seems to be an issue with the virtualenv installed by Conda. I have no idea how to fix it, so I stopped using a virtualenv installed by conda.
Related
I am trying to install pyscipopt. This should be done using pip install pyscipopt. The only catch is that pyscipopt requires the SCIP Optimization suite. Here is where I am having trouble. I wanted to make sure that my basic steps were correct.
The first way:
Following (https://github.com/SCIP-Interfaces/PySCIPOpt/blob/master/INSTALL.md)
Download SCIP optimization suite from https://www.scipopt.org/index.php#download
Then cd to my downloads folder. There, there is
scipoptsuite-7.0.1.tar
Run
tar xvfz scipoptsuite-7.0.1.tar
cd scipoptsuite-7.0.1
Then, following (https://scipopt.org/doc/html/CMAKE.php)
mkdir build
cd build
cmake ..
make
make check
make install
Alternatively, I also tried using make (https://scipopt.org/doc/html/MAKE.php)
I then either set export SCIPOPTDIR=<path_to_install_dir>, where <path_to_install_dir> is the path to scipoptsuite-7.0.1, in my downloads directory, or I added the scipoptsuite-7.0.1 directory to my $PATH. Either way, pip install pyscipopt worked.
However,
Ultimately I get (on one machine)
>>> import pyscipopt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/usr/anaconda/lib/python2.7/site-packages/pyscipopt/__init__.py", line 10, in <module>
from pyscipopt.scip import Model
ImportError: dlopen(/Users/usr/anaconda/lib/python2.7/site-packages/pyscipopt/scip.so, 2): Library not loaded: libscip.7.0.dylib
Referenced from: /Users/usr/anaconda/lib/python2.7/site-packages/pyscipopt/scip.so
Reason: image not found
On another machine I get
>>> import pyscipopt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/s/Library/Python/2.7/lib/python/site-packages/pyscipopt/__init__.py", line 10, in <module>
from pyscipopt.scip import Model
ImportError: dlopen(/Users/sam/Library/Python/2.7/lib/python/site-packages/pyscipopt/scip.so, 2): Library not loaded: libscip.7.0.dylib
Referenced from: /Users/s/Library/Python/2.7/lib/python/site-packages/pyscipopt/scip.so
Reason: unsafe use of relative rpath libscip.7.0.dylib in /Users/s/Library/Python/2.7/lib/python/site-packages/pyscipopt/scip.so with restricted binary
Is anyone able to install this software on a mac and give me a step by step guide, so at least I know I am not making any obvious mistakes? I guess there may be problems with my computers that can not be resolved, but I wanted to just make sure I had pieced together the instructions from various links correctly..
Another things I tried was just downloading the sh and running
sh scipopsuite-[VERSION].sh. This is considerably faster than make/cmake, and gives me what seems to be required in terms of lib, include and headers, but then when I try to pip install pyscipopt I usually get errors of the form scip/scip.h no such file or directory, command gcc failed as in (https://github.com/SCIP-Interfaces/PySCIPOpt/issues/54).
When installing pyopencl with pip, it complains about missing numpy:
Collecting pyopencl
Downloading https://files.pythonhosted.org/packages/bc/58/3ab1246e94986f1b6953e76d7ea7e69d2dbfef7b3f3874eded48524a024f/pyopencl-2018.2.2.tar.gz (341kB)
100% |████████████████████████████████| 348kB 9.5MB/s
Complete output from command python setup.py egg_info:
---------------------------------------------------------------------------
Pybind11 is not installed.
---------------------------------------------------------------------------
Very likely, the build process after this message will fail.
Simply press Ctrl+C and type
python -m pip install pybind11
to fix this. If you don't, the build will continue
in a few seconds.
[1] https://pybind11.readthedocs.io/en/stable/
---------------------------------------------------------------------------
Continuing in 1 seconds...
---------------------------------------------------------------------------
Mako is not installed.
---------------------------------------------------------------------------
That is not a problem, as most of PyOpenCL will be just fine
without it. Some higher-level parts of pyopencl (such as
pyopencl.reduction) will not function without the templating engine
Mako [1] being installed. If you would like this functionality to
work, you might want to install Mako after you finish
installing PyOpenCL.
Simply type
python -m pip install mako
either now or after the installation completes to fix this.
[1] http://www.makotemplates.org/
---------------------------------------------------------------------------
Hit Ctrl-C now if you'd like to think about the situation.
---------------------------------------------------------------------------
Continuing in 1 seconds...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-nccshu4t/pyopencl/setup.py", line 353, in <module>
main()
File "/tmp/pip-install-nccshu4t/pyopencl/setup.py", line 320, in main
language='c++',
File "/tmp/pip-install-nccshu4t/pyopencl/aksetup_helper.py", line 41, in __init__
self._include_dirs = self.include_dirs
File "/tmp/pip-install-nccshu4t/pyopencl/aksetup_helper.py", line 55, in get_include_dirs
return self._include_dirs + self.get_additional_include_dirs()
File "/tmp/pip-install-nccshu4t/pyopencl/aksetup_helper.py", line 52, in get_additional_include_dirs
return [self.get_numpy_incpath()]
File "/tmp/pip-install-nccshu4t/pyopencl/aksetup_helper.py", line 47, in get_numpy_incpath
file, pathname, descr = find_module("numpy")
File "/home/app/nb-ocl/.venv/lib/python3.5/imp.py", line 296, in find_module
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'numpy'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-nccshu4t/pyopencl/
This is an issue with the latest version of pyopencl published in PyPI. The latest version in the github repo already addresses this issue, using the new ExtensionUsingNumpy instead of the old NumpyExtension in setup.py::setup().
Still, if Mako and pybind11 dependencies are not installed, the installation of pyopencl wil fail. Why aren't these handled by setup.py/setuptools as well?
Since there are no binaries (wheels) available in PyPI, pip install defaults to pulling the software distribution (sdist) archive and tries to build it. The building process of pyopencl depends on numpy, pybind11 and mako, and for that reason pip fails.
OTOH if pyopencl provided binaries (wheels) this issue could be circumvented. There is a pull request awaiting review that should help with that: https://github.com/inducer/pyopencl/pull/264
I'm using libgpuarray (openCL) but can't seem to get the GPU working with Theano in anaconda 2. When I try to run the test I get:
ERROR (theano.gpuarray): pygpu was configured but could not be
imported Traceback (most recent call last): File
"C:\Users\username\Anaconda2\lib\site-packages\theano-0.9.0.dev1-py2.7.egg\theano\gpuarray__init__.py",
line 21, in import pygpu File
"C:\Users\username\Anaconda2\lib\site-packages\pygpu-0.2.1-py2.7-win-amd64.egg\pygpu__init__.py",
line 7, in from . import gpuarray, elemwise, reduction ImportError:
DLL load failed: The specified module could not be found.
Theano works fine with the cpu. I followed this documentation. I ran the command code "python setup.py build" and "python setup.py install" on setup.py in the libgpuarray folder (after I used cmake to install libgpuarray) to get pygpu and it ran successfully. After running cmake I get the gpuarray.dll file yet this error still occurs. Do I need to do something with it or is there something else happening?
I solved this problem by copying gpuarray.dll generated in your build dir into C:\Windows\System32.
I am using anaconda, and I've built vtk with conda using
conda install vtk
The default build of vtk is without gl2ps, and I want to know how can I build (or rather rebuild) vtk with gl2ps enabled using conda.
I am actually tring to execture this example code:
from mayavi import mlab
mlab.test_plot3d()
mlab.savefig(temp.eps)
but get the error
Saving as a vector PS/EPS/PDF/TeX file using GL2PS is either not supported by your version of VTK or you have not configured VTK to work with GL2PS -- read the documentation for the vtkGL2PSExporter class.
Thank You
EDIT
I downloaded the coda-recipes
git clone https://github.com/conda/conda-recipes.git
and even if I do not edit the files to include gl2ps
BUILD START: vtk-5.10.1-py27_1
Fetching package metadata: ......
Solving package specifications: .
The following packages will be downloaded:
package | build
---------------------------|-----------------
setuptools-13.0.2 | py27_0 435 KB
The following NEW packages will be INSTALLED:
cmake: 3.0.1-0
openssl: 1.0.1k-0
pip: 6.0.8-py27_0
python: 2.7.9-1
readline: 6.2-2
setuptools: 13.0.2-py27_0
sqlite: 3.8.4.1-0
system: 5.8-1
tk: 8.5.15-0
zlib: 1.2.8-0
Fetching packages ...
setuptools-13. 100% |######################################################################################################################################################################################| Time: 0:00:00 463.16 kB/s
Extracting packages ...
[ COMPLETE ] |########################################################################################################################################################################################################| 100%
Linking packages ...
[ COMPLETE ] |########################################################################################################################################################################################################| 100%
Source cache directory is: /home/users/user/anaconda/conda-bld/src_cache
Downloading source to cache: vtk-5.10.1.tar.gz
An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:
https://github.com/conda/conda-build/issues
Include the output of the command 'conda info' in your report.
Traceback (most recent call last):
File "/home/users/user/anaconda/bin/conda-build", line 5, in <module>
sys.exit(main())
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/main_build.py", line 110, in main
args_func(args, p)
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/main_build.py", line 308, in args_func
args.func(args, p)
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/main_build.py", line 268, in execute
build.build(m, verbose=not args.quiet, post=post)
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/build.py", line 304, in build
source.provide(m.path, m.get_section('source'))
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/source.py", line 221, in provide
unpack(meta)
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/source.py", line 60, in unpack
src_path = download_to_cache(meta)
File "/home/users/user/anaconda/lib/python2.7/site-packages/conda_build/source.py", line 48, in download_to_cache
download(meta['url'], path)
KeyError: 'url'
The recipe used to build vtk is here. You should modify it to include that package and build with with conda build vtk/.
I am trying to follow this tutorial and getting an error when I do the following:
(DataVizProject) $ pip install -r requirements.txt
It gives me a big error log, the last few lines of which are :
C:\python\new-coder\dataviz\DataVizProj\build\numpy\numpy\distutils\system_info.
py:1422: UserWarning:
Lapack (http://www.netlib.org/lapack/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [lapack_src]) or by setting
the LAPACK_SRC environment variable.
warnings.warn(LapackSrcNotFoundError.__doc__)
error: Unable to find vcvarsall.bat
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in C:\python\new-coder
\dataviz\DataVizProj\build\numpy
I know it doesn't work because when I do the following steps:
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>> import matplotlib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named matplotlib
>>>
Thanks a lot!
#Hasnain, when you are using python in windows, eventually you will see this error for some packages.
You have three options when it happen (in order of relevance):
1 - Try to download a MSI file. It will install the library without any problems. To numpy specifically you can download here (http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy)
2 - You can download the whole file and try a python setup.py install and modify the packacge if it raise some errors.
3 - You can compile your own library for your operational system.
Many MSI files you can find here (http://www.lfd.uci.edu/~gohlke/pythonlibs/)
The method recommended in that tutorial works well for Unix systems. If you are on Windows you will go through a lot of trouble trying to build numpy from sources with pip. I will save you some time, follow the official recomendation and try some binary install of the recommended ones in the official Scipy website. I personally recommend you the Anaconda or the Enthought distribution.
Usually when installing packages on windows, by default python searches for Visual Studio 2008 . You can either install it or use MinGW Compiler.
If you decide to use MinGW you should edit your distutils.cfg file in Python27\Lib\distutils directory :
[build]
compiler = mingw32
[build_ext]
compiler = mingw32