Is there a way to explicitly force the compiler for building Cython extensions when running python setup.py install? Where setup.py is of the form:
import os.path
import numpy as np
from setuptools import setup, find_packages, Extension
from Cython.Distutils import build_ext
setup(name='test',
packages=find_packages(),
cmdclass={'build_ext': build_ext},
ext_modules = [ Extension("test.func", ["test/func.pyx"]) ],
include_dirs=[np.get_include()]
)
I'm trying to install a package on Windows 8.1 x64 using Anaconda 3.16, Python 3.4, setuptools 18, NumPy 1.9 and Cython 0.24. The deployment script is adapted from the Cython wiki and this Stack Overflow answer:
Makefile.bat
:: create and activate a virtual environement with conda
conda create --yes -n test_env cython setuptools=18 pywin32 libpython numpy=1.9 python=3
call activate test_env
:: activate the MS SDK compiler as explained in the Cython wiki
cd C:\Program Files\Microsoft SDKs\Windows\v7.1\
set MSSdk=1
set DISTUTILS_USE_SDK=1
#call .\Bin\SetEnv /x64 /release
cd C:\test
python setup.py install
The problem is that in this case setup.py install still used the MinGW compiler included with conda instead of the MS Windows SDK 7.1 one.
So the DISTUTILS_USE_SDK=1 and MSSdk=1 don't seem to have an impact on the build. I'm not sure if activating the MS SDK from within a conda virtualenv might be an issue here.
Running python setup.py build_ext --compiler=msvc correctly builds the extension with the MS compiler, but subsequently running the setup.py install recompiles it with MinGW again. Same applies to python setup.py build --compiler=msvc.
Also tried running %COMSPEC% /E:ON /V:ON /K "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" as discussed in the answer linked above, but for me this produces a new terminal prompt, colored in yellow, and stops the install process.
Is there a way of forcing the compiler for building this package, for instance, by editing the setup.py?
You can provide (default) command line arguments for distutils in a separate file called setup.cfg (placed parallel to your setup.py). See the docs for more information. To set the compiler use something like:
[build]
compiler=msvc
Now calling python setup.py build is equivalent to calling python setup.py build --compiler=msvc. (You can still direct distutils to use an other complier by calling python setup.py build --compiler=someothercompiler)
Now you have (successfully directed distutils to use a msvc compiler. Unfortunately there is no option to tell it which msvc compiler to use. Basically there are two options:
One: Do nothing and distutils will try to locate vcvarsall.bat and use that to setup an environment. vcvarsall.bat (and the compiler it sets the environment up for) are part of Visual Studio, so you have to have installed that for it to work.
Two: Install the Windows SDK and tell distutils to use that. Be aware that the name DISUTILS_USE_SDK is rather missleading (at least in my opinion). It does NOT in fact tell distutils to use the SDK (and it's setenv.bat) to setup an environment, rather it means that distutils should assume the environment has already been set up. That is why you have to use some kind of Makefile.bat as you have shown in the OP.
Side Note: The specific version of VisualStudio or the Windows SDK depends on the targeted python version.
As a remark: on linux, you can use many of the autoconf environment variables. For the compiler
CC=mpicc python setup.py build_ext -i
Well I found a trick in my case : I wanted to use MSVC14.0 (from buildtools 2015) and NOT MSVC14.1 (buildtools 2017). I edited the file Lib\distutils_msvccompiler.py. There is a method
_find_vcvarsall
which is calling
best_version, best_dir = _find_vc2017()
I replaced this call by
best_version, best_dir = _find_vc2015()
Do not forget to undo this dirty trick once compiled.
I ended up putting this at the beginning of setup.py to force visual2015
useful when running in a non bat environment and starting vcvarsall from outside is not an option
if sys.platform == 'win32':
# patch env with vcvarsall.bat from vs2015 (vc14)
try:
cmd = '"{}..\\..\\VC\\vcvarsall.bat" x86_amd64 >nul 2>&1 && set'.format(environ['VS140COMNTOOLS'])
out = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True)
except:
print("Error executing {}".format(cmd))
raise
for key, _, value in (line.partition('=') for line in out.splitlines()):
if key and value:
os.environ[key] = value
# inform setuptools that the env is already set
os.environ['DISTUTILS_USE_SDK'] = '1'
There is a --skip-build option of install command which will help you with this. You can also specify multiple commands in a row, like so:
python setup.py build --compiler=msvc install --skip-build
Here is a list of supported compiler types (as returned by setup.py build_ext --help-compiler):
--compiler=bcpp Borland C++ Compiler
--compiler=cygwin Cygwin port of GNU C Compiler for Win32
--compiler=mingw32 Mingw32 port of GNU C Compiler for Win32
--compiler=msvc Microsoft Visual C++
--compiler=unix standard UNIX-style compiler
Note: To prevent compatibility issues extension modules should be built with the same compiler as the Python interpreter itself.
Related
I've read all of the other questions on this error and frustratingly enough, none give a solution that works.
If I run pip install sentencepiece in the cmd line, it gives me the following output.
src/sentencepiece/sentencepiece_wrap.cxx(2809): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
[end of output]
I'm running python 3.10.1 and pip 22.0.3 .
*I have the following Microsoft Visual C++ programs on my windows machine,which I've just done a fresh install of as it was complaining of not having a particular C++ program.
MS VC++
I've even added the .exe file to my PATH variables but still I get the same error.
Am I missing a particular Microsoft program on my pc?
I haven't seen this problem in Windows, but for Linux, I would normally reinstall Python after installing the dependencies (such as the MSVC thing). In that case this is especially helpful because I'm often rebuilding (compiling and other related steps) Python/Pip.
Could also just be an error specific to the module and Python version combo you're trying.
From a discussion in the comments:
I have the pyenv-win version manager, so I was able to create venvs and test this for you. With Python 3.10.2, it fails; with Python 3.8.10, it's successful. So, yes, reinstalling does seem to be worthy of your time.
With python3.10
On Windows: First, install vcpkg and install sentencepiece:x64-windows-static
copy the header and lib files from vcpkg/installed/x64-windows-static/include and lib to
C:/Program Files/python310/build/root/include and lib
This should work as the setup.py install expects the library in a ..\build\root\lib directory.
As the C:\Program Files\python310\lib is in the /LIBPATH, the resulting ..\build\root\lib will point to the sentencepiece.lib
If still encounting errors, then set the INCLUDE and LIB environment variables so that the cl.exe , which is called from pip install sentencetransformers finds them.
People having windows+python 3.10 env, here are exact steps to install it via vcpkg.
Other instructions are covered in zweistein's answer.
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install sentencepiece:x64-windows-static
This question already has an answer here:
Pybind11 doesn't work or C++ doesn't compile after upgrading to Mojave: -lstdc++ not found
(1 answer)
Closed 3 years ago.
I want to package my python binding for a c++ library using pybind11 and upload it to pip. It's easy to run Cmake and make to compile the binding project individually.
However, when I use python setup.py, Cmake throws ld: library not found for -lstdc++ on my Mac Mojave.
I've tried xcode-select --install
And open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
Both of them installed properly.
Also, I've tried sudo python setup.py sdist build and then python setup.py sdist build, amazingly it did work! But the the same error throws when I download and compile from pip install my-package.
Strangly, I used anaconda python3.6 on two Mac, it failed both, I create new conda env py3.7, It installed successfully!!! I also used /usr/local/bin/python3 (3.4) and also succeed!!!
My setup.py is like this:
import os
import sys
import pathlib
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext as build_ext_orig
class CMakeExtension(Extension):
def __init__(self, name):
# don't invoke the original build_ext for this special extension
super().__init__(name, sources=[])
class build_ext(build_ext_orig):
def run(self):
for ext in self.extensions:
self.build_cmake(ext)
super().run()
def build_cmake(self, ext):
# example of build args
build_args = ['--config', 'Release', '--', '-j4']
cwd = pathlib.Path().absolute()
build_temp = pathlib.Path(self.build_temp)
build_temp.mkdir(parents=True, exist_ok=True)
os.chdir(str(build_temp))
self.spawn(['cmake', str(cwd)])
if not self.dry_run:
self.spawn(['cmake', '--build', '.'] + build_args)
os.chdir(str(cwd))
setup(
...
install_requires=['numpy', 'pybind11'],
include_package_data=True,
ext_modules=[CMakeExtension('xxx')],
cmdclass={
'build_ext': build_ext,
}
)
which I mostly copied from https://stackoverflow.com/a/48015772/7961269
And my CMakeLists.txt is:
cmake_minimum_required(VERSION 2.8.12)
project(xxx)
set(CMAKE_BUILD_TYPE Release)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -stdlib=libc++")
find_package(pybind11 REQUIRED)
include_directories(...)
pybind11_add_module(xxx xxx/xxx.cpp)
Hope I can pip install my-package successfully without any errors
The error suggests that something in the python build is automatically adding -lstdc++ to the compiler flags, but because you've told clang to use -stdlib=libc++ it isn't looking in the directory that contains libstdc++ (which is correct, because you told it not to use libstdc++).
I think the python build system is not compatible with libc++, or you need to find some way to tell it to stop adding -lstdc++ automatically.
Before I get to far into this, I should note that I have seen a very similar question, but the solution presented did not work for me. Perhaps one reason why is because that was Linux build and my current difficulty is on a Windows 7 machine. I use Cygwin to get access to the gcc (5.2.0) compiler suite.
In any event, I have been attempting to try out Stan via PyStan. I am working with an Anaconda (2.4.1 64-bit) distribution which I just updated today (Python 2.7.11). I initially tried to install PyStan via pip, but the install keeps failing due to what looks like the following error:
Cannot build msvcr library: "msvcr90d.dll" not found
Consequently, I used conda instead, which seemed to install just fine. (I should note that the conda install pushed my numpy back to an earlier version, which created conflicts with the pandas upon import. I just updated anaconda to deal with these broken dependencies.) I was also able to import PyStan without any problems. However, when I actually tried to fit a model (inside of a Jupyter Notebook), the process failed with the exception in the title.
The first thing I did was confirm that gcc was where in the referenced location (not shown in the title). Indeed it was, and it seemed to working just fine. I then tried to run the model as a script from the command line (still using Python), and it failed with the same error. When I recreated the model via the REPL, it pointed to a different location that had a .bat file referencing the (verified) compiler, and that failed as well.
I am pretty sure this is because I have Visual Studio 2012, instead of Visual Studio 2008. While it is possible for me to run parallel installations, if this code is going to be useful for others in the future, these are not reasonable hoops to jump through to make it happen. I was hoping that someone else might have a better explanation. Any info would be greatly appreciated.
Beneficial from the post at https://github.com/stan-dev/pystan/issues/306
I have met various error message, but finally, I install PyStan successfully.
My machine is also on Windows 7, x64 with Anaconda3 installed.Here are the procedures to install PyStan from the sourced codes.
Install Visual Studio 2017 & Visual Studio C++ Build Tool 2015 at http://landinghub.visualstudio.com/visual-cpp-build-tools
Update Conda
conda update conda
conda update --all
check the dependencies
pip install setuptool
conda install numpy cython matplotlib scipy pandas
Install gcc compiler components
conda install libpython
conda install -c msys2 m2w64-toolchain=5.3.0
created distutils.cfg file inside Anaconda3\Lib\distutils folder with the following:
[build]
compiler = mingw32
Download Git at https://git-scm.com/downloads
git clone --recursive https://github.com/stan-dev/pystan.git
Compile from the source code
python setup.py build --compiler=mingw32
python setup.py install
P.S. The solution for the issue: Cannot build msvcr library: "vcruntime140d.dll" not found.
Copy vcruntime140d.dll from C:\Windows\System32 to any folder, which is reachable in the path in the advanced system settings/environment variables/ system variables.
I want to install PySide using PIP package manager. But it get this error message saying it didn't find nmake. This is no surprise because I do not have MSVC installed nor do I intend to.
Installing collected packages: pyside
Running setup.py install for pyside
Removing c:\users\cnyffele\appdata\local\temp\pip_build_cnyffele\pyside\pyside_package
Python architecture is 32bit
nmake not found. Trying to initialize the MSVC env...
Searching MSVC compiler version 9.0
error: Failed to find the MSVC compiler version 9.0 on your system.
However the setup.py program could simply run make:
C:\Users\cnyffele>where make
C:\MinGW32-xy\bin\make.exe
C:\Users\cnyffele>where mingw32-make
C:\MinGW32-xy\bin\mingw32-make.exe
But for some reason, it insists that if the platform is "win32" it should use msvc without trying anything else. It does, however, accept command-line options: I could specify "make-spec" to be "mingw" (see below).
From https://github.com/PySide/pyside-setup/blob/master/setup.py
OPTION_MAKESPEC = option_value("make-spec")
...
if sys.platform == "win32":
if OPTION_MAKESPEC is None:
OPTION_MAKESPEC = "msvc"
if not OPTION_MAKESPEC in ["msvc", "mingw"]:
print("Invalid option --make-spec. Available values are %s" % (["msvc", "mingw"]))
sys.exit(1)
How can I make setyp.py use the correct make when installing with PIP? Is there a way to have PIP provide command-line options to setup.py when it runs it? If this is not possible, how can I run setup.py manually after PIP downloaded it?
PIP allows passing options to setup via the options '--global-option' and '--install-option' as described in the pip reference guide.
The solution is:
pip install --global-option="--make-spec=mingw" PySide
Some additional information:
That prior to installing PySide using pip, you have to install cmake and Qt 4.8.
Build errors prevented me from downloading and installing PySide directly via pip. I needed to download the wheel binary packages from pypi.python.org.
Using a pre-downloaded .whl package, assuming the package is located in the current working directory:
pip install --global-option="--make-spec=mingw" PySide-1.2.4-cp27-none-win32.whl
Attempting to call cffi.FFI.verify() on windows will produce this error:
distutils.errors.DistutilsPlatformError: Unable to find vcvarsall.bat
I want to use mingw to compile instead of msvc. I tried to make distutils use mingw by creating c:\Python27\Lib\distutils\distutils.cfg with
[build]
compiler = mingw32
but this doesn't seem to affect cffi, I still get vcvarsall.bat missing error.
So how can I make cffi use gcc/mingw (or in general other C compiler)?
Try to reinstall cffi, now that distutils is properly configured.
For example using easy_install
easy_install cffi
Or even rebuild & install it from source using MinGW that way :
cd cffi-src-dir
python setup.py config --compiler=mingw32 build --compiler=mingw32 install
cd ..
This will make sure that cffi is correctly setup for use with MinGW... I guess...