I have a python project in which I have to execute this command every now and then.
python setup.py build_clib
followed by
python setup.py build_ext --inplace
I want this to happen with just a single line code of python setup.py build but that is not possible because I want my .so file inplace.
Is there any possible way of doing this?
You need to add these lines to your setup.cfg file.
[build_ext]
inplace=1
Related
I am trying to package a Python 3.6 project which additionally requires compiling CPP code. My test setup.py file currently looks like this:
print("Hello world from setup.py")
from distutils.command.build import build
from setuptools import setup
class TestBuild(build):
def run(self):
print("Hello world from TestBuild")
setup(
name="Test",
cmdclass={
'build': TestBuild,
},
)
The idea here is to place the CPP-compiling code inside TestBuild. When I run python setup.py build it seems to work well:
> python setup.py build
Hello world from setup.py
running build
Hello world from TestBuild
However, running python setup.py install does not seem to run the build step at all:
> python setup.py install
Hello world from setup.py
running install
running bdist_egg
running egg_info
writing Test.egg-info/PKG-INFO
writing top-level names to Test.egg-info/top_level.txt
writing dependency_links to Test.egg-info/dependency_links.txt
reading manifest file 'Test.egg-info/SOURCES.txt'
writing manifest file 'Test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying Test.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/Test-0.0.0-py3.5.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing Test-0.0.0-py3.5.egg
My questions:
Does the install command automatically invokes the build command?
If it does, what did I do wrong? If it does not, how can I ensure the build command runs?
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
Does the install command automatically invokes the build command?
Yes but it doesn't run python setup.py build — it calls the
corresponding code right inside one process hence you never see the
second print. If you really need to print it you have to override setuptools commands with custom classes; create such classes for build, bdist_egg and install commands.
Where can I find detailed description of the install process? I tried to dig inside the setuptools documentation and did not find anything like this.
I'm afraid distutils and setuptools source code is the only authoritative reference.
bdist_egg doesn't run build directly; it runs egg_info, build_clib and install_lib. install_lib calls distutils.build which runs build_py and build_ext. So it seems you have to override build_py, not build.
TL;DR: Running python setup.py develop --uninstall from a pypy environment created using tox results in an exception: error: [Error 32] The process cannot access the file because it is being used by another process: c:\users\shach\code\pydocstyle\.tox\pypy\site-packages\funniest.egg-link.
Hello folks,
I have a set of integration tests for my python package that do the following:
Call python setup.py develop (using subprocess.check_call)
Run all the tests for the package
Call python setup.py develop --uninstall (again, using subprocess.check_call)
The tests are being run by tox. Everything works fine on python versions 27, 33, 34, 35 and 36, but the code fails on pypy.
I'm not going to include the setup.py and project file here, you can assume they're fine. I've reproduced this using the minimal package shown here and it works (fails?) all the same.
To reproduce, I created a python script that runs the following:
import shlex
import subprocess
subprocess.check_call(shlex.split('python setup.py develop'))
print('----------')
subprocess.check_call(shlex.split('python setup.py develop --uninstall'))
Running the file using a regular pypy installed on the system works just fine:
C:\Users\shach\code\bla\funniest>pypy test.py
running develop
running egg_info
writing funniest.egg-info\PKG-INFO
writing dependency_links to funniest.egg-info\dependency_links.txt
writing top-level names to funniest.egg-info\top_level.txt
reading manifest file 'funniest.egg-info\SOURCES.txt'
writing manifest file 'funniest.egg-info\SOURCES.txt'
running build_ext
Creating c:\python\pypy\site-packages\funniest.egg-link (link to .)
Adding funniest 0.1 to easy-install.pth file
Installed c:\users\shach\code\bla\funniest
Processing dependencies for funniest==0.1
Finished processing dependencies for funniest==0.1
----------
running develop
Removing c:\python\pypy\site-packages\funniest.egg-link (link to .)
Removing funniest 0.1 from easy-install.pth file
C:\Users\shach\code\bla\funniest>
But when I run it from a tox environment for pypy:
C:\Users\shach\code\bla\funniest>tox
GLOB sdist-make: C:\Users\shach\code\bla\funniest\setup.py
pypy inst-nodeps: C:\Users\shach\code\bla\funniest\.tox\dist\funniest-0.1.zip
pypy installed: cffi==1.10.1,funniest==0.1,greenlet==0.4.12,readline==6.2.4.1
pypy runtests: PYTHONHASHSEED='122'
pypy runtests: commands[0] | python test.py
running develop
running egg_info
writing funniest.egg-info\PKG-INFO
writing dependency_links to funniest.egg-info\dependency_links.txt
writing top-level names to funniest.egg-info\top_level.txt
reading manifest file 'funniest.egg-info\SOURCES.txt'
writing manifest file 'funniest.egg-info\SOURCES.txt'
running build_ext
Creating c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
funniest 0.1 is already the active version in easy-install.pth
Installed c:\users\shach\code\bla\funniest
Processing dependencies for funniest==0.1
Finished processing dependencies for funniest==0.1
----------
running develop
Removing c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
error: [Error 32] The process cannot access the file because it is being used by another process: c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link
Traceback (most recent call last):
File "test.py", line 6, in <module>
subprocess.check_call(shlex.split('python setup.py develop --uninstall'))
File "C:\Python\pypy\lib-python\2.7\subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['python', 'setup.py', 'develop', '--uninstall']' returned non-zero exit status 1
ERROR: InvocationError: 'C:\\Users\\shach\\code\\bla\\funniest\\.tox\\pypy\\bin\\python.EXE test.py'
___________________________________ summary ___________________________________
ERROR: pypy: commands failed
C:\Users\shach\code\bla\funniest>
Here's the tox.ini I'm using to reproduce:
[tox]
envlist = pypy
[testenv]
commands=python test.py
I made sure that I have read/write permissions to the directory, and am kinda losing my mind here.
It works just fine on linux. Probably because you can remove a file that's being used (inodes, etc.) :^)
UPDATE 1:
I thought maybe the problem lies in that both develop and develop --uninstall are being run from the same python file, and that some resource is not being cleaned properly (perhaps an open file that locks the egg-link), so i ran it manually:
C:\Users\shach\code\bla\funniest>.tox\pypy\bin\activate.bat
(pypy) C:\Users\shach\code\bla\funniest>pypy setup.py develop
running develop
running egg_info
writing funniest.egg-info\PKG-INFO
writing dependency_links to funniest.egg-info\dependency_links.txt
writing top-level names to funniest.egg-info\top_level.txt
reading manifest file 'funniest.egg-info\SOURCES.txt'
writing manifest file 'funniest.egg-info\SOURCES.txt'
running build_ext
Creating c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
funniest 0.1 is already the active version in easy-install.pth
Installed c:\users\shach\code\bla\funniest
Processing dependencies for funniest==0.1
Finished processing dependencies for funniest==0.1
(pypy) C:\Users\shach\code\bla\funniest>pypy setup.py develop --uninstall
running develop
Removing c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
error: [Error 32] The process cannot access the file because it is being used by another process: c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link
(pypy) C:\Users\shach\code\bla\funniest>
Still fails :(
Please advise!
Shachar.
UPDATE 2:
I've tried to check if there's a process holding that file using Sysinternal's Process Explorer, and nada. I manually ran pypy setup.py develop, then checked in Process Explorer and made sure the file is not opened in any process, and then ran pypy setup.py develop --uninstall and the same error was raised.
UPDATE 3:
Still happens after closing CMD and opening a new one.
UPDATE 4:
REBOOTING THE MACHINE DOES NOT SOLVE THIS! Wha-----?
From the description it would seem that somewhere in the running of pypy setup.py develop --uninstall, a file or directory is being held open, which prevents its deletion on windows. You can try to run pypy with "-X track-resources" argument (pypy -X track-resources setup.py develop --uninstall) which should notify you if an object holding resources is deallocated before the resource is closed. Sometimes scanning for code like s - open(path).read() may be enough to find the problem, that code will leak the file descriptor until the garbage collector runs, notices the ghost object, and deletes it. The code should use a context manager; with open(path) as fid: s = fid.read()
TL;DR: Don't use setup.py directly, use pip install -e ..
It seems that the problem is in setuptools on windows. What happens is that the egg-link file is opened&closed just a couple of lines before it's being removed - and windows can't handle that.
Somehow pip does handle this case, I'll update when I have more info.
I've opened an issue in setuptools.
I have a simple Python package I've created. I copied the setup.py file from the Python docs, but when I run pip install . I get the following error:
~/Projects/wolfgang (master) $ pip install . Processing /Users/Cisplatin/Projects/wolfgang
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/wolfgang.egg-info
writing pip-egg-info/wolfgang.egg-info/PKG-INFO
writing top-level names to pip-egg-info/wolfgang.egg-info/top_level.txt
writing dependency_links to pip-egg-info/wolfgang.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/wolfgang.egg-info/SOURCES.txt'
error: package directory 'wolfgang' does not exist
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/k5/grp3jdnn5jvcd14ffvqyr9z00000gn/T/pip-pWAjtJ-build/
I've tried searching online and there seem to be some similar problems on GitHub or StackOverflow, but they all apply to something being broken in the project itself as opposed to the setup.py file. My program does not mess around with any command line arguments or anything like that, so I'm not sure why this is happening.
Here's a link to the actual setup.py file I'm using, if that helps.
There is no setup.py in your repository because you removed it.
There have to be a subdirectory wolfgang. Both __init__.py and vector.py must be in the subdirectory.
I'm trying to build a Python package for this project that includes the .json files in /fixtures and the *.html files in /templates. When I run the packaging script based off my setup.py, it says it should include those folders in the MANIFEST it generates, but the output just says it's hard-linking them and when I try to install via pip, it dies on the following:
Downloading/unpacking bluetrain
Running setup.py egg_info for package bluetrain
error: package directory 'bluetrain/fixtures' does not exist
Complete output from command python setup.py egg_info:
running egg_info
writing pip-egg-info/bluetrain.egg-info/PKG-INFO
writing top-level names to pip-egg-info/bluetrain.egg-info/top_level.txt
writing dependency_links to pip-egg-info/bluetrain.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
error: package directory 'bluetrain/fixtures' does not exist
EDIT: It looks like this was an issue with legacy crud that built up in the virtualenv I was installing succeeding versions of the package into. Trying a clean virtualenv fixed the issue.
Fixed this issue by creating a clean virtualenv and trying to build from there. It appears the constant build/ rebuild cycle in the original environment left legacy files behind that were stopping things from working as intended.
I'm building a Python extension module on Windows. The extension module is written in C++. When I call setup.py bdist then setup.py uses MinGW to compile the C++ code. Is there any way I can tell setup.py to use MSVS 2008 instead?
(Why do I want to do this? This issue is one reason.)
setup.py build --compiler=msvc
setup.py bdist_egg