We Keep Coding

Output directories for python setup.py sdist bdist_wheel

When doing
python setup.py sdist bdist_wheel
it creates build, dist, packagename.egg-info directories. I'd like to have them out of the current folder.
I tried:
--dist-dir=../dist: works with sdist but packagename.egg-info is still there
--bdist-dir=../dist: for example:
python setup.py sdist bdist_wheel --dist-dir=../dist --bdist-dir=../dist2
works and the final bdist package is in ../dist. But the current folder still gets new directories build, dist, packagename.egg-info, which I don't want.
Question: how to have everything (the output of sdist and bdist_wheel) outside of the current folder?
Of course I can write a script with mv, rm -r, etc. but I wanted to know if there exists a built-in solution.

setup.py takes a series of commands (setup.py --help shows usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]). Critically, each command is its own script with its own command line options (which is why setup.py bdist_wheel --help shows different options than setup.py sdist --help).
The subtlety here is that certain commands generate calls to other commands under the hood but aren't nice enough to pass along common flags to the generated commands. For example, bdist_wheel ends up calling build and also egg_info but it does not pass along any bdist-dir you may specify. There's no global "use such and such working directory for the whole setup.py command" because all the commands are running independently and without knowledge of each other.
In order to redirect all temp directories somewhere else you have to manually specify each command and use its temp directory flag. In my case for bdist_wheel the full invocation was:
python setup.py ^
build --build-base \path\to\working\dir ^
egg_info --egg-base \path\to\working\dir ^
bdist_wheel --dist-dir \path\to\final\output\dir
(Side note, I found that if build-base and egg-base didn't match I got a weird error about not having used relative paths.)
This was sufficient to put all temp directories outside of the source folder.
Unfortunately it's not directly obvious which temp directory is the result of which command. You can use the list of commands (setup.py --help-commands) and some guesswork to determine which command created each temp directory. Then use --help on that command to see how to change its working directory.

I tried some time again with -d, --dist-dir, --bdist-dir but I found no way to do it in one-line.
I'm afraid the shortest we could find (on Windows) is:
python setup.py sdist bdist_wheel
rmdir /s /q packagename.egg-info build ..\dist
move dist ..

Why don't you try the command:
python setup.py egg_info --egg-base /tmp sdist bdist_wheel
That will put the .egg_info folder in the tmp-folder.
At least, it will be outside of your source folder.

Run Makefile on pip install

I have some protocol buffer definitions which need to be built to Python source as part of the pip install process. I've subclassed the setuptools.command.install command in setup.py but I think it's trying to run the Makefile after the package is installed so the sources aren't recognised.
I can't find information about what happens during a pip installation. Can anyone shed any light?
setup.py:
import subprocess
import sys
from setuptools import setup
from setuptools.command.install import install
class Install(install):
"""Customized setuptools install command - builds protos on install."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
install.run(self)
setup(
name='myprotos',
version='0.0.1',
description='Protocol Buffers.',
install_requires=[],
cmdclass={
'install': Install,
}
)
Output of $ pip install -vvv .:
Processing /path/to/myprotos
Running setup.py (path:/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py) egg_info for package from file:///path/to/myprotos
Running command python setup.py egg_info
running egg_info
creating pip-egg-info/myprotos.egg-info
writing pip-egg-info/myprotos.egg-info/PKG-INFO
writing top-level names to pip-egg-info/myprotos.egg-info/top_level.txt
writing dependency_links to pip-egg-info/myprotos.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
reading manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
Source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build has version 0.0.1, which satisfies requirement myprotos==0.0.1 from file:///path/to/myprotos
Building wheels for collected packages: myprotos
Running setup.py bdist_wheel for myprotos: started
Destination directory: /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel-
Running command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel- --python-tag cp27
running bdist_wheel
running build
installing to build/bdist.macosx-10.12-x86_64/wheel
running install
# THIS IS MY MAKEFILE RUNNING
Grabbing github.com/google/protobuf...
Building Python protos...
# MAKEFILE COMPLETE
running install_egg_info
running egg_info
creating myprotos.egg-info
writing myprotos.egg-info/PKG-INFO
writing top-level names to myprotos.egg-info/top_level.txt
writing dependency_links to myprotos.egg-info/dependency_links.txt
writing manifest file 'myprotos.egg-info/SOURCES.txt'
reading manifest file 'myprotos.egg-info/SOURCES.txt'
writing manifest file 'myprotos.egg-info/SOURCES.txt'
Copying myprotos.egg-info to build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1.dist-info/WHEEL
Running setup.py bdist_wheel for myprotos: finished with status 'done'
Stored in directory: /Users/jds/Library/Caches/pip/wheels/92/0b/37/b5a50146994bc0b6774407139f01d648ba3a9b4853d2719c51
Removing source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build
Successfully built myprotos
Installing collected packages: myprotos
Found existing installation: myprotos 0.0.1
Uninstalling myprotos-0.0.1:
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/DESCRIPTION.rst
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/INSTALLER
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/METADATA
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/RECORD
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/WHEEL
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/metadata.json
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/top_level.txt
Successfully uninstalled myprotos-0.0.1
Successfully installed myprotos-0.0.1
Cleaning up...
Should my Makefile be running early in the process to generate the source files? Do the files need to be there before egg_info runs for example?
If I manually run the Makefile and then install the package then it works.
Update
Here is the structure of my project:
myprotos
├── Makefile
├── README.md
├── document.proto
├── myprotos # Generated by Makefile
│ ├── __init__.py # Generated by Makefile
│ └── proto_pb2.py # Generated by Makefile
└── setup.py
Here is the section of the Makefile which generates the Python source from Potocol Buffer definitions:
python: protoc deps
# the protoc and deps command above just downloads
# the `protoc` binary to a local bin directory
#echo "Building Python protos..."
#mkdir -p "${PYTHON_OUT}"
#touch "${PYTHON_OUT}"/__init__.py
#printf "__all__ = ['proto_pb2']" > "${PYTHON_OUT}"/__init__.py
#PATH="${LOCAL_BINARY_PATH}:$$PATH" protoc \
--proto_path="${BASE}" \
--proto_path="${GOPATH}/src/github.com/google/protobuf/src" \
--python_out="${PYTHON_OUT}/" \
${PROTOS}

OK, there are three things you need to change here:
Add Makefile and document.proto to a new file MANIFEST.in.
Makefile
document.proto
If you do that, the .zip file created by python setup.py sdist (which is also uploaded to PyPI) will contain those files.
You need to run your make command during python setup.py build, not during install. Since you are generating Python code, you will need to change the build_py command here:
import sys
import subprocess
from setuptools import setup
from setuptools.command.build_py import build_py
class Build(build_py):
"""Customized setuptools build command - builds protos on build."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
super().run()
setup(
name='buildtest',
version='1.0',
description='Python Distribution Utilities',
packages=['buildtest'],
cmdclass={
'build_py': Build,
}
)
If your Makefile generated machine code, i.e. from C or any other compiled language, you should change the build_ext command:
import sys
import subprocess
from setuptools import setup
from setuptools.command.build_ext import build_ext
class Build(build_ext):
"""Customized setuptools build command - builds protos on build."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
super().run()
setup(
name='buildtest',
version='1.0',
description='Python Distribution Utilities',
packages=['buildtest'],
has_ext_modules=lambda: True,
cmdclass={
'build_ext': Build,
}
)
Lastly, you need to tell setuptools to install the resulting package on install by defining an attribute packages in setup():
setup(
...
packages=['myprotos']
)
The reason for deciding to run build_py or build_ext lies in the situation when the two are run:
build_py is also run when creating source distributions, which have to be cross-platform. Compiled extensions are usually not cross-platform, so you cannot compile them in this step.
build_ext is only run when you are creating binary distributions, which are platform-specific. It is OK to compile to platform-specific machine code here.

Integrating setup.py with Makefile to run tests

I used to run the tests for my packages from the Makefile as a way to perform three tasks in one: setup a virtual environment, install requirements and call the testing suite with the corresponding arguments:
test: venv
env/bin/pip install -r test_requirements.txt
env/bin/nosetests --rednose --with-coverage --cover-pagacke my_module
Then I read that requirements.txt files are discouraged in favor of setup.py, so I modified the setup.py file aiming to get the same result:
setup(
...
tests_require=['nose', 'rednose', 'coverage'],
test_suite=['nose.collector'])
Now I could change the Makefile with
test: venv
coverage run --source=my_module/ setup.py test
But that requires installing the testing dependencies before running the setup.py file. I'm also not sure how to include other arguments such as rednose. What is the best way to do this?

Tox is good and all, but here's how to do it without having to install any other package beforehand.
List the testing dependencies as setup_requires instead of tests_require in the setup.py file
setup(
setup_requires=['nose', 'rednose', 'coverage'],
install_requires=[] # fill in other arguments as usual
)
Optionally add test parameters to the setup.cfg file.
[nosetests]
rednose=1
detailed-errors=1
with-coverage=1
cover-package=server
cover-xml=1
verbosity=2
Run the tests with the following command
python setup.py nosetests
Source: http://nose.readthedocs.io/en/latest/api/commands.html

Pass command line arguments to nose via "python setup.py test"

Package Settings
I have built a Python package which uses nose for testing. Therefore, setup.py contains:
..
test_suite='nose.collector',
tests_require=['nose'],
..
And python setup.py test works as expected:
running test
...
----------------------------------------------------------------------
Ran 3 tests in 0.065s
OK
Running with XUnit output
Since I'm using Jenkins CI, I would like to output the nose results to JUnit XML format:
nosetests <package-name> --with-xunit --verbose
However, python setup.py test is far more elegant, and it installs the test requirements without having to build a virtual environment.
Is there a way to pass the --with-xunit (or any other parameter) to nose, when calling nose via python setup.py test?

You can set nosetests option using setup.cfg
For example in you setup.cfg
[nosetests]
with-xunit=1
Further information can be found at http://nose.readthedocs.io/en/latest/api/commands.html

Nose provides its own setuptools command (nosetests) which accepts command line arguments:
python setup.py nosetests --with-xunit
More information can be found here:
http://nose.readthedocs.io/en/latest/setuptools_integration.html

Tox can't copy non-python file while installing the module

This is the tree structure of the module I'm writing the setup.py file for:
ls .
LICENSE
README.md
bin
examples
module
scratch
setup.py
tests
tox.ini
I configured my setup.py as follows:
from setuptools import setup, find_packages
setup(
name="package_name",
version="0.1",
packages=find_packages(),
install_requires=[
# [...]
],
extras_require={
# [...]
},
tests_require={
'pytest',
'doctest'
},
scripts=['bin/bootstrap'],
data_files=[
('license', ['LICENSE']),
],
# [...]
# could also include long_description, download_url, classifiers, etc.
)
If I install the package from my python environment (also a virtualenv)
pip install .
the LICENSE file gets correctly installed.
But running tox:
[tox]
envlist = py27, py35
[testenv]
deps =
pytest
git+https://github.com/djc/couchdb-python
docopt
commands = py.test \
{posargs}
I get this error:
running install_data
creating build/bdist.macosx-10.11-x86_64/wheel/leafline-0.1.data
creating build/bdist.macosx-10.11-x86_64/wheel/leafline-0.1.data/data
creating build/bdist.macosx-10.11-x86_64/wheel/leafline-0.1.data/data/license
error: can't copy 'LICENSE': doesn't exist or not a regular file
Removing the data_files part from the setup.py makes tox running correctly.

Your issue here is that setuptools is not able to find the 'LICENSE' file in the files that have been included for building the source distribution. You have 2 options, to tell setuptools to include that file (both have been pointed to here):
Add a MANIFEST.in file (like https://github.com/pypa/sampleproject/)
Use include_package_data=True in your setup.py file.
Using MANIFEST.in is often simpler and easier to verify due to https://pypi.org/project/check-manifest/, making it possible to use automation to verify that things are indeed correct (if you use a VCS like Git or SVN).
pip install . builds a wheel using python setup.py bdist_wheel which is installed by simply unpacking it appropriately, as defined in the Wheel Specification: https://www.python.org/dev/peps/pep-0427/
tox builds a source distribution using python setup.py sdist, which is then unpacked and installed using python setup.py install.
That might be a reason for the difference in behavior for you.

I have some resource files inside my packages which I use during the execution. To make setup store them in a package with python code, I use include_package_data=True and I access them using importlib.resources. You can use backport for an older Python version than 3.7 or another library.
Before each release I have a script which verifies, that all files I need are placed inside a bdist wheel to be sure that everything is on the place.