We Keep Coding

Running pip install twice to see changes ("developer mode") -- second install fails but first works

I am wondering how to use pip to develop a Python package which is going through many revisions rapidly. My work flow is to write C++ code, compile and install with pip install and test my code.
Then, I would like to change some underlying C++ code, recompile and reinstall with pip, test the new feature, change something else, go back etc. until my package is ready.
Why did pip install ./cmake_example work well the first time but when making changes to the code, reinstalling with recompiling produced an error? I just re-ran the command pip install ./cmake_example.
I changed a single line of C++ code in an innocuous way (adding +1 in the 'add' function just to see if I can change code and recompile) and the code compiled fine in my IDE without pip.
My basic idea was to use pip following this method to avoid having to hackishly insert my shared object into some python directory each time I make a change.
I used the cmake_example from pybind here and followed the steps and did pip install ./cmake_example and it worked very well. I ran the example fine in a Python console.
Then, I changed some code (just added +1 to the adding function), so nothing substantial and wanted to re-install the package.
I then got this error:
Building wheels for collected packages: cmake-example
Building wheel for cmake-example (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for cmake-example (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [65 lines of output]
running bdist_wheel
running build
running build_ext
CMake Error at CMakeLists.txt:2 (project):
Running
'/tmp/pip-build-env-q_8_pyjk/overlay/bin/ninja' '--version'
failed with:
No such file or directory
and
ERROR: Could not build wheels for cmake-example, which is required to
install pyproject.toml-based projects
I have tried pip uninstall cmake_example and then reinstall but to no avail. The upgrade function did not work either. Does pip change something in the project folder?
PS: The makers of pybind11 also provide a scikit example here where removing the cache for repeated builds is not necessary. The cmake_example seems to be intended for legacy projects that do not use the cmake extension scikit. So if the structure of the cmake_example is essential to the project, removal of the cache is the only way to go.

I found that deleting the build directory inside the cmake_example directory resolved the problem and pip install ./cmake_example worked again as it did the first time. You can combine the two commands:
rm -rf ./cmake_example/build && pip install ./cmake_example
Looking a little closer, (for me) it was sufficient to delete cmake_example/temp.linux-x86_64-3.8/CMakeCache.txt. I suspect the lines
//Program used to build from build.ninja files.
CMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-yqopewjn/overlay/bin/ninja
inside CMakeCache.txt mean that cmake caches the path to the ninja executable, but the second time around the temporary path is different, so it can no longer be found where cmake expects it.

Packaing a Wheel within a Wheel

Context
I have a wheel file of a Python CLI app (call it A) in an S3 bucket. Not the repo, not the source code, just the wheel, because "reasons". I also have access to the source code of a wrapper for A (call it B).
What's Needed
I would like to create a wheel file that will ideally install both A and B along with other dependencies available via PyPI, and distribute said wheel to multiple people many of whom may not (and need not) have access to the S3 bucket. I'm wondering if it's possible to package A within B's wheel such that when someone pip installs B.whl, A is picked up automatically.
What I have tried
I tried including a reference to A in B's setup.py under install_requires and the relative path to A (./deps/A.whl) under dependency_links, but that didn't work. The error I get is that pip could not find a version that satisfies the requirement of package A. I did not already know whether that would work or not for certain; just tried using the path instead of a URL.
Build command: python setup.py bdist_wheel

How do you build a wheel outside of a repo containing the package?

Question
Is there a way to build a wheel for a package while in a different repository such that the wheel has been built exactly as it would be if you built the wheel inside of the repository containing the package?
Example
Consider the following repo:
/repo-containing-your-package
|___ your_module/
|___ setup.py
Build method A
When I run python setup.py bdist_wheel from within repo-containing-your-package it builds the wheel as expected, including your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), I can run python -m your_module.foo from the command line.
When the package is building, I get output that verifies that my module has been picked up by the wheel:
creating 'dist/your_module-#.#.#-py3-none-any.whl' and adding 'build/bar' to it
adding 'your_module/__init__.py'
etc...
Build method B
However, if I run python ../repo-containing-your-package/setup.py bdist_wheel from a repository that is a sibling to repo-containing-your-package, it does not build the wheel as expected, as it fails to include your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), attempting python -m your_module.foo fails:
Error while finding module specification for 'your_module.foo' (ModuleNotFoundError: No module named 'your_module')
The fact that the module has not been properly installed with the package is confirmed by reviewing the build output, which does not include the adding 'your_module' output that method A includes.

Two solutions I know of:
change working directory in setup.py
If you can modify the setup script, you can change the working directory programmatically. Add an os.chdir call early enough in the setup script:
import os
from setuptools import setup
os.chdir(os.path.dirname(__file__))
setup(...)
You can also change the working directory with other means without having to modify the setup script, e.g. in bash:
$ pushd path/to/repo; python setup.py bdist_wheel; popd
Use pip wheel
pip has a subcommand wheel that builds a wheel from the given arg; this arg is usually the name of the package, but can be a directory containing the setup script. Pass -e in that case so the wheel has the correct name:
$ pip wheel -e path/to/repo

pip does not install my package dependencies

I have developed a python package on github that I released on PyPi. It installs with pip install PACKAGENAME, but does not do anything with the dependencies that are stated in the "install_requires" of the setup.py file.
Weirdly enough, the zip file of the associated release does install all dependencies.. I tried with different virtual environments and on different computers but it never installs the dependencies.. Any help appreciated.

pip install pythutils downloads a wheel if it's available — and it's available for your package.
When generating a wheel setuptools runs python setup.py locally but doesn't include setup.py into the wheel. Download your wheel file and unzip it (it's just a zip archive) — there is your main package directory pythutils and a directory with metadata pythutils-1.1.1.dist-info. In the metadata directory there is a file METADATA that usually lists static dependencies but your file doesn't list any. Because when you were generating wheels all your dependencies has already been installed so all your dynamic code paths were skipped.
The archive that you downloaded from Github release install dependencies because it's not a wheel so pip runs python setup.py install and your dynamic dependencies work.
What you can do? My advice is to avoid dynamic dependencies. Declare static dependencies and allow pip to decide what versions to install:
install_requires=[
'numpy==1.16.5; python_version>="2" and python_version<"3"',
'numpy; python_version>="3"',
],
Another approach would be to create version-specific wheel files — one for Python 2 and another for Python 3 — with fixed dependencies.
Yet another approach is to not publish wheels at all and only publish sdist (source distribution). Then pip is forced to run python setup.py install on the target machine. That not the best approach and it certainly will be problematic for packages with C extensions (user must have a compiler and developer tools to install from sources).

Your setup.py does a series of checks like
try:
import numpy
except ImportError:
if sys.version_info[0] == 2:
install_requires.append('numpy==1.16.5')
if sys.version_info[0] == 3:
install_requires.append("numpy")
Presumably the system where you ran it had all the required modules already installed, and so ended up with an empty list in install_requires. But this is the wrong way to do it anyway; you should simply make a static list (or two static lists, one each for Python 2 and Python 3 if you really want to support both in the same package).

Can Python setuptools install files outside dist-packages?

Is it actually possible to install data files from a sdist to arbitrary locations outside the sys.prefix directory using setuptools?
The documentation implies that it is possible to do so, using an absolute path in the data_files directory part, but in practice the leading slash seems to be ignored.
For instance with data_files=[('/opt/foo/bar', [...])], the files should be installed into the directory /opt/foo/bar. But they all end up in /usr/local/lib/python3.4/dist-packages/opt/foo/bar, which is no use to man nor beast.
I suspect that it used to work - has it been changed/broken recently?
(Using pip 8.1.1, Python 3.4, setuptools 20.9.0)

Best bet would be to override install with your own cmdclass and move the files yourself.

Ah, it's all pip's fault. Given a sdist, pip helpfully creates a wheel before installing that. But wheels can't install files outside dist-packages.
Luckily you can knock some sense into pip by chucking an error, causing it to fall back to installing the sdist you wanted in the first place.
# in setup.py
if 'bdist_wheel' in sys.argv:
raise RuntimeError("This setup.py does not support wheels")
(Thanks to Benjamin Bach)