I have a Python package that I'm distributing with pip. I need to add some custom code to be run at install time:
from setuptools import setup
from setuptools.command.install import install
class CustomInstall(install):
def run(self):
install.run(self)
print "TEST"
setup(
...
cmdclass={'install': CustomInstall},
...)
I thought the problem might pip suppressing stdout: Custom pip install commands not running. But then I replaced print "TEST" with creating a file and writing some text, and that didn't happen either.
It appears that my custom run method is only happening when I create and upload my_package to test PyPI:
python setup.py sdist bdist_wheel upload -r https://testpypi.python.org/pypi
and not when I pip install it:
pip install -i https://testpypi.python.org/pypi my_package
Maybe I am fundamentally not understanding how pip and setuptools work, but that is the opposite of the behavior I expected.
My questions are:
How can I get my CustomInstall class to work?
and
What actually happens when you call pip install?
I've looked a the setuptools docs and the PyPI docs, and I haven't been able to figure it out. It seems like other people have had success with this: Run custom task when call `pip install`, so I'm not sure what's going wrong.
So I'm not sure how much this will help, but I recently dealt with a similar issue, and here's what I learned.
Your custom install code appears to be correct. However, there are more methods than just run that can be overridden. Another useful one is finalize_options because you can write code to dynamically change the parameters of your setup.py (example here.)
This is a very good question to ask.pip install does various things depending on various factors. From where are you installing the package? From PyPI or some other package index? How was the package distributed? Is it a binary dist (.whl) or a source dist (.gz) file? Are you installing the package via a local directory? A remote repo via a VCS URL? Pip does not necessarily use the same approach for each of these cases. I would recommend using the -vvv flag to see what exactly pip is doing. It may not be running setuptools's install command for whatever reason...do you have
packages=setuptools.find_packages(),
include_package_data=True
in your setup.py file? Without these lines, pip could be installing your package's metadata but not the package itself.
Related
Installing with pip, I can write the following requirements.txt file:
git+https://repo#branch#egg=foo&subdirectory=this/bar/foo
numpy
And successfully install the requirements file:
python3 -m pip install -r requirements.tx
However, I have co-located in the directory a setup.py script that lists:
setuptools.setup(
...
install_requires = get_lines('requirements.txt'),
...
)
And installing this submodule using pip involves pip running setup.py...which fails to handle the module link:
git+https://github.com/repo#branch#egg=foo&subdirectory=this/bar/foo
I can see a lot of ways around this, but it seems like there should be one non-ambiguous way to do this which changes as little as possible in the setup.py script.
Is there such a way?
You probably need to change the line in requirements.txt to something like:
foo # git+https://repo#branch#egg=foo&subdirectory=this/bar/foo
References:
https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers
https://www.python.org/dev/peps/pep-0440/#direct-references
Although I am not entirely sure it will work. There might be subtle differences between the notations accepted in requirements.txt files, pip directly and setuptools. In particular I do not know how well things like egg and subdirectory are supported.
Advices:
Avoid calling python setup.py install or python setup.py develop from now on, and make sure to call python -m pip install . or python -m pip install --editable . instead.
I do consider reading requirements.txt from within setup.py as a red flag (or at least yellow). The contents of install_requires of setuptools and requirements.txt usually serve different purposes.
We have multiple versions of our package: package1 and package1-unstable - similar to tensorflow and tf-nightly. These are different packages on PyPi but install the same module. This then causes issues when both of these packages are installed as they overlap and write into the same directories in the site-packages folder. When one is removed, the other package stays but most of the module code is now removed, resulting in an even worse, dysfunctional state.
What is the cleanest way to detect colliding packages?
We can hardcode that package1 and package1-unstable are mutually incompatible. We use setup.py for the installation.
My thinking was to use a wrapper class around the install command class.
class Install(install):
def run(self):
if name == "package1":
self.ensure_not_installed("package1-unstable")
else:
self.ensure_not_installed("package1")
install.run(self)
def ensure_not_installed(pkg_name):
"""Raises an error when pkg_name is installed."""
...
...
cmdclass={'install': Install},
This approach seems to work as a general direction. However, I'm unsure yet about how to list exhaustively the installed packages. I'm testing the approaches with both pip install . and python setup.py install.
A couple of approaches that I tried are:
use site.getsitepackages(), iterate through the directories and check for the existence of the given package directories (i.e. package1-{version}.dist-info or pacakge1-unstable-{version}.dist-info - this can work, but this feels hacky / manual / I'm not confident yet that it's going to work in a portable way across all OSes and python distributions
try to call pip list or pip show package1 from within setup.py - this does not seem to work when the setup script is executed via pip install . as pip is not on the import path itself
pkg_resources.working_set works with python setup.py install but not with pip install . probably for similar reasons as calling pip doesn't work: the working set contains only wheel and setuptools when calling the setup with pip install .
in the general case you can't implement this as part of setup.py as pip will build your package to a wheel, cache it, and then never invoke setup.py after that. you're probably best to have some sort of post-installation tests which are run in a different way (makefile, tox.ini, etc.)
You can disable isolated builds by either
pip install --no-build-isolation .
or
PIP_NO_BUILD_ISOLATION=0 pip install .
However, some package installs rely on being invoked in an isolated environment.
Other times, the packaging routine uses a pyproject.toml.
This would be ignored in non-isolated builds.
I want to create a python package which will be cloned from its git repo when a build runs, so I will have the source inside the build agent. I would then like to run the python package as a command line tool, the package is called environment_manager.
Initially I thought I would follow a tutorial for creating a simple setup.py although this has proved to be a lot more difficult than I thought it would be and whenever I run python setup.py install --force I am not able to use my installed package, generally either module not found or the command is not recognised when I type it.
I have found that if I simply install with pip install . then I am actually able to use the tool from the command line and it works. I don't understand what the difference is, or why this only works when doing the pip install method.
Below is the setup.py file, I cannot see what is wrong with it:
from setuptools import setup, find_packages, find_namespace_packages
import pathlib
here = pathlib.Path(__file__).parent.resolve()
# Get the long description from the README file
long_description = (here / 'README.MD').read_text(encoding='utf-8')
setup(
name='environment_manager',
version='1.0.0',
package_dir={'': 'src'},
packages=find_namespace_packages(where='src', include='environment_manager.*'),
python_requires='>=3.8, <4',
install_requires=['boto3', 'botocore', 'pyyaml'],
extras_require={
'dev': ['pre-commit', 'black', 'pylint'],
'test': ['pytest', 'pytest-mock', 'coverage'],
},
entry_points={
'console_scripts': [
'environment-manager=environment_manager.environment_controller:main',
],
}
)
My project structure looks like:
environment_manager
/src
conf/
environment_manager/
environment_controller.py
config_parser.py
command.py
test/
unit_tests.py
I thought the correct way to install and run the tool from the command line was to use setup.py and setuptools but it seems like it is a lot easier and actually works if I just install it with pip.
Is installing it with pip over setup.py correct (as both ways the package appears when I type pip list) and are there any issues with my setup.py script? The script was taken from the pypa sample project and I removed most of what I didnt need.
setup.py is a python file, which usually tells you that the module/package you are about to install has been packaged and distributed with Distutils, which is the standard for distributing Python Modules. This allows you to easily install Python packages. Often it's enough to write: $ pip install .
In other words setup.py is a packaging file while pip is a package manager, therefore you should have setup.py file to be able to install with pip.
pip is a package manager which helps install, manage, and uninstall Python packages. It searches for them on PyPI, downloads them, and then runs their setup.py script.
Since you mentioned that you can run your binary executable after a pip install, but not a setup.py install, it is likely that each of them is installing the binary to separate locations.
One thing I would check is that you are using python and pip from the same version of Python, e.g:
% python --version
Python 3.8.6
% pip --version
pip 20.1.1 from /usr/lib/python3.8/site-packages/pip (python 3.8)
If these have different Python versions listed, they are likely installing to two separate directories - one in your PATH environment variable, and one which is not.
Next, I would check pip list -v after each install method, as this should list a Location header telling you where the package has been installed.
today I attempted to remove a file after my package (a python wheel) was installed via pip with the -t --target option.
Post-install script with Python setuptools
I am subclassing install in my setup.py like this:
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
# here I am using
p = os.path.join(self.install_libbase,"myPackage/folder/removeThisPyc.pyc")
if os.path.isfile(p):
os.unlink(p)
#there is also self.install_platlib and
#self.install_purelib which seem to be used by pip distutil scheme
#Have not tested those yet
when running
python setup.py install
this works the file is removed upon install.
But through
pip install path-to-my-wheel.whl
this does not work and the file is still there.
pip install -t /target/dir path-to-my-wheel.whl
does not work either...
So question is, what is pip doing with distutils and or setuptools and how can make this work?
Another thing I noticed is that pip does not seem to be printing anything, I am printing in my setup.py in verbose mode?
Is there away to see the full output from python instead of the "pip" only stuff?
Reading educates:
http://pythonwheels.com/
2. Avoids arbitrary code execution for installation. (Avoids setup.py)
As I am using wheels and wheels wont execute the setup.py, my concept of doing this is rubbish.
https://github.com/pypa/packaging-problems/issues/64
I guess this is between deployment and installation, though I would obviously count my little change to installation...
Is there a way to avoid pyc file creation upon a pip install whl ?
I am using
install_requires = [str(ir.req) for ir in parse_requirements("requirements.txt", session=PipSession())]
with pip install .. However, this does not seem to work with a requirements.txt that looks like this:
--trusted-host blah
--extra-index-url blah2
...
(support for --trusted-host was added in pip8.0.0). The install from blah fails because it complains about it not being an untrusted host as if it never processed the first line.
HOWEVER, pip install -r requirements.txt works perfectly, so these options are correct.
This means there is something parse_requirements is not doing. My question is: how do I fix or work around this using only pip install .? I could do something horrendous like:
os.system(pip install -r requirements.txt)
setup(...
in the setup.py file.
The implicit coupling of requirements.txt and setup.py is confusing to me. Nothing in setup calls requirements.txt unless you explicitly parse requirements.txt yourself, yet requirements.txt is a very standard python convention.
EDIT: We are using tools (Cloudify and sometimes Chef) that execute a pip install .. We cannot change this. I have to get this working as a pippable package, with the --trusted-host and --extra-index-urls without using a pip.conf either. Currently we are doing the os.system trick.
There has been much written about using setup.py vrs. requirements.txt. Setup.py is for Abstract requirements. Requirements.txt is for concrete requirements. In other words, they serve different purposes. Whereas requirements.txt is for an environment, setup.py is for a package. So it doesn't make sense for a setup.py to read from a requirement.txt just like it wouldn't make sense for a deb package to read from a Chef cookbook. Abstract vrs. Concrete Requirements
Often the reason people do this is they want to support people installing their package with pip install -r requirements.txt from within a check out without needing to list their dependencies twice. That's a reasonable thing to want which is why the requirements file format has a construct that enables it, simply make a requirements.txt file that contains "." or "-e ." and pip will automatically install the project and all of it's dependencies.
EDIT:
Since pip is not a library, using the most exposed part of the program is the safest (in my opinion). An alternative to os.system is
import pip
pip.main(['install','-r','requirements.txt'])
This answer from goCards is perfectly valid and you should probably import pip from your setup.py if there's no way around pip install .. But I would like to explain what actually happens here. Here's what you need to know:
install_requires is an option only supported by setuptools, a third-party package that improves upon distutils (the standard tool used in setup.py files and distributed with Python).
By design, setuptools only accepts actual requirements in install_requires, so options such as --trusted-host cannot be sent to install_requires.
This is why you're using parse_requirements("requirements.txt", session=PipSession()). This function only yields packages. The option lines such as --trusted-host bla are not returned, but sent to a PackageFinder if you gave one to parse_requirements. But you don't want these options to be returned, because setuptools would not know what do with those!
Long story short, if you want to use pip options, you need to use pip.