In my tox.ini file, the dependencies are installed via the requirements.txt file which is also used by setup.py, as follows:
The requirements.txt file contains the acceptable range of django packages, depending on the python version installed, as follows:
Django>=1.11,<2 ; python_version == '2.7'
Django>=1.11,<3 ; python_version > '3'
For python3, I want to make sure the tests run on django 2.0 as well as the latest django 2.1+ that will be installed by default, obeying the version constraints specified in the requirements.txt file. To achieve that, I force the installation of the desired django version with commands, as follows:
[tox]
envlist = {py27,py3}-django111,py3-django{20,21}
[testenv]
deps =
-r{toxinidir}/requirements.txt
commands =
django111: pip install 'Django>=1.11,<1.12'
py3-django20: pip install 'Django>=2.0,<2.1'
py3-django21: pip install 'Django>=2.1'
pytest
Ideally I could just add to the deps variable like so:
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
But pip does not support double requirements and will throw an error even though there is no conflict in how the version constraints are specified.
The drawback of using commands to override the installation is that it needs to remove the django package version installed via requirements.txt to install the desired one. Is there a way to avoid that extra step?
One trick is to move the requirement from requirements.txt into setup.py - where it's loosely pinned so that all your django versions are possible. For example
# setup.py
from setuptools import setup, find_packages
setup(
...
install_requires=[
"Django>=1.11,<2.1",
]
)
and then use your second suggestion in tox.ini
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
... so long as the Django requirement isn't listed in requirements.txt.
This works because the pip install is split in two parts, the first from tox:deps where you specify the hard requirement, and the second from the equivalent of pip install -e . where the setup.py has the looser requirement.
Related
Using setuptools, is it possible to list another editable package as a dependency for an editable package?
I'm trying to develop a collection of packages in order to use them across different production services, one of these packages (my_pkg_1) depends on a subset of my package collection (my_pkg_2, my_pkg_x, ...), so far, I've managed to put together this pyproject.toml:
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my_pkg_1"
version = "0.0.1"
dependencies = [
"my_pkg_2 # file:///somewhere/in/mysystem/my_pkg_2"
]
which does work when/for installing my_pkg_1 in editable mode, and it does install my_pkg_2 but not in editable mode. this is what I see when I pip list:
Package Version Editable project location
--------------- ------- -------------------------
my_pkg_2 0.0.1
my_pkg_1 0.0.1 /somewhere/in/mysystem/my_pkg_1
Is what I'm trying to do even possible? if so, how?
You may install my_pkg_2 explicitly in editable mode before installing my_pkg_1:
pip install --editable /somewhere/in/mysystem/my_pkg_2
Unfortunately, It is not possible to install dependencies (and dependencies of dependencies) automatically in editable mode by installing the main package. I am curious why it is not implemented.
Alternatively, you may add the package paths to the environment variable PYTHONPATH before running code from your main package. That way, you are able to import python modules from your other packages without having to install them.
This can not be done in pyproject.toml. At least not the way you want it and in a standard way.
If I were you I would write for myself a requirements.txt file (you could also give it a different name, obviously):
# install the current project as editable
--editable .
# install `my_pk_2` as editable
--editable /somewhere/in/mysystem/my_pkg_2
And you could use it like so:
path/to/venv/bin/python -m pip install --requirement 'path/to/requirements.txt'
for when you want to work on (edit) both pieces of software at the same time in the same environment.
Alternatively you could use a "development workflow tool" (such as PDM, Hatch, Poetry, and so on), and maybe one of those would be a better fit for your expectations.
I've read a discussion where a suggestion was to use the requirements.txt inside the setup.py file to ensure the correct installation is available on multiple deployments without having to maintain both a requirements.txt and the list in setup.py.
However, when I'm trying to do an installation via pip install -e ., I get an error:
Obtaining file:///Users/myuser/Documents/myproject
Processing /home/ktietz/src/ci/alabaster_1611921544520/work
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory:
'/System/Volumes/Data/home/ktietz/src/ci/alabaster_1611921544520/work'
It looks like pip is trying to look for packages that are available on pip (alabaster) on my local machine. Why? What am I missing here? Why isn't pip looking for the required packages on the PyPi server?
I have done it before the other way around, maintaining the setup file and not the requirements file. For the requirements file, just save it as:
*
and for setup, do
from distutils.core import setup
from setuptools import find_packages
try:
from Module.version import __version__
except ModuleNotFoundError:
exec(open("Module/version.py").read())
setup(
name="Package Name",
version=__version__,
packages=find_packages(),
package_data={p: ["*"] for p in find_packages()},
url="",
license="",
install_requires=[
"numpy",
"pandas"
],
python_requires=">=3.8.0",
author="First.Last",
author_email="author#company.com",
description="Description",
)
For reference, my version.py script looks like:
__build_number__ = "_LOCAL_"
__version__ = f"1.0.{__build_number__}"
Which Jenkins is replacing the build_number with a tag
This question consists of two separate questions, for the rather philosopihc choice of how to arrange setup requirements is actually unrelated to the installation error that you are experiencing.
First about the error: It looks like the project you are trying to install depends on another library (alabaster) of which you apparently also did an editable install using pip3 install -e . that points to this directory:
/home/ktietz/src/ci/alabaster_1611921544520/work
What the error tells you is that the directory where the install is supposed to be located does not exist anymore. You should only install your project itself in editable mode, but the dependencies should be installed into a classical system directory, i. e. without the option -e.
To clean up, I would suggest that you do the following:
# clean up references to the broken editable install
pip3 uninstall alabaster
# now do a proper non-editable install
pip3 install alabaster
Concerning the question how to arrange setup requirements, you should primarily use the install_requires and extras_require options of setuptools:
# either in setup.py
setuptools.setup(
install_requires = [
'dep1>=1.2',
'dep2>=2.4.1',
]
)
# or in setup.cfg
[options]
install_requires =
dep1>=1.2
dep2>=2.4.1
[options.extras_require]
extra_deps_a =
dep3
dep4>=4.2.3
extra_deps_b =
dep5>=5.2.1
Optional requirements can be organised in groups. To include such an extra group with the install, you can do pip3 install .[extra_deps_name].
If you wish to define specific dependency environments with exact versions (e. g. for Continuous Integration), you may use requirements.txt files in addition, but the general dependency and version constraint definitions should be done in setup.cfg or setup.py.
This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi
I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.
Solution 1 - dependency_links
I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:
install_requires=["requests", "proppkg==70.1.0"],
dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]
Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.
Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".
Solution 2 - PEP 508
Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:
install_requires=[
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.
Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:
Searching for proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).
Tox also fails in this scenario as it installs dependencies.
Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:
python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip
In that output I see "Collecting proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?
I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.
Workaround solution
My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.
Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.
Update - add setup.py
Here's a (slightly sanitized) version of my package's setup.py
from setuptools import setup, find_packages
def get_version():
"""
read version string
"""
version_globals = {}
with open("te/version.py") as fp:
exec(fp.read(), version_globals)
return version_globals['__version__']
setup(
name="te",
version=get_version(),
packages=find_packages(exclude=["tests.*", "tests"]),
author="My Name",
author_email="email#mycompany.com",
description="My Back-End Server",
entry_points={"console_scripts": [
"te-be=te.server:main"
]},
python_requires=">=3.7",
install_requires=["connexion[swagger-ui]",
"Flask",
"gevent",
"redis",
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
package_data={"te": ["openapi_te.yml"]},
include_package_data=True, # read MANIFEST.in
)
Is there a way to use an extra Python package index (ala pip --extra-index-url pypi.example.org mypackage) with setup.py so that running python setup.py install can find the packages hosted on pypi.example.org?
If you're the package maintainer, and you want to host one or more dependencies for your package somewhere other than PyPi, you can use the dependency_links option of setuptools in your distribution's setup.py file. This allows you to provide an explicit location where your package can be located.
For example:
from setuptools import setup
setup(
name='somepackage',
install_requires=[
'somedep'
],
dependency_links=[
'https://pypi.example.org/pypi/somedep/'
]
# ...
)
If you host your own index server, you'll need to provide links to the pages containing the actual download links for each egg, not the page listing all of the packages (e.g. https://pypi.example.org/pypi/somedep/, not https://pypi.example.org/)
setuptools uses easy_install under the hood.
It relies on either setup.cfg or ~/.pydistutils.cfg as documented here.
Extra paths to packages can be defined in either of these files with the find_links. You can override the registry url with index_url but cannot supply an extra-index-url. Example below inspired by the docs:
[easy_install]
find_links = http://mypackages.example.com/somedir/
http://turbogears.org/download/
http://peak.telecommunity.com/dist/
index-url = https://mypi.example.com
I wanted to post a latest answer to this since both the top answers are obsolete; use of easy_install has been deprecated by setuptools.
https://setuptools.pypa.io/en/latest/deprecated/easy_install.html
Easy Install is deprecated. Do not use it. Instead use pip. If you think you need Easy Install, please reach out to the PyPA team (a ticket to pip or setuptools is fine), describing your use-case.
Please use pip moving forward. You can do one of the following:
provide --index-url flag to pip command
define index-url in pip.conf file
define PIP_INDEX_URL environment variable
https://pip.pypa.io/en/stable/topics/configuration/
The following worked for me (develop, not install):
$ python setup.py develop --index-url https://x.com/n/r/pypi-proxy/simple
Where https://x.com/n/r/pypi-proxy/simple is a local PyPI repository.
Found solution when using Dockerfile:
RUN cd flask-mongoengine-0.9.5 && \
/bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple >> setup.cfg && \
python setup.py install
Which /bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple will exists in file setup.cfg:
[easy_install]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
this worked for me
PIP_INDEX_URL=<MY CUSTOM PIP INDEX URL> pip install -e .
I use setup.py and setup.cfg
As far as I know, you cant do that.
You need to tell pip this, or by passing a parameter like you mentioned, or by setting this on the user environment.
Check my ~/.pip/pip.conf:
[global]
download_cache = ~/.cache/pip
index-url = http://user:pass#localpypiserver.com:80/simple
timeout = 300
In this case, my local pypiserver also proxies all packages from pypi.python.org, so I dont need to add a 2nd entry.
You can include --extra-index-urls in a requirements.txt file. See: http://pip.readthedocs.org/en/0.8.3/requirement-format.html
I'm trying to port a python2 package to python3 (not my own) using six so that it's compatible with both. However one of the packages listed in requirements.txt is now included in the python3 stdlib and the pypi version doesn't work in python3 so I want to conditionally exclude it. Doing this in setup.py is easy, I can just do something like:
if sys.version_info[0] == 2:
requirements += py2_requirements
else:
requirements += py3_requirements
But I would like requirements.txt to reflect the correct list too. I can't find anything on this in the pip documentation. so does anyone know how to do it, or if it is even possible?
You can use the environment markers to achieve this in requirements.txt since pip 6.0:
SomeProject==5.4; python_version < '2.7'
SomeProject; sys_platform == 'win32'
It is supported by setuptools too by declaring extra requirements in setup.py:
setup(
...
install_requires=[
'six',
'humanize',
],
extras_require={
':python_version == "2.7"': [
'ipaddress',
],
},
)
See also requirement specifiers.
And Strings for the string versions of corresponding Python commands.
You can create multiple requirements files, put those common packages in a common file, and include them in another pip requirements file with -r file_path
requirements/
base.txt
python2.txt
python3.txt
python2.txt:
-r base.txt
Django==1.4 #python2 only packages
python3.txt:
-r base.txt
Django==1.5 #python3 only packages
pip install -r requirements/python2.txt