We Keep Coding

pip and tox ignore full path dependencies, instead look for "best match" in pypi

This is an extension of SO setup.py ignores full path dependencies, instead looks for "best match" in pypi
I am trying to write setup.py to install a proprietary package from a .tar.gz file on an internal web site. Unfortunately for me the prop package name duplicates a public package in the public PyPI, so I need to force install of the proprietary package at a specific version. I'm building a docker image from a Debian-Buster base image, so pip, setuptools and tox are all freshly installed, the image brings python 3.8 and pip upgrades itself to version 21.2.4.
Solution 1 - dependency_links
I followed the instructions at the post linked above to put the prop package in install_requires and dependency_links. Here are the relevant lines from my setup.py:
install_requires=["requests", "proppkg==70.1.0"],
dependency_links=["https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"]
Installation is successful in Debian-Buster if I run python3 setup.py install in my package directory. I see the proprietary package get downloaded and installed.
Installation fails if I run pip3 install . also tox (version 3.24.4) fails similarly. In both cases, pip shows a message "Looking in indexes" then fails with "ERROR: Could not find a version that satisfies the requirement".
Solution 2 - PEP 508
Studying SO answer pip ignores dependency_links in setup.py which states that dependency_links is deprecated, I started over, revised setup.py to have:
install_requires=[
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
Installation is successful in Debian-Buster if I run pip3 install . in my package directory. Pip shows a message "Looking in indexes" but still downloads and installs the proprietary package successfully.
Installation fails in Debian-Buster if I run python3 setup.py install in my package directory. I see these messages:
Searching for proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0
..
Reading https://pypi.org/simple/proppkg/
..
error: Could not find suitable distribution for Requirement.parse(...).
Tox also fails in this scenario as it installs dependencies.
Really speculating now, it almost seems like there's an ordering issue. Tox invokes pip like this:
python -m pip install --exists-action w .tox/.tmp/package/1/te-0.3.5.zip
In that output I see "Collecting proppkg# https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0" as the first step. That install fails because it fails to import package requests. Then tox continues collecting other dependencies. Finally tox reports as its last step "Collecting requests" (and that succeeds). Do I have to worry about ordering of install steps?
I'm starting to think that maybe the proprietary package is broken. I verified that the prop package setup.py has requests in its install_requires entry. Not sure what else to check.
Workaround solution
My workaround is installing the proprietary package in the docker image as a separate step before I install my own package, just by running pip3 install https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz. The setup.py has the PEP508 URL in install_requires. Then pip and tox find the prop package in the pip cache, and work fine.
Please suggest what to try for the latest pip and tox, or if this is as good as it gets, thanks in advance.
Update - add setup.py
Here's a (slightly sanitized) version of my package's setup.py
from setuptools import setup, find_packages
def get_version():
"""
read version string
"""
version_globals = {}
with open("te/version.py") as fp:
exec(fp.read(), version_globals)
return version_globals['__version__']
setup(
name="te",
version=get_version(),
packages=find_packages(exclude=["tests.*", "tests"]),
author="My Name",
author_email="email#mycompany.com",
description="My Back-End Server",
entry_points={"console_scripts": [
"te-be=te.server:main"
]},
python_requires=">=3.7",
install_requires=["connexion[swagger-ui]",
"Flask",
"gevent",
"redis",
"requests",
"proppkg # https://site.mycompany.com/path/to/proppkg-70.1.0.tar.gz#egg=proppkg-70.1.0"
],
package_data={"te": ["openapi_te.yml"]},
include_package_data=True, # read MANIFEST.in
)

How can I use git repos as dependencies for my PyPi package?

I have a package which I'm pushing to PyPi and some of the depedencies are not packages, but installable git repositories. My requirements.txt looks like this
sphinx_bootstrap_theme>=0.6.5
matplotlib>=2.2.0
numpy>=1.15.0
sphinx>=1.7.5
sphinx-argparse>=0.2.2
tensorboardX
tqdm>=4.24.0
Cython>=0.28.5
# git repos
git+git://github.com/themightyoarfish/svcca-gpu.git
Accordingly, my setup.py has this content:
#!/usr/bin/env python
from distutils.core import setup
import setuptools
import os
with open('requirements.txt', mode='r') as f:
requirements = f.read()
required_pkgs, required_repos = requirements.split('# git repos')
required_pkgs = required_pkgs.split()
required_repos = required_repos.split()
with open('README.md') as f:
readme = f.read()
setup(name=...
...
packages=setuptools.find_packages('.', include=[...]),
install_requires=required_pkgs,
dependency_links=required_repos,
zip_safe=False, # don't install egg, but source
)
But running pip install <package> does not actually install the git dependency. I assume that pip doesn't actually use the setup script. It works when I run python setup.py install manually.
Edit:
I also tried removing dependency_links and just using install_requires with the repository, but when installing my repository from GitHub (the project including the above files), I'm met with
Complete output from command python setup.py egg_info:
error in ikkuna setup command: 'install_requires' must be a string or
list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+git://g'"
It has been suggested in other answers that one can put something like
git+https://github.com/themightyoarfish/svcca-gpu.git#egg=svcca
into requirements.txt, but that fails with
error in <pkg> setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'
Question: (How) Can I list git repositories as dependencies for a pip package?

Out of the 50 or so different ways to specify git dependencies for Pip, the only one that did what I intended was this one (outline in PEP 508):
svcca # git+ssh://git#github.com/themightyoarfish/svcca-gpu
This can be used in install_requires, which solves the issue of dependency_links being ignored by pip.
An amusing side-effect is that the package cannot be uploaded to PyPi with such a dependency:
HTTPError: 400 Client Error: Invalid value for requires_dist. Error: Can't have direct dependency: 'svcca # git+ssh://git#github.com/themightyoarfish/svcca-gpu' for url: https://upload.pypi.org/legacy/

According to the next post related to How to state in requirements.txt a direct github source.
You could add a package from git remote repository with the next syntax:
-e git://github.com/themightyoarfish/svcca-gpu.git
Reference:
Install a project in editable mode (i.e. setuptools “develop mode”) from a local project path or a VCS url with-e

How to install dependencies from requirements.txt in a Yocto recipe for a local Python project

What I should have:
I want my Yocto Project to build a package for my Python project with all dependencies inside. The project has to run out of box on the resulting read-only sdcard image.
It simply should install all requirements in the required version to the package.
What I tried without luck:
Calling pip in do_install():
"pip/pip3 is not found", even it's in RDEPENDS.
Anyway, I really prefer this way.
With inherit pypi:
When trying with inherit pypi, it tries to get also my local sources (my pyton project) from pypi. And I have always to copy the requirements to the recipe. This is not my preferred way.
Calling pip in pkg_postinst():
It tries to install the modules on first start and fails, because the system has no internet connection and it's a read-only system. It must run out of the box without installation on first boot time. Does its stuff to late.
Where I'll get around:
There should be no need to change anything in the recipes when something changes in requirements.txt.
Background information
I'm working with Yocto Rocko in a Linux environment.
In the Hostsystem, there is no pip installed. I want to run this one installed from RDEPENDS in the target system.
Building the Package (only this recipe) with:
bitbake myproject
Building the whole sdcard image:
bitbake myProject-image-base
The recipe:
myproject.bb (relevant lines):
RDEPENDS_${PN} = "python3 python3-pip"
APP_SOURCES_DIR := "${#os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../../../../app-sources')}"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = " \
file://${APP_SOURCES_DIR}/myProject \
...
"
inherit allarch # tried also with pypi and setuptools3 for the pypi way.
do_install() { # Line 116
install -d -m 0755 ${D}/myProject
cp -R --no-dereference --preserve=mode,links -v ${APP_SOURCES_DIR}/myProject/* ${D}/myProject/
pip3 install -r ${APP_SOURCES_DIR}/myProject/requirements.txt
# Tried also python ${APP_SOURCES_DIR}/myProject/setup.py install
}
# Tried also this, but it's no option because the data MUST be included in the Package:
# pkg_postinst_${PN}() {
# #!/bin/sh -e
# pip3 install -r /myProject/requirements.txt
# }
FILES_${PN} = "/myProject/*"
Resulting Errors:
Expected to install the listed modules from requirements.txt into the myProject package, so that the python app will run directly on the resulting readonly sdcard image.
With pip, I get:
| /*/tmp/work/*/myProject/0.1.0-r0/temp/run.do_install: 116: pip3: not found
| WARNING: exit code 127 from a shell command.
| ERROR: Function failed: do_install ...
When using pypi:
404 Not Found
ERROR: myProject-0.1.0-r0 do_fetch: Fetcher failure for URL: 'https://files.pythonhosted.org/packages/source/m/myproject/myproject-0.1.0.tar.gz'. Unable to fetch URL from any source.
=> But it should not fetch myProject, since it is already local and nowhere remote.
Any ideas? What would be the best way to reach to a ready to use sdcard image without the need to change recipes when requirements.txt changes?

You should use RDEPENDS_${PN} to take care of your dependencies for your app in the recipe.
For example, assuming your python app needs aws-iot-device-sdk-python module, you should add it to RDEPENDS in the recipe. In your case, it would be like this:
RDEPENDS_${PN} = "python3 \
python3-pip \
python3-aws-iot-device-sdk-python \
"
Here's the link showing the Python modules supported by OpenEmbedded Layer.
https://layers.openembedded.org/layerindex/branch/master/layer/meta-python/
If the modules you need are not there, you will likely need to create recipes for the modules.

My newest findings:
Yocto/bitbake seems to suppress interpreting the requirements, because this breaks automatic dependency resolving what could lead to conflicts.
Reason: The required modules from setup.py would not be stored as independent packages, but as part of my package. So, bitbake does not know about this modules what could conflict with other packages that probably requires same modules in different versions.
What was in my recipe:
MY_INSTALL_ARGS = "--root=${D} \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
do_install() {
PYTHONPATH=${PYTHON_SITEPACKAGES_DIR} \
${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${MY_INSTALL_ARGS}
}
If I execute this outside of bitbake as python3 setup.py install ${MY_INSTALL_ARGS}, all will be installed correctly, but in the recipe, no requirements are installed.
There is a parameter --no-deps, but I didn't find where it is set.
I think there could be one possibility to exploit the requirements out of setup.py:
Find out where to disable --no-deps in the openembedded/poky layer for easy_install.
Creating a separate PYTHON_SITEPACKAGES_DIR
Install this separate PYTHON_SITEPACKAGES_DIR in eg the home directory as private python modules dir.
This way, no python module would trigger a conflict.
Since I do not have the time to experiment with this, I'll define now one recipe per requirement.

You try installing pip?
Debian
apt-get install python-pip
apt-get install python3-pip
Centos
yum install python-pip

Using an extra python package index url with setup.py

Is there a way to use an extra Python package index (ala pip --extra-index-url pypi.example.org mypackage) with setup.py so that running python setup.py install can find the packages hosted on pypi.example.org?

If you're the package maintainer, and you want to host one or more dependencies for your package somewhere other than PyPi, you can use the dependency_links option of setuptools in your distribution's setup.py file. This allows you to provide an explicit location where your package can be located.
For example:
from setuptools import setup
setup(
name='somepackage',
install_requires=[
'somedep'
],
dependency_links=[
'https://pypi.example.org/pypi/somedep/'
]
# ...
)
If you host your own index server, you'll need to provide links to the pages containing the actual download links for each egg, not the page listing all of the packages (e.g. https://pypi.example.org/pypi/somedep/, not https://pypi.example.org/)

setuptools uses easy_install under the hood.
It relies on either setup.cfg or ~/.pydistutils.cfg as documented here.
Extra paths to packages can be defined in either of these files with the find_links. You can override the registry url with index_url but cannot supply an extra-index-url. Example below inspired by the docs:
[easy_install]
find_links = http://mypackages.example.com/somedir/
http://turbogears.org/download/
http://peak.telecommunity.com/dist/
index-url = https://mypi.example.com

I wanted to post a latest answer to this since both the top answers are obsolete; use of easy_install has been deprecated by setuptools.
https://setuptools.pypa.io/en/latest/deprecated/easy_install.html
Easy Install is deprecated. Do not use it. Instead use pip. If you think you need Easy Install, please reach out to the PyPA team (a ticket to pip or setuptools is fine), describing your use-case.
Please use pip moving forward. You can do one of the following:
provide --index-url flag to pip command
define index-url in pip.conf file
define PIP_INDEX_URL environment variable
https://pip.pypa.io/en/stable/topics/configuration/

The following worked for me (develop, not install):
$ python setup.py develop --index-url https://x.com/n/r/pypi-proxy/simple
Where https://x.com/n/r/pypi-proxy/simple is a local PyPI repository.

Found solution when using Dockerfile:
RUN cd flask-mongoengine-0.9.5 && \
/bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple >> setup.cfg && \
python setup.py install
Which /bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple will exists in file setup.cfg:
[easy_install]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

this worked for me
PIP_INDEX_URL=<MY CUSTOM PIP INDEX URL> pip install -e .
I use setup.py and setup.cfg

As far as I know, you cant do that.
You need to tell pip this, or by passing a parameter like you mentioned, or by setting this on the user environment.
Check my ~/.pip/pip.conf:
[global]
download_cache = ~/.cache/pip
index-url = http://user:pass#localpypiserver.com:80/simple
timeout = 300
In this case, my local pypiserver also proxies all packages from pypi.python.org, so I dont need to add a 2nd entry.

You can include --extra-index-urls in a requirements.txt file. See: http://pip.readthedocs.org/en/0.8.3/requirement-format.html

"unrecognized .svn/entries format" using buildout

Since upgrading to subversion 1.7 I get "unrecognized .svn/entries format" when running buildout. I notice there is an unresolved bug reports for both distribute and setuptools for this error and it also seems that you can use setuptools_subversion to resolve the issue.
What I can't find out how to do is install setuptools_subversion so that buildout picks it up. Please can someone help?
I've tried
downloading it and running python setup.py install
adding it to the eggs list of the [buildout] part of my buildout configuration

You need to install it at the python site-packages level; easy_install (used under the hood by buildout) needs it available before it'll install anything else.
That said, the python setup.py install stanza should have installed it just fine; check by running the following test:
$ python -m setuptools_subversion
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools_subversion.py directory
That should print the installation path of the module, like it did for me in the above example. You could try to use pip or easy_install for automatic download:
$ pip install setuptools_subversion
or
$ easy_install setuptools_subversion
You can do that in a virtualenv if you want to isolate the installation. Because this is basically a dependency for svn 1.7, installing this at the same level as the svn binary (usually system wide) is certainly acceptable and the norm.
Note that the unrecognized .svn/entries format error message will not disappear, but your buildout will otherwise succeed. The message is printed no matter what as easy_install first tries the internal .svn parser before deferring to the external plugin.
If you really, really want to verify if the plugin is installed, run the following python code:
import pkg_resources
for entrypoint in pkg_resources.iter_entry_points('setuptools.file_finders'):
print entrypoint
On my system this prints:
svn = setuptools_subversion:listfiles
svn_cvs = setuptools.command.sdist:_default_revctrl
git = setuptools_git:gitlsfiles
hg = setuptools_hg:hg_file_finder