I'm new to python packaging and distributing. I have a python app which I want to pip install, so I created a setup.py for it.
Now the setup.py has install_requires which alerts if a dependency is missing, but I'm wondering if I can/should provide a way to automatically install missing dependencies. Currently the app requires one self-developed shared package, and no external packages.
EDIT:
My setup.py:
from setuptools import setup
setup(
name="TcpMonitor",
version="1.0",
packages=["tcpmonitor"],
py_modules=["tcp_monitor"],
install_requires=[
"CommonPyLib",
],
entry_points='''
[console_scripts]
tcp_monitor_gui=tcpmonitor:main
'''
)
Pip install output:
Collecting CommonPyLib (from TcpMonitor==1.0)
Could not find a version that satisfies the requirement CommonPyLib (from TcpMonitor==1.0) (from versions: )
No matching distribution found for CommonPyLib (from TcpMonitor==1.0)
As long as the dependency is listed in the install_requires list, it will automatically check for that module and if not present it will install it, providing that the module can be installed from PyPi. If not (where that package cannot be found on PyPi) you get a Could not find a version that satisfies the requirement error.
I could not find any packages with this name on PyPi so you need to add the dependency to be installed on PyPi to be installed via pip, if you host your package elsewhere (GitHub for example) this may provide a solution: How can I make setuptools install a package that's not on PyPI?.
Running pip install commonpylib returns
Could not find a version that satisfies the requirement commonlib (from versions: )
No matching distribution found for commonpylib
So you do not have a setup script problem but a problem with finding the package on PyPi (which does not seem to exist), or at least Python does not know where you have hosted it.
The other option is to integrate that package with what you are distributing without making it a dependency (i.e. add it to the __init__.py file ect).
Related
I am trying to build python wheel following the instructions as described in the link below. I am doing this first time.
https://packaging.python.org/en/latest/tutorials/packaging-projects/
I set up the folder structure, files and all. I have added this in pyproject.toml file.
[build-system]
requires= ["setuptools>=57.4.0","wheel>=0.37.1"]
build-backend = "setuptools.build_meta"
I have installed setuptoos and wheel on my virtual environment.
When I tried to run the build command, I am getting an SSL warnings and below error.
Could not find a version that satisfies the requirement wheel>=0.37.1
Could not fetch from URL https://pypi.org/simple
Even though have installed setuptools and wheel on my virtual environment, I think it is hitting the pypi to find and download these packages.
I don't know how the build module finds the modules/packages in "requires". I am not finding a way to direct to use the already installed setuptools and wheel on my machine instead of fetching from pypi.
Even if it tries to doanload again, how can derect to use our artifactory instead of pypi.
Any help in this is greatly appreciated.
I tried all below with differenct combinations but did not work. Obviously I am missing something.
1.
I added a pip.ini in my virtual environment (Lib\site-packages\pip).
Added the index-url with our organization's artifactory url.
Added trusted-host
Also tried pip.config
I downloaded the wheels for setuptools and wheel.
Added another argument in pyproject.toml
[easy-install]
find-links = c:\wheels
Added the wheels directly in the src folder.
Thanks.
accordingly to my research the following should work:
from setuptools import setup
from setuptools import find_packages
...
REQUIRES_INSTALL = [
'spacy==2.3.2',
'tensorflow==1.14.0',
'Keras==2.2.4',
'keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib',
'en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm'
]
...
setup(
name=NAME,
version=VERSION,
description=DESCRIPTION,
install_requires=REQUIRES_INSTALL,
...
)
When building a wheel or egg, everything is fine: python setup.py bdist_wheel.
But when trying to install the package (whl or egg) with pip install -U dist/mypack-....whl.
I get:
ERROR: Could not find a version that satisfies the requirement keras-contrib (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for keras-contrib (from mypack==0.3.5)
...
ERROR: Could not find a version that satisfies the requirement en-core-web-sm (from mypack==0.3.5) (from versions: none)
ERROR: No matching distribution found for en-core-web-sm (from mypack==0.3.5)
I have tried to same via setup.cfg but still no luck.
As reference - all these dependency are working when installing them first from requirments.txt and then installing the wheel.
spacy==2.3.2
tensorflow==1.14.0
Keras==2.2.4
keras-contrib#git+https://github.com/keras-team/keras-contrib.git#egg=keras-contrib
en-core-web-sm#https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.0/en_core_web_sm-2.3.0.tar.gz#egg=en-core-web-sm
pip install -r requirements.txt
pip install -U dist/mypack-....whl
But this is not clean way, since a wheel should be self contained.
Thank you for any hint!
Environment
Python: 3.7.0
Pip: 20.2.4
setuptools: 50.3.2
Some time ago it was possible to define a single requirements.txt or similar containing both specs for PyPI packages and links to repositories and archives.
That demanded to parse the requirements.txt and split them into "requirements" and "dependencies", where "requirements" would contain definitions of PyPI packages and "dependencies" — links.
Setuptools has different args for setup() for these: install_requires and dependency_links.
And it really worked: one was able to define a requirements.txt and install a package both as python setup.py install and as pip install .. Moreover, it was possible to install just dependencies via pip install -r requirements.txt. All ways worked and allowed to have a single place to define all requirements including non-PyPI links.
However, support of dependency_links arg was dropped by pip since v19. And here's the weird part: it is not dropped by setuptools. But there's more.
As of today, pip:
Supports only install_requires.
Prefers PEP 508 notation for dependencies in definitions of packages (install_requires) and standalone requirements.txt or similar.
Aborts installation of packages, which contain links in their install_requires.
Your definitions of dependencies mix 2 notations: prefixes like keras-contrib# are from PEP 508 and #egg= parts are from setuptools links notation.
This is not an issue: pip will ignore "eggs" as names are already defined before #.
I believe the installation of the package via pip works fine, i.e.:
pip install .
However, the issues will arise if the package is installed via setuptools, i.e.:
python setup.py install
setuptools does not understand PEP 508 notation and ignores links in install_requires. As of today, to make setuptools following links, both install_requires and dependency_links have to be used, e.g.:
setup(
...
install_requires=[
...
"keras_contrib==2.0.8",
...
],
dependency_links=[
"https://github.com/keras-team/keras-contrib/tarball/master#egg=keras_contrib-2.0.8",
...
],
)
Here are several tricky points:
A single dependency is defined in 2 places: a package name in install_requires and a link in dependency_links to resolve the package dependency.
The link is not git+https://.../....git, but it's a link to an archive: https://.../tarball/....
Egg name is in snake_case, not in dash-case. While it's possible to use dash-case, this will not allow specifying the version.
Version in install_requires is delimited via == and in dependency_links — via -.
It's possible to omit the version. But the only viable use case for that is if the package is not present in PyPI and is rarely updated. If the package is present in PyPI, but an unpublished version is needed, then the version must be specified.
And here's the bummer: fixing links for setuptools will break pip, as PEP 508 does not allow to specify versions. Keeping keras-contrib==x.y.z # ... in install_requires will make pip to search for the package keras-contrib==x.y.z, where ==x.y.z is not a version, but a part of the name. At the same time, not specifying a version will make setuptools to grab the latest version available at PyPI, not at the link from dependency_links.
In your case neither keras-contrib nor en-core-web-sm are present at PyPI, so using keras_contrib#git+https://... + dependency_links without version specified might work.
Otherwise, stick to pip install . and avoid using python setup.py install if the package depends on links.
See also:
PEP 508
PEP508: why either version requirement or URL but not both?
How can I make setuptools install a package that's not on PyPI?
pip install dependency links
pip3 setup.py install_requires PEP 508 git URL for private repo
Why is dependency links in setup.py deprecated?
Changing PEP 508 URLs in setup.py doesn't reinstall the dependency
Updating remote links with new URLs for PEP508 functionality
Requirements using PEP 508 direct references ignore the URL
Suggest alternatives for --process-dependency-links
Un-deprecate --process-dependency-links until an alternative is implemented
Changes to the pip dependency resolver in 20.3 (2020)
Trivia: several issues on GitHub are still open and PEP 508 is still in Active state since 2015. Digging around source code would reveal that setuptools is a wrapper around Python's distutils. setuptools is not a part of Python's stdlib, but the docs of distutils imply stdlib docs will be removed after docs of setuptools will be updated. At the same time pip is already bundled with Python's installations as Python's module. And yet we have pipfiles, pipenv, poetry, conda, pipx, pip-tools, shiv, spack, and the rest. Looks a bit overwhelming.
Looking as solutions in the past such as
pip ignores dependency_links in setup.py, this configuration should work.
Relevant content of my setup.py
packages=find_packages(),
dependency_links=['http://github.com/koji-project/koji/tarball/master#egg=koji'],
install_requires=['jira', 'PyYAML', 'requests', 'psycopg2',
'elasticsearch', 'beanbag', 'pyzabbix', 'enum34',
'beautifulsoup4', 'pytz', 'koji'],
tests_require=['flake8', 'autopep8', 'mock'],
include_package_data=True,
cmdclass={'test': setupTestRequirements}
The only thing I can think of is that my url is invalid. I don't see why it would be since it is of version 1.14.0.
Upon running pip install . I get.
Could not find a version that satisfies the requirement koji (from MARs==0.17.10) (from versions: ) No matching distribution found for koji (from MARs==0.17.10)
Upon running python setup.py develop --user, the output doesn't mention Koji
Your configuration is correct. However the problem lies elsewhere. Take a look at the koji repo on github: the project has no setup.py committed. As long as there's no setup.py script, neither pip nor setuptools (via setup.py install/setup.py develop) won't be able to install your project because they won't be able to install koji dependency as it is no valid python package at all.
Update:
The problem with koji repo on github is that it is only a mirror of the actual dev repo located on Fedora Pagure and is not synced with the upstream. So the correct answer is to use the real development repository instead of the github mirror:
dependency_links=['git+https://pagure.io/koji.git#egg=koji-1.14.0']
Easy peasy. :-)
Original answer (obsolete, only if you want to install from kojis repo mirror on Github):
I see two ways out of this situation:
Forking
fork koji on github
write your own setup.py script or copy it somewhere (see below for more info), commit and push
adapt the URL in dependency_links in your project's setup.py.
For testing, I prepared a fork of koji with a setup script; if I use its URL instead of the upstream repo, the installation succeeds. I also tagged my own "release" with koji-1.14.0.post1 to distinct the version with the setup script from the vanilla ones. Example setup.py with the new dependency:
from setuptools import setup, find_packages
setup(
name='spam',
version='0.1',
author='nobody',
author_email='nobody#nowhere.com',
url='www.example.com',
packages=[],
dependency_links=['https://github.com/hoefling/koji/tarball/master#egg=koji-1.14.0.post1'],
install_requires=['koji==1.14.0.post1'],
)
Testing the installation with pip yields:
$ pip install . --process-dependency-links
Obtaining file:///home/hoefling/python/spam
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting koji==1.14.0.post1 (from spam==0.1)
Downloading https://github.com/hoefling/koji/tarball/master (1.4MB)
100% |████████████████████████████████| 1.4MB 759kB/s
Collecting pyOpenSSL (from koji==1.14.0.post1->spam==0.1)
Using cached pyOpenSSL-17.5.0-py2.py3-none-any.whl
Collecting pycurl (from koji==1.14.0.post1->spam==0.1)
Using cached pycurl-7.43.0.1.tar.gz
...
Installing collected packages: six, idna, asn1crypto, pycparser, cffi,
cryptography, pyOpenSSL, pycurl, python-dateutil, chardet, certifi,
urllib3, requests, pykerberos, requests-kerberos, rpm-py-installer,
koji, spam
Running setup.py install for rpm-py-installer ... done
Running setup.py install for koji ... done
Running setup.py install for spam ... done
Successfully installed asn1crypto-0.23.0 certifi-2017.11.5 cffi-1.11.2
chardet-3.0.4 cryptography-2.1.4 idna-2.6 koji-1.14.0.post1 pyOpenSSL-17.5.0
pycparser-2.18 pycurl-7.43.0.1 pykerberos-1.1.14 python-dateutil-2.6.1
requests-2.18.4 requests-kerberos-0.11.0 rpm-py-installer-0.5.0 six-1.11.0
spam-0.1 urllib3-1.22
Installed packages look good:
$ pip list
Package Version
----------------- ------------
asn1crypto 0.23.0
certifi 2017.11.5
cffi 1.11.2
chardet 3.0.4
cryptography 2.1.4
idna 2.6
koji 1.14.0.post1
pip 9.0.1
pycparser 2.18
pycurl 7.43.0.1
pykerberos 1.1.14
pyOpenSSL 17.5.0
python-dateutil 2.6.1
requests 2.18.4
requests-kerberos 0.11.0
rpm-py-installer 0.5.0
rpm-python 4.11.3
setuptools 38.2.4
six 1.11.0
spam 0.1
urllib3 1.22
wheel 0.30.0
The downside of this method is the additional overhead you get in maintaining the fork until the setup script is merged into upstream. This includes testing and eventually adapting koji's setup.py in your fork each time you want to sync the upstream updates. I would probably create a separate branch with the setup script committed there, sync the fork as usual and then rebase the branch on top of fork's master, but if you are used to another update strategy, stick to it.
Use koji package from TestPyPI
Actually, I found some koji wheels of the most recent version on TestPyPI. This is also the place where I got the setup.py for the fork above - I downloaded the source tar, unpacked it and copied the setup script. This means that the koji devs are looking into distributing the project via PyPI and are working on the setup script, but didn't commit it yet. While they are working on it, you can use the testing package index as the workaround. This way, you will not build the package from sources, taking the wheel instead that koji devs built and uploaded:
setup(
...
dependency_links=['https://testpypi.python.org/pypi/koji'],
install_requires=['koji'],
)
The downsides of this method are:
You don't know if the koji package from TestPyPI is installable at all. Even if it is, there's no guarantee that the installed code will work as intended (although it should). When you have the fork, you can always fix the setup script yourself - here you are doomed if the wheel file has errors.
Packages on TestPyPI are removed on a regular basis. From the docs:
Note: The database for TestPyPI may be periodically pruned, so it is not unusual for user accounts to be deleted.
Last note
You can of course combine the two workarounds and use both URLs in dependency_links:
setup(
...
dependency_links=[
'https://testpypi.python.org/pypi/koji',
'https://github.com/hoefling/tarball/master#egg=koji-1.14.0.post1',
],
install_requires=['koji'],
)
This way, if the package is not found on TestPyPI, it will be built from your fork.
Last note 2
You will probably need to install some additional system packages; at least for my system CentOS Linux release 7.3.1611 (Core) I had to install curl-devel to satisfy pycurl.
I am trying to package my application for distribution on PyPi using setuptools. One of our dependencies is pysam and we require version >=8.3, so in our setup.py we have:
install_requires = ['pysam>=8.4']
setup(
###lots of stuff here
install_requires=install_requires)
When every time setup.py is run, it downloads and installs pysam, and then tells me it failed to find it:
Extracting pysam-0.2.3-py2.7-linux-x86_64.egg to /usr/local/lib/python2.7/dist-packages
pysam 0.2.3 is already the active version in easy-install.pth
Installed /usr/local/lib/python2.7/dist-packages/pysam-0.2.3-py2.7-linux-x86_64.egg
error: The 'pysam>=0.8.4' distribution was not found and is required by umi-tools
I was a bit worried by the fact it was talking about pysam 0.2.3, so I checked the version in python
>>> import pysam
>>> pysam.__version__
'0.9.1.4'
So pysam 0.9.1.4 is being installed. My questions are:
How does setup tools determine if a requirement has been met, and could it be thinking pysam 0.2.3 is installed even though 0.9 is.
What else could be going on such that setuptools thinks pysam isn't install when it is?
This is a very simple problem, but I’m completely stuck on how to solve it. I couldn’t even found a way for just downloading.
The website don’t give anything for downloading it outside pypi and without smug.
$ pip install --allow-external gitlib gitlib
Collecting gitlib
Could not find a version that satisfies the requirement gitlib (from versions: )
Some insecure and unverifiable files were ignored (use --allow-unverified gitlib to allow).
No matching distribution found for gitlib
Using the required options doesn’t solve the problem.
$ pip install --allow-all-external --allow-unverified gitlib gitlib==0.5
Collecting gitlib==0.5
Could not find a version that satisfies the requirement gitlib==0.5 (from versions: )
No matching distribution found for gitlib==0.5
Pypi is only a list of registered projects -- if the package is listed at a download URL, you will need to follow that path to get the actual code to install.
In your case, you could git clone or pip install from git://mcnabbs.org/smug.git. If there's issues cloning or downloading, that's not something anyone here would be able to help you out with -- the package maintainer has decided to self-host his code.
More generally, pip interacts with pypi. Pypi is a package index for code that is not in the standard library. Much of this code is third-party/non-official. It's important to note that there is no promise of quality, functionality, or ability to install this code directly from pip.
In the case of "Gitlab" as per pypi, there's no promise that the code is actually true to its marketing of being a "Pythonic low-level Git library", there's no guarantee that the download URL is direct (or correct or even working), and it's a perfect example of the value of knowing that this information is maintained by the person that registered that package on pypi.