I'm using pbr which uses a requirements.txt file to look for dependencies.
I've a line in requirements.txt like git+ssh://git#github.com/user/repo.git and it works when I run pip install -r requirements.txt
However, when I run python setup.py build I run in to the error:
error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+ssh://g'"
There are many Stack Overflow answers that deal with this problem exclusively when using setuptools and all of them suggest putting the Git dependency into the dependency_links list in setup.py.
I would like pbr to be able to deal with my Git dependency directly from requirements.txt in a way that works when I run both python setup.py build and pip install -r requirements.txt.
Is this possible? Are there any close workarounds?
In the example you provided, pbr is propagating the whole line to install_requires, which produces an invalid line.
Provide the requirement name via #egg=name
For it to work as intended, the url needs a #egg suffix to tell pbr what requirement is provided by that URL. If the URL looks like this, pbr will scrape a requirement out of the #egg part and propagate only repo to install_requires:
git+ssh://git#github.com/user/repo.git#egg=repo
Version constraints
If a version is included, pbr will add a >= constraint on it. So this would become repo>=1.2.3 in install_requires:
git+ssh://git#github.com/user/repo.git#egg=repo-1.2.3
Dependency Links
It'll also extract a dependency_link item that contains the full URL. You can use it by passing --process-dependency-links to pip. By default, pip will return the error Could not find a version that satisfies the requirement repo unless the package is also available via PyPI. If --process-dependency-links is specified, then it'll fetch it from the Git URL instead.
Use the -e flag, or require pbr>=1.9.0
Prior to version 1.9.0, pbr only recognized http and https URLs, unless the line started with -e. It added support for git://, git+ssh://, git+https:// without -e in this commit.
Related
I am trying to run:
pip3 install -e .
in my Python project where I have the following setup.py:
from setuptools import setup
setup(
name='mypackage',
install_requires=[
"anotherpackage#git+git#bitbucket.org:myorg/anotherpackage.git"
]
)
but it fails with:
error in mypackage setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL given
I guess it is correct about the format of my URL as PEP 508 doesn't allow specifying git user name for ssh clone URLs.
What is the correct syntax for PEP 508 URLs with git+ssh protocol for install_requires dependency for private git repositories (in this case hosted on BitBucket)? What is the syntax for specifying a specific branch, tag or sha?
More context to avoid XY problem
I have an internal Python project that depends on multiple internally developed Python packages. I would like to avoid the necessity for hosting my own PIP repository in the organisation and thus I am trying to use git URLs directly. I need to use ssh protocol for git URLs as all the users have their ssh keys configured and it would be cumbersome to ask all the users to configure their app passwords in BitBuckets (I have 2FA required and the regular user password doesn't work).
I have already tried to use:
dependency_links
setup(
name='mypackage',
install_requires=[
"anotherpackage==0.0.1"
],
dependency_links=[
"git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage-0.0.1"
]
)
But they are deprecated and they are ignored by pip3 install -e .. According to documentation I've found, PEP 508 URLs should be used instead.
requirements.txt file with entries duplicated from install_requires entries
I have a requirements.txt file with:
-e git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage
and I use pip3 install -r requirements.txt instead of pip3 install -e .. It works but is suboptimal as I have to keep both setyp.py and requirements.txt in sync.
If there is any other recommended solution for my problem I would like to learn about it :)
After checking pip source code I found the correct syntax for private BitBucket repositories.
The general form for the packages with URLs is <package name>#<URI> and the URI must start with a <scheme>://.
So I fixed it to:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git
and then I was getting a different error - this time git command (invoked by pip) was complaining about repository URL ssh://git#bitbucket.org:myorg/anotherpackage.git.
I checked the git documentation for the ssh:// URLs format and found out that hostname and organisation parts must be separated with / instead of ::
ssh://git#bitbucket.org/myorg/anotherpackage.git
This URL works fine. I also learned from the pip source code that the actual revision/branch/tag can be specified by appending #<rev-spec> so I can specify for example the tag 0.0.1 with the following in install_requires:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git#0.0.1
The only issue that I still have is that when I change the revision and run pip3 install -e . again it doesn't detect the change (even when run with --upgrade). I have to manually uninstall the package (pip3 uninstall anotherpackage) and run pip3 install -e . again.
I have a package which I'm pushing to PyPi and some of the depedencies are not packages, but installable git repositories. My requirements.txt looks like this
sphinx_bootstrap_theme>=0.6.5
matplotlib>=2.2.0
numpy>=1.15.0
sphinx>=1.7.5
sphinx-argparse>=0.2.2
tensorboardX
tqdm>=4.24.0
Cython>=0.28.5
# git repos
git+git://github.com/themightyoarfish/svcca-gpu.git
Accordingly, my setup.py has this content:
#!/usr/bin/env python
from distutils.core import setup
import setuptools
import os
with open('requirements.txt', mode='r') as f:
requirements = f.read()
required_pkgs, required_repos = requirements.split('# git repos')
required_pkgs = required_pkgs.split()
required_repos = required_repos.split()
with open('README.md') as f:
readme = f.read()
setup(name=...
...
packages=setuptools.find_packages('.', include=[...]),
install_requires=required_pkgs,
dependency_links=required_repos,
zip_safe=False, # don't install egg, but source
)
But running pip install <package> does not actually install the git dependency. I assume that pip doesn't actually use the setup script. It works when I run python setup.py install manually.
Edit:
I also tried removing dependency_links and just using install_requires with the repository, but when installing my repository from GitHub (the project including the above files), I'm met with
Complete output from command python setup.py egg_info:
error in ikkuna setup command: 'install_requires' must be a string or
list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+git://g'"
It has been suggested in other answers that one can put something like
git+https://github.com/themightyoarfish/svcca-gpu.git#egg=svcca
into requirements.txt, but that fails with
error in <pkg> setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'
Question: (How) Can I list git repositories as dependencies for a pip package?
Out of the 50 or so different ways to specify git dependencies for Pip, the only one that did what I intended was this one (outline in PEP 508):
svcca # git+ssh://git#github.com/themightyoarfish/svcca-gpu
This can be used in install_requires, which solves the issue of dependency_links being ignored by pip.
An amusing side-effect is that the package cannot be uploaded to PyPi with such a dependency:
HTTPError: 400 Client Error: Invalid value for requires_dist. Error: Can't have direct dependency: 'svcca # git+ssh://git#github.com/themightyoarfish/svcca-gpu' for url: https://upload.pypi.org/legacy/
According to the next post related to How to state in requirements.txt a direct github source.
You could add a package from git remote repository with the next syntax:
-e git://github.com/themightyoarfish/svcca-gpu.git
Reference:
Install a project in editable mode (i.e. setuptools “develop mode”) from a local project path or a VCS url with-e
I'm having serious trouble with using setup.py to pip install my package which also has dependency links. I have read this answer and this one thoroughly and none of the answers including the accepted ones help.
Here is the setup.py for the package trying to install.
Basically, it reads the requirements.txt to fill install_requires and dependency_links, most of the rest of the code is boilerplate from cookie-cutter. requirements.txt has a private github repo in it which is causing the issues. e.g. git+https://${GITHUB_OAUTH_TOKEN}#github.com/jmerkow/pripy.git#egg=pripy
When I run pip install -r requirements.txt everything works great, it installs the private repository. However, if I try to install using pip install . --process-dependency-links, I get this error:
Could not find a version that satisfies the requirement pripy (from mypackage==<sha>) (from versions: )
No matching distribution found for pripy (from mypackage==<sha>)
If I take off the #egg=xxx from the link in requirements, the private repo package is completely ignored by pip install . but not by pip install -r requirements.txt.
I have confirmed that dependency_links contains 'git+https://<actual-token>#github.com/jmerkow/pripy.git#egg=pripy' and that install_requires includes 'pripy'
How do you get setup to properly Is this a problem with the sub-package? setup.py in that repo is done pretty much the same, except there are no private links.
Ugh, this always happens. I put in all the work to the question, then I figure it out myself.
The issue is two things, first, all dependeny_links need to have a version, second to pull the version from the requirements file properly you need to do some magic on the string.
Comparing to the above setup.py, I changes the way requirements are added to the two lists (updated here). Then add the version to #egg=xxx on the link e.g.
git+https://${GITHUB_OAUTH_TOKEN}#github.com/jmerkow/pripy.git#egg=pripy-0.
Now setup.py will parse that file, take the egg version info, convert it to a pip version (basically replace the first '-' with an '==') for the install_requires, and you're good to go.
I have a requirements.txt file with the line below
Pillow==2.7.0
I download the requirements for offline mode like this and put it in vendor director
pip install --download vendor -r requirements.txt
This downloads the file Pillow-2.7.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel to my vendor directory
But when I try to deploy (push) my app I get the error below:
Collecting Pillow==2.7.0 (from -r requirements.txt (line 4))
Could not find a version that satisfies the requirement Pillow==2.7.0 (from -r requirements.txt (line 4)) (from versions: )
No matching distribution found for Pillow==2.7.0 (from -r requirements.txt (line 4))
I am deploying the app to Cloud Foundry using the Python Buildpack following this documentation http://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring
UPDATE: Another post points out how you can specify to recent versions of pip a flag not to pull binaries (which I'm assuming is the issue). If that resolves the issues then you should pick his post as solution. I'd also poke cloudfoundry just needing to point this out.
The --download option just downloads the file for the dependency, but not actually install it (that might have contributed to why the flag was deprecated, to get rid of this confusion).
pip download replaces the --download option to pip install, which is now deprecated and will be removed in pip 10.
It's the same as the newer:
pip download ....
https://pip.pypa.io/en/stable/reference/pip_download/
Now assuming your cloud provider is reading the requirements.txt and looking themselves in your vendor folder to install (a-not-so-wise-approach), you'd potentially have all kinds of issues with any binaries, symbolic links, etc... unless the environment you're deploying to matches the local environment. Part of the idea with package systems like pip and a requirements file is so different platforms can pull down the needed libraries for their specific architecture, OS, etc...
This approach has so many things could go wrong...
For example, your mac probably is using a case insensitive file system. If the cloudfoundry is using linux, that's case-sensitive. On mac, this wouldn't matter if they saved it all lower case, but might be an issue when copying the files to a linux system expecting to match a capitlized "P".
In example at http://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring you can see the following comment
vendors all the pip *.tar.gz into vendor/
*.tar.gz packages are source packages whereas recent versions of pip download binary packages by default. To force pip to download only source packages use --no-binary :all: If you want to download binary packages you have to specify target platform as per this remark from pip download documentation:
pip download with the --platform, --python-version,
--implementation, and --abi options provides the ability to fetch
dependencies for an interpreter and system other than the ones that
pip is running on. --only-binary=:all: is required when using any of
these options. It is important to note that these options all default
to the current system/interpreter, and not to the most restrictive
constraints (e.g. platform any, abi none, etc). To avoid fetching
dependencies that happen to match the constraint of the current
interpreter (but not your target one), it is recommended to specify
all of these options if you are specifying one of them.
I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?
Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.
“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils
requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support
First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.
Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.
It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.
I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname
For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.