I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?
Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.
“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils
requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support
First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.
Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.
It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.
I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname
For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.
Related
I am trying to install a package from a private repository on Git.
I am using Personal Access Token in my Git URL in order to bypass the manual authentication step. (You can read about Personal Access Tokens here)
If I add this git URL in requirements file and then use the requirements file in pip to install build it works.
requirements.txt
<package name> # git+https://<Personal Access Token>#<git server address>/<username>/<repository name>.git#<branch name>#egg=<package name>
But, if I use the same URL directly it asks for password, how do I avoid this password prompt (as mentioned below):
pip install git+https://<Personal Access Token>#<git server address>/<username>/<repository name>.git#<branch name>#egg=<package name>
This issue is not observed on all machines that i tested on. It worked on Win 10 x64 and Win 10 x86. But it didn't work on Ubuntu x64. I made sure all the 3 systems has same Python version (3.8.0) and same Pip version (19.3.1).
Use environment variables with the syntax ${VARIABLE} (POSIX format, upper case and underscores allowed) so you're not hard-coding your secrets.
Pip will replace when installing from requirements.txt.
So you can refer to a token to clone the private repo, for example:
in requirements.txt
Github
git+https://${GITHUB_TOKEN}#github.com/user/project.git#{version}
Gitlab
git+https://${GITLAB_TOKEN_USER}:${GITLAB_TOKEN}#gitlab.com/user/project.git#{version}
Bitbucket
git+https://${BITBUCKET_USER}:${BITBUCKET_APP_PASSWORD}#bitbucket.org/user/project.git#{version}
More info here:
https://docs.readthedocs.io/en/stable/guides/private-python-packages.html
Go to GitLab profile settings and generate an read access token:
Select access tokens
give it a name (you can leave expiration date empty)
give it access to read all repositories you have access
generate it
Now edit your requirement file:
pandas==1.0.5
git+https://yourgitlabuser:<generated_token>#gitlab/group/repo#hash#egg=piplib
requests==2.24.0
I just had the same issue. In the end, I could install the package as follows.
from the command line:
pip install mypackagename --no-deps --index-url https://gitlab+deploy-token-mytokenname:tokenpassword#gitlab.mycompany.com/api/v4/projects/123456789/packages/pypi/simple
by specifying it in the requirements.txt file:
(Note that the flask and flask-cors package requirements in the example below are just an example, because it may seem really weird to a reader that the other lines in the example are really content that can be written in a requirements.txt.)
flask==1.1.1
flask-cors==3.0.8
--index-url https://pypi.org/simple --extra-index-url https://gitlab+deploy-token-mytokenname:tokenpassword#gitlab.mycompany.com/api/v4/projects/123456789/packages/pypi/simple
mypackagename
Then of course run pip install -r requirements.txt.
Note that both fragments above show how to provide your password, as you asked.
I'm having serious trouble with using setup.py to pip install my package which also has dependency links. I have read this answer and this one thoroughly and none of the answers including the accepted ones help.
Here is the setup.py for the package trying to install.
Basically, it reads the requirements.txt to fill install_requires and dependency_links, most of the rest of the code is boilerplate from cookie-cutter. requirements.txt has a private github repo in it which is causing the issues. e.g. git+https://${GITHUB_OAUTH_TOKEN}#github.com/jmerkow/pripy.git#egg=pripy
When I run pip install -r requirements.txt everything works great, it installs the private repository. However, if I try to install using pip install . --process-dependency-links, I get this error:
Could not find a version that satisfies the requirement pripy (from mypackage==<sha>) (from versions: )
No matching distribution found for pripy (from mypackage==<sha>)
If I take off the #egg=xxx from the link in requirements, the private repo package is completely ignored by pip install . but not by pip install -r requirements.txt.
I have confirmed that dependency_links contains 'git+https://<actual-token>#github.com/jmerkow/pripy.git#egg=pripy' and that install_requires includes 'pripy'
How do you get setup to properly Is this a problem with the sub-package? setup.py in that repo is done pretty much the same, except there are no private links.
Ugh, this always happens. I put in all the work to the question, then I figure it out myself.
The issue is two things, first, all dependeny_links need to have a version, second to pull the version from the requirements file properly you need to do some magic on the string.
Comparing to the above setup.py, I changes the way requirements are added to the two lists (updated here). Then add the version to #egg=xxx on the link e.g.
git+https://${GITHUB_OAUTH_TOKEN}#github.com/jmerkow/pripy.git#egg=pripy-0.
Now setup.py will parse that file, take the egg version info, convert it to a pip version (basically replace the first '-' with an '==') for the install_requires, and you're good to go.
I have a requirements.txt file with the line below
Pillow==2.7.0
I download the requirements for offline mode like this and put it in vendor director
pip install --download vendor -r requirements.txt
This downloads the file Pillow-2.7.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel to my vendor directory
But when I try to deploy (push) my app I get the error below:
Collecting Pillow==2.7.0 (from -r requirements.txt (line 4))
Could not find a version that satisfies the requirement Pillow==2.7.0 (from -r requirements.txt (line 4)) (from versions: )
No matching distribution found for Pillow==2.7.0 (from -r requirements.txt (line 4))
I am deploying the app to Cloud Foundry using the Python Buildpack following this documentation http://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring
UPDATE: Another post points out how you can specify to recent versions of pip a flag not to pull binaries (which I'm assuming is the issue). If that resolves the issues then you should pick his post as solution. I'd also poke cloudfoundry just needing to point this out.
The --download option just downloads the file for the dependency, but not actually install it (that might have contributed to why the flag was deprecated, to get rid of this confusion).
pip download replaces the --download option to pip install, which is now deprecated and will be removed in pip 10.
It's the same as the newer:
pip download ....
https://pip.pypa.io/en/stable/reference/pip_download/
Now assuming your cloud provider is reading the requirements.txt and looking themselves in your vendor folder to install (a-not-so-wise-approach), you'd potentially have all kinds of issues with any binaries, symbolic links, etc... unless the environment you're deploying to matches the local environment. Part of the idea with package systems like pip and a requirements file is so different platforms can pull down the needed libraries for their specific architecture, OS, etc...
This approach has so many things could go wrong...
For example, your mac probably is using a case insensitive file system. If the cloudfoundry is using linux, that's case-sensitive. On mac, this wouldn't matter if they saved it all lower case, but might be an issue when copying the files to a linux system expecting to match a capitlized "P".
In example at http://docs.cloudfoundry.org/buildpacks/python/index.html#vendoring you can see the following comment
vendors all the pip *.tar.gz into vendor/
*.tar.gz packages are source packages whereas recent versions of pip download binary packages by default. To force pip to download only source packages use --no-binary :all: If you want to download binary packages you have to specify target platform as per this remark from pip download documentation:
pip download with the --platform, --python-version,
--implementation, and --abi options provides the ability to fetch
dependencies for an interpreter and system other than the ones that
pip is running on. --only-binary=:all: is required when using any of
these options. It is important to note that these options all default
to the current system/interpreter, and not to the most restrictive
constraints (e.g. platform any, abi none, etc). To avoid fetching
dependencies that happen to match the constraint of the current
interpreter (but not your target one), it is recommended to specify
all of these options if you are specifying one of them.
Is there a way to use an extra Python package index (ala pip --extra-index-url pypi.example.org mypackage) with setup.py so that running python setup.py install can find the packages hosted on pypi.example.org?
If you're the package maintainer, and you want to host one or more dependencies for your package somewhere other than PyPi, you can use the dependency_links option of setuptools in your distribution's setup.py file. This allows you to provide an explicit location where your package can be located.
For example:
from setuptools import setup
setup(
name='somepackage',
install_requires=[
'somedep'
],
dependency_links=[
'https://pypi.example.org/pypi/somedep/'
]
# ...
)
If you host your own index server, you'll need to provide links to the pages containing the actual download links for each egg, not the page listing all of the packages (e.g. https://pypi.example.org/pypi/somedep/, not https://pypi.example.org/)
setuptools uses easy_install under the hood.
It relies on either setup.cfg or ~/.pydistutils.cfg as documented here.
Extra paths to packages can be defined in either of these files with the find_links. You can override the registry url with index_url but cannot supply an extra-index-url. Example below inspired by the docs:
[easy_install]
find_links = http://mypackages.example.com/somedir/
http://turbogears.org/download/
http://peak.telecommunity.com/dist/
index-url = https://mypi.example.com
I wanted to post a latest answer to this since both the top answers are obsolete; use of easy_install has been deprecated by setuptools.
https://setuptools.pypa.io/en/latest/deprecated/easy_install.html
Easy Install is deprecated. Do not use it. Instead use pip. If you think you need Easy Install, please reach out to the PyPA team (a ticket to pip or setuptools is fine), describing your use-case.
Please use pip moving forward. You can do one of the following:
provide --index-url flag to pip command
define index-url in pip.conf file
define PIP_INDEX_URL environment variable
https://pip.pypa.io/en/stable/topics/configuration/
The following worked for me (develop, not install):
$ python setup.py develop --index-url https://x.com/n/r/pypi-proxy/simple
Where https://x.com/n/r/pypi-proxy/simple is a local PyPI repository.
Found solution when using Dockerfile:
RUN cd flask-mongoengine-0.9.5 && \
/bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple >> setup.cfg && \
python setup.py install
Which /bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple will exists in file setup.cfg:
[easy_install]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
this worked for me
PIP_INDEX_URL=<MY CUSTOM PIP INDEX URL> pip install -e .
I use setup.py and setup.cfg
As far as I know, you cant do that.
You need to tell pip this, or by passing a parameter like you mentioned, or by setting this on the user environment.
Check my ~/.pip/pip.conf:
[global]
download_cache = ~/.cache/pip
index-url = http://user:pass#localpypiserver.com:80/simple
timeout = 300
In this case, my local pypiserver also proxies all packages from pypi.python.org, so I dont need to add a 2nd entry.
You can include --extra-index-urls in a requirements.txt file. See: http://pip.readthedocs.org/en/0.8.3/requirement-format.html
I am having trouble configuring my pip.conf file to stipulate that PIP should look for downloads from
https://pypi.python.org/simple/ rather than http://pypi.python.org/simple/
I have a related question posted PIP Could not find any downloads that satisfy the requirement SQLAlchemy where a couple of people diagnosed what was going wrong. However I am having trouble putting in place the solution.
I opend my pip.conf file located at /home/user/.pip/pip.conf using the pip config file documentation I added the below. There is nothing else in the file.
[global]
timeout = 60
find-links = https://pypi.python.org/simple/
[install]
find-links = https://pypi.python.org/simple/
After saving an exiting there is no change. I still cannot run commands such as pip install
$ sudo pip install SQLAlchemy
Downloading/unpacking SQLAlchemy
Cannot fetch index base URL http://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement SQLAlchemy
No distributions at all found for SQLAlchemy
Storing complete log in /home/user/.pip/pip.log
My setup
Ubunto 12.04 VM
You should use index-url = https://pypi.python.org/simple/ rather than find-links in your config. This will replace the default rather than just adding another option (which is what find-links does).
From latest guide: https://pip.pypa.io/en/stable/topics/configuration/
The new default configuration file is: $HOME/.config/pip/pip.conf
Legacy per-user configuration file which is also respected:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
Update from 2011-04-04 version to latest one
pip version 1.0 is fairly old, pypi shows date 2011-04-04. Consider installing latest version.
Since 2011 there were some significant changes (security fixes, support for wheel format...)
I would first remove completely the pip installed by apt-get, incl. cleaning whatever is in directories around. And then install using get-pip.py as described on pypa.
I know, that it is often recommended following Linux distribution packages, but with pip it is simply not practical.
I think your changes are not being used.
pip install --find-links=https://pypi.python.org/simple/ SQLAlchemy works on my system.
. The log says it is checking https://pypi.python.org/simple.