I am having trouble understanding how pip works in a specific environment. The thing is that I am trying to install OpenStack using ansible-openstack deployment method. It provides playbooks to prepare the complete environment and install all components. Deployment fails at the step when python modules should be installed from a private repo (it is being built during previous deployment steps). This repo is nothing else as nginx server exposing directories with python wheels via http protocol. When OpenStack nodes are reaching over to this repo with pip in order to install packages, they seem not to find any of the packages that satisfy their constraints (I am not posting the whole command using the requirement file, but required versions are there). Each node has similarly looking HOME/./pip/pip.conf:
[global]
no-index = true
pre = True
timeout = 120
trusted-host =
172.21.51.152
[install]
upgrade = True
find-links =
http://172.21.51.152:8181/os-releases/14.0.7/
where 172.21.51.152 is that repo that is supposed to replace PyPI.
When playbook tasks that should install packages are running they all fail with similar error:
root#control1-galera-container-434df170:~# pip install MySQL-python
Collecting MySQL-python
Could not find a version that satisfies the requirement MySQL-python (from versions: )
No matching distribution found for MySQL-python
even though the file is there and I can wget it from the URL where the file is located.
ls /var/www/repo/os-releases/14.0.7/ | grep -i mysql_python
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
I checked with tcpdump what kind of request pip is sending when "pip install" command is used. It sends a GET get to http://172.21.51.152:8181/os-releases/14.0.7/ where all wheel files are. Server replies with the index.html file:
<html>\r\n
<head><title>Index of /</title></head>\r\n
<body bgcolor="white">\r\n
<h1>Index of /</h1><hr><pre>../\r\n
links/
openstackgit/
os-releases/
pkg-cache/
pools/
venvs/
</pre><hr></body>\r\n
</html>\r\n
After getting this reply pip seems to be satisfied with the outcome and decides that the package it was looking for is not there. I am trying to figure out if that is expected behaviour and there is something wrong with that repo or if pip is being launched with insufficient configuration that doesn't allow it to perform the search.
I tried to delete the pip.conf altogether and pip worked normally just downloading files from PyPI. What am I missing here in order to make it work from the private repository?
Thanks for you help!
Resolved this question in OpenStack Operators mailing list. Was an issue of nginx server and not working autoindex. After fixing autoindes the complete directory index started to show up which enabled pip to download and install packages.
FWIW, I had the very same symptom Danil described:
root#control1-galera-container-434df170:~# pip install MySQL-python
Collecting MySQL-python Could not find a version that satisfies the requirement MySQL-python (from versions: )
No matching distribution found for MySQL-python
The root cause in my case was a (restrictive) umask setting of 077.
# root#controller1
$ umask # octal
0077
$ umask -S # symbolic
u=rwx,g=,o=
The default seems to be 022 (symbolic: u=rwx,g=rx,o=rx), details at https://en.wikipedia.org/wiki/Umask
This umask of 077 affects the nginx config files (which, for example, set the
'autoindex' behavior, which is needed to serve the python wheels):
# root#controller1-repo-container-e0445a8f
$ cd /etc/nginx/sites-available/
$ ll openstack-slushee.vhost
-rw------- 1 root root [...] openstack-slushee.vhost
If also affects the python packages:
# root#controller1-repo-container-e0445a8f
$ cd /var/www/repo/pools/ubuntu-16.04-x86_64/mysql_python/
$ ll mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
-rw------- 1 nginx root [...] mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
And these python packages get requested from the other containers in an openstack-ansible setup:
# root#controller1-galera-container-564eedea
$ cat .pip/pip.conf
[...]
[install]
upgrade = True
find-links =
http://172.29.236.11:8181/os-releases/15.1.2/ubuntu-16.04-x86_64/
And ultimately, it affects the file which specifies the python package versions:
# root#controller1-repo-container-e0445a8f
$ cd /var/www/repo/os-releases/15.1.2/ubuntu-16.04-x86_64/
$ ll requirements_absolute_requirements.txt
-rw------- 1 root root [...] requirements_absolute_requirements.txt
You will notice that nginx complains a lot in its error log:
# root#controller1-repo-container-e0445a8f
$ grep -i error /var/log/nginx/openstack-slushee.error.log | grep -i mysql
2017/07/03 10:30:34 [error] 17458\#17458: *642 open()
"/var/www/repo/os-releases/15.1.2/ubuntu-16.04-x86_64/
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl" failed
(13: Permission denied), client: 172.29.236.11,
server: openstack-slushee,
request: "GET /os-releases/15.1.2/ubuntu-16.04-x86_64/
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl HTTP/1.1",
host: "172.29.236.11:8181"
Long story short, set the umask to a less restrictive mode before starting openstack-ansible:
$ umask 0022
Side notes:
here's the discussion where I want to figure out how to handle umask in openstack-ansible: http://lists.openstack.org/pipermail/openstack-operators/2017-July/013892.html
this question might have been better suited at https://ask.openstack.org/en/questions/
Related
I'm trying to install a custom package from a different index other than pypi in my pod.
I created a pip.conf and I run this commands in my dockerfile:
ADD docker_files/pip.conf /etc/pip.conf
ENV PIP_CONFIG_FILE /etc/pip.conf
and when I run RUN echo python3 -m pip config debug I can see in the output that pip does recognise my pip conf and all of the indexes in it. Output:
env_var: PIP_CONFIG_FILE='/etc/pip.conf' env: /etc/pip.conf, exists: True global.index-url: <custom> global.extra-index-url: <more custom> global: /etc/xdg/pip/pip.conf, exists: False /etc/pip.conf, exists: True global.index-url: <custom> global.extra-index-url: <more custom> site: /usr/local/pip.conf, exists: False
but when it searches for the package and it can't find it in pypi, it doesn't search my other indexes:
Searching for <package>
Reading https://pypi.org/simple/<package>/
Couldn't find index page for <package> (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for <package>
error: Could not find suitable distribution for Requirement.parse(<package>)
The command '/bin/sh -c python setup.py install' returned a non-zero code: 1
I also read that sometimes if the path $HOME/Library/Application Support exists it overwrite everything, but I checked and it does not. I'm using python 3.8.13, I also tried copying the pip conf file to /usr/local/pip.conf and removing the var PIP_CONFIG_FILE from the dockerfile, Nothing worked.
Did I miss something?
I was working on a Python/Flask project, and I am not sure what happened but all of a sudden when I opened my Terminal, it seemed like pip was deleted; when I tried running things like 'pip list' from different directories (including my global one), I got the following
zsh: command not found: pip
Needless to say, when I tried running packages that I had previously installed with pip like Flask, I got the same zsh error as per above.
And when I reinstalled pip and ran pip list, it seems like all of my Python packages got deleted?:
pip 22.1.2
setuptools 58.1.0
wheel 0.37.1
Any ideas what could have caused this? I can reinstall the packages that is no issue. But I fear this will happen again in the future and I have no idea what I did that could have caused this...
Could this have been a $PATH issue?
When I run echo $PATH in my global directory, I get:
/Library/Frameworks/Python.framework/Versions/3.10/bin
/opt/homebrew/bin
/opt/homebrew/sbin
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Library/Apple/usr/bin
#mast o;
Here is what I get when running find / -name pip:
/Users/patrickchong/PycharmProjects/Live_Server/venv/bin/pip
/Users/patrickchong/PycharmProjects/Live_Server/venv/lib/python2.7/site-packages/pip
/Users/patrickchong/PycharmProjects/JP interview/venv/bin/pip
/Users/patrickchong/PycharmProjects/JP interview/venv/lib/python2.7/site-packages/pip
/Users/patrickchong/opt/anaconda3/bin/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py310hecd8cb5_0/bin/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py310hecd8cb5_0/lib/python3.10/site-packages/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py39hecd8cb5_0/bin/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py39hecd8cb5_0/lib/python3.9/site-packages/pip
/Users/patrickchong/opt/anaconda3/lib/python3.9/site-packages/pip
find: /Users/patrickchong/.Trash: Operation not permitted
/Users/patrickchong/.conda/envs/Crypto Trading Bot project/bin/pip
/Users/patrickchong/.conda/envs/Crypto Trading Bot project/lib/python3.10/site-packages/pip
/Users/patrickchong/.conda/envs/AlgoExpert Questions/bin/pip
/Users/patrickchong/.conda/envs/AlgoExpert Questions/lib/python3.10/site-packages/pip
/Applications/iFunia YouTube Downloader.app/Contents/Frameworks/ASVDSDK.framework/Versions/A/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip
/opt/homebrew/lib/python3.9/site-packages/pip
/opt/homebrew/Cellar/python#3.9/3.9.13_1/libexec/bin/pip
/opt/homebrew/Cellar/python#3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/share/doc/python3.9/examples/Tools/msi/pip
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
find: /dev/fd/6: Not a directory
(I omitted a lot of the "find ....", because it was a very long list)
Here is what I get when I run pip3:
patrickchong#Patricks-MacBook-Pro ~ % pip3
Usage:
pip3 <command> [options]
Commands:
install Install packages.
download Download packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
check Verify installed packages have compatible dependencies.
config Manage local and global configuration.
search Search PyPI for packages.
cache Inspect and manage pip's wheel cache.
index Inspect information available from package indexes.
wheel Build wheels from your requirements.
hash Compute hashes of package archives.
completion A helper command used for command completion.
debug Show information useful for debugging.
help Show help for commands.
General Options:
-h, --help Show help.
--debug Let unhandled exceptions propagate outside the
main subroutine, instead of logging them to
stderr.
--isolated Run pip in an isolated mode, ignoring
environment variables and user configuration.
--require-virtualenv Allow pip to only run in a virtual environment;
exit with an error otherwise.
-v, --verbose Give more output. Option is additive, and can be
used up to 3 times.
-V, --version Show version and exit.
-q, --quiet Give less output. Option is additive, and can be
used up to 3 times (corresponding to WARNING,
ERROR, and CRITICAL logging levels).
--log <path> Path to a verbose appending log.
--no-input Disable prompting for input.
--proxy <proxy> Specify a proxy in the form
scheme://[user:passwd#]proxy.server:port.
--retries <retries> Maximum number of retries each connection should
attempt (default 5 times).
--timeout <sec> Set the socket timeout (default 15 seconds).
--exists-action <action> Default action when a path already exists:
(s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
--trusted-host <hostname> Mark this host or host:port pair as trusted,
even though it does not have valid or any HTTPS.
--cert <path> Path to PEM-encoded CA certificate bundle. If
provided, overrides the default. See 'SSL
Certificate Verification' in pip documentation
for more information.
--client-cert <path> Path to SSL client certificate, a single file
containing the private key and the certificate
in PEM format.
--cache-dir <dir> Store the cache data in <dir>.
--no-cache-dir Disable the cache.
--disable-pip-version-check
Don't periodically check PyPI to determine
whether a new version of pip is available for
download. Implied with --no-index.
--no-color Suppress colored output.
--no-python-version-warning
Silence deprecation warnings for upcoming
unsupported Pythons.
--use-feature <feature> Enable new functionality, that may be backward
incompatible.
--use-deprecated <feature> Enable deprecated functionality, that will be
removed in the future.
I'm creating a private package repository using PyPiServer. I'm serving this on one VPS instance and trying to pip install from another.
Traceback from pip:
root#managersvr:~# pip install --index-url http://<IP>:8080/simple/ global_toolkit --trusted-host <IP>
Could not find a version that satisfies the requirement global_toolkit (from versions: )
No matching distribution found for global_toolkit
root#managersvr:~#
Server logs from PyPi:
[02/May/2020 23:57:10] "GET /simple/global-toolkit/ HTTP/1.1" 200 313
I'm struggling to debug. Steps taken so far:
- I have run pip install --upgrade pip
- I have navigated to the PyPi IP address in browser, which correctly displays:
Welcome to pypiserver!
This is a PyPI compatible package index serving 1 packages.
To use this server with pip, run the following command:
pip install --index-url http://<IP>:8080/simple/ PACKAGE [PACKAGE2...]
To use this server with easy_install, run the following command:
easy_install --index-url http://<IP>:8080/simple/ PACKAGE [PACKAGE2...]
The complete list of all packages can be found here or via the simple index.
This instance is running version 1.3.2 of the pypiserver software.
Navigating to the "here" link for a complete list of all packages yields:
Index of packages
global_toolkit.tar.gz
And navigating to the simple index yields:
Simple Index
global-toolkit
I also tried using global-toolkit instead of global_toolkit.
I am looking for steps to identify the layer at which there is an error, and debug it. Thanks.
UPDATE
I found the pip search command. The output is, strange:
(venv) root#managersvr:~# pip search --index http://<IP>:8080 global_toolkit
global_toolkit () -
I've resolved the issue. In hindsight I should have foreseen the problem. I failed to include my ext_modules value in the top level setup.py (so the cython modules weren't compiled). A good tip for debugging: use the -v flag!
I am trying to run:
pip3 install -e .
in my Python project where I have the following setup.py:
from setuptools import setup
setup(
name='mypackage',
install_requires=[
"anotherpackage#git+git#bitbucket.org:myorg/anotherpackage.git"
]
)
but it fails with:
error in mypackage setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL given
I guess it is correct about the format of my URL as PEP 508 doesn't allow specifying git user name for ssh clone URLs.
What is the correct syntax for PEP 508 URLs with git+ssh protocol for install_requires dependency for private git repositories (in this case hosted on BitBucket)? What is the syntax for specifying a specific branch, tag or sha?
More context to avoid XY problem
I have an internal Python project that depends on multiple internally developed Python packages. I would like to avoid the necessity for hosting my own PIP repository in the organisation and thus I am trying to use git URLs directly. I need to use ssh protocol for git URLs as all the users have their ssh keys configured and it would be cumbersome to ask all the users to configure their app passwords in BitBuckets (I have 2FA required and the regular user password doesn't work).
I have already tried to use:
dependency_links
setup(
name='mypackage',
install_requires=[
"anotherpackage==0.0.1"
],
dependency_links=[
"git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage-0.0.1"
]
)
But they are deprecated and they are ignored by pip3 install -e .. According to documentation I've found, PEP 508 URLs should be used instead.
requirements.txt file with entries duplicated from install_requires entries
I have a requirements.txt file with:
-e git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage
and I use pip3 install -r requirements.txt instead of pip3 install -e .. It works but is suboptimal as I have to keep both setyp.py and requirements.txt in sync.
If there is any other recommended solution for my problem I would like to learn about it :)
After checking pip source code I found the correct syntax for private BitBucket repositories.
The general form for the packages with URLs is <package name>#<URI> and the URI must start with a <scheme>://.
So I fixed it to:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git
and then I was getting a different error - this time git command (invoked by pip) was complaining about repository URL ssh://git#bitbucket.org:myorg/anotherpackage.git.
I checked the git documentation for the ssh:// URLs format and found out that hostname and organisation parts must be separated with / instead of ::
ssh://git#bitbucket.org/myorg/anotherpackage.git
This URL works fine. I also learned from the pip source code that the actual revision/branch/tag can be specified by appending #<rev-spec> so I can specify for example the tag 0.0.1 with the following in install_requires:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git#0.0.1
The only issue that I still have is that when I change the revision and run pip3 install -e . again it doesn't detect the change (even when run with --upgrade). I have to manually uninstall the package (pip3 uninstall anotherpackage) and run pip3 install -e . again.
To preface, I have already seen this question Is it possible to use pip to install a package from a private github repository?
I am trying to install a package from a private repository that I have access to using pip.
I am able to directly clone it like so:
(myenv)robbie#ubuntu:~/git$ git clone git#github.com:matherbk/django-messages.git
Cloning into 'django-messages'...
remote: Counting objects: 913, done.
remote: Compressing objects: 100% (345/345), done.
remote: Total 913 (delta 504), reused 913 (delta 504)
Receiving objects: 100% (913/913), 165.73 KiB, done.
Resolving deltas: 100% (504/504), done.
But when I try to install it via pip (my virtualenv is activated):
(myenv)robbie#ubuntu:~/git$ pip install git+https://git#github.com/matherbk/django-messages.gitDownloading/unpacking git+https://git#github.com/matherbk/django-messages.git
Cloning https://git#github.com/matherbk/django-messages.git to /tmp/pip-13ushS-build
Password for 'https://git#github.com':
fatal: Authentication failed
Complete output from command /usr/bin/git clone -q https://git#github.com/matherbk/django-messages.git /tmp/pip-13ushS-build:
----------------------------------------
Command /usr/bin/git clone -q https://git#github.com/matherbk/django-messages.git /tmp/pip-13ushS-build failed with error code 128 in None
Storing complete log in /home/robbie/.pip/pip.log
I tried typing in my password but it failed. However I am ssh authenticated for git#github.com:
(myenv)robbie#ubuntu:~/git$ ssh -T git#github.com
Hi robpodosek! You've successfully authenticated, but GitHub does not provide shell access.
I can switch git#github.com to robpodosek#github.com and it lets me install via pip just fine:
(myenv)robbie#ubuntu:~/git$ pip install git+https://robpodosek#github.com/matherbk/django-messages.git
Downloading/unpacking git+https://robpodosek#github.com/matherbk/django-messages.git
Cloning https://robpodosek#github.com/matherbk/django-messages.git to /tmp/pip-SqEan9-build
Password for 'https://robpodosek#github.com':
Running setup.py egg_info for package from git+https://robpodosek#github.com/matherbk/django-messages.git
warning: no files found matching 'README'
Installing collected packages: django-messages
Running setup.py install for django-messages
warning: no files found matching 'README'
Successfully installed django-messages
Cleaning up...
However I want to do what the first mentioned article does by using git#github.com so that I don't have to add my username into a requirements.txt file and add that to version control.
Any thoughts? I previously had this working but had to boot up a fresh image. Thanks ahead of time.
It worked by using oxyum's suggestion of changing the : to a /:
pip install git+ssh://git#github.com/matherbk/django-messages.git
Make sure you use github.com/account instead of github.com:account
see Git+SSH dependencies have subtle (yet critical) differences from git clone
Had virtualenv activated and had to install a series of applications from github.com from a text file.
(venv)$ cat requirements.txt
-e git://github.com/boto/botocore.git#develop#egg=botocore
-e git://github.com/boto/jmespath.git#develop#egg=jmespath
-e git://github.com/boto/s3transfer.git#develop#egg=s3transfer
nose==1.3.3
mock==1.3.0
wheel==0.24.0
unittest2==0.5.1; python_version == '2.6'
(venv)$ pip install -r requirements.txt
Ignoring unittest2: markers 'python_version == "2.6"' don't match your environment Obtaining botocore from git+git://github.com/boto/botocore.git#develop#egg=botocore (from -r requirements.txt (line 1))
Cloning git://github.com/boto/botocore.git (to develop) to ./venv/src/botocore
fatal: unable to connect to github.com:
github.com[0: 192.30.253.112]: errno=Connection timed out
github.com[1: 192.30.253.113]: errno=Connection timed out
Command "git clone -q git://github.com/boto/botocore.git
/home/ubuntu/utils/boto3/venv/src/botocore" failed with error code 128 in None
However, as #Robeezy suggested, edited the requirement.txt and changed from
-e git://github.com...
to
-e git+https://github.com...
That is the link provided if you were to clone from the site (only options were Clone or Download).
So, thank you! It did work finally.
If you're installing with pip install git+https://github.com/repo and getting this error, make sure your username and password are correct. I was getting this error because I was incorrectly entering my password.