To preface, I have already seen this question Is it possible to use pip to install a package from a private github repository?
I am trying to install a package from a private repository that I have access to using pip.
I am able to directly clone it like so:
(myenv)robbie#ubuntu:~/git$ git clone git#github.com:matherbk/django-messages.git
Cloning into 'django-messages'...
remote: Counting objects: 913, done.
remote: Compressing objects: 100% (345/345), done.
remote: Total 913 (delta 504), reused 913 (delta 504)
Receiving objects: 100% (913/913), 165.73 KiB, done.
Resolving deltas: 100% (504/504), done.
But when I try to install it via pip (my virtualenv is activated):
(myenv)robbie#ubuntu:~/git$ pip install git+https://git#github.com/matherbk/django-messages.gitDownloading/unpacking git+https://git#github.com/matherbk/django-messages.git
Cloning https://git#github.com/matherbk/django-messages.git to /tmp/pip-13ushS-build
Password for 'https://git#github.com':
fatal: Authentication failed
Complete output from command /usr/bin/git clone -q https://git#github.com/matherbk/django-messages.git /tmp/pip-13ushS-build:
----------------------------------------
Command /usr/bin/git clone -q https://git#github.com/matherbk/django-messages.git /tmp/pip-13ushS-build failed with error code 128 in None
Storing complete log in /home/robbie/.pip/pip.log
I tried typing in my password but it failed. However I am ssh authenticated for git#github.com:
(myenv)robbie#ubuntu:~/git$ ssh -T git#github.com
Hi robpodosek! You've successfully authenticated, but GitHub does not provide shell access.
I can switch git#github.com to robpodosek#github.com and it lets me install via pip just fine:
(myenv)robbie#ubuntu:~/git$ pip install git+https://robpodosek#github.com/matherbk/django-messages.git
Downloading/unpacking git+https://robpodosek#github.com/matherbk/django-messages.git
Cloning https://robpodosek#github.com/matherbk/django-messages.git to /tmp/pip-SqEan9-build
Password for 'https://robpodosek#github.com':
Running setup.py egg_info for package from git+https://robpodosek#github.com/matherbk/django-messages.git
warning: no files found matching 'README'
Installing collected packages: django-messages
Running setup.py install for django-messages
warning: no files found matching 'README'
Successfully installed django-messages
Cleaning up...
However I want to do what the first mentioned article does by using git#github.com so that I don't have to add my username into a requirements.txt file and add that to version control.
Any thoughts? I previously had this working but had to boot up a fresh image. Thanks ahead of time.
It worked by using oxyum's suggestion of changing the : to a /:
pip install git+ssh://git#github.com/matherbk/django-messages.git
Make sure you use github.com/account instead of github.com:account
see Git+SSH dependencies have subtle (yet critical) differences from git clone
Had virtualenv activated and had to install a series of applications from github.com from a text file.
(venv)$ cat requirements.txt
-e git://github.com/boto/botocore.git#develop#egg=botocore
-e git://github.com/boto/jmespath.git#develop#egg=jmespath
-e git://github.com/boto/s3transfer.git#develop#egg=s3transfer
nose==1.3.3
mock==1.3.0
wheel==0.24.0
unittest2==0.5.1; python_version == '2.6'
(venv)$ pip install -r requirements.txt
Ignoring unittest2: markers 'python_version == "2.6"' don't match your environment Obtaining botocore from git+git://github.com/boto/botocore.git#develop#egg=botocore (from -r requirements.txt (line 1))
Cloning git://github.com/boto/botocore.git (to develop) to ./venv/src/botocore
fatal: unable to connect to github.com:
github.com[0: 192.30.253.112]: errno=Connection timed out
github.com[1: 192.30.253.113]: errno=Connection timed out
Command "git clone -q git://github.com/boto/botocore.git
/home/ubuntu/utils/boto3/venv/src/botocore" failed with error code 128 in None
However, as #Robeezy suggested, edited the requirement.txt and changed from
-e git://github.com...
to
-e git+https://github.com...
That is the link provided if you were to clone from the site (only options were Clone or Download).
So, thank you! It did work finally.
If you're installing with pip install git+https://github.com/repo and getting this error, make sure your username and password are correct. I was getting this error because I was incorrectly entering my password.
Related
I am trying to install python packages from the local file system with pip as discussed in this question.
I have used pip2pi as suggested in the accepted answer in the above question.
dmanna#ubuntu:~$ mkdir -p pyt/pkg
dmanna#ubuntu:~$ pip2tgz pyt/pkg/ patroni[zookeeper]
The downloaded packages
dmanna#ubuntu:~$ ls /home/dmanna/pyt/pkg/
cdiff-1.0.tar.gz
certifi-2018.4.16-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
click-6.7-py2.py3-none-any.whl
idna-2.7-py2.py3-none-any.whl
kazoo-2.5.0-py2.py3-none-any.whl
patroni-1.4.4.tar.gz
prettytable-0.7.2.tar.bz2
psutil-5.4.6.tar.gz
psycopg2-2.7.4.tar.gz
python_dateutil-2.7.3-py2.py3-none-any.whl
pytz-2018.4-py2.py3-none-any.whl
PyYAML-3.12.tar.gz
requests-2.19.1-py2.py3-none-any.whl
six-1.11.0-py2.py3-none-any.whl
tzlocal-1.5.1.tar.gz
urllib3-1.23-py2.py3-none-any.whl
Then when I am trying to install the required package from the above local directory. It is giving me the below error
dmanna#ubuntu:~$ sudo pip install --no-index --find-links=pyt/pkg/patroni-1.4.4.tar.gz patroni[zookeeper]
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking patroni[zookeeper]
Running setup.py (path:/tmp/pip_build_root/patroni/setup.py) egg_info for package patroni
Installing extra requirements: 'zookeeper'
Downloading/unpacking urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Could not find any downloads that satisfy the requirement urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Cleaning up...
No distributions at all found for urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Storing debug log for failure in /home/dmanna/.pip/pip.log
Can someone let me know what I am doing wrong?
Python - 2.7.6
PIP - 1.5.4
Ubuntu 14.04
--find-links=pyt/pkg
Find links in the directory, not in the patroni-1.4.4.tar.gz archive.
I have a Pyramid server on an Openshift Python 3 cartridge that does not like the module Pandas, but I cannot figure how to figure out why it is so.
In setup.py the method setuptools.setup is called with the argument install_requires with a few modules (such as pyramid,biopython,numpy), which run fine. But when I add pandas it fails. This is what it says during pushing (✱ = censoring added for security):
remote: Installing collected packages: six, python-dateutil, pytz, pandas
remote: Found existing installation: six 1.3.0
remote: Not uninstalling six at /opt/rh/python33/root/usr/lib/python3.3/site-packages, outside environment /var/lib/openshift/✱✱✱✱✱/python/virtenv/venv
remote: Running setup.py install for pandas: started
remote: Running setup.py install for pandas: still running...
remote: Running setup.py install for pandas: still running...
remote: Running setup.py install for pandas: still running...
remote: Running setup.py install for pandas: still running...
remote: Running setup.py install for pandas: still running...
remote: Running setup.py install for pandas: still running...
Connection to pedel2-matteoferla.rhcloud.com closed by remote host.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
To ssh://pedel2-matteoferla.rhcloud.com/~/git/pedel2.git/
✱✱✱ master -> master
error: failed to push some refs to 'ssh://✱✱✱✱#pedel2-matteoferla.rhcloud.com/~/git/pedel2.git/'
Adding a requirements.txt file does not affect it due to permission issues seen also with when I ssh into the server and pip install it manually. Namely /var/lib/openshift isn't my virtual environment.
remote: Activating virtenv
remote: Checking for pip dependency listed in requirements.txt file..
remote: The directory '/var/lib/openshift/✱✱✱✱/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Pip3 within the virtual python does the same.
cd /python/virtenv/venv/bin/
./pip3 install pandas
Easy_install does it right but has no permission, so this approach is wrong anyway.
pedel2-matteoferla.rhcloud.com bin]\> ./easy_install pandas
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 122] Disk quota exceeded: '/var/lib/openshift/✱✱✱✱/python/virtenv/venv/lib/python3.3/site-packages/test-easy-install-247598.write-test'
I assume I ought to solve the problem with setuptools module. Which means I need to figure out why from there. But I am stuck.
EDIT. It is not my disk quota.
Matteos-Air-3:pedel2 matteo$ rhc show-app pedel2 --gears quota
Gear Cartridges Used Limit
------------------------ ---------- ------ -----
✱✱✱✱ python-3.3 268 MB 1 GB
I think you have a few issues.
size issue - it's difficult to reproduce it. But maybe we don't need it (see details below)
Modern Pandas versions are NOT compatible with Python 3.3 - see this
I'd recommend you to use "ready to go" distributives like Anaconda with already preinstalled numpy, pandas and lot more.
Beside that - don't overwrite systme Python - it might cause another problems.
Just install Anaconda into a separate directory and set Anaconda environment everytime you want to use it.
PS alternatively you can have Anaconda as Docker
I am having trouble understanding how pip works in a specific environment. The thing is that I am trying to install OpenStack using ansible-openstack deployment method. It provides playbooks to prepare the complete environment and install all components. Deployment fails at the step when python modules should be installed from a private repo (it is being built during previous deployment steps). This repo is nothing else as nginx server exposing directories with python wheels via http protocol. When OpenStack nodes are reaching over to this repo with pip in order to install packages, they seem not to find any of the packages that satisfy their constraints (I am not posting the whole command using the requirement file, but required versions are there). Each node has similarly looking HOME/./pip/pip.conf:
[global]
no-index = true
pre = True
timeout = 120
trusted-host =
172.21.51.152
[install]
upgrade = True
find-links =
http://172.21.51.152:8181/os-releases/14.0.7/
where 172.21.51.152 is that repo that is supposed to replace PyPI.
When playbook tasks that should install packages are running they all fail with similar error:
root#control1-galera-container-434df170:~# pip install MySQL-python
Collecting MySQL-python
Could not find a version that satisfies the requirement MySQL-python (from versions: )
No matching distribution found for MySQL-python
even though the file is there and I can wget it from the URL where the file is located.
ls /var/www/repo/os-releases/14.0.7/ | grep -i mysql_python
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
I checked with tcpdump what kind of request pip is sending when "pip install" command is used. It sends a GET get to http://172.21.51.152:8181/os-releases/14.0.7/ where all wheel files are. Server replies with the index.html file:
<html>\r\n
<head><title>Index of /</title></head>\r\n
<body bgcolor="white">\r\n
<h1>Index of /</h1><hr><pre>../\r\n
links/
openstackgit/
os-releases/
pkg-cache/
pools/
venvs/
</pre><hr></body>\r\n
</html>\r\n
After getting this reply pip seems to be satisfied with the outcome and decides that the package it was looking for is not there. I am trying to figure out if that is expected behaviour and there is something wrong with that repo or if pip is being launched with insufficient configuration that doesn't allow it to perform the search.
I tried to delete the pip.conf altogether and pip worked normally just downloading files from PyPI. What am I missing here in order to make it work from the private repository?
Thanks for you help!
Resolved this question in OpenStack Operators mailing list. Was an issue of nginx server and not working autoindex. After fixing autoindes the complete directory index started to show up which enabled pip to download and install packages.
FWIW, I had the very same symptom Danil described:
root#control1-galera-container-434df170:~# pip install MySQL-python
Collecting MySQL-python Could not find a version that satisfies the requirement MySQL-python (from versions: )
No matching distribution found for MySQL-python
The root cause in my case was a (restrictive) umask setting of 077.
# root#controller1
$ umask # octal
0077
$ umask -S # symbolic
u=rwx,g=,o=
The default seems to be 022 (symbolic: u=rwx,g=rx,o=rx), details at https://en.wikipedia.org/wiki/Umask
This umask of 077 affects the nginx config files (which, for example, set the
'autoindex' behavior, which is needed to serve the python wheels):
# root#controller1-repo-container-e0445a8f
$ cd /etc/nginx/sites-available/
$ ll openstack-slushee.vhost
-rw------- 1 root root [...] openstack-slushee.vhost
If also affects the python packages:
# root#controller1-repo-container-e0445a8f
$ cd /var/www/repo/pools/ubuntu-16.04-x86_64/mysql_python/
$ ll mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
-rw------- 1 nginx root [...] mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
And these python packages get requested from the other containers in an openstack-ansible setup:
# root#controller1-galera-container-564eedea
$ cat .pip/pip.conf
[...]
[install]
upgrade = True
find-links =
http://172.29.236.11:8181/os-releases/15.1.2/ubuntu-16.04-x86_64/
And ultimately, it affects the file which specifies the python package versions:
# root#controller1-repo-container-e0445a8f
$ cd /var/www/repo/os-releases/15.1.2/ubuntu-16.04-x86_64/
$ ll requirements_absolute_requirements.txt
-rw------- 1 root root [...] requirements_absolute_requirements.txt
You will notice that nginx complains a lot in its error log:
# root#controller1-repo-container-e0445a8f
$ grep -i error /var/log/nginx/openstack-slushee.error.log | grep -i mysql
2017/07/03 10:30:34 [error] 17458\#17458: *642 open()
"/var/www/repo/os-releases/15.1.2/ubuntu-16.04-x86_64/
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl" failed
(13: Permission denied), client: 172.29.236.11,
server: openstack-slushee,
request: "GET /os-releases/15.1.2/ubuntu-16.04-x86_64/
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl HTTP/1.1",
host: "172.29.236.11:8181"
Long story short, set the umask to a less restrictive mode before starting openstack-ansible:
$ umask 0022
Side notes:
here's the discussion where I want to figure out how to handle umask in openstack-ansible: http://lists.openstack.org/pipermail/openstack-operators/2017-July/013892.html
this question might have been better suited at https://ask.openstack.org/en/questions/
I was trying to install keras (for using LSTM) on my Windows 10 system in Anaconda using the method provided by Yelaman. But after running the command pip install git+git://github.com/Theano/Theano.git, I received the following fatal error -
C:\Anaconda>pip install git+git://github.com/Theano/Theano.git
Collecting git+git://github.com/Theano/Theano.git
Cloning git://github.com/Theano/Theano.git to
c:\users\krishna\appdata\local\temp\pip-reettr-build fatal: protocol
error: bad line length character:
Er
Command "git clone -q git://github.com/Theano/Theano.git
c:\users\krishna\appdata\local\temp\pip-reettr-build" failed with
error code 128 in None
I don't have a github account but i use git using bitbucket (if that matters).
Could anyone explain to me what error I am committing and if there is a way out? My main goal is to use keras in Anaconda in Windows.
Thanks!
It seems that the error may be because we have a git repository listed as a dependency. Not still fully sure, though. (Source)
The solution is to replace pip install git+git://github.com/Theano/Theano.git with pip install git+http://github.com/Theano/Theano.git i.e. replacing the second git with http (this works since the repository is public)
When trying to install a git repo through pip, for example:
pip install git+https://github.com/kayluhb/django-admin-bootstrapped.git#cms-2.4
it just works. But then when putting that same line in a requirements file (if that matters this requirements file is called from another requirements file) then pip can't find the branch:
Could not find a tag or branch 'cms-2.4 '
What did I miss here? An argument like -e or something?
UPDATE:
I have tried another one without the '#' version:
git+http://github.com/jmoiron/johnny-cache.git
When I manually pip install it, it just works too. But then in the requirements file it fails with:
error: The requested URL returned error: 400 Bad request while accessing http://github.com/jmoiron/johnny-cache.git /info/refs?service=git-upload-pack
fatal: HTTP request failed
Command /usr/bin/git clone -q "http://github.com/jmoiron/johnny-cache.git " /tmp/pip-GE7AOI-build failed with error code 128 in None
Have you tried with git+git instead of git+https?