We Keep Coding

My computer could not locate pip all of a sudden, and when I reinstalled it, all of my Python packages were gone?

I was working on a Python/Flask project, and I am not sure what happened but all of a sudden when I opened my Terminal, it seemed like pip was deleted; when I tried running things like 'pip list' from different directories (including my global one), I got the following
zsh: command not found: pip
Needless to say, when I tried running packages that I had previously installed with pip like Flask, I got the same zsh error as per above.
And when I reinstalled pip and ran pip list, it seems like all of my Python packages got deleted?:
pip 22.1.2
setuptools 58.1.0
wheel 0.37.1
Any ideas what could have caused this? I can reinstall the packages that is no issue. But I fear this will happen again in the future and I have no idea what I did that could have caused this...
Could this have been a $PATH issue?
When I run echo $PATH in my global directory, I get:
/Library/Frameworks/Python.framework/Versions/3.10/bin
/opt/homebrew/bin
/opt/homebrew/sbin
/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin
/Library/Apple/usr/bin
#mast o;
Here is what I get when running find / -name pip:
/Users/patrickchong/PycharmProjects/Live_Server/venv/bin/pip
/Users/patrickchong/PycharmProjects/Live_Server/venv/lib/python2.7/site-packages/pip
/Users/patrickchong/PycharmProjects/JP interview/venv/bin/pip
/Users/patrickchong/PycharmProjects/JP interview/venv/lib/python2.7/site-packages/pip
/Users/patrickchong/opt/anaconda3/bin/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py310hecd8cb5_0/bin/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py310hecd8cb5_0/lib/python3.10/site-packages/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py39hecd8cb5_0/bin/pip
/Users/patrickchong/opt/anaconda3/pkgs/pip-21.2.4-py39hecd8cb5_0/lib/python3.9/site-packages/pip
/Users/patrickchong/opt/anaconda3/lib/python3.9/site-packages/pip
find: /Users/patrickchong/.Trash: Operation not permitted
/Users/patrickchong/.conda/envs/Crypto Trading Bot project/bin/pip
/Users/patrickchong/.conda/envs/Crypto Trading Bot project/lib/python3.10/site-packages/pip
/Users/patrickchong/.conda/envs/AlgoExpert Questions/bin/pip
/Users/patrickchong/.conda/envs/AlgoExpert Questions/lib/python3.10/site-packages/pip
/Applications/iFunia YouTube Downloader.app/Contents/Frameworks/ASVDSDK.framework/Versions/A/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip
/opt/homebrew/lib/python3.9/site-packages/pip
/opt/homebrew/Cellar/python#3.9/3.9.13_1/libexec/bin/pip
/opt/homebrew/Cellar/python#3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/share/doc/python3.9/examples/Tools/msi/pip
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
find: /dev/fd/6: Not a directory
(I omitted a lot of the "find ....", because it was a very long list)
Here is what I get when I run pip3:
patrickchong#Patricks-MacBook-Pro ~ % pip3
Usage:
pip3 <command> [options]
Commands:
install Install packages.
download Download packages.
uninstall Uninstall packages.
freeze Output installed packages in requirements format.
list List installed packages.
show Show information about installed packages.
check Verify installed packages have compatible dependencies.
config Manage local and global configuration.
search Search PyPI for packages.
cache Inspect and manage pip's wheel cache.
index Inspect information available from package indexes.
wheel Build wheels from your requirements.
hash Compute hashes of package archives.
completion A helper command used for command completion.
debug Show information useful for debugging.
help Show help for commands.
General Options:
-h, --help Show help.
--debug Let unhandled exceptions propagate outside the
main subroutine, instead of logging them to
stderr.
--isolated Run pip in an isolated mode, ignoring
environment variables and user configuration.
--require-virtualenv Allow pip to only run in a virtual environment;
exit with an error otherwise.
-v, --verbose Give more output. Option is additive, and can be
used up to 3 times.
-V, --version Show version and exit.
-q, --quiet Give less output. Option is additive, and can be
used up to 3 times (corresponding to WARNING,
ERROR, and CRITICAL logging levels).
--log <path> Path to a verbose appending log.
--no-input Disable prompting for input.
--proxy <proxy> Specify a proxy in the form
scheme://[user:passwd#]proxy.server:port.
--retries <retries> Maximum number of retries each connection should
attempt (default 5 times).
--timeout <sec> Set the socket timeout (default 15 seconds).
--exists-action <action> Default action when a path already exists:
(s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
--trusted-host <hostname> Mark this host or host:port pair as trusted,
even though it does not have valid or any HTTPS.
--cert <path> Path to PEM-encoded CA certificate bundle. If
provided, overrides the default. See 'SSL
Certificate Verification' in pip documentation
for more information.
--client-cert <path> Path to SSL client certificate, a single file
containing the private key and the certificate
in PEM format.
--cache-dir <dir> Store the cache data in <dir>.
--no-cache-dir Disable the cache.
--disable-pip-version-check
Don't periodically check PyPI to determine
whether a new version of pip is available for
download. Implied with --no-index.
--no-color Suppress colored output.
--no-python-version-warning
Silence deprecation warnings for upcoming
unsupported Pythons.
--use-feature <feature> Enable new functionality, that may be backward
incompatible.
--use-deprecated <feature> Enable deprecated functionality, that will be
removed in the future.

Hortonworks 2.6.5 yum install python-pip not working

I am trying to install python pip on HDP2.6.5, but its constantly failing with Error 403.
[root#sandbox-hdp ~]# yum install python-pip
Loaded plugins: fastestmirror, ovl
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
To address this issue please refer to the below wiki article
https://wiki.centos.org/yum-errors
If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
One of the configured repositories failed (HDP-2.6-repo-1),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=HDP-2.6-repo-1 ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable HDP-2.6-repo-1
or
subscription-manager repos --disable=HDP-2.6-repo-1
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=HDP-2.6-repo-1.skip_if_unavailable=true
failure: repodata/repomd.xml from HDP-2.6-repo-1: [Errno 256] No more mirrors to try.
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden

Hortonworks repos have been behind a pay wall for the last few years, so installing or upgrading of packages within the sandbox will no longer work.

Python pip install gives “Command ”python setup.py egg_info“ failed with error code 1”

Edit: Yes I know this question already exists, except my question is a bit different and none of the solutions fixed it.
I do most of my Python stuff when I'm at work and not on my personal machine, but I decided to install it on my personal computer as well. I fresh installed python 3.6.1, and created a virtual environment with virtualenv. Then within the virtualenv I tried to pip install urllib (or any module) and I received the error:
(pdbot) C:\Users\user\Documents\pdbot>pip install urllib
Collecting urllib
Using cached urllib-1.21.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\setup.py", line 191
s.connect((base64.b64decode(rip), 017620))
^
SyntaxError: invalid token
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\
I read elsewhere that this error had something to do with setuptools not being properly installed. So I ran this to attempt to fix the issue:
easy_install -U setuptools
I ended up receiving an even weirder error next:
(pdbot) C:\Users\zeke\Documents\pdbot>easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Downloading https://pypi.python.org/packages/a9/23/720c7558ba6ad3e0f5ad01e0d6ea2288b486da32f053c73e259f7c392042/setuptools-36.0.1.zip#md5=430eb106788183eefe9f444a300007f0
Best match: setuptools 36.0.1
Processing setuptools-36.0.1.zip
Writing C:\Users\zeke\AppData\Local\Temp\easy_install-jhg1val_\setuptools-36.0.1\setup.cfg
Running setuptools-36.0.1\setup.py -q bdist_egg --dist-dir C:\Users\zeke\AppData\Local\Temp\easy_install-jhg1val_\setuptools-36.0.1\egg-dist-tmp-8apak7kn
warning: no files found matching '*' under directory 'setuptools\_vendor'
Copying setuptools-36.0.1-py3.6.egg to c:\users\zeke\documents\pdbot\lib\site-packages
Adding setuptools 36.0.1 to easy-install.pth file
Installing easy_install-script.py script to c:\users\zeke\documents\pdbot\Scripts
Installing easy_install.exe script to c:\users\zeke\documents\pdbot\Scripts
error: [WinError 5] Access is denied: 'c:\\users\\zeke\\documents\\pdbot\\Scripts\\easy_install.exe'
This looks like a permissions error, but I ran these both in an administrator command prompt (Windows 10) and got the same result. I am the only user on this computer and I have all admin permissions. Is this virtualenv causing an issue? How do I remedy it?
EDIT: I was able to fix the permissions issue by leveraging the python executable like so:
python -m easy_install -U setuptools
But it didn't fix the python setup.py egg_info issue. I still get this error message when trying to pip install anything:
Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\
I have tried both python -m pip install urllib and pip install urllib and neither work.

I had the same problem when trying to install urllib, but after doing a pip search urllib, I discovered that the problem was due to the version of urllib. From the search:
$ pip search urllib
...
> urllib5 (5.0.0) - Just increment the number and create a new lib. Never fix the original one.
At the end, a simple
pip install urllib5
within an elevated shell solved it.

Your problem has to do with permissions. The related/similar tools setup_tools, easy_install, and pip all tend to set a default set of permissions on files and folders they try to create in the package installation folder(s), rather than trying to match access permissions of the location they're installing in.
On Linux systems, where files and folders individually have permissions, this is frequently bypassed with the sudo command. On Windows, the equivalent is to run the installer as an Administrator. Since you're in the console, you have to open a console with Administrator privileges to run the pip command in.
Notable under Windows, the modules installed with pip from an Administrator console are still accessible to all users of the system that have the proper path in the PYTHONPATH system environment variable. Under Linux however, the problem is exacerbated by the fact that the files themselves may not be created with read and execute access for other users and may need to have their permissions manually modified after installation.
WARNING: urllib vs urllib2 vs urllibx
Both other answers claim that the problem is you're not specifying the correct "version" of the module in the call to pip. Neither is correct, as the error clearly indicates an installation folder access permissions violation causing the failure, but they also incorrectly recommended VERY unsafe behavior.
pip install urllib != pip install urllib5 these are two completely different packages.
The documentation for pip (https://packaging.python.org/tutorials/installing-packages/#id17) clearly says the way to specify a module version explicitly is pip install 'urllib==5'.
As part of how the package management engine implemented by pip works, running the command pip install urllib will always try to use the latest version of the urllib package, so you shouldn't need to specify the version unless you have some reason that you need a very specific version of the module.

There are two points to make in order to answer your question:
1. You are lucky you did not install that package!
The package you were trying to install was a maliciously created python package that was designed to look like a real package (in this case urllib3). If you had installed it, the package would have operated as normal except it would have sent some basic information about the system on which you installed the package to a URL (you can see more details on this here). You can read more about this fake package at either of the following links:
https://app.threatconnect.com/auth/incident/incident.xhtml?incident=5256822&owner=Common%20Community (you can sign up for a free account to view this one)
http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/index.html
Sending basic information about your systems to an unknown source isn't the worst thing you could do, but is certainly something you want to avoid when possible.
2. To properly install a package...
Specifically urllib:
To install urllib, you need to specify the version of the package you would like to install. For example, pip install urllib3.
Any package in general:
As #Elisabete Coelho suggested, you can use the pip search <package-name> feature to view the available packages. This is not perfect, however, as it may list malicious libraries like the one you were trying to install. A good guideline is that you should follow the installation instructions in a package's documentation closely to avoid any unforeseen issues. This is just an unfortunate necessity of living in a world where people make pretend python packages.

using repo other then pypi with pip

I am having trouble understanding how pip works in a specific environment. The thing is that I am trying to install OpenStack using ansible-openstack deployment method. It provides playbooks to prepare the complete environment and install all components. Deployment fails at the step when python modules should be installed from a private repo (it is being built during previous deployment steps). This repo is nothing else as nginx server exposing directories with python wheels via http protocol. When OpenStack nodes are reaching over to this repo with pip in order to install packages, they seem not to find any of the packages that satisfy their constraints (I am not posting the whole command using the requirement file, but required versions are there). Each node has similarly looking HOME/./pip/pip.conf:
[global]
no-index = true
pre = True
timeout = 120
trusted-host =
172.21.51.152
[install]
upgrade = True
find-links =
http://172.21.51.152:8181/os-releases/14.0.7/
where 172.21.51.152 is that repo that is supposed to replace PyPI.
When playbook tasks that should install packages are running they all fail with similar error:
root#control1-galera-container-434df170:~# pip install MySQL-python
Collecting MySQL-python
Could not find a version that satisfies the requirement MySQL-python (from versions: )
No matching distribution found for MySQL-python
even though the file is there and I can wget it from the URL where the file is located.
ls /var/www/repo/os-releases/14.0.7/ | grep -i mysql_python
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
I checked with tcpdump what kind of request pip is sending when "pip install" command is used. It sends a GET get to http://172.21.51.152:8181/os-releases/14.0.7/ where all wheel files are. Server replies with the index.html file:
<html>\r\n
<head><title>Index of /</title></head>\r\n
<body bgcolor="white">\r\n
<h1>Index of /</h1><hr><pre>../\r\n
links/
openstackgit/
os-releases/
pkg-cache/
pools/
venvs/
</pre><hr></body>\r\n
</html>\r\n
After getting this reply pip seems to be satisfied with the outcome and decides that the package it was looking for is not there. I am trying to figure out if that is expected behaviour and there is something wrong with that repo or if pip is being launched with insufficient configuration that doesn't allow it to perform the search.
I tried to delete the pip.conf altogether and pip worked normally just downloading files from PyPI. What am I missing here in order to make it work from the private repository?
Thanks for you help!

Resolved this question in OpenStack Operators mailing list. Was an issue of nginx server and not working autoindex. After fixing autoindes the complete directory index started to show up which enabled pip to download and install packages.

FWIW, I had the very same symptom Danil described:
root#control1-galera-container-434df170:~# pip install MySQL-python
Collecting MySQL-python Could not find a version that satisfies the requirement MySQL-python (from versions: )
No matching distribution found for MySQL-python
The root cause in my case was a (restrictive) umask setting of 077.
# root#controller1
$ umask # octal
0077
$ umask -S # symbolic
u=rwx,g=,o=
The default seems to be 022 (symbolic: u=rwx,g=rx,o=rx), details at https://en.wikipedia.org/wiki/Umask
This umask of 077 affects the nginx config files (which, for example, set the
'autoindex' behavior, which is needed to serve the python wheels):
# root#controller1-repo-container-e0445a8f
$ cd /etc/nginx/sites-available/
$ ll openstack-slushee.vhost
-rw------- 1 root root [...] openstack-slushee.vhost
If also affects the python packages:
# root#controller1-repo-container-e0445a8f
$ cd /var/www/repo/pools/ubuntu-16.04-x86_64/mysql_python/
$ ll mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
-rw------- 1 nginx root [...] mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl
And these python packages get requested from the other containers in an openstack-ansible setup:
# root#controller1-galera-container-564eedea
$ cat .pip/pip.conf
[...]
[install]
upgrade = True
find-links =
http://172.29.236.11:8181/os-releases/15.1.2/ubuntu-16.04-x86_64/
And ultimately, it affects the file which specifies the python package versions:
# root#controller1-repo-container-e0445a8f
$ cd /var/www/repo/os-releases/15.1.2/ubuntu-16.04-x86_64/
$ ll requirements_absolute_requirements.txt
-rw------- 1 root root [...] requirements_absolute_requirements.txt
You will notice that nginx complains a lot in its error log:
# root#controller1-repo-container-e0445a8f
$ grep -i error /var/log/nginx/openstack-slushee.error.log | grep -i mysql
2017/07/03 10:30:34 [error] 17458\#17458: *642 open()
"/var/www/repo/os-releases/15.1.2/ubuntu-16.04-x86_64/
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl" failed
(13: Permission denied), client: 172.29.236.11,
server: openstack-slushee,
request: "GET /os-releases/15.1.2/ubuntu-16.04-x86_64/
mysql_python-1.2.5-cp27-cp27mu-linux_x86_64.whl HTTP/1.1",
host: "172.29.236.11:8181"
Long story short, set the umask to a less restrictive mode before starting openstack-ansible:
$ umask 0022
Side notes:
here's the discussion where I want to figure out how to handle umask in openstack-ansible: http://lists.openstack.org/pipermail/openstack-operators/2017-July/013892.html
this question might have been better suited at https://ask.openstack.org/en/questions/

Fatal error while installing keras

I was trying to install keras (for using LSTM) on my Windows 10 system in Anaconda using the method provided by Yelaman. But after running the command pip install git+git://github.com/Theano/Theano.git, I received the following fatal error -
C:\Anaconda>pip install git+git://github.com/Theano/Theano.git
Collecting git+git://github.com/Theano/Theano.git
Cloning git://github.com/Theano/Theano.git to
c:\users\krishna\appdata\local\temp\pip-reettr-build fatal: protocol
error: bad line length character:
Er
Command "git clone -q git://github.com/Theano/Theano.git
c:\users\krishna\appdata\local\temp\pip-reettr-build" failed with
error code 128 in None
I don't have a github account but i use git using bitbucket (if that matters).
Could anyone explain to me what error I am committing and if there is a way out? My main goal is to use keras in Anaconda in Windows.
Thanks!

It seems that the error may be because we have a git repository listed as a dependency. Not still fully sure, though. (Source)
The solution is to replace pip install git+git://github.com/Theano/Theano.git with pip install git+http://github.com/Theano/Theano.git i.e. replacing the second git with http (this works since the repository is public)