We Keep Coding

pip ignores pip.conf file for pod

I'm trying to install a custom package from a different index other than pypi in my pod.
I created a pip.conf and I run this commands in my dockerfile:
ADD docker_files/pip.conf /etc/pip.conf
ENV PIP_CONFIG_FILE /etc/pip.conf
and when I run RUN echo python3 -m pip config debug I can see in the output that pip does recognise my pip conf and all of the indexes in it. Output:
env_var: PIP_CONFIG_FILE='/etc/pip.conf' env: /etc/pip.conf, exists: True global.index-url: <custom> global.extra-index-url: <more custom> global: /etc/xdg/pip/pip.conf, exists: False /etc/pip.conf, exists: True global.index-url: <custom> global.extra-index-url: <more custom> site: /usr/local/pip.conf, exists: False
but when it searches for the package and it can't find it in pypi, it doesn't search my other indexes:
Searching for <package>
Reading https://pypi.org/simple/<package>/
Couldn't find index page for <package> (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
No local packages or working download links found for <package>
error: Could not find suitable distribution for Requirement.parse(<package>)
The command '/bin/sh -c python setup.py install' returned a non-zero code: 1
I also read that sometimes if the path $HOME/Library/Application Support exists it overwrite everything, but I checked and it does not. I'm using python 3.8.13, I also tried copying the pip conf file to /usr/local/pip.conf and removing the var PIP_CONFIG_FILE from the dockerfile, Nothing worked.
Did I miss something?

Why can't I delete broken Python packages? - "WARNING: Ignoring invalid distribution -ycopg2" [duplicate]

After running pip freeze I noticed the following warning on top of the list:
WARNING: Could not parse requirement: -atplotlib
So I checked the installed packages using pip list, and indeed the following is considered a package:
Package Version
--------------- -------
-atplotlib 3.0.3
I assume I probably had a typo when installing/upgrading matplotlib, which led to the aforementioned "package" being installed.
But I am not able to remove it as pip uninstall -atplotlib is read as a command and returns the following error:
No such option: -a
I found the following folders:
C:\Users\name\Anaconda3\Lib\site-packages\~atplotlib
C:\Users\name\Anaconda3\Lib\site-packages\~atplotlib-3.0.3-py3.7.egg-info
Is it safe, and sufficient, to remove them?

It is safe to delete the offending folder(s) from your site-packages directory.
More info below (source):
When uninstalling a package, pip will now rename it in place to a name
that cannot be imported, and once it has confirmed that everything
will succeed (including installing new versions if it’s doing an
upgrade), only then will it delete those folders. If something fails,
it renames them back.
Previously, it would copy the entire contents to another directory,
and potentially another drive, and then copy them back if it needed.
So this change is a significant performance improvement, especially
for packages with a lot of files in them.
What you’re seeing here is that the deletion failed for some reason -
perhaps pip crashed? - and so the directories were not removed. I
thought pip ignored them completely, but perhaps something else
changed since I tested that?
The directories are safe to delete.

EDIT: According to this link, provided by Lawrence in his answer
looking for and deleting the incorrectly named folders in your site-package directory should solve the issue.
If this is not sufficient, continue the cleaning as explained below.
Searching for the name of the broken package (without the leading dash) allowed me to find the following two folders:
C:\Users\name\Anaconda3\Lib\site-packages~atplotlib
C:\Users\name\Anaconda3\Lib\site-packages~atplotlib-3.0.3-py3.7.egg-info
Following Hoefling's comment (below)
I checked the SOURCES.txt file in the egg-info directory %dir%/~atplotlib-3.0.3-py3.7.egg-info/SOURCES.txt. Went through the list of paths in this file and made sure all paths listed did not contained ~. Then I renamed the directory ~atplotlib-3.0.3-py3.7.egg-info into atplotlib-3.0.3-py3.7.egg-info (removed the tilde ~).
Finally, I ran pip uninstall atplotlib, which prompted the following:
Uninstalling atplotlib-3.0.3:
Would remove:
C:\Users\name\Anaconda3\Lib\site-packages\atplotlib-3.0.3-py3.7.egg-info
C:\Users\name\Anaconda3\Lib\site-packages\matplotlib
C:\Users\name\Anaconda3\Lib\site-packages\pylab.py
Proceeding with the removal solved the issue (the warning disappeared and the package is not anymore on the package list.

I received the following error myself, after executing:
python -m pip install --upgrade pip --user
WARNING: Ignoring invalid distribution -ip (c:\python310\lib\site-packages)
I went to the file specification within the C:\
Located the -ip (it was located at the top) and I deleted it.
Executed the code again
Results:
Requirement already satisfied: pip in c:\users\GhostFace\appdata\roaming\python\python310\site-packages (21.3.1)

I had to do rm -rf atplotlib after removing ~ character as beginning of the file name.
$ pip freeze | grep atplotlib
$ cd /home/user/.local/lib/python3.8/site-packages
$ mv \~ atplotlib atplotlib
$ pip freeze | grep atplotlib
atplotlib ==3.0.3
$ pip uninstall atplotlib
WARNING: Skipping atplotlib as it is not installed.
$ rm -rf atplotlib

How do prevent pip and easy_install from removing the temporary directories?

I need to debug some failures to install packages using pip and it seems that once I get the error pip is removing all the temporary files, preventing me from looking inside and finding what happened.
The gist below contains the error that I get but I do want to find-out which config file is generating this error and all the files from tmp are removed by the time the script ends.
https://gist.github.com/ssbarnea/b373062dd45de92735c7482b2735c5fb

From my experience with pip 19.0.3, it is sufficient to use the --no-clean option, which is available to both pip install and pip wheel.
It can also be specified in the respective sections of the pip configuration file:
[install]
no-clean = true
[wheel]
no-clean = true

pip: Missing distribution spec. How to fix this?

I am getting following error while running this command on my virtual machine:
$ sudo pip install -U -v --no-deps -b /tmp/piyush/ /tmp/piyush/common_bundle
File "/usr/local/lib/python2.7/dist-packages/pip-6.0.6-py2.7.egg/pip/_vendor/pkg_resources/__init__.py", line 2807, in parse_requirements
raise ValueError("Missing distribution spec", line)
ValueError: ('Missing distribution spec', '/tmp/piyush/common_bundle')
My input file is an archive.
$ file common_bundle
common_bundle: Zip archive data, at least v2.0 to extract
The version of pip is:
$ pip -V
pip 6.0.6 from /usr/local/lib/python2.7/dist-packages/pip-6.0.6-py2.7.egg (python 2.7)
Here is the pip manifest present in the common_bundle:
# This is a pip bundle file, that contains many source packages
# that can be installed as a group. You can install this like:
# pip this_file.zip
# The rest of the file contains a list of all the packages included:
PyYAML==3.10
boto==2.6.0
msgpack-python==0.2.2
tornado==2.1.1
ujson==1.22
virtualenv==1.8.2
bottle==0.10.7
raven==2.0.3
protobuf==2.4.1
# These packages were installed to satisfy the above requirements:
simplejson==2.4.0
distribute==0.6.32
Does anything look fishy?
Thanks.

When pip installing from an archive, the file must either have the proper extension or you need to explicitly use the file:// protocol. Rename your file to /tmp/piyush/common_bundle.zip or use file:///tmp/piyush/common_bundle.
The error you're seeing has nothing to do with the contents of the archive, pip hasn't even unpacked it yet.

Check to see if there are any spaces in your path. This was my problem. Also try keeping the path to the file as short as possible.

How to state in requirements.txt a direct github source

I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?

Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.

“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils

requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support

First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt

Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.

None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.

Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.

It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.

I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname

For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.