As part of a gitlab CI job, I am using pip to install internally-developed packages that are stored using gitlab pypi. When I attempt to run pip install <PACKAGE_NAME>, I get the following error:
WARNING: 401 Error, Credentials not correct for https://<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple/<PACKAGE_NAME>/
ERROR: Could not find a version that satisfies the requirement <PACKAGE_NAME> (from versions: none)
ERROR: No matching distribution found for <PACKAGE_NAME>
I am using a pip.conf file like the following:
[global]
extra-index-url =
https://__token__:<CI_JOB_TOKEN_SUBBED_IN_EXTERNALLY>#<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
The install using the same pip.conf works fine on a workstation using my personal api token instead of the CI Job token. What might be causing this problem?
It turns out that while __token__ works fine for personal tokens, it does not work for CI tokens; it needs to be gitlab-ci-token. The following file works fine:
[global]
extra-index-url =
https://gitlab-ci-token:<CI_JOB_TOKEN_SUBBED_IN_EXTERNALLY>#<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
Related
I created a private package for my employer. Since I’m forbidden to upload it to PyPI (it’s proprietary), I uploaded it to the packages index for my project on our private Gitlab hub. I can install it manually with:
$ pip install my-package --extra-index-url https://__token__:my-token-xxx#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple
Now I also want setuptools to be able to find it when listed in the install_requires argument to setup(). I tried:
setup(
install_requires=[
f"my-package # https://__token__:{API_TOKEN}#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple",
...
],
...
pip install -e . results in
ERROR: HTTP error 404 while getting https://__token__:****#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple
This is different than
my-package # git+https://user:password#gitlab.company-domain.com/..../my-package.git
That works, but I want to be able to download it as a pre-built wheel.
I’m not sure whether this is a setuptools issue or a Gitlab issue. The 404 response tells me that it might be a gitlab issue, yet the same URI works perfectly when used with the pip install CLI command.
This question is similar to Include python packages from gitlab's package registry and other external indexes directly into setup.py dependencies, but I don't think that one got sufficient response. I posted the same question to discuss.python.org, but that discussion is old and I think I might get a quicker response here.
I also found this response to a similar question, which wasn't encouraging. It recommends Poetry or Pipenv. I've tried both, and found each to be excruciatingly slow when resolving dependencies, so I fell back on setuptools.
Only include the package name in install_requires. Then, configure your (extra) index URL in your pip configuration (either environment variables or pip.conf/.pypirc or CLI argument). Then using pip install as normal will work.
For example:
In setup.py:
# ...
install_requires=[
'my-package-name',
# ...
],
# ...
Then the install command (assuming the environment variable API_TOKEN exists):
GITLAB_INDEX="https://__token__:${API_TOKEN}#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple"
pip install --extra-index-url "${GITLAB_INDEX}" -e .
I have a private PyPi registry that my company uses to host internal Python packages. I have setup my machine to handle installing dependencies via the private registry with a ~/.pip/pip.conf as such:
[global]
extra-index-url = https://****:****#company.jfrog.io/artifactory/api/pypi/pypi-local/simple
trusted-host = company.jfrog.io
Now, when I try and pip install internal_pkg, all works just fine. However, as I try to pip install pandas (for example), I get the following error:
Looking in indexes: https://pypi.org/simple, https://****:****#parkmobile.jfrog.io/artifactory/api/pypi/pypi-local/simple
Collecting pandas
ERROR: Could not install packages due to an EnvironmentError: Failed to parse: https://****:****#company.jfrog.io/artifactory/api/pypi/pypi-local/simple/pandas/
Note, my username is my company email, hence has # and . symbols.
I am trying to understand why this is failing. How can I setup a pip.conf such that is looks first in my private registry, but looks at pypi.org for example if no package name is found on private.
Thanks!
Like explained here: PIP how escape character # in password?
You can URL encode the 'problematic' character.
In your case # can be encoded into %40
I'm creating a private package repository using PyPiServer. I'm serving this on one VPS instance and trying to pip install from another.
Traceback from pip:
root#managersvr:~# pip install --index-url http://<IP>:8080/simple/ global_toolkit --trusted-host <IP>
Could not find a version that satisfies the requirement global_toolkit (from versions: )
No matching distribution found for global_toolkit
root#managersvr:~#
Server logs from PyPi:
[02/May/2020 23:57:10] "GET /simple/global-toolkit/ HTTP/1.1" 200 313
I'm struggling to debug. Steps taken so far:
- I have run pip install --upgrade pip
- I have navigated to the PyPi IP address in browser, which correctly displays:
Welcome to pypiserver!
This is a PyPI compatible package index serving 1 packages.
To use this server with pip, run the following command:
pip install --index-url http://<IP>:8080/simple/ PACKAGE [PACKAGE2...]
To use this server with easy_install, run the following command:
easy_install --index-url http://<IP>:8080/simple/ PACKAGE [PACKAGE2...]
The complete list of all packages can be found here or via the simple index.
This instance is running version 1.3.2 of the pypiserver software.
Navigating to the "here" link for a complete list of all packages yields:
Index of packages
global_toolkit.tar.gz
And navigating to the simple index yields:
Simple Index
global-toolkit
I also tried using global-toolkit instead of global_toolkit.
I am looking for steps to identify the layer at which there is an error, and debug it. Thanks.
UPDATE
I found the pip search command. The output is, strange:
(venv) root#managersvr:~# pip search --index http://<IP>:8080 global_toolkit
global_toolkit () -
I've resolved the issue. In hindsight I should have foreseen the problem. I failed to include my ext_modules value in the top level setup.py (so the cython modules weren't compiled). A good tip for debugging: use the -v flag!
I am getting the following error when trying to deploy my build release for a python/flask web app:-
I created a python command task with the following -
pip install --upgrade
However, error persists.
I'd suggest you try to disable\suppress this error\warning (because its pretty harmless):
[global]
disable-pip-version-check = True
On many linux the default location for the pip configuration file is $HOME/.config/pip/pip.conf. You can always look that up in the docs.
I use conda environment.yml for CircleCI continuous integration. It all works fine, but now I need to add a private package.
I install it from git via pip, but it can't access the page:
Could not install requirement SE from
https://github.com/organisation/research.git#egg=SE because of error 404
Client Error: Not Found for url:
https://github.com/organisation/research Could not install requirement
SE from https://github.com/organisation/research.git#egg=SE because of
HTTP error 404 Client Error: Not Found for url:
https://github.com/organisation/research for URL
http://github.com/organisation/research.git#egg=SE
So I wonder, how can I pass / setup github credentials to/in the conda environment?
Just to be clear, this package dependency is happily tested on CircleCI by itself...
Actually, CircleCi offers to add your SSH key in project settings.