We Keep Coding

poetry install fail build with private dependency

I have a module that depends on another module (module_a) stored in my private repository (nexus) that reuqest another module (module_b) at build time, even stored in the private repository.
I add repository source on poetry.toml in order to add my private repo.
[[tool.poetry.source]]
name = "nexus"
url = "https://my_nexus_url/private_repo/simple"
secondary = true
Then I specify the dependency on th toml
[tool.poetry.dependencies]
python = ">=3.9.0,<3.11,"
module_a="1.0.0"
When I run poetry install it download module_a and build it. During the build process I get this error:
ERROR: Could not find a version that satisfies the requirement module_b==1.0.1
ERROR: No matching distribution found for module_b==1.0.1
When I try to install the module using pip with --extra-index-url <my repo> everything works fine.
pip install module_a --extra-index-url https://my_nexus_url/private_repo/simple
I guess that the problem is releted on the pip command, executed by poetry. It dose not specify the extra-index-url that point to my repo so it try to download dependency (module_b) from pypi repository instead from my repo.
There is a way to instruct poetry in order to use my private repo when source build is required?
I have already try with this:
[tool.poetry.dependencies]
python = ">=3.9.0,<3.11,"
module_b={version="1.0.1", source="nexus"}
module_a={version="1.0.0", source="nexus"}
without any success.

Error pulling packages from gitlab private repository in CI job

As part of a gitlab CI job, I am using pip to install internally-developed packages that are stored using gitlab pypi. When I attempt to run pip install <PACKAGE_NAME>, I get the following error:
WARNING: 401 Error, Credentials not correct for https://<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple/<PACKAGE_NAME>/
ERROR: Could not find a version that satisfies the requirement <PACKAGE_NAME> (from versions: none)
ERROR: No matching distribution found for <PACKAGE_NAME>
I am using a pip.conf file like the following:
[global]
extra-index-url =
https://__token__:<CI_JOB_TOKEN_SUBBED_IN_EXTERNALLY>#<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
The install using the same pip.conf works fine on a workstation using my personal api token instead of the CI Job token. What might be causing this problem?

It turns out that while __token__ works fine for personal tokens, it does not work for CI tokens; it needs to be gitlab-ci-token. The following file works fine:
[global]
extra-index-url =
https://gitlab-ci-token:<CI_JOB_TOKEN_SUBBED_IN_EXTERNALLY>#<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple

Pip install from private Git repo, with Personal access token in Git URL

I am trying to install a package from a private repository on Git.
I am using Personal Access Token in my Git URL in order to bypass the manual authentication step. (You can read about Personal Access Tokens here)
If I add this git URL in requirements file and then use the requirements file in pip to install build it works.
requirements.txt
<package name> # git+https://<Personal Access Token>#<git server address>/<username>/<repository name>.git#<branch name>#egg=<package name>
But, if I use the same URL directly it asks for password, how do I avoid this password prompt (as mentioned below):
pip install git+https://<Personal Access Token>#<git server address>/<username>/<repository name>.git#<branch name>#egg=<package name>
This issue is not observed on all machines that i tested on. It worked on Win 10 x64 and Win 10 x86. But it didn't work on Ubuntu x64. I made sure all the 3 systems has same Python version (3.8.0) and same Pip version (19.3.1).

Use environment variables with the syntax ${VARIABLE} (POSIX format, upper case and underscores allowed) so you're not hard-coding your secrets.
Pip will replace when installing from requirements.txt.
So you can refer to a token to clone the private repo, for example:
in requirements.txt
Github
git+https://${GITHUB_TOKEN}#github.com/user/project.git#{version}
Gitlab
git+https://${GITLAB_TOKEN_USER}:${GITLAB_TOKEN}#gitlab.com/user/project.git#{version}
Bitbucket
git+https://${BITBUCKET_USER}:${BITBUCKET_APP_PASSWORD}#bitbucket.org/user/project.git#{version}
More info here:
https://docs.readthedocs.io/en/stable/guides/private-python-packages.html

Go to GitLab profile settings and generate an read access token:
Select access tokens
give it a name (you can leave expiration date empty)
give it access to read all repositories you have access
generate it
Now edit your requirement file:
pandas==1.0.5
git+https://yourgitlabuser:<generated_token>#gitlab/group/repo#hash#egg=piplib
requests==2.24.0

I just had the same issue. In the end, I could install the package as follows.
from the command line:
pip install mypackagename --no-deps --index-url https://gitlab+deploy-token-mytokenname:tokenpassword#gitlab.mycompany.com/api/v4/projects/123456789/packages/pypi/simple
by specifying it in the requirements.txt file:
(Note that the flask and flask-cors package requirements in the example below are just an example, because it may seem really weird to a reader that the other lines in the example are really content that can be written in a requirements.txt.)
flask==1.1.1
flask-cors==3.0.8
--index-url https://pypi.org/simple --extra-index-url https://gitlab+deploy-token-mytokenname:tokenpassword#gitlab.mycompany.com/api/v4/projects/123456789/packages/pypi/simple
mypackagename
Then of course run pip install -r requirements.txt.
Note that both fragments above show how to provide your password, as you asked.

Certificate error uploading python package to local Artificatory repo [duplicate]

I am new to setuptools on python.
I added a package 'numpy' and 'tensorflow' to install_requires list and running python setup.py install. It does not install due to SSL issue. We use self signed SSL for https based urls.
In case of pip for individual packages I can use --cert option. As I know setuptools use pip indirectly. If so, is there anyway to force it to use --cert option when setup.py is ran?

It turns out that we have actually two problems here.
The problem about tensorflow is related some difference between egg and wheel packages.
To solve the problem about numpy and other packages it is enough to add the following line into ~/.pip/pip.conf file:
[global]
cert=/path/to/cert.crt
If necessarry you may add any other option values (including proxy) with same format.

pip3 setup.py install_requires PEP 508 git URL for private repo

I am trying to run:
pip3 install -e .
in my Python project where I have the following setup.py:
from setuptools import setup
setup(
name='mypackage',
install_requires=[
"anotherpackage#git+git#bitbucket.org:myorg/anotherpackage.git"
]
)
but it fails with:
error in mypackage setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL given
I guess it is correct about the format of my URL as PEP 508 doesn't allow specifying git user name for ssh clone URLs.
What is the correct syntax for PEP 508 URLs with git+ssh protocol for install_requires dependency for private git repositories (in this case hosted on BitBucket)? What is the syntax for specifying a specific branch, tag or sha?
More context to avoid XY problem
I have an internal Python project that depends on multiple internally developed Python packages. I would like to avoid the necessity for hosting my own PIP repository in the organisation and thus I am trying to use git URLs directly. I need to use ssh protocol for git URLs as all the users have their ssh keys configured and it would be cumbersome to ask all the users to configure their app passwords in BitBuckets (I have 2FA required and the regular user password doesn't work).
I have already tried to use:
dependency_links
setup(
name='mypackage',
install_requires=[
"anotherpackage==0.0.1"
],
dependency_links=[
"git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage-0.0.1"
]
)
But they are deprecated and they are ignored by pip3 install -e .. According to documentation I've found, PEP 508 URLs should be used instead.
requirements.txt file with entries duplicated from install_requires entries
I have a requirements.txt file with:
-e git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage
and I use pip3 install -r requirements.txt instead of pip3 install -e .. It works but is suboptimal as I have to keep both setyp.py and requirements.txt in sync.
If there is any other recommended solution for my problem I would like to learn about it :)

After checking pip source code I found the correct syntax for private BitBucket repositories.
The general form for the packages with URLs is <package name>#<URI> and the URI must start with a <scheme>://.
So I fixed it to:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git
and then I was getting a different error - this time git command (invoked by pip) was complaining about repository URL ssh://git#bitbucket.org:myorg/anotherpackage.git.
I checked the git documentation for the ssh:// URLs format and found out that hostname and organisation parts must be separated with / instead of ::
ssh://git#bitbucket.org/myorg/anotherpackage.git
This URL works fine. I also learned from the pip source code that the actual revision/branch/tag can be specified by appending #<rev-spec> so I can specify for example the tag 0.0.1 with the following in install_requires:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git#0.0.1
The only issue that I still have is that when I change the revision and run pip3 install -e . again it doesn't detect the change (even when run with --upgrade). I have to manually uninstall the package (pip3 uninstall anotherpackage) and run pip3 install -e . again.