I use conda environment.yml for CircleCI continuous integration. It all works fine, but now I need to add a private package.
I install it from git via pip, but it can't access the page:
Could not install requirement SE from
https://github.com/organisation/research.git#egg=SE because of error 404
Client Error: Not Found for url:
https://github.com/organisation/research Could not install requirement
SE from https://github.com/organisation/research.git#egg=SE because of
HTTP error 404 Client Error: Not Found for url:
https://github.com/organisation/research for URL
http://github.com/organisation/research.git#egg=SE
So I wonder, how can I pass / setup github credentials to/in the conda environment?
Just to be clear, this package dependency is happily tested on CircleCI by itself...
Actually, CircleCi offers to add your SSH key in project settings.
Related
As part of a gitlab CI job, I am using pip to install internally-developed packages that are stored using gitlab pypi. When I attempt to run pip install <PACKAGE_NAME>, I get the following error:
WARNING: 401 Error, Credentials not correct for https://<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple/<PACKAGE_NAME>/
ERROR: Could not find a version that satisfies the requirement <PACKAGE_NAME> (from versions: none)
ERROR: No matching distribution found for <PACKAGE_NAME>
I am using a pip.conf file like the following:
[global]
extra-index-url =
https://__token__:<CI_JOB_TOKEN_SUBBED_IN_EXTERNALLY>#<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
The install using the same pip.conf works fine on a workstation using my personal api token instead of the CI Job token. What might be causing this problem?
It turns out that while __token__ works fine for personal tokens, it does not work for CI tokens; it needs to be gitlab-ci-token. The following file works fine:
[global]
extra-index-url =
https://gitlab-ci-token:<CI_JOB_TOKEN_SUBBED_IN_EXTERNALLY>#<GITLAB_URL>/api/v4/projects/<PROJECT_ID>/packages/pypi/simple
I created a private package for my employer. Since I’m forbidden to upload it to PyPI (it’s proprietary), I uploaded it to the packages index for my project on our private Gitlab hub. I can install it manually with:
$ pip install my-package --extra-index-url https://__token__:my-token-xxx#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple
Now I also want setuptools to be able to find it when listed in the install_requires argument to setup(). I tried:
setup(
install_requires=[
f"my-package # https://__token__:{API_TOKEN}#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple",
...
],
...
pip install -e . results in
ERROR: HTTP error 404 while getting https://__token__:****#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple
This is different than
my-package # git+https://user:password#gitlab.company-domain.com/..../my-package.git
That works, but I want to be able to download it as a pre-built wheel.
I’m not sure whether this is a setuptools issue or a Gitlab issue. The 404 response tells me that it might be a gitlab issue, yet the same URI works perfectly when used with the pip install CLI command.
This question is similar to Include python packages from gitlab's package registry and other external indexes directly into setup.py dependencies, but I don't think that one got sufficient response. I posted the same question to discuss.python.org, but that discussion is old and I think I might get a quicker response here.
I also found this response to a similar question, which wasn't encouraging. It recommends Poetry or Pipenv. I've tried both, and found each to be excruciatingly slow when resolving dependencies, so I fell back on setuptools.
Only include the package name in install_requires. Then, configure your (extra) index URL in your pip configuration (either environment variables or pip.conf/.pypirc or CLI argument). Then using pip install as normal will work.
For example:
In setup.py:
# ...
install_requires=[
'my-package-name',
# ...
],
# ...
Then the install command (assuming the environment variable API_TOKEN exists):
GITLAB_INDEX="https://__token__:${API_TOKEN}#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple"
pip install --extra-index-url "${GITLAB_INDEX}" -e .
I'm having an issue where when I install the requests package on a fresh anaconda install (onto an environment), it breaks my anaconda in a way where I cannot download any further packages due to an HTTP error.
The process I've gone through a number of times now is:
Uninstall anaconda (using anaconda-clean and add/remove programs)
Re-install anaconda
Run conda update conda on my base environment
Run conda create -n auckland-index python=3.7 to create a new environment
I install pandas with conda install pandas to make sure I can download packages in the new environment
I then run conda install requests to install requests, which downloads and installs successfully
Then when I try to install any other packages I get the below CondaHTTPError across both base and new environments
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/win-64/current_repodata.json>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.
'https://repo.anaconda.com/pkgs/main/win-64'
I have tried installing requests before pandas and after other packages, but I get an CondaHTTPError always after installing requests.
I have tried the following from other stack questions to fix (of which none work):
Set ssl_verify: false in .condarc
Copy both libcrypto-1_1-x64.dll and libssl-1_1-x64.dll to the anaconda3/DLLs folder
I'm at a bit of a loss as to why installing requests is causing this issue, as I'm downloading it through the conda install command. This isn't a network issue either as the downloading and installing of packages was working before requests was installed. Once I have installed requests I cannot download any more packages for any environment.
Has anyone had this issue or know of a fix? Is this a bug?
EDIT: from each environment I have run conda list requests and python -c "import requests; print(requests.__spec__)", with the results below.
Base
# Name Version Build Channel
requests 2.22.0 py37_0
ModuleSpec(name='requests', loader=<_frozen_importlib_external.SourceFileLoader object at 0x00000284138E7FC8>, origin='%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\requests\\__init__.py', submodule_search_locations=['%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\requests'])
auckland-index
# Name Version Build Channel
requests 2.25.1 pyhd3eb1b0_0
ModuleSpec(name='requests', loader=<_frozen_importlib_external.SourceFileLoader object at 0x00000284138E7FC8>, origin='%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\requests\\__init__.py', submodule_search_locations=['%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\requests'])
EDIT 2: Adding path variables from base environment
['', '%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\python37.zip',
'%USERPROFILE%\\open-data-api',
'%USERPROFILE%\\Auckland-Index-Update',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\DLLs',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index',
'%USERPROFILE%',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\locket-0.2.1-py3.7.egg',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\win32',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\win32\\lib',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\Pythonwin',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\envs\\auckland-index\\lib\\site-packages\\IPython\\extensions',
'%USERPROFILE%\\.ipython',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\python37.zip',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\DLLs',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\lib',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\win32',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\win32\\lib',
'%USERPROFILE%\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\Pythonwin']
Issue was caused by PYTHONPATH windows environment variable, once this was deleted problem was solved. Thanks to #merv for help getting there.
So I am trying to install ipywidgets-0.0.1 into my Anaconda environment. I am running Anaconda3. Unfortunately, the machine I need to install it on does not have internet access.
The basic way to install the package is to run conda install ipywidgets, but of course this gives the following error:
Fetching package metadata: Could not connect to https://repo.continuum.io/pkgs/p
ro/noarch/
Could not connect to https://repo.continuum.io/pkgs/free/win-64/
Could not connect to https://repo.continuum.io/pkgs/pro/win-64/
.Could not connect to https://repo.continuum.io/pkgs/free/noarch/
...
Solving package specifications: .
Package plan for installation in environment C:\Program Files\Anaconda3:
The following packages will be downloaded:
package | build
---------------------------|-----------------
ipywidgets-0.0.1 | np19py34_0 7 KB
The following NEW packages will be INSTALLED:
ipywidgets: 0.0.1-np19py34_0
Proceed ([y]/n)? y
Fetching packages ...
Could not connect to https://repo.continuum.io/pkgs/free/win-64/ipywidgets-0.0.1
-np19py34_0.tar.bz2
Error: Connection error: HTTPSConnectionPool(host='repo.continuum.io', port=443)
: Max retries exceeded with url: /pkgs/free/win-64/ipywidgets-0.0.1-np19py34_0.t
ar.bz2 (Caused by ProtocolError('Connection aborted.', gaierror(11004, 'getaddri
nfo failed'))):
https://repo.continuum.io/pkgs/free/win-64/ipywidgets-0.0.1-np19py34_0.tar.bz2
This makes sense, since the box can't access the internet.
I downloaded the ipywidgets tarball from the listed URL at https://repo.continuum.io/pkgs/free/win-64/ipywidgets-0.0.1-np19py34_0.tar.bz2, and I ssh'ed it onto the box.
What command can I run on the box (it is a Windows 7 environment) so that it will run the install as local, looking at the downloaded tarball to unpack rather than trying to access the internet to get the file?
Download the whole package and its dependencies to some local directory
You can get it from PYPI ipywidgets
Let's consider you donwloaded from PYPI and you have ~/src/ipywidgets with ~/src/ipywidgets/setup.py which will have install_requires with a list of dep. in your /pypi directory. Like so:
install_requires=['hello', 'etc', ...]
To install your app and its dep. into the current python env., you've to do something like:
$ cd ~/src/ipywidgets
$ easy_install --always-unzip --allow-hosts=None --find-links=/pypi
Note that the build will fail if one or more dependencies are missing
If you want to use conda for this, might as well if you are already using anaconda, you should also be able to do:
conda install --offline PATH_TO_FILE/ipywidgets-0.0.1-np19py34_0.tar.bz2
Where, of course, PATH_TO_FILE is just the path to the file you've downloaded.
I'm trying to configure a notification for a Google Cloud Storage bucket object change however I'm stuck when trying to make gsutil use a service account with this command.
gcloud auth activate-service-account service-account-email --key-file path/to/key.p12
The error I get in the command line is:
ERROR: (gcloud.auth.activate-service-account) PyOpenSSL is not available. If you have already installed PyOpenSSL, you will need to enable site packages by setting the environment variable CLOUDSDK_PYTHON_SITEPACKAGES to 1. If that does not work, See https://developers.google.com/cloud/sdk/crypto for details.
I followed this instructions to get pyOpenSSL. If I ask pip about that package it tells me it is installed
$ pip show pyopenssl
---
Name: pyOpenSSL
Version: 0.14
Location: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Requires: cryptography, six
I also see the environment variable if I call the env command
$ env
...
CLOUDSDK_PYTHON_SITEPACKAGES=1
Am I doing something wrong?
This indicates that PyOpenSSL was not properly installed. Since PyOpenSSL includes crypto routines, the Cloud SDK can not easily package it and we rely on third-party installation.