We Keep Coding

pip: dependency installation not houring private package server

My Case:
I have a isolated server(without internet and python 3.8.2) which as python virtual env at /apps/interface/usr/,
All my wheels are stored in https://my.private.org/
I am using pip 19.2.3 to install the packages.
Now,
I have pacakge named py, which has condition like "setup_requires=["setuptools-scm"]," (dependency).
py package is downloaded from https://my.private.org/
now, while trying to download setuptools-scm, its referring to https://files.pythonhosted.org/packages/... instead of https://my.private.org/
I have tried the below methods to redirect the pip to install wheels from https://my.private.org/ .
Point: The main package is getting installed from my personal repo, But the dependency packages is trying to download from public web which we don't have access to.
Method 1:
pip3.8 install py==1.8.1 --index-url https://user:psw#my.private.org/ --extra-index-url https://user:psw#my.private.org/ -v
Method 2:
updated /apps/interface/usr/lib/python3.8/distutils/distutils.cfg
[easy_install]
find_links = https://user:psw#my.private.org/
index-url = https://user:psw#my.private.org/
but no use.
Method 3:
updated below files (tried with each file)
/etc/xdg/pip/pip.conf
/etc/pip.conf
/root/.pip/pip.conf
/root/.config/pip/pip.conf
/apps/interface/usr/pip.conf
Below is the error I am getting
raise DistutilsError("Download error for %s: %s"
distutils.errors.DistutilsError: Download error for https://files.pythonhosted.org/packages/ad/d3/e54f8b4cde0f6fb4f231629f570c1a33ded18515411dee6df6fe363d976f/setuptools_scm-4.1.2-py2.py3-none-any.whl#sha256=69258e2eeba5f7ce1ed7a5f109519580fa3578250f8e4d6684859f86d1b15826: [Errno 101] Network is unreachable
Hoping to find what is the issue.
Note: Below is Error details
Downloading from URL https://my.private.org/py/py-1.8.1.tar.gz (from https://my.private.org/py/)
|████████████████████████████████| 215kB 74kB/s
Added py==1.8.1 from https://my.private.org/py/py-1.8.1.tar.gz to build tracker '/tmp/pip-req-tracker-gohcv4uc'
Running setup.py (path:/tmp/pip-install-xp37jgnc/py/setup.py) egg_info for package py
Running command python setup.py egg_info
Traceback (most recent call last):
File "/usr/local/lib/python3.8/urllib/request.py", line 1319, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/local/lib/python3.8/http/client.py", line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1004, in _send_output
self.send(msg)
File "/usr/local/lib/python3.8/http/client.py", line 944, in send
self.connect()
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/ssl_support.py", line 173, in connect
sock = socket.create_connection(
File "/usr/local/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/local/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 766, in open_url
return open_with_auth(url, self.opener)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 961, in _socket_timeout
return func(*args, **kwargs)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 1080, in open_with_auth
fp = opener(request)
File "/usr/local/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/local/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/local/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/ssl_support.py", line 160, in https_open
return self.do_open(
File "/usr/local/lib/python3.8/urllib/request.py", line 1322, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-xp37jgnc/py/setup.py", line 42, in <module>
main()
File "/tmp/pip-install-xp37jgnc/py/setup.py", line 5, in main
setup(
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 144, in setup
_install_setup_requires(attrs)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/dist.py", line 716, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
File "/apps/interface/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 780, in resolve
dist = best[req.key] = env.best_match(
File "/apps/interface/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1065, in best_match
return self.obtain(req, installer)
File "/apps/interface/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in obtain
return installer(requirement)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/dist.py", line 786, in fetch_build_egg
return cmd.easy_install(req)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 665, in easy_install
dist = self.package_index.fetch_distribution(
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 655, in fetch_distribution
dist = find(requirement)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 635, in find
loc = self.download(dist.location, tmpdir)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 579, in download
found = self._download_url(scheme.group(1), spec, tmpdir)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 824, in _download_url
return self._attempt_download(url, filename)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 830, in _attempt_download
headers = self._download_to(url, filename)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 729, in _download_to
fp = self.open_url(url)
File "/apps/interface/usr/lib/python3.8/site-packages/setuptools/package_index.py", line 779, in open_url
raise DistutilsError("Download error for %s: %s"
distutils.errors.DistutilsError: Download error for https://files.pythonhosted.org/packages/ad/d3/e54f8b4cde0f6fb4f231629f570c1a33ded18515411dee6df6fe363d976f/setuptools_scm-4.1.2-py2.py3-none-any.whl#sha256=69258e2eeba5f7ce1ed7a5f109519580fa3578250f8e4d6684859f86d1b15826: [Errno 101] Network is unreachable
Cleaning up...
Removing source in /tmp/pip-install-xp37jgnc/py
Removed py==1.8.1 from https://packages.zeomega.org/py/py-1.8.1.tar.gz from build tracker '/tmp/pip-req-tracker-gohcv4uc'
Removed build tracker '/tmp/pip-req-tracker-gohcv4uc'
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Finally found the issue,
the easy_install was not honoring index_url configured. Thats main because of the hard coded url in easy_install.py. I changed the url as suggested in below link and its worked fine.
you can find the anwere here

urllib.request.urlopen is giving me errors everytime Python 3.6

Here is my code, very simple.
import urllib.request
x = urllib.request.urlopen('https://www.google.com/')
print(x.read())
and it is giving me this error:
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /Users/myName/Library/Preferences/PyCharmCE2017.1/scratches/scratch_3.py
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/MattAltro/Library/Preferences/PyCharmCE2017.1/scratches/scratch_3.py", line 3, in <module>
x = urllib.request.urlopen('https://www.google.com/')
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>
Process finished with exit code 1
Please help!!

It appears you have a security problem with SSL.
It works for me.
Does it produce the same error all the times?
Have you the lock when you go to https://www.google.com/ in your browser? If not, this is not a python problem, but a connection problem.
In some cases, you want to disable security checks with additional arguments verify=False.

Looks like you are on OS X and Python 3.6. Python 3.6 needs a post-install step to make SSL working. There are two options evidently:
install the certifi package
run the /Applications/Python 3.6/Install Certificates.command
I had my python installed through Homebrew, so I didn't have the .command file, and this tried with just the certifi package. The package would install fine, but SSL still wouldn't work. I ended up downloading Python 3.6 (from python.org) and running the Install Certificates.command. It worked fine from there.

I get CERTIFICATE_VERIFY_FAILED when I try to install the spaCy English language model

I'm running OS X El Capitan on Python 3.5.2 via Anaconda and have spaCy 0.101.0.
I'm trying to install the spaCy English language model using python -m spacy.en.download. However when I do that, I get an error that says urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>. The complete Traceback is as follows:
Traceback (most recent call last):
File "/Users/bsherman/anaconda/lib/python3.5/urllib/request.py", line 1254, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Users/bsherman/anaconda/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/Users/bsherman/anaconda/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/Users/bsherman/anaconda/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/Users/bsherman/anaconda/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/Users/bsherman/anaconda/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/Users/bsherman/anaconda/lib/python3.5/http/client.py", line 1260, in connect
server_hostname=server_hostname)
File "/Users/bsherman/anaconda/lib/python3.5/ssl.py", line 377, in wrap_socket
_context=self)
File "/Users/bsherman/anaconda/lib/python3.5/ssl.py", line 752, in __init__
self.do_handshake()
File "/Users/bsherman/anaconda/lib/python3.5/ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "/Users/bsherman/anaconda/lib/python3.5/ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/bsherman/anaconda/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/Users/bsherman/anaconda/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/spacy/en/download.py", line 13, in <module>
plac.call(main)
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/plac_core.py", line 328, in call
cmd, result = parser.consume(arglist)
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/plac_core.py", line 207, in consume
return cmd, self.func(*(args + varargs + extraopts), **kwargs)
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/spacy/en/download.py", line 9, in main
download('en', force)
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/spacy/download.py", line 24, in download
package = sputnik.install(about.__title__, about.__version__, about.__models__[lang])
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/sputnik/__init__.py", line 37, in install
index.update()
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/sputnik/index.py", line 84, in update
index = json.load(session.open(request, 'utf8'))
File "/Users/bsherman/anaconda/lib/python3.5/site-packages/sputnik/session.py", line 43, in open
r = self.opener.open(request)
File "/Users/bsherman/anaconda/lib/python3.5/urllib/request.py", line 466, in open
response = self._open(req, data)
File "/Users/bsherman/anaconda/lib/python3.5/urllib/request.py", line 484, in _open
'_open', req)
File "/Users/bsherman/anaconda/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/Users/bsherman/anaconda/lib/python3.5/urllib/request.py", line 1297, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Users/bsherman/anaconda/lib/python3.5/urllib/request.py", line 1256, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

Run in terminal:
/Applications/Python 3.5.2/Install Certificates.command
or update certifi package via pip

I've come cross this issue several times before and pip install pip-system-certs did the trick for me.
https://pypi.org/project/pip-system-certs/

How to pass proxy to pythonbrew for downloading setuptools?

I installed pythonbrew on an ubuntu precise system. Had some issues using proxy during the installation (see an earlier post). Now I'm able to install different versions of python and switch between them, however setuptools is not installed during python installation:
ERROR: Failed to install setuptools. See /usr/local/pythonbrew/log/build.log to see why.
Skip installation of setuptools.
The build.log contains:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz
Traceback (most recent call last):
File "distribute_setup.py", line 556, in <module>
sys.exit(main())
File "distribute_setup.py", line 552, in main
tarball = download_setuptools(download_base=options.download_base)
File "distribute_setup.py", line 211, in download_setuptools
src = urlopen(url)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 124, in urlopen
return _opener.open(url, data, timeout)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 389, in open
response = meth(req, response)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 502, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 421, in error
result = self._call_chain(*args)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
result = func(*args)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 597, in http_error_302
return self.parent.open(new)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 383, in open
response = self._open(req, data)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 401, in _open
'_open', req)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
result = func(*args)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 1138, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/local/pythonbrew/pythons/Python-2.6/lib/python2.6/urllib2.py", line 1105, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>
Looks to me like the issue is with the proxy again and I'm not sure what script to modify so that the setuptools installation uses my proxy. I modified distribute_setup.py and added proxy information there (based on this post), but it looks like this file is overwritten each time pythonbrew install is called. I am new to python and can't find where this file is generated from. Any pointers as to how I can pass proxy to the setuptools part of the installation will be greatly appreciated.

Setting HTTP_PROXY and HTTPS_PROXY in the env solved this particular problem (after that the error changed from 110 to 113; for a follow up see this other question).

Installing distribute for Python 3.3

I try to install distribute with python 3.3 on Ubuntu 12.04 to install pymongo.
I downloaded distribute_setup.py with
curl -O http://python-distribute.org/distribute_setup.py
Afterwards I tried
python3 distribute_setup.py install
But I recive
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.49.tar.gz
Traceback (most recent call last):
File "distribute_setup.py", line 556, in <module>
sys.exit(main())
File "distribute_setup.py", line 552, in main
tarball = download_setuptools(download_base=options.download_base)
File "distribute_setup.py", line 211, in download_setuptools
src = urlopen(url)
File "/usr/local/lib/python3.3/urllib/request.py", line 156, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.3/urllib/request.py", line 475, in open
response = meth(req, response)
File "/usr/local/lib/python3.3/urllib/request.py", line 587, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.3/urllib/request.py", line 507, in error
result = self._call_chain(*args)
File "/usr/local/lib/python3.3/urllib/request.py", line 447, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.3/urllib/request.py", line 692, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/local/lib/python3.3/urllib/request.py", line 469, in open
response = self._open(req, data)
File "/usr/local/lib/python3.3/urllib/request.py", line 492, in _open
'unknown_open', req)
File "/usr/local/lib/python3.3/urllib/request.py", line 447, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.3/urllib/request.py", line 1310, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>
Can someone please help me? It seems as if distribute uses code, that does not compile with python 3.3..
I installed Python 3.3.2 in my homefolder following these steps:
./configure
make
make test
sudo make install
If I try to install pymongo from source I recive:
Extracting in /tmp/tmpth91z4
Traceback (most recent call last):
File "setup.py", line 22, in <module>
from setuptools import setup, Feature
ImportError: No module named 'setuptools'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dev/pymongo/ez_setup.py", line 130, in use_setuptools
import pkg_resources
ImportError: No module named 'pkg_resources'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 25, in <module>
use_setuptools()
File "/home/dev/pymongo/ez_setup.py", line 132, in use_setuptools
return _do_download(version, download_base, to_dir, download_delay)
File "/home/dev/pymongo/ez_setup.py", line 111, in _do_download
_build_egg(egg, tarball, to_dir)
File "/home/dev/pymongo/ez_setup.py", line 83, in _build_egg
tar = tarfile.open(tarball)
File "/usr/local/lib/python3.3/tarfile.py", line 1571, in open
raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully
Thank you

Are you sure you installed openssl-devel before compiling Python? I have had the same issue, however I fixed it by previously installing openssl-devel and then compiling Python 3 + distribute.