We Keep Coding

Getting self signed certificate error with pip install within Docker, but only for certain packages

I'm just playing with a simple example to get a basic understanding of Docker going. Here is my Docker image file:
FROM python:3.7-alpine
# copy all the files to the container
COPY . /test
WORKDIR /test
# install dependencies
RUN pip install pip_system_certs --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org
RUN pip install -r requirements.txt
# run the command
CMD ["python", "./test_script.py"]
The trusted-host options are what allow us to get around corporate network security settings and install packages internally on windows and they seem to work in Docker too but only for some packages. For instance if my requirements.txt includes flask and requests everything is fine, but pandas and numpy give me
WARNING: Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'SSLError(SSLCertVerificationError(1, '[SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed
certificate in certificate chain (_ssl.c:1076)'))': /simple/numpy/
and fails. I think it's weird that this is working for some packages but not others.
Any help appreciated.
Using Docker Desktop in Windows 10.

I know my company's big corporate proxy removes (most) normal certificates and re-wraps them in a self-signed cert. This caused lots of similar headaches for me. I resolved it by:
Figuring out what our root cert was by visiting an internet site in Chrome, clicking on the lock in the address bar, and viewing the certification path for the site's certificate. The root CA was our internal one.
Going to the certificate management in Windows control panel and under "Trusted Root Certification" found my company's internal root cert and exported it as a "Base-64 encoded X.509" file.
Copied that certificate file into my Docker container and added it as a CA certificate to the "os" inside my container. After that, everything I ran in my container just worked.
The catch with step 3 here is that exactly how you do this is different for different flavors of linux. I don't know much about alpine, but these links might get you pointed in roughly the right direction:
https://blog.confirm.ch/adding-a-new-trusted-certificate-authority/
https://github.com/gliderlabs/docker-alpine/issues/260
Also, bonus catch - if you use python's requests library in your application, it doesn't use the system CA certs by default. If this is a problem for you, read about setting the REQUESTS_CA_BUNDLE in the accepted answer here: Python Requests - How to use system ca-certificates (debian/ubuntu)?

Installation of python packages from terminal started to fail, Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

I was not able to solve this for days, so if you can give me any advice I would appreciate it!
When I try to install any package from python I get this message (in this case lets take flask for example):
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/flask/
Could not find a version that satisfies the requirement Flask (from versions: )
No matching distribution found for Flask "
I do not really understand what caused it because previously everything worked well and I was able to install packages. And, I did not install anything new in between. Also, I am really a beginner in python programming and I do not fully understand what exactly the message is trying to say. I definitely have an internet connection and I have installed e.g. Flask previously and have used it as well. The only thing that happened before everything got stack was that my internet provider has "rebooted" my internet connection (I am not sure that "reboot" is the right termini for that).
I am using Windows subsystem for Linux, Visual studio code and python version: Python 3.6.8 (default, Oct 7 2019, 12:59:55) [GCC 8.3.0] on Linux.

This sounds like a DNS issue. Pip would be trying to resolve https://pypi.org. But ultimately that has to resolve to an IP Address.
Most ISPs provide a DNS service by default. Perhaps it wasn't able to resolve it?
When you added nameserver 8.8.8.8 you're using a Google DNS server which, it seems worked as expected by resolving https://pypi.org

I solved it by adding:
nameserver 8.8.8.8
nameserver 8.8.4.4
to the file:
/etc/resolv.conf
(via $sudo nano /etc/resolv.conf and save)
BUT I still do not understand fully why it worked? and Where came the problem from?

Private PyPi Repository in Closed Network

I am on a closed network (aka no access to the internet) thus I need to create my own PyPi server. I am following the instructions here:
How to set up and use a private PyPI repo
The only differences:
I did not do was set up Apache authentication. No need for it.
I put everything in /src/pypi (all pointers were updated)
I was able to get Apache operational and (supposedly) got my ~/.pip/pip.conf file operational. Here are the contents:
[global]
extra-index-url=https://pypi.myserver.com/pypi/
trusted-host = pypi.myserver.com
When I try to run the command:
pip install foobar-utils
I get the following:
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at xxx>: Failed to establish a new connection: [Errorno -2] Name or service not known',)' /simple/foobar-utils
I know it's hitting apache, I see it in my logs.
Any help would be great. Thanks

If you want to completely replace PyPI you must use index-url instead of extra-index-url. With extra-index-url pip still search packages at https://pypi.org, extra-index-url is an additional server to search.
Your pip.conf must be
[global]
index=https://pypi.myserver.com/pypi/
index-url=https://pypi.myserver.com/pypi/
trusted-host = pypi.myserver.com
index is for pip search, index-url for pip install.

How to install Python packages over SSH Port Forwarding?

I am controlling a remote unit over SSH and OPENVPN.
On the remote unit I want to install some Python packages using pip but:
the remote company firewall allows only traffic on port 22 (and not 443, needed by pip);
DNS is not installed on the remote unit;
I cannot modify any OPENVPN settings (or I would like to avoid this option as it means to access some remote sysadmin and try to convince him that the configuration must be changed);
all systems are Linux (Ubuntu + Debian). Non Windows involved.
Stripping down hours of attempts (I am not a system admin and my knowledge on this subject is very limited), the idea was to open an obvious SSH port forwarding:
ssh -R 9999:pypi.python.org:443 xxxx#XX.XX.XX.XX
and then, on the remote unit play with pip install:
pip install pymodbus==1.3.2 --proxy localhost:9999
But this command returns:
Cannot fetch index base URL https://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement pymodbus==1.3.2
/root/.pip/pip.log is:
Getting page https://pypi.python.org/simple/pymodbus/
Could not fetch URL https://pypi.python.org/simple/pymodbus/: connection error: ('Connection aborted.', BadStatusLine("''",))
Will skip URL https://pypi.python.org/simple/pymodbus/ when looking for download links for pymodbus==1.3.2
Getting page https://pypi.python.org/simple/
Could not fetch URL https://pypi.python.org/simple/: connection error: ('Connection aborted.', BadStatusLine("''",))
Will skip URL https://pypi.python.org/simple/ when looking for download links for pymodbus==1.3.2
Cannot fetch index base URL https://pypi.python.org/simple/
URLs to search for versions for pymodbus==1.3.2:
* https://pypi.python.org/simple/pymodbus/1.3.2
* https://pypi.python.org/simple/pymodbus/
Getting page https://pypi.python.org/simple/pymodbus/1.3.2
Could not fetch URL https://pypi.python.org/simple/pymodbus/1.3.2: connection error: ('Connection aborted.', BadStatusLine("''",))
Will skip URL https://pypi.python.org/simple/pymodbus/1.3.2 when looking for download links for pymodbus==1.3.2
Getting page https://pypi.python.org/simple/pymodbus/
It is obvious the remote unit cannot read the index page on pypi.pthon.org because the connection is refused.
What is the correct syntax for what I am trying to achieve?

Proxy is going to be tricky. I suggest that you scp the pip module source file and install it locally from source. Use
pip install package —download="/pth/to/downloaded/file” to get the package, scp it to the dest server and use pip install “/pth/to/scp/file”

It's look like my problem. after exploration, I have found a solution.
And because in my region, pypi.python.org is slow, so I change my pip.conf and use pypi.douban.com/simple, as my index-url. this website use http protocol. so in my solution. I use 80 port as my target port.
Problem:
I have two host. host1 could connect Pypi.douban.com. and host2 couldn't.
but I can connect host2 in host1 through ssh.
so in host2, I open a tmux session and open a ssh tunnel by local port forwarding(not remote port forwarding):
ssh -L 9999:pypi.douban.com:80 username#host1
after this redirect, I can use
pip install scikit-learn --proxy localhost:9999
to install package in host2.

easy_install ReviewBoard [Errno 104] Connection reset by peer

I have a Kubuntu 10.04 VM image and am trying to install ReviewBoard by following The Linux Installation Wiki. When I get to the step to easy_install ReviewBoard, I encounter a problem I cannot find a solution to. Below is the console output:
>> sudo easy_install ReviewBoard
Searching for ReviewBoard
Best match: ReviewBoard 1.0.8
Processing ReviewBoard-1.0.8-py2.6.egg
ReviewBoard 1.0.8 is already the active version in easy-install.pth
Installing rb-site script to /usr/local/bin
Using /usr/local/lib/python2.6/dist-packages/ReviewBoard-1.0.8-py2.6.egg
Processing dependencies for ReviewBoard
Searching for pytz
Reading http://downloads.reviewboard.org/mirror/
Download error: [Errno 104] Connection reset by peer -- Some packages may not be found!
Reading http://downloads.reviewboard.org/releases/ReviewBoard/1.0/
Download error: [Errno 104] Connection reset by peer -- Some packages may not be found!
Reading http://pypi.python.org/simple/pytz/
Download error: [Errno 104] Connection reset by peer -- Some packages may not be found!
Reading http://pypi.python.org/simple/pytz/
Download error: [Errno 104] Connection reset by peer -- Some packages may not be found!
Couldn't find index page for 'pytz' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
Download error: [Errno 104] Connection reset by peer -- Some packages may not be found!
No local packages or download links found for pytz
error: Could not find suitable distribution for Requirement.parse('pytz')
I am new to python, but it seems like easy_install cannot decide on a version of pytz. I have read
104, 'Connection reset by peer' socket error, or When does closing a socket result in a RST rather than FIN?
http://homepage.mac.com/s_lott/iblog/architecture/C551260341/E20081031204203/index.html
and it seems like the problem described in those articles has more to do with development than my problem, but I could be wrong.
Has anyone encountered a problem like this? If there is any missing information that would help troubleshoot this, please let me know.
#Ben Hoffstein
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Link partner advertised link modes: Not reported
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: No
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
Link detected: yes
EDIT:
I'm not sure if this will be more useful information, but I tried installing the gitplugin for Trac and I received the same issue. So, it's looking like it may be a problem with easy_install.
>> sudo easy_install http://trac-hacks.org/svn/gitplugin/0.11
Downloading http://trac-hacks.org/svn/gitplugin/0.11
error: Download error for http://trac-hacks.org/svn/gitplugin/0.11: [Errno 104] Connection reset by peer

This is possibly a temporary problem with PyPI host, try again: easy_install pytz

like #Tarantula said, it is a temporary pypi service problem, i hit the same yesterday. i tried to ping pypi.python.org (or dig or host) a few times and find there are 2 servers returned by dns server. and it is because one of the pypi server keep throwing this error for one of the python package request.
so i hard coded the mapping pypi.python.org to the good one in /etc/hosts basically force pypi.python.org to resolve to the server i wanted and it relieved my pain. of course it is a temporary workaround when we have no control of the external service.
# add a line to /etc/hosts
151.101.200.223 pypi.python.org
get rid of the line when the service get back to normal.