We Keep Coding

Python Docker SDK "Error while fetching server API version"

In the Python Docker SDK, When I do
import docker
docker.from_env()
I see
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
I have docker desktop running and this works in the terminal
$ docker run -it ubuntu
If I add a version number
docker.from_env(version="6.0.1")
it stops erroring, but it doesn't seem to matter what number I use. I also then see an error on
client.containers.run("ubuntu")
of
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
I'm on Ubuntu 22.04 and I'm seeing the problem with both Poetry and plain pip + venv. I've looked through the dozen or so questions about that error message and tried everything that looked relevant.

My final solution was to uninstall Docker Desktop and install Docker Engine instead. I had to manually delete some Docker config files left over after installation too.

SSL error only in python command window with apify request

I am trying to use endpoint from apify.com. When I run my request in web browser with token everything is fine but if I run my request via requests library from python console I am getting following error:
SSLError: HTTPSConnectionPool(host='', port=443): Max retries exceeded with url: /endpoint?token=token (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))
Moreover if I set verify = False in my request than request is working. Does anyone have an idea what can be wrong? Thanks in advance

I had this issue come up a few weeks ago.
>>> pip install certifi
>>> python -m certifi
I'm not certain that one needs to actually call the module to get it's functionality, but I did and it solved the error. More info on Certifi here. It is also a recommended package extension to requests from their website. I added those lasts bits because I was wary of installing a package that ostensibly was never called after installation.

Solution was to install internal company SSL package for managing SSL connection from python. There was a recent change.

How to fix requests.get() failure on gitlab rest api?

I am using python reqests.get() to invoke one of gitlab api's in my Python script running on Windows. It works fine on my dev machine, but it fails with below error message when I run the script in TeamCity job runner.
HTTPSConnectionPool(host='gitlab.myservice.com', port=443): Max retries exceeded with url: /api/v4/projects/1111/pipeline_schedules/111?private_token=******* (Caused by SSLError(SSLError(1, '_ssl.c:503: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'),))
What is very strange is that very similar script that calls Artifactory rest api runs just fine in the same script even when it runs in TeamCity.
Can anybody shed light on this mystery?
Update
The infra is using Python 2.7.16. When I tested with Python 3.x it worked fine. However, the org is not ready to move to 3.x due to dependencies they cannot solve yet.

Databricks CLI: SSLError, can't find local issuer certificate

I have installed and configured the Databricks CLI, but when I try using it I get an error indicating that it can't find a local issuer certificate:
$ dbfs ls dbfs:/databricks/cluster_init/
Error: SSLError: HTTPSConnectionPool(host='dbc-12345678-1234.cloud.databricks.com', port=443): Max retries exceeded with url: /api/2.0/dbfs/list?path=dbfs%3A%2Fda
tabricks%2Fcluster_init%2F (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer
certificate (_ssl.c:1123)')))
Does the above error indicate that I need to install a certificate, or somehow configure my environment so that it knows how to find the correct certificate?
My environment is Windows 10 with WSL (Ubuntu 20.04) (the command above is from WSL/Ubuntu command line).
The Databricks CLI was installed into an Anaconda environment including the following certificates and SSL packages:
$ conda list | grep cert
ca-certificates 2020.6.20 hecda079_0 conda-forge
certifi 2020.6.20 py38h32f6830_0 conda-forge
$ conda list | grep ssl
openssl 1.1.1g h516909a_1 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
I get a similar error when I attept to use the REST API with curl:
$ curl -n -X GET https://dbc-12345678-1234.cloud.databricks.com/api/2.0/clusters/list
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This problem can be solved by disabling the SSL certificate verification. In Databricks CLI you can do so by specifying insecure = True in your Databricks configuration file .databrickscfg.

I established trust to my Databricks instance by setting the environment variable REQUESTS_CA_BUNDLE.
➜ databricks workspace list
Error: SSLError: HTTPSConnectionPool(host='HOSTNAME.azuredatabricks.net', port=443): Max retries exceeded with url: /api/2.0/workspace/list?path=%2F (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
➜ export REQUESTS_CA_BUNDLE=path/to/ca-bundle
➜ databricks workspace list
Users
Shared
Repos
From GitHub Issue:
Download the root CA certificate used to sign the Databricks certificate. Determine the path to the CA bundle and set the environment variable REQUESTS_CA_BUNDLE. See SSL Cert Verification for more information.

There is a similar issue in GitHub for Azure CLI. The solution is practically the same. Combining that with the Erik's answer:
Download the certificate using your browser and save it to disk
Open you Chrome and go to the Databricks website
Press CTRL + SHIFT + I to open the dev tools
Click Security tab
Click View certificate button
Click Details tab
On the Certification Hierarchy, (the top panel), click the highest node in the tree
Click Export the selected certificate
Choose where you want to save (eg. /home/cert/certificate.crt)
Use the SET command on Windows or the export on Linux to create a env variable called REQUESTS_CA_BUNDLE and point it to the downloaded file in the Step 1. (keep in mind that this need to be done in the same machine as you are trying to use the dbfs not in the cluster) For instance:
Linux
export REQUESTS_CA_BUNDLE=/home/cert/certificate.crt
Windows
set REQUESTS_CA_BUNDLE=c:\temp\cert\certificate.crt
Try to run your command dbfs ls dbfs:/databricks/cluster_init/ again
$ dbfs ls dbfs:/databricks/cluster_init/
It should work!

How to install Python packages over SSH Port Forwarding?

I am controlling a remote unit over SSH and OPENVPN.
On the remote unit I want to install some Python packages using pip but:
the remote company firewall allows only traffic on port 22 (and not 443, needed by pip);
DNS is not installed on the remote unit;
I cannot modify any OPENVPN settings (or I would like to avoid this option as it means to access some remote sysadmin and try to convince him that the configuration must be changed);
all systems are Linux (Ubuntu + Debian). Non Windows involved.
Stripping down hours of attempts (I am not a system admin and my knowledge on this subject is very limited), the idea was to open an obvious SSH port forwarding:
ssh -R 9999:pypi.python.org:443 xxxx#XX.XX.XX.XX
and then, on the remote unit play with pip install:
pip install pymodbus==1.3.2 --proxy localhost:9999
But this command returns:
Cannot fetch index base URL https://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement pymodbus==1.3.2
/root/.pip/pip.log is:
Getting page https://pypi.python.org/simple/pymodbus/
Could not fetch URL https://pypi.python.org/simple/pymodbus/: connection error: ('Connection aborted.', BadStatusLine("''",))
Will skip URL https://pypi.python.org/simple/pymodbus/ when looking for download links for pymodbus==1.3.2
Getting page https://pypi.python.org/simple/
Could not fetch URL https://pypi.python.org/simple/: connection error: ('Connection aborted.', BadStatusLine("''",))
Will skip URL https://pypi.python.org/simple/ when looking for download links for pymodbus==1.3.2
Cannot fetch index base URL https://pypi.python.org/simple/
URLs to search for versions for pymodbus==1.3.2:
* https://pypi.python.org/simple/pymodbus/1.3.2
* https://pypi.python.org/simple/pymodbus/
Getting page https://pypi.python.org/simple/pymodbus/1.3.2
Could not fetch URL https://pypi.python.org/simple/pymodbus/1.3.2: connection error: ('Connection aborted.', BadStatusLine("''",))
Will skip URL https://pypi.python.org/simple/pymodbus/1.3.2 when looking for download links for pymodbus==1.3.2
Getting page https://pypi.python.org/simple/pymodbus/
It is obvious the remote unit cannot read the index page on pypi.pthon.org because the connection is refused.
What is the correct syntax for what I am trying to achieve?

Proxy is going to be tricky. I suggest that you scp the pip module source file and install it locally from source. Use
pip install package —download="/pth/to/downloaded/file” to get the package, scp it to the dest server and use pip install “/pth/to/scp/file”

It's look like my problem. after exploration, I have found a solution.
And because in my region, pypi.python.org is slow, so I change my pip.conf and use pypi.douban.com/simple, as my index-url. this website use http protocol. so in my solution. I use 80 port as my target port.
Problem:
I have two host. host1 could connect Pypi.douban.com. and host2 couldn't.
but I can connect host2 in host1 through ssh.
so in host2, I open a tmux session and open a ssh tunnel by local port forwarding(not remote port forwarding):
ssh -L 9999:pypi.douban.com:80 username#host1
after this redirect, I can use
pip install scikit-learn --proxy localhost:9999
to install package in host2.