We Keep Coding

can connect to URL with curl but not with requests (i.e. requests ignoring my CA bundle?)

I am able to connect to a certain URL with cURL, after I installed the corresponding SSL certificates:
$ export MY_URL=https://www.infosubvenciones.es/bdnstrans/GE/es/convocatoria/616783
$ curl -vvvv $MY_URL # Fails
$ sudo openssl x509 -inform pem -outform pem -in /tmp/custom-cert.pem -out /usr/local/share/ca-certificates/custom-cert.crt
$ sudo update-ca-certificates
$ curl -vvvv $MY_URL # OK
However, requests (or httpx, or any other library I use) refuses to do so:
In [1]: import os
...: import requests
...: requests.get(os.environ["MY_URL"])
---------------------------------------------------------------------------
SSLCertVerificationError Traceback (most recent call last)
...
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
My understanding is that requests uses certifi and as such these custom certificates are not available here:
In [1]: import certifi
In [2]: certifi.where()
Out[2]: '/tmp/test_ca/.venv/lib/python3.10/site-packages/certifi/cacert.pem'
I have already tried a number of things, like trying to use the system CA bundle:
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt same error
requests.get(..., verify="/etc/ssl/certs/ca-certificates.crt") same error
switched to httpx + a custom SSL context as explained in the docs, same error
attempted truststore as discussed in this httpx issue, same error
How can I make Python (requests, httpx, raw ssl, anything) use the same certificates that cURL is successfully using?
The only thing that worked so far, inspired by this hackish SO answer, is to do verify=False. But I don't want to do that.
In [9]: requests.get(
...: my_url,
...: verify=False,
...: )
/tmp/test_ca/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

i tried your stuff on my system (Manjaro Linux, python 3.10) i can make a connection. I downloaded the complete certificate chain from the website (with my browser). After that i can use it with:
r = requests.get(url=URL, verify=<path to pem file>)
and with
export REQUESTS_CA_BUNDLE=<path to pem>
r = requests.get(url=URL)
I tried the export within pyCharm.
So the python stuff is working and you may have a problem in your certificates. Without this stuff i get the ssl error (of course), because python does not use the system certs as you mentioned correct. In my pem-file i have 3 certificates. Maybe you have only 1 and the others are in the global store, so that curl does not need the complete chain, instead of python. You should try to download the complete chain with your browser and try again.

How can I solve python openssl load_cert_chain function error?

I'm setting up a new server using Pytoh, tornado.
I'm going to set HTTPS.
But when server loads server certificate files, load_cert_chain function generates an error.
I purchased those certificate files from GoDaddy and it was okay on Apache Server before.
So certificate file and key file match and ca file is also okay.
python version: 3.7.4
tornado version : 4.5.2
openssl version: OpenSSL 1.1.1c 28 May 2019
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl_ctx.load_verify_locations("../ca.crt")
ssl_ctx.verify_mode = ssl.CERT_REQUIRED
https_server = tornado.httpserver.HTTPServer(app, ssl_options=ssl_ctx)
This is error message.
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)

I solved this problem. I use NginX server to host python server with digital signed certificates.

Let's encrypt after installing Django has errors

After installing Django I tried to install SSL using Let's Encrypt with Nginx,
sudo certbot --nginx -d [mydomain.com]
But I get this error,
Domain: [mydomain.com]
Type: unknownHost
Detail: No valid IP addresses found for [mydomain.com]
My domain is accessible and there are A entries.
What is wrong?

Python Requests 'Permission denied' but works with sudo

I am using Python Requests library to connect to a REST server by using .pem certificates to authenticate who I am to establish a connection so I can start collecting data, parsing it, etc. When I run my program via Eclipse or the terminal, I get this error:
[('system library', 'fopen', 'Permission denied'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'system lib')]
However, if I run as 'sudo' - the requests library works as intended and I am able to retrieve the data. Unfortunately, running as 'sudo' has side effects that where the default Python interpreter is the root interpreter, which is Python2. However, there are a lot of library dependencies that are needed from Anaconda.
For context, here is the function I am using to establish a connection:
PEM_FILE = os.path.expanduser("path/to/pem/file.pem") #Path is to a folder in root level
def set_up_connection(self):
#URL's to request a connection with
rest_auth = 'https://www.restwebsite.com/get/data'
ip_address = self.get_ip_address()
body = json.dumps({'userid':'user', 'password':'pass', 'ip_address':ip_address})
try:
resp = self.session.post(rest_auth, data=body, cert=PEM_FILE, verify=False)
values = resp.json()
token = values['token']
except Exception as e:
print(e)
token = None
return token, ip_address
TLDR; Using 'python rest_connector.py' renders an error. Running that command as sudo works.
Context for the certificates: The .pem cert permissions is set to 600 (rw-------).
To try and solve my problem running as sudo, I have started a terminal and run 'sudo -Es' which sets the terminal to run as root and uses Anaconda as my default interpreter, BUT I end up with a handshake error:
[('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert unknown ca'), ('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]
If someone can help me solve it this way, it would be a nice temporary fix, but I still need to be able to run this without sudo.
Thanks, in advance.

The username needs to be able to read the file. You can verify by running ls path_to_file.pem.
If you have changed ownership of the file, you might still be missing executable permissions on the directories containing the file.
You can potentially fix that with chmod -R +x ~/path/to_directory_containing_perm

Ok, so I managed to solve this and I'll post what I did in case anyone else stumbles upon this with a similar problem.
Permissions being set to 600 for the certs and pem file, ownership being set to root, and performing the openssl hashing function, the only problem was where the certs were placed in the sub directories.
While I placed the certs into 'etc/pki/tls/certs', they actually belonged in 'etc/ssl/certs'. The same goes for the .pem file except that .pem would be placed in a restricted folder, 'private'. After moving the files to the correct folder. Then I was able to set the verify param for the request to the cert_path and everything worked like I needed it to.
resp = self.session.post(rest_auth, data=body, cert=PEM_FILE, verify=cert_path)
'etc/pki/tls/certs' is the directory for Fedora distribution of Linux. 'etc/ssl/certs' is the directory for the Ubuntu distribution of Linux.
I hope this helps someone.

Passing crt/pem with paho

At the moment I am trying to figure out how to do an MQTT publish using the .crt and .pem files for our application.
below is some pseudo code for what I am trying to do:
connect to an existing topic over tls
publish a message
import paho.mqtt.publish as mqtt
import paho.mqtt.client as mqttclient
topic = "Some/Topic/Goes/Here"
my_ca_cert = open("<path_to_cert>\ca.crt", 'rb').read()
my_pri_cert = open("<path_to_cert>\private.pem", 'rb').read()
my_key_cert = open("<path_to_cert>\certificate.pem", 'rb').read()
mqttc = mqttclient.Client("Python_Ex_Pub")
mqttc.tls_set(my_ca_cert, certfile=my_pri_cert, keyfile=my_key_cert)
mqttc.connect("<gateway_address>", 8883)
mqttc.publish(topic_name, "This is a test pub from Python.")
mqttc.loop(timeout=2.0, max_packets=1)
When I run the script the following error is thrown:
Traceback (most recent call last):
File "mqtt_pub_test.py", line 9, in <module>
mqttc.tls_set(my_ca_cert, certfile=my_pri_cert, keyfile=my_key_cert)
File "C:\Python27\lib\site-packages\paho\mqtt\client.py", line 557, in tls_set
raise IOError(ca_certs+": "+err.strerror)
IOError: -----BEGIN CERTIFICATE-----
<cert_info_here>
-----END CERTIFICATE-----: No such file or directory
I read through the TLS example on the paho doc page, but just do not understand how I should be passing the crt / pem files in my code.
At one point I was only pointing to the folder containing the files, and even went as far as to chmod 777 the folder but at that point I was getting access denied from python at run time.
Any help is appreshiated

Do you want to do TLS client authentication (that is, your Python script needs to authenticate to the server / MQTT broker)? Or do you want your Python script to behave like a web browser, and just validate the server certificate?
If you only want the latter, I have had success using the tls_set() method in the Paho Python client when I point it to a PEM file containing the server's certificate. And that's the only argument you need to pass on tls_set() to have the Paho client validate the server certificate, and connect to the broker using TLS. For example:
mqttc.tls_set("/home/bob/certificates/mqttbrokercertificate.pem")
How do you obtain the mqtt broker's certificate in PEM format? The easiest way is with openssl:
openssl s_client -host mqtt.broker.hostname.com -port 8883 -showcerts
Redirect the output to a file, and delete every line in the file except what is between the "BEGIN CERTIFICATE" and "END CERTIFICATE" lines (inclusive -- be sure to include these lines as well). This is a good article here on StackOverflow about how to save a server's SSL certificate using openssl:
How to save server SSL certificate to a file
Lastly, you need to be sure of what version of TLS your broker supports, and make sure your Python client also supports it. For example, the IBM Watson IoT Platform requires TLS 1.2. The ssl module in Python 2.7 (which is built on openssl) does not support TLS 1.2. Generally, you need Python 3.X, and openssl of at least 1.0.1. Here is how you can set the TLS version on the Paho client (don't forget to import ssl):
mqttc.tls_set("/home/bob/certificates/mqttbrokercertificate.pem", tls_version=ssl.PROTOCOL_TLSv1_2)
If you want TLS client authentication, that's probably better handled in an entirely separate article. But I hope this helps with TLS server authentication using the Paho Python client.

Looking at the error it would suggest that the tls_set function is expecting paths to the files not the contents of the file to be passed in.