We Keep Coding

Serving over HTTPS with Gunicorn/Flask: ERR_CERT_AUTHORITY_INVALID

Looking for a quick way to serve an API over HTTPS for testing purposes. The API app is created using flask and being served on port 443 using gunicorn.
gunicorn --certfile=server.crt --keyfile=server.key --bind 0.0.0.0:443 wsgi:app
When my React app (served over HTTPS) sends a POST request to one of the routes via HTTPS, the browser console is showing
POST https://1.2.3.4/foo net::ERR_CERT_AUTHORITY_INVALID
My key and certs are created using
openssl genrsa -aes128 -out server.key 2048
openssl rsa -in server.key -out server.key
openssl req -new -days 365 -key server.key -out server.csr
openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 365
Is there a solution to solve ERR_CERT_AUTHORITY_INVALID raised by the browser, without using a reverse proxy like nginx/caddy? And without each user having to manually trust the self-signed cert?

Your browser/computer/device need to trust the certificate presented by gunicorn...
You should add the hostname of your PC in the certificate (Common name or Subject Alternative Name) and add the Certificate to your Trusted List of Certificates

i ran into a similar problem recently on firefox creating the cert using open ssl.
i opted for an alternative solution using mkcert
sudo apt install libnss3-tools
sudo apt install mkcert
wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.4/mkcert-v1.4.4-linux-amd64
sudo cp mkcert-v1.4.4-linux-amd64 /usr/local/bin/mkcert
sudo chmod +x /usr/local/bin/mkcert
mkcert -install
mkcert test.example.com '*.test.example.com' localhost 127.0.0.1 ::1
you'll want to modify /etc/hosts to include test.example.com
127.0.0.1 localhost test.example.com
don't forget to logout and log back in to update changes in hosts
if firefox still complains go to settings -> privacy/security and open View Certificates.
under the server tab, add an exception for https://test.example.com:(port #) and select Get Certificate.
then Confirm Security Exception
now fire up gunicorn using the pem format files generated by mkcert.
in my case it was something like...
gunicorn --certfile test.example.com+4.pem --keyfile test.example.com+4-key.pem
your cert should be accepted now.
each member of our team has to set this up locally. (specifically, we use an installer script to build the dev project, but the dev is responsible for installing the cert on the browser of their choosing.)
for us it was a small inconvenience for the payoff.
if this doesn't suit your needs then unfortunately yes, you might have to opt for an alternative such as caddy or nginx to reverse-proxy your requests. but you'd still have to supply a certificate using some version of the example above or via tools like certbot ect
i'd recommend a pre-config'd docker container, or a custom installer script if you're working on a team based project.

Mqtt TLS certificate verify failed : self signed certificate

Am trying to implement TLS for mqtt and has followed the tutorials from the link below
http://www.steves-internet-guide.com/mosquitto-tls/
I followed exactly how it has been instructed to generate certificates using openssl and pasted in the location of mqtt and changed the conf of mqtt and restarted the service.
But when I try to connect to mqtt using tls it shows the below error message
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1124)
And python code is
client1 = paho.Client("control1")
client1.tls_set(ca_certs="ca.crt")
client1.tls_insecure_set(True)
client1.connect("localhost", 8883)
client1.loop_forever()
where ca.crt is in the project directory.

The message you are receiving indicates that the broker's server certificate is not trusted (because it is self-signed), therefore paho is not being correctly told it is trustworthy.
It is possible your fake certificate authority's root certificate (the ca.crt file you feed to paho) is not properly signed or generated, or the certificates that Mosquitto is using are not signed correctly. Either way, you likely need to start the entire process over to be 100% certain everything was done right.
Generate the fake certificate authority's (CA) signing key
$ openssl genrsa -des3 -out ca.key 2048
Generate a certificate signing request for the fake CA
$ openssl req -new -key ca.key -out ca-cert-request.csr -sha256
Give the organization a name like "Fake Authority" and do not enter a common name (since your fake CA does not actually live on a server with a name)
Create the fake CA's root certificate
$ openssl x509 -req -in ca-cert-request.csr -signkey ca.key -out ca-root-cert.crt -days 365 -sha256
Create the server / mqtt broker's keypair
$ openssl genrsa -out server.key 2048
Create a certificate signing request using the server key to send to the fake CA for identity verification
$ openssl req -new -key server.key -out server-cert-request.csr -sha256
Give the organization a name like "Localhost MQTT Broker Inc." and the common name should be localhost or the exact domain you use to connect to the mqtt broker
Now acting as the fake CA, you receive the server's request for your signature. You have verified the server is who it says it is (an MQTT broker operating on localhost), so create a new certificate & sign it with all the power of your fake authority.
$ openssl x509 -req -in server-cert-request.csr -CA ca-root-cert.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360
Now you have everything you need. Make sure (as in Steve's tutorial) Mosquitto is loading the following in mosquitto.conf:
listener 8883
cafile certs\ca-root-cert.crt
keyfile certs\server.key
certfile certs\server.crt
Make sure paho-mqtt is loading the fake CA's root certificate.
client1.tls_set(ca_certs="ca-root-cert.crt")
This is how it knows that mosquitto's server.crt is legitimately signed by a "real and trusted authority" and is not "self-signed" and thus untrusted. Mosquitto and paho should now be able to securely connect and communicate.

Vault returns CERTIFICATE_VERIFY_ERROR when authenticating with tls

I am newbie to vault and currently trying to enable client tls authentication (all on the only CentOS 7 node) via self-signed certificates.
Of course, first of all i ran: vault auth-enable cert
Then, I created a policy.hcl file with such contents:
path "secret/policy/test/foo" {policy="read"}
Then I did the following:
$ vault policy-write test policy.hcl
$ vault write "secret/policy/test/foo" vaule=s3cr3t policy=test
Generate client certificate:
$ openssl genrsa -out client.key 2048
$ openssl req -new -key client.key -out client.csr
$ openssl x509 -req -in client.csr -signkey client.key -out client.crt
And the final action:
$ vault write auth/cert/certs/test display_name=test policies=test certificate=#/home/vagrant/ssl/client.crt
And it all execute without any errors or warnings. Then, i wrote a simple script which must read just a one value after authentication with TLS:
import os
import hvac
client = hvac.Client(url='https://127.0.0.1:8200', cert=('client.crt', 'client.key'))
print(client.read('secret/policy/test/foo'))
But when I run it I get the following error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=8200): Max retries exceeded with url: /v1/secret/policy/test/foo (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),))
Server log says:
http: TLS handshake error from 127.0.0.1:33744: remote error: tls: unknown certificate authority
Any help is highly appreciated
EDIT:
to clarify what the actual problem is, I'd like to mention that when i authenticate to the Vault using CLI:
$ vault auth -method=cert -client-cert=client.crt -client-key=client.key
# returns:
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: 8afdb5bd-b6f1-e33e-a391-ced99fa18b5f
token_duration: 2764800
token_policies: [default test]
... it seems to be working just fine cuz the policy I've created is being attached and I can read the data from secret/policy/test/foo, but Python+HVAC approach does not work.

Ansible for Windows: WinRM HTTPS setup

I have some Ansible playbooks I want to run against some Windows hosts. I've followed the various Ansible guides for setting up WinRM and they have worked fine, but the default setups are very insecure and I want something more production ready. However, the instructions for how to do this are incredibly sparse. So far I've done the following:
On my Windows box:
Enabled WinRM using the supplied ConfigureRemotingForAnsible.ps1 script
Configured target machine to to use a valid cert on HTTPS/5986 rather than the self-signed one generated by the above script
Enabled both Kerberos and CredSSP auth methods on the target machine in WinRM. Some of my role steps require CredSSP to work reliably.
So far so good, the Windows side seems to work fine. However, getting Ansible to connect is proving a nightmare. I can't figure out how to get Ansible to trust the HTTPS cert on the target despite adding it. On my Centos 7 'push box' I've done the following:
Install Ansible and pip with the pywinrm, requests_kerberos and requests_credssp modules
Added my CA certificate to both /etc/pki/tls/certs and /etc/pki/ca-cert
Set my inventory to the following:
ansible_user=ADMINISTRATOR#DOMAIN.COM
ansible_password=Password1
ansible_port=5986
ansible_connection=winrm
ansible_winrm_scheme=https
/#ansible_winrm_server_cert_validation=ignore
ansible_winrm_transport=credssp
With certification validation turned on the connection fails with the following error:
fatal: [host.domain.com]:
UNREACHABLE! => {
"changed": false,
"msg": "credssp: HTTPSConnectionPool(host='host.domain.com', port=5986): Max retries exceeded with url: /wsman (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))",
"unreachable": true}
With cert validation turned off it works fine.
So my question: How do I get Ansible to trust my CA cert?

This isn't simple. The problem is there are 3 different groups that don't really want to play:
Ansible: doesn't want to be responsible for OpenSSL or Windows details
OpenSSL: doesn't want to be used as a certficate authority
Microsoft: doesn't want you to use anything but Windows tools
First, you will need 3 sets of OpenSSL certs/keys: the self-signed Certificate Authority (CA), the server winRM HTTPS, and the target user.
There is nothing magic about the CA cert/key; it can be generated with the following:
openssl req -new -out caroot.req -keyout caroot.key -days 730 ...
openssl x509 -req -in caroot.req -extensions v3_ca -signkey caroot.key -out cacert.pem -days 730
There are a myriad of other options to these OpenSSL commands which may (or not) work in your specific environment. This is one of those instances of "don't want to be responsible for OpenSSL".
Second, you will need the WinRM HTTPS cert/key. First, you have to generate the server cert. The key detail of generating the cert is that the 'common name' attribute on the cert MUST match the target hostname, as with any HTTPS cert.
openssl req -new -out server.req -keyout server.key -days 730 -nodes ...
Now, here comes the tricky part. You can't just sign the key as-is; the signing process must add a couple of additional attributes to the cert or it won't work. Create a text file I'll call 'attributes.txt' with the following two lines:
subjectAltName=DNS:hostname.mycompany.com,DNS:hostname
extendedKeyUsage=serverAuth
The first line, the 'subjectAltName', is required or the Python/OpenSSL client will reject the key. You need to substitute the proper values for the target hostname. The second line is required by Windows, or Windows will not use this as for an HTTPS server.
Now sign the key with the CA generated earlier:
openssl x509 -req -in server.req -out server.pem -days 730 -CA caroot.pem -CAkey caroot.key -extfile attributes.txt
Another little wrinkle: Windows will not import a server key unless you combine the key and cert into a PKCS#12 file. This can be done with the following command, which creates a PKCS#12 file with no password:
openssl pkcs12 -export -password pass: -inkey server.key -in server.pem -out windows.pfx
This will generate a file 'windows.pfx' which we'll need later.
Bored yet?
Next we need another key/cert for the user login. This is assuming that we're going to be using a local user on the system, not a domain user. (You're on your own for that.) Generating the key is almost the same as for the server key, except that the 'commom name' attribute (CN) MUST match the target username. (In later usage, I'll call our user 'ansible').
openssl req -new -out winlogin.req -keyout winlogin.key -days 730
This private key is going to be the login key, analogous to an SSH private key. Next, we must sign the req. As with the server key, we need to add some extended attributes, or it won't work. Our 'attributes.txt' file must contain the following lines:
subjectAltName=otherName:1.3.6.1.4.1.311.20.2.3;UTF8:ansible#localhost
extendedKeyUsage=clientAuth
The oddball identifier in the 'altNames' is some proprietary Microsoft thingy, which must contain the 'username#localhost'. As I mentioned before, I'm using a local user 'ansible'. The extendedKeyUsage attribute is required or Windows will not allow the key to be used for user authentication. Finally, we sign the cert:
openssl x509 -req -in winlogin.req -out winlogin.pem -days 730 -CA caroot.pem -CAkey caroot.key -extfile attributes.txt
Just so you know, I'm starting to get on my own nerves here. We're almost there. Finally, copy the CA cert (caroot.pem), the server PKCS#12 cert/key (windows.pfx), and the user cert (winlogin.pem) to the target windows system somewhere. Run the following PowerShell script from the same directory. It will create the local ansible user and import all of the SSL artifacts into their proper target locations. Sorry if this isn't commercial quality code. This also has a hardcoded password for the ansible user, but that is irrelevant and can be freely changed. If this actually runs successfully, and you survive the shock, the file artifacts (the PFX and PEM files) can be removed from the Windows server.
Also note that this script adds a firewall rule at the end. This, of course, will have to be appropriately modified for your environment.
# master script to enable winRM via HTTPS and allow certificate-based
# authentication
# add the local 'ansible' user
$username = "ansible"
$password = ConvertTo-SecureString -string "Ans!b123" -AsPlainText -Force
New-LocalUser -Name $username -AccountNeverExpires -Description "Ansible Remote Management" -Password $password
Add-LocalGroupMember -Member "ansible" -Group "Administrators"
$credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $password
# import the Ansible root CA; certlm should show this in the
# 'Trusted Root Certification Authorities' folder
$caroot = Import-Certificate -FilePath ".\caroot.pem" -CertStoreLocation "Cert:\LocalMachine\Root"
# import the user cert; certlm should show this in the 'Trusted People' folder
$userkey = Import-Certificate -FilePath ".\winlogin.pem" -CertStoreLocation "Cert:\LocalMachine\TrustedPeople"
New-Item -Path WSMan:\localhost\ClientCertificate -subject "ansible#localhost" -URI "*" -Issuer $caroot.Thumbprint -Credential $credential -Force
# import the server certs - should appear in 'Personal' folder. The PFX file
# must contain both the cert and private key
$srvcert = Get-ChildItem -Path ".\windows.pfx" | Import-PFXCertificate -CertStoreLocation "Cert:\LocalMachine\MY" -Exportable
# Now create the winRM instance
$selector_set = #{Address= "*" Transport = "HTTPS" }
$value_set = #{CertificateThumbprint = $srvcert.Thumbprint}
New-WSManInstance -ResourceURI "winrm/config/Listener" -SelectorSet $selector_set -ValueSet $value_set
Set-Item -Path WSMan:\localhost\Service\Auth\Certificate -value $true
# add a firewall rule
New-NetFirewallRule -DisplayName "Ansible WinRM" -Direction Inbound -Protocol "TCP" -LocalPort "5986" -Action Allow -RemoteAddress #("192.16.2.3")
Okay, now that Windows is done, we should be ready for Ansible. Within your inventory, you will need the following variables to apply to the Windows systems:
ansible_port: 5986
ansible_connection: winrm
ansible_winrm_ca_trust_path: /path/to/caroot.pem
ansible_winrm_transport: certficate
ansible_winrm_cert_pem: /path/to/winlogin.pem
ansible_winrm_cert_key_pem: /path/to/winlogin.key
At this point, Ansible should connect to the Windows server using HTTPS and certificate authentication. I hope I hit all of the steps, it was a while ago I set this up.

Run this on the ansible control host:
python -c "import ssl; print(ssl.get_default_verify_paths())"
this way you will see where python expects to find the certificate
Ensure that the CA trust certificate is pem-encoded
regards

This worked for me..
https://dodgydudes.se/validate-ca-certificate-in-ansible-connecting-with-winrm/
ansible_winrm_ca_trust_path: /etc/ssl/certs #TLS 1.2
ansible_winrm_transport: ntlm

Python SSL wrap_socket failed with SSLError, Errno 336265218

I have simple client/server SSL code which worked fine on Python 3.2. However, I decided to switch over to 2.7 (due to abundance of third party modules), and now the code is failing. The code is as follows:
Client:
def connect(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(('localhost', 4430))
self.ssl_sock = ssl.wrap_socket(self.sock, cert_reqs = ssl.CERT_NONE, ssl_version = ssl.PROTOCOL_TLSv1)
Server:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((self.hostname, 4430))
self.sock.listen(5)
while True:
self.newsocket, self.fromaddr = self.sock.accept()
self.ssl_sock = ssl.wrap_socket(
self.newsocket,
server_side = True,
certfile = "cert.pem",
ssl_version=ssl.PROTOCOL_TLSv1
)
self._handle_client(self.ssl_sock)
The cert.pem (located in same directory as server .py file):
-----BEGIN RSA PRIVATE KEY-----
(812 "random" characters here)
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
(1260 "random" characters here)
-----END CERTIFICATE-----
The failure is as follows:
1. server waits on self.sock.accept()
2. client connects with self.sock.connect()
3. server proceeds to wrap_socket, which fails with the following error:
Traceback (most recent call last):
File "C:\Program Files\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\workspace\projectc\server\server\clientlistener.py", line 49, in run
ssl_version=ssl.PROTOCOL_TLSv1
File "C:\Program Files\Python27\lib\ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "C:\Program Files\Python27\lib\ssl.py", line 141, in __init__
ciphers)
SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
P.S.
1) I have suspicions about the wrap_socket call because even when I use a non-existent file for certfile, the same error results.
2) I tried the alternative wrapping on the python documentation on ssl (i.e. with ssl.SSLContext) and it's odd that 'module' object has no attribute 'SSLContext', when it is part of the ssl module. This wasn't an issue when I tried it with Python 3.2.
Update:
I have found that the problem only occurs when I'm doing "run" from within Eclipse, not so when I run both files from separate command prompts. Still investigating the issue...
Update 2:
I tried a very simple client/server script with the SAME code and it works. Now, with the server code sitting inside a multiprocessing subprocess (launched by the main server process), it seems not to work. Related?

Problem solved. I believe Eclipse was holding some residual information, though I never figured out what information it was holding. It might have to do with the migration from python 3.2 to 2.7, though I previously already changed that in the run config. Resetting the Eclipse environment worked (simply rebooting the computer wasn't enough).

my solution was that when creating my .pem file i set a blank password and assumed it meant no pasword. so the server was still expecting to use a password. i had to manually remove the password.
here is a little how to guide if it helps anyone (also, please note i was coding python for a server side to an iOS app)
NOTE: need to follow directions from apple’s developer website to create certificate first
then export the .p12 file,
by exporting the embedded private key that is created (in ‘keychain access’),
NOT the actual certificate
————————————————————————————————————
————————————————————————————————————
FOR DEVELOPMENT CERT:
After getting the p12 file, it needs to be converted to the PEM format by executing this command from the terminal:
$ openssl pkcs12 -clcerts -nokeys -out apns-dev-cert.pem -in apns_dev.p12
$ openssl pkcs12 -nocerts -out apns-dev-key.pem -in apns_dev.p12
If you wish to remove the passphrase execute the following:
(NOTE: using a ‘blank’ password when exporting/converting, is still indeed setting a password,
hence you should still execute the following if you intend to have no password)
$ openssl rsa -in apns-dev-key.pem -out apns-dev-key-noenc.pem
Finally, you need to combine the key and cert files into a apns-dev.pem file we will use when connecting to APNS:
$ cat apns-dev-cert.pem apns-dev-key-noenc.pem > apns-dev.pem
————————————————————————————————————
FOR PRODUCTION CERT:
After getting the p12 file, it needs to be converted to the PEM format by executing this command from the terminal:
$ openssl pkcs12 -clcerts -nokeys -out apns-prod-cert.pem -in apns_prod.p12
$ openssl pkcs12 -nocerts -out apns-prod-key.pem -in apns_prod.p12
If you wish to remove the passphrase execute the following:
(NOTE: using a ‘blank’ password when exporting/converting, is still indeed setting a password,
hence you should still execute the following if you intend to have no password)
$ openssl rsa -in apns-prod-key.pem -out apns-prod-key-noenc.pem
Finally, you need to combine the key and cert files into a apns-dev.pem file we will use when connecting to APNS:
$ cat apns-prod-cert.pem apns-prod-key-noenc.pem > apns-prod.pem