I am trying to figure out how to setup a SSL link using the Python library Twisted. I have managed to create a certificate that works on the server side, but I am totally stuck when it comes to the client side.
The example from the twisted website states:
The following examples rely on the files server.pem (private key and
self-signed certificate together) and public.pem (the server’s public
certificate by itself).
I have generated myself a certificate and key using OpenSSL:
# Generate Private Key:
openssl genrsa -des3 -out certs/server.key 2048
# Generate Certificate Signing Request:
openssl req -new -key certs/server.key -sha256 -out certs/server.csr
# Generate a Self-Signed Certificate:
openssl x509 -req -days 365 -in certs/server.csr -signkey certs/server.key -sha256 -out certs/server.crt
# Convert the CRT to PEM format:
openssl x509 -in certs/server.crt -out certs/server.pem -outform PEM
For the server-side I am combining certs/server.crt and certs/server.key to create server.pem and trying to use server.crt for public.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData)
I get an error about not starting line. Which certificate should I be using for the client if it's not server.crt please?
In case you want to have certificate based authentication for the clients as well:
I had that issue some time ago and wrote a blog post about my solution.
It also contains a guide to create certificates and sign them with an own certificate authority. You can find the python example code at GitHub.
It uses Twisted for a simple JSONRPCServer
with certificate based authentication for both, server as well as for the clients.
The main thing is to define an own AltCtxFactory for the clients:
# Use our own context factory to use our certificate to authenticate
# against the server and ensure that we are using a strong SSL/TLS
# encryption method
class AltCtxFactory(ssl.ClientContextFactory):
def getContext(self):
# Used TLS/SSL encryption method
sslMethod = SSL.TLSv1_2_METHOD
# Clients private Key, used for authentication
privKey = "<PATH TO YOUR PRIVATE KEY>"
# Clients certificate, used for authentication
certificate = "<PATH TO YOUR CERTIFICATE>"
# Our trusted Certificate Authority for server connections
accepted_ca = "<PATH TO YOUR ACCEPTED CERTIFICATE AUTHORITY>"
self.method = sslMethod
ctx = ssl.ClientContextFactory.getContext(self)
# Ensure that we verify server's certificate and use our own
# verifyCallback method to get further details of invalid certificates
ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
verifyCallback)
# Ensure that we only trust our CA
ctx.load_verify_locations(accepted_ca)
# Use our own Callback mehtod if a password is needed to decrypt our
# private key
ctx.set_passwd_cb(password_cb)
# Use our certificate for authentication against server
ctx.use_certificate_file(certificate)
# Use our private key for authentication against server
ctx.use_privatekey_file(privKey)
return ctx
Feel free to use the code in your projects.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData) I get an error
about not starting line. Which certificate should I be using for the
client if it's not server.crt please?
This should be ssl.Certificate.LoadPEM(certData) if you look at the example on the Twisted howto page.
Related
My aim is to achieve SHA1 fingerprint of a third party website's certificate. I am able to get it successfully using openssl command line however, it's not getting same when I tried to achieve it using python code. The SHA1 fingerprint obtained using python code is totally different than the one obtained via openssl.
openssl steps -->
openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443
The above command output contains chain and root certificate;
Certificate chain
0 s:/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=*.actions.githubusercontent.com
i:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
-----BEGIN CERTIFICATE-----
MIIG9jCCBd6gAwIBAgIQCFCR4fqbkQJJbzQZsc87qzANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMjAxMTEwMDAwMDBa
Save the chain certificate with .crt extension as MaingithubOIDC.crt and running below command gives SHA1 fingerprint;
❯ openssl x509 -in MaingithubOIDC.crt -fingerprint -noout
SHA1 Fingerprint=15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31
Reference link - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
Python code (version 3.8/3.9) -->
import ssl
import socket
import hashlib
addr = 'token.actions.githubusercontent.com'
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
print (wrappedSocket)
except:
response = False
else:
der_cert = wrappedSocket.getpeercert(True)
pem_cert = ssl.DER_cert_to_PEM_cert(wrappedSocket.getpeercert(True))
print(pem_cert)
#Print SHA1 Thumbprint
thumb_sha1 = hashlib.sha1(der_cert).hexdigest()
print("SHA1: " + thumb_sha1)
Python code output;
SHA1: 55a7ef500a3a99f64c99c665daaf3f07403cff3d
So, the SHA1 fingerprint doesn't match with the one obtained using openssl. Am I missing something in python code?
The problem is not the wrong fingerprint calculation from the certificate but that you get the wrong certificate. The server in question is a multi-domain setup which will return different certificates based on the server_name given in the TLS handshake - see Server Name Indication.
The following code will not provide a server_name, which results in a certificate returned for *.azureedge.net, not *.actions.githubusercontent.com as the openssl s_client code gets:
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
To fix this the server_name need to be given:
ctx = ssl.create_default_context()
wrappedSocket = ctx.wrap_socket(sock,
server_hostname='token.actions.githubusercontent.com')
try:
wrappedSocket.connect((addr, 443))
With this change the expected certificate is send by the server and the fingerprint is properly calculated on it.
I'm trying to connect to "mqtts://broker.emqx.io:8883" using paho mqtt but getting the invalid certificate error.
tls = { 'ca_certs': "certificate.pem", 'insecure': False, 'tls_version': ssl.PROTOCOL_TLS_CLIENT }
publish.single( topic="testtopic\", payload=payload, hostname="broker.emqx.io", auth={}, port=8883, tls=tls, protocol=mqtt.MQTTv311 )
I don't want it to be insecure and want to get the certificate from the server instead of providing a certificate file manually. It would be a great help if someone could guide me in a right direction.
The ca_certs file should not come from the server, it's suppose to contain the certs for the Certificate Authority chain that signed the certificate used by the broker.
So in this case it should hold1 the certificate chain from:
Sectigo RSA Domain Validation Secure Server CA
USERTrust RSA Certification Authority
AAA Certificate Services
You need to provide the full chain or a file containing all the trusted public CA certs.
1 found with openssl s_client see here
I am attempting to show proof of concept for iOS Push Notifications from a Google AppEngine application instance using this RPC handler...
PAYLOAD = {'aps': {'alert':'Push!','sound':'default'}}
TOKEN = '[...]'
class APNsTest(BaseRPCHandler):
def get(self, context, name):
self._call_method(context, name)
def send_push(self):
# certificate files
filename = 'VisitorGuidePush'
abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../archive/certificate'))
ca_certs = os.path.abspath(os.path.join(abs_path, '%s.ca'%filename))
certfile = os.path.abspath(os.path.join(abs_path, '%s.crt'%filename))
keyfile = os.path.abspath(os.path.join(abs_path, '%s.key'%filename))
# serialize payload
payload = json.dumps(PAYLOAD)
# APNS server address...
# apns_address = ('api.development.push.apple.com', 443) # Development server
# apns_address = ('api.development.push.apple.com', 2197) # Development server
# apns_address = ('api.push.apple.com', 443) # Production server
apns_address = ('api.push.apple.com', 2197) # Production server
# a socket to connect to APNS over SSL
_sock = socket.socket()
_ssl = ssl.wrap_socket(_sock, keyfile=keyfile,
certfile=certfile,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=ca_certs)
_ssl.connect(apns_address)
# Generate a notification packet
token = binascii.unhexlify(TOKEN)
fmt = '!cH32sH{0:d}s'.format(len(payload))
cmd = '\x00'
message = struct.pack(fmt, cmd, len(token), token, len(payload), payload)
_ssl.write(message)
_ssl.close()
return self.response_result(PAYLOAD)
And need help resolving this error when executing "_ssl.connect(apns_address)"
SSLError: [Errno 1] _ssl.c:507: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
My PEM file (derived from a .p12) and device token were generated a week ago by a mobile developer on our team, suggestions for validating these would be helpful. For now I believe there are current and valid.
While the TLSv1 protocol is being specified, I've notice the handshake failure identifies sslv3.
I have attempted many variations and combination of wrap_socket and apns_address, and am consistently stopped by the handshake failure. Which leads me to suspect a problem with the way I am applying the pem certificate.
The primary references I have been using for wrap_socket are Using OpenSSL and TLS/SSL wrapper for socket objects, not to mention more than a few StackOverflow posts.
Please provide advice concerning the appropriate keyfile, certfile, and ca_certs values and any other advice or resources available for GAE based APNs communication. Thanks ~
Updated question...
The original .p12 has been validated using Pusher, and divided via openssl...
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.key -nodes -nocerts
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.crt -nodes -nokeys
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.ca -nodes -cacerts
I'm receiving a new error which appears related to the ca_certs...
SSLError: [Errno 0] _ssl.c:343: error:00000000:lib(0):func(0):reason(0)
Removing the ca_certs requirement or passing in other files like the .p12 or the .crt result in a return to the original handshake failure.
Look into using a library such as pyapns, which is what I used to get push notifications to work on GAE. To test whether you're using the correct key/cert file, you can use apps like Pusher. Also, I know that to get SSL capabilities on GAE you have to enable billing, so maybe that's the problem. Good luck!
The appropriate support files start with Creating a Universal Push Notification Client SSL Certificate as a p12 file.
Next, utilizing command line openssl to parse the p12 into the desired certificate and key files...
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.key -nodes -nocerts
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.crt -nodes -nokeys
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.pem -nodes
And finally to obtain a qualified Certificate Authority file (from Troubleshooting Push Notifications)
In addition to the SSL identity (certificate and associated private
key) created by Member Center, you should also install the Entrust CA
(2048) root certificate on your provider.
Entrust.net Certificate Authority (2048) download ~ entrust_2048_ca.cer
Note that every GAE instance hosts its own Certificate Authority at /etc/ca-certificates.crt as described here, Using OpenSSL.
With these files added to your project you can make one of two, equally valid, ssl socket objects...
_ssl = ssl.wrap_socket(_sock, keyfile=VisitorGuide.key,
certfile=VisitorGuide.crt,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=entrust_2048_ca.cer)
...or...
_ssl = ssl.wrap_socket(_sock, certfile=VisitorGuide.pem,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=entrust_2048_ca.cer)
TLS/SSL wrapper for socket objects 17.3.4.3. Combined key and certificate explains why both are valid parameter options.
Before I provide the final code block, I have to point out something concerning the APNs address (this proved to be the key point, allowing me to resolve the handshake failure and obtain an SSL connection between GAE and APNs)
According to the iOS Developer Library APNs Provider API
The first step in sending a remote notification is to establish a connection with the appropriate APNs server:
Development server: api.development.push.apple.com:443
Production server: api.push.apple.com:443
Note: You can alternatively use port 2197 when communicating with APNs. You might do this, for example, to allow APNs traffic through your firewall but to block other HTTPS traffic.
But it wasn't until I dug into the Pusher source that I discovered the APNs addresses to which I could connect...
gateway.sandbox.push.apple.com:2195
gateway.push.apple.com:2195
Without further ado...
class APNsTest(BaseRPCHandler):
def get(self, context, name):
self._call_method(context, name)
def send_push(self):
# certificate files
abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../cert'))
pem_file = os.path.abspath(os.path.join(abs_path, 'VisitorGuide.pem'))
ca_certs = '/etc/ca-certificates.crt'
# APNS server address...
apns_address = ('gateway.sandbox.push.apple.com', 2195)
# apns_address = ('gateway.push.apple.com', 2195)
# a socket to connect to APNS over SSL
_sock = socket.socket()
_ssl = ssl.wrap_socket(_sock, certfile=pem_file,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=ca_certs)
_ssl.connect(apns_address)
# a notification packet
payload = json.dumps(PAYLOAD)
token = binascii.unhexlify(TOKEN)
fmt = '!cH32sH{0:d}s'.format(len(payload))
cmd = '\x00'
message = struct.pack(fmt, cmd, len(token), token, len(payload), payload)
_ssl.write(message)
_ssl.close()
return self.response_result(PAYLOAD)
...executes without error.
Also, from experimenting, I've found out that SSL related errors on App Engine usually go away when I apply the built-in 3rd party ssl lib:
libraries:
- name: ssl
version: latest
or:
libraries:
- name: ssl
version: "2.7"
I am trying to write a client script in Python that accesses a web application and uses SSL client certificates for authentication. Though I am able to access the application from both Firefox and Chrome as long as I have the client certificate loaded, I get the following response whenever I send the request via Python 'requests:
400 No required SSL certificate was sent
nginx/1.4.6 (Ubuntu)
I have also tried Python httplib, s_client, and curl, and get the same error message. I am using the same client certificate for all the testing, in pkcs12 format for the web browsers and pulled out into certificate and key PEM files for the command line tools. My python code looks like:
import requests
CERT = r'/path/to/cert.crt' #Client certificate
KEY = r'/path/to/key.key' #Client private key
CACERT = r'/path/to/ca.crt' #Server certificate chain
session = requests.Session()
session.cert = (CERT, KEY)
resp = session.get('https://my.webapp.com/',
verify=CACERT)
print resp.content
session.close()
s_client gives more information about what is happening. Here's an abbreviated version of the output:
$ openssl s_client -cert cert.crt -key key.key -CAfile ca.crt -connect <host>:<port>
CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify return:1
depth=1 /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
verify return:1
depth=0 /serialNumber=tgBIwyM-p18O/aDyvyWNKHDnOezzDJag/OU=GT89519184/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.rexdb.us
verify return:1
---
Certificate chain
...
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=...
---
No client certificate CA names sent
---
SSL handshake has read 3519 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES128-SHA
Session-ID: ...
Session-ID-ctx:
Master-Key: ...
Key-Arg : None
Start Time: 1409269239
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
GET / HTTP/1.1
Host: my.webapp.com
HTTP/1.1 400 Bad Request
Server: nginx/1.4.6 (Ubuntu)
Date: Thu, 28 Aug 2014 23:41:04 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close
...
<head><title>400 No required SSL certificate was sent</title></head>
...
closed
I'm reasonably sure this is not an issue with the server because authentication works in a browser. However, the s_client output says 'No client certificate CA names sent', which sounds like a server problem (e.g. the client certificate is not being sent on to the server because the server isn't asking for it). Here is the relevant part of the nginx configuration:
ssl on;
ssl_certificate_key /path/to/server/key.pem;
ssl_certificate /path/to/server/certificate.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
ssl_session_timeout 5m;
ssl_client_certificate /path/to/client/ca/certificate.crt;
ssl_verify_depth 10;
ssl_verify_client on;
The application uses uwsgi through nginx. There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
The only other potentially relevant difference I can find is that in Firefox the connection is keep-alive and in s_client it is close. I've tried setting Connection: keep-alive in the header, with the same result.
There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
I assume that means you have multiple certificates behind the same IP address and that the client has to use SNI (Server Name Indication) to send the server the expected hostname inside the SSL handshake. openssl s_client does not use SNI by default and I don't know if python does - it might depend on the version of python you use.
Because the client only sends a certificate if the server tells it do to it might be, that because of missing SNI you run into the wrong configuration part, that is the default part with another certificate and without requirement for client certificates.
I would recommend to try with openssl s_client again, but use the command line option -servername (not documented in man page but shown if called with -h) to explicitly set the expected server name. If this works you need to find a way to use SNI in python too. If this does not work please make a packet dump and make sure with wireshark, that the server really requires the client to send a certificate and the client really does not send a certificate.
I'm stuck with this for several days and could not figure it out still. I just want to build a simple TLS c/s communication in python. For server I use EC2, client I use my own laptop. I setup and test normal socket communication and everything works fine.
When I try this tutorial from the official doc, I run into problem. For the following client code:
# require a certificate from the server
ssl_sock = ssl.wrap_socket(s,
ca_certs="/etc/ca_certs_file",
cert_reqs=ssl.CERT_REQUIRED)
As far as I know the part /etc/ca_certs_file should be some certificates from CAs. I am confused where should I look for them. I actually find some .pem files in /etc/ssl/certs on EC2 server but nothing on the client, my laptop.
I also tried to generate a user certificate according to this tutorial on openssl, I followed the steps, generating cakey.pem, cacert.pem for the server, userkey.pem, usercert-req.pem for the client, all in a same directory in my EC2 server. When I execute openssl ca -in usercert-req.pem -out usercert.pem, it outputs error:
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem:
unable to load certificate
140420412405408:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:696:Expecting: TRUSTED CERTIFICATE
So actually how should this cert file get generated? Generate at server side, then wait for client to request them over the air, or generate at client side, or obtain from 3rd party and directly use on client side?
Could anyone give any guidance? Any help is appreciated.
This will create a self signed certificate pair, the private key will be in the same file:
openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem
And then from python on the server side:
new_client_socket, address = server_socket.accept()
secured_client_socket = ssl.wrap_socket(new_client_socket,
server_side=True,
certfile='cert.pem',
keyfile='cert.pem',
ssl_version=ssl.PROTOCOL_TLSv1)
And the client application:
unsecured_client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket = ssl.wrap_socket(unsecured_client_socket,
ca_certs='cert.pem',
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1)