We Keep Coding

Certificate Generation for Python Twisted SSL

I am trying to figure out how to setup a SSL link using the Python library Twisted. I have managed to create a certificate that works on the server side, but I am totally stuck when it comes to the client side.
The example from the twisted website states:
The following examples rely on the files server.pem (private key and
self-signed certificate together) and public.pem (the server’s public
certificate by itself).
I have generated myself a certificate and key using OpenSSL:
# Generate Private Key:
openssl genrsa -des3 -out certs/server.key 2048
# Generate Certificate Signing Request:
openssl req -new -key certs/server.key -sha256 -out certs/server.csr
# Generate a Self-Signed Certificate:
openssl x509 -req -days 365 -in certs/server.csr -signkey certs/server.key -sha256 -out certs/server.crt
# Convert the CRT to PEM format:
openssl x509 -in certs/server.crt -out certs/server.pem -outform PEM
For the server-side I am combining certs/server.crt and certs/server.key to create server.pem and trying to use server.crt for public.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData)
I get an error about not starting line. Which certificate should I be using for the client if it's not server.crt please?

In case you want to have certificate based authentication for the clients as well:
I had that issue some time ago and wrote a blog post about my solution.
It also contains a guide to create certificates and sign them with an own certificate authority. You can find the python example code at GitHub.
It uses Twisted for a simple JSONRPCServer
with certificate based authentication for both, server as well as for the clients.
The main thing is to define an own AltCtxFactory for the clients:
# Use our own context factory to use our certificate to authenticate
# against the server and ensure that we are using a strong SSL/TLS
# encryption method
class AltCtxFactory(ssl.ClientContextFactory):
def getContext(self):
# Used TLS/SSL encryption method
sslMethod = SSL.TLSv1_2_METHOD
# Clients private Key, used for authentication
privKey = "<PATH TO YOUR PRIVATE KEY>"
# Clients certificate, used for authentication
certificate = "<PATH TO YOUR CERTIFICATE>"
# Our trusted Certificate Authority for server connections
accepted_ca = "<PATH TO YOUR ACCEPTED CERTIFICATE AUTHORITY>"
self.method = sslMethod
ctx = ssl.ClientContextFactory.getContext(self)
# Ensure that we verify server's certificate and use our own
# verifyCallback method to get further details of invalid certificates
ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
verifyCallback)
# Ensure that we only trust our CA
ctx.load_verify_locations(accepted_ca)
# Use our own Callback mehtod if a password is needed to decrypt our
# private key
ctx.set_passwd_cb(password_cb)
# Use our certificate for authentication against server
ctx.use_certificate_file(certificate)
# Use our private key for authentication against server
ctx.use_privatekey_file(privKey)
return ctx
Feel free to use the code in your projects.

When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData) I get an error
about not starting line. Which certificate should I be using for the
client if it's not server.crt please?
This should be ssl.Certificate.LoadPEM(certData) if you look at the example on the Twisted howto page.

Python Failed to Verify any CRLs for SSL/TLS connections

In Python 3.4, a verify_flags that can be used to check if a certificate was revoked against CRL, by set it to VERIFY_CRL_CHECK_LEAF or VERIFY_CRL_CHECK_CHAIN.
I wrote a simple program for testing. But on my systems, this script failed to verify ANY connections even if it's perfectly valid.
import ssl
import socket
def tlscheck(domain, port):
addr = domain
ctx = ssl.create_default_context()
ctx.options &= ssl.CERT_REQUIRED
ctx.verify_flags = ssl.VERIFY_CRL_CHECK_LEAF
ctx.check_hostname = True
#ctx.load_default_certs()
#ctx.set_default_verify_paths()
#ctx.load_verify_locations(cafile="/etc/ssl/certs/ca-certificates.crt")
sock = ctx.wrap_socket(socket.socket(), server_hostname=addr)
sock.connect((addr, port))
import pprint
print("TLS Ceritificate:")
pprint.pprint(sock.getpeercert())
print("TLS Version:", sock.version())
print("TLS Cipher:", sock.cipher()[0])
exit()
tlscheck("stackoverflow.com", 443)
My code always quits with ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645).
First I suspected that the certificate database was not loaded properly. But after I tried load_default_certs(), set_default_verify_paths(), load_verify_locations(cafile="/etc/ssl/certs/ca-certificates.crt"), and none of them worked.
Also, ctx.options &= ssl.CERT_REQUIRED works as expected, it can tell if a certificate chain is trusted or not. But not for CRLs... It also indicates that my CAs are correct.
I know "/etc/ssl/certs/ca-certificates.crt" contains valid CAs. What is the problem?

To check against CRL you have to manually download the CRL and put them in the right place so that the underlying OpenSSL library will find it. There is no automatic downloading of CRL and specifying the place where to look for the CRL is not intuitive either. What you can do:
get the CRL distribution points from the certificate. For stackoverflow.com one is http://crl3.digicert.com/sha2-ha-server-g5.crl
download the current CRL from there
convert it from the DER format to PEM format, because this is what is expected in the next step:
openssl crl -inform der -in sha2-ha-server-g5.crl > sha2-ha-server-g5.crl.pem
add the location to the verify_locations:
ctx.load_verify_locations(cafile="./sha2-ha-server-g5.crl.pem")
This way you can verify the certificate against the CRL.

iOS Push Notifications (APNs) over GAE, SSL Handshake Failure

I am attempting to show proof of concept for iOS Push Notifications from a Google AppEngine application instance using this RPC handler...
PAYLOAD = {'aps': {'alert':'Push!','sound':'default'}}
TOKEN = '[...]'
class APNsTest(BaseRPCHandler):
def get(self, context, name):
self._call_method(context, name)
def send_push(self):
# certificate files
filename = 'VisitorGuidePush'
abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../archive/certificate'))
ca_certs = os.path.abspath(os.path.join(abs_path, '%s.ca'%filename))
certfile = os.path.abspath(os.path.join(abs_path, '%s.crt'%filename))
keyfile = os.path.abspath(os.path.join(abs_path, '%s.key'%filename))
# serialize payload
payload = json.dumps(PAYLOAD)
# APNS server address...
# apns_address = ('api.development.push.apple.com', 443) # Development server
# apns_address = ('api.development.push.apple.com', 2197) # Development server
# apns_address = ('api.push.apple.com', 443) # Production server
apns_address = ('api.push.apple.com', 2197) # Production server
# a socket to connect to APNS over SSL
_sock = socket.socket()
_ssl = ssl.wrap_socket(_sock, keyfile=keyfile,
certfile=certfile,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=ca_certs)
_ssl.connect(apns_address)
# Generate a notification packet
token = binascii.unhexlify(TOKEN)
fmt = '!cH32sH{0:d}s'.format(len(payload))
cmd = '\x00'
message = struct.pack(fmt, cmd, len(token), token, len(payload), payload)
_ssl.write(message)
_ssl.close()
return self.response_result(PAYLOAD)
And need help resolving this error when executing "_ssl.connect(apns_address)"
SSLError: [Errno 1] _ssl.c:507: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
My PEM file (derived from a .p12) and device token were generated a week ago by a mobile developer on our team, suggestions for validating these would be helpful. For now I believe there are current and valid.
While the TLSv1 protocol is being specified, I've notice the handshake failure identifies sslv3.
I have attempted many variations and combination of wrap_socket and apns_address, and am consistently stopped by the handshake failure. Which leads me to suspect a problem with the way I am applying the pem certificate.
The primary references I have been using for wrap_socket are Using OpenSSL and TLS/SSL wrapper for socket objects, not to mention more than a few StackOverflow posts.
Please provide advice concerning the appropriate keyfile, certfile, and ca_certs values and any other advice or resources available for GAE based APNs communication. Thanks ~
Updated question...
The original .p12 has been validated using Pusher, and divided via openssl...
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.key -nodes -nocerts
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.crt -nodes -nokeys
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.ca -nodes -cacerts
I'm receiving a new error which appears related to the ca_certs...
SSLError: [Errno 0] _ssl.c:343: error:00000000:lib(0):func(0):reason(0)
Removing the ca_certs requirement or passing in other files like the .p12 or the .crt result in a return to the original handshake failure.

Look into using a library such as pyapns, which is what I used to get push notifications to work on GAE. To test whether you're using the correct key/cert file, you can use apps like Pusher. Also, I know that to get SSL capabilities on GAE you have to enable billing, so maybe that's the problem. Good luck!

The appropriate support files start with Creating a Universal Push Notification Client SSL Certificate as a p12 file.
Next, utilizing command line openssl to parse the p12 into the desired certificate and key files...
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.key -nodes -nocerts
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.crt -nodes -nokeys
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.pem -nodes
And finally to obtain a qualified Certificate Authority file (from Troubleshooting Push Notifications)
In addition to the SSL identity (certificate and associated private
key) created by Member Center, you should also install the Entrust CA
(2048) root certificate on your provider.
Entrust.net Certificate Authority (2048) download ~ entrust_2048_ca.cer
Note that every GAE instance hosts its own Certificate Authority at /etc/ca-certificates.crt as described here, Using OpenSSL.
With these files added to your project you can make one of two, equally valid, ssl socket objects...
_ssl = ssl.wrap_socket(_sock, keyfile=VisitorGuide.key,
certfile=VisitorGuide.crt,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=entrust_2048_ca.cer)
...or...
_ssl = ssl.wrap_socket(_sock, certfile=VisitorGuide.pem,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=entrust_2048_ca.cer)
TLS/SSL wrapper for socket objects 17.3.4.3. Combined key and certificate explains why both are valid parameter options.
Before I provide the final code block, I have to point out something concerning the APNs address (this proved to be the key point, allowing me to resolve the handshake failure and obtain an SSL connection between GAE and APNs)
According to the iOS Developer Library APNs Provider API
The first step in sending a remote notification is to establish a connection with the appropriate APNs server:
Development server: api.development.push.apple.com:443
Production server: api.push.apple.com:443
Note: You can alternatively use port 2197 when communicating with APNs. You might do this, for example, to allow APNs traffic through your firewall but to block other HTTPS traffic.
But it wasn't until I dug into the Pusher source that I discovered the APNs addresses to which I could connect...
gateway.sandbox.push.apple.com:2195
gateway.push.apple.com:2195
Without further ado...
class APNsTest(BaseRPCHandler):
def get(self, context, name):
self._call_method(context, name)
def send_push(self):
# certificate files
abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../cert'))
pem_file = os.path.abspath(os.path.join(abs_path, 'VisitorGuide.pem'))
ca_certs = '/etc/ca-certificates.crt'
# APNS server address...
apns_address = ('gateway.sandbox.push.apple.com', 2195)
# apns_address = ('gateway.push.apple.com', 2195)
# a socket to connect to APNS over SSL
_sock = socket.socket()
_ssl = ssl.wrap_socket(_sock, certfile=pem_file,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=ca_certs)
_ssl.connect(apns_address)
# a notification packet
payload = json.dumps(PAYLOAD)
token = binascii.unhexlify(TOKEN)
fmt = '!cH32sH{0:d}s'.format(len(payload))
cmd = '\x00'
message = struct.pack(fmt, cmd, len(token), token, len(payload), payload)
_ssl.write(message)
_ssl.close()
return self.response_result(PAYLOAD)
...executes without error.

Also, from experimenting, I've found out that SSL related errors on App Engine usually go away when I apply the built-in 3rd party ssl lib:
libraries:
- name: ssl
version: latest
or:
libraries:
- name: ssl
version: "2.7"

Python imaplib display certificate key

I want imaplib to display the md5 (or SHA) key of an IMAP Server Certificate to make sure, that there's no MITM (I don't trust the CA, so verifying the chain isn't enough in this case).
Displaying the whole certificate would also be okay.
I'd appreciate any help!!
Chris

You can use the M2Crypto package to parse the full SSL certificate from the IMAP connection's SSL socket. Here is an example:
import imaplib
from M2Crypto import X509
cn = imaplib.IMAP4_SSL('imap.gmail.com', 993)
sock = cn.ssl()
data = sock.getpeercert(1)
cert = X509.load_cert_string(data, X509.FORMAT_DER)
print cert.get_fingerprint()
Prints:
2029AF27C0A55390D670C0BD7AB9747
Use the other attributes on cert to get further information.

I don't know how to do it from imaplib, but you can connect to a secure IMAP server and display the certificate using M2Crypto:
from M2Crypto import SSL
ctx = SSL.Context('sslv3')
c = SSL.Connection(ctx)
c.connect(('localhost', 993)) # automatically checks cert matches host
cert = c.get_peer_cert()
print cert.as_pem()
print cert.as_text()
Note that cert is an X509 object.

Use Python to get an smtp server certificate

What's the easiest way to connect to a SMTP server that supports STARTTLS and get its server SSL certificate? I know it can be done using openssl with something like this
openssl s_client -starttls smtp -crlf -connect 192.168.0.1:25
How can I do it from within Python and I don't want to call openssl and parse its output. I looked at M2Crypto which is an openssl wrapper, but as far as I can tell that doesn't support starttls. An example of how to do it with a Python library would be very much appreciated.

This returns a certificate in binary format (DER-encoded):
import socket, ssl
s = socket.socket()
s.connect(("host", 25))
s.send("STARTTLS\n")
s.recv(1000)
ss = ssl.wrap_socket(s)
certificate_der = ss.getpeercert(True)
This is jus to give you an idea, error handling, etc. is required of course. If you want to decode the information from the certificate you either have to prodivde a certificate authorities bundle/directory for acceptable CAs (getpeercert() will return a meaningfull dict in this case), or use a more capable ssl library, M2Crypto for example.

You could try something like:
import ssl
cert = ssl.get_server_certificate(('imap.gmail.com',993))
to get server's certificate

As I can't comment abbot answer, just remark that depending on the server config you may need to send an EHLO before STARTTLS:
import socket, ssl
hostname = 'test.com'
port = 25
context = ssl.create_default_context()
with socket.create_connection((hostname, port)) as sock:
sock.recv(1000)
sock.send(b'EHLO\nSTARTTLS\n')
sock.recv(1000)
with context.wrap_socket(sock, server_hostname=hostname) as sslsock:
der_cert = sslsock.getpeercert(True)

smtplib provides the starttls() method which should deal with all issues:
http://docs.python.org/library/smtplib.html