I'm stuck with this for several days and could not figure it out still. I just want to build a simple TLS c/s communication in python. For server I use EC2, client I use my own laptop. I setup and test normal socket communication and everything works fine.
When I try this tutorial from the official doc, I run into problem. For the following client code:
# require a certificate from the server
ssl_sock = ssl.wrap_socket(s,
ca_certs="/etc/ca_certs_file",
cert_reqs=ssl.CERT_REQUIRED)
As far as I know the part /etc/ca_certs_file should be some certificates from CAs. I am confused where should I look for them. I actually find some .pem files in /etc/ssl/certs on EC2 server but nothing on the client, my laptop.
I also tried to generate a user certificate according to this tutorial on openssl, I followed the steps, generating cakey.pem, cacert.pem for the server, userkey.pem, usercert-req.pem for the client, all in a same directory in my EC2 server. When I execute openssl ca -in usercert-req.pem -out usercert.pem, it outputs error:
Using configuration from /usr/lib/ssl/openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem:
unable to load certificate
140420412405408:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:696:Expecting: TRUSTED CERTIFICATE
So actually how should this cert file get generated? Generate at server side, then wait for client to request them over the air, or generate at client side, or obtain from 3rd party and directly use on client side?
Could anyone give any guidance? Any help is appreciated.
This will create a self signed certificate pair, the private key will be in the same file:
openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem
And then from python on the server side:
new_client_socket, address = server_socket.accept()
secured_client_socket = ssl.wrap_socket(new_client_socket,
server_side=True,
certfile='cert.pem',
keyfile='cert.pem',
ssl_version=ssl.PROTOCOL_TLSv1)
And the client application:
unsecured_client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket = ssl.wrap_socket(unsecured_client_socket,
ca_certs='cert.pem',
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1)
Related
first i create a ca-cert key pair with
openssl req -new -x509 -keyout private_key.pem -out public_cert.pem -days 365 -nodes
Generating a RSA private key
..+++++
.................................+++++
writing new private key to 'private_key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:.
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:35.222.65.55 <----------------------- this ip should be server ip very important
Email Address []:
now i run a server with python code
# libraries needed:
from http.server import HTTPServer, SimpleHTTPRequestHandler
import ssl , socket
# address set
server_ip = '0.0.0.0'
server_port = 3389
# configuring HTTP -> HTTPS
httpd = HTTPServer((server_ip, server_port), SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, certfile='./public_cert.pem',keyfile='./private_key.pem', server_side=True)
httpd.serve_forever()
now this server can be connected for both secure ca-cert case and ingore-ca-cert server connections when using SSL case
that is
curl --cacert public_cert.pem --cert-type PEM https://35.222.65.55:3389
and
curl -k https://35.222.65.55:3389
will work
how to detect if the request is ingnore-ca-cert or not from server side ?
how to not allow insecure connection from server side ?
The server side has no control over the certificate validation done at the client side. The server has no knowledge if the client has verified the certificate or not. Nothing in the exchanged data indicates if the client is doing a curl -k or a curl without this option. Thus it is not possible to stop clients with broken or disabled validation from connecting to the server.
My aim is to achieve SHA1 fingerprint of a third party website's certificate. I am able to get it successfully using openssl command line however, it's not getting same when I tried to achieve it using python code. The SHA1 fingerprint obtained using python code is totally different than the one obtained via openssl.
openssl steps -->
openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443
The above command output contains chain and root certificate;
Certificate chain
0 s:/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=*.actions.githubusercontent.com
i:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
-----BEGIN CERTIFICATE-----
MIIG9jCCBd6gAwIBAgIQCFCR4fqbkQJJbzQZsc87qzANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMjAxMTEwMDAwMDBa
Save the chain certificate with .crt extension as MaingithubOIDC.crt and running below command gives SHA1 fingerprint;
❯ openssl x509 -in MaingithubOIDC.crt -fingerprint -noout
SHA1 Fingerprint=15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31
Reference link - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
Python code (version 3.8/3.9) -->
import ssl
import socket
import hashlib
addr = 'token.actions.githubusercontent.com'
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
print (wrappedSocket)
except:
response = False
else:
der_cert = wrappedSocket.getpeercert(True)
pem_cert = ssl.DER_cert_to_PEM_cert(wrappedSocket.getpeercert(True))
print(pem_cert)
#Print SHA1 Thumbprint
thumb_sha1 = hashlib.sha1(der_cert).hexdigest()
print("SHA1: " + thumb_sha1)
Python code output;
SHA1: 55a7ef500a3a99f64c99c665daaf3f07403cff3d
So, the SHA1 fingerprint doesn't match with the one obtained using openssl. Am I missing something in python code?
The problem is not the wrong fingerprint calculation from the certificate but that you get the wrong certificate. The server in question is a multi-domain setup which will return different certificates based on the server_name given in the TLS handshake - see Server Name Indication.
The following code will not provide a server_name, which results in a certificate returned for *.azureedge.net, not *.actions.githubusercontent.com as the openssl s_client code gets:
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
To fix this the server_name need to be given:
ctx = ssl.create_default_context()
wrappedSocket = ctx.wrap_socket(sock,
server_hostname='token.actions.githubusercontent.com')
try:
wrappedSocket.connect((addr, 443))
With this change the expected certificate is send by the server and the fingerprint is properly calculated on it.
I am writing a python socket server with ssl and I am encountering certificate unknown error during ssl handshake.
I have created private key and certificate with openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes command on my own.
This server is intended to run in intranet(under wifi) in my PC, and users will contact my PC IPaddress with their browser. Hence I have not registered this certificate with any CA. and I don't think its mandatory in my case.
Below more details..
echoserver.py
import socket
import ssl
import threading
class echoserver:
def __init__(self,i,p):
self.ip=i
self.port=p
def handler(self,c,a):
msg=c.recv(1024)
c.send(msg)
def serve(self):
echoserver = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
echoserver = ssl.wrap_socket(echoserver, keyfile='keys/key.pem', certfile='keys/cert.pem', server_side=True)
echoserver.bind((self.ip, self.port))
echoserver.listen()
while True:
(c,a)=echoserver.accept()
threading.Thread(target=self.handler, args=(c,a)).start()
es=echoserver('192.168.43.124',443) #My PC's ip assigned under wifi network
es.serve()
#Connecting from mobile phone within same network as https://192.163.43.124
Error in server during ssl handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1108)
What I tried
Adding cert_reqs=ssl.CERT_NONE and ca_certs="/location/to/keys" parameters in wrap_socket function.
Doesn't seems to work. I assume these options are for client side.
Adding do_handshake_on_connect=False in wrap_socket function
In Chrome, When connected server throws same error and thread/connection closed with exception. and chrome seems to send same connection request immediately again, and the second request works flawlessly !!.
In firefox, First connection closed with same error and there is no second request.
Assigning common name in certificate same as IP address
Not working.
Checked certificate_unknown error in ietf specification here. It gives no clue except this explanation certificate_unknown: Some other (unspecified) issue arose in processing the certificate, rendering it unacceptable.
One other thing I noted is, if I use built-in ThreadedHTTPServer in the same way as below, It works beautifully, without any issues I mentioned above.
httpd = self.ThreadedHTTPServer((self.ip, self.port), self.handler)
httpd.socket = ssl.wrap_socket(httpd.socket, keyfile='keys/key.pem', certfile='keys/cert.pem', server_side=True)
I am not sure why this happens and how should I proceed with this. and not sure how built-in server modules works fine.
Appreciate any leads. Thanks.
Below Python builtin HTTPServer works fine with ssl, not showing any error. Not sure how?
import ssl
from http.server import HTTPServer, BaseHTTPRequestHandler
class requesthandler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-type","text/html")
self.end_headers()
self.wfile.write("<html><body>It works</body></html>".encode('utf8'))
httpd = HTTPServer(('192.168.43.124', 443), requesthandler)
httpd.socket = ssl.wrap_socket(httpd.socket, keyfile='keys/key.pem', certfile='keys/cert.pem', server_side=True)
httpd.serve_forever()
ssl.SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1108)
This means the client (browser) does not trust your certificate since it is issued by an unknown entity. If you want to use self-signed certificates you have to explicitly import these as trusted for all clients you want to use.
There is no way around this. The certificate in TLS is to make sure that the connection is done with the expected server and not some man in the middle claiming to be the expected server. If a client would trust arbitrary certificates then it would also trust certificates created by a man in the middle attacker.
Below Python builtin HTTPServer works fine with ssl, not showing any error. Not sure how?
The browser will still complain.
The only difference is that the server captures the exception and thus will not crash but continue. You can do the same in your code:
while True:
try:
(c,a)=echoserver.accept()
threading.Thread(target=self.handler, args=(c,a)).start()
except:
pass # ignore error
I am trying to figure out how to setup a SSL link using the Python library Twisted. I have managed to create a certificate that works on the server side, but I am totally stuck when it comes to the client side.
The example from the twisted website states:
The following examples rely on the files server.pem (private key and
self-signed certificate together) and public.pem (the server’s public
certificate by itself).
I have generated myself a certificate and key using OpenSSL:
# Generate Private Key:
openssl genrsa -des3 -out certs/server.key 2048
# Generate Certificate Signing Request:
openssl req -new -key certs/server.key -sha256 -out certs/server.csr
# Generate a Self-Signed Certificate:
openssl x509 -req -days 365 -in certs/server.csr -signkey certs/server.key -sha256 -out certs/server.crt
# Convert the CRT to PEM format:
openssl x509 -in certs/server.crt -out certs/server.pem -outform PEM
For the server-side I am combining certs/server.crt and certs/server.key to create server.pem and trying to use server.crt for public.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData)
I get an error about not starting line. Which certificate should I be using for the client if it's not server.crt please?
In case you want to have certificate based authentication for the clients as well:
I had that issue some time ago and wrote a blog post about my solution.
It also contains a guide to create certificates and sign them with an own certificate authority. You can find the python example code at GitHub.
It uses Twisted for a simple JSONRPCServer
with certificate based authentication for both, server as well as for the clients.
The main thing is to define an own AltCtxFactory for the clients:
# Use our own context factory to use our certificate to authenticate
# against the server and ensure that we are using a strong SSL/TLS
# encryption method
class AltCtxFactory(ssl.ClientContextFactory):
def getContext(self):
# Used TLS/SSL encryption method
sslMethod = SSL.TLSv1_2_METHOD
# Clients private Key, used for authentication
privKey = "<PATH TO YOUR PRIVATE KEY>"
# Clients certificate, used for authentication
certificate = "<PATH TO YOUR CERTIFICATE>"
# Our trusted Certificate Authority for server connections
accepted_ca = "<PATH TO YOUR ACCEPTED CERTIFICATE AUTHORITY>"
self.method = sslMethod
ctx = ssl.ClientContextFactory.getContext(self)
# Ensure that we verify server's certificate and use our own
# verifyCallback method to get further details of invalid certificates
ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
verifyCallback)
# Ensure that we only trust our CA
ctx.load_verify_locations(accepted_ca)
# Use our own Callback mehtod if a password is needed to decrypt our
# private key
ctx.set_passwd_cb(password_cb)
# Use our certificate for authentication against server
ctx.use_certificate_file(certificate)
# Use our private key for authentication against server
ctx.use_privatekey_file(privKey)
return ctx
Feel free to use the code in your projects.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData) I get an error
about not starting line. Which certificate should I be using for the
client if it's not server.crt please?
This should be ssl.Certificate.LoadPEM(certData) if you look at the example on the Twisted howto page.
I am attempting to show proof of concept for iOS Push Notifications from a Google AppEngine application instance using this RPC handler...
PAYLOAD = {'aps': {'alert':'Push!','sound':'default'}}
TOKEN = '[...]'
class APNsTest(BaseRPCHandler):
def get(self, context, name):
self._call_method(context, name)
def send_push(self):
# certificate files
filename = 'VisitorGuidePush'
abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../archive/certificate'))
ca_certs = os.path.abspath(os.path.join(abs_path, '%s.ca'%filename))
certfile = os.path.abspath(os.path.join(abs_path, '%s.crt'%filename))
keyfile = os.path.abspath(os.path.join(abs_path, '%s.key'%filename))
# serialize payload
payload = json.dumps(PAYLOAD)
# APNS server address...
# apns_address = ('api.development.push.apple.com', 443) # Development server
# apns_address = ('api.development.push.apple.com', 2197) # Development server
# apns_address = ('api.push.apple.com', 443) # Production server
apns_address = ('api.push.apple.com', 2197) # Production server
# a socket to connect to APNS over SSL
_sock = socket.socket()
_ssl = ssl.wrap_socket(_sock, keyfile=keyfile,
certfile=certfile,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=ca_certs)
_ssl.connect(apns_address)
# Generate a notification packet
token = binascii.unhexlify(TOKEN)
fmt = '!cH32sH{0:d}s'.format(len(payload))
cmd = '\x00'
message = struct.pack(fmt, cmd, len(token), token, len(payload), payload)
_ssl.write(message)
_ssl.close()
return self.response_result(PAYLOAD)
And need help resolving this error when executing "_ssl.connect(apns_address)"
SSLError: [Errno 1] _ssl.c:507: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
My PEM file (derived from a .p12) and device token were generated a week ago by a mobile developer on our team, suggestions for validating these would be helpful. For now I believe there are current and valid.
While the TLSv1 protocol is being specified, I've notice the handshake failure identifies sslv3.
I have attempted many variations and combination of wrap_socket and apns_address, and am consistently stopped by the handshake failure. Which leads me to suspect a problem with the way I am applying the pem certificate.
The primary references I have been using for wrap_socket are Using OpenSSL and TLS/SSL wrapper for socket objects, not to mention more than a few StackOverflow posts.
Please provide advice concerning the appropriate keyfile, certfile, and ca_certs values and any other advice or resources available for GAE based APNs communication. Thanks ~
Updated question...
The original .p12 has been validated using Pusher, and divided via openssl...
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.key -nodes -nocerts
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.crt -nodes -nokeys
openssl pkcs12 -in vgp.p12 -out VisitorGuidePush.ca -nodes -cacerts
I'm receiving a new error which appears related to the ca_certs...
SSLError: [Errno 0] _ssl.c:343: error:00000000:lib(0):func(0):reason(0)
Removing the ca_certs requirement or passing in other files like the .p12 or the .crt result in a return to the original handshake failure.
Look into using a library such as pyapns, which is what I used to get push notifications to work on GAE. To test whether you're using the correct key/cert file, you can use apps like Pusher. Also, I know that to get SSL capabilities on GAE you have to enable billing, so maybe that's the problem. Good luck!
The appropriate support files start with Creating a Universal Push Notification Client SSL Certificate as a p12 file.
Next, utilizing command line openssl to parse the p12 into the desired certificate and key files...
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.key -nodes -nocerts
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.crt -nodes -nokeys
openssl pkcs12 -in VisitorGuide.p12 -out VisitorGuide.pem -nodes
And finally to obtain a qualified Certificate Authority file (from Troubleshooting Push Notifications)
In addition to the SSL identity (certificate and associated private
key) created by Member Center, you should also install the Entrust CA
(2048) root certificate on your provider.
Entrust.net Certificate Authority (2048) download ~ entrust_2048_ca.cer
Note that every GAE instance hosts its own Certificate Authority at /etc/ca-certificates.crt as described here, Using OpenSSL.
With these files added to your project you can make one of two, equally valid, ssl socket objects...
_ssl = ssl.wrap_socket(_sock, keyfile=VisitorGuide.key,
certfile=VisitorGuide.crt,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=entrust_2048_ca.cer)
...or...
_ssl = ssl.wrap_socket(_sock, certfile=VisitorGuide.pem,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=entrust_2048_ca.cer)
TLS/SSL wrapper for socket objects 17.3.4.3. Combined key and certificate explains why both are valid parameter options.
Before I provide the final code block, I have to point out something concerning the APNs address (this proved to be the key point, allowing me to resolve the handshake failure and obtain an SSL connection between GAE and APNs)
According to the iOS Developer Library APNs Provider API
The first step in sending a remote notification is to establish a connection with the appropriate APNs server:
Development server: api.development.push.apple.com:443
Production server: api.push.apple.com:443
Note: You can alternatively use port 2197 when communicating with APNs. You might do this, for example, to allow APNs traffic through your firewall but to block other HTTPS traffic.
But it wasn't until I dug into the Pusher source that I discovered the APNs addresses to which I could connect...
gateway.sandbox.push.apple.com:2195
gateway.push.apple.com:2195
Without further ado...
class APNsTest(BaseRPCHandler):
def get(self, context, name):
self._call_method(context, name)
def send_push(self):
# certificate files
abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../cert'))
pem_file = os.path.abspath(os.path.join(abs_path, 'VisitorGuide.pem'))
ca_certs = '/etc/ca-certificates.crt'
# APNS server address...
apns_address = ('gateway.sandbox.push.apple.com', 2195)
# apns_address = ('gateway.push.apple.com', 2195)
# a socket to connect to APNS over SSL
_sock = socket.socket()
_ssl = ssl.wrap_socket(_sock, certfile=pem_file,
server_side=False,
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1,
ca_certs=ca_certs)
_ssl.connect(apns_address)
# a notification packet
payload = json.dumps(PAYLOAD)
token = binascii.unhexlify(TOKEN)
fmt = '!cH32sH{0:d}s'.format(len(payload))
cmd = '\x00'
message = struct.pack(fmt, cmd, len(token), token, len(payload), payload)
_ssl.write(message)
_ssl.close()
return self.response_result(PAYLOAD)
...executes without error.
Also, from experimenting, I've found out that SSL related errors on App Engine usually go away when I apply the built-in 3rd party ssl lib:
libraries:
- name: ssl
version: latest
or:
libraries:
- name: ssl
version: "2.7"