I'm trying to have a simple function collect certificates from servers. Using Python 3.10.8 and my code looks something this:
import ssl
def certgrab(dom):
address = (dom, 443)
try:
f = ssl.get_server_certificate(address)
except Exception as clanger:
return {'clanger': clanger}
print(f)
This is fine when I try it against 'google.com' or 'microsoft.com'. But most websites return the following error:{'clanger': ConnectionRefusedError(10061, 'No connection could be made because the target machine actively refused it', None, 10061, None)}.
I was wondering if it was a rejection because the sites don't like the user-agent (requests works fine with everything I test against, but obviously cannot grab the cert (unless it secretly can - which would be great!). But I cannot find a way of specifying one in the SSL library.
I'm at a bit of a loss as it works against 'google.com' and 'microsoft.com' (but then I suppose they may have set their sites to be generous / forgiving regarding what types of connections they support).
ConnectionRefusedError(10061, 'No connection could be made because the target machine actively refused it'
This has nothing to do with certificates, not even with TLS. This is a connection error at the TCP level, i.e. even before any TLS and certificates are in effect.
But most websites return the following error ...
If this is really "most websites" then you might have serious problems in your infrastructure which limit access to large parts of the internet. Or, you might need to use a proxy - but ssl.get_server_certificate does not support a proxy.
Related
I am wanting to use Python to retrieve the remote server certificate (not validate or check it in any way). I have retrieved the server certificate using both methods ``ssl.get_server_certificateandSSLSocket.getpeercert`.
The main reason I had to try SSLSocket.getpeercert over ssl.get_server_certificate was that the timeout value on the TLS handshake was not being honored with the ssl.get_server_certificate. One of the hosts I was trying to the the server certificate had some problem and would hang my python script during the TLS handshake and only the SSLSocket.getpeercert would time this out.
I also notice I cannot retrieve the server certificates from very old systems that use TLS 1.0 or even SSL with SSLSocket.getpeercert and there is no place to specify to the ssl_version like there is in ssl.get_server_certificate.
So I see both methods retrieve the server certificate and each seems to have different issues. But what are the differences with what each does? When would I use one over the other?
The context related to my question is that I work for a company in the networking area. This company has several stores around the country where DVRs are accessible through port 2781 and a domain for people to access security cameras, the problem is that in order for these people to successfully access DVRs through the domain and port you must have a DMZ configured in the modem of the stores. To corroborate the DMZ I'm trying to use Python with the sockets module but I don't understand the best way to do it yet.
import socket
s = socket.socket()
s.connect((domain, port))
s.close()
Once I make the proper connection which is the best way to check if there is a communication? Work it with an exception or just use socket.recv and detect if it is empty?
In order for connect to succeed there already has to be some kind of connection be done. Otherwise the TCP handshake would fail. Thus the first step would be to check if connect succeeds or throws an exception.
It can still be possible that there is some deep packet inspection firewall in place which does not block the initial connection but only blocks the later data exchange. To find out if this is the case you have to do actual bidirectional communication. But how this communication should look like depends on the specific application protocol which is unknown in your case. Still you need to check that a) sending and receiving works (catching exceptions) and b) that a response returns the expected data.
My code :-
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("www.python.org" , 80))
s.sendall(b"GET https://www.python.org HTTP/1.0\n\n")
print(s.recv(4096))
s.close()
Why the output shows me this:-
b'HTTP/1.1 500 Domain Not Found\r\nServer: Varnish\r\nRetry-After: 0\r\ncontent-type: text/html\r\nCache-Control: private, no-cache\r\nconnection: keep-alive\r\nContent-Length: 179\r\nAccept-Ranges: bytes\r\nDate: Tue, 11 Jul 2017 15:23:55 GMT\r\nVia: 1.1 varnish\r\nConnection: close\r\n\r\n\n\n\nFastly error: unknown domain \n\n\nFastly error: unknown domain: . Please check that this domain has been added to a service.'
How can I fix it?
This is wrong on multiple levels:
to access a HTTPS resource you need to create a TLS connection (i.e. ssl_wrap on top of an existing TCP connection, with proper certificate checking etc) and then send the HTTP request. Of course the TCP connection in this case should go to port 443(https) not 80 (http).
the HTTP request should only contain the path, not the full URL
the line end must be \r\n not \n
you better send a Host header too since many severs require it
And that's only the request. Properly handling the response is a different topic.
I really really recommend to use an existing library like requests. HTTP(S) is considerably more complex as most think who only had a look at a few traffic captures.
import requests
x = requests.get('https://www.python.org')
print x.text
With the requests library, HTTPS requests are very simple! If you're doing this with raw sockets, you have to do a lot more work to negotiate a cipher and etc. Try the above code (python 2.7).
I would also note that, in my experience, Python is excellent for doing things quickly. If you are learning about networking and cryptography, try writing a HTTPS client on your own using sockets. If you want to automate something quickly, use the tools that are available to you. I almost always use requests for this type of task. As an additional note, if you're interested in parsing HTML content, check out the PyQuery library. I've used it to automate interaction with many web services.
Requests
PyQuery
I have already implemented SSL communication where client application verifies the identity of the SSL server application using flask. Now I want SSL server application to verify the identity of the SSL-client application. Is it possible with flask ? How do I verify client certificate ? During first handshake client is sending CSR and in response I am sending back certificate signed by Self Signed CA certificate.
But I am not yet clear how the client will be verified by server while next communication. Is there any callback for cert verification. Link on Google groups says its not possible to have ssl authentication on Flask. in order to do this one need to use webserver like apache,ngnix. Is this the only way to authenticate client ?
There is one more thing that I want to achieve that I need to identify each client based on their certificate. is that even possible with flask.
my question could be naive as I am not yet much familiar to flask
Disclaimer
Before I start I would note #Emanuel Ey's comment. That you would want to consider if this was being done on a production or development server first. For example; if you are using Apache WebServer the HTTPS component can be done from Apache. The only thing you would do differently is pass through the certificate details as options and your server app would then verify the serial number within the app itself.
It is Possible
But it the way it is possible is not considered good programming practice. Unfortunately, it's not accessible from flask.request and not possible with the Flask package. However, Flask uses Werkzeug and it is possible by patching the werkzeug.serving package where will be writing your main Flask code. It is not recommended because you may want to update Flask or Werkzeug later and your patch might break and need to be re-factored. i.e. from 0.9 to 1.0.
This provides a solution without using a web server. But I would recommend the web server/environment variable combo. It is cleaner and comparatively good practice.
I have done some testing to see if this is easy to implement. I was able to confirm that this method can work using the latest development codebase 'Werkzeug-0.10_devdev_20141223-py2.7'.
You'll probably want to verify of the serial number (seed number) found in each certificate (and maybe even some other variables). As you may know, the serial is unique to each certificate and is determined during the certificate generation process by you on the server side. It helps to store this along with the clients record and certificate information (where appropriate) in order to verify client certificate serial number later on. Note: It may require alterations between hex and base 10 decimal.
Werkzeug dev_2014122
What I did was to add in the following options to the werkzeug.serving.BaseWSGIServer.__init__ call to wrap_socket().
Use these;
server_side=True, ca_certs= '/etc/apache2/ssl/ca.pem', cert_reqs=ssl.CERT_REQUIRED
ca_certs: Use this to verify against, this is the CA cert used to generate the client certificates)
ssl.CERT_REQUIRED: require client certificate verification against ca_certs
Note: If the client certificate is does not pass initial verification you will not be able to fetch the client certificate. It will be None.
Then in my Flask test class I patched verify_request
where
def verify_request(self, request, client_address):
cert = request.getpeercert(True)
raw = decoder.decode(cert)[0]
print "Serial Number of your certificate is: % " % str(raw[0][1])
# todo: do checks & if serial no is ok then return true
return True
werkzeug.serving.BaseWSGIServer.verify_request = verify_request
This proved it is possible but you'll probably want to investigate the request handlers of the HTTPServer class that the BaseWSGIServer inherits to find a better way to do a call back or override.
Werkzeug 0.9.X
If you are using Werkzeug 0.9.X I'm assuming you are using the import from OpenSSL import SSL. see code snippet here. I have not tested this.
Some of the calls you may be interested in for this version would be;
- Context.set_verify(mode, callback)
- Connection.get_peer_certificate()
Clarification
What I do not understand is your reference to sending a CSR during the first handshake. If this is your process of client certificate generation you may want to rethink how you do this in the context of your system and environment. If I could have some more information I could comment further..
Also, 'handshake' in an SSL/TLS context generally refers to the action of creating the secure connection in the first place using an existing certificate. Immediately after handshaking, loosely speaking, a connection is established.
I'm trying to set up a server and client in python where the server authenticates clients using SSL with certificates. There are a lot of examples of SSL certificates online, but everything I've found has the server providing a certificate to the client and the client checking it. I need the server to ensure that the client has the authority to connect to the server. I understand how to generate and send certificates and the basics of how they work. I would type out my code, but my client/server without SSL is working fine and I've been referencing this for SSL. The client/server example at the bottom of that page summarizes my understanding of SSL certs in python.
I realize this isn't much to go on, but if someone could explain the basic modifications to that example to have the server authenticate the client instead of the other way around, that would be awesome. Alternatively, a link to an example or even just some socket methods to investigate would be very helpful. Let me know if more information is needed. I don't mean to be vague and promise I've spent all morning looking for info myself :).
Edit: I'm trying to stick to the basic ssl library. Aka "import ssl".
You would use SSLSocket.getpeercert to get the certificate. The client would need to specify a key and certificate when wrapping the socket just like the server side. On the server side, you will also need to pass ca_certs="path_to_ca_cert_file" and probably also want to specify cert_reqs=CERT_REQUIRED (see. args for ssl.wrap_socket.
In addition to this, it sounds like you might be looking to do certificate based client authentication/authorization? This is a simple matter of using getpeercert to get the client certificate and accessing fields within the certificate to use in a normal authentication path (i.e. Common Name == User Id)
Not really sure what your question refers, however you can see SSL in Python, other resource for SSL in Python, Validating SSL, get SSL Certificate information and you probably found other good links.