I've been following the steps in this tutorial to create a IoT hub Module, so I can experiment with module twins for my sample device. Every works well as long as I create the module with SAS tokens, as described in the tutorial.
But now, I'd like to be able to create a module by using a sample X.509 CA cert. The problem is that when I try to use the following to create my module:
module_client = IoTHubModuleClient.create_from_x509_certificate(cert, HOSTNAME, DEVICE_ID, MODULE_ID)
...with cert being the X.509 cert I created for my sample device using those instructions (i.e. ./certGen.sh create_device_certificate SampleDevice4), I get the following error:
ConnectOperation: completing with error UnauthorizedError('Connection
Refused: not authorised.')
I know that my sample device cert is valid because I'm able to use it successfully in this tutorial about device twins. So I assume that I need to create a new X.509 cert/key pair specifically for that module, as opposed to using the device cert. But I couldn't find how to create those. Does anyone know how to generate X.509 CA certs for a module as opposed to a device?
I found that one can generate a module X.509 cert by using the following syntax:
./certGen.sh create_device_certificate SampleDevice4\/mymodule. Note that the slash must be escaped so that openssl can properly generate the common name (CN) for the cert.
As far as I can tell, this is undocumented behavior, so I don't know if this convention can be used reliably in the future. I left feedback on the tutorial to ask if this is the case.
My final goal is to port over a simple mqtt-paho-python script to C++ for integration within a large application.
The python example using paho is quite simple:
client = mqtt.Client(transport="websockets")
client.username_pw_set(settings['username'], password=settings['password'])
client.tls_set_context(context=ssl.create_default_context())
They set up the default TLS context, authenticate with a username and password, and then connect. This works great!
However, now I want to try to get the same secure configuration using paho-mqtt-cpp. The basic example, borrowing from their async examples, goes like this:
mqtt::connect_options connOpts;
connOpts.set_keep_alive_interval(20);
connOpts.set_clean_session(true);
connOpts.set_user_name("username");
connOpts.set_password("password123");
mqtt::ssl_options sslOpts;
connOpts.set_ssl(sslOpts);
mqtt::async_client client("wss://test.mosquitto.org:8081", "myClient");
callback cb(client, connOpts);
client.set_callback(cb);
However, ssl.get_default_context() in python's ssl library seems to do quite a bit of setup for me that isn't replicated in C++; from python's own documentation:
"For client use, if you don’t have any special requirements for your security policy, it is highly recommended that you use the create_default_context() function to create your SSL context. It will load the system’s trusted CA certificates, enable certificate validation and hostname checking, and try to choose reasonably secure protocol and cipher settings."
Most WSS connections I've tried require a certificate, and create_default_context() seems to be able to provide the proper certificates without me generating any myself.
So my questions:
(1) Where are Windows' System Default Certificates that I can use for secure connections? and
(2) What other settings do I need to manually configure that create_default_context() might be setting up for me under the hood?
I've tried looking at the source, but it's not easily discernible where the OS-specific options are.
I am trying to connect to a crate database with python
from crate import client
url = '434.342.435.2:4400' # Faked these numbers for purposes of this post
conn = client.connect(url)
It seems like I need to pass the cert_file and key_file arguments to client.connect which point to my .pem and .key files. Looking in the documentation, I cannot find any resource to create or download these files.
Any advice? Even a comment pointing me to a good resource for beginners would be appreciated.
So cert and key files are part of the TLS encryption of a HTTP(S) connection that are required if you use a self-signed certificate :)
This seems to be a very good explanation of the file types
As mfussenegger explained in the comment, these files are optional and only required if your CrateDB instance is "hidden" behind a reverse proxy server like NGINX or Apache with a self-signed certificate.
A small green lock on the far left of your browser's address bar indicates HTTPS (and therefore TLS) with known certificates.
Typically certificates signed by an unknown CA - like yourself - result in a warning page and a red indicator:
Since you are also referring to username and password, they usually indicate some sort of auth (maybe basic auth) which is not yet supported by crate-python :(
I'm using Python 2.7.5 (not 3.X) and I need to verify a FTPS (FTP-TLS) public certificate. That is, I want to verify it against the standard certificate authority, not a custom key. (Similar to HTTPS.)
I see some options but I cannot get them to work:
The FTP_TLS() class doesn't seem to offer the ability to verify certificates, unless I'm mistaken:
class ftplib.FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, timeout]]]]]]])
I've read into the certifi and also M2Crypto, but while I can connect and transfer using FTP/TLS, I can't seem to find a way to verify the certificate.
Also, I don't think I will be able to use the CURL libraries in this case :( Just a note.
Let's try to make it into a possible answer: http://heikkitoivonen.net/blog/2008/10/14/ssl-in-python-26
The resource referenced by mcepl is no longer available over http, but only using https.
https://heikkitoivonen.net/blog/2008/10/14/ssl-in-python-26
So much for 301 redirects.
Today I faced one interesting issue.
I'm using the foursquare recommended python library httplib2 raise
SSLHandshakeError(SSLError(1, '_ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed'),)
while trying to request an oauth token
response, body = h.request(url, method, headers=headers, body=data)
in
_process_request_with_httplib2 function
does anyone know why this happens?
If you know that the site you're trying to get is a "good guy", you can try creating your "opener" like this:
import httplib2
if __name__ == "__main__":
h = httplib2.Http(".cache", disable_ssl_certificate_validation=True)
resp, content = h.request("https://site/whose/certificate/is/bad/", "GET")
(the interesting part is disable_ssl_certificate_validation=True )
From the docs:
http://bitworking.org/projects/httplib2/doc/html/libhttplib2.html#httplib2.Http
EDIT 01:
Since your question was actually why does this happen, you can check this or this.
EDIT 02:
Seeing how this answer has been visited by more people than I expected, I'd like to explain a bit when disabling certificate validation could be useful.
First, a bit of light background on how these certificates work. There's quite a lot of information in the links provided above, but here it goes, anyway.
The SSL certificates need to be verified by a well known (at least, well known to your browser) Certificate Authority. You usually buy the whole certificate from one of those authorities (Symantec, GoDaddy...)
Broadly speaking, the idea is: Those Certificate Authorities (CA) give you a certificate that also contains the CA information in it. Your browsers have a list of well known CAs, so when your browser receives a certificate, it will do something like: "HmmmMMMmmm.... [the browser makes a supiciuous face here] ... I received a certificate, and it says it's verified by Symantec. Do I know that "Symantec" guy? [the browser then goes to its list of well known CAs and checks for Symantec] Oh, yeah! I do. Ok, the certificate is good!
You can see that information yourself if you click on the little lock by the URL in your browser:
However, there are cases in which you just want to test the HTTPS, and you create your own Certificate Authority using a couple of command line tools and you use that "custom" CA to sign a "custom" certificate that you just generated as well, right? In that case, your browser (which, by the way, in the question is httplib2.Http) is not going to have your "custom" CA among the list of trusted CAs, so it's going to say that the certificate is invalid. The information is still going to travel encrypted, but what the browser is telling you is that it doesn't fully trust that is traveling encrypted to the place you are supposing it's going.
For instance, let's say you created a set of custom keys and CAs and all the mambo-jumbo following this tutorial for your localhost FQDN and that your CA certificate file is located in the current directory. You could very well have a server running on https://localhost:4443 using your custom certificates and whatnot. Now, your CA certificate file is located in the current directory, in the file ./ca.crt (in the same directory your Python script is going to be running in). You could use httplib2 like this:
h = httplib2.Http(ca_certs='./ca.crt')
response, body = h.request('https://localhost:4443')
print(response)
print(body)
... and you wouldn't see the warning anymore. Why? Because you told httplib2 to go look for the CA's certificate to ./ca.crt)
However, since Chrome (to cite a browser) doesn't know about this CA's certificate, it will consider it invalid:
Also, certificates expire. There's a chance you are working in a company which uses an internal site with SSL encryption. It works ok for a year, and then your browser starts complaining. You go to the person that is in charge of the security, and ask "Yo!! I get this warning here! What's happening?" And the answer could very well be "Oh boy!! I forgot to renew the certificate! It's ok, just accept it from now, until I fix that." (true story, although there were swearwords in the answer I received :-D )
Recent versions of httplib2 is defaulting to its own certificate store.
# Default CA certificates file bundled with httplib2.
CA_CERTS = os.path.join(
os.path.dirname(os.path.abspath(__file__ )), "cacerts.txt")
In case if you're using ubuntu/debian, you can explicitly pass the path to system certificate file like
httplib2.HTTPSConnectionWithTimeout(HOST, ca_certs="/etc/ssl/certs/ca-certificates.crt")
Maybe this could be the case:
I got the same problem and debugging the Google Lib I found out that the reason was that I was using an older version of httplib2(0.9.2). When I updated to the most recent (0.14.0) it worked.
If you already install the most recent, make sure that some lib is not installing an older version of httplib2 inside its dependencies.
When you see this error with a self-signed certificate, as often happens inside a corporate proxy, you can point httplib2 to your custom certificate bundle using an environment variable. When, for example, you don't want to (or can't) modify the code to pass the ca_certs parameter.
You can also do this when you don't want to modify the system certificate store to append your CA cert.
export HTTPLIB2_CA_CERTS="\path\to\your\CA_certs_bundle"