At the moment I am trying to figure out how to do an MQTT publish using the .crt and .pem files for our application.
below is some pseudo code for what I am trying to do:
connect to an existing topic over tls
publish a message
import paho.mqtt.publish as mqtt
import paho.mqtt.client as mqttclient
topic = "Some/Topic/Goes/Here"
my_ca_cert = open("<path_to_cert>\ca.crt", 'rb').read()
my_pri_cert = open("<path_to_cert>\private.pem", 'rb').read()
my_key_cert = open("<path_to_cert>\certificate.pem", 'rb').read()
mqttc = mqttclient.Client("Python_Ex_Pub")
mqttc.tls_set(my_ca_cert, certfile=my_pri_cert, keyfile=my_key_cert)
mqttc.connect("<gateway_address>", 8883)
mqttc.publish(topic_name, "This is a test pub from Python.")
mqttc.loop(timeout=2.0, max_packets=1)
When I run the script the following error is thrown:
Traceback (most recent call last):
File "mqtt_pub_test.py", line 9, in <module>
mqttc.tls_set(my_ca_cert, certfile=my_pri_cert, keyfile=my_key_cert)
File "C:\Python27\lib\site-packages\paho\mqtt\client.py", line 557, in tls_set
raise IOError(ca_certs+": "+err.strerror)
IOError: -----BEGIN CERTIFICATE-----
<cert_info_here>
-----END CERTIFICATE-----: No such file or directory
I read through the TLS example on the paho doc page, but just do not understand how I should be passing the crt / pem files in my code.
At one point I was only pointing to the folder containing the files, and even went as far as to chmod 777 the folder but at that point I was getting access denied from python at run time.
Any help is appreshiated
Do you want to do TLS client authentication (that is, your Python script needs to authenticate to the server / MQTT broker)? Or do you want your Python script to behave like a web browser, and just validate the server certificate?
If you only want the latter, I have had success using the tls_set() method in the Paho Python client when I point it to a PEM file containing the server's certificate. And that's the only argument you need to pass on tls_set() to have the Paho client validate the server certificate, and connect to the broker using TLS. For example:
mqttc.tls_set("/home/bob/certificates/mqttbrokercertificate.pem")
How do you obtain the mqtt broker's certificate in PEM format? The easiest way is with openssl:
openssl s_client -host mqtt.broker.hostname.com -port 8883 -showcerts
Redirect the output to a file, and delete every line in the file except what is between the "BEGIN CERTIFICATE" and "END CERTIFICATE" lines (inclusive -- be sure to include these lines as well). This is a good article here on StackOverflow about how to save a server's SSL certificate using openssl:
How to save server SSL certificate to a file
Lastly, you need to be sure of what version of TLS your broker supports, and make sure your Python client also supports it. For example, the IBM Watson IoT Platform requires TLS 1.2. The ssl module in Python 2.7 (which is built on openssl) does not support TLS 1.2. Generally, you need Python 3.X, and openssl of at least 1.0.1. Here is how you can set the TLS version on the Paho client (don't forget to import ssl):
mqttc.tls_set("/home/bob/certificates/mqttbrokercertificate.pem", tls_version=ssl.PROTOCOL_TLSv1_2)
If you want TLS client authentication, that's probably better handled in an entirely separate article. But I hope this helps with TLS server authentication using the Paho Python client.
Looking at the error it would suggest that the tls_set function is expecting paths to the files not the contents of the file to be passed in.
Related
I am running a python script - ssh.py on my local machine to transfer file and directory from one remote server (ip = 35.189.168.20) to another remote server (ip = 10.243.96.94)
This is how my code looks:
HOST = "35.189.168.207"
USER = "sovith"
PASS = "xxx"
destHost = "10.243.96.94"
destUser = "root"
destPass = "xxx"
#SSH Connection
client1=paramiko.SSHClient()
client1.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client1.connect(HOST,username=USER,password=PASS)
#SFTP inside SSH server connection
with pysftp.Connection(host=destHost, username=destUser, password=destPass) as sftp:
#put build directory - sftp.put_r(source, destination)
sftp.put_r('/home/sovith/build' , '/var/tmp')
sftp.close()
client1.close()
Let me just tell you that all directory paths and everything is correct. I just feel that there's some logically mistake inside the code. The output i got after execution is :
Traceback (most recent call last):
File "ssh.py", line 108, in <module>
func()
File "ssh.py", line 99, in func
sftp.put_r('/home/sovith/nfmbuild' , '/var/tmp')
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pysftp/__init__.py", line 427, in put_r
os.chdir(localpath)
FileNotFoundError: [Errno 2] No such file or directory: '/home/sovith/build'
Can you just correct the mistake from my code or suggest any better method to accomplish the task. In simpler words, i want a script to copy files and directory between two remote server.
That's not possible. Not the way you do it. The fact that you open a connection to one remote server, does not make the following code magically work, as if it was executed on that server. It still runs on the local machine. So the code is trying to upload local files (which do not exist).
There's actually no way to transfer files between two remote SFTP servers from local machine.
In general, you will need to download the files from the first server to a local temporary directory. And then upload them to the second server.
See Python PySFTP transfer files from one remote server to another remote server
Another option is to connect to one remote server using SSH and then run SFTP client on the server to transfer the files to/from the second server.
But that's not copying from one SFTP server to another SFTP server. That's copying from one SSH server to SFTP server. You need an SSH access, mere SFTP access is not enough.
To execute a command of a remote SSH server, use pysftp Connection.execute. Though using pysftp to execute a command on a server is a bit overkill. You can use directly Paramiko instead:
Python Paramiko - Run command
(pysftp is just a wrapper around Paramiko with more advanced SFTP functions)
From your error message, can you confirm that the source directory /home/sovith/build (or /home/bgnanasekaran/nfmbuild) exists?
A good practice would be to check that before the call to sftp.put_r(), ie :
from pathlib import Path
d = Path('/home/sovith/build')
if d.exists():
sftp.put_r('/home/sovith/build' , '/var/tmp')
else:
print("Your source doesn't exist")
I'm setting up a new server using Pytoh, tornado.
I'm going to set HTTPS.
But when server loads server certificate files, load_cert_chain function generates an error.
I purchased those certificate files from GoDaddy and it was okay on Apache Server before.
So certificate file and key file match and ca file is also okay.
python version: 3.7.4
tornado version : 4.5.2
openssl version: OpenSSL 1.1.1c 28 May 2019
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl_ctx.load_verify_locations("../ca.crt")
ssl_ctx.verify_mode = ssl.CERT_REQUIRED
https_server = tornado.httpserver.HTTPServer(app, ssl_options=ssl_ctx)
This is error message.
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)
I solved this problem. I use NginX server to host python server with digital signed certificates.
I am using Python Requests library to connect to a REST server by using .pem certificates to authenticate who I am to establish a connection so I can start collecting data, parsing it, etc. When I run my program via Eclipse or the terminal, I get this error:
[('system library', 'fopen', 'Permission denied'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'system lib')]
However, if I run as 'sudo' - the requests library works as intended and I am able to retrieve the data. Unfortunately, running as 'sudo' has side effects that where the default Python interpreter is the root interpreter, which is Python2. However, there are a lot of library dependencies that are needed from Anaconda.
For context, here is the function I am using to establish a connection:
PEM_FILE = os.path.expanduser("path/to/pem/file.pem") #Path is to a folder in root level
def set_up_connection(self):
#URL's to request a connection with
rest_auth = 'https://www.restwebsite.com/get/data'
ip_address = self.get_ip_address()
body = json.dumps({'userid':'user', 'password':'pass', 'ip_address':ip_address})
try:
resp = self.session.post(rest_auth, data=body, cert=PEM_FILE, verify=False)
values = resp.json()
token = values['token']
except Exception as e:
print(e)
token = None
return token, ip_address
TLDR; Using 'python rest_connector.py' renders an error. Running that command as sudo works.
Context for the certificates: The .pem cert permissions is set to 600 (rw-------).
To try and solve my problem running as sudo, I have started a terminal and run 'sudo -Es' which sets the terminal to run as root and uses Anaconda as my default interpreter, BUT I end up with a handshake error:
[('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert unknown ca'), ('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]
If someone can help me solve it this way, it would be a nice temporary fix, but I still need to be able to run this without sudo.
Thanks, in advance.
The username needs to be able to read the file. You can verify by running ls path_to_file.pem.
If you have changed ownership of the file, you might still be missing executable permissions on the directories containing the file.
You can potentially fix that with chmod -R +x ~/path/to_directory_containing_perm
Ok, so I managed to solve this and I'll post what I did in case anyone else stumbles upon this with a similar problem.
Permissions being set to 600 for the certs and pem file, ownership being set to root, and performing the openssl hashing function, the only problem was where the certs were placed in the sub directories.
While I placed the certs into 'etc/pki/tls/certs', they actually belonged in 'etc/ssl/certs'. The same goes for the .pem file except that .pem would be placed in a restricted folder, 'private'. After moving the files to the correct folder. Then I was able to set the verify param for the request to the cert_path and everything worked like I needed it to.
resp = self.session.post(rest_auth, data=body, cert=PEM_FILE, verify=cert_path)
'etc/pki/tls/certs' is the directory for Fedora distribution of Linux. 'etc/ssl/certs' is the directory for the Ubuntu distribution of Linux.
I hope this helps someone.
I am trying to connect my python paho mqtt client to my broker through tls using my own certificate authority. I generated necessary files and configured my RabbitMQ broker to use them. My idea is that the client authenticates the server but client itself doesn't need to be authenticated.
ca.cert.pem
cert.pem
key.pem
I know these should work correctly because I also have a scala paho mqtt client that works correctly.
Currently I am running the file directly from the terminal on my mac. I am using an virtuanenv created by python 3.5.2 and I have a file subscribe.py
import paho.mqtt.client as paho
import ssl
def on_message(clnt, userdata, msg):
print(msg.topic+" "+str(msg.payload))
def on_connect(client, userdata, rc):
print("Connected with result code "+str(rc))
mqttc.subscribe("foo")
mqttc = paho.Client()
mqttc.on_message = on_message
mqttc.on_message = on_message
mqttc.tls_set("ca.cert.pem", tls_version=ssl.PROTOCOL_TLSv1_2)
mqttc.connect("address", 8883, 60)
mqttc.loop_forever()
When I run the file I receive the following error
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)
I have also tried changing the tls_version to TLSv1_1, TLSv1 and leaving it out completely. ca.cert.pem is in the same folder as the subscribe.py file
The example on paho website seems very simple so what am I missing here? Why do I receive the error on my python client?
Have you tried mosquitto clients? There are lots of possibility for the problem you have encountered, so best way to figured it out is trying another way.
mosquitto_sub -h address -p 8883 --cafile ca.cert.pem -t "#" -d -v
d flag is for debug messages, v flag is verbose mode.
If mosquitto client cannot connect with ca, you can try:
mosquitto_sub -h address -p 8883 --cafile ca.cert.pem -t "#" -d -v --insecure
The insecure flag will ignore CA checking for server. If add the insecure flag can make you connected, then maybe the CA is wrong. You can use openssl to debug it then.
I have simple client/server SSL code which worked fine on Python 3.2. However, I decided to switch over to 2.7 (due to abundance of third party modules), and now the code is failing. The code is as follows:
Client:
def connect(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(('localhost', 4430))
self.ssl_sock = ssl.wrap_socket(self.sock, cert_reqs = ssl.CERT_NONE, ssl_version = ssl.PROTOCOL_TLSv1)
Server:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((self.hostname, 4430))
self.sock.listen(5)
while True:
self.newsocket, self.fromaddr = self.sock.accept()
self.ssl_sock = ssl.wrap_socket(
self.newsocket,
server_side = True,
certfile = "cert.pem",
ssl_version=ssl.PROTOCOL_TLSv1
)
self._handle_client(self.ssl_sock)
The cert.pem (located in same directory as server .py file):
-----BEGIN RSA PRIVATE KEY-----
(812 "random" characters here)
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
(1260 "random" characters here)
-----END CERTIFICATE-----
The failure is as follows:
1. server waits on self.sock.accept()
2. client connects with self.sock.connect()
3. server proceeds to wrap_socket, which fails with the following error:
Traceback (most recent call last):
File "C:\Program Files\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\workspace\projectc\server\server\clientlistener.py", line 49, in run
ssl_version=ssl.PROTOCOL_TLSv1
File "C:\Program Files\Python27\lib\ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "C:\Program Files\Python27\lib\ssl.py", line 141, in __init__
ciphers)
SSLError: [Errno 336265218] _ssl.c:351: error:140B0002:SSL routines:SSL_CTX_use_PrivateKey_file:system lib
P.S.
1) I have suspicions about the wrap_socket call because even when I use a non-existent file for certfile, the same error results.
2) I tried the alternative wrapping on the python documentation on ssl (i.e. with ssl.SSLContext) and it's odd that 'module' object has no attribute 'SSLContext', when it is part of the ssl module. This wasn't an issue when I tried it with Python 3.2.
Update:
I have found that the problem only occurs when I'm doing "run" from within Eclipse, not so when I run both files from separate command prompts. Still investigating the issue...
Update 2:
I tried a very simple client/server script with the SAME code and it works. Now, with the server code sitting inside a multiprocessing subprocess (launched by the main server process), it seems not to work. Related?
Problem solved. I believe Eclipse was holding some residual information, though I never figured out what information it was holding. It might have to do with the migration from python 3.2 to 2.7, though I previously already changed that in the run config. Resetting the Eclipse environment worked (simply rebooting the computer wasn't enough).
my solution was that when creating my .pem file i set a blank password and assumed it meant no pasword. so the server was still expecting to use a password. i had to manually remove the password.
here is a little how to guide if it helps anyone (also, please note i was coding python for a server side to an iOS app)
NOTE: need to follow directions from apple’s developer website to create certificate first
then export the .p12 file,
by exporting the embedded private key that is created (in ‘keychain access’),
NOT the actual certificate
————————————————————————————————————
————————————————————————————————————
FOR DEVELOPMENT CERT:
After getting the p12 file, it needs to be converted to the PEM format by executing this command from the terminal:
$ openssl pkcs12 -clcerts -nokeys -out apns-dev-cert.pem -in apns_dev.p12
$ openssl pkcs12 -nocerts -out apns-dev-key.pem -in apns_dev.p12
If you wish to remove the passphrase execute the following:
(NOTE: using a ‘blank’ password when exporting/converting, is still indeed setting a password,
hence you should still execute the following if you intend to have no password)
$ openssl rsa -in apns-dev-key.pem -out apns-dev-key-noenc.pem
Finally, you need to combine the key and cert files into a apns-dev.pem file we will use when connecting to APNS:
$ cat apns-dev-cert.pem apns-dev-key-noenc.pem > apns-dev.pem
————————————————————————————————————
FOR PRODUCTION CERT:
After getting the p12 file, it needs to be converted to the PEM format by executing this command from the terminal:
$ openssl pkcs12 -clcerts -nokeys -out apns-prod-cert.pem -in apns_prod.p12
$ openssl pkcs12 -nocerts -out apns-prod-key.pem -in apns_prod.p12
If you wish to remove the passphrase execute the following:
(NOTE: using a ‘blank’ password when exporting/converting, is still indeed setting a password,
hence you should still execute the following if you intend to have no password)
$ openssl rsa -in apns-prod-key.pem -out apns-prod-key-noenc.pem
Finally, you need to combine the key and cert files into a apns-dev.pem file we will use when connecting to APNS:
$ cat apns-prod-cert.pem apns-prod-key-noenc.pem > apns-prod.pem