good evening!
I am having some issues connecting to an external FTPS server using Airflow's FTPS Hook.
Here's a condensed version of my function which runs in a python operator:
def ftps_connection_test(conn_id, source_path,
source_connection_type):
if source_connection_type == 'FTPS':
ftps_hook=FTPSHook(ftp_conn_id=conn_id)
ftps_hook.get_conn().prot_p()
print('Testing Connection:', ftps_hook.test_connection())
I am getting a handshake error when running the DAG because I need to provide a certificate file when connecting. If I try connecting in WinSCP without this certificate file I get a very similar error.
I've been fiddling with this for a day but I couldn't figure out how to create a connection with a key/cert file using this hook.
Would someone know how to resolve this?
Many thanks in advance!
Tried searching online for similar errors
Tried searching through Airflow's Docs
It seems the default hook uses FTP FTP class which does not support TLS support. Source code here. I think you need the connection of class FTP_TLS. So either you have to manipulate the connection hook or I'd recommend writing a new hook using the other class and passing the certificate and other details as parameters.
Related
I would like to access remote SSH server files within Python, and found fsspec. However, there seems to be few code usage examples.
In particular, I can connect by specifying all SSH config options in the function as:
fsspec.filesystem('sftp', host='XXX.XXX.XXX.XXX', port=XXX, username='XXX', password='XXX')
However, I would like to connect simply as fsspec.filesystem('sftp', host='nickname') as I would do as sftp nickname on console, where I have already set all the config options in .ssh/config.
This is both for convenience and the fact that I do not want to parse my password in plain text.
I have read the API documentation (https://filesystem-spec.readthedocs.io/en/latest/api.html) and searched a bit but could not find a way yet. May I ask if anyone can point me some direction?
Many thanks!
My final goal is to port over a simple mqtt-paho-python script to C++ for integration within a large application.
The python example using paho is quite simple:
client = mqtt.Client(transport="websockets")
client.username_pw_set(settings['username'], password=settings['password'])
client.tls_set_context(context=ssl.create_default_context())
They set up the default TLS context, authenticate with a username and password, and then connect. This works great!
However, now I want to try to get the same secure configuration using paho-mqtt-cpp. The basic example, borrowing from their async examples, goes like this:
mqtt::connect_options connOpts;
connOpts.set_keep_alive_interval(20);
connOpts.set_clean_session(true);
connOpts.set_user_name("username");
connOpts.set_password("password123");
mqtt::ssl_options sslOpts;
connOpts.set_ssl(sslOpts);
mqtt::async_client client("wss://test.mosquitto.org:8081", "myClient");
callback cb(client, connOpts);
client.set_callback(cb);
However, ssl.get_default_context() in python's ssl library seems to do quite a bit of setup for me that isn't replicated in C++; from python's own documentation:
"For client use, if you don’t have any special requirements for your security policy, it is highly recommended that you use the create_default_context() function to create your SSL context. It will load the system’s trusted CA certificates, enable certificate validation and hostname checking, and try to choose reasonably secure protocol and cipher settings."
Most WSS connections I've tried require a certificate, and create_default_context() seems to be able to provide the proper certificates without me generating any myself.
So my questions:
(1) Where are Windows' System Default Certificates that I can use for secure connections? and
(2) What other settings do I need to manually configure that create_default_context() might be setting up for me under the hood?
I've tried looking at the source, but it's not easily discernible where the OS-specific options are.
I am attempting to run a script written in Python 2.7.5 (not using Django). When it tries to connect to a remote mysql server with the MySQLdb.connect() method it throws the following error:
_mysql_exceptions.OperationalError: (2049, "Connection using old (pre-4.1.1) authentication protocol refused (client option 'secure_auth' enabled)")
I have done reading about this issue:
Django/MySQL-python - Connection using old (pre-4.1.1) authentication protocol refused (client option 'secure_auth' enabled)
mysql error 2049 connection using old (pre-4-1-1) authentication from mac
Is there a way to set a parameter in the MySQLdb.connect() method to set secure_auth to false? Without having to change any passwords or running the command from the cmd line. I have looked at the official docs and there does not appear to be anything in there.
I have tried adding secure_auth=False to the parameters but throws an error (shown in the code below).
Python:
def get_cursor():
global _cursor
if _cursor is None:
try:
db = MySQLdb.connect(user=_usr, passwd=_pw, host='external.website.com', port=3306, db=_usr, charset="utf8")
# tried this but it doesnt work (as expect but tried anyway) which throws this error
# TypeError: 'secure_auth' is an invalid keyword argument for this function
# db = MySQLdb.connect(user=_usr, passwd=_pw, host='external.website.com', port=3306, db=_usr, charset="utf8", secure_auth=false)
_cursor = db.cursor()
except MySQLdb.OperationalError:
print "error connecting"
raise
return _cursor
I spent an inordinate amount of time working through the MySQLdb source code and determined that this simply cannot be done without patching the MySQLdb's C wrapping code. Theoretically, you should be able to pass the SECURE_CONNECTION flag to specify that do not want to use the insecure old passwords:
MySQLdb.connect(..., client_flags=MySQLdb.constant.CLIENT.SECURE_CONNECTION)
But the MySQLdb code never actually checks that flag, and never configures the secure_connection option when calling the MySQL connection code, so it always defaults to requiring new-style passwords.
Possible fixes include:
Patch the MySQLdb code
Use an old version of the MySQL client libraries
Update the passwords on the MySQL server
Create a single new user with a new-style password
Sorry I don't have a better answer. I just ran into this problem myself!
I know Moses answer as been validated but I wanted to offer my work around based on what he suggested.
I had previously installed mysql_python for my python and had the brew version of mysql installed.
I deteleted all of that.
I look for a way to install MySQLdb by looking for it last stable version with the source.
I compiled them (followed the isntructions here), installed them and then I looked for a stable version of MySQL client (MySQL website is the best place for that) and install the 5.5 version which was perfectly fitting my requirements.
I made mysql to launch itself automatically and then restarted my computer (but you can just restart apache) and check that all path were correct and the right includes are in the right places (you can check that against the link above).
And now it all works fine!
Hope it helps.
SSL is a separate paramter that you can set in the connection paramter...Here is a note from the source code...Try checking mysql_ssl_set() documentation.
ssl
dictionary or mapping, contains SSL connection parameters;
see the MySQL documentation for more details
(mysql_ssl_set()). If this is set, and the client does not
support SSL, NotSupportedError will be raised.
This document talks about all the secure parameters - http://dev.mysql.com/doc/refman/5.0/en/mysql-ssl-set.html...
I don't see anything to disable secure auth in glance..
I am trying to add authentication to a xmlrpc server (which will be running on nodes of a P2P network) without using user:password#host as this will reveal the password to all attackers. The authentication is so to basically create a private network, preventing unauthorised users from accessing it.
My solution to this was to create a challenge response system very similar to this but I have no clue how to add this to the xmlrpc server code.
I found a similar question (Where custom authentication was needed) here.
So I tried creating a module that would be called whenever a client connected to the server. This would connect to a challenge-response server running on the client and if the client responded correctly would return True. The only problem was that I could only call the module once and then I got a reactor cannot be restarted error. So is there some way of having a class that whenever the "check()" function is called it will connect and do this?
Would the simplest thing to do be to connect using SSL? Would that protect the password? Although this solution would not be optimal as I am trying to avoid having to generate SSL certificates for all the nodes.
Don't invent your own authentication scheme. There are plenty of great schemes already, and you don't want to become responsible for doing the security research into what vulnerabilities exist in your invention.
There are two very widely supported authentication mechanisms for HTTP (over which XML-RPC runs, therefore they apply to XML-RPC). One is "Basic" and the other is "Digest". "Basic" is fine if you decide to run over SSL. Digest is more appropriate if you really can't use SSL.
Both are supported by Twisted Web via twisted.web.guard.HTTPAuthSessionWrapper, with copious documentation.
Based on your problem description, it sounds like the Secure Remote Password Protocol might be what you're looking for. It's a password-based mechanism that provides strong, mutual authentication without the complexity of SSL certificate management. It may not be quite as flexible as SSL certificates but it's easy to use and understand (the full protocol description fits on a single page). I've often found it a useful tool for situations where a trusted third party (aka Kerberos/CA authorities) isn't appropriate.
For anyone that was looking for a full example below is mine (thanks to Rakis for pointing me in the right direction). In this the user and password is stored in a file called 'passwd' (see the first useful link for more details and how to change it).
Server:
#!/usr/bin/env python
import bjsonrpc
from SRPSocket import SRPSocket
import SocketServer
from bjsonrpc.handlers import BaseHandler
import time
class handler(BaseHandler):
def time(self):
return time.time()
class SecureServer(SRPSocket.SRPHost):
def auth_socket(self, socket):
server = bjsonrpc.server.Server(socket, handler_factory=handler)
server.serve()
s = SocketServer.ForkingTCPServer(('', 1337), SecureServer)
s.serve_forever()
Client:
#! /usr/bin/env python
import bjsonrpc
from bjsonrpc.handlers import BaseHandler
from SRPSocket import SRPSocket
import time
class handler(BaseHandler):
def time(self):
return time.time()
socket, key = SRPSocket.SRPSocket('localhost', 1337, 'dht', 'testpass')
connection = bjsonrpc.connection.Connection(socket, handler_factory=handler)
test = connection.call.time()
print test
time.sleep(1)
Some useful links:
http://members.tripod.com/professor_tom/archives/srpsocket.html
http://packages.python.org/bjsonrpc/tutorial1/index.html
I'm in a situation where I need to pass some texts to a prompt generate by a API (seems for API it's a pretty weird behavior, this is the first time I ran into this), like below:
kvm_cli = libvirt.open("qemu+ssh://han#10.0.10.8/system")
then a prompt shows up asking for the ssh password (password for 10.0.10.8 is:), I have to manually type it there in order to move on and yield the kvm_cli object I needed.
I tried to use the pexpect module to deal with this however it's for OS command line instead of API.
It's also possible to work around this by using ssh certification files but it's not a favorable authentication approach in our scenario.
Since our wrapper to the 'open' method is not interactive, we cannot ask the user to input the password, do you guys have any thought how could I address it?
I am not a libvirt user, but I believe that the problem is not in the library, but in the connection method. You seem to be connecting via ssh, so you need to authenticate yourself.
I've been reading the libvirt page on ArchWiki, and I think that you could try:
setting up the simple (TCP/IP socket) connection method, or
setting up key-based, password-less SSH login for your virtual host.