I'd like to catch nicely the error when "No hostkey for host *** is found" and give an appropriate message to the end user. I tried this:
import pysftp, paramiko
try:
with pysftp.Connection('1.2.3.4', username='root', password='') as sftp:
sftp.listdir()
except paramiko.ssh_exception.SSHException as e:
print('SSH error, you need to add the public key of your remote in your local known_hosts file first.', e)
but unfortunately the output is not very nice:
SSH error, you need to add the public key of your remote in your local known_hosts file first. No hostkey for host 1.2.3.4 found.
Exception ignored in: <function Connection.__del__ at 0x00000000036B6D38>
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\pysftp\__init__.py", line 1013, in __del__
self.close()
File "C:\Python37\lib\site-packages\pysftp\__init__.py", line 784, in close
if self._sftp_live:
AttributeError: 'Connection' object has no attribute '_sftp_live'
How to nicely avoid these last lines / this "exception ignored" with a try: except:?
I had the same problem. I solved this by disabling the hostkeys in the cnops:
import pysftp as sftp
FTP_HOST = "sftp.abcd.com"
FTP_USER = "root"
FTP_PASS = ""
cnopts = sftp.CnOpts()
cnopts.hostkeys = None
with sftp.Connection(host=FTP_HOST, username=FTP_USER, password=FTP_PASS, cnopts=cnopts) as sftp:
print("Connection succesfully stablished ... ")
sftp.cwd('/folder/') # Switch to a remote directory
directory_structure = sftp.listdir_attr() # Obtain structure of the remote directory
for attr in directory_structure:
print(attr.filename, attr)
The analysis by #reverse_engineer is correct. However:
It seems that an additional attribute, self._transport, also is defined too late.
The problem can be temporarily corrected until a permanent fix comes by subclassing the pysftp.Connection class as follows:
import pysftp
import paramiko
class My_Connection(pysftp.Connection):
def __init__(self, *args, **kwargs):
self._sftp_live = False
self._transport = None
super().__init__(*args, **kwargs)
try:
with My_Connection('1.2.3.4', username='root', password='') as sftp:
l = sftp.listdir()
print(l)
except paramiko.ssh_exception.SSHException as e:
print('SSH error, you need to add the public key of your remote in your local known_hosts file first.', e)
Update
I could not duplicate this error on my desktop. However, I see in the source for pysftp in the code where it initializes its _cnopts attribute with self._cnopts = cnopts or CnOpts() where cnopts is a keyword parameter to the pysftp.Connection constructor and there is a possibilty of the CnOpts constructor throwing a HostKeysException exception if no host keys are found resulting in the _cnopts attribute not being set.
Try the following updated code and let me know if it works:
import pysftp
import paramiko
class My_Connection(pysftp.Connection):
def __init__(self, *args, **kwargs):
try:
if kwargs.get('cnopts') is None:
kwargs['cnopts'] = pysftp.CnOpts()
except pysftp.HostKeysException as e:
self._init_error = True
raise paramiko.ssh_exception.SSHException(str(e))
else:
self._init_error = False
self._sftp_live = False
self._transport = None
super().__init__(*args, **kwargs)
def __del__(self):
if not self._init_error:
self.close()
try:
with My_Connection('1.2.3.4', username='root', password='') as sftp:
l = sftp.listdir()
print(l)
except paramiko.ssh_exception.SSHException as e:
print('SSH error, you need to add the public key of your remote in your local known_hosts file first.', e)
I think that's a bug in pysftp. You will always have that behavior when pysftp.Connection fails on a No hostkey for XXX found exception, because the failed Connection object (it fails so you can't access it, but it exists in the Python interpreter) gets cleaned up by the GC, which deletes it, and as you can see here, that tries to close the connection first.
We see that close() checks whether the connection is live by checking self._sftp_live. However, the Exception was thrown in the constructor of Connection before that attribute is defined (the exception happens line 132, while _sftp_live is defined line 134), so that leaves the failed Connection object in an inconsistent state and hence the uncaught exception you see.
This has no easy solution that I can think of except introducing a nice bug fix to the pysftp project ;)
Related
I'm trying to open a file which got created as part of the Paramiko exec_command. The file gets created, but when I check if the file exists, it always returns false. How do I check if the file exists and then open the file for reading?
The code written is:
ip = '10.30.2.104'
username = 'root'
password = 'docker'
def checkSSH(ip, username, password, retries=1):
ssh = SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.load_system_host_keys()
for x in range(retries):
try:
ssh.connect(ip, username=username, password=password, timeout=10)
return ssh
except (
paramiko.BadHostKeyException, paramiko.AuthenticationException, paramiko.SSHException, socket.error) as e:
print('Exception occured.. {0}'.format(e))
return False
hdl = checkSSH(ip, username, password)
cmd = 'ls > test.log'
hdl.exec_command(cmd)
a = os.path.exists('/root/test.log')
print(a) >>> if the file exists, this returns true as per the documentation but in this case, it always returns false.
First, you are not waiting for the command to complete.
For that see:
Wait to finish command executed with Python Paramiko
Once the command completes, you can retrieve its exit code using Channel.recv_exit_status():
How can you get the SSH return code using Paramiko?
If it returns 0, you know that everything went fine and no additional check is needed.
Anyway, if you want to check, use SFTP (SFTPClient.stat):
sftp = hdl.open_sftp()
try:
sftp.stat('/root/test.log')
print("exists")
except Exception as e:
print("something is wrong: " + e)
Though the execution of ls command seems strange to me.
If you want to retrieve the directory listing, use SFTP: SFTPClient.listdir_attr
I am using pysftp with Python 3.7 to setup an SFTP client script.
My code (simplified and minimal):
import pysftp
import sys
# Variables
destination_dir = 'BOGUS_DIR'
server = 'myserver.mydomain.com'
user = 'my_user'
key = 'my_key'
port = 22
# cnopts
mycnopts = pysftp.CnOpts()
mycnopts.log = True
mycnopts.compression = True
mycnopts.ciphers = None
mycnopts.hostkeys = None
try:
with pysftp.Connection(server, username=user, private_key=key, port=port, cnopts=mycnopts) as sftp:
try:
with sftp.cd(destination_dir):
print("OK cd worked")
except:
print("NOT OK cd failed")
e = sys.exc_info()
print("Exception: {0}".format(e))
if sftp.isdir(destination_dir):
print("OK isdir")
else:
print("NOT OK isdir")
except:
print("Connection failure.")
e = sys.exc_info()
print("Exception: {0}".format(e))
The output is: OK cd worked
But I know for a fact that BOGUS_DIR does not exist. It is like pysftp does not raise the exception on cd(), or I am catching it wrong (hence my python code is not properly done).
Same for isdir(), whatever I put as parameter, it always returns True even if the directory does not exist.
If I change my connection parameters for something wrong, I do catch the connection failure exception.
Is pyftp processing exceptions wrong, or is my code at fault here? Should I not trust pysftp and use Paramiko directly?
If directory is not exist you getting an error from remote shell where you trying to run the command. In this code you trying to catch the exception that can be raised only by sftp. Maybe, you should check the status code that should be returned by sftp module after each executed shell command
Ok I figured it out, I think.
sftp.cd() does not raise an exception if the directory does not exist. Only operations on the bad directory do. So if I modify my code like this:
....
try:
with sftp.cd(destination_dir):
sftp.listdir()
print("OK ce worked")
except:
print("NOT OK cd failed")
e = sys.exc_info()
print("Exception: {0}".format(e))
....
This way I get an exception since sftp.listdir() cannot work with an inexistent directory.
Almost like sftp.cd does not do anything other than set the value of the current directory, without actually doing anything with it.
I'm trying to connect through Paramiko to an external SFTP and passing an encrypted config file for the credentials. When attempting to connect with the following, I get an error for the exception client.close that states AttributeError: 'Nonetype' object has not attribute 'close':
client = None
try:
client = paramiko.Transport(hostname, port)
client.connect(username=username, password=password)
except Exception as e:
client.close()
return 'Cannot connect to SFTP server: ' + str(e.args[-1]), []
# Go
sftp = paramiko.SFTPClient.from_transport(client)
In the exception you're calling method client.close() which throws an error because the method client.connect did not succeeded, remove this method or create an exception for each possible error with to code handling that specific error.
This is one of those rare cases that it's nearly impossible to reproduce, but I've seen it happen 4 times out of 20.
Here's my open_session method:
def open_session:
self.session = paramiko.SSHClient()
self.host = host
self.username = username
self.password = password
self.session.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
self.session.connect(self.host, username=self.username, port=port, password=self.password, timeout=self.connect_timeout)
except paramiko.ssh_exception.BadHostKeyException:
message = "host key could not be verified"
self.sys_conf.logger.warning(message)
raise BadCredentials(message)
except paramiko.ssh_exception.AuthenticationException:
message = "authentication failed."
self.sys_conf.logger.warning(message)
raise BadCredentials(message)
except (paramiko.ssh_exception.SSHException, socket.error) as e:
message = "could not establish connection, an error occurred: {}".format(e)
self.sys_conf.logger.warning(message)
raise ConnectionFail(message)
except socket.timeout:
message = "could not establish connection, time's out!"
self.sys_conf.logger.warning(message)
raise ConnectionFail(message)
Any particular reason why Paramiko does this?
It should be:
self.session = paramiko.SSHClient()
self.host = host
self.username = username
self.password = password
self.session.load_system_host_keys() # u missed this
self.session.set_missing_host_key_policy(paramiko.AutoAddPolicy())
load_system_host_keys(filename=None)
Load host keys from a system (read-only) file. Host keys read with this method will not be saved back by save_host_keys.
poaramiko
Upon further investigation, it turns out that it's an issue with the machine itself. The code mentioned in my initial question is good as is and worked 300 times in a row without fail on another, more stable machine.
In particular, I am currently trying to check if a connection to a client is valid using the following function:
def mongodb_connect(client_uri):
try:
return pymongo.MongoClient(client_uri)
except pymongo.errors.ConnectionFailure:
print "Failed to connect to server {}".format(client_uri)
I then use this function like this:
def bucket_summary(self):
client_uri = "some_client_uri"
client = mongodb_connect(client_uri)
db = client[tenant_id]
ttb = db.timebucket.count() # If I use an invalid URI it hangs here
Is there a way to catch and throw an exception at the last line if an invalid URI is given? I initially thought that's what the ConnectionFailure was for (so this could be caught when connecting) but I was wrong.
If I run the program with an invalid URI, which fails to run, issuing a KeyboardInterrupt yields:
File "reportjob_status.py", line 58, in <module>
tester.summarize_timebuckets()
File "reportjob_status.py", line 43, in summarize_timebuckets
ttb = db.timebucket.count() #error
File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 1023, in count
return self._count(cmd)
File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 985, in _count
with self._socket_for_reads() as (sock_info, slave_ok):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/Library/Python/2.7/site-packages/pymongo/mongo_client.py", line 699, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/Library/Python/2.7/site-packages/pymongo/mongo_client.py", line 663, in _get_socket
server = self._get_topology().select_server(selector)
File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 121, in select_server
address))
File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 106, in select_servers
self._condition.wait(common.MIN_HEARTBEAT_INTERVAL)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 358, in wait
_sleep(delay)
The serverSelectionTimeoutMS keyword parameter of pymongo.mongo_client.MongoClient controls how long the driver will try to connect to a server. The default value is 30s.
Set it to a very low value compatible with your typical connection time¹ to immediately report an error. You need to query the DB after that to trigger a connection attempt :
>>> maxSevSelDelay = 1 # Assume 1ms maximum server selection delay
>>> client = pymongo.MongoClient("someInvalidURIOrNonExistantHost",
serverSelectionTimeoutMS=maxSevSelDelay)
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> client.server_info()
This will raise pymongo.errors.ServerSelectionTimeoutError.
¹ Apparently setting serverSelectionTimeoutMS to 0 might even work in the particular case your server has very low latency (case of a "local" server with very light load for example)
It is up to you to catch that exception and to handle it properly. Something like that:
try:
client = pymongo.MongoClient("someInvalidURIOrNonExistantHost",
serverSelectionTimeoutMS=maxSevSelDelay)
client.server_info() # force connection on a request as the
# connect=True parameter of MongoClient seems
# to be useless here
except pymongo.errors.ServerSelectionTimeoutError as err:
# do whatever you need
print(err)
will display:
No servers found yet
Hi to find out that the connection is established or not you can do that :
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure
client = MongoClient()
try:
# The ismaster command is cheap and does not require auth.
client.admin.command('ismaster')
except ConnectionFailure:
print("Server not available")
serverSelectionTimeoutMS
This defines how long to block for server selection before throwing an
exception. The default is 30,000 (milliseconds). It MUST be
configurable at the client level. It MUST NOT be configurable at the
level of a database object, collection object, or at the level of an
individual query.
This default value was chosen to be sufficient for a typical server
primary election to complete. As the server improves the speed of
elections, this number may be revised downward.
Users that can tolerate long delays for server selection when the
topology is in flux can set this higher. Users that want to "fail
fast" when the topology is in flux can set this to a small number.
A serverSelectionTimeoutMS of zero MAY have special meaning in some
drivers; zero's meaning is not defined in this spec, but all drivers
SHOULD document the meaning of zero.
https://github.com/mongodb/specifications/blob/master/source/server-selection/server-selection.rst#serverselectiontimeoutms
# pymongo 3.5.1
from pymongo import MongoClient
from pymongo.errors import ServerSelectionTimeoutError
client = MongoClient("mongodb://localhost:27000/", serverSelectionTimeoutMS=10, connectTimeoutMS=20000)
try:
info = client.server_info() # Forces a call.
except ServerSelectionTimeoutError:
print("server is down.")
# If connection create a new one with serverSelectionTimeoutMS=30000
serverSelectionTimeoutMS doesn't work for me (Python 2.7.12, MongoDB 3.6.1, pymongo 3.6.0). A. Jesse Jiryu Davis suggested in a GitHub issue that we attempt a socket-level connection first as a litmus test. This does the trick for me.
def throw_if_mongodb_is_unavailable(host, port):
import socket
sock = None
try:
sock = socket.create_connection(
(host, port),
timeout=1) # one second
except socket.error as err:
raise EnvironmentError(
"Can't connect to MongoDB at {host}:{port} because: {err}"
.format(**locals()))
finally:
if sock is not None:
sock.close()
# elsewhere...
HOST = 'localhost'
PORT = 27017
throw_if_mongodb_is_unavailable(HOST, PORT)
import pymongo
conn = pymongo.MongoClient(HOST, PORT)
print(conn.admin.command('ismaster'))
# etc.
There are plenty of problems this won't catch, but if the server isn't running or isn't reachable, this'll show you right away.
Can also be checked this way:
from pymongo import MongoClient
from pymongo.errors import OperationFailure
def check_mongo_connection(client_uri):
connection = MongoClient(client_uri)
try:
connection.database_names()
print('Data Base Connection Established........')
except OperationFailure as err:
print(f"Data Base Connection failed. Error: {err}")
check_mongo_connection(client_uri)
For pymongo >= 4.0 the preferred method is to use ping command instead of deprecated ismaster:
from pymongo.errors import ConnectionFailure
client = MongoClient()
try:
client.admin.command('ping')
except ConnectionFailure:
print("Server not available")
To handle auth failures, include OperationFailure:
except OperationFailure as err:
print(f"Database error encountered: {err}")
Source: mongo_client.py