I'm trying to use paramiko to get a file via SFTP.
It connects, I can list directories and it even downloads the first megabyte or so of the file but then it just hangs. No exception, no error, nothing. It just hangs there indefinitely.
Here's the code I'm working with:
import paramiko
t = paramiko.Transport( host )
t.connect( username=uname, password=passwd )
f = paramiko.SFTPClient.from_transport( t )
print f.listdir()
f.get( fname, fname ) #it hangs on this line :\
I have sftp access to the host in question but no shell access.
The host contains a single file that I need to fetch regularly and process in a python script.
Any help with this problem or alternate solutions to doing SFTP in Python are greatly appreciated :)
I was experiencing the same problem as Ulfur. He posted his own fix/workaround as a comment to another answer, so I decided to add it as a proper answer to make it more visible.
The basic idea is to not use the .get() method, but to loop over the lines. The following is a Python 3 implementation.
transport = None
sftp = None
sftp_path = 'some/sftp/path'
dst_path = '/some/local/path'
try:
transport = paramiko.Transport((SFTP_HOST, SFTP_PORT))
transport.set_log_channel('delapi')
transport.connect(username=SFTP_USER, password=SFTP_PASS)
sftp = paramiko.SFTPClient.from_transport(transport)
with sftp.open(sftp_path, 'r') as remote:
with open(dst_path, 'w') as local:
for line in remote:
local.write(line)
except SSHException:
print("SSH error")
finally:
if sftp:
sftp.close()
if transport:
transport.close()
I suggest you fire up Wireshark on the client and see what's happening at the protocol level. You won't be able to read the data in the packets as it will be encrypted, but you will see what's going on at the TCP/IP level and that might provide a clue.
Related
I have a simple script that successfully downloads a 75MB file over FTP:
try:
ftp = ftplib.FTP(host)
ftp.login(username, password)
ftp.cwd(source_dir)
except ftplib.all_errors as e:
print('Ftp error = ', e)
return False
# Check filename exists
if filename in ftp.nlst():
local_filename = os.path.join(dest_dir, filename)
lf = open(local_filename, "wb")
ftp.retrbinary("RETR " + filename, lf.write)
lf.close()
print(filename, ' successfully downloaded')
else:
print(filename, ' not found in the path ', source_dir)
ftp.quit()
This script works fine on both my home and work laptops when run from Spyder IDE or a Windows scheduled task.
I have deployed the exact same script to a Windows Virtual Machine on Azure.
Files less than 10MB seem to download ok.
Files larger than 30MB return an exception:
421 Data timeout. Reconnect. Sorry.
I get around 700 Mbps on Azure and only around 8Mbps on my home network.
It looks like a timeout. I can see the file is partially downloaded.
I tried setting ftp.set_pasv(False), but this then returns me 500 Illegal Port, which is to be expected. I understand passive is the preferred approach anyhow.
What else can I do to troubleshoot and resolve this issue?
Just some suggestions for you.
According to the wiki page for File Transfer Protocol, FTP may run in active or passive mode, as the figure below. In active mode, the client requires a listening port for incoming data from the server. However, due to the listening port of client for FTP server is random assigned, you can not prepare in advance to add the port in NSG inbound rules. So you should use passive mode in the client side on Azure VM with FTP.set_pasv(True) or without FTP.set_pasv(False).
For the issue 421 Data timeout. Reconnect. Sorry., please check the timeout setting in your FTP server, such as the data_connection_timeout property of vsftpd.conf file of vftp, to set enough long value of time out
Try to set a timeout value longer then the global default setting for ftplib.FTP(host='', user='', passwd='', acct='', timeout=None, source_address=None) function.
Try to use function FTP.set_debuglevel(level) to debug output more details for your script to find out the possible reason.
Hope it helps.
I am uploading files to a remote sftp server using the pysftp module in Python. About 25% of the time a file won't upload and I receive the error: '[Errno 2] No such file.'
I am connecting to the remote sftp server with simply a username and password. No SSH key is being used. Upon establishing the connection I pass an instance of cnopts() with hostkeys set to None since no SSH keys are being used. I then loop through each file and and perform a put() to upload each file to the sftp server. The first couple files typically upload successfully, but then by the thrid or fourth file I typically receive the [Errno 2] error. If I rerun the script on the same file that just failed, it will upload just fine. Therefore, it doesn't seem to be an issue with the local or remote path since everything is the same the first run to the second.
Connection Code
elif self.ctype == 'sftp':
if self.pkpath == None:
#set pysftp to not check for ssh key, only use password
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
#connecting to sftp server
self.conn = pysftp.Connection(self.address, self.user, password=self.password,
cnopts=cnopts)
else:
self.conn = pysftp.Connection(self.address, self.user, password=self.password,
private_key=self.pkpath)
Put Code (uploadFile function)
def uploadFile(self,fileName,fdestname=None):
...
if fdestname is None:
fattr = self.conn.put(fileName, confirm=True, preserve_mtime=True)
else:
fattr = self.conn.put(fileName, fdestname, confirm=True, preserve_mtime=True)
Looping Code
elif inputdict['source'] == 'unprocessed':
Sftp.uploadFile(os.path.join(inputdict['unprocessedfolder'],vimsf), vimsfm)
print Sftp.lst()
Since the local and remote file paths are correct and subsequent loops upload the previously failed files just fine, I'd expect that all files would upload successfully on the first try.
In this particular case, the '[Errno 2] No such file.'error was showing up because the input on the put method was set to True. The files were being removed from the destination folder so quickly after the upload, the confirmation process could not complete. It would then return the '[Errno 2] No such file.' error. The issue was intermittent since about 50% of the time, the files would stick around long enough in the destination directory that the confirmation could complete successfully. Setting the confirm bool to False solved the issue.
Very good catch here. Was pulling my hair out, but turned out that on the post side, they were processing the file quickly and removing it. This caused the error.
head, tail = os.path.split(filename)
with pysftp.Connection('hotname', username='uname', password='pass',
cnopts=cnopts) as sftp:
with sftp.cd('/mydir/subdir/'):
sftp.put(filename,tail,confirm=False)
I use Paramiko to put a file to an SFTP server:
import paramiko
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put(local_path, remote_path)
Now, I would like to check if it worked. The idea is that I compare the checksum of the local file and the remote one (that is located on the SFTP server).
Does Paramiko functionality allows to do that? If it is the case, how exactly it works?
With the SFTP, running over an encrypted SSH session, there's no chance the file contents could get corrupted while transferring. So unless it gets corrupted when reading the local file or writing the remote file, you can be pretty sure that the file was uploaded correctly, if the .put does not throw any error.
try:
sftp.put(local_path, remote_path)
except:
# Something went wrong
If you want to test explicitly anyway:
While there's the check-file extension to the SFTP protocol to calculate a remote file checksum, it's not widely supported. Particularly it's not supported by the most widespread SFTP server implementation, the OpenSSH. See What SFTP server implementations support check-file extension.
If you are lucky to connect to another SFTP server that supports the extension, you can use the Paramiko's SFTPFile.check method.
If not, your only option is to download the file back and compare locally.
If you have a shell access to the server, you can of course try to run some shell checksum command (sha256sum) over a separate shell/SSH connection (or channel) and parse the results. But that's not an SFTP solution anymore. See Comparing MD5 of downloaded files against files on an SFTP server in Python.
if the file doesn't upload then the method will throw an error, so u can check for error
import paramiko
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
try:
sftp.put(local_path, remote_path)
except IOError:
#'Failure'
except OSError:
#'Failure'
I wrote a simple code to upload a file to a SFTP server in Python. I am using Python 2.7.
import pysftp
srv = pysftp.Connection(host="www.destination.com", username="root",
password="password",log="./temp/pysftp.log")
srv.cd('public') #chdir to public
srv.put('C:\Users\XXX\Dropbox\test.txt') #upload file to nodejs/
# Closes the connection
srv.close()
The file did not appear on the server. However, no error message appeared. What is wrong with the code?
I have enabled logging. I discovered that the file is uploaded to the root folder and not under public folder. Seems like srv.cd('public') did not work.
I found the answer to my own question.
import pysftp
srv = pysftp.Connection(host="www.destination.com", username="root",
password="password",log="./temp/pysftp.log")
with srv.cd('public'): #chdir to public
srv.put('C:\Users\XXX\Dropbox\test.txt') #upload file to nodejs/
# Closes the connection
srv.close()
Put the srv.put inside with srv.cd
Do not use pysftp it's dead. Use Paramiko directly. See also pysftp vs. Paramiko.
The code with Paramiko will be pretty much the same, except for the initialization part.
import paramiko
with paramiko.SSHClient() as ssh:
ssh.load_system_host_keys()
ssh.connect(host, username=username, password=password)
sftp = ssh.open_sftp()
sftp.chdir('public')
sftp.put('C:\Users\XXX\Dropbox\test.txt', 'test.txt')
To answer the literal OP's question: the key point here is that pysftp Connection.cd works as a context manager (so its effect is discarded without with statement), while Paramiko SFTPClient.chdir does not.
import pysftp
with pysftp.Connection(host="www.destination.com", username="root",
password="password",log="./temp/pysftp.log") as sftp:
sftp.cwd('/root/public') # The full path
sftp.put('C:\Users\XXX\Dropbox\test.txt') # Upload the file
No sftp.close() is needed, because the connection is closed automatically at the end of the with-block
I did a minor change with cd to cwd
Syntax -
# sftp.put('/my/local/filename') # upload file to public/ on remote
# sftp.get('remote_file') # get a remote file
I have this 30 virtual machines and I am doing everything manually right now which is a kind of trouble for me. I am trying now to write a script so I can connect to them all automatically and do my desired task. I made a flow diagram of what I need to do. Can anybody just give me some hints on how will i achieve this task programatically. I am attaching the flow diagram.
Thanks in Advance.
Please Right Click On Image and Click View Image to See the Flow Diagram.
Reading a text file and getting the data is trivial:
with open('host.txt', 'r') as inf:
lines = inf.readlines()
hostlist = [ln.split() for ln in lines]
Now hostlist should be a list of lists;
[['192.168.0.23', 'root', 'secret'], ['192.168.0.24', 'root', 'secret2'] ...
But your list shouldn't have to contain more than the hostnames. IP adresses can be gotten from DNS that you have to configure anyway, and ssh can login without passwords if configured correctly.
Putting the passwords for all your virtual hosts in a plain text file has security concerns. If you want to go that route, make sure to restrict access to that file!
You can use subprocess to execute commands. I would suggest using rsync to push the desired files to the virtual machines. That minimizes network traffic. And you can deploy directly from filesystem to filesystem without having to roll a tarball. It can be as simple as
status = subprocess.check_output(['rsync', '-av', localdir, remotedir])
Where localdir is the directory where the files for the virtual machine in question are stored (it should end with a '/'), and 'remotedir' is the hostname::directory on the virtual machine where the data should land (this should not end with a '/').
For executing commands remotely, ssh is the way to go. Configure that for passwordless login, using ssh's authorized_keys file on each remote host. Then you don't need to put passwords in your list of hosts.
Fabric is best solution for you. Fabric based on paramiko ( which based on libssh2), it makes very easy to work with commands on remote host and provides function to upload and download fiels from remote host.
Here it is http://docs.fabfile.org/en/1.5/
Docs about put function here
I don't get your question in the flow diagram, however you can use paramiko as suggested and I have a large number of background utilities written on top of paramiko which enables support people to monitor remote web servers on the browser. Snippet below,
client = paramiko.SSHClient()
client.set_missing_host_key_policy( paramiko.AutoAddPolicy() )
client.load_system_host_keys()
client.connect( '192.168.100.1', port=4001, username='monitor', password='XXXX' )
cmds = [ "sed -i 's/\/bin\/date -u/\/bin\/date/g' /etc/cron.hourly/reboot" ]
for cmd in cmds:
if __DEBUG_MODE__:
print 'Executing..... ' + cmd
stdin, stdout, stderr = client.exec_command( cmd )
Also if you want to push files below is the snippet,
def setupSFTPClient(self, ip_add):
print 'Setting SFTP client: ' + ip_add
tb = 'Finished SFTP-ing. '
try:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.load_system_host_keys()
client.connect(ip_add, port=4001, username='monitor', password='XXXX')
sftp_client = client.open_sftp()
# NB you need the filename for remote path otherwise
# paramiko barfs with IOError: Failure
sftp_client.put( '/home/projects/portal_release.tgz', '/var/ND/portal_release.tgz' )
sftp_client.put( '/home/projects/portal_installer.sh', '/var/ND/portal_installer.sh' )
sftp_client.close()
except Exception, e:
print e
tb = traceback.format_exc()
finally:
print tb
client.close()