I have an SFTP server. I can take data by transferring/downloading files. Is there a way that I can do without downloading files?
My code is as below:
# Connection to the SFTP server
with pysftp.Connection(hostname, username, passowrd, port) as sftp:
with sftp.cd('directory'):
sftp.get('filename.txt')
This code downloads file to my local machine.
Yes and no. You can use the data from the remote (SFTP) server without storing the files to a local disk.
But you cannot use data locally without downloading them. That's impossible. You have to transfer the data to use them – at least to a memory of the local machine.
See A way to load big data on Python from SFTP server, not using my hard disk.
My answer there talks about Paramiko. But pysftp is a just a thin wrapper around Paramiko. Its Connection.open is directly mapped to underlying Paramiko's SFTPClient.open. So you can keep using pysftp:
with sftp.open('filename.txt', bufsize=32768) as f:
# use f as if you have opened a local file with open()
Though I'd recommend you not to: pysftp vs. Paramiko.
Related
i found this vulnerability CVE-2021-4189 (https://bugzilla.redhat.com/show_bug.cgi?id=2036020)
in ftblib library in python
CVE description : A flaw was found in Python, specifically in the FTP (File Transfer Protocol) client library in PASV (passive) mode. The issue is how the FTP client trusts the host from the PASV response by default. This flaw allows an attacker to set up a malicious FTP server that can trick FTP clients into connecting back to a given IP address and port. This vulnerability could lead to FTP client scanning ports, which otherwise would not have been possible.
now i am confused if this vulnerability affected my code using a ftp upload in passive mode as below or should i use ftp.set_pasv(False) to use active mode only
# Init Connection
ftp = FTP()
ftp.connect(FTP_ADDRESS, FTP_PORT)
ftp.login(*FTP_CREDENTIALS)
Yes, your code would be vulnerable, if you did not update to a fixed version of Python yet. And if it is using IPv4 (PASV) for data connections (what is likely does).
I'd not recommend switching to the active mode though, as that will likely cause you problems.
Rather fix your code the same way the ftplib fix works – by ignoring the IP returned by the server in the FTP.makepasv.
See SmartFTP implementation in my answer to:
Cannot list FTP directory using ftplib – but FTP client works
Having that said, I do not consider the vulnerability serious enough to even worry about – unless your code connects to random FTP servers.
I can't find anything about this exception. I am trying to rename a remote file on a local (Windows) SFTP server with fsspec. Paramiko behind the scenes is doing a posix_rename(). What does the error mean?
fs.rename(old_file_path, new_file_path)
Paths look like /folder/file.ext.
I can rename files with other FTP clients on that same server.
Indeed, fsspec SFTPFileSystem.mv calls Paramiko SFTPClient.posix_rename. That's imo a bad choice. The SFTPClient.posix_rename internally uses a proprietary OpenSSH posix-rename#openssh.com extension, which is naturally not supported by most other SFTP servers (such as yours).
I do not know what is the best solution/workaround. You can probably add your own "file system" implementation based on SFTPFileSystem, reimplementing SFTPFileSystem.mv to call standard Paramiko SFTPClient.rename (which uses standard SFTP rename request).
Actually, I just found that the SFTPClient is exposed through the SFTPFileSystem and I can call rename() on it directly, which worked!
fs.ftp.rename("testfile.txt", "x")
This question already has an answer here:
How to send CSV file directly to an FTP server
(1 answer)
Closed 3 years ago.
Is there any way to directly send files from one API to another FTP server without downloading them to local in Python 3.
Currently we downloading from one API to local and then sending it to FTP server, want to avoid that hop from data flow by directly sending files to server.
you can use byte data of the file(it will store as in-memory) and pass that to another API.
One of the options would be having another API function (TransferFile, ...), which will transfer data from API server to FTP site. Then you just call that API method from your code without downloading data to the local server.
The FTP protocol has provision for initiating a data transfert between two remote hosts from a third party client. This is called the proxy mode. Unfortunately, most servers disable it for security reasons, because it used to be a very efficient way for DOS attacks.
If you have control on both servers and if both use FTP and if they are not publicly exposed, this can be very efficient.
In any other use case, the data will have to pass through the client. The best that can be done is to open both connections and transfer data to the target host as soon as it has been received from the source without storing it on disk.
I am using python 3.4 and pysftp , (pysftp suspected to be working on 3.4)
Pysftp is a wrapper over paramiko.
I have no problem downloading files.
I can also upload small files.
When i am uploading files that take longer than a few seconds to complete however i get an error.
I monitored my interent connection, after about 3 seconds there is no more uploading taking place.
after ~5 minutes i recieve an EOFError
I also experimented with the paramiko module with the same results.
I can upload files using open ssh as well as filezilla without problem.
with pysftp.Connection(host="host",username="python",
password="pass",port=2222) as srv:
print('server connected')
srv.put(file_name)
I would like to be able to upload files greater than a few kb... what am i missing?
It seems that paramiko is not adjusting window during file uploading. You can increase window_size manualy:
with pysftp.Connection(host="host",username="python",
password="pass",port=2222) as srv:
print('server connected')
channel = srv.sftp_client.get_channel()
channel.lock.acquire()
channel.out_window_size += os.stat(file_name).st_size
channel.out_buffer_cv.notifyAll()
channel.lock.release()
srv.put(file_name)
It works for me, but sometimes it is not enough for large files so I add some extra bytes. I think, that some packets may be lost and it depends on the connection.
I'm trying to send over multiple files from one server to another using Python. I've found a few ssh2 libraries, but either I can't find documentation on them (e.g. ssh), or they don't seem to support mput.
Anyone know of any sftp libraries which support mput?
Paramiko is a library that handles SSH and related things, such as SFTP, but it only supports a regular put, no mput that I can see.
What exactly does mput do? My sftp client doesn't have that command...
Guessing from the name, I'm thinking "multiple puts" or something like that, to send multiple files in one go? If that is the case, I suggest just looping over your list of files and using put.
I was faced with a similar problem a long time ago and could not get a satisfactory mput-method to run in Python. So the paramiko library seemed to make the most sense to me.
To enable progress output or other actions in your Python application, it is advantageous to send the files individually in a for-loop. However, the overhead increases minimally with this variant.
A small Paramiko-example code :
import pysftp
import os
list_files_to_transfer = []
# Check if List is empty
if list_files_to_transfer:
# Advanced connection options beyond authentication
cnopts = pysftp.CnOpts()
# Compression for lower Transfer-Load
cnopts.compression = True
cnopts.hostkeys = None
# Establish a connection with the SFTP server
with pysftp.Connection(host=host, username=username, port=port,
private_key=os.path.abspath(path_private_key),
cnopts=cnopts) as sftp:
# Change to the specified remote directory
with sftp.cd(self.path_remote_sink_folder):
# browse the list of files
for file in list_files_to_transfer:
# Upload the file to the server
sftp.put(file)