I've a working code which works good when I mention the file name exactly how it is in folder. But these files have date and time added to its name. How can I make the file path for the same to read? Below is my code.
import paramiko
import os
paramiko.util.log_to_file('logfile.log')
host = "ftp.servername.com"
port = 22
transport = paramiko.Transport((host, port))
password = "mypass"
username = "myuser"
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
filepath = '/Home/user/Automation_2021-07-14_170139.csv'
localpath = 'file/dkc.csv'
sftp.get(filepath, localpath)
sftp.close()
transport.close()
How can I pass the * in the filepath? I wanted to make it like below.
filepath = '/Home/user/Automation_*.csv'
localpath = 'file/dkc.csv'
sftp.get(filepath, localpath)
You cannot pass a wildcard directly to Paramiko's SFTPClient.get.
You have to first find out the exact name and then use it with the get.
See List files on SFTP server matching wildcard in Python using Paramiko
Below code helped me to fix the issue.
latest = 0
latestfile = None
for fileattr in sftp.listdir_attr():
if fileattr.filename.startswith('Automation_DKC') and fileattr.st_mtime > latest:
latest = fileattr.st_mtime
latestfile = fileattr.filename
I am trying to connect to an SFTP server but I am getting an error:
pysftp.exceptions.ConnectionException: ('NAMEOFSERVER#sftp.NAMEOFHOST.gov', 22)
I have no clue why this is happening, here is the portion of the code that is causing problems:
myHostname = "NAMEOFHOST"
myUsername = "" #there is no username
myPassword = "PASSWORD"
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,cnopts=cnopts) as sftp:
print ("Connection succesfully stablished ... ")
Any help is appreciated, thank you!
If there is no username, why don't you put None, instead of an empty string. None != ""
So:
myHostname = "NAMEOFHOST"
myPassword = "PASSWORD"
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, password=myPassword,cnopts=cnopts) as sftp:
print ("Connection succesfully stablished ... ")
Give it a try and say whether this solves your issue.
I have a file >500 MB to download using sftp connection, I tried using pysptp and getting error SSHException: Server connection dropped:
import pysftp
import sys
myHostname = "dbfiles.xyz.org"
myUsername = "XXXX"
myPassword = "YYYY"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword,cnopts=cnopts) as sftp:
print("Connection succesfully stablished ... ")
localFilePath = 'c:/....'
remoteFilePath = sftp.listdir('/folder/')
for filename in remoteFilePath:
if 'string_to_match' in filename:
local_path = localFilePath + filename
print (filename)
print (local_path)
sftp.get("folder/" + filename, local_path)
And getting SSHException: Server connection dropped: EOF error after 18MB of file is downloaded. Is there any way I can put limit on amount to data downloaded or can delay this get process in order to get full file, i tried several ways but because of large file size, unable to download complete file. Any help appreciated.
Go to Paramiko library in stfp_file.py and change the MAX_REQUEST_SIZE to 1024. It worked for me. You can find the package here:
/home//.local/lib/python3.8/site-packages/paramiko/sftp_file.py
So recently I wrote a code that put files from one folder to an SFTP folder. It works generally, but many times it fails to upload the data, and I have no clue way. Do you have any tips?
import os
import pysftp
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
#Set these to make the code run
shared_aim_fd='' #the folder where you copy from
sftp_user=''
sftp_pw=''
sftp_site=''
sftp_folder_name='' #where you want to copy to
for od_files in os.listdir(shared_aim_fd):
myUsername =str(sftp_user)
myPassword =str(sftp_pw)
myHostname=str(sftp_site)
mySFTPFolderName=str(sftp_folder_name)
try:
with pysftp.Connection(host=myHostname, username=myUsername, password=myPassword, cnopts=cnopts) as sftp:
print ("Connection succesfully stablished ... ")
#Make sure we have the folder where we want to push it
if not sftp.isdir(mySFTPFolderName):
#if not, then create one
sftp.makedirs(mySFTPFolderName)
sftp.put(shared_aim_fd+od_files,str(mySFTPFolderName+'/'+od_files))
sftp.close()
except:
print("Upload failed")
pass
I'm working on a simple tool that transfers files to a hard-coded location with the password also hard-coded. I'm a python novice, but thanks to ftplib, it was easy:
import ftplib
info= ('someuser', 'password') #hard-coded
def putfile(file, site, dir, user=(), verbose=True):
"""
upload a file by ftp to a site/directory
login hard-coded, binary transfer
"""
if verbose: print 'Uploading', file
local = open(file, 'rb')
remote = ftplib.FTP(site)
remote.login(*user)
remote.cwd(dir)
remote.storbinary('STOR ' + file, local, 1024)
remote.quit()
local.close()
if verbose: print 'Upload done.'
if __name__ == '__main__':
site = 'somewhere.com' #hard-coded
dir = './uploads/' #hard-coded
import sys, getpass
putfile(sys.argv[1], site, dir, user=info)
The problem is that I can't find any library that supports sFTP. What's the normal way to do something like this securely?
Edit: Thanks to the answers here, I've gotten it working with Paramiko and this was the syntax.
import paramiko
host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))
password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)
sftp = paramiko.SFTPClient.from_transport(transport)
import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)
sftp.close()
transport.close()
print 'Upload done.'
Thanks again!
Paramiko supports SFTP. I've used it, and I've used Twisted. Both have their place, but you might find it easier to start with Paramiko.
You should check out pysftp https://pypi.python.org/pypi/pysftp it depends on paramiko, but wraps most common use cases to just a few lines of code.
import pysftp
import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
host = "THEHOST.com" #hard-coded
password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
with pysftp.Connection(host, username=username, password=password) as sftp:
sftp.put(localpath, path)
print 'Upload done.'
Here is a sample using pysftp and a private key.
import pysftp
def upload_file(file_path):
private_key = "~/.ssh/your-key.pem" # can use password keyword in Connection instead
srv = pysftp.Connection(host="your-host", username="user-name", private_key=private_key)
srv.chdir('/var/web/public_files/media/uploads') # change directory on remote server
srv.put(file_path) # To download a file, replace put with get
srv.close() # Close connection
pysftp is an easy to use sftp module that utilizes paramiko and pycrypto. It provides a simple interface to sftp.. Other things that you can do with pysftp which are quite useful:
data = srv.listdir() # Get the directory and file listing in a list
srv.get(file_path) # Download a file from remote server
srv.execute('pwd') # Execute a command on the server
More commands and about PySFTP here.
If you want easy and simple, you might also want to look at Fabric. It's an automated deployment tool like Ruby's Capistrano, but simpler and of course for Python. It's build on top of Paramiko.
You might not want to do 'automated deployment' but Fabric would suit your use case perfectly none the less. To show you how simple Fabric is: the fab file and command for your script would look like this (not tested, but 99% sure it will work):
fab_putfile.py:
from fabric.api import *
env.hosts = ['THEHOST.com']
env.user = 'THEUSER'
env.password = 'THEPASSWORD'
def put_file(file):
put(file, './THETARGETDIRECTORY/') # it's copied into the target directory
Then run the file with the fab command:
fab -f fab_putfile.py put_file:file=./path/to/my/file
And you're done! :)
fsspec is a great option for this, it offers a filesystem like implementation of sftp.
from fsspec.implementations.sftp import SFTPFileSystem
fs = SFTPFileSystem(host=host, username=username, password=password)
# list a directory
fs.ls("/")
# open a file
with fs.open(file_name) as file:
content = file.read()
Also worth noting that fsspec uses paramiko in the implementation.
With RSA Key then refer here
Snippet:
import pysftp
import paramiko
from base64 import decodebytes
keydata = b"""AAAAB3NzaC1yc2EAAAADAQABAAABAQDl"""
key = paramiko.RSAKey(data=decodebytes(keydata))
cnopts = pysftp.CnOpts()
cnopts.hostkeys.add(host, 'ssh-rsa', key)
with pysftp.Connection(host=host, username=username, password=password, cnopts=cnopts) as sftp:
with sftp.cd(directory):
sftp.put(file_to_sent_to_ftp)
Twisted can help you with what you are doing, check out their documentation, there are plenty of examples. Also it is a mature product with a big developer/user community behind it.
There are a bunch of answers that mention pysftp, so in the event that you want a context manager wrapper around pysftp, here is a solution that is even less code that ends up looking like the following when used
path = "sftp://user:p#ssw0rd#test.com/path/to/file.txt"
# Read a file
with open_sftp(path) as f:
s = f.read()
print s
# Write to a file
with open_sftp(path, mode='w') as f:
f.write("Some content.")
The (fuller) example: http://www.prschmid.com/2016/09/simple-opensftp-context-manager-for.html
This context manager happens to have auto-retry logic baked in in the event you can't connect the first time around (which surprisingly happens more often than you'd expect in a production environment...)
The context manager gist for open_sftp: https://gist.github.com/prschmid/80a19c22012e42d4d6e791c1e4eb8515
Paramiko is so slow. Use subprocess and shell, here is an example:
remote_file_name = "filename"
remotedir = "/remote/dir"
localpath = "/local/file/dir"
ftp_cmd_p = """
#!/bin/sh
lftp -u username,password sftp://ip:port <<EOF
cd {remotedir}
lcd {localpath}
get {filename}
EOF
"""
subprocess.call(ftp_cmd_p.format(remotedir=remotedir,
localpath=localpath,
filename=remote_file_name
),
shell=True, stdout=sys.stdout, stderr=sys.stderr)
PyFilesystem with its sshfs is one option. It uses Paramiko under the hood and provides a nicer paltform independent interface on top.
import fs
sf = fs.open_fs("sftp://[user[:password]#]host[:port]/[directory]")
sf.makedir('my_dir')
or
from fs.sshfs import SSHFS
sf = SSHFS(...
Here's a generic function that will download any given sftp url to a specified path
from urllib.parse import urlparse
import paramiko
url = 'sftp://username:password#hostname/filepath.txt'
def sftp_download(url, dest):
url = urlparse(url)
with paramiko.Transport((url.hostname, 22)) as transport:
transport.connect(None,url.username,url.password)
with paramiko.SFTPClient.from_transport(transport) as sftp:
sftp.get(url.path, dest)
Call it with
sftp_download(url, "/tmp/filepath.txt")