How To Download Folder From SFTP Using Python's Paramiko [duplicate] - python

I want to download a directory with unknown contents recursively via SSH and have been trying Paramiko. I have seen several examples how to upload directories but none that covers recursive download.
I can list all items in a directory but haven't been able to find a way of knowing if the item is a file (to download) or a directory (to call recursively).
transport = paramiko.Transport((MY_IP, 22))
transport.connect(username=MY_NAME, password=MY_PASS)
sftp = paramiko.SFTPClient.from_transport(transport)
file_list = sftp.listdir(path='/home/MY_HOME_DIR')
for item in file_list:
# Here is an item name... but is it a file or directory?
print(item)
sftp.close()
transport.close()
So how do I know if an item is a file or if it is a directory?


from stat import S_ISDIR
def isdir(path):
try:
return S_ISDIR(sftp.stat(path).st_mode)
except IOError:
#Path does not exist, so by definition not a directory
return False
...assuming sftp is an open Paramiko SFTP connection.


An old question, but a solution I came up with that works quite well, it's a little bit sloppy (typecasting and slashes and all) - but it does work.
Note this uses fabric.api.local to make the directories in the destination.
def sftp_get_recursive(path, dest, sftp=sftp):
item_list = sftp.listdir(path)
dest = str(dest)
if not os.path.isdir(dest):
local("mkdir %s" % dest)
for item in item_list:
item = str(item)
if is_directory(path + "/" + item, sftp):
sftp_get_recursive(path + "/" + item, dest + "/" + item, sftp)
else:
sftp.get(path + "/" + item, dest + "/" + item)


Paramiko does not support recursive operations.
But it's easy to implement:
import os
from stat import S_ISDIR, S_ISREG
def get_r_portable(sftp, remotedir, localdir):
for entry in sftp.listdir_attr(remotedir):
remotepath = remotedir + "/" + entry.filename
localpath = os.path.join(localdir, entry.filename)
mode = entry.st_mode
if S_ISDIR(mode):
try:
os.mkdir(localpath)
except OSError:
pass
get_r_portable(sftp, remotepath, localpath)
elif S_ISREG(mode):
sftp.get(remotepath, localpath)
You also can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
pysftp.Connection.put_r()
pysftp.Connection.get_r()
Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.
But pysftp seems to be an abandoned project, so you better stick with Paramiko.


A small update to Dan LaManna's answer that works in 2021.
import paramiko
import os
from stat import S_ISDIR, S_ISREG
def sftp_get_recursive(path, dest, sftp):
item_list = sftp.listdir_attr(path)
dest = str(dest)
if not os.path.isdir(dest):
os.makedirs(dest, exist_ok=True)
for item in item_list:
mode = item.st_mode
if S_ISDIR(mode):
sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)
else:
sftp.get(path + "/" + item.filename, dest + "/" + item.filename)
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp_get_recursive(remote_path, local_path, sftp)
sftp.close()


You can use the stat() method of your sftp object:
http://www.lag.net/paramiko/docs/paramiko.SFTPClient-class.html


stat() method among other attributes returns permissions. d (for example drwxrwxrwx) shows that it is directory.
As example:
dir = oct(sftp.stat(path).st_mode)
print dir[0:2]
output interpritation:
01 fifo
02 character special
04 directory
06 block special
10 regular file
12 symbolic link
14 socket


Here is an answer for 2022.
pysftp is abandoned. paramiko does not implement recursivity for SFTP. So I made a library named sftputil (to make a parallel with ftputil) based on Paramiko.
sftputil implements functionalities such as walk, glob and sync. To copy an entire directory, the easiest way is to use the synchronization feature :
from sftputil import SFTP
sftp = SFTP("hostname", "username", password="password")
# To copy recursively
sftp.sync_pull("remote/path", "local/dir")
# To copy only the files
sftp.sync_pull("remote/path", "local/dir", recursive=False)


If u using Linux or Unix-like. U can use 'file' utility with popen.
Or simple u can use os.path.isdir() =)

Related

Download new files only, ignoring file extensions, from remote SFTP server in Python

from datetime import datetime
import pysftp
import fnmatch
import os
from stat import S_IMODE, S_ISDIR, S_ISREG
Hostname = "Hostname"
Username = "Username"
Password = "Password"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=Hostname, username=Username, password=Password,
cnopts=cnopts) as sftp:
print("Connection successfully established ... ")
local_dir = r"D:\testing"
remote_dir = r'C:\Users\admin\Documents\personal'
for entry in sftp.listdir(remote_dir):
root, ext = os.path.splitext(entry)
for entry1 in sftp.listdir(local_dir):
root1, ext1 = os.path.splitext(entry1)
if root == root1:
if ext != ext1:
pass
elif root != root1:
remotepath = os.path.join(remote_dir, entry)
localpath = os.path.join(local_dir, entry)
sftp.get(remotepath, localpath, preserve_mtime=False)
I'm trying to export files from SFTP server to local. In order to do that I need to compare files by filename and file extension from server and local folder.
For instance abc.xlsx, abc.csv, adc.txt from server and
local folder has got abc.xlsx, then my script shouldn't copy any files with same name and same extension or different extension.
Here I need to compare the filenames and extension
if the same name then don't download
if same name and different extension don't download

You have to use os.listdir for the local path.
There's no need to re-read the local folder every time (unless you somehow aim to cater for multiple files with same base name in the remote directory).
Your code for identifying new files is not correct either.
You should not use os.path.join for SFTP paths.
This should do:
localfiles = os.listdir(local_dir)
for rentry in sftp.listdir(remote_dir):
rroot, rext = os.path.splitext(rentry)
found = False
for lentry in localfiles:
lroot, lext = os.path.splitext(lentry)
if rroot == lroot:
found = True
break
if not found:
remotepath = remote_dir + "/" rentry
localpath = os.path.join(local_dir, rentry)
sftp.get(remotepath, localpath, preserve_mtime=False)
Though these days, you should not use pysftp, as it is dead. Use Paramiko directly instead. See pysftp vs. Paramiko. The above code will work with Paramiko too with its SFTPClient.listdir.
Obligatory warning: Do not set cnopts.hostkeys = None, unless you do not care about security. For the correct solution see Verify host key with pysftp.

How to download folder by checking item inside it from remote server to local dir using Python?

In my root folder I have few folders.
For example:
root
AAA
a.txt
a.xml
BBB
b.txt
b.xml
CCC
c.xml
DDD
......
I need to download all folder which contain .txt file inside it except DDD.
For example AAA, BBB has .txt file so I need to download it except CCC(as CCC didn't contain .txt)
I used the below code which download all file except DDD but am not able to check whether it contains .txt or not
Code I tried:
from paramiko import SSHClient
import paramiko
from stat import S_ISDIR
ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('host', username='test', password='123')
remote_dir ='/root'
local_path = 'C:/download/'
def download_dir(remote_dir, local_dir):
import os
os.path.exists(local_dir) or os.makedirs(local_dir)
dir_items = sftp.listdir_attr(remote_dir)
for item in dir_items:
# assuming the local system is Windows and the remote system is Linux
# os.path.join won't help here, so construct remote_path manually
if(item.filename != "DDD"):
remote_path = remote_dir + '/' + item.filename
local_path = os.path.join(local_dir, item.filename)
if S_ISDIR(item.st_mode):
download_dir(remote_path, local_path)
else:
sftp.get(remote_path, local_path)
download_dir(remote_dir,local_path)
How can I check the files inside it and download the folder based on that?

You can use glob.glob(pathname, "*.txt") to get all txt files in a directory.
The documentation can be found here

Something like this:
import fnmatch
if S_ISDIR(item.st_mode):
if any(fnmatch.fnmatch(f, "*.txt") for f in sftp.listdir(remote_path)):
download_dir(remote_path, local_path)
else:
sftp.get(remote_path, local_path)
It's bit inefficient, as you will list all subfolders twice. But you can optimize it easily, if needed.
Based on List files on SFTP server matching wildcard in Python using Paramiko.

I can copy to remote computer but cannot copy from using Python

Any ideas on why I cannot copy from remote computer?
The first snippet works and I can copy to the 'servername'.
The second snippet gives me a 'No such file or directory.' error when I want to copy from 'servername' to my local computer.
UPDATE 2
This does not work either...
def copyfrom():
source_path = "\\computername\c$\test"
dest_path= "C:\localtest"
file_name = "testfile.txt"
shutil.copyfile(os.path.join(source_path, file_name), os.path.join(dest_path, file_name))
UPDATE I'm reading that you cannot copy from a remote computer using shutil. Anyone have any ideas on what my options are?
I have been copying to a list of computers using this...
import os
import shutil
import fileinput
import re
import sys # some of these use for other code in my program
source = os.listdir("C:/Users/jm/Desktop/PythonUpdate/")
destination = '//' + servername + r'/c$/test/'
for files in source:
if files.endswith("myname.config"):
try:
os.makedirs(destination, exist_ok=True)
shutil.copy(files,destination)
except:
copyerror()
os.system('cls' if os.name == 'nt' else 'clear')
array = []
with open("C:/Users/jm/Desktop/PythonUpdate/serverlist.txt", "r") as f:
for servername in f:
copyfiles(servername.strip())
Why is the opposite not working?
Here is what I am trying:
def copyfrom(servername):
# copy config from server
source = os.listdir('//' + servername + r'/c$/test') # directory where original configs are located
destination = 'C:/Users/jm/Desktop/PythonUpdate/' # destination server directory
for files in source:
if files.endswith("myname.config"):
try:
os.makedirs(destination, exist_ok=True)
shutil.copy(files,destination)
except:
copyerror()
readconfig(servername)
# begin here
os.system('cls' if os.name == 'nt' else 'clear')
array = []
with open("C:/Users/jm/Desktop/PythonUpdate/serverlist.txt", "r") as f:
for servername in f:
copyfrom(servername.strip())

You're not running the script from a directory called C:/Users/jm/Desktop/PythonUpdate/ by any chance?
os.listdir would be giving you the names of the entries in a directory at the specified path, not the full network name including the path.
https://docs.python.org/2/library/os.html
the modified example is using backslashes instead of forward slashes. if you want to do that, it's easier to use raw strings.
def copyfrom():
source_path = r"\\computername\c$\test"
dest_path= r"C:\localtest"
file_name = "testfile.txt"
shutil.copyfile(os.path.join(source_path, file_name), os.path.join(dest_path, file_name)
that said, there's really no reason to use backslashes in path names. unless you're running on a windows PC with no network share support, forward slash will be treated the same as backslash.

Specify file pattern in pysftp get

We can write a simple get like this:
import pysftp
hostname = "somehost"
user = "bob"
password = "123456"
filename = 'somefile.txt'
with pysftp.Connection(hostname, username=user, private_key='/home/private_key_file') as sftp:
sftp.get(filename)
However, I want to specify a pattern in the filename, something like: '*.txt'
Any idea on how to do this using pysftp ?

There's no function to download files matching a file mask in pysftp.
You have to:
list the directory, using Connection.listdir or Connection.walktree (if you need recursion)
iterate the list of files, filtering the files you want
call Connection.get individually for each.
For a trivial implementation, see:
List files on SFTP server matching wildcard in Python using Paramiko
It's about Paramiko, but the file matching part will be the same with pysftp:
import fnmatch
for filename in sftp.listdir('/remote/path'):
if fnmatch.fnmatch(filename, "*.txt"):
sftp.get("/remote/path/" + filename, "/local/path/" + filename)
Though your should really be using Paramiko anyway: pysftp vs. Paramiko
For a recursive example (you have to add the file matching), see:
Python pysftp get_r from Linux works fine on Linux but not on Windows.
See also how Connection.get_d or Connection.get_r are implemented.

Can confirm after going through the documentation that you can't list using a pattern. So i did something like this:
import pysftp
import re
server = pysftp.Connection(host=FTP_HOST,
username=FTP_USERNAME,
password=FTP_PASSWORD)
server.cwd(YOUR_FILES_PATH)
filelist = server.listdir()
for filename in filelist:
filedate = re.search(".*\.txt$", filename)
if filedate:
print "FOUND FILE " + filename

import pysftp
import sys
[...]
dn = datetime.now().strftime("%Y%m%d%H");
with pysftp.Connection(myHost, myUsername, password=myPassword) as sftp:
myFileList = sftp.listdir("files/")
for filename in myFileList:
if (filename.rfind("ArrivalList_" + dn) != -1):
sftp.get("files/" + filename, "/tmp/" + filename)

Recursive directory copy with Paramiko in Python

I am new to Python scripting. I need to copy few folders from my local machine (windows) to Linux server. As of now, I am copying the folders by opening WinSCP console. I need to automate this process. I have written a below code in Python using Paramiko module library.
import paramiko
import os
transport = paramiko.Transport(('10.10.10.10', 22))
transport.connect(username='weblogic', password='weblogic')
sftp = paramiko.SFTPClient.from_transport(transport)
filepath = '/apps/logs'
localpath = 'C:\\Users\\Public\\test'
sftp.put(localpath,filepath)
Above is not working properly and giving below error. Can you please help me to copy the folder present in the windows path C:\Users\Public\test to Linux server path /apps/logs?
Traceback (most recent call last):
File "C:\Users\Desktop\python\execute_script.py", line 28, in <module>
sftp.put(localpath,filepath)
File "C:\Python27\lib\paramiko\sftp_client.py", line 548, in put
fl = file(localpath, 'rb')
IOError: [Errno 13] Permission denied: 'C:\\Users\\Public\\test'

Please check the below code from the link https://gist.github.com/johnfink8/2190472. I have used put_all method in the snippet.
import paramiko
import socket
import os
from stat import S_ISDIR
class SSHSession(object):
# Usage:
# Detects DSA or RSA from key_file, either as a string filename or a
# file object. Password auth is possible, but I will judge you for
# using it. So:
# ssh=SSHSession('targetserver.com','root',key_file=open('mykey.pem','r'))
# ssh=SSHSession('targetserver.com','root',key_file='/home/me/mykey.pem')
# ssh=SSHSession('targetserver.com','root','mypassword')
# ssh.put('filename','/remote/file/destination/path')
# ssh.put_all('/path/to/local/source/dir','/path/to/remote/destination')
# ssh.get_all('/path/to/remote/source/dir','/path/to/local/destination')
# ssh.command('echo "Command to execute"')
def __init__(self,hostname,username='root',key_file=None,password=None):
#
# Accepts a file-like object (anything with a readlines() function)
# in either dss_key or rsa_key with a private key. Since I don't
# ever intend to leave a server open to a password auth.
#
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((hostname,22))
self.t = paramiko.Transport(self.sock)
self.t.start_client()
keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
key = self.t.get_remote_server_key()
# supposed to check for key in keys, but I don't much care right now to find the right notation
if key_file is not None:
if isinstance(key,str):
key_file=open(key,'r')
key_head=key_file.readline()
key_file.seek(0)
if 'DSA' in key_head:
keytype=paramiko.DSSKey
elif 'RSA' in key_head:
keytype=paramiko.RSAKey
else:
raise Exception("Can't identify key type")
pkey=keytype.from_private_key(key_file)
self.t.auth_publickey(username, pkey)
else:
if password is not None:
self.t.auth_password(username,password,fallback=False)
else: raise Exception('Must supply either key_file or password')
self.sftp=paramiko.SFTPClient.from_transport(self.t)
def command(self,cmd):
# Breaks the command by lines, sends and receives
# each line and its output separately
#
# Returns the server response text as a string
chan = self.t.open_session()
chan.get_pty()
chan.invoke_shell()
chan.settimeout(20.0)
ret=''
try:
ret+=chan.recv(1024)
except:
chan.send('\n')
ret+=chan.recv(1024)
for line in cmd.split('\n'):
chan.send(line.strip() + '\n')
ret+=chan.recv(1024)
return ret
def put(self,localfile,remotefile):
# Copy localfile to remotefile, overwriting or creating as needed.
self.sftp.put(localfile,remotefile)
def put_all(self,localpath,remotepath):
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file),os.path.join(remotepath,walker[0],file))
def get(self,remotefile,localfile):
# Copy remotefile to localfile, overwriting or creating as needed.
self.sftp.get(remotefile,localfile)
def sftp_walk(self,remotepath):
# Kindof a stripped down version of os.walk, implemented for
# sftp. Tried running it flat without the yields, but it really
# chokes on big directories.
path=remotepath
files=[]
folders=[]
for f in self.sftp.listdir_attr(remotepath):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print (path,folders,files)
yield path,folders,files
for folder in folders:
new_path=os.path.join(remotepath,folder)
for x in self.sftp_walk(new_path):
yield x
def get_all(self,remotepath,localpath):
# recursively download a full directory
# Harder than it sounded at first, since paramiko won't walk
#
# For the record, something like this would gennerally be faster:
# ssh user#host 'tar -cz /source/folder' | tar -xz
self.sftp.chdir(os.path.split(remotepath)[0])
parent=os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in self.sftp_walk(parent):
try:
os.mkdir(os.path.join(localpath,walker[0]))
except:
pass
for file in walker[2]:
self.get(os.path.join(walker[0],file),os.path.join(localpath,walker[0],file))
def write_command(self,text,remotefile):
# Writes text to remotefile, and makes remotefile executable.
# This is perhaps a bit niche, but I was thinking I needed it.
# For the record, I was incorrect.
self.sftp.open(remotefile,'w').write(text)
self.sftp.chmod(remotefile,755)

In addition to the answer #user1041177, but here a way to do it when you are on windows to linux host (not really sure which kind of host actually).
I don't know why, but if I keep backslash onto remote path, I get a FileNotFoundException. The only way to work was to replace all '\' by '/'
Maybe someone could tell me the proper way to avoid this situation at all ?
Here a part of the exact same code above to give you breadcrumbs if you encounter the same issue :
def sftp_walk(socket, remotepath):
remotepath = remotepath.replace('\\', '/')
path = remotepath
files = []
folders = []
for f in socket.listdir_attr(remotepath.replace('\\', '/')):
if S_ISDIR(f.st_mode):
folders.append(f.filename)
else:
files.append(f.filename)
print(path, folders, files)
yield path, folders, files
for folder in folders:
new_path = os.path.join(remotepath.replace('\\', '/'), folder)
for x in sftp_walk(socket, new_path):
yield x
def get_all(socket, remotepath, localpath):
remotepath = remotepath.replace('\\', '/')
socket.chdir(os.path.split(remotepath)[0])
parent = os.path.split(remotepath)[1]
try:
os.mkdir(localpath)
except:
pass
for walker in sftp_walk(socket, parent):
try:
os.mkdir(os.path.join(localpath, walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
socket.get(os.path.join(walker[0], file).replace('\\', '/'), os.path.join(localpath, walker[0], file).replace('\\', '/'))
BTW, I am not using those function inside an object, that's why their is 'socket' instead of 'self' because I call those function by passing the SFTP socket to them.
Finally have to say thank you to #user1041177, working like a charm.

I was trying to copy from a windows box to a linux box and got the same error as #Apex above. I was using the put_all method and I had to do some "replace" on parts of the code.
def put_all(self,localpath,remotepath):
remotepath = remotepath.replace('\\', '/')
# recursively upload a full directory
os.chdir(os.path.split(localpath)[0])
parent=os.path.split(localpath)[1]
for walker in os.walk(parent):
try:
self.sftp.mkdir(os.path.join(remotepath,walker[0]).replace('\\', '/'))
except:
pass
for file in walker[2]:
self.put(os.path.join(walker[0],file).replace('\\', '/'),os.path.join(remotepath,walker[0],file).replace('\\', '/'))

I found a few shortcomings with the above methods - first, the putter/getter doesn't function in the way you'd expect - if you want to put /foo/bar into /some/folder, you can't as it won't let you put files from a source folder to a different destination folder - the only thing you can do is put /foo/bar into /some/bar. In addition, you have to specify the source as /foo/bar and the destination as /some to end up with /some/bar - I find this confusing as it's not how most operating/ftp systems handle putting/getting/copying/etc. So, I improved on the answers listed above:
If you're going from Windows to Linux:
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), posixpath.join(dest, ''.join(root.rsplit(source))[1:].replace('\\', '/'), file))
source = '%USERPROFILE%\\Downloads\\'
dest = '/foo/bar'
put_dir(source, dest)
If you're just doing Windows then swap out posixpath.join with os.path.join and remove .replace('\\', '/'):
def put_dir(source, dest):
source = os.path.expandvars(source).rstrip('\\').rstrip('/')
dest = os.path.expandvars(dest).rstrip('\\').rstrip('/')
for root, dirs, files in os.walk(source):
for dir in dirs:
try:
sftp.mkdir(os.path.join(dest, ''.join(root.rsplit(source))[1:], dir))
except:
pass
for file in files:
sftp.put(os.path.join(root, file), os.path.join(dest, ''.join(root.rsplit(source))[1:], file))
source = '%USERPROFILE%\\Downloads\\'
dest = 'foo\\bar'
put_dir(source, dest)
The reason for the try statement is that sftp.mkdir errors out if the folder already exists.

Paramiko does not support recursive operations.
You can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
pysftp.Connection.put_r()
pysftp.Connection.get_r()
Or you can just base your code on pysftp source code. Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.

Categories