We can write a simple get like this:
import pysftp
hostname = "somehost"
user = "bob"
password = "123456"
filename = 'somefile.txt'
with pysftp.Connection(hostname, username=user, private_key='/home/private_key_file') as sftp:
sftp.get(filename)
However, I want to specify a pattern in the filename, something like: '*.txt'
Any idea on how to do this using pysftp ?
There's no function to download files matching a file mask in pysftp.
You have to:
list the directory, using Connection.listdir or Connection.walktree (if you need recursion)
iterate the list of files, filtering the files you want
call Connection.get individually for each.
For a trivial implementation, see:
List files on SFTP server matching wildcard in Python using Paramiko
It's about Paramiko, but the file matching part will be the same with pysftp:
import fnmatch
for filename in sftp.listdir('/remote/path'):
if fnmatch.fnmatch(filename, "*.txt"):
sftp.get("/remote/path/" + filename, "/local/path/" + filename)
Though your should really be using Paramiko anyway: pysftp vs. Paramiko
For a recursive example (you have to add the file matching), see:
Python pysftp get_r from Linux works fine on Linux but not on Windows.
See also how Connection.get_d or Connection.get_r are implemented.
Can confirm after going through the documentation that you can't list using a pattern. So i did something like this:
import pysftp
import re
server = pysftp.Connection(host=FTP_HOST,
username=FTP_USERNAME,
password=FTP_PASSWORD)
server.cwd(YOUR_FILES_PATH)
filelist = server.listdir()
for filename in filelist:
filedate = re.search(".*\.txt$", filename)
if filedate:
print "FOUND FILE " + filename
import pysftp
import sys
[...]
dn = datetime.now().strftime("%Y%m%d%H");
with pysftp.Connection(myHost, myUsername, password=myPassword) as sftp:
myFileList = sftp.listdir("files/")
for filename in myFileList:
if (filename.rfind("ArrivalList_" + dn) != -1):
sftp.get("files/" + filename, "/tmp/" + filename)
Related
I want to download a directory with unknown contents recursively via SSH and have been trying Paramiko. I have seen several examples how to upload directories but none that covers recursive download.
I can list all items in a directory but haven't been able to find a way of knowing if the item is a file (to download) or a directory (to call recursively).
transport = paramiko.Transport((MY_IP, 22))
transport.connect(username=MY_NAME, password=MY_PASS)
sftp = paramiko.SFTPClient.from_transport(transport)
file_list = sftp.listdir(path='/home/MY_HOME_DIR')
for item in file_list:
# Here is an item name... but is it a file or directory?
print(item)
sftp.close()
transport.close()
So how do I know if an item is a file or if it is a directory?
from stat import S_ISDIR
def isdir(path):
try:
return S_ISDIR(sftp.stat(path).st_mode)
except IOError:
#Path does not exist, so by definition not a directory
return False
...assuming sftp is an open Paramiko SFTP connection.
An old question, but a solution I came up with that works quite well, it's a little bit sloppy (typecasting and slashes and all) - but it does work.
Note this uses fabric.api.local to make the directories in the destination.
def sftp_get_recursive(path, dest, sftp=sftp):
item_list = sftp.listdir(path)
dest = str(dest)
if not os.path.isdir(dest):
local("mkdir %s" % dest)
for item in item_list:
item = str(item)
if is_directory(path + "/" + item, sftp):
sftp_get_recursive(path + "/" + item, dest + "/" + item, sftp)
else:
sftp.get(path + "/" + item, dest + "/" + item)
Paramiko does not support recursive operations.
But it's easy to implement:
import os
from stat import S_ISDIR, S_ISREG
def get_r_portable(sftp, remotedir, localdir):
for entry in sftp.listdir_attr(remotedir):
remotepath = remotedir + "/" + entry.filename
localpath = os.path.join(localdir, entry.filename)
mode = entry.st_mode
if S_ISDIR(mode):
try:
os.mkdir(localpath)
except OSError:
pass
get_r_portable(sftp, remotepath, localpath)
elif S_ISREG(mode):
sftp.get(remotepath, localpath)
You also can use pysftp. It's a wrapper around Paramiko that has more Python-ish look and feel and supports recursive operations. See
pysftp.Connection.put_r()
pysftp.Connection.get_r()
Or see my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows.
But pysftp seems to be an abandoned project, so you better stick with Paramiko.
A small update to Dan LaManna's answer that works in 2021.
import paramiko
import os
from stat import S_ISDIR, S_ISREG
def sftp_get_recursive(path, dest, sftp):
item_list = sftp.listdir_attr(path)
dest = str(dest)
if not os.path.isdir(dest):
os.makedirs(dest, exist_ok=True)
for item in item_list:
mode = item.st_mode
if S_ISDIR(mode):
sftp_get_recursive(path + "/" + item.filename, dest + "/" + item.filename, sftp)
else:
sftp.get(path + "/" + item.filename, dest + "/" + item.filename)
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp_get_recursive(remote_path, local_path, sftp)
sftp.close()
You can use the stat() method of your sftp object:
http://www.lag.net/paramiko/docs/paramiko.SFTPClient-class.html
stat() method among other attributes returns permissions. d (for example drwxrwxrwx) shows that it is directory.
As example:
dir = oct(sftp.stat(path).st_mode)
print dir[0:2]
output interpritation:
01 fifo
02 character special
04 directory
06 block special
10 regular file
12 symbolic link
14 socket
Here is an answer for 2022.
pysftp is abandoned. paramiko does not implement recursivity for SFTP. So I made a library named sftputil (to make a parallel with ftputil) based on Paramiko.
sftputil implements functionalities such as walk, glob and sync. To copy an entire directory, the easiest way is to use the synchronization feature :
from sftputil import SFTP
sftp = SFTP("hostname", "username", password="password")
# To copy recursively
sftp.sync_pull("remote/path", "local/dir")
# To copy only the files
sftp.sync_pull("remote/path", "local/dir", recursive=False)
If u using Linux or Unix-like. U can use 'file' utility with popen.
Or simple u can use os.path.isdir() =)
from datetime import datetime
import pysftp
import fnmatch
import os
from stat import S_IMODE, S_ISDIR, S_ISREG
Hostname = "Hostname"
Username = "Username"
Password = "Password"
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with pysftp.Connection(host=Hostname, username=Username, password=Password,
cnopts=cnopts) as sftp:
print("Connection successfully established ... ")
local_dir = r"D:\testing"
remote_dir = r'C:\Users\admin\Documents\personal'
for entry in sftp.listdir(remote_dir):
root, ext = os.path.splitext(entry)
for entry1 in sftp.listdir(local_dir):
root1, ext1 = os.path.splitext(entry1)
if root == root1:
if ext != ext1:
pass
elif root != root1:
remotepath = os.path.join(remote_dir, entry)
localpath = os.path.join(local_dir, entry)
sftp.get(remotepath, localpath, preserve_mtime=False)
I'm trying to export files from SFTP server to local. In order to do that I need to compare files by filename and file extension from server and local folder.
For instance abc.xlsx, abc.csv, adc.txt from server and
local folder has got abc.xlsx, then my script shouldn't copy any files with same name and same extension or different extension.
Here I need to compare the filenames and extension
if the same name then don't download
if same name and different extension don't download
You have to use os.listdir for the local path.
There's no need to re-read the local folder every time (unless you somehow aim to cater for multiple files with same base name in the remote directory).
Your code for identifying new files is not correct either.
You should not use os.path.join for SFTP paths.
This should do:
localfiles = os.listdir(local_dir)
for rentry in sftp.listdir(remote_dir):
rroot, rext = os.path.splitext(rentry)
found = False
for lentry in localfiles:
lroot, lext = os.path.splitext(lentry)
if rroot == lroot:
found = True
break
if not found:
remotepath = remote_dir + "/" rentry
localpath = os.path.join(local_dir, rentry)
sftp.get(remotepath, localpath, preserve_mtime=False)
Though these days, you should not use pysftp, as it is dead. Use Paramiko directly instead. See pysftp vs. Paramiko. The above code will work with Paramiko too with its SFTPClient.listdir.
Obligatory warning: Do not set cnopts.hostkeys = None, unless you do not care about security. For the correct solution see Verify host key with pysftp.
In my root folder I have few folders.
For example:
root
AAA
a.txt
a.xml
BBB
b.txt
b.xml
CCC
c.xml
DDD
......
I need to download all folder which contain .txt file inside it except DDD.
For example AAA, BBB has .txt file so I need to download it except CCC(as CCC didn't contain .txt)
I used the below code which download all file except DDD but am not able to check whether it contains .txt or not
Code I tried:
from paramiko import SSHClient
import paramiko
from stat import S_ISDIR
ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('host', username='test', password='123')
remote_dir ='/root'
local_path = 'C:/download/'
def download_dir(remote_dir, local_dir):
import os
os.path.exists(local_dir) or os.makedirs(local_dir)
dir_items = sftp.listdir_attr(remote_dir)
for item in dir_items:
# assuming the local system is Windows and the remote system is Linux
# os.path.join won't help here, so construct remote_path manually
if(item.filename != "DDD"):
remote_path = remote_dir + '/' + item.filename
local_path = os.path.join(local_dir, item.filename)
if S_ISDIR(item.st_mode):
download_dir(remote_path, local_path)
else:
sftp.get(remote_path, local_path)
download_dir(remote_dir,local_path)
How can I check the files inside it and download the folder based on that?
You can use glob.glob(pathname, "*.txt") to get all txt files in a directory.
The documentation can be found here
Something like this:
import fnmatch
if S_ISDIR(item.st_mode):
if any(fnmatch.fnmatch(f, "*.txt") for f in sftp.listdir(remote_path)):
download_dir(remote_path, local_path)
else:
sftp.get(remote_path, local_path)
It's bit inefficient, as you will list all subfolders twice. But you can optimize it easily, if needed.
Based on List files on SFTP server matching wildcard in Python using Paramiko.
I created a python script to copy files from a source folder to a destination folder, the script runs fine in my local machine.
However, when I tried to change the source to a path located in a server installed in a DMZ and the destination to a folder in a local servers I got the following error:
FileNotFoundError: [WinError 3] The system cannot find the path specified: '\reports'
And Here is the script:
import sys, os, shutil
import glob
import os.path, time
fob= open(r"C:\Log.txt","a")
dir_src = r"\reports"
dir_dst = r"C:\Dest"
dir_bkp = r"C:\Bkp"
for w in list(set(os.listdir(dir_src)) - set(os.listdir(dir_bkp))):
if w.endswith('.nessus'):
pathname = os.path.join(dir_src, w)
Date_File="%s" %time.ctime(os.path.getmtime(pathname))
print (Date_File)
if os.path.isfile(pathname):
shutil.copy2(pathname, dir_dst)
shutil.copy2(pathname, dir_bkp)
fob.write("File Name: %s" % os.path.basename(pathname))
fob.write(" Last modified Date: %s" % time.ctime(os.path.getmtime(pathname)))
fob.write(" Copied On: %s" % time.strftime("%c"))
fob.write("\n")
fob.close()
os.system("PAUSE")
Okay, we first need to see what kind of remote folder you have.
If your remote folder is shared windows network folder, try mapping it as a network drive: http://windows.microsoft.com/en-us/windows/create-shortcut-map-network-drive#1TC=windows-7
Then you can just use something like Z:\reports to access your files.
If your remote folder is actually a unix server, you could use paramiko to access it and copy files from it:
import paramiko, sys, os, posixpath, re
def copyFilesFromServer(server, user, password, remotedir, localdir, filenameRegex = '*', autoTrust=True):
# Setup ssh connection for checking directory
sshClient = paramiko.SSHClient()
if autoTrust:
sshClient.set_missing_host_key_policy(paramiko.AutoAddPolicy()) #No trust issues! (yes this could potentially be abused by someone malicious with access to the internal network)
sshClient.connect(server,user,password)
# Setup sftp connection for copying files
t = paramiko.Transport((server, 22))
t.connect(user, password)
sftpClient = paramiko.SFTPClient.from_transport(t)
fileList = executeCommand(sshclient,'cd {0}; ls | grep {1}'.format(remotedir, filenameRegex)).split('\n')
#TODO: filter out empties!
for filename in fileList:
try:
sftpClient.get(posixpath.join(remotedir, filename), os.path.join(localdir, filename), callback=None) #callback for showing number of bytes transferred so far
except IOError as e:
print 'Failed to download file <{0}> from <{1}> to <{2}>'.format(filename, remotedir, localdir)
If your remote folder is something served with the webdav protocol, I'm just as interested in an answer as you are.
If your remote folder is something else still, please explain. I have not yet found a solution that treats all equally, but I'm very interested in one.
I am working on server 1. I need to write a Python script where I need to connect to a server 2 and get certain files (files whose name begins with the letters 'HM') from a directory and put them into another directory, which needs to be created at the run time (because for each run of the program, a new directory has to be created and the files must be dumped in there), on server 1.
I need to do this in Python and I'm relatively new to this language. I have no idea where to start with the code. Is there a solution that doesn't involve 'tarring' the files? I have looked through Paramiko but that just transfers one file at a time to my knowledge. I have even looked at glob but I cannot figure out how to use it.
to transfer the files you might wanna check out paramiko
import os
import paramiko
localpath = '~/pathNameForToday/'
os.system('mkdir ' + localpath)
ssh = paramiko.SSHClient()
ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
ssh.connect(server, username=username, password=password)
sftp = ssh.open_sftp()
sftp.get(remotepath, localpath)
sftp.close()
ssh.close()
I you wanna use glob you can do this:
import os
import re
import glob
filesiwant = re.compile('^HM.+') #if your files follow a more specific pattern and you don't know regular expressions you can give me a sample name and i'll give you the regex4it
path = '/server2/filedir/'
for infile in glob.glob( os.path.join(path, '*') ):
if filesiwant.match(infile):
print "current file is: " + infile
otherwise an easier alternative is to use os.listdir()
import os
for infile in os.listdir('/server2/filedir/'):
...`
does that answer your question? if not leave comments
Python wouldn't be my first choice for this task, but you can use calls to the system and run mkdir and rsync. In particular you could do
import os
os.system("mkdir DIRECTORY")
os.system("rsync -cav user#server2:/path/to/files/HM* DIRECTORY/")
Just use ssh and tar. No need to get Python involved
$ ssh server2 tar cf - HM* | tar xf -
The remote tar can pipe straight into the local tar
You could use fabric. Create fabfile.py on server1:
import os
from fabric.api import get, hosts
#hosts('server2')
def download(localdir):
os.makedirs(localdir) # create dir or raise an error if it already exists
return get('/remote/dir/HM*', localdir) # download HM files to localdir
And run: fab download:/to/dir from the same directory in a shell (fabfile.py is to fab as Makefile is to make).