I have been working all day trying to figure out how to use the python ftplib module to download folders, subfolders, and files from an ftp server but I could only come up with this.
from ftplib import FTP
import sys, ftplib
sys.tracebacklimit = 0 # Does not display traceback errors
sys.stderr = "/dev/null" # Does not display Attribute errors
Host = "ftp.debian.org"
Port = 21
Username = ""
Password = ""
def MainClass():
global ftp
global con
Host
Port
ftp = FTP()
con = ftp.connect(Host, Port) # Connects to the host with the specified port
def grabfile():
source = "/debian/"
filename = "README.html"
ftp.cwd(source)
localfile = open(filename, 'wb')
ftp.retrbinary('RETR ' + filename, localfile.write)
ftp.quit()
localfile.close()
try:
MainClass()
except Exception:
print "Not Connected"
print "Check the address", Host + ":" + str(Port)
else:
print "Connected"
if ftplib.error_perm and not Username == "" and Password == "":
print "Please check your credentials\n", Username, "\n", Password
credentials = ftp.login(Username, Password)
grabfile()
This python script will download a README.html file from ftp.debian.org but, I would like to be able to download whole folders with files and subfolders in them and I cannot seem to figure that out. I have searched around for different python scripts using this module but I cannot seem to find any that do what I want.
Any suggestions or help would be greatly appreciated.
Note:
I would still like to use python for this job but it could be a different module such as ftputil or any other one out there.
Thanks in advance,
Alex
The short solution:
You could possibly just run: "wget -r ftp://username:password#ftp.debian.org/debian/*" to get all the files under the debian directory.
Then you can process the files in python.
The long solution:
You can go over every directory listing by using ftplib, getting a directory listing parsing it and then getting every file and recursing into directories.
If you search the web you'd find previous posts on stackoverlow which solve this issue
Related
I am using paramiko to upload files to SFTP server. I want to transfer all files in folder. The names of files are in a manner one.txt, two.txt.... . I want the files to be sent sequentially like one.txt then two.txt then three.txt.....the below code for transfer of one file runs fine, but the last code which I have tried to transfer all files is not working....
import paramiko
source = r'/home/netcs/b/one.txt'
dest = r'/home/tein/c/pickle.txt'
hostname = '10.10.10.9'
port = 22 # default port for SSH
username = 'tein'
password = 'po'
try:
t = paramiko.Transport((hostname, port))
t.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(t)
sftp.put(source, dest)
finally:
t.close()
Transfer all files (not working):
import paramiko
source = r'/home/netcs/b/'
dest = r'/home/tein/c/'
hostname = '10.10.10.9'
port = 22 # default port for SSH
username = 'tein'
password = 'po'
for file in source:
if file.endswith('.txt'):
try:
t = paramiko.Transport((hostname, port))
t.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(t)
sftp.put(source, dest)
finally:
t.close()
break
else:
print('No txt file found')
I have files in b/:
But the script's output is:
no txt file found
no txt file found
no txt file found
Your code never reads the local directory. Your for loop iterates characters in the '/home/netcs/b/' string, not files in the /home/netcs/b/ folder.
For listing files in a folder, use os.listdir:
How do I list all files of a directory?
Also the os.listdir returns filenames only, so you have to combine them with the source when calling SFTPClient.put.
Similarly the remotepath argument of SFTPClient.put must also be a full file path.
You do not want to break after the first found file
Other issue is that the print('No txt file found') is misplaced. You will print that for every file that does not have .txt extension.
files = os.listdir(source)
for file in files:
if file.endswith('.txt'):
try:
t = paramiko.Transport((hostname, port))
t.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(t)
sftp.put(os.path.join(source, file), dest + "/" + file)
finally:
t.close()
For a recursive upload, see:
Python pysftp put_r does not work on Windows
I'm programming in a MacOS and using Python + Django. I must to get some files in our private network (Windows network) and move them to our server. There, Python/Django will read these files and save the data in a database. How can I do that?
What I have tried
source_path = "smb://server-name/GRUPOS/TECNOLOGIA_INFORMACAO/Dashboard Diretoria/"
dest_path = "./static/reports/". # This is my static folder where I want to move the file
file_name = "general_reports.csv"
shutil.copyfile(source_path + file_name, dest_path + file_name)
It gives the follow error:
[Errno 2] No such file or directory:
'smb://server-name/GRUPOS/TECNOLOGIA_INFORMACAO/Dashboard
Diretoria/general_reports.csv'
This path (source_path) I just copied and past from the Finder, so... I think that it's correct. I have already searched at StackOverflow and I have tried other methods like put "r" before the path... Nothing....
Technologies used
Python 3.6;
Django 3.0.5;
Mac OSX;
Windows Network.
Thank you for your help and patience.
You need to import an SMB client libary for python or you mount that drive before you work there
First of all, thank you #Van de Wack.
This is the complete solution:
Install the pysmb library (https://pypi.org/project/pysmb/):
pip install pysmb
Import the library to your code:
from smb.SMBConnection import SMBConnection
The follow code is a example to list all directories:
server_ip = "10.110.10.10" # Take your server IP - I have put a fake IP :)
server_name = 'myserver' # The servername for the IP above
share_name = "GRUPOS" # This is the principal folder of your network that you want's to connect
network_username = 'myuser' # This is your network username
network_password = '***' # This is your network password
machine_name = 'myuser#mac-mc70006405' # Your machine name
conn = SMBConnection(network_username, network_password, machine_name, server_name, use_ntlm_v2 = True)
assert conn.connect(server_ip, 139)
files = conn.listPath(share_name, "/TECNOLOGIA_INFORMACAO/Dashboard Diretoria/")
for item in files:
print(item.filename)
I created a python script to copy files from a source folder to a destination folder, the script runs fine in my local machine.
However, when I tried to change the source to a path located in a server installed in a DMZ and the destination to a folder in a local servers I got the following error:
FileNotFoundError: [WinError 3] The system cannot find the path specified: '\reports'
And Here is the script:
import sys, os, shutil
import glob
import os.path, time
fob= open(r"C:\Log.txt","a")
dir_src = r"\reports"
dir_dst = r"C:\Dest"
dir_bkp = r"C:\Bkp"
for w in list(set(os.listdir(dir_src)) - set(os.listdir(dir_bkp))):
if w.endswith('.nessus'):
pathname = os.path.join(dir_src, w)
Date_File="%s" %time.ctime(os.path.getmtime(pathname))
print (Date_File)
if os.path.isfile(pathname):
shutil.copy2(pathname, dir_dst)
shutil.copy2(pathname, dir_bkp)
fob.write("File Name: %s" % os.path.basename(pathname))
fob.write(" Last modified Date: %s" % time.ctime(os.path.getmtime(pathname)))
fob.write(" Copied On: %s" % time.strftime("%c"))
fob.write("\n")
fob.close()
os.system("PAUSE")
Okay, we first need to see what kind of remote folder you have.
If your remote folder is shared windows network folder, try mapping it as a network drive: http://windows.microsoft.com/en-us/windows/create-shortcut-map-network-drive#1TC=windows-7
Then you can just use something like Z:\reports to access your files.
If your remote folder is actually a unix server, you could use paramiko to access it and copy files from it:
import paramiko, sys, os, posixpath, re
def copyFilesFromServer(server, user, password, remotedir, localdir, filenameRegex = '*', autoTrust=True):
# Setup ssh connection for checking directory
sshClient = paramiko.SSHClient()
if autoTrust:
sshClient.set_missing_host_key_policy(paramiko.AutoAddPolicy()) #No trust issues! (yes this could potentially be abused by someone malicious with access to the internal network)
sshClient.connect(server,user,password)
# Setup sftp connection for copying files
t = paramiko.Transport((server, 22))
t.connect(user, password)
sftpClient = paramiko.SFTPClient.from_transport(t)
fileList = executeCommand(sshclient,'cd {0}; ls | grep {1}'.format(remotedir, filenameRegex)).split('\n')
#TODO: filter out empties!
for filename in fileList:
try:
sftpClient.get(posixpath.join(remotedir, filename), os.path.join(localdir, filename), callback=None) #callback for showing number of bytes transferred so far
except IOError as e:
print 'Failed to download file <{0}> from <{1}> to <{2}>'.format(filename, remotedir, localdir)
If your remote folder is something served with the webdav protocol, I'm just as interested in an answer as you are.
If your remote folder is something else still, please explain. I have not yet found a solution that treats all equally, but I'm very interested in one.
I am trying to list the folders in a ftp directory so that the output is as follows:
[32114, 32115, 32116, ..., 42123]
The following script accesses the ftp site. How can I access the folders in the ftp directory using the attached script as a start?
import arcpy, ftplib, os, socket
HOST = 'atlas.ca.gov'
DIRN = '/pub/naip/2012/doqqs_combined_color-nir/'
workspace = 'F:/download/location'
# Get current working directory
print 'The current working directory is %s' % os.getcwd()
# Change the current working directory to a download folder
os.chdir(workspace)
print 'The workspace has been changed to %s' % workspace
try:
f = ftplib.FTP(HOST)
except (socket.error, socket.gaierror), e:
print 'ERROR: cannot reach "%s"' % HOST
print '*** Connected to host "%s"' % HOST
try:
f.login()
except ftplib.error_perm:
print 'Error: cannot login anonymously'
f.quit()
print '*** Logged in as "anonymous"'
try:
f.cwd(DIRN)
except ftplib.error_perm:
print 'ERROR: cannot CD to "%s"' % DIRN
f.quit()
print '*** Changed to "%s" folder' % DIRN
If you look at the ftplib docs, there are two obvious functions for this: nlst and dir.
Either one will give you a list of all members in the directory, both files and subdirectories.
With nlst, you don't get any information beyond the name. But if you were planning to chdir or otherwise use each one of them, that's OK; you will get exceptions for the ones that turned out to be regular file, and can just skip over them.
With dir, you get a full directory listing, in human-readable form. Which you will have to capture (by passing a callback function) and then parse manually—which is a lot less fun, but the only way you can know in advance which members are files and which are directories.
If the server is a windows machine, the output of
ftp.dir() will look like 01-14-14 04:21PM <DIR> Output
If the server is of the *nix variety, the output will look like drwx------ 3 user group 512 Apr 22 2005 Mail
If you want to check if an item is a directory, use a regular expression on each line.
Windows FTP server will contain <DIR> if the item is a directory and *nix ftp servers will start with d.
Do you need help with Python regexes?
Building on the answer by abarnert, the following produces the list:
from ftplib import FTP
ftp = FTP('atlas.ca.gov')
ftp.login()
ftp.cwd('/pub/naip/2012/doqqs_combined_color-nir/')
print ftp.nlst()
I've a package with .gz and filename as (xxxxxxxxxxxx_PARSERxxxxx.tar.gz) extension placed in a remote directory say 1.1.1.1(Should be a input as a variable), i'm currently running the script from machine say 2.2.2.2, i need to place the package in machine 3.3.3.3 (Should be input as avariable), prepared a script for the same but its not working, pls help me out on the same,
Here is the code which i scripted so far(Not working):
#!/usr/bin/python
#ftp.py
import sys
import telnetlib
from ftplib import FTP
import os
def handleDownload(block):
file.write(block)
print ".",
hstip= raw_input('Enter the hstip: ') #ip of the machine where packages ae placed
uid = raw_input('Enter the username: ') #uid os the machine whr packages are placed
pwd = raw_input('Enter the user password: ') #pwd of hte machine where packages are palced
path = raw_input('Enter the path where packages are present: ') #path for the packages.
rmthstip= raw_input('Enter the hstip: ') #ip of the machine where packages to be placed
rmtuid = raw_input('Enter the username: ') #uid os the machine whr packages to be placed
rmtpwd = raw_input('Enter the user password: ') #pwd of hte machine where packages to be palced
cwd = os.getcwd()
os.chdir(cwd)
logout = 'parser files downloaded succesfully'
tn = telnetlib.Telnet(rmtip)
tn.read_until("login: ")
tn.write(rmtuid + "\n")
tn.read_until("Password:")
tn.write(rmtpwd + "\n")
ftp=FTP(hstip,uid,pwd)
print 'Logging in.'
ftp.login(uid,pwd)
ftp.cwd(path)
parserfile = 'find . -name "*PARSER*.gz"'
filenm = os.system(parserfile)
print filenm
ftp.retrbinary(' RETR ', 'filenm', logout )
ftp.quit()
tn.close()
It would really help if you told us how the program is not working.
By glancing quickly over it, I'd say you perserfile bit and filenm usage are wrong.
os.system returns an integer, so filenm is not a file name, but the exist status of the find command. You want to use pipe = subprocess.Popen(['find', '.', '-name', '*PARSER*.gz'], stdout=subprocess.PIPE and a for loop over pipe.stdout to read all the lines produced by the command
the ftp.retrbinary line looks suspicious: you probably want to remove the quotes around 'filenm' to use the path found above, and to use your callback function which needs fixing (see below)
in handleDownload, the call to file.write will fail because you are calling on the file class and not on a file object
Finally, it looks strange to me to perform a search on local file system to find a path on the remote ftp server which looks like what your code is doing, unless you have some specificities to your local path. Or are you trying to use the telnet connection to perform a remote search? IMO, if the FTP server allows you to do so and you know the remote path, you'd better set the CWD on the FTP server to that dir, retrieve the contents and look for a path that matches your requirements.
The downloading per se should be rewritten as:
local_fobj = file('downloaded_file.gz', 'wb')
def download_cb(data):
local_fobj.write(data)
ftp.retrbinary('RETR %s' % filenm, download_cb)
local_fobj.close()
Other misc stuff:
cwd = os.getwd() followed by os.chdir(cwd) can be safely removed
rmtip is not defined, should by rmthstip