We Keep Coding

download remote gz files that reside in a tree like directories does snot work

I have been scratching my head for more than 2 days, but still cannot figure out how to do the following!
I want to download all Geo data sets that are in ftp://ftp.ncbi.nlm.nih.gov and then in each data set, I need to see if they contain the keywords that I am interested in. I was able to manually download one of the data sets and checked the file for the desired keywords. However, since the number of data sets are huge, I cannot do it manually. I want to write a program to do it for me. For the first step, I just tried to see if I can download them.
The structure is as follows:
hots->
/geo/
-> datasets/
-> GDS1nnn/ .... all the way through GDS6nnn and each of them
contain more than 600 directories; ordered by number i.e.
GDS1001. Now, in each of these directories:
---> soft inside this folder there are 2 files that are named
like this: folder name (GDS1001)+_full.soft.gz
this is the file that I think I need to download and then see if the keywords that I am looking for are inside that file.
Here is my code:
ftp = FTP('ftp.ncbi.nlm.nih.gov') # remember that you ONLY need to provide the host name not the complete address!
ftp.login()
#ftp.retrlines('LIST')
ftp.cwd("/geo/datasets/GDS1nnn/")
ftp.retrlines('LIST')
filenames = ftp.nlst()
count = len(filenames)
curr = 0
print ("found {} files".format(count))
for filename in filenames:
first_path=filename+"/soft/"
second_path=first_path+filename+"_full.soft.gz"
#print(second_path)
local_filename = os.path.join(r'full path to a folder that I
created')
file = open(local_filename, 'wb')
ftp.retrbinary('RETR ' + second_path, file.write)
file.close()
ftp.quit()
Output:
file = open(local_filename, 'wb')
PermissionError: [Errno 13] Permission denied: full path to a folder that I created'
However, I have both read and write permission on this folder.
Thanks for your help

The following code shows how you can create a folder for each dataset and save their content into that folder.
import sys, ftplib, os, itertools
from ftplib import FTP
from zipfile import ZipFile
ftp = FTP('ftp.ncbi.nlm.nih.gov')
ftp.login()
#ftp.retrlines('LIST')
ftp.cwd("/geo/datasets/GDS1nnn/")
ftp.retrlines('LIST')
filenames = ftp.nlst()
curr = 0
#print ("found {} files".format(count))
count = 0
for filename in filenames:
array_db=[]
os.mkdir( os.path.join('folder called "output' + filename ) )
first_path=filename+"/soft/"
os.mkdir( os.path.join('folder called "output' + first_path ) )
second_path=first_path+filename+"_full.soft.gz"
array_db.append(second_path)
for array in array_db:
print(array)
local_filename = os.path.join('folder called "output' + array )
file = open(local_filename, 'wb')
ftp.retrbinary('RETR ' + array, file.write)
file.flush()
file.close()
ftp.quit()

"Permission denied" error from downloading all files from FTP folder

So far I have the gotten the names of the files I need from the FTP site. See code below.
from ftplib import FTP
import os, sys, os.path
def handleDownload(block):
file.write(block)
ddir='U:/Test Folder'
os.chdir(ddir)
ftp = FTP('sidads.colorado.edu')
ftp.login()
print ('Logging in.')
directory = '/pub/DATASETS/NOAA/G02158/unmasked/2012/04_Apr/'
print ('Changing to ' + directory)
ftp.cwd(directory)
ftp.retrlines('LIST')
print ('Accessing files')
filenames = ftp.nlst() # get filenames within the directory
print (filenames)
Where I am running into trouble is the download of the files into a folder. The code below is something I have tried however I receive the permission error due to the file not being created before I write to it.
for filename in filenames:
local_filename = os.path.join('C:/ArcGis/New folder', filename)
file = open(local_filename, 'wb')
ftp.retrbinary('RETR '+ filename, file.write)
file.close()
ftp.quit()
Here is the error and callback.

The directory listing includes the . reference to the folder (and probably also .. reference to the parent folder).
You have to skip it, you cannot download it (them).
for filename in filenames:
if (filename != '.') and (filename != '..'):
local_filename = os.path.join('C:/ArcGis/New folder', filename)
file = open(local_filename, 'wb')
ftp.retrbinary('RETR '+ filename, file.write)
file.close()
Actually you have to skip all folders in the listing.

Going through ftp directories in python

I'm trying to download several folders from an ftp server with Python 3 using ftplib.
I have a list of the names of the folders. They are all located in a folder 'root'. The problem is that I don't know how to navigate through them. When I use cwdI can go to a deeper directory, but how do I get up again?
I'm trying to get something like
list = ["folder1", "folder2", "folder3"]
for folder in list:
##navigate to folder
##do something

You can retrieve current directory using FTP.pwd method. Remember that directory before change directory.
parent_dir = ftp_object.pwd()
list = ["folder1", "folder2", "folder3"]
for folder in list:
ftp_object.cwd('{}/{}'.format(parent_dir, folder))
ftp_object.cwd(parent_dir) # go to parent directory

I made some changes to code I found here
You have to make the destination folder before running the code.
Also, the site I used did not require username or pass.
Please let me know if this works. I am wondering if I should "put this in my back pocket" and save it to my external hard drive.
#!/usr/bin/python
import sys
import ftplib
import urllib.request
import os
import time
import errno
server = "ftp.analog.com"
#user = ""
#password = ""
source = "/pub/MicroConverter/ADuCM36x/"
destination0 = "C:/NewFolder/" # YOU HAVE TO UT THIS NEW FOLDER IN C: BEFORE RUNNING
interval = 0.05
ftp = ftplib.FTP(server)
ftp.login()#(user, password)
count = 0 #We need this variable to make the first folder correctly
def downloadFiles(path, destination):
try:
ftp.cwd(path)
os.chdir(destination)
mkdir_p(destination[0:len(destination)-1] + path)
print ("Created: " + destination[0:len(destination)-1] + path )
except OSError:
pass
except ftplib.error_perm:
print ( "Error: could not change to " + path )
sys.exit("Ending Application")
filelist=ftp.nlst()
print(filelist)
for file in filelist:
time.sleep(interval)
if "." in file :
url = ("ftp://" + server + path + file)
urllib.request.urlretrieve(url, destination + path + file)
else:
try:
ftp.cwd(path + file + "/")
downloadFiles(path + file + "/", destination)
except ftplib.error_perm:
os.chdir(destination[0:len(destination)-1] + path)
try:
ftp.retrbinary("RETR " + file, open(os.path.join(destination + path, file),"wb").write)
print ("Downloaded: " + file)
except:
print ("Error: File could not be downloaded " + file)
return
def mkdir_p(path):
try:
os.makedirs(path)
except OSError as exc:
if exc.errno == errno.EEXIST and os.path.isdir(path):
pass
else:
raise
downloadFiles(source, destination0)
#ftp.quit()

Python shutil move I/O error

I am trying to find some files, create a folder and move the files in there.
def test():
try:
logfile = "C:\\Users\\alkis\\Desktop\\testouter\\test"
result_dir = os.path.join(logfile, "testzip")
print result_dir
os.makedirs(result_dir)
os.chmod(result_dir, stat.S_IWRITE)
kpath = logfile + "\\*.jpg"
print kpath
files = glob.glob(kpath)
for file in files:
filename = os.path.splitext(file)[0]
print filename
os.chmod(filename, stat.S_IWRITE)
shutil.move(filename, result_dir)
except Exception, e:
#shutil.rmtree(result_dir)
print e.__doc__ + "\r\n"
print e.message
return
The error I am getting is: MS-Windows OS call failed
I check the permissions on my files and they are not read only.

You are listing each file, removing the extension, then trying to move that filename.
The extension is part of the filename, don't remove it. Windows Exlorer hides the extension only when displaying files.
You also don't need to call os.chmod() on the filename; just skip that step:
for file in files:
filename = os.path.splitext(file)[0]
print filename
shutil.move(filename, result_dir)

Copy files in ftp server Python

I was able to copy files in the ftp server to a different location by writing the file to the webserver and then uploading it again.
Is there any way that I can write the file contents to memory without writing it to the harddisk and uploading them to the server. This is my code.
filepath = os.path.join(os.path.dirname(os.path.dirname(__file__)),'OpenTable','tempfiles')
ftp = ftplib.FTP('ftphost')
ftp.login('username','password')
filenames = []
ftp.retrlines('NLST', filenames.append)
for filename in filenames:
if filename[len(filename)-3:] == 'zip':
ftp.cwd("/")
filepath1 = os.path.join(filepath,filename)
print filepath1
r = open(filepath1,'w')
ftp.retrbinary('RETR ' + filename, r.write)
ftp.cwd("/BackUp")
r.close()
r = open(filepath1,'r')
ftp.storbinary('STOR ' + filename, r)
r.close()
os.remove(filepath1)
print 'Successfully Backed up ', filename
ftp.quit()
I tried using the StringIO. It doesn't seem to work.
Thanks.