I have a piece of code in Python to download files from an ftp. The code downloads the very first file in the list of available days but fails to download the second. What could be the problem?
import os, ftplib
destdir='D:\precipitation\dl'
ftp = ftplib.FTP('ftp.itc.nl')
ftp.login('anonymous', '')
ftp.cwd('pub/mpe/msg')
available_days=['summsgmpe_20100101.zip','summsgmpe_20100102.zip', 'summsgmpe_20100103.zip', 'summsgmpe_20100104.zip', 'summsgmpe_20100105.zip', 'summsgmpe_20100106.zip', 'summsgmpe_20100107.zip', 'summsgmpe_20100108.zip']
hdfs = list()
for day in available_days :
file = available_days[available_days.index(day)]
print 'file=', file
local_file = os.path.join(destdir, file)
ftp.retrbinary('RETR %s' %file, open(local_file, 'wb').write)
hdfs.append(os.path.abspath(local_file))
ftp.cwd('..')
ftp.quit()
Remove your call to ftp.cwd(..)
That's moving up a directory for each iteration of the list, instead of staying in the correct folder where the files are.
Related
Using "ftplib" Python package, I connect to a host, post which I send a binary file using "storbinary: function to this directory. However, the file transfer does not happen, the source file is over-written and made 0 bytes
Tried using "cwd" within ftplib to change to destination directory before uploading file.
Tried 'rsync' on the command line which works fine (thus establishing file integrity).
print ('Establishing FTP connection')
ftp = FTP(dest_dir, 'user', 'password')
print '+++++', ftp.pwd()
ftp.cwd(dest_dir)
print 'work dir now', ftp.pwd()
ftp.retrlines('LIST')
f_name = /home/test/file_to_upload
with open(f_name, 'rb') as cfile:
ftp.storbinary('STOR %s' % f_name, cfile)
ftp.quit()
The file: /home/test/file_to_upload should be uploaded to dest_dir, instead it is getting over-written and also becomes an empty file, i.e. file contents are erased
f_name contains directory of your local system. Try to extract the filename only. Also be careful adding a space between ‘STOR’ and ‘filename’
Example:
ftp.storbinary('STOR '+ os.path.basename(f_name), open(f_name, 'rb'))
I want to download files from an FTP server and archive them locally in a (zip) archive.
It is well known how to download files and save them individually:
import ftplib
remote = ftplib.FTP(ftp_server)
remote.login(username, password)
for filename in file_list:
remote.retrbinary("RETR " + filename, open(filename, 'wb').write)
remote.quit()
It is also well known how to add files to an archive:
import zipfile
archive = zipfile.ZipFile(archive_file)
archive.write(filename)
archive.close()
But it seems not possible to use both at the same time:
remote.retrbinary("RETR " + filename, archive.write(filename))
This leads to a FileNotFoundError, because filename has not been saved to a local (temporary) directory in between.
Is there a way to directly send the file stream from FTP into a zip archive? Or would it be more efficient to download all files straight, add them to the archive, and then delete the files? I would like to keep harddisk I/O as low as possible.
Download the file to memory and use ZipFile.writestr:
import ftplib
import zipfile
from io import BytesIO
# ...
archive = zipfile.ZipFile(archive_file, "w")
for filename in file_list:
flo = BytesIO()
ftp.retrbinary('RETR ' + filename, flo.write)
archive.writestr(filename, flo.getvalue())
archive.close()
I'm trying to upload all my files to a time_name directory (such as "170905_161330") in my ftp server.
# -*- coding:utf-8 -*-
import ftplib
import os
import datetime
CWD = os.getcwd()
NOW_STR = str(datetime.datetime.now())
LOCAL_STR = "HDD1/AutoBackUp/" + NOW_STR[2:4]+NOW_STR[5:7]+NOW_STR[8:10]+"_"+NOW_STR[11:13]+NOW_STR[14:16]+NOW_STR[17:19]+"/"
FTP_ADDRESS = 'FTP_ADDRESS'
FTP_ID = '1'
FTP_PW = '1'
ftp = ftplib.FTP(FTP_ADDRESS, FTP_ID, FTP_PW)
for i in os.listdir(CWD):
FILENAME = str(i)
print(FILENAME)
ftp.mkd(LOCAL_STR)
LOCAL_NAME = LOCAL_STR + FILENAME
print(str(LOCAL_NAME))
with open(FILENAME, 'rb') as fileOBJ:
ftp.storlines('STOR ' + str(LOCAL_NAME), fileOBJ)
ftp.quit()
But the error
ftplib.error_perm: 550 HDD1/AutoBackUp/170905_160635/: Directory not empty
continues to appear, while the first file is uploaded correctly. After that, it doesn't work.
I can check my first file in ftp server, but yeah. second file doesn't exist.
I guess... storlines function only works when upload folder is empty.
How can I solve this problem?
From a very rapid read of your code, I suspect that the problem is in ftp.mkd. You already created the directory at the first iteration of the for loop.
To test this error on your local system, open the terminal:
write a mkdir test command
write a mkdir test again
You'll see an error: File Exist. I think the directory not empty is gnerated from this error in the server.
Modify your code to put ftp.mkd before the for loop:
ftp.mkd(LOCAL_STR)
for i in os.listdir(CWD):
FILENAME = str(i)
print(FILENAME)
LOCAL_NAME = LOCAL_STR + FILENAME
print(str(LOCAL_NAME))
with open(FILENAME, 'rb') as fileOBJ:
ftp.storlines('STOR ' + str(LOCAL_NAME), fileOBJ)
ftp.quit()
and test it again. Please remember to remove the directory from the server before testing it.
I need some help in order to create a script to download multiple .csv files from FTP every 24 hours, ignoring the old files and to continue downloading the new ones to keep an update. I'm having trouble writing the pattern because the name of the files vary from 01150728.csv, 01150904.csv to 02xxxxxx.csv, 03xxxxx.csv and currently it reached 30151007.csv.
The script that I'm currently using downloads all the files but I need a command line in order to do what I described earlier.
from ftplib import FTP
import sys
import ftplib
import os
import fnmatch
os.chdir(r'______________') # Directory where the files need to be downloaded
ftp=ftplib.FTP('xxxxxxxx', 'xxxxx', 'xxxxxx') # ftp host info
ftp.cwd('______')
filematch='*csv'
for filename in ftp.nlst(filematch):
fhandle=open(filename, 'wb')
print 'Getting ' + filename
ftp.retrbinary('RETR '+ filename, fhandle.write)
fhandle.close()
ftp.quit()
You should keep a list or set of the files already fetched. The following assumes you run the code once and don't exit.
from ftplib import FTP
import sys
import ftplib
import os
import fnmatch
os.chdir(r'______________') # Directory where the files need to be downloaded
ftp=ftplib.FTP('xxxxxxxx', 'xxxxx', 'xxxxxx') # ftp host info
ftp.cwd('______')
filematch='*csv'
import time
downloaded = []
while True: # runs forever
skipped = 0
for filename in ftp.nlst(filematch):
if filename not in downloaded:
fhandle=open(filename, 'wb')
print 'Getting ' + filename
ftp.retrbinary('RETR '+ filename, fhandle.write)
fhandle.close()
downloaded.append(filename)
else:
skipped += 1
print 'Downloaded %s, skipped %d files' % (downloaded[-1], skipped)
time.sleep(24*60*60) # sleep 24 hours after finishing last download
ftp.quit()
If you run the script each day, omit the while loop and use pickle or simply write the list/set in a file, and load it at the start of the script.
I've been trying to download files from an FTP server. For this I've found this Python-FTP download all files in directory and examined it. Anyways, I extracted the code I needed and it shows as follows:
import os
from ftplib import FTP
ftp = FTP("ftp.example.com", "exampleUsername", "examplePWD")
file_names = ftp.nlst("\public_html")
print file_names
for filename in file_names:
if os.path.splitext(filename)[1] != "":
local_filename = os.path.join(os.getcwd(), "Download", filename)
local_file = open(filename, 'wb')
ftp.retrbinary('RETR ' + filename, local_file.write)
local_file.close()
ftp.close()
But when it tries to open the file, it keeps saying:
ftplib.error_perm: 550 Can't open CHANGELOG.php: No such file or directory
I've tried w+, a+, rw, etc. and I keep getting the same error all the time. Any ideas?
Note: I am using OSX Mavericks and Python 2.7.5.
This question may have been asked several times and believe me I researched and found some of them and none of them worked for me.
open() in Python does not create a file if it doesn't exist
ftplib file select
It looks like you are listing files in a directory and then getting files based on the returned strings. Does nlst() return full paths or just filenames? If its just filenames than retrbinary might be expecting "/Foo/file" but getting "file", and there might not be anything named file in the root dir of the server.