I have a simple script that successfully downloads a 75MB file over FTP:
try:
ftp = ftplib.FTP(host)
ftp.login(username, password)
ftp.cwd(source_dir)
except ftplib.all_errors as e:
print('Ftp error = ', e)
return False
# Check filename exists
if filename in ftp.nlst():
local_filename = os.path.join(dest_dir, filename)
lf = open(local_filename, "wb")
ftp.retrbinary("RETR " + filename, lf.write)
lf.close()
print(filename, ' successfully downloaded')
else:
print(filename, ' not found in the path ', source_dir)
ftp.quit()
This script works fine on both my home and work laptops when run from Spyder IDE or a Windows scheduled task.
I have deployed the exact same script to a Windows Virtual Machine on Azure.
Files less than 10MB seem to download ok.
Files larger than 30MB return an exception:
421 Data timeout. Reconnect. Sorry.
I get around 700 Mbps on Azure and only around 8Mbps on my home network.
It looks like a timeout. I can see the file is partially downloaded.
I tried setting ftp.set_pasv(False), but this then returns me 500 Illegal Port, which is to be expected. I understand passive is the preferred approach anyhow.
What else can I do to troubleshoot and resolve this issue?
Just some suggestions for you.
According to the wiki page for File Transfer Protocol, FTP may run in active or passive mode, as the figure below. In active mode, the client requires a listening port for incoming data from the server. However, due to the listening port of client for FTP server is random assigned, you can not prepare in advance to add the port in NSG inbound rules. So you should use passive mode in the client side on Azure VM with FTP.set_pasv(True) or without FTP.set_pasv(False).
For the issue 421 Data timeout. Reconnect. Sorry., please check the timeout setting in your FTP server, such as the data_connection_timeout property of vsftpd.conf file of vftp, to set enough long value of time out
Try to set a timeout value longer then the global default setting for ftplib.FTP(host='', user='', passwd='', acct='', timeout=None, source_address=None) function.
Try to use function FTP.set_debuglevel(level) to debug output more details for your script to find out the possible reason.
Hope it helps.
Related
Actually we are using python3.6.8 in our server, we are trying to connect the ftp server and pushing files to the ftp server through an api call, here when we try to push the files from local it is running fine and files are being pushed but when calling api to the server it is redirecting to a 502 bad gateway error after 14.8s time when tried with postman. the server we use is AWS EC2
ftp = ftplib.FTP()
host = config.FTP_HOST
port = 21
ftp.connect(host, port)
try:
ftp.login(config.FTP_USERNAME, config.FTP_PASSWORD)
file = open(path_image, 'rb')
ftp.cwd("/DailyDump/target/")
ftp.storbinary("STOR sample_file_name" + str(yesterday_date) + ".csv", file)
file.close()
ftp.close()
except:
pass
This problem was caused due to the maximum timeout reached on the API call, hence I transferred the codes from API to stand alone code to make it run for longer time period without giving error. so there is no error in ftp logging.
I'm downloading two files from a C-More industrial HMI FTP server. I don't know what OS the HMI is running but I suspect that its FTP server has some quirks. Using Jython 2.7, one file can be read without difficulty but the other has a space in the file name and the normal wrap-in-quotes solution doesn't work - yet.
The following works in the Windows 10 FTP client.
ftp> get NO_SPACES.csv
200 PORT command successful.
150 Opening ASCII mode data connection for NO_SPACES.csv.
226 Transfer complete.
ftp: 12774 bytes received in 0.27Seconds 47.66Kbytes/sec.
ftp> get "WITH SPACE.csv"
200 PORT command successful.
150 Opening ASCII mode data connection for WITH SPACE.csv.
226 Transfer complete.
ftp: 6328 bytes received in 0.02Seconds 316.40Kbytes/sec.
So far, so good. Now try it in Python:
ftp = FTP(myIP) # Connect.
ftp.login(userName, password) # Login.
ftp.set_pasv(False) # Required by the C-More panel for some reason.
with io.BytesIO() as binary_buffer:
# read all of products into a binary buffer
# ftp.retrbinary("RETR NO_SPACES.csv", binary_buffer.write) # This line works.
ftp.retrbinary('RETR "WITH SPACE.csv"', binary_buffer.write) # This one doesn't.
The script console in my development system reports:
ftplib.error_perm: 550 "WITH SPACE.csv": Requested action not taken.
Filenames have been changed to protect the innocent.
Windows FTP likes the get command. Python seems to favour RETR.
I've tried 'RETR "WITH SPACE.csv"' and "RETR 'WITH SPACE.csv'". Same result.
If I have to I can rename the files in the HMI but that will require some validation and paperwork and that's no fun.
I'm developing this on the latest version of Inductive Automation's Ignition! SCADA system which uses Jython 2.7.
Has anyone got any ideas for me to try?
The ftplib has no issue with spaces. The problem are the quotes you add to the RETR command. There should be no quotes:
ftp.retrbinary('RETR WITH SPACE.csv', binary_buffer.write)
If you enable the debug mode in ftp using the -d switch, you will see that it also sends no quotes to the FTP server in the RETR command:
ftp> get "WITH SPACE.csv"
---> PORT 127,0,0,1,15,145
200 Port command successful
---> RETR WITH SPACE.csv
150 Opening data channel for file download from server of "/WITH SPACE.csv"
226 Successfully transferred "/WITH SPACE.csv"
ftp: 12 bytes received in 0.00Seconds 12000.00Kbytes/sec.
Note that the get is a commandline ftp client user command that translates to the FTP protocol RETR command.
I'm trying to run a simple ftps script to upload a file on a scheduled basis from a Linux box to a ftps instance running on Windows Server 2012. When I try and test the script on my desktop (OS x), the script errors out:
Error uploading file: [Errno 54] Connection reset by peer
If I run the script on a linux box, same error, except 104 instead of 54:
Error uploading file: [Errno 104] Connection reset by peer
The files that I'm uploading have either been empty or are 8 bytes. I've verified that ftps is working with 2 other clients on my desktop. What am I missing / overlooking?
#!/usr/bin/env python
from ftplib import FTP_TLS
import fnmatch
import os
import ssl
import sys
server = '192.168.1.2'
user = 'myUsername'
passwd = 'myPassword'
def connect_ftp():
ftp = FTP_TLS(server, user, passwd)
ftp.set_pasv(True)
ftp.prot_p()
return ftp
def upload_file(ftp_connection, upload_file_path):
try:
upload_file = open("/tmp/test/" + upload_file_path, 'r')
print('Uploading ' + upload_file_path + "...")
ftp_connection.storbinary('STOR ' + upload_file_path, upload_file)
ftp_connection.quit()
ftp_connection.close()
upload_file.close()
print('Upload finished.')
except Exception, e:
print("Error uploading file: " + str(e))
ftp_conn = connect_ftp()
for file in os.listdir('/tmp/test'):
if fnmatch.fnmatch(file, 'bt_*.txt'):
upload_file(ftp_conn, file)
I think this problem only shows on MS FTP server.
First of all turn on debug
ftp.set_debuglevel(2)
In my case transfer hangs on
put 'STOR test.xml\r\n'
get '125 Data connection already open; Transfer starting.\n'
resp '125 Data connection already open; Transfer starting.'
Then I found this suggession http://www.sami-lehtinen.net/blog/python-32-ms-ftps-ssl-tls-lockup-fix
I've tried (commented conn.unwrap() in storbinary) it and it worked!
In my case in was line 513
# shutdown ssl layer
if _SSLSocket is not None and isinstance(conn, _SSLSocket):
pass #conn.unwrap()
This is obliviously very bad hack, but I couldn't find anything better.
I had the same issue and succeeded to solve it with this lines :
ftps = FTP_TLS(server)
ftps.set_debuglevel(2) # To show logs
ftps.ssl_version = ssl.PROTOCOL_TLS
ftps.set_pasv(True)
ftps.login(user="user", passwd="passwd")
And I switched from Python 3.5.1 to Python 3.8.3.
I have this 30 virtual machines and I am doing everything manually right now which is a kind of trouble for me. I am trying now to write a script so I can connect to them all automatically and do my desired task. I made a flow diagram of what I need to do. Can anybody just give me some hints on how will i achieve this task programatically. I am attaching the flow diagram.
Thanks in Advance.
Please Right Click On Image and Click View Image to See the Flow Diagram.
Reading a text file and getting the data is trivial:
with open('host.txt', 'r') as inf:
lines = inf.readlines()
hostlist = [ln.split() for ln in lines]
Now hostlist should be a list of lists;
[['192.168.0.23', 'root', 'secret'], ['192.168.0.24', 'root', 'secret2'] ...
But your list shouldn't have to contain more than the hostnames. IP adresses can be gotten from DNS that you have to configure anyway, and ssh can login without passwords if configured correctly.
Putting the passwords for all your virtual hosts in a plain text file has security concerns. If you want to go that route, make sure to restrict access to that file!
You can use subprocess to execute commands. I would suggest using rsync to push the desired files to the virtual machines. That minimizes network traffic. And you can deploy directly from filesystem to filesystem without having to roll a tarball. It can be as simple as
status = subprocess.check_output(['rsync', '-av', localdir, remotedir])
Where localdir is the directory where the files for the virtual machine in question are stored (it should end with a '/'), and 'remotedir' is the hostname::directory on the virtual machine where the data should land (this should not end with a '/').
For executing commands remotely, ssh is the way to go. Configure that for passwordless login, using ssh's authorized_keys file on each remote host. Then you don't need to put passwords in your list of hosts.
Fabric is best solution for you. Fabric based on paramiko ( which based on libssh2), it makes very easy to work with commands on remote host and provides function to upload and download fiels from remote host.
Here it is http://docs.fabfile.org/en/1.5/
Docs about put function here
I don't get your question in the flow diagram, however you can use paramiko as suggested and I have a large number of background utilities written on top of paramiko which enables support people to monitor remote web servers on the browser. Snippet below,
client = paramiko.SSHClient()
client.set_missing_host_key_policy( paramiko.AutoAddPolicy() )
client.load_system_host_keys()
client.connect( '192.168.100.1', port=4001, username='monitor', password='XXXX' )
cmds = [ "sed -i 's/\/bin\/date -u/\/bin\/date/g' /etc/cron.hourly/reboot" ]
for cmd in cmds:
if __DEBUG_MODE__:
print 'Executing..... ' + cmd
stdin, stdout, stderr = client.exec_command( cmd )
Also if you want to push files below is the snippet,
def setupSFTPClient(self, ip_add):
print 'Setting SFTP client: ' + ip_add
tb = 'Finished SFTP-ing. '
try:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.load_system_host_keys()
client.connect(ip_add, port=4001, username='monitor', password='XXXX')
sftp_client = client.open_sftp()
# NB you need the filename for remote path otherwise
# paramiko barfs with IOError: Failure
sftp_client.put( '/home/projects/portal_release.tgz', '/var/ND/portal_release.tgz' )
sftp_client.put( '/home/projects/portal_installer.sh', '/var/ND/portal_installer.sh' )
sftp_client.close()
except Exception, e:
print e
tb = traceback.format_exc()
finally:
print tb
client.close()
I'm on Windows 7.
I cannot connect to my iPad with a simple Python script:
HOST = '192.168.1.122'
try:
f = ftplib.FTP(HOST)
except (socket.error, socket.gaierror), e:
MessageBox.Show('ERROR: cannot reach "%s"' % HOST)
return
try:
f.connect(HOST,2121)
f.login()
except ftplib.error_perm:
MessageBox.Show('ERROR: cannot login anonymously')
f.quit()
return
The errors I have is "getaddrinfo returns an empty list" and the "cannot reach..." message... Cannot solve it...
I tried to FTP with several programs on the iPad without success. If I FTP via DOS box or using a FTP software it works. I tried as well another FTP server on my PC and it works.
I am forced to use port 2121, so can't change it.
Any clue or experience?
You should read docs before anything:
class ftplib.FTP([host[, user[,
passwd[, acct[, timeout]]]]]) Return a
new instance of the FTP class. When
host is given, the method call
connect(host) is made. When user is
given, additionally the method call
login(user, passwd, acct) is made
(where passwd and acct default to the
empty string when not given). The
optional timeout parameter specifies a
timeout in seconds for blocking
operations like the connection attempt
(if is not specified, the global
default timeout setting will be used).
So, if you do f = ftplib.FTP(HOST) it fails because it will try to connect to standard port (21) and not 2121.
You should get an instance of ftplib and later use f.connect(HOST, 2121).
http://docs.python.org/library/ftplib.html