We Keep Coding

Socket is closed error in python script (paramiko)

I am new to python and have written this script through references online. This script is basically connecting to a sftp server outside the network and lists a directory in that server to collect .xml files (oldest to newest) and transfers file 1 by 1 in a loop.
After few mins, the script ends the session when it reaches connection timeout or no files to transfer. Sleeps for 5 seconds and connects again to get the next set of files, like near real-time.
This script was working fine for several months before we started having
'Socket is closed error' like every 10 -15 mins. The script would run normally as expected and starts transferring files, then all of sudden will hang for 2-3 mins and eventually through the below error.
Sometimes, the moment the script connects to sftp server and starts transferring files, after few files, will send up with the same error again
Error:
return self._send(s, m) File "C:\ProgramData\Anaconda3\lib\site-packages\paramiko\channel.py", line 1198, in _send
raise socket.error("Socket is closed")OSError: Socket is closed
import os
import shutil
import paramiko
from time import sleep
from datetime import datetime
import fnmatch
from lxml import etree
import lxml
localpath=r"D:/Imported_Files/"
logpath=r"D:/Imported_Files/log/"
temppath=r"D:/Imported_Files/file_rename_temp/"
while True:
#########
try: #### providing server credentials and connection to the sftp server with username and private key
host = "Hostname"
port = 22
transport = paramiko.Transport((host, port))
username = "username"
mykey = paramiko.RSAKey.from_private_key_file("C:/<PATH>",password='#########')
transport.connect(username = username, pkey = mykey)
sftp = paramiko.SFTPClient.from_transport(transport)
except Exception as e:
print(str(e))
sleep(30)
continue
try:
sftp.listdir()
sftp.chdir("outbox")
sftp.listdir("")
file_list=[x.filename for x in sorted(sftp.listdir_attr(),key= lambda f: f.st_mtime)] ## listing directory and get oldest files first in the list to process
file_list
except Exception as e: #### continue if there is an exception
print(str(e))
sleep(30)
continue
dttxt=str(datetime.now().strftime('%Y%m%d'))
for file in file_list: #### getting only files with .xml extension
if fnmatch.fnmatch(file,"*.xml"):
tempfilepath=temppath+file
localfilepath=localpath+file
file_info=sftp.stat(file)
file_mod_time=datetime.fromtimestamp(file_info.st_mtime) ### assigning modified timestamp of file to variable
file_acc_time=datetime.fromtimestamp(file_info.st_atime) ### assigning create timestamp of file to variable
try:
sftp.get(file,tempfilepath) ### performing sftp of the selected file from the list
except:
file_error_log = open(logpath+"files_not_processed"+dttxt+".log", "a") #### writing info to log
file_error_log.write("Failed:"+file+"\n")
file_error_log.close()
print("getting file "+file+" failed!")
continue
try:
sftp.remove(file) #### removing the file from sftp server after successful transfer
except:
print("deleteing file "+file+" failed")
os.remove(tempfilepath)
print("exception, moving on to next file")
file_error_ftp_remove = open(logpath+"files_not_deleted_ftp"+dttxt+".log", "a")
file_error_ftp_remove.write("Failed:"+file+"\n")
file_error_ftp_remove.close()
continue
try:
root = lxml.etree.parse(tempfilepath) #### parsing the file to extract a tag from .xml
system_load_id=root.find('system_load_id')
if system_load_id.text==None:
system_load_id_text=""
else:
system_load_id_text=system_load_id.text
new_filename=localpath+os.path.splitext(os.path.basename(tempfilepath))[0]+"-"+system_load_id_text+os.path.splitext(os.path.basename(localfilepath))[1]
sleep(0.3)
os.rename(tempfilepath, new_filename)
except:
sleep(0.3)
os.rename(tempfilepath, localpath+file)
print('Cant parse xml, hence moving the file as it is')
pass
########### file moved to final location after parsing .xml. writing to log and processing next file in the list
file_processed_log = open(logpath+"files_processed"+ str(datetime.now().strftime('%Y%m%d'))+".log", "a")
file_processed_log.write(str(datetime.now().strftime('%Y%m%d %H:%M:%S'))+" : "+file+","+str(file_mod_time)+","+str(file_acc_time)+"\n")
file_processed_log.close()
print(datetime.now())
sleep(5) ######## 1 session complete , sleeping and connecting to server again to get the next set of files
The issue is not consistent. Sometimes, we have this error like 5 times in a day and some days 100+ times in a day
I have researched online and not sure where the issue is since the script ran fine for several months and processed 1000s of files per day in near real-time without any issues

try except not running except part

I have a short code that ftps a small file into a server.
session = ftplib.FTP("192.168.0.164", "admin", "admin")
file = open("path/test.txt", "rb")
try:
session.storbinary("STOR application/test.txt", file)
except:
print("failed")
else:
print("success!")
file.close()
In the above piece I changed the IP address to so it would fail (there is no 192.168.0.164 device) and it doesn't print failed like it's supposed to. In the terminal I get a
"connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond" error.
if I mistype the path, I also don't get a failed print on the terminal.
If I type in the correct IP the success part does work.
Am I using the try/except wrong?
UPDATE:
Current code looks like this:
file = open("path/test.txt", "rb")
try:
session = ftplib.FTP("192.168.0.161", "admin", "admin")
session.storbinary("STOR application/test.txt", file)
except:
print("Unable to reach host")
else:
print("success!")
session.quit()
finally:
print ("DONE!!")
file.close()
I figure the ftplib.all_errors will catch all errors (host unreachable, and file not found). It seems to catch the unable to reach host errors, but no file not found errors.

Am I using the try/except wrong?
Your syntax is correct, but Python is not actually reaching the try block at all.
When you call session = ftplib.FTP(host, ...) where host is unreachable, the code will stop in its tracks there. That's because FTP.__init__() greedily calls self.connect(). This will in turn call socket.create_connection(), which will not succeed for an unreachable host.
So, you'd need to modify to:
with open("path/test.txt", "rb") as file:
try:
session = ftplib.FTP("192.168.0.164", "admin", "admin")
session.storbinary("STOR application/test.txt", file)
except Exception as e:
print("failed")
print(e)
else:
print("success!")
finally:
session.quit()

New to python need to send data over already connected TCP port without waiting for request from client

7 socket listener setup. It works great and keeps the connection open, non blocking, all that. From time to time a file will show up that I need to hand back to the client. That works to, but it only send the data in the file if the client sends a character first. I need to have it send the data when the file shows up and not wait. I am coming from php and know what I am doing there. Python is new to me so there are some nuances I don't understand about this code.
while True:
try:
#I want this bit here to fire without waiting for the client to send anything
#right now it works except the client has to send a character first
#check for stuff to send back
for fname in os.listdir('data/%s/in' % dirname):
print(fname)
f = open('data/%s/in/%s' % (dirname, fname), "r")
client.send(f.readline())
data = client.recv(size)
if data:
bucket=bucket+data
else:
raise error('Client disconnected')
except Exception as e:
client.close()
print(e)
return False

Python: File download using ftplib hangs forever after file is successfully downloaded

I have been trying to troubleshoot an issue where in when we are downloading a file from ftp/ftps. File gets downloaded successfully but no operation is performed post file download completion. No error has occurred which could give more information about the issue.
I tried searching for this on stackoverflow and found this link which talks about similar problem statement and looks like I am facing similar issue, though I am not sure. Need little more help in resolving the issue.
I tried setting the FTP connection timeout to 60mins but of less help.
Prior to this I was using retrbinary() of the ftplib but same issue occurs there. I tried passing different blocksize and windowsize but with that also issue was reproducible.
I am trying to download the file of size ~3GB from AWS EMR cluster. Sample code is written below.
def download_ftp(self, ip, port, user_name, password, file_name, target_path):
try:
os.chdir(target_path)
ftp = FTP(host=ip)
ftp.connect(port=int(port), timeout=3000)
ftp.login(user=user_name, passwd=password)
if ftp.nlst(file_name) != []:
dir = os.path.split(file_name)
ftp.cwd(dir[0])
for filename in ftp.nlst(file_name):
sock = ftp.transfercmd('RETR ' + filename)
def background():
fhandle = open(filename, 'wb')
while True:
block = sock.recv(1024 * 1024)
if not block:
break
fhandle.write(block)
sock.close()
t = threading.Thread(target=background)
t.start()
while t.is_alive():
t.join(60)
ftp.voidcmd('NOOP')
logger.info("File " + filename + " fetched successfully")
return True
else:
logger.error("File " + file_name + " is not present in FTP")
except Exception, e:
logger.error(e)
raise
Another option suggested in the above mentioned link is to close the connection post downloading small chunk of the file and then restart the connection. Can someone suggest how can this be achieved, not sure how to resume the download from the same point where the file download was stopped last time before closing the connection. Will this method be full proof of downloading the entire file.
I don't know much about FTP server level timeout settings so didn't know what and how it needs to be altered. I basically want to write a generic FTP down-loader which can help in downloading the files from FTP/FTPS.
When I use retrbinary() method of ftplib and set debug level to 2.
ftp.set_debuglevel(2)
ftp.retrbinary('RETR ' + filename, fhandle.write)
Below logs are getting printed.
cmd 'TYPE I'
put 'TYPE I\r\n'
get '200 Type set to I.\r\n'
resp '200 Type set to I.'
cmd 'PASV'
put 'PASV\r\n'
get '227 Entering Passive Mode (64,27,160,28,133,251).\r\n'
resp '227 Entering Passive Mode (64,27,160,28,133,251).'
cmd 'RETR FFFT_BRA_PM_R_201711.txt'
put 'RETR FFFT_BRA_PM_R_201711.txt\r\n'
get '150 Opening BINARY mode data connection for FFFT_BRA_PM_R_201711.txt.\r\n'
resp '150 Opening BINARY mode data connection for FFFT_BRA_PM_R_201711.txt.'

Before doing anything, note that there is something very wrong with your connection, and diagnosing that and getting it fixed is far better than working around it. But sometimes, you just have to deal with a broken server, and even sending keepalives doesn't help. So, what can you do?
The trick is to download a chunk at a time, then abort the download—or, if the server can't handle aborting, close and reopen the connection.
Note that I'm testing everything below with ftp://speedtest.tele2.net/5MB.zip, which hopefully this doesn't cause a million people to start hammering their servers. Of course you'll want to test it with your actual server.
Testing for REST
The entire solution of course relies on the server being able to resume transfers, which not all servers can do—especially when you're dealing with something badly broken. So we'll need to test for that. Note that this test will be very slow, and very heavy on the server, so do not testing with your 3GB file; find something much smaller. Also, if you can put something readable there, it will help for debugging, because you may be stuck comparing files in a hex editor.
def downit():
with open('5MB.zip', 'wb') as f:
while True:
ftp = FTP(host='speedtest.tele2.net', user='anonymous', passwd='test#example.com')
pos = f.tell()
print(pos)
ftp.sendcmd('TYPE I')
sock = ftp.transfercmd('RETR 5MB.zip', rest=pos)
buf = sock.recv(1024 * 1024)
if not buf:
return
f.write(buf)
You will probably not get 1MB at a time, but instead something under 8KB. Let's assume you're seeing 1448, then 2896, 4344, etc.
If you get an exception from the REST, the server does not handle resuming—give up, you're hosed.
If the file goes on past the actual file size, hit ^C, and check it in a hex editor.
If you see the same 1448 bytes or whatever (the amount you saw it printing out) over and over again, again, you're hosed.
If you have the right data, but with extra bytes between each chunk of 1448 bytes, that's actually fixable. If you run into this and can't figure out how to fix it by using f.seek, I can explain—but you probably won't run into it.
Testing for ABRT
One thing we can do is try to abort the download and not reconnect.
def downit():
with open('5MB.zip', 'wb') as f:
ftp = FTP(host='speedtest.tele2.net', user='anonymous', passwd='test#example.com')
while True:
pos = f.tell()
print(pos)
ftp.sendcmd('TYPE I')
sock = ftp.transfercmd('RETR 5MB.zip', rest=pos)
buf = sock.recv(1024 * 1024)
if not buf:
return
f.write(buf)
sock.close()
ftp.abort()
You're going to want to try multiple variations:
No sock.close.
No ftp.abort.
With sock.close after ftp.abort.
With ftp.abort after sock.close.
All four of the above repeated with TYPE I moved to before the loop instead of each time.
Some will raise exceptions. Others will just appear to hang forever. If that's true for all 8 of them, we need to give up on aborting. But if any of them works, great!
Downloading a full chunk
The other way to speed things up is to download 1MB (or more) at a time before aborting or reconnecting. Just replace this code:
buf = sock.recv(1024 * 1024)
if buf:
f.write(buf)
with this:
chunklen = 1024 * 1024
while chunklen:
print(' ', f.tell())
buf = sock.recv(chunklen)
if not buf:
break
f.write(buf)
chunklen -= len(buf)
Now, instead of reading 1442 or 8192 bytes for each transfer, you're reading up to 1MB for each transfer. Try pushing it farther.
Combining with keepalives
If, say, your downloads were failing at 10MB, and the keepalive code in your question got things up to 512MB, but it just wasn't enough for 3GB—you can combine the two. Use keepalives to read 512MB at a time, then abort or reconnect and read the next 512MB, until you're done.

how to abort and retry ftp download after specified time?

I have a Python script which connects to a remote FTP server and downloads a file. As the server I connect to is not very reliable, it often happens that the transfer stalls and transfer rates become extremely low. However, no error is raised, so that my script stalls as well.
I use the ftplib module, with the retrbinary function. I would like to be able to set a timeout value after which the download aborts, and then automatically retry/restart the transfer (resuming would be nice, but that's not strictly necessary, as the files are only ~300M).

I managed what I need to do using the threading module:
conn = FTP(hostname, timeout=60.)
conn.set_pasv(True)
conn.login()
while True:
localfile = open(local_filename, "wb")
try:
dlthread = threading.Thread(target=conn.retrbinary,
args=("RETR {0}".format(remote_filename), localfile.write))
dlthread.start()
dlthread.join(timeout=60.)
if not dlthread.is_alive():
break
del dlthread
print("download didn't complete within {timeout}s. "
"waiting for 10s ...".format(timeout=60))
time.sleep(10)
print("restarting thread")
except KeyboardInterrupt:
raise
except:
pass
localfile.close()

What about the timeout argument of the FTP class http://docs.python.org/2/library/ftplib.html#ftplib.FTP