We Keep Coding

Check if a socket is already opened - python

I have an azure timer function which runs every minute to trigger a socket which gets data from a website. I don't want to establish a connection everytime the timer runs the function. So, is there a way in Python which I can check if a socket is open for a particular website on particular port?
Or, is there a way to re-use a socket in time-triggered applications?
# Open socket
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(20) # 20 sec timeout
if is_socket_open(sock):
logging.info("Socket is already open")
else:
logging.info("No socket was open. Opening a new one...")
sock.connect(server_address)
sock.settimeout(None)
logging.info(f"Connected to {sock}")
return sock
except socket.gaierror as e:
logging.exception(f"Error connecting to remote server {e}")
time.sleep(20)
except socket.error as e:
logging.exception(f"Connection error {e}")
time.sleep(20)
except Exception as e:
logging.exception(f"An exception occurred: {e}")
time.sleep(20)
def is_socket_open(sock: socket.socket) -> bool:
try:
# this will try to read bytes without blocking and also without removing them from buffer (peek only)
data = sock.recv(16, socket.MSG_PEEK)
if len(data) == 0:
return True
except socket.timeout:
return False # socket is not connected yet, therefore receiving timed out
except BlockingIOError:
return False # socket is open and reading from it would block
except ConnectionResetError:
return True # socket was closed for some other reason
except Exception as e:
logging.exception(f"unexpected exception when checking if a socket is closed: {e}")
return False
return False
So this entire process runs every minute.

You can always use global variables to reuse objects in future invocations. The following example was copied from Google Cloud Platform documentation, but you can apply the same concept to your Azure Function:
# Global (instance-wide) scope
# This computation runs at instance cold-start
instance_var = heavy_computation()
def scope_demo(request):
# Per-function scope
# This computation runs every time this function is called
function_var = light_computation()
return 'Instance: {}; function: {}'.format(instance_var, function_var)
In your case, you can declare the sock as a global variable and reuse it in future warm start invocations. You should also increase the timeout to above 60 seconds, giving that you're triggering your azure function every minute.
However, keep in mind that there is no guarantee that the state of the function will be preserved for future invocations. For instance, in auto-scaling situations, a new socket would be open.
Microsoft Azure also says the following in regards to client connections:
To avoid holding more connections than necessary, reuse client instances rather than creating new ones with each function invocation. We recommend reusing client connections for any language that you might write your function in.
See also:
Manage connections in Azure Functions

Socket is closed error in python script (paramiko)

I am new to python and have written this script through references online. This script is basically connecting to a sftp server outside the network and lists a directory in that server to collect .xml files (oldest to newest) and transfers file 1 by 1 in a loop.
After few mins, the script ends the session when it reaches connection timeout or no files to transfer. Sleeps for 5 seconds and connects again to get the next set of files, like near real-time.
This script was working fine for several months before we started having
'Socket is closed error' like every 10 -15 mins. The script would run normally as expected and starts transferring files, then all of sudden will hang for 2-3 mins and eventually through the below error.
Sometimes, the moment the script connects to sftp server and starts transferring files, after few files, will send up with the same error again
Error:
return self._send(s, m) File "C:\ProgramData\Anaconda3\lib\site-packages\paramiko\channel.py", line 1198, in _send
raise socket.error("Socket is closed")OSError: Socket is closed
import os
import shutil
import paramiko
from time import sleep
from datetime import datetime
import fnmatch
from lxml import etree
import lxml
localpath=r"D:/Imported_Files/"
logpath=r"D:/Imported_Files/log/"
temppath=r"D:/Imported_Files/file_rename_temp/"
while True:
#########
try: #### providing server credentials and connection to the sftp server with username and private key
host = "Hostname"
port = 22
transport = paramiko.Transport((host, port))
username = "username"
mykey = paramiko.RSAKey.from_private_key_file("C:/<PATH>",password='#########')
transport.connect(username = username, pkey = mykey)
sftp = paramiko.SFTPClient.from_transport(transport)
except Exception as e:
print(str(e))
sleep(30)
continue
try:
sftp.listdir()
sftp.chdir("outbox")
sftp.listdir("")
file_list=[x.filename for x in sorted(sftp.listdir_attr(),key= lambda f: f.st_mtime)] ## listing directory and get oldest files first in the list to process
file_list
except Exception as e: #### continue if there is an exception
print(str(e))
sleep(30)
continue
dttxt=str(datetime.now().strftime('%Y%m%d'))
for file in file_list: #### getting only files with .xml extension
if fnmatch.fnmatch(file,"*.xml"):
tempfilepath=temppath+file
localfilepath=localpath+file
file_info=sftp.stat(file)
file_mod_time=datetime.fromtimestamp(file_info.st_mtime) ### assigning modified timestamp of file to variable
file_acc_time=datetime.fromtimestamp(file_info.st_atime) ### assigning create timestamp of file to variable
try:
sftp.get(file,tempfilepath) ### performing sftp of the selected file from the list
except:
file_error_log = open(logpath+"files_not_processed"+dttxt+".log", "a") #### writing info to log
file_error_log.write("Failed:"+file+"\n")
file_error_log.close()
print("getting file "+file+" failed!")
continue
try:
sftp.remove(file) #### removing the file from sftp server after successful transfer
except:
print("deleteing file "+file+" failed")
os.remove(tempfilepath)
print("exception, moving on to next file")
file_error_ftp_remove = open(logpath+"files_not_deleted_ftp"+dttxt+".log", "a")
file_error_ftp_remove.write("Failed:"+file+"\n")
file_error_ftp_remove.close()
continue
try:
root = lxml.etree.parse(tempfilepath) #### parsing the file to extract a tag from .xml
system_load_id=root.find('system_load_id')
if system_load_id.text==None:
system_load_id_text=""
else:
system_load_id_text=system_load_id.text
new_filename=localpath+os.path.splitext(os.path.basename(tempfilepath))[0]+"-"+system_load_id_text+os.path.splitext(os.path.basename(localfilepath))[1]
sleep(0.3)
os.rename(tempfilepath, new_filename)
except:
sleep(0.3)
os.rename(tempfilepath, localpath+file)
print('Cant parse xml, hence moving the file as it is')
pass
########### file moved to final location after parsing .xml. writing to log and processing next file in the list
file_processed_log = open(logpath+"files_processed"+ str(datetime.now().strftime('%Y%m%d'))+".log", "a")
file_processed_log.write(str(datetime.now().strftime('%Y%m%d %H:%M:%S'))+" : "+file+","+str(file_mod_time)+","+str(file_acc_time)+"\n")
file_processed_log.close()
print(datetime.now())
sleep(5) ######## 1 session complete , sleeping and connecting to server again to get the next set of files
The issue is not consistent. Sometimes, we have this error like 5 times in a day and some days 100+ times in a day
I have researched online and not sure where the issue is since the script ran fine for several months and processed 1000s of files per day in near real-time without any issues

New to python need to send data over already connected TCP port without waiting for request from client

7 socket listener setup. It works great and keeps the connection open, non blocking, all that. From time to time a file will show up that I need to hand back to the client. That works to, but it only send the data in the file if the client sends a character first. I need to have it send the data when the file shows up and not wait. I am coming from php and know what I am doing there. Python is new to me so there are some nuances I don't understand about this code.
while True:
try:
#I want this bit here to fire without waiting for the client to send anything
#right now it works except the client has to send a character first
#check for stuff to send back
for fname in os.listdir('data/%s/in' % dirname):
print(fname)
f = open('data/%s/in/%s' % (dirname, fname), "r")
client.send(f.readline())
data = client.recv(size)
if data:
bucket=bucket+data
else:
raise error('Client disconnected')
except Exception as e:
client.close()
print(e)
return False

Python: File download using ftplib hangs forever after file is successfully downloaded

I have been trying to troubleshoot an issue where in when we are downloading a file from ftp/ftps. File gets downloaded successfully but no operation is performed post file download completion. No error has occurred which could give more information about the issue.
I tried searching for this on stackoverflow and found this link which talks about similar problem statement and looks like I am facing similar issue, though I am not sure. Need little more help in resolving the issue.
I tried setting the FTP connection timeout to 60mins but of less help.
Prior to this I was using retrbinary() of the ftplib but same issue occurs there. I tried passing different blocksize and windowsize but with that also issue was reproducible.
I am trying to download the file of size ~3GB from AWS EMR cluster. Sample code is written below.
def download_ftp(self, ip, port, user_name, password, file_name, target_path):
try:
os.chdir(target_path)
ftp = FTP(host=ip)
ftp.connect(port=int(port), timeout=3000)
ftp.login(user=user_name, passwd=password)
if ftp.nlst(file_name) != []:
dir = os.path.split(file_name)
ftp.cwd(dir[0])
for filename in ftp.nlst(file_name):
sock = ftp.transfercmd('RETR ' + filename)
def background():
fhandle = open(filename, 'wb')
while True:
block = sock.recv(1024 * 1024)
if not block:
break
fhandle.write(block)
sock.close()
t = threading.Thread(target=background)
t.start()
while t.is_alive():
t.join(60)
ftp.voidcmd('NOOP')
logger.info("File " + filename + " fetched successfully")
return True
else:
logger.error("File " + file_name + " is not present in FTP")
except Exception, e:
logger.error(e)
raise
Another option suggested in the above mentioned link is to close the connection post downloading small chunk of the file and then restart the connection. Can someone suggest how can this be achieved, not sure how to resume the download from the same point where the file download was stopped last time before closing the connection. Will this method be full proof of downloading the entire file.
I don't know much about FTP server level timeout settings so didn't know what and how it needs to be altered. I basically want to write a generic FTP down-loader which can help in downloading the files from FTP/FTPS.
When I use retrbinary() method of ftplib and set debug level to 2.
ftp.set_debuglevel(2)
ftp.retrbinary('RETR ' + filename, fhandle.write)
Below logs are getting printed.
cmd 'TYPE I'
put 'TYPE I\r\n'
get '200 Type set to I.\r\n'
resp '200 Type set to I.'
cmd 'PASV'
put 'PASV\r\n'
get '227 Entering Passive Mode (64,27,160,28,133,251).\r\n'
resp '227 Entering Passive Mode (64,27,160,28,133,251).'
cmd 'RETR FFFT_BRA_PM_R_201711.txt'
put 'RETR FFFT_BRA_PM_R_201711.txt\r\n'
get '150 Opening BINARY mode data connection for FFFT_BRA_PM_R_201711.txt.\r\n'
resp '150 Opening BINARY mode data connection for FFFT_BRA_PM_R_201711.txt.'

Before doing anything, note that there is something very wrong with your connection, and diagnosing that and getting it fixed is far better than working around it. But sometimes, you just have to deal with a broken server, and even sending keepalives doesn't help. So, what can you do?
The trick is to download a chunk at a time, then abort the download—or, if the server can't handle aborting, close and reopen the connection.
Note that I'm testing everything below with ftp://speedtest.tele2.net/5MB.zip, which hopefully this doesn't cause a million people to start hammering their servers. Of course you'll want to test it with your actual server.
Testing for REST
The entire solution of course relies on the server being able to resume transfers, which not all servers can do—especially when you're dealing with something badly broken. So we'll need to test for that. Note that this test will be very slow, and very heavy on the server, so do not testing with your 3GB file; find something much smaller. Also, if you can put something readable there, it will help for debugging, because you may be stuck comparing files in a hex editor.
def downit():
with open('5MB.zip', 'wb') as f:
while True:
ftp = FTP(host='speedtest.tele2.net', user='anonymous', passwd='test#example.com')
pos = f.tell()
print(pos)
ftp.sendcmd('TYPE I')
sock = ftp.transfercmd('RETR 5MB.zip', rest=pos)
buf = sock.recv(1024 * 1024)
if not buf:
return
f.write(buf)
You will probably not get 1MB at a time, but instead something under 8KB. Let's assume you're seeing 1448, then 2896, 4344, etc.
If you get an exception from the REST, the server does not handle resuming—give up, you're hosed.
If the file goes on past the actual file size, hit ^C, and check it in a hex editor.
If you see the same 1448 bytes or whatever (the amount you saw it printing out) over and over again, again, you're hosed.
If you have the right data, but with extra bytes between each chunk of 1448 bytes, that's actually fixable. If you run into this and can't figure out how to fix it by using f.seek, I can explain—but you probably won't run into it.
Testing for ABRT
One thing we can do is try to abort the download and not reconnect.
def downit():
with open('5MB.zip', 'wb') as f:
ftp = FTP(host='speedtest.tele2.net', user='anonymous', passwd='test#example.com')
while True:
pos = f.tell()
print(pos)
ftp.sendcmd('TYPE I')
sock = ftp.transfercmd('RETR 5MB.zip', rest=pos)
buf = sock.recv(1024 * 1024)
if not buf:
return
f.write(buf)
sock.close()
ftp.abort()
You're going to want to try multiple variations:
No sock.close.
No ftp.abort.
With sock.close after ftp.abort.
With ftp.abort after sock.close.
All four of the above repeated with TYPE I moved to before the loop instead of each time.
Some will raise exceptions. Others will just appear to hang forever. If that's true for all 8 of them, we need to give up on aborting. But if any of them works, great!
Downloading a full chunk
The other way to speed things up is to download 1MB (or more) at a time before aborting or reconnecting. Just replace this code:
buf = sock.recv(1024 * 1024)
if buf:
f.write(buf)
with this:
chunklen = 1024 * 1024
while chunklen:
print(' ', f.tell())
buf = sock.recv(chunklen)
if not buf:
break
f.write(buf)
chunklen -= len(buf)
Now, instead of reading 1442 or 8192 bytes for each transfer, you're reading up to 1MB for each transfer. Try pushing it farther.
Combining with keepalives
If, say, your downloads were failing at 10MB, and the keepalive code in your question got things up to 512MB, but it just wasn't enough for 3GB—you can combine the two. Use keepalives to read 512MB at a time, then abort or reconnect and read the next 512MB, until you're done.

Resume FTP download after timeout

I'm downloading files from a flaky FTP server that often times out during file transfer and I was wondering if there was a way to reconnect and resume the download. I'm using Python's ftplib. Here is the code that I am using:
#! /usr/bin/python
import ftplib
import os
import socket
import sys
#--------------------------------#
# Define parameters for ftp site #
#--------------------------------#
site = 'a.really.unstable.server'
user = 'anonymous'
password = 'someperson#somewhere.edu'
root_ftp_dir = '/directory1/'
root_local_dir = '/directory2/'
#---------------------------------------------------------------
# Tuple of order numbers to download. Each web request generates
# an order numbers
#---------------------------------------------------------------
order_num = ('1','2','3','4')
#----------------------------------------------------------------#
# Loop through each order. Connect to server on each loop. There #
# might be a time out for the connection therefore reconnect for #
# every new ordernumber #
#----------------------------------------------------------------#
# First change local directory
os.chdir(root_local_dir)
# Begin loop through
for order in order_num:
print 'Begin Proccessing order number %s' %order
# Connect to FTP site
try:
ftp = ftplib.FTP( host=site, timeout=1200 )
except (socket.error, socket.gaierror), e:
print 'ERROR: Unable to reach "%s"' %site
sys.exit()
# Login
try:
ftp.login(user,password)
except ftplib.error_perm:
print 'ERROR: Unable to login'
ftp.quit()
sys.exit()
# Change remote directory to location of order
try:
ftp.cwd(root_ftp_dir+order)
except ftplib.error_perm:
print 'Unable to CD to "%s"' %(root_ftp_dir+order)
sys.exit()
# Get a list of files
try:
filelist = ftp.nlst()
except ftplib.error_perm:
print 'Unable to get file list from "%s"' %order
sys.exit()
#---------------------------------#
# Loop through files and download #
#---------------------------------#
for each_file in filelist:
file_local = open(each_file,'wb')
try:
ftp.retrbinary('RETR %s' %each_file, file_local.write)
file_local.close()
except ftplib.error_perm:
print 'ERROR: cannot read file "%s"' %each_file
os.unlink(each_file)
ftp.quit()
print 'Finished Proccessing order number %s' %order
sys.exit()
The error that I get:
socket.error: [Errno 110] Connection timed out
Any help is greatly appreciated.

Resuming a download through FTP using only standard facilities (see RFC959) requires use of the block transmission mode (section 3.4.2), which can be set using the MODE B command. Although this feature is technically required for conformance to the specification, I'm not sure all FTP server software implements it.
In the block transmission mode, as opposed to the stream transmission mode, the server sends the file in chunks, each of which has a marker. This marker may be re-submitted to the server to restart a failed transfer (section 3.5).
The specification says:
[...] a restart procedure is provided to protect users from gross system failures (including failures of a host, an FTP-process, or the underlying network).
However, AFAIK, the specification does not define a required lifetime for markers. It only says the following:
The marker information has meaning only to the sender, but must consist of printable characters in the default or negotiated language of the control connection (ASCII or EBCDIC). The marker could represent a bit-count, a record-count, or any other information by which a system may identify a data checkpoint. The receiver of data, if it implements the restart procedure, would then mark the corresponding position of this marker in the receiving system, and return this information to the user.
It should be safe to assume that servers implementing this feature will provide markers that are valid between FTP sessions, but your mileage may vary.

A simple example for implementing a resumable FTP download using Python ftplib:
def connect():
ftp = None
with open('bigfile', 'wb') as f:
while (not finished):
if ftp is None:
print("Connecting...")
FTP(host, user, passwd)
try:
rest = f.tell()
if rest == 0:
rest = None
print("Starting new transfer...")
else:
print(f"Resuming transfer from {rest}...")
ftp.retrbinary('RETR bigfile', f.write, rest=rest)
print("Done")
finished = True
except Exception as e:
ftp = None
sec = 5
print(f"Transfer failed: {e}, will retry in {sec} seconds...")
time.sleep(sec)
More fine-grained exception handling is advisable.
Similarly for uploads:
Handling disconnects in Python ftplib FTP transfers file upload

To do this, you would have to keep the interrupted download, then figure out which parts of the file you are missing, download those parts and then connect them together. I'm not sure how to do this, but there is a download manager for Firefox and Chrome called DownThemAll that does this. Although the code is not written in python (I think it's JavaScript), you could look at the code and see how it does this.
DownThemll - http://www.downthemall.net/