I have an azure timer function which runs every minute to trigger a socket which gets data from a website. I don't want to establish a connection everytime the timer runs the function. So, is there a way in Python which I can check if a socket is open for a particular website on particular port?
Or, is there a way to re-use a socket in time-triggered applications?
# Open socket
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(20) # 20 sec timeout
if is_socket_open(sock):
logging.info("Socket is already open")
else:
logging.info("No socket was open. Opening a new one...")
sock.connect(server_address)
sock.settimeout(None)
logging.info(f"Connected to {sock}")
return sock
except socket.gaierror as e:
logging.exception(f"Error connecting to remote server {e}")
time.sleep(20)
except socket.error as e:
logging.exception(f"Connection error {e}")
time.sleep(20)
except Exception as e:
logging.exception(f"An exception occurred: {e}")
time.sleep(20)
def is_socket_open(sock: socket.socket) -> bool:
try:
# this will try to read bytes without blocking and also without removing them from buffer (peek only)
data = sock.recv(16, socket.MSG_PEEK)
if len(data) == 0:
return True
except socket.timeout:
return False # socket is not connected yet, therefore receiving timed out
except BlockingIOError:
return False # socket is open and reading from it would block
except ConnectionResetError:
return True # socket was closed for some other reason
except Exception as e:
logging.exception(f"unexpected exception when checking if a socket is closed: {e}")
return False
return False
So this entire process runs every minute.
You can always use global variables to reuse objects in future invocations. The following example was copied from Google Cloud Platform documentation, but you can apply the same concept to your Azure Function:
# Global (instance-wide) scope
# This computation runs at instance cold-start
instance_var = heavy_computation()
def scope_demo(request):
# Per-function scope
# This computation runs every time this function is called
function_var = light_computation()
return 'Instance: {}; function: {}'.format(instance_var, function_var)
In your case, you can declare the sock as a global variable and reuse it in future warm start invocations. You should also increase the timeout to above 60 seconds, giving that you're triggering your azure function every minute.
However, keep in mind that there is no guarantee that the state of the function will be preserved for future invocations. For instance, in auto-scaling situations, a new socket would be open.
Microsoft Azure also says the following in regards to client connections:
To avoid holding more connections than necessary, reuse client instances rather than creating new ones with each function invocation. We recommend reusing client connections for any language that you might write your function in.
See also:
Manage connections in Azure Functions
Related
I know that Django Channels can be used to make websocket server, not client. So I used websockets to relay websocket incoming message to my Django like this:
async def relay():
source_server = 'ws://source.example/ws/message' # This is an external server
target_server = 'ws://target.example/ws/message' # This is my Django server
async for target in websockets.connect(target_server):
try:
async for source in websockets.connect(source_server):
try:
while True:
try:
message = await source.recv()
await target.send()
# log message
except websockets.ConnectionClosed as e:
# lost source server or target server or both
raise(e)
except Exception as e:
# did not lose servers
continue
except websockets.ConnectionClosed as e:
# lost source server or target server or both
if target.close_code is not None:
# lost target server and need to start from the outer for loop
# (to get a new target websocket connection)
source.disconnect()
raise(e)
# lost source server and will continue the inner for loop
# (to get a new source websocket connection)
continue
except Exception as e:
# did not lose any server and will continue the inner for loop
# (to get a new source websocket connection)
continue
except websockets.ConnectionClosed as e:
# lost target server and will continue the outer for loop
# (to get a new target websocket connection)
continue
except Exception as e:
# did not lose any server and will start the outer for loop
# (to get a new target websocket connection)
continue
asyncio.run(relay())
Understandably, this is not the most efficient code-up. But this is what I can think of.
I run this code as a Docker container (let's call it relay container) along side with my Django Docker containers (with the same Docker image as Django of course).
Here's my questions:
Is there a way to make Django a websocket client? (I want to save one container for the relay). For your information, I run Django container (using Daphne), two Celery (one for beat and one for worker) containers.
If I bring down the relay container, it takes long time (five to ten seconds) to be down. The exit code is Out Of Memory. What causes the problem? How can I shutdown the container gracefully?
Thanks.
I am creating a socket where the server creates a new log file daily. At midnight, a backup of the previous day's log needs to be taken. The log file is huge and it could take time to compress and upload it on the cloud. I don't want the server to wait until backup is taken before it continues to listen to new incoming requests. I have written pseudo code from the 'if' condition in my code below as I am not sure how to implement it. I am guessing I'll have to use multiprocessing here?
while True:
clientsocket, address = s.accept()
enter_log("SUCCESS", "Established incoming connection from {}".format(address))
if midnight:
call_another_function_parallelly_for_backup_and_let_the_loop_continue_listening
while True:
clientsocket, address = s.accept()
enter_log("SUCCESS", "Established incoming connection from {}".format(address))
if midnight:
t = threading.Thread(target=your_func, args=(your_arg,))
t.start()
t.join()
def your_func(your_arg):
try:
your logic
except Exception as e:
print(e)
This will create new thread until looping and if is true...
Using pyBluez, I use the following code to advertise and listen for a bluetooth connection:
def connect_socket():
global client_sock
try:
server_sock = BluetoothSocket(RFCOMM)
server_sock.bind(("", PORT_ANY))
server_sock.listen(1)
port = server_sock.getsockname()[1]
uuid = "00001101-0000-1000-8000-00805F9B34FB"
advertise_service(server_sock, "GSA",
service_id=uuid,
service_classes=[uuid, SERIAL_PORT_CLASS],
profiles=[SERIAL_PORT_PROFILE])
print("Waiting for connection on RFCOMM channel %d" % port)
client_sock, client_info = server_sock.accept()
print("Accepted connection from ", client_info)
except Exception as e: (yes, I know I'm catching all exceptions)
print(e)
I use the following to call the above and send data out from the socket. (I wind up waiting for a connection on every possible channel, which is not desirable, but that's not my only problem or the one that's prompting this question, though I'd like to fix it, too.)
def write_bt(message):
global client_sock
if client_sock is None:
threading.Thread(target=connect_socket).start()
if client_sock is not None:
try:
client_sock.send(message)
except Exception as e:
gsa_msg.message(e)
client_sock = None
I also need to receive data from the socket and write it to a usb connection. For this, I use the following:
def forward_bt_to_usb():
global client_sock
global serUSB
if (client_sock is not None) and (serUSB is not None):
try:
data = client_sock.recv(1024)
serUSB.write(data)
except Exception as e:
gsa_msg.error(e)
client_sock = None
Both write_bt() and forward_bt_to_usb() get called continuously from a loop and are communicating with the same client, but there isn't always data being received over the socket, and forward_bt_to_usb() seems to block everything in that case.
I believe that I probably have all of this structured improperly for what I'm trying to do, or perhaps I just need to have separate threads for sending and receiving data, but it's not obvious to me how to do that (Initially I just put some of the code from forward_bt_to_usb() in a separate thread, without realizing that that would just keep creating new threads as forward_bt_to_usb() kept getting called.)
It seems that what I'm trying to do should be pretty straightforward and certainly not novel, but I haven't been able to find examples or an explanation that I've been able to implement.
I have a big question I couldn't get answer from the web about
sockets in python.
I'm making a simple client program (python) based on socket:
Connecting to a server.
I would like to make a function that its purpose is just to try to connect to the server, otherwise the application will not work.
Because I had trouble with "global" socket variable across the whole class, I decided to make a local socket variable inside my main and pass it through all functions.
I wanted to make sure that I understand it 100% :
Should I return the socket from the function that's trying to connect to the server ( otherwise sleeps for 0.5 a second and tries again )
OR I don't need to return the socket at all and the socket variable itself will be updated ?
UPDATE
#will try to connect to the server
def try_connecting_to_server(socket_to_server):
connected = False
while not connected:
try:
socket_to_server.connect((HOST, PORT)) # connect to the server
connected = True
except:
print "couldn't connect to server, sleeping for 0.5 seconds"
time.sleep(0.5)
return socket_to_server
def main():
# start the socket to the server
socket_to_server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # setup the socket for future use
try:
socket_to_server = try_connecting_to_server(socket_to_server)
handle_missions(socket_to_server) # pass the socket
except:
print "encountered an error"
finally:
socket_to_server.sendall(PROTOCOL_CLOSE_KEY)
socket_to_server.close()
if __name__ == "__main__":
main()
def try_connecting_to_server(socket_to_server):
connected = False
while not connected:
try:
socket_to_server.connect((HOST, PORT)) # connect to the server
connected = True
except:
print "couldn't connect to server, sleeping for 0.5 seconds"
time.sleep(0.5)
return socket_to_server
There is no reason for this function to return socket_to_server.
Since a socket object is mutable, any changes to it inside the function (e.g. connecting it to a server) are visible to the function which called it.
You can verify that by making this change in main():
returned_sock = try_connecting_to_server(socket_to_server)
if returned_sock is socket_to_server:
print "They refer to the exact same object!"
See How do I pass a variable by reference?
I have a python program where I use a server socket to send data. There is a class which has some Threading methods. Each method checks a queue and if the queue is not empty, it sends the data over the sever socket. Queues are being filled with what clients send to server(server is listening for input requests). Sending is accomplished with a method call:
def send(self, data):
self.sqn += 1
try:
self.clisock.send(data)
except Exception, e:
print 'Send packet failed with error: ' + e.message
When the program starts, sending rate is around 500, but after a while it decreases instantly to 30 with this exception:
Send packet failed with error: <class 'socket.error'>>>[Errno 32] Broken pipe
I don't know what causes the rate to increase! Any idea?
That error is from your send function trying to write to a socket closed on the other side. If that is intended then catch the exception using
import errno, socket
try:
self.clisock.send(data)
except socket.error, err:
if err[0] == errno.EPIPE:
# do something
else:
pass # do something else
If this isn't intended behavior on the part of the client then you'll have to update your post with the corresponding client code.