I want to have some kind of server that receives events (i.e using sockets), and each event has a different ID (i.e dst port number).
Is there a way that from the moment I see the first packet of an specific ID, I start some kind of timeout (i.e, 1ms), and if in that time nothing else with the same ID is received an event is triggered, but if something is received the timeout is reset to 1ms.
I have seen that something like that can be done by using signals and the SIGALARM signal. However, I want to keep multiple "timers" for every different ID.
Sounds like a job for select. As you are using sockets, you have a socket descriptor for a client (presumably one for each client but as long as you have one, it works). So you either want to wait until a packet arrives on one of your sockets or until a timeout occurs. This is exactly what select does.
So calculate the expiration time for each client when you receive a message, then in your main loop, simply calculate the soonest-to-expire timeout and provide that as the timeout parameter to select.select (with all the socket descriptors as the rlist parameter). Then you get awakened when a new packet/message arrives or when the oldest timeout expires. If it's a new packet, you process the packet and reset that provider's timeout to 1ms; otherwise, you do whatever you do when the timeout expires.
Then calculate the next-to-expire timeout. Rinse. Lather. Repeat.
Something like this:
now = time.time()
timeout = min([(client.expiration - now) for client in clients_list])
rrdy, wrdy, xrdy = select.select([client.sock for client in clients_list], [], [], timeout)
if not rrdy:
# Timeout
now = time.time()
for client in clients_list:
if client.expiration < now:
process_timeout(client)
else:
# Process incoming messages
for rsock in rrdy:
process_message(rsock.recv(4096))
client.expiration = time.time() + .001
See the sched built-in module, which has a scheduler.
You can construct a new scheduler instance, then use scheduler.enter to schedule a function to be called after a delay; and if you receive a message within the time limit, you can remove its event from the queue using scheduler.cancel(event); you can use the scheduler.run() to run the scheduler in another thread, or you can use scheduler.run(blocking=False) in a select-multiplexing thread with timeouts.
Related
I'm struggling understanding a "weird" behavior of my simple script. Basically, it works as expected if time.sleep() is set as 60s but as soon as I put a value above 90 (90 is the limit apparently in my case), the loop doesn't work properly. I discovered this when I was trying to pause the script for 3 mins.
Here's my script
from gpiozero import CPUTemperature
import time
import paho.mqtt.client as mqtt #import the client1
import psutil
broker_address="192.168.1.17"
client = mqtt.Client("P1") #create new instance
client.connect(broker_address) #connect to broker
#time.sleep(60)
while True:
cpu = CPUTemperature()
print(cpu.temperature)
#a=cpu.temperature
#print(psutil.cpu_percent())
#print(psutil.virtual_memory()[2])
#print(a)
client.publish("test/message",cpu.temperature)
#client.publish("test/ram", psutil.virtual_memory()[2])
#client.publish("test/cpu", psutil.cpu_percent())
time.sleep(91)
In this case, with 91s it just prints the value of cpu.temperature every 91s, whereas with a value like 60s, besides printing, it also publishes the value via mqtt every cycle.
Am I doing something wrong here? Or for a longer sleep I need to change my code? I'm running this on a RaspberryPi.
Thanks in advance
EDIT:
I solved modifying the script, in particular how mqtt was handling the timing
here's the new script
mqttc=mqtt.Client("P1")
#mqttc.on_connect = onConnect
#mqttc.on_disconnect = onDisconnect
mqttc.connect("192.168.1.17", port=1883, keepalive=60)
mqttc.loop_start()
while True:
cpu = CPUTemperature()
print(cpu.temperature)
mqttc.publish("test/message",cpu.temperature)
time.sleep(300)
The MQTT client uses a network thread to handle a number of different aspects of the connection to the broker.
Firstly, it handles sending ping request to the broker in order to keep the connection alive. The default period for the keepalive period is 60 seconds. The connection will be dropped by the broker if it does not receive any messages in 1.5 times this value, which just happens to be 90 seconds.
Secondly, the thread handles any incoming messages that the client may have subscribed to.
Thirdly, if you try to publish a message that is bigger than the MTU of the network link, calling mqttc.publish() will only send the first packet and the loop is needed to send the rest of the payload.
There are 2 ways to run the network tasks.
As you have found, you can start a separate thread with the mqttc.loop_start()
The other option is to call mqttc.loop() within your own while loop
I have a Python script that is used to receive data associated with a radio station audio event (such as a song or commercial) from the machine playing the audio. The script will parse and process the data and then send portions of it to various other destinations.
First the socket is set up:
client_socket_1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print 'trying to open socket 1'
client_socket_1.connect((TCP_RCV_IP_CR1, TCP_RCV_PORT_CR1))
client_socket_1.setblocking(0)
except socket.error, e:
print 'Error', e, TCP_RCV_IP_CR1, '\n\n\n'
else:
SOCK1 = 1
print 'Successful connection to ',TCP_RCV_IP_CR1,'\n'
Now we wait until data is available to be read. I used select() and when the socket is ready to be read, the thread that parses and processes the data is spawned.
ready_1 = select.select([client_socket_1], [], [], 1) # select tells us when data is available at the socket
if ready_1[0] and SOCK1: # Don't run this code if there is no connection on client_socket_1 or no data available
t1 = Thread(target=processdata1) # Set up the thread
t1.start() # Call the process to process available data as a thread
It is important that the data be read as quickly as possible as it will be transported via TCP or UDP (depending on the particular data chunks and program specifications) along with the associated audio, and the function of one of the data items we are handling can create an on-air 'hiccup' in the audio if not received in a timely fashion. (TMI: It causes a 'replacement' commercial to play at the receiving end which is supposed to 'cover' the commercial audio we are sending. If the replacement spot doesn't start quickly enough listeners will hear the beginning of the commercial we are sending, then the local replacement one will start when our data is received and it sounds like a hiccup on the air.)
To confirm that my script is not always receiving the data quickly enough I telnetted to the port it is listening to and watched the data as it is received in the telnet window, then look at the Python output (which sends received data to stdout as soon as it is received) and I see about a 1.5-second delay between the telnet output and the Python output. This is the same amount of delay we have observed in normal on-air operation.
I chose to use select() because I was asked to multi-thread the script and I thought that would be a good way to know when to trigger a data-processing thread. My original idea was to simply loop through attempting to read data from each of the three systems we are monitoring and, when data is found, process it.
The thought was that if data is being processed from one system when another system has data ready to be read, it might cause a delay in processing and sending out the data from that machine. However, I can't see that delay being as significant as what I am experiencing now. I am considering going back to the original plan.
I would rather stick with what I have which is working flawlessly as long as data is received in a timely fashion. Any thoughts on why the excessively long delay?
I think it has to do with your timeout parameter in combination with the wlist and xlist parameters
Consider this piece of code
write_list = []
exception_list = []
select.select([client_socket_1], write_list, exception_list)
It takes an optional timeout parameter, like you use it. The documentation says
select() also takes an optional fourth parameter which is the number
of seconds to wait before breaking off monitoring if no channels have
become active. Using a timeout value lets a main program call select()
as part of a larger processing loop, taking other actions in between
checking for network input.
It might be that the call will always wait one second before returning because of the empty lists. Try
ready_1 = select.select(
[client_socket_1],
[client_socket_1],
[client_socket_1], 1
)
Or you can use a timeout value of 0, which
specifies a poll and never blocks.
A have a application with two threads. Its a network controlled game,
1. thread (Server)
Accept socket connections and receive messages
When message is sent, create an event and add it to the queue
Code:
class SingleTCPHandler(SocketServer.StreamRequestHandler):
def handle(self):
try:
while True:
sleep(0.06)
message = self.rfile.readline().strip()
my_event = pygame.event.Event(USEREVENT, {'control':message})
print message
pygame.event.post(my_event)
2. thread (pygame)
In charge of game rendering
Receives messages via event queue which Server populates
Renders the game based on messages every 60ms
This is how the game looks. The control messages are just speeds for the little square.
For the purpose of debug i connect to the server from a virtual machine with:
ncat 192.168.56.1 2000
And then send control messages. In production, these messages will be sent every 50ms by an Android device.
The problem
In my debug environment, i manually type messages with a period of a few seconds. During the time i don't type anything the game gets rendered many times. What happens is that the message (in server code) is constantly rendered with the previously received value.
I send the following:
1:0.5
On the console where the app is started i receive the following due to line print message in Server code:
alan#alan ~/.../py $ python main.py
1:0.5
What the game does is it acts as it is constantly (with the period it renders, and not every few seconds as i type) receiving this value.
SInce that is happenig i would expect that the print message which is in while True also outputs constantly and that the output is:
alan#alan ~/.../py $ python main.py
1:0.5
1:0.5
1:0.5
1:0.5
....
However that is not the case. Please advise (I'm also open for proposals to what to change the subject to if it isn't explanatory enough)
Your while True loop is polling the socket, which is only going to get messages when they are sent; it has no idea or care what the downstream event consumer is doing with those messages, it is just going to dispatch an event for and print the contents of the next record on the socket queue every .6 seconds. If you want the game to print the current command every render loop, you'll have to put the print statement in the render loop itself, not in the socket poller. Also, since you seem to want to have the last command "stick" and not post a new event unless the user actually inputs something, you might want to put an if message: block around the event dispatch code in the socket handler you have here. Right now, you'll send an empty event every .6 seconds if the user hasn't provided you any input since the last time you checked.
I also don't think it's probably advisable to put a sleep, or the loop you have for that matter, in your socket handler. The SocketServer is going to be calling it every time you receive data on the socket, so that loop is effectively being done for you, and all doing it here is going to do is open you up to overflowing the buffer, I think. If you want to control how often you post events to pygame, you probably want to do that by either blocking events of a certain type from being added if there is already 1 queued, or by grabbing all events of a given type from the queue each game loop and then just ignoring all but the first or last one. You could also control it by checking in the handler if it has been some amount of time since the last event was posted, but then you have to make sure the event consumer is capable of handling an event queue with multiple events waiting on it, and does the appropriate queue flushing when needed.
Edit:
Docs:
The difference is that the readline() call in the second handler will call recv() multiple times until it encounters a newline character, while the single recv() call in the first handler will just return what has been sent from the client in one sendall() call.
So yes, reading the whole line is guaranteed. In fact, I don't think the try is necessary either, since this won't even be called unless there is input to handle.
I am using asyncore and asynchat modules to build a SMTP server (I used code from smtpd lib to build the SMTP server) but I have a problem with connection timeouts. When I open telnet connection to the SMTP server and leave it so, the connection is established althought no data exchange happens. I want to set timeout e.g 30 seconds and to close the idle connection by server if nothing comes from the client (else there could be an easy DOS vulnerability). I googled for a solution, read source codes and documentation but didn't found anything usable.
Thanks
According to asyncore documentation, asyncore.loop() has a timeout parameter, which defaults to 30 seconds. So apparently default already should be 30 seconds, you can try and play with it to suit your own needs.
The timeout argument sets the timeout parameter for the appropriate
select() or poll() call, measured in seconds; the default is 30
seconds.
Ok, the above actually refers to poll() or select() timeout and not the idle timeout.
As per this page, you can hack asyncore to support timeouts like this:
Add the following block to your own copy of asyncore.poll just after the for fd in e: block...
#handle timeouts
rw = set(r) + set(w)
now = time.time()
for f in (i for i in rw if i in map):
map[f].lastdata = now
for j in (map[i] for i in map if i not in rw):
if j.timeout+j.lastdata now:
#timeout!
j.handle_close()
You ARE going to need to initialize .timeout and .lastdata members for
every instance, but that shouldn't be so bad (for a socket that
doesn't time out, I would actually suggest a 1 hour or 1 day timeout).
If I connect to an inexistent socket with pyzmq I need to hit CTRL_C to stop the program. Could someone explay why this happens?
import zmq
INVALID_ADDR = 'ipc:///tmp/idontexist.socket'
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect(INVALID_ADDR)
socket.send('hello')
poller = zmq.Poller()
poller.register(socket, zmq.POLLIN)
conn = dict(poller.poll(1000))
if conn:
if conn.get(socket) == zmq.POLLIN:
print "got result: ", socket.recv(zmq.NOBLOCK)
else:
print 'got no result'
This question was also posted as a pyzmq Issue on GitHub. I will paraphrase my explanation here (I hope that is appropriate, I am fairly new to SO):
A general rule: When in doubt, hangs at the end of your zeromq program are due to LINGER.
The hang here is caused by the LINGER socket option, and happens in the context.term() method called during garbage collection at the very end of the script. The LINGER behavior is described in the zeromq docs, but to put it simply, it is a timeout (in milliseconds) to wait for any pending messages in the queue to be handled after closing the socket before dropping the messages. The default behavior is LINGER=-1, which means to wait forever.
In this case, since no peer was ever started, the 'hello' message that you tried to send is still waiting in the send queue when the socket tries to close. With LINGER=-1, ZeroMQ will wait until a peer is ready to receive that message before shutting down. If you bind a REP socket to 'ipc:///tmp/idontexist.socket' while this script is apparently hanging, the message will be delivered and the script will finish exiting cleanly.
If you do not want your script to wait (as indicated by your print statements that you have already given up on getting a reply), set LINGER to any non-negative value (e.g. socket.linger = 0), and context.term() will return after waiting the specified number of milliseconds.
I should note that the INVALID_ADDR variable name suggests an understanding that connection to an interface that does not yet have a listener is not valid - this is incorrect. zeromq allows bind/connect events to happen in any order, as illustrated by the behavior described above, of binding a REP socket to the interface while the sending script is blocking on term().
In most cases, you can bind and connect ZMQ sockets in either order, so your connect()/send() is simply waiting for the corresponding bind() at the other end, which never comes, so the program appears to hang. Check where the program is hanging by printing out some logging statements...