I'm struggling understanding a "weird" behavior of my simple script. Basically, it works as expected if time.sleep() is set as 60s but as soon as I put a value above 90 (90 is the limit apparently in my case), the loop doesn't work properly. I discovered this when I was trying to pause the script for 3 mins.
Here's my script
from gpiozero import CPUTemperature
import time
import paho.mqtt.client as mqtt #import the client1
import psutil
broker_address="192.168.1.17"
client = mqtt.Client("P1") #create new instance
client.connect(broker_address) #connect to broker
#time.sleep(60)
while True:
cpu = CPUTemperature()
print(cpu.temperature)
#a=cpu.temperature
#print(psutil.cpu_percent())
#print(psutil.virtual_memory()[2])
#print(a)
client.publish("test/message",cpu.temperature)
#client.publish("test/ram", psutil.virtual_memory()[2])
#client.publish("test/cpu", psutil.cpu_percent())
time.sleep(91)
In this case, with 91s it just prints the value of cpu.temperature every 91s, whereas with a value like 60s, besides printing, it also publishes the value via mqtt every cycle.
Am I doing something wrong here? Or for a longer sleep I need to change my code? I'm running this on a RaspberryPi.
Thanks in advance
EDIT:
I solved modifying the script, in particular how mqtt was handling the timing
here's the new script
mqttc=mqtt.Client("P1")
#mqttc.on_connect = onConnect
#mqttc.on_disconnect = onDisconnect
mqttc.connect("192.168.1.17", port=1883, keepalive=60)
mqttc.loop_start()
while True:
cpu = CPUTemperature()
print(cpu.temperature)
mqttc.publish("test/message",cpu.temperature)
time.sleep(300)
The MQTT client uses a network thread to handle a number of different aspects of the connection to the broker.
Firstly, it handles sending ping request to the broker in order to keep the connection alive. The default period for the keepalive period is 60 seconds. The connection will be dropped by the broker if it does not receive any messages in 1.5 times this value, which just happens to be 90 seconds.
Secondly, the thread handles any incoming messages that the client may have subscribed to.
Thirdly, if you try to publish a message that is bigger than the MTU of the network link, calling mqttc.publish() will only send the first packet and the loop is needed to send the rest of the payload.
There are 2 ways to run the network tasks.
As you have found, you can start a separate thread with the mqttc.loop_start()
The other option is to call mqttc.loop() within your own while loop
Related
I have a large python script with a thread that listens to a serial port and puts new data to a queue whenever it's received. I've been trying to improve the performance of the script, as right now even when nothing is happening it's using ~ 12% of my Ryzen 3600 CPU. That seems very excessive.
Here's the serial listener thread:
def listen(self):
"""
Main listener
"""
while self.doListen:
# wait for a message
if self.bus.ser.in_waiting:
# Read rest of message
msg = self.bus.read(self.bus.ser.in_waiting)
# Put into queue
self.msgQueue.put_nowait(msg)
I profiled the script using yappi and found that the serial.in_waiting call seems to be hogging the majority of the cycles. See the KCachegrind screenshot below:
I tried the trick suggested in this question, doing a blocking read(1) call to wait for data. However read(1) just continuously returns empty data and never actually blocks (and yes, I've made sure my pyserial timeout is set to None)
Is there a more elegant and CPU-friendly way to wait for data on the serial bus? My messages are of variable length, so doing a blocking read(n) wouldn't work. They also don't end in newlines or any specific terminators, so readline() wouldn't work either.
Aaron's suggestion was great. A simple time.sleep(0.01) in my serial thread dramatically cut down on CPU usage. So far it looks like I'm not missing any messages either, which was my big fear with adding in sleeps.
The new listener thread:
def listen(self):
"""
Main listener
"""
while self.doListen:
# wait for a message
if self.bus.ser.in_waiting:
# Read rest of message
msg = self.bus.read(self.bus.ser.in_waiting)
# Put into queue
self.msgQueue.put_nowait(msg)
# give the CPU a break
time.sleep(0.01)
I want to have some kind of server that receives events (i.e using sockets), and each event has a different ID (i.e dst port number).
Is there a way that from the moment I see the first packet of an specific ID, I start some kind of timeout (i.e, 1ms), and if in that time nothing else with the same ID is received an event is triggered, but if something is received the timeout is reset to 1ms.
I have seen that something like that can be done by using signals and the SIGALARM signal. However, I want to keep multiple "timers" for every different ID.
Sounds like a job for select. As you are using sockets, you have a socket descriptor for a client (presumably one for each client but as long as you have one, it works). So you either want to wait until a packet arrives on one of your sockets or until a timeout occurs. This is exactly what select does.
So calculate the expiration time for each client when you receive a message, then in your main loop, simply calculate the soonest-to-expire timeout and provide that as the timeout parameter to select.select (with all the socket descriptors as the rlist parameter). Then you get awakened when a new packet/message arrives or when the oldest timeout expires. If it's a new packet, you process the packet and reset that provider's timeout to 1ms; otherwise, you do whatever you do when the timeout expires.
Then calculate the next-to-expire timeout. Rinse. Lather. Repeat.
Something like this:
now = time.time()
timeout = min([(client.expiration - now) for client in clients_list])
rrdy, wrdy, xrdy = select.select([client.sock for client in clients_list], [], [], timeout)
if not rrdy:
# Timeout
now = time.time()
for client in clients_list:
if client.expiration < now:
process_timeout(client)
else:
# Process incoming messages
for rsock in rrdy:
process_message(rsock.recv(4096))
client.expiration = time.time() + .001
See the sched built-in module, which has a scheduler.
You can construct a new scheduler instance, then use scheduler.enter to schedule a function to be called after a delay; and if you receive a message within the time limit, you can remove its event from the queue using scheduler.cancel(event); you can use the scheduler.run() to run the scheduler in another thread, or you can use scheduler.run(blocking=False) in a select-multiplexing thread with timeouts.
I am unable to grasp this with the help of Programming concepts in general with the following scenario:
Note: All Data transmission in this scenario is done via UDP packets using socket module of Python3
I have a Server which sends some certain amount of data, assume 300 Packets over a WiFi Channel
At the other end, I have a receiver which works on a certain Decoding process to decode the data. This Decoding Process is kind of Infinite Loop which returns Boolean Value true or false at every iteration depending on certain aspects which can be neglected as of now
a Rough Code Snippet is as follows:Python3
incomingPacket = next(bringNextFromBuffer)
if decoder.consume_data(incomingPacket):
# this if condition is inside an infinite loop
# unless the if condition becomes True keep
# keep consuming data in a forever for loop
print("Data has been received")
Everything as of moment works since the Server and Client are in proximity and the data can be decoded. But in practical scenarios I want to check the loop that is mentioned above. For instance, after a certain amount of time, if the above loop is still in the Forever (Infinite) state I would like to send out something back to the server to start the data sending again.
I am not much clear with multithreading concept, but can I use a thread over here in this scenario?
For Example:
Thread a Process for a certain amount of time and keep checking the decoder.consume_data() function and if the time expires and the output is still False can I then send out a kind of Feedback to the server using struct.pack() over sockets.
Of course the networking logic, need NOT be addressed as of now. But is python capable of MONITORING THIS INFINITE LOOP VIA A PARALLEL THREAD OR OTHER CONCEPT OF PROGRAMMING?
Caveats
Unfortunately the Receiver in question is a dumb receiver i.e. No user control is specified. Only thing Receiver can do is decode the data and perhaps send a Feedback to the Server stating whether the data is received or not and that is possible only when the above mentioned LOOP is completed.
What is a possible solution here?
(Would be happy to share more information on request)
Yes you can do this. Roughly it'll look like this:
from threading import Thread
from time import sleep
state = 'running'
def monitor():
while True:
if state == 'running':
tell_client()
sleep(1) # to prevent too much happening here
Thread(target=monitor).start()
while state == 'running':
receive_data()
A have a application with two threads. Its a network controlled game,
1. thread (Server)
Accept socket connections and receive messages
When message is sent, create an event and add it to the queue
Code:
class SingleTCPHandler(SocketServer.StreamRequestHandler):
def handle(self):
try:
while True:
sleep(0.06)
message = self.rfile.readline().strip()
my_event = pygame.event.Event(USEREVENT, {'control':message})
print message
pygame.event.post(my_event)
2. thread (pygame)
In charge of game rendering
Receives messages via event queue which Server populates
Renders the game based on messages every 60ms
This is how the game looks. The control messages are just speeds for the little square.
For the purpose of debug i connect to the server from a virtual machine with:
ncat 192.168.56.1 2000
And then send control messages. In production, these messages will be sent every 50ms by an Android device.
The problem
In my debug environment, i manually type messages with a period of a few seconds. During the time i don't type anything the game gets rendered many times. What happens is that the message (in server code) is constantly rendered with the previously received value.
I send the following:
1:0.5
On the console where the app is started i receive the following due to line print message in Server code:
alan#alan ~/.../py $ python main.py
1:0.5
What the game does is it acts as it is constantly (with the period it renders, and not every few seconds as i type) receiving this value.
SInce that is happenig i would expect that the print message which is in while True also outputs constantly and that the output is:
alan#alan ~/.../py $ python main.py
1:0.5
1:0.5
1:0.5
1:0.5
....
However that is not the case. Please advise (I'm also open for proposals to what to change the subject to if it isn't explanatory enough)
Your while True loop is polling the socket, which is only going to get messages when they are sent; it has no idea or care what the downstream event consumer is doing with those messages, it is just going to dispatch an event for and print the contents of the next record on the socket queue every .6 seconds. If you want the game to print the current command every render loop, you'll have to put the print statement in the render loop itself, not in the socket poller. Also, since you seem to want to have the last command "stick" and not post a new event unless the user actually inputs something, you might want to put an if message: block around the event dispatch code in the socket handler you have here. Right now, you'll send an empty event every .6 seconds if the user hasn't provided you any input since the last time you checked.
I also don't think it's probably advisable to put a sleep, or the loop you have for that matter, in your socket handler. The SocketServer is going to be calling it every time you receive data on the socket, so that loop is effectively being done for you, and all doing it here is going to do is open you up to overflowing the buffer, I think. If you want to control how often you post events to pygame, you probably want to do that by either blocking events of a certain type from being added if there is already 1 queued, or by grabbing all events of a given type from the queue each game loop and then just ignoring all but the first or last one. You could also control it by checking in the handler if it has been some amount of time since the last event was posted, but then you have to make sure the event consumer is capable of handling an event queue with multiple events waiting on it, and does the appropriate queue flushing when needed.
Edit:
Docs:
The difference is that the readline() call in the second handler will call recv() multiple times until it encounters a newline character, while the single recv() call in the first handler will just return what has been sent from the client in one sendall() call.
So yes, reading the whole line is guaranteed. In fact, I don't think the try is necessary either, since this won't even be called unless there is input to handle.
I am using asyncore and asynchat modules to build a SMTP server (I used code from smtpd lib to build the SMTP server) but I have a problem with connection timeouts. When I open telnet connection to the SMTP server and leave it so, the connection is established althought no data exchange happens. I want to set timeout e.g 30 seconds and to close the idle connection by server if nothing comes from the client (else there could be an easy DOS vulnerability). I googled for a solution, read source codes and documentation but didn't found anything usable.
Thanks
According to asyncore documentation, asyncore.loop() has a timeout parameter, which defaults to 30 seconds. So apparently default already should be 30 seconds, you can try and play with it to suit your own needs.
The timeout argument sets the timeout parameter for the appropriate
select() or poll() call, measured in seconds; the default is 30
seconds.
Ok, the above actually refers to poll() or select() timeout and not the idle timeout.
As per this page, you can hack asyncore to support timeouts like this:
Add the following block to your own copy of asyncore.poll just after the for fd in e: block...
#handle timeouts
rw = set(r) + set(w)
now = time.time()
for f in (i for i in rw if i in map):
map[f].lastdata = now
for j in (map[i] for i in map if i not in rw):
if j.timeout+j.lastdata now:
#timeout!
j.handle_close()
You ARE going to need to initialize .timeout and .lastdata members for
every instance, but that shouldn't be so bad (for a socket that
doesn't time out, I would actually suggest a 1 hour or 1 day timeout).