We Keep Coding

bind takes a long time

I am using over 20 raspberry pis to control heating and other functions in my house. For instance each room has a pi (client) with a temperature sensor, which sends a message to another pi (server) at one of three manifolds to open or close a valve on the heating system.
My problem is the amount of time the server takes to execute the bind instruction. A newly set up pi will typically take a few millisecond to execute the line, but soon the time becomes variable, sometimes milliseconds, usually seconds, but frequently tens of seconds or even hundreds of seconds.
typical code,
elapselp = default_timer() # timing start
port = 12571 # port number, many different ones used
while bound == 0:
try:
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.bind(('', port))
bound = 1
elapselp1 = default_timer() - elapselp
print ('bind time = ', elapselp1)
As stated above elapselp1 can be at the low millisecond level, or 200 seconds which causes significant problems.
I usually find I can get back to short times by changing the port number used, which means changing on all the clients, but after some time running happily the times will increase again. Rebooting or powering down does not help. I am using Raspberry 4s and Zero Ws, no difference.
I suspect that some register is not clearing, but outside that am clueless. Can somebody advise please.
I am running Rasparian 3.6 and Python3
Thank you for all your replies, here is the remainder of this section of the code.
As I’m not sure how to input his I’m breaking it down over a number of comments.
elapselp = default_timer()
port =12571
while bound == 0:
try:
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.bind(('', port))
bound = 1
elapselp1 = default_timer()- elapselp
print (‘bind time = ‘, elapselp2)
except:
dummy = 1
s1.listen(5)
print ('waiting for input ', end='')
while True:
conn, addr = s1.accept()
print ('connected by ', addr)
conn.send('thank you'.encode())
data = conn.recv(1024).decode()
print ('data = ', data)
time.sleep(5)
conn.close()
s1.close()
if data == '10':
startdaw = default_timer()
GPIO.output(22, GPIO.LOW)
print ('switch dining area heating off')
h1 = 0
dawl = edaw
startdawr = default_timer()
elif data == '11':
GPIO.output(22, GPIO.HIGH)
## A lot of other actions depending on data received here
else:
print ('another code ', data)
if not data:
break
print ('yet another')
I have a number of Raspberry pi Zeros running the code for test along with 7 doing so as part of my control system. The performance is very erratic.
One unit running through the code every 30 seconds in response to a client has been running over 5 days, majority of the bind times have been under 1 millsec, I see some between 1 and 10, but it has recorded one time of 130 seconds.
Another unit which failed to bind for over two hours has now completed over 33,000 binds (this one just closes after the bind, no if statements), and the longest time for a bind is 710 msec.
If I take out the try/except I get no error message, it just drops through and loops back.
I hope there is some useful information in there.

Using Arduino Ultrasonic Sensor with Pyfirmata

I'm trying to use pyfirmata to use an Arduino Ultrasonic sensor. I used Arduino Uno board and HC-SR04 Ultrasonic sensor. Here is the code I'm using. The code ran smoothly, it's just that it seems the echo pin failed to get an impulse from the trigger ultrasonic sound, so it keeps on getting False (LOW reading) and thus giving me false distance reading. Does anyone have a solution for this problem?
import pyfirmata
import time
board = pyfirmata.Arduino('COM16')
start = 0
end = 0
echo = board.get_pin('d:11:i')
trig = board.get_pin('d:12:o')
LED = board.get_pin('d:13:o')
it = pyfirmata.util.Iterator(board)
it.start()
trig.write(0)
time.sleep(2)
while True:
time.sleep(0.5)
trig.write(1)
time.sleep(0.00001)
trig.write(0)
print(echo.read())
while echo.read() == False:
start = time.time()
while echo.read() == True:
end = time.time()
TimeElapsed = end - start
distance = (TimeElapsed * 34300) / 2
print("Measured Distance = {} cm".format(distance) )
I've tried changing the time.sleep() to several value and it still doesn't work. It works just fine when I'm using Arduino code dirrectly from Arduino IDE.

I haven't done the exact math but given a range of 50cm you're at about 3ms travel time. That would mean you need to turn off the pulse and poll the pin state within that time.
That's not going to happen. The echo probably arrives befor you have turned off the emitter through PyFirmata. You should do the delay measurement on the Arduino.

I solve this false data problem by counting. I observe that false data comes after 2 or 3 sec. So if it takes More than 2 or 3 sec I clear count and restarts it from 0;
Sudo code:
cnt = 0;
if sensorvalue <= 20 && sensorvalue <= 30:
cnt++;
if cnt>=5:
detected = true;
cnt =0;
if cnt<5 && lastDecttime>2 (2 sec):
cnt = 0; // Here we handle the false value and clear the data

I'm currently trying to work this exact problem out. I can get the sensor to work using the Arduino IDE directly, but not with python and pyfirmata. I am getting some output, but its mostly non-sensical.
Here's an example output I'm getting, while keeping the sensor at the same distance from my object:
817.1010613441467
536.828875541687
0.0
546.0820078849792
0.0
0.0
1060.0213408470154
Regarding your code, the only thing I can see that you could do differently is to use the board.pass_time function instead of time.sleep(). Let me know if you get anywhere!
import pyfirmata as pyf
import time
def ultra_test():
board = pyf.Arduino("COM10")
it = pyf.util.Iterator(board)
it.start()
trigpin = board.get_pin("d:7:o")
echopin = board.get_pin("d:8:i")
while True:
trigpin.write(0)
board.pass_time(0.5)
trigpin.write(1)
board.pass_time(0.00001)
trigpin.write(0)
limit_start = time.time()
while echopin.read() != 1:
if time.time() - limit_start > 1:
break
pass
start = time.time()
while echopin.read() != 0:
pass
stop = time.time()
time_elapsed = stop - start
print((time_elapsed) * 34300 / 2)
board.pass_time(1)

Simple python script using 100% CPU on rPi3B+

I have raspberry pi model 3b+ with a HC-SR04 ultrasonic distance sensor (there is also a couple of ds18b20 and a DHT21 but I think they're unrelated to my problem).
I have found a python script to make measurements from it and modified it to my needs - mostly to take a couple of reading spanned in time, take an average from it and map the value to range from 0 to 100, as the percentage and commit it to the influx database for grafana and domoticz.
The code:
#source: https://tutorials-raspberrypi.com/raspberry-pi-ultrasonic-sensor-hc-sr04/
#Libraries
import RPi.GPIO as GPIO
import time
from influxdb import InfluxDBClient
import sys
# https://www.domoticz.com/wiki/Domoticz_API/JSON_URL%27s#Python
import requests
client = InfluxDBClient(database='pellet')
series = []
#GPIO Mode (BOARD / BCM)
GPIO.setmode(GPIO.BCM)
#set GPIO Pins
GPIO_TRIGGER = 23
GPIO_ECHO = 22
#set GPIO direction (IN / OUT)
GPIO.setup(GPIO_TRIGGER, GPIO.OUT)
GPIO.setup(GPIO_ECHO, GPIO.IN)
def distance():
# set Trigger to HIGH
GPIO.output(GPIO_TRIGGER, True)
# set Trigger after 0.01ms to LOW
time.sleep(0.00001)
GPIO.output(GPIO_TRIGGER, False)
StartTime = time.time()
StopTime = time.time()
# save StartTime
while GPIO.input(GPIO_ECHO) == 0:
StartTime = time.time()
# save time of arrival
while GPIO.input(GPIO_ECHO) == 1:
StopTime = time.time()
# time difference between start and arrival
TimeElapsed = StopTime - StartTime
# multiply with the sonic speed (34300 cm/s)
# and divide by 2, because there and back
distance = (TimeElapsed * 34300) / 2
return distance
def pellet(dist):
# zmierzona odleglosc
# dist = distance()
# do zmierzenia poziom maksymalny
# 63 - do pokrywy
in_min = 63
# do zmierzenia poziom minimalny
in_max = in_min + 100
#wyjscie jako procent, od 0 do 100
out_min = 100
out_max = 0
# map z arduino: https://www.arduino.cc/reference/en/language/functions/math/map/
return (dist - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
def loop():
# nie wiecej jak 200 iteracji
loop = 200
# suma
total = 0
# tabelka z pojedynczmi wynikami
measurements = []
# liczba pomiarow do zrobienia
counter = 10
counter1 = 0
# czas pomiedzy pomiarami
sleep =30
#
while loop > 0:
loop -= 1
time.sleep(sleep)
# koniec, jesli wykonano liczbe pomiarow
if counter == 0:
#print(total/10)
return pellet(total/10), measurements
break
if loop == 0 and counter1 != 0:
return pellet(total/counter1), measurements
break
if loop == 0 and (counter1 == 0 or total == 0):
GPIO.cleanup()
sys.exit()
dist = distance()
# jesli wynik jest zly
if dist < 63 or dist > 163:
print("nie ok")
continue
counter -= 1
measurements.append(dist)
counter1 += 1
total += dist
print("total po ",counter1 , "sek: ", total, "dist: ", dist)
print(total/10)
#return total/10
if __name__ == '__main__':
try:
#dist = distance()
#print ("Measured Distance = %.1f cm" % dist)
#print (pellet(dist))
loop=loop()
print("avg :", loop[0])
#print("measurs :", loop[1])
#print("test :", loop[1][2])
if (1):
point = {
"measurement": "pellet",
"tags": {
"location": "piwnica",
"type": "hc-sr04"
},
"fields": {
"value": loop[0],
"raw_measurement1": loop[1][0],
"raw_measurement2": loop[1][1],
"raw_measurement3": loop[1][2],
"raw_measurement4": loop[1][3],
"raw_measurement5": loop[1][4],
"raw_measurement6": loop[1][5],
"raw_measurement7": loop[1][6],
"raw_measurement8": loop[1][7],
"raw_measurement9": loop[1][8],
"raw_measurement10": loop[1][9]
}
}
series.append(point)
client.write_points(series)
url = 'http://localhost:8080/json.htm?type=command&param=udevice&nvalue=0&idx=13&svalue='+str(loop[0])
r = requests.get(url)
GPIO.cleanup()
# Reset by pressing CTRL + C
except KeyboardInterrupt:
print("Measurement stopped by User")
GPIO.cleanup()
The problem is I noticed that the CPU temperature graph was elevated, with many short valleys to the about correct temperature.
When I ssh'd to the pi and run htop I saw that it was this script that is using 100% cpu.
But the weirdest thing is that the script is running in crontab every 15 minutes since yesterday, from about 14:30 and raise CPU temp started today at around 11:00.
I'm not a developer or a programmer and I just mostly copied the code from around the web so I don't know if this is some part of the code that did this (but why after 21 hours?) or what and why, and how to debug and fix it.
so it isn't just enviromental thing as the pi is in the attic where is about 5C to 10C.
Thank you for your help.

Here:
while GPIO.input(GPIO_ECHO) == 0:
StartTime = time.time()
this says "if the pin is 0, save the time, if the pin is zero, save the time, if the pin...." incessantly. You'll want to wait a little time after each check
while GPIO.input(GPIO_ECHO) == 0:
time.sleep(0.001) # 1 ms
StartTime = time.time()
The check itself probably takes ~us, so this will reduce CPU usage by 99%. You might want to do the same for the pin==1 case, depending on how accurate you need the times to be.

While it is impossible to know for sure where the issue lies without debugging directly on your system, and a glance at the code reveals several possible bugs in the logic, the one place that is most likely to cause the issue is the distance function.
As #mdurant already pointed out, your read loops will jump the CPU usage to 100%, but I suspect there is also another issue:
The trigger code and the read code are time sensitive!
The problem is, we don't know how much time actually passes between
# set Trigger to HIGH
GPIO.output(GPIO_TRIGGER, True)
# set Trigger after 0.01ms to LOW
time.sleep(0.00001)
GPIO.output(GPIO_TRIGGER, False)
and:
# save StartTime
while GPIO.input(GPIO_ECHO) == 0:
StartTime = time.time()
# save time of arrival
while GPIO.input(GPIO_ECHO) == 1:
StopTime = time.time()
While this simple algorithm - pulse trigger, count return interval will work on a microcontroller like Arduino, it is not reliable on a full blown computer like Raspberry Pi.
Microcontrollers run a single thread, with no OS or task scheduling, so they run code in real time or as close to it as possible (borrowing a few interrupts here and there).
But in your case you are running an interpreted language on a multitasking operating system, without explicitly giving it any high priority.
This means, your process could be suspended just enough time to miss the return "ping" and get stuck in the first loop.
This may only happen rarely when something else puts a load on the Pi, which would explain why you only noticed the issue after 21 hours running.
You should implement some form of timeout for GPIO reading loops and return an error value from the distance function if that timeout is reached to ensure you do not have an infinite loop when something goes wrong with the hardware, or you miss the return ping due to scheduling issues.
I suggest something like this:
def distance():
MAX_READ_ATTEMPTS = 10000 #this is a random value I chose. You will need to fine-tune it based on actual hardware performance
# set Trigger to HIGH
GPIO.output(GPIO_TRIGGER, True)
# set Trigger after 0.01ms to LOW
time.sleep(0.00001)
GPIO.output(GPIO_TRIGGER, False)
# lets not have any unneeded code here, just in case
# save StartTime
while GPIO.input(GPIO_ECHO) == 0 and retry_counter < MAX_READ_ATTEMPTS:
retry_counter += 1
StartTime = time.time()
# maximum number of retries reached, returning with error condition
if retry_counter == MAX_READ_ATTEMPTS:
return -1
reatry_counter = 0
# save time of arrival
while GPIO.input(GPIO_ECHO) == 1 and retry_counter < MAX_READ_ATTEMPTS:
retry_counter += 1
StopTime = time.time()
# maximum number of retries reached, returning with error condition
if retry_counter == MAX_READ_ATTEMPTS:
return -1
# time difference between start and arrival
TimeElapsed = StopTime - StartTime
# multiply with the sonic speed (34300 cm/s)
# and divide by 2, because there and back
distance = (TimeElapsed * 34300) / 2
return distance
I am intentionally not adding a delay between reads, because I am not sure what the tolerance for this measurement is in terms of timing, and sleep functions can't guarantee an exact delay (again, due to OS / CPU scheduling).
A brief 100% CPU load should be worth it to ensure accurate and valid measurements, as long as it is not kept up for too long, which our retry counting method should prevent.
Note that my only experience with ultrasonic sensors is using Arduino where a special pulseIn function is used that takes care of the implementation details of measurement so this solution is mostly an educated guess and I have no way of testing it.

start reading incoming data from a serial port, using pyserial, when a specific character appears

I'm trying to read incoming data from a weight scale (Lexus Matrix One). I want the code to start reading the 8 characters after = appears.
The problem is that sometimes the code does that, and other times it starts reading the data at the middle of the measurement sent by the scale, making it impossible to read properly. I'm using the pyserial module on python 3 on windows.
import serial
ser=serial.Serial("COM4", baudrate=9600)
a=0
while a<10:
b=ser.read(8)
a=a+1
print(b)
the expected result is: b'= 0004.0'
but sometimes I get: b'4.0= 000'

I think we would need a little more information on the format of data coming from your weight scale to provide a complete answer. But your current code only reads the first 80 bytes from the stream, 8 bytes at a time.
If you want to read the next 8 bytes following any equals sign, you could try something like this:
import serial
ser = serial.Serial("COM4", baudrate=9600)
readings = []
done = False
while not done:
current_char = ser.read()
# check for equals sign
if current_char == b'=':
reading = ser.read(8)
readings.append(reading)
# this part will depend on your specific needs.
# in this example, we stop after 10 readings
# check for stopping condition and set done = True
if len(readings) >= 10:
done = True

How to quickly read single byte serial data using Python

I am reading serial data in Python using the following code:
port = "COM11"
baud = 460800
timeout=1
ser = serial.Serial()
ser.port = port
ser.baudrate = baud
ser.timeout = timeout
while 1:
# Read from serial port, blocking
data =ser.read(1)
print data
# some further processing of data
I am sending data at very fast rate but when I use this code I am getting data at a very slow rate, maybe around 2 - 3 data per second. This is too slow because I want to do real time plotting.
So, instead of above code I tried:
while 1:
# Read from serial port, blocking
data =ser.read(1)
data1=(data)
# If there is more than 1 byte, read the rest
n = ser.inWaiting()
data1 = (data1 + ser.read(n))
print data1
Now the speed at which data is updated is the same but instead of a single byte I am checking a number of bytes in input queue and reading them. I am receiving around 3850 bytes per loop so this one appears much faster to me but in fact it is almost the same, the only change is that I am not reading a greater number of bytes.
I want to read a single byte and check for the real time it was received. To do so I cannot use second method where I use ser.inWaiting(). How can I read single byte data faster than using the approaches above?

Here's some test code I wrote for a project that you can try different baud settings with. Basically it sends out some data on Tx (which could be connected directly to Rx) and expects the data to be echoed back. It then compares the returned data with the sent data and lets you know if/when errors occur. Note that if there are no errors then the output will remain blank and at the end of test it will print "0 Comm Errors".
import serial, time
test_data = "hello this is so freakin cool!!!" + '\r' #Must always be terminated with '\r'
echo_timeout = 1 #the time allotted to read back the test_data string in seconds
cycleNum = 0
errors = 0
try:
ser = serial.Serial(port="COM1", baudrate=115200, timeout=1)
ser.flush()
print "starting test"
for x in xrange(100):
cycleNum += 1
d = ser.write(test_data)
ret_char = returned = ''
start_time = time.time()
while (ret_char <> '\r') and (time.time() - start_time < echo_timeout):
ret_char = ser.read(1)
returned += ret_char
if not returned == test_data:
errors += 1
print "Cycle: %d Sent: %s Received: %s" % (cycleNum, repr(test_data), repr(returned) )
except Exception as e:
print 'Python Error:', e
finally:
if 'ser' in locals():
print "%d Comm Errors" % errors
if ser.isOpen():
ser.close()
print "Port Was Successfully Closed"
else:
print "Port Already Closed"
else:
print "Serial Variable Was Never Initialized"