We Keep Coding

Read multiple DS18B20 temperature sensors faster using Raspberry Pi

My custom sensor dashboard requests new readings every second.
This worked well, until I hooked up 3 DS18B20 temperature sensors (1-wire protocol, so all on 1 pin), which each take 750ms to provide new data.
This is the class I currently use to read the temperature of each sensor:
# ds18b20.py
# written by Roger Woollett
import os
import glob
import time
class DS18B20:
# much of this code is lifted from Adafruit web site
# This class can be used to access one or more DS18B20 temperature sensors
# It uses OS supplied drivers and one wire support must be enabled
# To do this add the line
# dtoverlay=w1-gpio
# to the end of /boot/config.txt
#
# The DS18B20 has three pins, looking at the flat side with the pins pointing
# down pin 1 is on the left
# connect pin 1 to GPIO ground
# connect pin 2 to GPIO 4 *and* GPIO 3.3V via a 4k8 (4800 ohm) pullup resistor
# connect pin 3 to GPIO 3.3V
# You can connect more than one sensor to the same set of pins
# Only one pullup resistor is required
def __init__(self):
# Load required kernel modules
os.system('modprobe w1-gpio')
os.system('modprobe w1-therm')
# Find file names for the sensor(s)
base_dir = '/sys/bus/w1/devices/'
device_folder = glob.glob(base_dir + '28*')
self._num_devices = len(device_folder)
self._device_file = list()
i = 0
while i < self._num_devices:
self._device_file.append(device_folder[i] + '/w1_slave')
i += 1
def _read_temp(self, index):
# Issue one read to one sensor
# You should not call this directly
# First check if this index exists
if index >= len(self._device_file):
return False
f = open(self._device_file[index], 'r')
data = f.read()
f.close()
return data
def tempC(self, index=0):
# Call this to get the temperature in degrees C
# detected by a sensor
data = self._read_temp(index)
retries = 0
# Check for error
if data == False:
return None
while (not "YES" in data) and (retries > 0):
# Read failed so try again
time.sleep(0.1)
#print('Read Failed', retries)
data = self._read_temp(index)
retries -= 1
if (retries == 0) and (not "YES" in data):
return None
(discard, sep, reading) = data.partition(' t=')
if reading == 85000:
# 85ºC is the boot temperature of the sensor, so ignore that value
return None
temperature = float(reading) / 1000.0
return temperature
def device_count(self):
# Call this to see how many sensors have been detected
return self._num_devices
I already tried to return the previous temperature reading if the current one isn't finished yet, however this didn't reduce the time it took to read a sensor, so I guess the only way is to do things asynchronously.
I could reduce the precision to reduce the time it takes per reading, but ideally I would read all of the sensors simultaneously on separate threads.
How can I best implement this? Or are there other ways to improve the reading speed of multiple DS18B20 sensors?
Thanks for any insights!

You're facing some limitations introduced by the Linux kernel driver. If you were interacting with the OneWire protocol directly, you would only have a single 750ms read cycle for all three sensors, rather than (3 * 750ms). When speaking the 1-wire protocol directly, you can issue a single "convert temperature" command to all devices on the bus, as described here, and then read all the sensors.
The Linux driver explicitly doesn't support this mode of operation:
If none of the devices are parasite powered it would be possible to convert all the devices at the same time and then go back to read individual sensors. That isn’t currently supported. The driver also doesn’t support reduced precision (which would also reduce the conversion time) when reading values.
That means you're stuck with a 750ms per device read cycle. Your best option is probably placing the sensor reading code in a separate thread, e.g.:
import glob
import threading
import time
# Note that we're inheriting from threading.Thread here;
# see https://docs.python.org/3/library/threading.html
# for more information.
class DS18B20(threading.Thread):
default_base_dir = "/sys/bus/w1/devices/"
def __init__(self, base_dir=None):
super().__init__()
self._base_dir = base_dir if base_dir else self.default_base_dir
self.daemon = True
self.discover()
def discover(self):
device_folder = glob.glob(self._base_dir + "28*")
self._num_devices = len(device_folder)
self._device_file: list[str] = []
for i in range(self._num_devices):
self._device_file.append(device_folder[i] + "/w1_slave")
self._values: list[float | None] = [None] * self._num_devices
self._times: list[float] = [0.0] * self._num_devices
def run(self):
"""Thread entrypoint: read sensors in a loop.
Calling DS18B20.start() will cause this method to run in
a separate thread.
"""
while True:
for dev in range(self._num_devices):
self._read_temp(dev)
# Adjust this value as you see fit, noting that you will never
# read actual sensor values more often than 750ms * self._num_devices.
time.sleep(1)
def _read_temp(self, index):
for i in range(3):
with open(self._device_file[index], "r") as f:
data = f.read()
if "YES" not in data:
time.sleep(0.1)
continue
disacard, sep, reading = data.partition(" t=")
temp = float(reading) / 1000.0
self._values[index] = temp
self._times[index] = time.time()
break
else:
print(f"failed to read device {index}")
def tempC(self, index=0):
return self._values[index]
def device_count(self):
"""Return the number of discovered devices"""
return self._num_devices
Because this is a thread, you need to .start() it first, so your
code would look something like:
d = DS18B20()
d.start()
while True:
for i in range(d.device_count()):
print(f'dev {i}: {d.tempC(i)}')
time.sleep(0.5)
You can call the tempC method as often as you want, because it's
just return a value from the _values array. The actual update
frequency is controlled by the loop in the run method (and the
minimum cycle time imposed by the sensors).

Amending Apogee sensor open-source Python 2 code to operate on Raspberry pi 4 through USB port

Working on a project to connect several sensors to Raspberry Pi 4 Model B and stream collected data to Google Cloud Platform for storage/reporting/analysis
I have an Apogee sq520 PAR sensor which connects through USB and is designed to operate on Windows and Mac (but not Linux) with an accompanying GUI to view the data. I've connected the sensor to the Raspberry PI through the USB port (since IO pins are out of the picture) and now need to write a program for the Pi to read it. After reaching out to the Apogee support team, they've shared with me the source-code (below). However, as I am a complete beginner I am having a hard time amending the code to be read by the Raspberry Pi. Need all the help and support I can get.
Notes:
Code utilizes Pyserial library
Code is written on Python 2.7 (according to website). - I am using Python 3
When compiling the code as-is I receive a lot of syntax errors, regardless whether compiling in Python 2 or 3
Sensor connects through 5V USB
More info on sensor: https://www.apogeeinstruments.com/sq-520-full-spectrum-smart-quantum-sensor-usb/#product-tab-description
Code:
from serial import Serial
from time import sleep
import struct
GET_VOLT = '\x55!'
READ_CALIBRATION = '\x83!'
SET_CALIBRATION = '\x84%s%s!'
READ_SERIAL_NUM = '\x87!'
GET_LOGGING_COUNT = '\xf3!'
GET_LOGGED_ENTRY = '\xf2%s!'
ERASE_LOGGED_DATA = '\xf4!'
class Quantum(object):
def __init__(self):
"""Initializes class variables, and attempts to connect to device"""
self.quantum = None
self.offset = 0.0
self.multiplier = 0.0
self.connect_to_device()
def connect_to_device(self):
"""This function creates a Serial connection with the defined comport
and attempts to read the calibration values"""
port = 'COM1' # you'll have to check your device manager and put the actual com port here
self.quantum = Serial(port, 115200, timeout=0.5)
try:
self.quantum.write(READ_CALIBRATION)
multiplier = self.quantum.read(5)[1:]
offset = self.quantum.read(4)
self.multiplier = struct.unpack('<f', multiplier)[0]
self.offset = struct.unpack('<f', offset)[0]
except (IOError, struct.Error), data:
print data
self.quantum = None
def get_micromoles(self):
"""This function converts the voltage to micromoles"""
voltage = self.read_voltage()
if voltage == 9999:
# you could raise some sort of Exception here if you wanted to
return
# this next line converts volts to micromoles
micromoles = (voltage - self.offset) * self.multiplier * 1000
if micromoles < 0:
micromoles = 0
return micromoles
def read_voltage(self):
"""This function averages 5 readings over 1 second and returns
the result."""
if self.quantum == None:
try:
self.connect_to_device()
except IOError:
# you can raise some sort of exception here if you need to
return
# store the responses to average
response_list = []
# change to average more or less samples over the given time period
number_to_average = 5
# change to shorten or extend the time duration for each measurement
# be sure to leave as floating point to avoid truncation
number_of_seconds = 1.0
for i in range(number_to_average):
try:
self.quantum.write(GET_VOLT)
response = self.quantum.read(5)[1:]
except IOError, data:
print data
# dummy value to know something went wrong. could raise an
# exception here alternatively
return 9999
else:
if not response:
continue
# if the response is not 4 bytes long, this line will raise
# an exception
voltage = struct.unpack('<f', response)[0]
response_list.append(voltage)
sleep(number_of_seconds/number_to_average)
if response_list:
return sum(response_list)/len(response_list)
return 0.0

Pyserial package doesnot read all the data from the COM port ( reads only 6000 to 6150 bytes always)

I wrote a small pyserial interface to read the data from the COM port after issuing a command. For eg : in my case my system has a lot of network interface so i need to validate whether all the interfaces are up using ifconfig command. But when i gave this command , the output of the command is getting truncated at the last few lines. The approximate size of the output in bytes would be 6500-7000 bytes but i am receiving only around 6000-6150 bytes all the time. Please find my code below
'''
import serial
import time
com_serial = serial.Serial("COM6", 115200, timeout = 10)
com_serial.reset_input_buffer()
com_serial.write(b"ifconfig\n")
data_all = b" "
time.sleep(5)
while True:
bytetoread = com_serial.inWaiting()
time.sleep(2)
print ("Bytetoread: " , bytetoread)
data = com_serial.read(bytetoread)
data_all += data
if bytetoread < 1:
break
print ("Data:", data_all)
com_serial.close()
'''
**Output:
Bytetoread: 3967
Bytetoread: 179
Bytetoread: 2049
Bytetoread: 0
**
Data: *********with missing data at the end.
I am not sure why the logs are missing?
I have tried another approach.
'''
import serial
import time
com_serial = serial.Serial("COM6", 115200, timeout = 10)
com_serial.reset_input_buffer()
com_serial.write(b"ifconfig\n")
time.sleep(5)
data_all = b" "
data_all = com_serial.read(100000000)
print (data_all)
com_serial.close()
'''
Here also the last few logs are getting truncated.

The root cause seems to be inadequate buffer size of the Tx and Rx serial buffer. By increasing the buffer size using .set_buffer_size() resolved the issue.
'''
import serial
import time
com_serial = serial.Serial("COM6", 115200, timeout = 10)
com_serial.set_buffer_size(rx_size = 12800, tx_size = 12800)
com_serial.reset_input_buffer()
com_serial.write(b"ifconfig\n")
data_all = b" "
data_all = com_serial.read(100000000)
print (data_all)
com_serial.close()
'''

Micro Switch with pyserial RS232 starts/stops a timer in a tkinter thread but continues to run even when stopped

I have been using a micro switch connected to an RS232/USB serial converter cable on my windows PC to start stop and reset a timer.
The program runs smoothly most of the time but every so often updating the timer widget gets stuck running and the timer will not stop.
With the serial protocol i want to receive 1 byte b'\x00' for off and anything that's not b'\x00' should signify on.
I have replaced the micro switch with button widgets to simulate the switch and don't get the same error or i just have not kept at it for long enough.
It could be an issue with the RS232 causing an error i cannot see but my knowledge on this is sketchy and have exhausted all avenues looking online for any information on this.
import time
import sys
import serial
import threading
from tkinter import *
from tkinter import ttk
class Process(Frame):
def __init__(self, root, parent=None, **kw):
Frame.__init__(self, parent, kw)
self.root = root
self._cycStart = 0.0
self._cycTimeElapsed = 0.0
self._cycRunning = 0.0
self.cycTimeStr = StringVar()
self.cycTime_label_widget()
self.ser = serial.Serial(
port='COM4',
baudrate=1200,
timeout=0
)
self.t1 = threading.Thread(target=self.start_stop, name='t1')
self.t1.start()
def initUI(self):
root.focus_force()
root.title("")
root.bind('<Escape>', lambda e: root.destroy())
def cycTime_label_widget(self):
# Make the time label
cycTimeLabel = Label(root, textvariable=self.cycTimeStr, font=
("Ariel 12"))
self._cycleSetTime(self._cycTimeElapsed)
cycTimeLabel.place(x=1250, y=200)
cycTimeLabel_2 = Label(root, text="Cycle Timer:", font=("Ariel
12"))
cycTimeLabel_2.place(x=1150, y=200)
def _cycleUpdate(self):
""" Update the label with elapsed time. """
self._cycTimeElapsed = time.time() - self._cycStart
self._cycleSetTime(self._cycTimeElapsed)
self._cycTimer = self.after(50, self._cycleUpdate)
def _cycleSetTime(self, elap):
""" Set the time string to Minutes:Seconds:Hundreths """
minutes = int(elap/60)
seconds = int(elap - minutes*60.0)
hseconds = int((elap - minutes*60.0 - seconds)*100)
self.cycTimeStr.set('%02d:%02d:%02d' % (minutes, seconds,
hseconds))
return
def cycleStart(self):
""" Start the stopwatch, ignore if running. """
if not self._cycRunning:
self._cycStart = time.time() - self._cycTimeElapsed
self._cycleUpdate()
self._cycRunning = 1
else:
self.cycleReset()
def cycleStop(self):
""" Stop the stopwatch, ignore if stopped. """
if self._cycRunning:
self.after_cancel(self._cycTimer)
self._cycTimeElapsed = time.time() - self._cycStart
self._cycleSetTime(self._cycTimeElapsed)
self._cycRunning = 0
self._cycTimeElapsed = round(self._cycTimeElapsed, 1)
self.cycleTimeLabel = Label(root, text=(self._cycTimeElapsed,
"seconds"), font=("Ariel 35"))
self.cycleTimeLabel.place(x=900, y=285)
self.cycleReset()
def cycleReset(self):
""" Reset the stopwatch. """
self._cycStart = time.time()
self._cycTimeElapsed = 0
self._cycleSetTime(self._cycTimeElapsed)
def start_stop(self):
while True :
try:
data_to_read = self.ser.inWaiting()
if data_to_read != 0: # read if there is new data
data = self.ser.read(size=1).strip()
if data == bytes(b'\x00'):
self.cycleStop()
print("Off")
elif data is not bytes(b'\x00'):
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
if __name__ == '__main__':
root = Tk()
application = Process(root)
root.mainloop()
I expect the timer to start running when the micro switch is pressed. when depressed it should stop and reset back to zero and wait for the next press

With a better understanding of what you're trying to do better solutions come to mind.
As it turns out, you're not using your serial port to send or receive serial data. What you're actually doing is wiring a switch to its RX line and toggling it manually with a mechanical switch, feeding a high or low level depending on the position of the switch.
So what you're trying to do is emulating a digital input line with the RX line of your serial port. If you take a look a how a serial port works you'll see that when you send a byte the TX line toggles from low to high at the baud rate, but on top of the data you have to consider the start and stop bits. So, why your solution works (at least sometimes): that's easy to see when you look at a scope picture:
This is a screenshot of the TX line sending the \x00 byte, measured between pins 3 (TX) and 5 (GND) with no parity bit. As you can see the step only lasts for 7.5 ms (with a 1200 baud rate). What you are doing with your switch is something similar but ideally infinitely long (or until you toggle your switch back, which will be way after 7.5 ms no matter how fast you do it). I don't have a switch to try but if I open a terminal on my port and use a cable to shortcircuit the RX line to pin 4 (on a SUB-D9 connector) sometimes I do get a 0x00 byte, but mostly it's something else. You can try this experiment yourself with PuTTy or RealTerm and your switch, I guess you'll get better results but still not always the byte you expect because of the contacts bouncing.
Another approach: I'm sure there might be ways to improve on what you have, maybe reducing the baud rate to 300 or 150 bps, checking for a break in the line or other creative ideas.
But what you're trying to do is more akin to reading a GPIO line, and actually, the serial port has several digital lines intended (in the old days) for flow control.
To use these lines you should connect the common pole on your switch to the DSR line (pin 6 on a SUB-D9) and the NO and NC poles to lines DTR (pin 4) and RTS (pin 7).
The software side would be actually simpler than reading bytes: you just have to activate hardware flow control :
self.ser = serial.Serial()
self.ser.port='COM4'
self.ser.baudrate=1200 #Baud rate does not matter now
self.ser.timeout=0
self.ser.rtscts=True
self.ser.dsrdtr=True
self.ser.open()
Define the logical levels for your switch:
self.ser.setDTR(False) # We use DTR for low level state
self.ser.setRTS(True) # We use RTS for high level state
self.ser.open() # Open port after setting everything up, to avoid unkwnown states
And use ser.getDSR() to check the logical level of the DSR line in your loop:
def start_stop(self):
while True :
try:
switch_state = self.ser.getDSR()
if switch_state == False and self._cycRunning == True:
self.cycleStop()
print("Off")
elif switch_state == True and self._cycRunning == False:
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")
I defined your self._cycRunning variable as boolean (in your initialization code you had defined it as float, but that was probably a typo).
This code works with no glitches at all even using a stripped wire as a switch.

You don't explain very well how your protocol works (I mean what is your switch supposed to be sending, or if it's sending a state change only once or several times or continuously).
But there are some red flags on your code anyway:
-With data = self.ser.read(size=1).strip() you read 1 byte but immediately you check if you have received 2 bytes. Is there a reason to do that?
-Your timer stop condition works comparing with the NULL character. That should not be a problem, but depending on your particular configuration it might (in some configurations the NULL character is read as something else, so it's wise to make sure you're really receiving it correctly).
-Your timer start condition seems too loose. Whatever you receive on the port, if it's one byte, you start your timer. Again, I don't know if that's the way your protocol works but it seems prone to trouble.
-When you replace your hardware switch with a software emulation it works as intended, but that is not surprising since you're probably imposing the condition. When you read from the serial port you have to deal with real world issues like noise, communication errors or the switch bouncing back and forth from ON to OFF. Maybe for a very simple protocol you don't need to use any error checking method, but it seems wise to at least check for parity errors. I'm not completely sure it would be straight-forward to do that with pyserial; on a quick glance I found this issue that's been open for a while.
-Again, the lack of info on your protocol: should you be using XON-XOFF flow control and two stop bits? I guess you have a reason to do it, but you should be very aware of why and how you're using those.
EDIT: With the comments below I can try to improve a bit my answer. This is just an idea for you to develop: instead of making the stop condition comparing exactly with 0x00 you can count the number of bits set to 1 and stop the counter if it's less or equal to 2. That way you can account for bits that are not received correctly.
You can do the same with the start condition but I don't know what hex value you send.
Credits for the bit counting function go to this question.
...
def numberOfSetBits(i):
i = i - ((i >> 1) & 0x55555555)
i = (i & 0x33333333) + ((i >> 2) & 0x33333333)
return (((i + (i >> 4) & 0xF0F0F0F) * 0x1010101) & 0xffffffff) >> 24
def start_stop(self):
while True :
try:
data_to_read = self.ser.inWaiting()
if data_to_read != 0: # read if there is new data
data = self.ser.read(size=1).strip()
if numberOfSetBits(int.from_bytes(data, "big")) <= 2:
self.cycleStop()
print("Off")
elif numberOfSetBits(int.from_bytes(data, "big")) >= 3: #change the condition here according to your protocol
self.cycleStart()
print("On")
except serial.SerialException as e:
print("Error")

Tornado Server using most of the cpu while using tornado-sockjs and only two clients.

I am using Tornado Server, 4.4.2 and pypy 5.9.0 and python 2.7.13,
hosted on Ubuntu 16.04.3 LTS
A new client logs in and a new class is created and passed the socket, so dialog can be maintained. I am using a global clients[] list to contain the classes. initial dialog looks like :
clients = []
class RegisterWebSocket(SockJSConnection):
# intialize the class and handle on-open (some things left out)
def on_open(self,info):
self.ipaddress = info.headers['X-Real-Ip']
def on_message(self, data):
coinlist = []
msg = json.loads(data)
if 'coinlist' in msg:
coinlist = msg['coinlist']
if 'currency' in msg:
currency = msg['currency']
tz = pendulum.timezone('America/New_York')
started = pendulum.now(tz).to_day_datetime_string()
ws = WebClientUpdater(self, self.clientid, coinlist,currency,
started, self.ipaddress)
clients.append(ws)
The ws class is shown below and I use a tornado periodiccallback to update the clients with their specific info every 20 seconds
class WebClientUpdater(SockJSConnection):
def __init__(self, ws,id, clist, currency, started, ipaddress):
super(WebClientUpdater,self).__init__(ws.session)
self.ws = ws
self.id = id
self.coinlist = clist
self.currency = currency
self.started = started
self.ipaddress = ipaddress
self.location = loc
self.loop = tornado.ioloop.PeriodicCallback(self.updateCoinList,
20000, io_loop=tornado.ioloop.IOLoop.instance())
self.loop.start()
self.send_msg('welcome '+ id)
def updateCoinList(self):
pdata = db.getPricesOfCoinsInCurrency(self.coinlist,self.currency)
self.send(dict(priceforcoins = pdata))
def send_msg(self,msg):
self.send(msg)
I also start at 60 second periodiccallback at startup, to monitor the clients for closed connections and remove them from the client[] list. Which I put on the startup line to call a def internally like
if __name__ == "__main__":
app = make_app()
app.listen(options.port)
ScheduleSocketCleaning()
and
def ScheduleSocketCleaning():
def cleanSocketHouse():
print "checking sockets"
for x in clients:
if x.is_closed:
x = None
clients[:] = [y for y in clients if not y.is_closed ]
loop = tornado.ioloop.PeriodicCallback(cleanSocketHouse, 60000,
io_loop=tornado.ioloop.IOLoop.instance())
loop.start()
If I monitor the server using TOP I see that it uses 4% cpu typical with bursts to 60+ immediately, but later, say after a few hours it becomes in the 90% and stays there.
I have used strace and I see an enormous amount of Stat calls on the same files with errors shown in the strace -c view, but I cannot find any errors in a text file using -o trace.log. How can I find those errors ?
But I also notice that most of the time is consumed in epoll_wait.
%time
41.61 0.068097 7 9484 epoll_wait
26.65 0.043617 0 906154 2410 stat
15.77 0.025811 0 524072 read
10.90 0.017840 129 138 brk
2.41 0.003937 9 417 madvise
2.04 0.003340 0 524072 lseek
0.56 0.000923 3 298 sendto
0.06 0.000098 0 23779 gettimeofday
100.00 0.163663 1989527 2410 total
Notice 2410 errors above.
When I view the strace output stream using attached pid, I just see endless Stat calls on the same files..
Can someone advise me as to how to better debug this situation? With only two clients and 20 seconds between client updates, I would expect the CPU usage (there are no other users of the site during this prototype stage) would be less than 1% or thereabouts.

You need to close PeriodicCallbacks, otherwise its a memory leak. You do that by simply calling .close() on a PeriodicCallback object. One way to deal with that is in your periodic cleaning task:
def cleanSocketHouse():
global clients
new_clients = []
for client in clients:
if client.is_closed:
# I don't know why you call it loop,
# .timer would be more appropriate
client.loop.close()
else:
new_clients.append(client)
clients = new_clients
I'm not sure how accurate .is_closed is (some testing is required). The other way is to alter updateCoinList. The .send() method should fail when the client is no longer connected, right? Therefore try: except: should do the trick:
def updateCoinList(self):
global clients
pdata = db.getPricesOfCoinsInCurrency(self.coinlist,self.currency)
try:
self.send(dict(priceforcoins = pdata))
except Exception:
# log exception?
self.loop.close()
clients.remove(self) # you should probably use set instead of list
If ,send() actually doesn't fail (for whatever reason, I'm not that familiar with Tornado) then stick to the first solution.