We Keep Coding

How to properly read most recent value from serial?

I am reading values from a pressure sensing mat which has 32x32 individual pressure points. It outputs the readings on serial as 1024 bytes between 1 and 250 + 1 'end token' byte which is always 255 (or xFF).
I thought the function bellow would flush/reset the input buffer and then take a 'fresh' reading and return the max pressure value from that reading whenever I call it.
However, none of the ser.reset_input_buffer() and similar methods seem to actually empty the buffer. When I press down on the mat, run the program and immediately release the pressure, I don't see the max value drop immediately. Instead, it seems to be going through the buffer one by one.
import serial
import numpy as np
import time
def read_serial():
ser_bytes = bytearray([0])
# none of these seem to make a differece
ser.reset_input_buffer()
ser.flushInput()
ser.flush()
# 2050 bytes should always contain a whole chunk of 1025 bytes ending with 255 (xFF)
while len(ser_bytes) <= 2050:
ser_bytes = ser_bytes + ser.read_until(b'\xFF')
ser_ints = np.array(ser_bytes, dtype='int32') #bytes to ints
last_end_byte_index = np.max( np.where(ser_ints == 255) ) #find the last end byte
# get only the 1024 sensor readings as 32x32 np array
mat_reading = np.array( ser_ints[last_end_byte_index-1024: last_end_byte_index]).reshape(32,32)
return np.amax(mat_reading)
ser = serial.Serial('/dev/tty.usbmodem14201', 115200, timeout=1)
while True:
print(read_serial())
time.sleep(1)
The best solution I found so far is having a designated thread which keeps reading the buffer and updating a global variable. It works but seems a bit unresourceful if I only want to read the value about every 60 seconds. Is there a better way?
Also... is there a better way to read the 1025-byte chunk representing the entire mat? There are no line breaks, so ser.readline() won't work.
Thanks! Not sure how to make an MWE with serial, sorry about that ;)

unable to unpack information between custom Preamble in Python and telnetlib

I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)

Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.

I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.

Python: Can I read two bytes in one transaction with the smbus module?

I'm trying to read the temperature and humidity using a Texas Instruments HDC1008 from Adafruit, product 2635. I'm on a rasberry pi 2, using the smbus module. According to TI's PDF, when getting a reading, the number will be sent in two bytes that you put together. I found this code that does what I'm trying to do with micropython, where they have a recv function that seems to simply sends them back a list with two bytes. The SMBus module doesn't seem to have any equivalent for what I'm trying to do. Here's some of my code.
class HDC1008:
I2C_BUS = 1
#Registers
REG_TEMP = 0
REG_HUMID = 1
REG_CONFIG = 2
#Configuration bits
CFG_RST = 1<<15
CFG_MODE_SINGLE = 0 << 12
CFG_MODE_BOTH = 1 << 12
ADDRESS = 0x40
def __init__(self, bus_num=I2C_BUS):
self.bus=smbus.SMBus(bus_num)
def readTemperature(self):
#configure the HDC1008 for one reading
config = 0
config |= self.CFG_MODE_SINGLE
self.bus.write_byte_data(self.ADDRESS, self.REG_CONFIG, config)
#tell the thing to take a reading
self.bus.write_byte(self.ADDRESS, self.REG_TEMP)
time.sleep(0.015)
#get the reading back from the thing
raw = self.bus.read_byte(self.ADDRESS)
raw = (raw<<8) + self.bus.read_byte(self.ADDRESS)
#use TI's formula to turn it into people numbers
temperature = (raw/65536.0)*165.0 - 40
#convert temp to f
temperature = temperature * (9.0/5.0) + 32
return temperature
When I'm getting the value for raw from bus.read_byte, I'm able to get the first half of the temperature bits, but the second reading is just zeros, presumably because the first transaction is over. How do I get two bytes in one transaction?

tnx a lot for sharing this code. I'm happy to get it working in Python.
I do not exactly understand the problem, I can read the Temperature and Humidity with our code (the only Python code I could find and works)
I did change it a little bit (make it a Class):
import smbus
class HDC:
#Registers
REG_TEMP = 0
REG_HUMID = 1
REG_CONFIG = 2
I2C_BUS = 2 #2 for PCDuino, 1 for PI
#Configuration bits
CFG_RST = 1<<15
CFG_MODE_SINGLE = 0 << 12
CFG_MODE_BOTH = 1 << 12
ADDRESS = 0x40
def __init__(self, bus_num=I2C_BUS):
self.bus=smbus.SMBus(bus_num)
def readTemperature(self):
#configure the HDC1008 for one reading
config = 0
config |= self.CFG_MODE_SINGLE
self.bus.write_byte_data(self.ADDRESS, self.REG_CONFIG, config)
#tell the thing to take a reading
self.bus.write_byte(self.ADDRESS, self.REG_TEMP)
time.sleep(0.015)
#get the reading back from the thing
raw = self.bus.read_byte(self.ADDRESS)
raw = (raw<<8) + self.bus.read_byte(self.ADDRESS)
#use TI's formula to turn it into people numbers
temperature = (raw/65536.0)* 165 - 40
#convert temp to farenheid
#temperature = temperature * (9.0/5.0) + 32
return temperature
def readHum(self):
#configure the HDC1008 for one reading
config = 0
config |= self.CFG_MODE_SINGLE
self.bus.write_byte_data(self.ADDRESS, self.REG_CONFIG, config)
#tell the thing to take a reading
self.bus.write_byte(self.ADDRESS, self.REG_HUMID)
time.sleep(0.015)
#get the reading back from the thing
raw = self.bus.read_byte(self.ADDRESS)
raw = (raw<<8) + self.bus.read_byte(self.ADDRESS)
hum=(raw/(2.0**16))*100
return hum
In the program:
from hdc1008 import HDC
HDC1008=HDC()
print HDC1008.readTemperature()
print HDC1008.readHum()

As I can distinguish different data from Serial?

I trying to read by serial differents values, but i dont know hoy to split that, because the two values are numbers but from different source
First i have a PICAXE sending converted data by ADC of light sensor by serial to python.
Second i have a PICAXE sending data of temperature sensor by serial to python.
Light code PICAXE
symbol puerto = B.5
main: readadc10 puerto,w1 ; read value into w1
sertxd(#w1,cr,lf)
goto main ; loop back to start
Temp code PICAXE
symbol temp = B.4
readtemp temp, w0 ; read value into w1
debug
sertxd(#w0,cr,lf)
goto main
Python code
import pygame
import sys, serial
from pygame.locals import *
ser = serial.Serial()
ser.port = 3
ser.baudrate = 4800
while True:
datos = ser.readline()
grados = float(datos)
print grados
The problem is that picaxe send simultaneus data from light and temp, but when python receive data, i dont know how to recognized each data.
Anyone can help me??
Thank!

If you have a temperature reading and a light level reading to send at the same time, you could put them on one line separated by a space.
PICAXE:
sertxd(#w0," ",#w1,cr,lf)
Python:
readings = ser.readline()
[reading1, reading2] = readings.split()
temperature = float(reading1)
lightlevel = float(reading2)
If the two types of reading are produced irregularly, you could transmit a character before each one to identify what type it is.
PICAXE:
sertxd("T ",#w0,cr,lf)
...
sertxd("L ",#w1,cr,lf)
Python:
reading = ser.readline()
[readingtype, readingvalue] = reading.split()
if readingtype == "T":
temperature = float(readingvalue)
elif readingtype == "L":
lightlevel = float(readingvalue)

Read latest character sent from Arduino in Python

I'm a beginner in both Arduino and Python, and I have an idea but I can't get it to work. Basically, when in Arduino a button is pressed, it sends "4" through the serial port. What I want in Python is as soon as it reads a 4, it should do something. This is what I got so far:
import serial
ser = serial.Serial('/dev/tty.usbserial-A900frF6', 9600)
var = 1
while var == 1:
if ser.inWaiting() > 0:
ser.readline(1)
print "hello"
But obviously this prints hello no matter what. What I would need is something like this:
import serial
ser = serial.Serial('/dev/tty.usbserial-A900frF6', 9600)
var = 1
while var == 1:
if ser.inWaiting() > 0:
ser.readline(1)
if last.read == "4":
print "hello"
But how can I define last.read?

I don't know a good way of synchronising the comms with readLine since it's not a blocking call. You can use ser.read(numBytes) which is a blocking call. You will need to know how many bytes Arduino is sending though to decode the byte stream correctly. Here is a simple example that reads 8 bytes and unpacks them into 2 unsigned shorts and a long (the <HHL part) in Python
try:
data = [struct.unpack('<HHL', handle.read(8)) for i in range(PACKETS_PER_TRANSMIT)]
except OSError:
self.emit(SIGNAL("connectionLost()"))
self.connected = False
Here's a reference to the struct.unpack()
The Arduino code that goes with that. It reads two analog sensor values and the micro timestamp and sends them over the serial.
unsigned int SensA, SensB;
byte out_buffer[64];
unsigned int buffer_head = 0;
unsigned int buffer_size = 64;
SensA = analogRead(SENSOR_A);
SensB = analogRead(SENSOR_B);
micr = micros();
out_buffer[buffer_head++] = (SensA & 0xFF);
out_buffer[buffer_head++] = (SensA >> 8) & 0xFF;
out_buffer[buffer_head++] = (SensB & 0xFF);
out_buffer[buffer_head++] = (SensB >> 8) & 0xFF;
out_buffer[buffer_head++] = (micr & 0xFF);
out_buffer[buffer_head++] = (micr >> 8) & 0xFF;
out_buffer[buffer_head++] = (micr >> 16) & 0xFF;
out_buffer[buffer_head++] = (micr >> 24) & 0xFF;
Serial.write(out_buffer, buffer_size);
The Arduino playground and Processing Forums are good places to look around for this sort of code as well.
UPDATE
I think I might have misled you with readLine not blocking. Either way, the above code should work. I also found this other thread on SO regarding the same subject.
UPDATE You don't need to use the analog sensors, that's just what the project I did happened to be using, you are of course free to pass what ever values over the serial. So what the Arduino code is doing is it has a buffer of type byte where the output is being stored before being sent. The sensor values and micros are then written to the buffer and the buffer sent over the serial. The (SensA & 0xFF) is a bit mask operator that takes the bit pattern of the SensA value and masks it with the bit pattern of 0xFF or 255 in decimal. Essetianlly this takes the first 8 bits from the 16 bit value of SensA which is an Arduino short. the next line does the same thing but shifts the bits right by 8 positions, thus taking the last 8 bits.
You'll need to understand bit patterns, bit masking and bit shifting for this. Then the buffer is written to the serial.
The Python code in turn does reads the bits from the serial port 8 bits at a time. Have a look at the struct.unpack docs. The for comprehension is just there to allow sending more than one set of values. Because the Arduino board and the Python code are running out of sync I added that to be able to send more than one "lines" per transmit. You can just replace that with struct.unpack('<HHL',handle.read(8)). Remember that the ´handle.read()´ takes a number of bytes where as the Arduino send code is dealing with bits.

I think it might work with this modifications:
import serial
ser = serial.Serial('/dev/tty.usbserial-A900frF6', 9600)
var = 1
while var == 1:
if (ser.inWaiting() > 0):
ser.readline(1)
print "hello"