We Keep Coding

What is a faster way to read binary as uint8, create sliding window of uint32, and find indices where the uint32 value == x?

I have some binary data to parse, and need to find where packets start. All packets start with the same header, but packet size is variable. The header is a 32 bit unsigned integer.
Below is my implementation, but it's slow. Is there some numpy functionality or other options to make this operation faster?
"""Example of binary data:
d9 37 b2 a5 08 31 03 ... 46 00 00 01 b9 1e 43 ... d9 37 b2 a5 30 90 06 00 cb... 08 00 30 43 d9 37 b2 a5 ... 04 01 c8 f4 ...
"""
def sliding_window(iterable, n=2):
"""Return a zipped object where each item is a sliding group of n elements from iterable.
Example:
in = [1,2,3,4,5,6,7,8]
out = [[1,2,3,4],[2,3,4,5],[3,4,5,6],[4,5,6,7],[5,6,7,8]]
"""
iterables = itertools.tee(iterable, n)
for iterable, num_skipped in zip(iterables, itertools.count()):
for _ in range(num_skipped):
next(iterable, None)
return zip(*iterables)
packet_header = 0xd937b2a5
dat_file = "path/to/file"
dat = np.fromfile(file=dat_file,dtype = np.uint8)
sw_u8s = sliding_window(dat,4)
#this is really slow
sw_u32s = [struct.unpack('>I', bytes(bb)) for bb in sw_u8s]
# then do something like
# packet_start = np.argwhere(sw_u32s == packet_header)
# to find the indices of the packet headers

Read your array as np.uint8:
import numpy as np
# -v- packet header starts at byte 3
data_bytes = bytes([0x32, 0x12, 0x8a, 0xd9, 0x37, 0xb2, 0xa5, 0xf8, 0x3d])
dat = np.frombuffer(data_bytes, dtype=np.uint8)
Make a view of "windowed int32" values:
dat_32win = np.ndarray((len(dat) - 3,), dtype=np.uint32, buffer=dat, strides=(1,))
Now you take your packet header in the byte order of your machine. In the likely chance that it is little endian, you need to reverse the order of the bytes (you can do this programmatically with int.to_bytes and int.from_bytes but it doesn't seem worth the trouble):
packet_header_le = 0xa5b237d9
Now you just need to find the index of that value:
idx = np.argmax(dat_32win == packet_header_le)
print(idx)
# 3
One note, if the packet header is not in the byte sequence, then that last np.argmax would return 0 (because its argument would be an array full of False values), which is the same result you would get if the packet header started at the first byte. You may want to handle that error condition checking that the packet header value is actually there.

Encode/decode data from RS232 serial port

This is the first time I've had to connect to a device via RS232 serial to read/write data and I'm stuck on the encoding/decoding procedures.
I'm doing everything in Python 3 using the library "pyserial". Here is what I've done so far:
import serial
ser = serial.Serial()
ser.port = '/dev/ttyUSB0'
ser.baudrate = 115200
ser.bytesize = serial.EIGHTBITS
ser.parity = serial.PARITY_NONE
ser.stopbits = serial.STOPBITS_ONE
ser.timeout = 3
ser.open()
device_write = ser.write(bytearray.fromhex('AA 55 00 00 07 00 12 19 00'))
device_read = ser.read_until()
The connection/communication appears to be working as intended. The output of device_read is
b'M1830130A2IMU v3.2.9.1 26.04.19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x0527641\x00\x00\x00IMHF R.1.0.0 10.28.2018 td: 6.500ms\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00'
and this is where I'm stuck. I don't know how to interpret this. Attached is an image from the datasheet which explains what the output is suppose to represent.
The datasheet says "fields in bytes 98 to 164 are empty" for the device I have. Can someone help me understand what needs to be done to convert the output of ser.read_until() to a form that is "human readable" and represents the data in the image? I don't need someone to write the code for me, but I'm not even sure where to start. Again, this is my first time doing this so I'm a bit lost on what is going on.

If you are trying to write a single byte with hex value 12 (decimal 18), I believe what you need to do is ser.write(bytes([0x12])), which is equivalent to ser.write(bytes([18])).
It looks like your output is 154 bytes rather than 98, and much of it non-human-readable.
But if you did have the data described in the graph, you could break it up like this:
ID_sn = device_read[0:8].decode('ascii')
ID_fw = device_read[8:48].decode('ascii')
Press_Sens = device_read[48]
and so on.

This isn't an answer, just #ozangds' idea fleshed-out (might save you some typing):
def decode_bytes(data, start, stop):
return data[start:stop+1].decode('ascii').rstrip('\x00')
device_read = b'M1830130A2IMU v3.2.9.1 26.04.19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x0527641\x00\x00\x00IMHF R.1.0.0 10.28.2018 td: 6.500ms\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00'
ID_sn = decode_bytes(device_read, 0, 7)
ID_fw = decode_bytes(device_read, 8, 47)
Press_sens = device_read[48]
IMU_type = device_read[49]
IMU_sn = decode_bytes(device_read, 50, 57)
IMU_fw = decode_bytes(device_read, 58, 97)
label_fmt = '{:>10}: {!r}'
print(label_fmt.format('ID_sn', ID_sn))
print(label_fmt.format('ID_fw', ID_fw))
print(label_fmt.format('Press_sens', Press_sens))
print(label_fmt.format('IMU_type', IMU_type))
print(label_fmt.format('IMU_sn', IMU_sn))
print(label_fmt.format('IMU_fw', IMU_fw))
Output:
ID_sn: 'M1830130'
ID_fw: 'A2IMU v3.2.9.1 26.04.19'
Press_sens: 2
IMU_type: 5
IMU_sn: '27641'
IMU_fw: 'IMHF R.1.0.0 10.28.2018 td: 6.500ms'

Raspberry pi adc: MPC3001 using python and spidev

I am trying to use a MPC3001 ADC with the Raspberry pi over SPI using Python.
While doing so I am getting some strange results:
The code I am using:
import sys
import spidev
spi = spidev.SpiDev()
spi.open(0,0)
def readAdc(channel):
r = spi.xfer2([1, 8 + channel << 4, 0])
return ((r[1]&3) << 8) + r[2]
while True:
print readAdc(0)
time.sleep(0.5)
running the script above, while measuring the center point of a voltage divider, yields a random switching between 2 values: 504 and 1016.
Since 504 is the value I would expect to be correct, in combination with the binary representation of the two results;
504 --> 00111111000
1016 --> 01111111000
I assume I am accidentally 'creating' a 1 somewhere.
Can someone point me in the right direction.
Thanks in advance
BTW: Is it me, or is there no decent documentation for the spidev lib?

In the data sheet figure 5-2 it shows what the issue is. The data returned from the device, bit by bit, looks like this (where X = don't care, and Bx is bit number x of the data; B0 is LSB and B9 MSB):
BYTE0: X X 0 B9 B8 B7 B6 B5
BYTE1: B4 B3 B2 B1 B0 B1 B2 B3
BYTE2: B4 B5 B6 B7 B8 B9 X X
If you change the return statement to this:
return ((r[0] & 0x1F) << 5) | ((r[1] >> 3) & 0x1F)
it might work.
But if I were you I wouldn't trust me (I no longer have a hardware setup where I could test anything). I think it's best to print the individual values of the bytes in r and ponder the result until it starts to make sense. Be aware that some SPI implementations let you read in the data on either the rising or the falling edge. If you have that wrong your data will never make sense.
Another clue here is that the three LSBs of an A/D converter reading shouldn't be consistently 000, for reading after reading. There is almost always noise in the least significant bit, at least, if not two or three bits.
BTW I agree with you about spidev. I switched to quick2wire and then things went much more smoothly. It's better written and much better documented.

Python delays on Raspberry Pi

I'm trying to simulate a compound action potential for calibrating research instruments. The goal is to output a certain 10 µV signal at 250 Hz. The low voltage will be dealt with later, the main problem for me is the frequency. The picture below shows an overview of the system I'm trying to make.
By data acquisition from a live animal, and processing the data in MATLAB, I've made a low noise signal, with 789 values in 12-bit format. I then cloned the repository where I stored this in csv-format to a Raspberry Pi using Git. Below is the Python script I've written on the RPi. You can skip to def main in the script to see functionality.
#!/usr/bin/python
import spidev
from time import sleep
import RPi.GPIO as GPIO
import csv
import sys
import math
DEBUG = False
spi_max_speed = 20 * 1000000
V_Ref = 5000
Resolution = 2**12
CE = 0
spi = spidev.SpiDev()
spi.open(0,CE)
spi.max_speed_hz = spi_max_speed
LDAQ = 22
GPIO.setmode(GPIO.BOARD)
GPIO.setup(LDAQ, GPIO.OUT)
GPIO.output(LDAQ,GPIO.LOW)
def setOutput(val):
lowByte = val & 0b11111111 #Make bytes using MCP4921 data sheet info
highByte = ((val >> 8) & 0xff) | 0b0 << 7 | 0b0 << 6 | 0b1 << 5 | 0b1 << 4
if DEBUG :
print("Highbyte = {0:8b}".format(highByte))
print("Lowbyte = {0:8b}".format(lowByte))
spi.xfer2([highByte, lowByte])
def main():
with open('signal12bit.csv') as signal:
signal_length = float(raw_input("Please input signal length in ms: "))
delay = float(raw_input("Please input delay after signal in ms: "))
amplitude = float(raw_input("Please input signal amplitude in mV: "))
print "Starting Simulant with signal length %.1f ms, delay %.1f ms and amplitude %.1f mV." % (signal_length, delay, amplitude)
if not DEBUG : print "Press ctrl+c to close."
sleep (1) #Wait a sec before starting
read = csv.reader(signal, delimiter=' ', quotechar='|')
try:
while(True):
signal.seek(0)
for row in read: #Loop csv file rows
if DEBUG : print ', '.join(row)
setOutput(int(row)/int((V_Ref/amplitude))) #Adjust amplitude, not super necessary to do in software
sleep (signal_length/(data_points*1000) #Divide by 1000 to make into ms, divide by length of data
sleep (delay/1000)
except (KeyboardInterrupt, Exception) as e:
print(e)
print "Closing SPI channel"
setOutput(0)
GPIO.cleanup()
spi.close()
if __name__ == '__main__':
main()
This script almost works as intended. Connecting the output pin of an MCP4921 DAC to an oscilloscope shows that it reproduces the signal very well, and it outputs the subsequent delay correctly.
Unfortunately, the data points are seperated much further than I need them to be. The shortest time I can cram the signal into is about 79 ms. This is due to dividing by 789000 in the sleep function, which I know is too much to ask from Python and from the Pi, because reading the csv file takes time. However, if I try making an array manually, and putting those values out instead of reading the csv file, I can achieve a frequency over 6 kHz with no loss.
My question is this
How can I get this signal to appear at a frequency of 250 Hz, and decrease it reliably from the user's input? I've thought about manually writing the 789 values into an array in the script, and then changing the SPI speed to whatever value fits with 250 Hz. This would eliminate the slow csv reader function, but then you can't reduce the frequency from user input. In any case, eliminating the need for csv.read would help a lot. Thanks!

Figured it out earlier today, so I thought I'd post an answer here, in case someone comes upon a similar problem in the future.
The problem with the internal delay between data points cannot be solved with sleep(), for several reasons. What I ended up doing was the following
Move all math and function calling out of the critical loop
Do a linear regression analysis on the time it takes to transfer the values with no delay
Increase the number of datapoints in the CSV file to "plenty" (9600) in MATLAB
Calculate the number of points needed to meet the user's wanted signal length
Take evenly seperated entries from the now bigger CSV file to fit that number of points as closely as possible.
Calculate these values and then calculate the SPI bytes explicitly
Save the two byte lists, and output them directly in the critical loop
The new code, with a bit of input checking, is below
#!/usr/bin/python
import spidev
from time import sleep
import RPi.GPIO as GPIO
import sys
import csv
import ast
spi_max_speed = 16 * 1000000 # 16 MHz
V_Ref = 5000 # 5V in mV
Resolution = 2**12 # 12 bits for the MCP 4921
CE = 0 # CE0 or CE1, select SPI device on bus
total_data_points = 9600 #CSV file length
spi = spidev.SpiDev()
spi.open(0,CE)
spi.max_speed_hz = spi_max_speed
LDAQ=22
GPIO.setmode(GPIO.BOARD)
GPIO.setup(LDAQ, GPIO.OUT)
GPIO.output(LDAQ,GPIO.LOW)
def main():
#User inputs and checking for digits
signalLengthU = raw_input("Input signal length in ms, minimum 4: ")
if signalLengthU.isdigit():
signalLength = signalLengthU
else:
signalLength = 4
delayU = raw_input("Input delay after signal in ms: ")
if delayU.isdigit():
delay = delayU
else:
delay = 0
amplitudeU = raw_input("Input signal amplitude in mV, between 1 and 5000: ")
if amplitudeU.isdigit():
amplitude = amplitudeU
else:
amplitude = 5000
#Calculate data points, delay, and amplitude
data_points = int((1000*float(signalLength)-24.6418)/12.3291)
signalDelay = float(delay)/1000
setAmplitude = V_Ref/float(amplitude)
#Load and save CSV file
datain = open('signal12bit.csv')
read = csv.reader(datain, delimiter=' ', quotechar='|')
signal = []
for row in read:
signal.append(ast.literal_eval(row[0]))
#Downsampling to achieve desired signal length
downsampling = int(round(total_data_points/data_points))
signalSpeed = signal[0::downsampling]
listlen = len(signalSpeed)
#Construction of SPI bytes, to avoid calling functions in critical loop
lowByte = []
highByte = []
for i in signalSpeed:
lowByte.append(int(i/setAmplitude) & 0b11111111)
highByte.append(((int(i/setAmplitude) >> 8) & 0xff) | 0b0 << 7 | 0b0 << 6 | 0b1 << 5 | 0b1 << 4)
print "Starting Simulant with signal length %s ms, delay %s ms and amplitude %s mV." % (signalLength, delay, amplitude)
print "Press ctrl+c to stop."
sleep (1)
try:
while(True): #Main loop
for i in range(listlen):
spi.xfer2([highByte[i],lowByte[i]]) #Critical loop, no delay!
sleep (signalDelay)
except (KeyboardInterrupt, Exception) as e:
print e
print "Closing SPI channel"
lowByte = 0 & 0b11111111
highByte = ((0 >> 8) & 0xff) | 0b0 << 7 | 0b0 << 6 | 0b1 << 5 | 0b1 << 4
spi.xfer2([highByte, lowByte])
GPIO.cleanup()
spi.close()
if __name__ == '__main__':
main()
The result is exactly what I wanted. Below is seen an example from the oscilloscope with a signal length of 5 ms; 200 Hz. Thanks for your help, guys!

As I can distinguish different data from Serial?

I trying to read by serial differents values, but i dont know hoy to split that, because the two values are numbers but from different source
First i have a PICAXE sending converted data by ADC of light sensor by serial to python.
Second i have a PICAXE sending data of temperature sensor by serial to python.
Light code PICAXE
symbol puerto = B.5
main: readadc10 puerto,w1 ; read value into w1
sertxd(#w1,cr,lf)
goto main ; loop back to start
Temp code PICAXE
symbol temp = B.4
readtemp temp, w0 ; read value into w1
debug
sertxd(#w0,cr,lf)
goto main
Python code
import pygame
import sys, serial
from pygame.locals import *
ser = serial.Serial()
ser.port = 3
ser.baudrate = 4800
while True:
datos = ser.readline()
grados = float(datos)
print grados
The problem is that picaxe send simultaneus data from light and temp, but when python receive data, i dont know how to recognized each data.
Anyone can help me??
Thank!

If you have a temperature reading and a light level reading to send at the same time, you could put them on one line separated by a space.
PICAXE:
sertxd(#w0," ",#w1,cr,lf)
Python:
readings = ser.readline()
[reading1, reading2] = readings.split()
temperature = float(reading1)
lightlevel = float(reading2)
If the two types of reading are produced irregularly, you could transmit a character before each one to identify what type it is.
PICAXE:
sertxd("T ",#w0,cr,lf)
...
sertxd("L ",#w1,cr,lf)
Python:
reading = ser.readline()
[readingtype, readingvalue] = reading.split()
if readingtype == "T":
temperature = float(readingvalue)
elif readingtype == "L":
lightlevel = float(readingvalue)