Stop reading bytes from serial port (Python/RPi/UART) - python

This problem is quite complicated. Seemingly I have simple two-way communication beetwen 2 devices where reading side is Raspberry pi 3. I'm trying to transfer file. I'm sending it part by part (5kb part). It looks like:
1) Sending side send first 5kb part (exactly 5136 bytes where first 16 bytes are ID/SIZE/ADD_INFO/CRC)
2) RPi3 read 5136 bytes, and calculate CRC for chunk.
3) RPi3 compare CRC received from sending side and calculated by RPi
4a) If CRC doesn't match I switch lanes from RX to TX line using Dual Bus Buffer Gate With 3-State Outputs and set High State at TX line (I keep it for 30ms).
4b) If CRC match I just wait for next chunk of file
5) Sending side switch lanes too and read my TX state if state is HIGH/1 (when CRC doesn't match) it sends same chunk (retransmission) if state is LOW/0 sends another chunk. (changing state take 10 ms)
On osciloscope it looks like this (4a):
and this (4b):
Beetwen 5136 chunk there is free time to calculate CRC then we have on RPi3 side changing state if we have to because CRC doesn't match (red lines at 4a) and reading TX side from Sending side.
Ok now some simplified code from RPi:
def recv():
while True:
#Set line to low because I want to read data
GPIO.output(16, GPIO.LOW)
ID = ser.read(4)
SIZE = ser.read(4)
ADD_INFO = ser.read(4)
CRC = ser.read(4)
#get crc from sending side
crc_chunk = int.from_bytes(CRC, byteorder='little')
data = ser.read(5120)
#calculating CRC from chunk
rpiCRC = crcSTM(data)
while True:
<--- here I wait about 20ms to calculate CRC
if rpiCRC != crc_chunk:
#Crc doesn't match I want retransmission
GPIO.output(16, GPIO.HIGH)
<--- keep it HIGH for about 30ms
break
else:
#keep it low because crc match and we are waiting for next chunk
GPIO.output(16, GPIO.HIGH)
break
All this looks legit but to the point. I always get only first chunk after that RPi just stop reading bytes. It only happen in case '4b' I get first chunk properly and I'm waiting for next one and then RPi just stop reading bytes or give me just some scratches from time to time. I test what If i get first chunk properly and set retransmission thing but everything looks great and I was getting all the time retransmission of first chunk and get it all the time. I came to this that changing line on sending side when I have LOW/1 state affects on RPi and it just stop reading bytes properly. Don't know why it's messing it and don't know how to fight it. I tryied flushing buffer, closing port before sending side chaning line and open it after it changes line again but all this just do nothing what can I do more ?
P.S This waiting things i do in my own timer but there is no need to put here code like code from sending side because problem is on RPi side.
P.S sorry for chaotic language but I was trying do explane it as simply as i can

Ok I fight it. I have enabled option "Would you like a login shell to be accessible over serial" in sudo raspi-congif. Don't know why this was messing everything but disabling this fix it. This is quite strange because I'm playing with raspberry and serial some time and there was no problem if RPi3 was sending something via uart or when it was just reading without changing lines etc :)

Related

Receiving the correct length of data over serial

Originally, I am using putty to display the output of my embedded device. However, I want to manipulate the data as I receive it, thus I used python to connect to the port(w timeout 0). All the C code does is print the buffer (lets say file length of size = 22112 bytes) as following:
for (x = 0; x < size; x++) {
printf("%x", buffer[x]);
}
printf("\r\n");
The python code just reads the data as a line since I end the C code with a new line after the loop:
ser_bytes = ser.readline()
print(len(ser_bytes))
But python prints the length to be 34742 when it should be 22112. As you may have noticed, I print the values in hex, other formats give other values.
So what is the correct way to receive the data to get the correct size?
And is there any other way to do this?
UPDATE:
Currently I am using:
write(1, fileBuffer, size);
To write the raw data, but Python STILL does not receive all the data. Out of 22112, only around 8000 is received. Printing to Putty shows the correct number of data though.
For some visuals, this is the end of the file (around 8000 bytes) which is stored by python when the data is received over serial.
This is the original file itself (directly from the SD card). I highlighted where the data stops exactly.
UPDATE:
The problem seems to be a buffer issue. Since I am running on windows, I used the following to increase the buffer size:
ser.set_buffer_size(rx_size = 25000, tx_size = 25000)
ser.inWaiting(), shows a total of 22208 in the buffer when it should be 22112. I read it and store it anyways using ser.read(ser.inWaiting()). To see where the extra bytes are coming from, I print two lines from the original file and from the file after it is received over serial:
There seems to be an 'extra' carriage return in the file over serial. Where did it come from?

reading 14-bit data using i2c, python, and raspberry pi

I'm trying to read data from this barometric pressure sensor on a raspberry pi using python & i2c/smbus.
The sensor's data sheet (page 10) says it will output a digital value in the range 0-16383 (2**14). So far it seems like I have to read whole bytes, so I'm not sure how to get a 14 bit value. (I had a link to the data sheet, but SO says I need more reputation before I can add more links to posts.)
This sample uses Adafruit's I2C python library, which is basically a wrapper around SMBus.
import Adafruit_I2C
import time
# sensor returns a 14-bit reading
max_output = 2**14
# per data sheet, max_output == 1.6 bar
max_bar = 1.6
# i2c address specified in data sheet
sensor = Adafruit_I2C.Adafruit_I2C(0x78)
while True:
reading = sensor.readU16(0, little_endian=False)
# reading is sometimes, but not always, greater than 2**14
# this adjustment feels pretty hacky/wrong
while reading > max_output:
reading = reading >> 1
bar = reading / float(max_output) * max_bar
print bar
time.sleep(1)
I compare these readings to the output from my handheld GPS, which includes a barometer. I sometimes get readings which are somewhat close (1030 millibar when the GPS reads 1001 millibar), but the sensor then dips drastically (down to 930 millibar) for a few readings. I have a suspicion that this is due to how I'm reading the data, but no real evidence to back that up.
At this point, I'm not sure what to try next.
Some things I've guessed at, but would appreciate some more-informed help with:
How can I read just the 14 bits that the sensor is outputting?
What endian-ness are the returned values? Assuming big-endian produced values which seemed more sane, but I may be conflating multiple problems here.
How can I tell which register to read from? This isn't mentioned in the data-sheet anywhere. I guessed that register 0 is probably the only one.
You should be masking the output of the sensor, not shifting it. e.g. reading = reading & (max_output-1) should probably do it.
The top two bits are the status bits, so if they are set sometimes they could mean things like: normal mode or stale data indicator.

Capturing audio with python on the raspberry pi fails

I am currently working on an easy-to-use audio capturing device for digitizing old casette tapes (i.e. low fidelity). This device is based on a raspberry pi with an usb sound card, which does nothinge else than starting the listed python script on bootup.
import alsaaudio
import wave
import os.path
import RPi.GPIO as GPIO
import key
import usbstick
import time
try:
# Define storage
path = '/mnt/usb/'
prefix = 'Titel '
extension = '.wav'
# Configure GPIOs
GPIO.setmode(GPIO.BOARD)
button_shutdown = key.key(7)
button_record = key.key(11)
GPIO.setup(12, GPIO.OUT)
GPIO.setup(15, GPIO.OUT)
GPIO.output(12, GPIO.HIGH)
# Start thread to detect external memory
usb = usbstick.usbstick(path, 13)
# Configure volume
m = alsaaudio.Mixer('Mic', 0, 1)
m.setvolume(100, 0, 'capture')
# Only run until shutdown button gets pressed
while not (button_shutdown.pressed()):
# Only record if record button is pressed and memory is mounted
if (button_record.pressed() and usb.ismounted()):
# Create object to read input
inp = alsaaudio.PCM(alsaaudio.PCM_CAPTURE, alsaaudio.PCM_NORMAL, 'sysdefault:CARD=Device')
inp.setchannels(1)
inp.setrate(44100)
inp.setformat(alsaaudio.PCM_FORMAT_S16_LE)
inp.setperiodsize(1024)
# Find next file name
i = 0
filename = ''
while (True):
i += 1
filename = path + prefix + str(i) + extension
if not (os.path.exists(filename)):
break
print 'Neue Aufnahme wird gespeichert unter ' + filename
# Create wave file
wavfile = wave.open(filename, 'w')
wavfile.setnchannels(1)
wavfile.setsampwidth(2)
wavfile.setframerate(44100)
# Record sound
while (button_record.pressed()):
l, data = inp.read()
wavfile.writeframes(data)
GPIO.output(15, GPIO.HIGH)
# Stop record an save
print 'Aufnahme beendet\n'
inp.close()
wavfile.close()
GPIO.output(15, GPIO.LOW)
# Record has been started but no memory is mounted
elif (button_record.pressed() and not usb.ismounted()):
print 'Massenspeichergeraet nicht gefunden'
print 'Warte auf Massenspeichergeraet'
# Restart after timeout
timestamp = time.time()
while not (usb.ismounted()):
if ((time.time() - timestamp) > 120):
time.sleep(5)
print 'Timeout.'
#reboot()
#myexit()
print 'Massenspeichergeraet gefunden'
myexit()
except KeyboardInterrupt:
myexit()
According to the documentation pyaudio, the routine inp.read() or alsaaudio.PCM.read() respectively should usually wait until a full period of 1024 samples has been captured. It then should return the number of captured samples as well as the samples itself. Most of the time it returns exactly one period of 1024 samples. I don't think that I have a performance problem, since I would expect it to return several periods then.
THE VERY MYSTERIOUS BEHAVIOR: After 01:30 of recording, inp.read() takes some milliseconds longer than normal to process (this is a useful information in my ignorant opinion) and then returns -32 and faulty data. Then the stream continues. After half a minute at 02:00 it takes about a second (i.e. longer than the first time) to process and returns -32 and faulty data again. This procedere repeats then every minute (02:30-03:00, 03:30-04:00, 04:30-05:00). This timing specification was roughly taken by hand.
-32 seems to result from the following code line in /pyalsaaudio-0.7/alsaaudio.c
return -EPIPE;
Some words about how this expresses: If the data stream is directly written into the wave file, i.e. including the faulty period, the file contains sections of white noise. These sections last 30 seconds. This is because the samples usually consist of 2 bytes. When the faulty period (1 byte) is written, the byte order gets inverted. With the next faulty period it gets inverted again and therefore is correct. If faulty data is refused and only correct data is written into the wave file, the file 'jumps' every 30 seconds.
I think the problem can either be found in
1. the sound card (but I tested 2 different)
2. the computing performance of the raspberry pi
3. the lib pyaudio
Further note: I am pretty new to the linux and python topic. If you need any logs or something, please describe how I can find them.
To cut a long story short: Could someone please help me? How can I solve this error?
EDIT: I already did this usb firmware updating stuff, which is needed, since the usb can be overwhelmed. BTW: What exactly is this EPIPE failure?
Upon further inverstigation, I found out, that this error is not python/pyaudio specific. When I try to record a wave file with arecord, I get the following output. The overrun messages are sent according to the timing described above.
pi#raspberrypi ~ $ sudo arecord -D sysdefault:CARD=Device -B buffersize=192000 -f cd -t wav /mnt/usb/test.wav
Recording WAVE '/mnt/usb/test.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo
overrun!!! (at least -1871413807.430 ms long)
overrun!!! (at least -1871413807.433 ms long)
overrun!!! (at least -1871413807.341 ms long)
overrun!!! (at least -1871413807.442 ms long)
Referring to this thread at raspberrypi.org, the problem seems to be the (partly) limited write speed to the SD card or the usb storage device with a raspberry pi. Recording to RAM (with tmpfs) or compressing the audio data (e.g. to mp3 with lame) before writing it somewhere else could be a good solution in some cases.
I can't say why the write speed is to low. In my opinion, the data stream is 192 kByte/s for 48 kSamples/s, 16 Bit, stereo. Any SD card or usb mass storage should be able to handle this. As seen above, buffering the audio stream doesn't help.

Checking for parity errors with pyserial

I'm currently writing a small utility in python to monitor the communications on a serial line. This is being used to debug some hardware that is connected via rs232 so being able to see exactly what's going over the line is extremely important. How do I check for parity errors using pyserial?
Specifically I'm wondering if there is a platform independent way of finding the value of the parity bit using pyserial. I'd strongly prefer to not need termios to do this as this is used on some windows machines.
I monitored the parity bit by bit banging with the GPIO4 on my Pi.
Inspiration here
My solution is outputting the parity bit in a second byte and writing all into a file:
import time
import pigpio # http://abyz.me.uk/rpi/pigpio/python.html
RXD=4 # number of GPIO pin
pi = pigpio.pi()
if not pi.connected:
exit(0)
pigpio.exceptions = False # Ignore error if already set as bit bang read.
handle = pi.file_open("/home/pi/Documents/bit_bang_output.txt",pigpio.FILE_WRITE) #assuming that the file /opt/pigpio/access (yes without extension) contains a line /home/pi/Domcuments/* w
pi.bb_serial_read_open(RXD, 9600,9) # Set baud rate and number of data bits here. Reading 9 data bits will read the parity bit.
pigpio.exceptions = True
stop = time.time() + 5.0 # recording 5.0 seconds
while time.time() < stop:
(count, data) = pi.bb_serial_read(RXD)
if count:
#print(data.hex(),end="")
pi.file_write(handle, data.hex())
pi.bb_serial_read_close(RXD)
pi.stop()

Read data from load cell

Your help is badly needed...
I'm trying to read data and print it to the python console from a load cell. My setup is as follow:
The load cell is a MD type from Eilersen connected to a load cell signal converter of type MCE2040 Seriel Communication Module also from Eilersen. The MCE2040 is connected to my PC through a USB to seriel connector like this link_http://www.usbgear.com/USB-COM-I-SI.html (I'm only allowed two links) one.
The load cell is connected to COM 1.
I have tried to run this snippet:
import serial
ser = serial.Serial(0) # open first serial port
print ser.portstr # check which port was really used
#ser.write("hello") # write a string
ser.close()
...and that prints 'COM1' to the console so I guess my connection should be okay.
My problem is that I don't know how to proceed. In the end I'd like to plot a graph of the incoming data and output a data file with time stamps, but for starters I'd like to print some load cell data to the console.
Any help will be highly appreciated. If further information is needed, please let me know.
Thx in advance.
Edit:
I have some documentation re MCE2040:
3.1 EVC Mode (without time stamp)
Specification: RS232/RS4422
Baudrate: 115200 bps
38400 bps (select with SW1.5)
Data bits: 7
Parity: Even
Stop bits: 1
Protocol: EVC protocol described below (Transmit Only)
3.1.1 EVC Protocol Format
After each sample period a new weight telegram is transmitted. The transmitted telegram has the following format:
<LF>WWWWWWWW<CR>
Each telegram contains a line feed character, a weight result and a carriage return character. The telegram contains:
<LF> Line Feed character (ASCII 0Ah).
WWWWWWWW Weight value for the loadcell. The value is an 8 byte ASCII hex number with MSB first.
<CR> Carriage Return character (ASCII 0Dh).
I was able to get some output from the following code:
import serial
ser = serial.Serial(0, baudrate=115000 ,timeout=100)
print ser.portstr
x = ser.read(50)
print x
ser.close()
print 'close'
Output:
COM1
ÆÆÆÆA0·5
ÆÆÆÆA0·6
ÆÆÆÆA0·5
ÆÆÆÆA0·±
ÆÆÆÆA0·±
close
First of all make sure it's really your com port, since COM1 is used by a lot of computers i'm not sure it's your com port.
You can use a simple wire to loop back info by connecting TX to RX at the USB to Serial converter, it will result in an echo (you will read what you write) it's a very simple way to verify that you are talking with the right com port.
Regarding how to continue:
Useful basic commands:
ser.write("command") with this command you send to the device some command.
ser.read(n) is for read n bytes from the device
ser.readline() will read line until it reached \n (new line)
Steps:
Send a command to your device.
Read all the data by some end byte (Frame Synchronization).
Parse data to structure (list or something like that..)
Plot it to graph.
Useful Links:
pyserial docs
tips for reading serial
plotly for graphs in python

Categories