We Keep Coding

How to do non blocking usb serial input in circuit python?

I am trying to interface a micropython board with python on my computer using serial read and write, however I can't find a way to read usb serial data in micropython that is non-blocking.
Basicly I want to call the input function without requiring an input to move on. (something like https://github.com/adafruit/circuitpython/pull/1186 but for usb)
I have tried using tasko, uselect (it failed to import the library and I can't find a download), and await functions. I'm not sure it there is a way to do this, but any help would be appreciated.

For CircuitPython there's supervisor.runtime.serial_bytes_available which may provide the building block for what you want to do.
This is discussed on Adafruit Forums: Receive commands from the computer via USB.

based on the relative new added usb_cdc buildin (>= 7.0.0) you can do something like this:
def read_serial(serial):
available = serial.in_waiting
while available:
raw = serial.read(available)
text = raw.decode("utf-8")
available = serial.in_waiting
return text
# main
buffer = ""
serial = usb_cdc.console
while True:
buffer += read_serial(serial)
if buffer.endswith("\n"):
# strip line end
input_line = buffer[:-1]
# clear buffer
buffer = ""
# handle input
handle_your_input(input_line)
do_your_other_stuff()
this is a very simple example.
the line-end handling could get very complex if you want to support universal line ends and multiple commands send at once...
i have created a library based on this: CircuitPython_nonblocking_serialinput

Reading python serial data without new line

I am reading serial port data in python. The data does not have newline, so I am receiving data continuously. The data packet has a termination with '|'. I want to read data continuously and print it on newline after '|'
My current data looks like this (highlighted fields are data of my interest in each packet, how shall I extract it and plot )
b'\x01\x063\x011;790.10,3.73,203.45;0.00;28|1;503.36,2.88,129.87;2.00;28|1;1.00,1.60,0.23;4.00;28|1;167.10,1.13,44.98;6.00;28|1;0.07,0.34,2.02;8.00;28|1;100.44,1.04,26.24;10.00;28|1;0.33,0.89,1.65;12.00;28|1;71.72,0.13,19.10;14.00;28|1;0.07,0.41,1.76;16.00;28|1;55.70,0.08,14.89;18.00;28|1;0.19,0.61,2.07;20.00;28|1;45.84,0.46,11.70;22.00;28|1;0.07,0.44,1.76;24.00;28|1;38.87,0.53,9.90;26.00;28|1;0.12,0.11,1.62;28.00;28|1;33.65,0.26,8.57;30.00;28|1;0.07,0.11,1.58;32.00;28|1;29.80,0.36,7.51;34.00;28|1;0.09,0.37,1.48;36.00;28|1;26.65,0.17,6.80;38.00;28|1;0.07,0.28,1.43;40.00;28|1;24.07,0.11,6.32;42.00;28|1;0.06,0.14,1.66;44.00;28|1;22.11,0.09,5.65;46.00;28|1;0.07,0.15,1.66;48.00;28|1;20.41,0.22,5.13;50.00;28|1;0.05,0.08,1.61;52.00;28|1;18.93,0.05,4.80;54.00;28|1;0.06,0.12,1.77;56.00;28|1;17.74,0.14,4.24;58.00;28|1;0.03,0.04,1.57;60.00;28|1;16.61,0.06,4.03;62.00;28|1;0.06,0.07,1.55;64.00;28|1;15.59,0.14,3.86;66.00;28|1;0.02,0.11,1.68;68.00;28|1;14.78,0.12,3.49;70.00;28|1;0.06,0.18,1.57;72.00;28|1;14.03,0.05,3.39;74.00;28|1;0.01,0.09,1.67;76.00;28|1;13.35,0.04,3.15;78.00;28|1;0.05,0.14,1.72;80.00;28|1;12.81,0.18,2.85;82.00;28|1;0.00,0.10,1.60;84.00;28|1;12.26,0.16,2.75;86.00;28|1;0.05,0.07,1.61;88.00;28|1;11.73,0.08,2.58;90.00;28|1;0.01,0.08,1.58;92.00;28|1;11.31,0.10,2.46;94.00;28|1;0.04,0.16,1.54;96.00;28|1;10.87,0.07,2.40;98.00;28|1;0.01,0.08,1.57;100.00;28|1;10.48,0.06,2.32;102.00;28|1;0.04,0.06,1.66;104.00;28|1;10.19,0.06,2.15;106.00;28|1;0.02,0.03,1.64;108.00;28|1;9.87,0.09,2.05;110.00;28|1;0.03,0.07,1.65;112.00;28|1;9.57,0.02,1.94;114.00;28|1;0.02,0.04,1.69;116.00;28|1;9.33,0.09,1.75;118.00;28|1;0.03,0.04,1.57;120.00;28|1;9.05,0.03,1.75;122.00;28|1;0.02,0.02,1.61;124.00;28|1;8.79,0.08,1.68;126.00;28|1;0.02,0.10,1.65;128.00;28|1;8.61,0.05,1.55;130.00;28|1;0.03,0.11,1.58;132.00;28|1;8.40,0.06,1.56;134.00;28|1;0.02,0.04,1.67;136.00;28|1;8.22,0.05,1.42;138.00;28|1;0.03,0.07,1.65;140.00;28|1;8.08,0.13,1.31;142.00;28|1;0.01,0.07,1.59;144.00;28|1;7.90,0.08,1.30;146.00;28|1;0.03,0.06,1.62;148.00;28|1;7.73,0.03,1.20;150.00;28|1;0.01,0.07,1.58;152.00;28|1;7.60,0.04,1.17;154.00;28|1;0.03,0.10,1.57;156.00;28|1;7.45,0.07,1.16;158.00;28|1;0.01,0.02,1.62;160.00;28|1;7.33,0.04,1.10;162.00;28|1;0.03,0.04,1.65;164.00;28|1;7.26,0.05,1.01;166.00;28|1;0.00,0.01,1.64;168.00;28|1;7.14,0.04,0.96;170.00;28|1;0.03,0.05,1.66;172.00;28|1;7.03,0.05,0.88;174.00;28|1;0.00,0.02,1.65;176.00;28|1;6.95,0.07,0.78;178.00;28|1;0.03,0.05,1.57;180.00;28|1;6.84,0.02,0.81;182.00;28|1;0.01,0.03,1.62;184.00;28|1;6.75,0.04,0.74;186.00;28|1;0.03,0.09,1.62;188.00;28|1;6.71,0.06,0.68;190.00;28|1;0.01,0.07,1.59;192.00;28|1;6.64,0.06,0.70;194.00;28|1;0.02,0.02,1.67;196.00;28|1;6.58,0.06,0.57;198.00;28|1;0.01,0.04,1.62;200.00;28|1;6.54,0.10,0.53;202.00;28|1;0.02,0.08,1.59;204.00;28|1;6.46,0.04,0.52;206.00;28|1;0.02,0.06,1.62;208.00;28|1;6.40,0.01,0.45;210.00;28|1;0.02,0.06,1.58;212.00;28|1;6.37,0.03,0.45;214.00;28|1;0.02,0.06,1.60;216.00;28|1;6.32,0.07,0.43;218.00;28|1;0.02,0.01,1.64;220.00;28|1;6.29,0.02,0.36;222.00;28|1;0.02,0.03,1.65;224.00;28|1;6.29,0.05,0.31;226.00;28|1;0.01,0.02,1.64;228.00;28|1;6.24,0.01,0.26;230.00;28|1;0.02,0.04,1.66;232.00;28|1;6.21,0.06,0.18;234.00;28|1;0.01,0.03,1.62;236.00;28|1;6.20,0.05,0.14;238.00;28|1;0.02,0.06,1.57;240.00;28|1;6.16,0.01,0.16;242.00;28|1;0.01,0.04,1.62;244.00;28|1;6.16,0.02,0.09;246.00;28|1;0.02,0.09,1.60;248.00;28|1;6.18,0.07,0.09;250.00;28|1;0.00,0.04,1.60;252.00;28|1;6.17,0.06,0.08;254.00;28|DATAEND|
I am currenlty reading 3480 bytes. but want to read data continuously
import serial
ser = serial.Serial('/dev/ttyUSB0', 115200, serial.EIGHTBITS, serial.PARITY_NONE, serial.STOPBITS_ONE)
buff = list()
values = bytearray([1,6,51,1]) # serial port command to read serial data
#print(type(values))
ser.write(values)
while True:
s = ser.read(3480)
print(s)
I want to separate all fields and print the data that is highlighted

I would recommend the use of read_until which allows you to define the character which works like a newline (LF is the default value). The call signature is read_until(expected=LF, size=None). For details consult https://pyserial.readthedocs.io/en/latest/pyserial_api.html . Alternatively you could read everything which is available and use a regular expression to find the single packages and extract the fields. This can be done by re.find_iter.

Python - Reading Serial Data that is constantly sending for parsing

I have a control box and a Raspberry Pi which communicate over Serial (Serial to RJ45), and I need the commands sent from the control box which are sent every 50ms. I am able to read the code, but here's the issue. When I start reading, the starting byte is incorrect so I am unable to parse it.
For example (The output I am currently getting):
b'\0x21\0x21\0x98\0x98\0x21\0x21\0x18\0x12\0x21\0x12\0x02\0x32\0x11
The starting byte I need has to be 0x98, so I need it to be like this
b'\0x98\0x98\0x21\0x21\0x18\0x12\0x21\0x12\0x02\0x32\0x11\0x‌​12\0x11
I need it this way so I can parse the line and say grab Byte[4]-(0x21) or something like that.
In terms of research, I ran into Struct. I have no idea how to use this though, and I have no idea if I even need to use it.
I currently don't have a full version of the code on me at this moment, but here is a quick example of what I currently have:
import serial
import time
port = serial.Serial("/dev/ttyS0", baudrate=9600)
while True:
output = port.read(13) # --- In Total there are 13 Bytes
print(output)

Since you are getting another lot of data every 50mS, you need to be able to sync with the start of the data:
buffer = b''
header = b'\0x98'
while True:
if port.in_waiting:
buffer += port.read(port.in_waiting)
while len(buffer) >= 2:
if buffer[0] == header and buffer[1] == header:
break
buffer=buffer[1:]
if len(buffer) >= 13:
print(buffer[:13]) # or otherwise process latest buffer
buffer = buffer[13:]
This code starts with an empty buffer and then reads whatever data arrives at the serial port. While the buffer does not start with the two header bytes, any excess at the front is discarded. Once the buffer starts with the right header and is long enough, the 13 bytes are printed here (but you might want to call another function to process a whole packet), and then that packet is thrown away, ready to start with whatever arrives next.

Unexpected behaviour of read(n) in pySerial 3.3

I'm writing a project using an STM32F407_VG board that uses an RS232 connection to send batch of data of different sizes (~400 bytes) on a serial port, and those data must be written on file. On desktop side I'm using a python 3 script with pyserial 3.3.
I've tried reading a single byte at time with ser.read() but I think it's too slow because I'm losing some of the data. So I'm trying to send the size of the batch as an integer before the batch itself, in order to reduce the overhead, and write data to file during time interval within a batch and the following one.
PROBLEM IS: ser.read(n) behave in a very strange way, and 99% of the times it blocks when it's time to read the batch and do not return. It also happens that sometimes it can read the first batch and writes it to file successfully, but it blocks at the second loop iteration. It's strange because I can use ser.read(4) to get the batch size with zero problem, and I use ser.readline() at the beginning of the script when listening to a starting signal, but I cannot read the data.
I'm sure that data are there and are well formed because I checked with a logic analyzer, and I've already tried with enabling and disabling flow control or set different baud rates on both the board and the script. I think it could be a config problem of python, but actually I've run out of ideas.
PYTHON SCRIPT CODE -SNIPPET
ser = serial.Serial(str(sys.argv[1]), \
int(sys.argv[2]), \
stopbits=serial.STOPBITS_ONE, \
parity=serial.PARITY_NONE, \
bytesize=serial.EIGHTBITS, \
timeout=None \
)
outputFile = open(sys.argv[3],"wb")
# wait for begin string
beginSignal = "ready"
word = ""
while word != beginSignal:
word = ser.readline().decode()
word = sample.split("\n")[0]
print("Started receiving...")
while True:
# read size of next batch
nextBatchSize = ser.read(4)
nextBatchSize = int.from_bytes(nextBatchSize,byteorder='little', signed=True)
# reads the batch:
# THIS IS THE ONE THAT CREATES PROBLEMS
batch = ser.read(nextBatchSize)
# write data to file
outputFile.write(batch)
BOARD CODE - SNIPPET
// this function sends the size of the batch and the batch itself
void sendToSerial(unsigned char* mp3data, int size){
// send actual size of the batch
write(STDOUT_FILENO,&size,sizeof(int));
// send the batch of data
write(STDOUT_FILENO,mp3data,size);
}
Any idea? Thanks!

You might have already tried this, but print out nextBatchSize to confirm that it is what you expect, just in case the byte order is reversed. If this is wrong your Python code could be trying to read too many bytes, and would therefore block.
Also you can check ser.in_waiting to see how many bytes are available to be read from the input buffer before your read attempt:
print(ser.in_waiting)
batch = ser.read(nextBatchSize)
You should also check the return value of write() in your C code, which is the number of bytes actually written, or -1 if there is an error. write() is not guaranteed to write all of the bytes in the given buffer, so there might remain some that have not been written. Or there might have been an error.
Data loss suggests a flow control issue. You need to ensure that it is enabled at both ends. You've said that you tried it, but in your posted code it is not enabled. How are you opening and configuring the serial port at the board end?

Converting data from serial/usb using PySerial

I have a UBlox receiver connected to my computer and I am trying to read it using PySerial however I am new to python and was hoping to get some clarification/help on understanding the data.
My code looks like:
import serial
# open the connection port
connection = serial.Serial('/dev/ttyACM0', 9600)
# open a file to print the data. I am doing this to make
# sure it is working
file1 = open('output_file', 'wb+')
# All messages from ublox receivers end with a carriage return
# and a newline
msg = connection.readline()
# print the message to the file
print >> file1, msg
What I get in the file, and when I print the 'type' of msg it is a list:
['\xb5b\x01\x064\x00\xe0\x88\x96#\xd3\xb9\xff\xffX\x07\x03\xdd6\xc31\xf6\xfd)\x18\xea\xe6\x8fd\x1d\x00\x01\x00\x00\x00\x00\x00\x00\xfd\xff\xff\xff\x01\x00\x00\x00\x02\x00\x00\x00p\x00\x02\x0f\x16\xa2\x02\x00\x9c\xeb\xb5b\x01\x07\\x00\xe0\x88\x96#\xe0\x07\x01\x17\x15237\x04\x00\x00\x00\xd6\xb9\xff\xff\x03\x01\n']
["\x1a\x0c\x04\x19'y\x00$\xf7\xff\xff\x1a\x1d\x04\x01\x00\x007\x00\x00\x00\x00\x00\x02\x1f\x0c\x01\x00+:\x00\x00\x00\x00\x00\x01 \r\x07&-\x9f\x00\xff\x01\x00\x00\x17\xc1\x0c\x04\x16\n"]
In order to interpret/decode the ublox messages have two format types. Some of the messages are in NMEA format(basically comma delimited)
$MSG, 1, 2, 3, 4
Where the other messages are straight hexidecimal, where each byte or set of bytes represent some information
[AA BB CC DD EE]
So my question is: is there a way I can interpret/convert the data from serial connection to a readable or more usable format so I can actually work with the messages. Like I said, I am new to python and more used to C++ style strings or array of characters
`

A typical parsing task. In this case, it'll probably be the simplest to make tokenization two-stage:
read the data until you run into a message boundary (you didn't give enough info on how to recognize it)
split the read message into its meaningful parts
for type 1, it's likely re.split(", *",text)
for type 2, none needed
display the parts however you want
Regarding why serial.Serial.readline returns a list. I consulted the sources - serial.Serial delegates readline to io.IOBase, and its source indeed shows that it should return a bytestring.
So, the function might be overridden in your code by something. E.g. what do print connection.readline and print serial.Serial.readline show?