We Keep Coding

Python pyserial one write delay

I'm having a weird issue with pyserial, using Python 3.6.9, running under WSL Ubuntu 18.4.2 LTS
I've set up a simple function to send GCODE commands to a serial port:
def gcode_send(data):
print("Sending: " + data.strip())
data = data.strip() + "\n" # Strip all EOL characters for consistency
s.write(data.encode()) # Send g-code block to grbl
grbl_out = s.readline().decode().strip()
print(grbl_out)
It sort of works, but every command I send is 'held' until the next is sent.
e.g.
I send G0 X0 > the device doesn't react
I send G0 X1 > the device reacts to G0 X0
I send G1 X0 > the device reacts to G0 X1
and so on...
My setup code is:
s = serial.Serial(com, 115200)
s.write("\r\n\r\n".encode()) # Wake up grbl
time.sleep(2) # Wait for grbl to initialize
s.flushInput() # Flush startup text in serial input
I can work around the delay for now, but it's quite annoying and I can't find anyone else experiencing the same. Any idea what could be causing this?

There might be a lot of problems here, but rest assured that the pyserial is not causing it. It uses the underlying OS's API to communicate with the UART driver. That being said you first have to test your code with real Linux to see whether WSL is causing it. I.e. whether a Linux and Windows UART buffers are correctly synced.
I am sorry that I cannot tell whether a problem is in your code or not because I do not know the device you are using, so I cannot guess what is happening on its end of communication channel. Have in mind that Windows alone can act weirdly in best of circumstances, so, prepare yourself for some frustrations here. Check your motherboard or USB2Serial converter drivers or whatever hw you are using.
Next thing, you should know that sometimes, communication gets confusing if timeouts aren't set. Why? Nobody really knows. So try setting timeouts. Check whether you need software Xon/Xoff turned on or not, and other RS232 parameters that might be required by the device you are communicating with.
Also, see what is going on with s.readline(), I wouldn't personally use it. Timeouts might help or you can use s.read(1024) with timeouts. I do not remember right now, but see whether pyserial supports asynchronous communication. If it does, you can try using it instead of standard blocking mode.
Also, check whether you have to forcefully flush the serial buffer after s.write() or add a sleep after it. It might happen that the device doesn't get the message but the read request is activated. As the device didn't receive the command it doesn't respond. After you send another command, IO buffer is flushed and the previous one is delivered and so forth. Serial communication is fun, but when it hits a snag it can be a real P in the A, believe me.
Ow, a P.S. Check whether the device sends "\r\n\r\n" or "\r\n" only, or "\r" or "\n" in response. s.readline() might get confused. For a start, try putting there two s.readline()s one after another and print out each output. If the device sends double EOL then the one s.readline() is stopping on the empty line and your program receives an empty response, when you send another command s.readline() goes through the buffer and returns a full line that is already there but not read before.

Here it goes. The code promissed in the comment. Big portions of it removed and error checks too.
It is a typing terminal for using PyS60 Python console on Nokia smartphones in the Symbian series via bluetooth. Works fantastically.
from serial import *
from thread import start_new_thread as thread
from time import sleep
import sys, os
# Original code works on Linux too
# The following code for gettin one character from stdin without echoing it on terminal
# has its Linux complement using tricks from Python's stdlib getpass.py module
# I.e. put the terminal in non-blocking mode, turn off echoing and use sys.stdin.read(1)
# Here is Win code only (for brevity):
import msvcrt
def getchar ():
return msvcrt.getch()
def pause ():
raw_input("\nPress enter to continue . . .")
port = raw_input("Portname: ")
if os.name=="nt":
nport = ""
for x in port:
if x.isdigit(): nport += x
port = int(nport)-1
try:
s = Serial(port, 9600)
except:
print >> sys.stderr, "Cannot open the port!\nThe program will be closed."
pause()
sys.exit(1)
print "Port ready!"
running = 1
def reader():
while running:
try:
msg = s.read()
# If timeout is set
while msg=="":
msg = s.read()
sys.stdout.write(msg)
except: sleep(0.001)
thread(reader,())
while 1:
try: c = getchar()
except Exception, e:
running = 0
print >> sys.stderr, e
s.write('\r\n\x04')
break
if c=='\003' or c=='\x04':
running = 0
s.write('\r\n\x04')
break
s.write(c)
s.close()
pause()

pySerial Capturing a long response

Hi guys I'm working a on script that will get data from a host using the Data Communications Standard (Developed by: Data Communication Standard Committee Lens Processing Division of The Vision Council), by serial port and pass the data into ModBus Protocol for the device to perform it's operations.
Since I don't fiscally have access to the host machine I'm trying to develop a secondary script to emulate the host. I am currently on the stage where I need to read a lot of information from the serial port and I get only part of the data. I was hoping to get the whole string sent on the send_job() function on my host emulator script.
Guys also can any of you tell me if this would be a good approach? the only thing the machine is supposed to do is grab 2 values from the host response and assign them to two modbus holding registers.
NOTE: the initialization function is hard coded because it will always be the same and the actual response data will not matter except for status. Also the job request is hard coded i only pass the job # that i get from a modbus holding register, the exact logic on how the host resolved this should not matter i only need to send the job number scanned from the device in this format.
main script:
def request_job_modbus(job):
data = F'[06][1c]req=33[0d][0a]job={job}[0d][0a][1e][1d]'.encode('ascii')
writer(data)
def get_job_from_serial():
response = serial_client.read_all()
resp = response.decode()
return resp
# TODO : SEND INIT SEQUENCE ONCE AND VERIFY IF REQUEST status=0
initiation_request()
init_response_status = get_init_status()
print('init method being active')
print(get_init_status())
while True:
# TODO: get job request data
job_serial = get_job_from_serial()
print(job_serial)
host emulation script:
def send_job():
job_response = '''[06][1c]ans=33[0d]job=30925[0d]status=0;"ok"[0d]do=l[0d]add=;2.50[0d]ar=1[0d]
bcerin=;3.93[0d]bcerup=;-2.97[0d]crib=;64.00[0d]do=l[0d]ellh=;64.00[0d]engmask=;613l[0d]
erdrin=;0.00[0d]erdrup=;10.00[0d]ernrin=;2.00[0d]ernrup=;-8.00[0d]ersgin=;0.00[0d]
ersgup=;4.00[0d]gax=;0.00[0d]gbasex=;-5.30[0d]gcrosx=;-7.96[0d]kprva=;275[0d]kprvm=;0.55[0d]
ldpath=\\uscqx-tcpmain-at\lds\iot\do\800468.sdf[0d]lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]'''.encode('ascii')
writer(job_response)
def get_init_request():
req = p.readline()
print(req)
request = req.decode()[4:11]
# print(request)
if request == 'req=ini':
print('request == req=ini??? <<<<<<< cumple condicion y enviala respuesta')
send_init_response()
send_job()
while True:
# print(get_init_request())
get_init_request()
what I get in screen: main script
init method being active
bce
erd
condition was met init status=0
outside loop
ers
condition was met init status=0
inside while loop
trigger reset <<<--------------------
5782
`:lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]
outside loop
condition was met init status=0
outside loop
what I get in screen: host emulation script
b'[1c]req=ini[0d][0a][1e][1d]'
request == req=ini??? <<<<<<< cumple condicion y enviala respuesta
b''
b'[06][1c]req=33[0d][0a]job=5782[0d][0a][1e][1d]'
b''
b''
b''
b''
b''
b''

I'm suspect you're trying to write too much at once to a hardware buffer that is fairly small. Especially when dealing with low power hardware, assuming you can stuff an entire message into a buffer is not often correct. Even full modern PC's sometimes have very small buffers for legacy hardware like serial ports. You may find when you switch from development to actual hardware, that the RTS and DTR lines need to be used to determine when to send or receive data. This will be up to whoever designed the hardware unfortunately, as they are often also ignored.
I would try chunking your data transfer into smaller bits as a test to see if the whole message gets through. This is a quick and dirty first attempt that may have bugs, but it should get you down the right path:
def get_job_from_serial():
response = b'' #buffer for response
while True:
try:
response += serial_client.read() #read any available data or wait for timeout
#this technically could only be reading 1 char at a time, but any
#remotely modern pc should easily keep up with 9600 baud
except serial.SerialTimeoutException: #timeout probably means end of data
#you could also presumably check the length of the buffer if it's always
#a fixed length to determine if the entire message has been sent yet.
break
return response
def writer(command):
written = 0 #how many bytes have we actually written
chunksize = 128 #the smaller you go, the less likely to overflow
# a buffer, but the slower you go.
while written < len(command):
#you presumably might have to wait for p.dtr() == True or similar
#though it's just as likely to not have been implemented.
written += p.write(command[written:written+chunksize])
p.flush() #probably don't actually need this
P.S. I had to go to the source code for p.read_all (for some reason I couldn't find it online), and it does not do what I think you expect it does. The exact code for it is:
def read_all(self):
"""\
Read all bytes currently available in the buffer of the OS.
"""
return self.read(self.in_waiting)
There is no concept of waiting for a complete message, it just a shorthand for grab everything currently available.

Python serial.readline() not blocking

I'm trying to use hardware serial port devices with Python, but I'm having timing issues. If I send an interrogation command to the device, it should respond with data. If I try to read the incoming data too quickly, it receives nothing.
import serial
device = serial.Serial("/dev/ttyUSB0", 9600, timeout=0)
device.flushInput()
device.write("command")
response = device.readline()
print response
''
The readline() command isn't blocking and waiting for a new line as it should. Is there a simple workaround?

readline() uses the same timeout value you passed to serial.Serial().
If you want readline to be blocking, just delete the timeout argument, the default value is None.
You could also set it to None before calling readline(), if you want to have a timeout for openening the device:
import serial
try:
device = serial.Serial("/dev/ttyUSB0", 9600, timeout=0.5)
except:
#Exception handeling
device.flushInput()
device.write("command")
device.timeout=None
response = device.readline()
print response

I couldn't add a commend so I will just add this as an answer. You can reference this stackoverflow thread. Someone attempted something similar to your question.
Seems they put their data reading in a loop and continuously looped over it while data came in. You have to ask yourself one thing if you will take this approach, when will you stop collecting data and jump out of the loop? You can try and continue to read data, when you are already collecting, if nothing has come in for a few milliseconds, jump out and take that data and do what you want with it.
You can also try something like:
While True:
serial.flushInput()
serial.write(command)
incommingBYTES = serial.inWaiting()
serial.read(incommingBYTES)
#rest of the code down here

Socket issues in Python

I'm building a simple server-client app using sockets. Right now, I am trying to get my client to print to console only when it received a specific message (actually, when it doesn't receive a specific message), but for some reason, every other time I run it, it goes through the other statement in my code and is really inconsistent - sometimes it will work as it should and then it will randomly break for a couple uses.
Here is the code on my client side:
def post_checker(client_socket):
response = client_socket.recv(1024)
#check if response is "NP" for a new post from another user
if response == "NP":
new_response = client_socket.recv(1024)
print new_response
else: #print original message being sent
print response
where post_checker is called in the main function as simply "post_checker(client_socket)" Basically, sometimes I get "NPray" printed to my console (when the client only expects to receive the username "ray") and other times it will print correctly.
Here is the server code correlated to this
for sublist in user_list:
client_socket.send("NP")
client_socket.send(sublist[1] + " ")
where user_list is a nested list and sublist[1] is the username I wish to print out on the client side.
Whats going on here?

The nature of your problem is that TCP is a streaming protocol. The bufsize in recv(bufsize) is a maximum size. The recv function will always return data when available, even if not all of the bytes have been received.
See the documentation for details.
This causes problems when you've only sent half the bytes, but you've already started processing the data. I suggest you take a look at the "recvall" concept from this site or you can also consider using UDP sockets (which would solve this problem but may create a host of others as UDP is not a guaranteed protocol).
You may also want to let the python packages handle some of the underlying framework for you. Consider using a SocketServer as documented here:

buffer = []
def recv(sock):
global buffer
message = b""
while True:
if not (b"\r\n" in b"".join(buffer)):
chunk = sock.recv(1024)
if not chunk:
break
buffer.append(chunk)
concat = b"".join(buffer)
if (b"\r\n" in concat):
message = concat[:concat.index(b"\r\n")]
concat = concat[concat.index(b"\r\n") + 2:]
buffer = [concat]
break
return message
def send(sock, data):
sock.send(data + b"\r\n")
I have tested this, and in my opinion, it works perfectly
My use case: I have two scripts that send data quickly, it ends up that one time or another, the buffers receive more than they should, and gather the data, with this script it leaves everything that receives more saved, and continues receiving until there is a new line between the data, and then, it gathers the data, divides in the new line, saves the rest and returns the data perfectly separated
(I translated this, so please excuse me if anything is wrong or misunderstood)

Arduino serial timeouts after several serial writes

I noticed with my board from DIY drones a strange behavior when I use my custom firmware.
Here is an example function which is called in my firmware running on an Arduino board:
void send_attitude(float roll, float pitch, float yaw) {
hal.console->printf("{\"type\":\"sens_attitude\",\"roll\":%.4f,\"pitch\":%.4f,\"yaw\":%.4f}\n",
roll, pitch, yaw);
}
As you can see, the code just writing a message in the serial port set in setup (hal.uartA).
I call this function every 0.5s:
inline void medium_loop() {
static int timer = 0;
int time = hal.scheduler->millis() - timer;
// send every 0.5 s
if(time > 500) {
send_attitude(OUT_PIT, OUT_ROL, OUT_YAW);
timer = hal.scheduler->millis();
}
}
Now to the strange thing. If I use the serial monitor or read the board with another program or script everything is fine. Every 0.5s the proper LED is blinking and message is shown. But if I don't read it out, after appr. 10s the LED is flushing up continuously and no connection/communication is possible anymore. I have to unplug the board then. The same behavior is observed the other way round. If I send to my board over serial port (in my case USB) and don't flush the input buffer, the LED is flushing up continuously and I get a timeout. The following code works:
def send_data(line):
# calc checksum
chk = chksum(line)
# concatenate msg and chksum
output = "%s*%x\r\n" % (line, chk)
try:
bytes = ser.write(output)
except serial.SerialTimeoutException as e:
logging.error("Write timeout on serial port '{}': {}".format(com_port, e))
# Flush input buffer, if there is still some unprocessed data left
# Otherwise the APM 2.5 control boards stucks after some command
ser.flush() # Try to send old message
ser.flushInput() # Delete what is still inside the buffer
If I comment out this line:
ser.flushInput() # Delete what is still inside the buffer
I don't use more settings then this.
I get (depending on the message interval) a timeout sooner or later. In my case I send every 20ms a signal which results in a timeout after ~10s. Also dependent on the length of message. Bigger messages cause it faster than smaller ones.
My settings are shown in the following snippets. Client side python code:
com_port = '/dev/ttyACM0'
baud_rate = '115200'
try:
ser = serial.Serial(com_port, baud_rate, timeout=0.1, writeTimeout=0.1, rtscts=1)
The if these timeouts happen, then I also get one if I set the timeout to something like 2s. In my case I need a very low latency, which is indeed possible if I keep reading and flushing. Firmware code from my Arduino:
void setup() {
// Set baud rate when connected to RPi
hal.uartA->begin(115200);
hal.console->printf("Setup device ..\n");
// Followed by motor, compass, barometer initialization
My questions are:
What exactly happens with my board?
Why it is not reacting anymore if I just write in my serial port without reading or flushing the buffer?
Is it really a buffer or driver problem associated with this strange behavior and is this problem related to all Arduino boards or maybe just mine APM 2.5 from DIY drones?
Last but not least: I was finding no functions in the library which are targeting such problems. Are there maybe any I don't know?
The complete source code is #google code: https://code.google.com/p/rpicopter/source/browse/

What board are you using and what processor does it have? My guess would be that your board is based on the ATmega32U4, or some other microcontroller that has a built-in USB module. If so, I have seen similar behavior before here is what I think is happening:
There is a buffer on your microcontroller to hold serial data going to the computer. There is a buffer in the computer's USB serial driver to hold serial received from the chip. Since you are not reading bytes from the COM port, the buffer on the computer will fill up. Once the buffer on the computer fills up, it stops requesting data from the microcontroller. Therefore, the buffer on the microcontroller will eventually fill up.
Once the microcontroller's buffer is full, how do you expect printf command to behave? For simplicity, the printf you are using is probably designed to just wait in a blocking loop until buffer space is available and then send the next character, until the message is done. Since buffer space will never be available, your program gets stuck in an infinite loop.
A better strategy would be to check to see if enough buffer space is available before calling printf. The code might look something like this:
if(console_buffer_space() > 80)
{
hal.console->printf(...);
}
I don't know if this is possible in the DIY drones firmware, and I don't know if the max buffer space can actually ever reach 80, so you will have to research this a bit.

I don't understand the use of:
ser.flush() # Try to send old message
ser.flushInput() # Delete what is still inside the buffer
Lets say your device is connected to PC and the python code is writing the (line, chk):
ser.flush() - why are you using it?
ser.flushInput() - will "delete" the Serial input buffer at the PC

It looks like other people have the same problem. And thanks to the Mod-Braniac who deleted my minimal example. My bet is, that's a problem with Arduino USB controller chip or the firmware on it.