We Keep Coding

Python pyserial one write delay

I'm having a weird issue with pyserial, using Python 3.6.9, running under WSL Ubuntu 18.4.2 LTS
I've set up a simple function to send GCODE commands to a serial port:
def gcode_send(data):
print("Sending: " + data.strip())
data = data.strip() + "\n" # Strip all EOL characters for consistency
s.write(data.encode()) # Send g-code block to grbl
grbl_out = s.readline().decode().strip()
print(grbl_out)
It sort of works, but every command I send is 'held' until the next is sent.
e.g.
I send G0 X0 > the device doesn't react
I send G0 X1 > the device reacts to G0 X0
I send G1 X0 > the device reacts to G0 X1
and so on...
My setup code is:
s = serial.Serial(com, 115200)
s.write("\r\n\r\n".encode()) # Wake up grbl
time.sleep(2) # Wait for grbl to initialize
s.flushInput() # Flush startup text in serial input
I can work around the delay for now, but it's quite annoying and I can't find anyone else experiencing the same. Any idea what could be causing this?

There might be a lot of problems here, but rest assured that the pyserial is not causing it. It uses the underlying OS's API to communicate with the UART driver. That being said you first have to test your code with real Linux to see whether WSL is causing it. I.e. whether a Linux and Windows UART buffers are correctly synced.
I am sorry that I cannot tell whether a problem is in your code or not because I do not know the device you are using, so I cannot guess what is happening on its end of communication channel. Have in mind that Windows alone can act weirdly in best of circumstances, so, prepare yourself for some frustrations here. Check your motherboard or USB2Serial converter drivers or whatever hw you are using.
Next thing, you should know that sometimes, communication gets confusing if timeouts aren't set. Why? Nobody really knows. So try setting timeouts. Check whether you need software Xon/Xoff turned on or not, and other RS232 parameters that might be required by the device you are communicating with.
Also, see what is going on with s.readline(), I wouldn't personally use it. Timeouts might help or you can use s.read(1024) with timeouts. I do not remember right now, but see whether pyserial supports asynchronous communication. If it does, you can try using it instead of standard blocking mode.
Also, check whether you have to forcefully flush the serial buffer after s.write() or add a sleep after it. It might happen that the device doesn't get the message but the read request is activated. As the device didn't receive the command it doesn't respond. After you send another command, IO buffer is flushed and the previous one is delivered and so forth. Serial communication is fun, but when it hits a snag it can be a real P in the A, believe me.
Ow, a P.S. Check whether the device sends "\r\n\r\n" or "\r\n" only, or "\r" or "\n" in response. s.readline() might get confused. For a start, try putting there two s.readline()s one after another and print out each output. If the device sends double EOL then the one s.readline() is stopping on the empty line and your program receives an empty response, when you send another command s.readline() goes through the buffer and returns a full line that is already there but not read before.

Here it goes. The code promissed in the comment. Big portions of it removed and error checks too.
It is a typing terminal for using PyS60 Python console on Nokia smartphones in the Symbian series via bluetooth. Works fantastically.
from serial import *
from thread import start_new_thread as thread
from time import sleep
import sys, os
# Original code works on Linux too
# The following code for gettin one character from stdin without echoing it on terminal
# has its Linux complement using tricks from Python's stdlib getpass.py module
# I.e. put the terminal in non-blocking mode, turn off echoing and use sys.stdin.read(1)
# Here is Win code only (for brevity):
import msvcrt
def getchar ():
return msvcrt.getch()
def pause ():
raw_input("\nPress enter to continue . . .")
port = raw_input("Portname: ")
if os.name=="nt":
nport = ""
for x in port:
if x.isdigit(): nport += x
port = int(nport)-1
try:
s = Serial(port, 9600)
except:
print >> sys.stderr, "Cannot open the port!\nThe program will be closed."
pause()
sys.exit(1)
print "Port ready!"
running = 1
def reader():
while running:
try:
msg = s.read()
# If timeout is set
while msg=="":
msg = s.read()
sys.stdout.write(msg)
except: sleep(0.001)
thread(reader,())
while 1:
try: c = getchar()
except Exception, e:
running = 0
print >> sys.stderr, e
s.write('\r\n\x04')
break
if c=='\003' or c=='\x04':
running = 0
s.write('\r\n\x04')
break
s.write(c)
s.close()
pause()

pySerial Capturing a long response

Hi guys I'm working a on script that will get data from a host using the Data Communications Standard (Developed by: Data Communication Standard Committee Lens Processing Division of The Vision Council), by serial port and pass the data into ModBus Protocol for the device to perform it's operations.
Since I don't fiscally have access to the host machine I'm trying to develop a secondary script to emulate the host. I am currently on the stage where I need to read a lot of information from the serial port and I get only part of the data. I was hoping to get the whole string sent on the send_job() function on my host emulator script.
Guys also can any of you tell me if this would be a good approach? the only thing the machine is supposed to do is grab 2 values from the host response and assign them to two modbus holding registers.
NOTE: the initialization function is hard coded because it will always be the same and the actual response data will not matter except for status. Also the job request is hard coded i only pass the job # that i get from a modbus holding register, the exact logic on how the host resolved this should not matter i only need to send the job number scanned from the device in this format.
main script:
def request_job_modbus(job):
data = F'[06][1c]req=33[0d][0a]job={job}[0d][0a][1e][1d]'.encode('ascii')
writer(data)
def get_job_from_serial():
response = serial_client.read_all()
resp = response.decode()
return resp
# TODO : SEND INIT SEQUENCE ONCE AND VERIFY IF REQUEST status=0
initiation_request()
init_response_status = get_init_status()
print('init method being active')
print(get_init_status())
while True:
# TODO: get job request data
job_serial = get_job_from_serial()
print(job_serial)
host emulation script:
def send_job():
job_response = '''[06][1c]ans=33[0d]job=30925[0d]status=0;"ok"[0d]do=l[0d]add=;2.50[0d]ar=1[0d]
bcerin=;3.93[0d]bcerup=;-2.97[0d]crib=;64.00[0d]do=l[0d]ellh=;64.00[0d]engmask=;613l[0d]
erdrin=;0.00[0d]erdrup=;10.00[0d]ernrin=;2.00[0d]ernrup=;-8.00[0d]ersgin=;0.00[0d]
ersgup=;4.00[0d]gax=;0.00[0d]gbasex=;-5.30[0d]gcrosx=;-7.96[0d]kprva=;275[0d]kprvm=;0.55[0d]
ldpath=\\uscqx-tcpmain-at\lds\iot\do\800468.sdf[0d]lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]'''.encode('ascii')
writer(job_response)
def get_init_request():
req = p.readline()
print(req)
request = req.decode()[4:11]
# print(request)
if request == 'req=ini':
print('request == req=ini??? <<<<<<< cumple condicion y enviala respuesta')
send_init_response()
send_job()
while True:
# print(get_init_request())
get_init_request()
what I get in screen: main script
init method being active
bce
erd
condition was met init status=0
outside loop
ers
condition was met init status=0
inside while loop
trigger reset <<<--------------------
5782
`:lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]
outside loop
condition was met init status=0
outside loop
what I get in screen: host emulation script
b'[1c]req=ini[0d][0a][1e][1d]'
request == req=ini??? <<<<<<< cumple condicion y enviala respuesta
b''
b'[06][1c]req=33[0d][0a]job=5782[0d][0a][1e][1d]'
b''
b''
b''
b''
b''
b''

I'm suspect you're trying to write too much at once to a hardware buffer that is fairly small. Especially when dealing with low power hardware, assuming you can stuff an entire message into a buffer is not often correct. Even full modern PC's sometimes have very small buffers for legacy hardware like serial ports. You may find when you switch from development to actual hardware, that the RTS and DTR lines need to be used to determine when to send or receive data. This will be up to whoever designed the hardware unfortunately, as they are often also ignored.
I would try chunking your data transfer into smaller bits as a test to see if the whole message gets through. This is a quick and dirty first attempt that may have bugs, but it should get you down the right path:
def get_job_from_serial():
response = b'' #buffer for response
while True:
try:
response += serial_client.read() #read any available data or wait for timeout
#this technically could only be reading 1 char at a time, but any
#remotely modern pc should easily keep up with 9600 baud
except serial.SerialTimeoutException: #timeout probably means end of data
#you could also presumably check the length of the buffer if it's always
#a fixed length to determine if the entire message has been sent yet.
break
return response
def writer(command):
written = 0 #how many bytes have we actually written
chunksize = 128 #the smaller you go, the less likely to overflow
# a buffer, but the slower you go.
while written < len(command):
#you presumably might have to wait for p.dtr() == True or similar
#though it's just as likely to not have been implemented.
written += p.write(command[written:written+chunksize])
p.flush() #probably don't actually need this
P.S. I had to go to the source code for p.read_all (for some reason I couldn't find it online), and it does not do what I think you expect it does. The exact code for it is:
def read_all(self):
"""\
Read all bytes currently available in the buffer of the OS.
"""
return self.read(self.in_waiting)
There is no concept of waiting for a complete message, it just a shorthand for grab everything currently available.

Python's pyserial with interrupt mode

I have a device which works on serial communication. I am writing python code which will send some commands to get the data from the device.
There are three commands.
1.COMMAND - sop
Device does its internal calculation and sends below data
Response - "b'SOP,0,921,34,40,207,0,x9A\r\n'"
2.COMMAND - time
This gives a date time values which normally do not change untill the device is restarted
3.START - "\r\r" or (<cr><cr>)
This command puts the device in responsive mode after which it responds to above commands. This command is basically entering <enter> twice & only have to do once at the start.
Now the problem which I am facing is that, frequency of data received from sop command is not fixed and hence the data is received anytime. This command can also not be stopped once started, so if I run another command like time, and read the data, I do not receive time values and they are merged with the sop data sometime. Below is the code, I am using:
port = serial.Serial('/dev/ttyS0',115200) #Init serial port
port.write(("\r\r".encode())) #Sending the start command
bytesToRead = port.in_waiting #Checking data bytesize
res = port.read(bytesToRead) #Reading the data which is normally a welcome msg
port.reset_input_buffer() #Clearing the input serial buffer
port.reset_output_buffer() #Clearing the output serial buffer
port.write(("sop\r".encode())) #Sending the command sop
while True:
time.sleep(5)
bytesToRead = port.in_waiting
print(bytesToRead)
res = port.read(bytesToRead)
print(res)
port.reset_input_buffer()
port.write(("time\r".encode()))
res = port.readline()
print(res)
Using the above command I sometimes do not receive the value of time after executing its command or sometimes it is merged with the sop command. Also with the sop command, I received a lot of data during the sleep(5) out of which I need to get the latest data. If I do not include sleep(5), I miss the sop data and it is then received after executing the time command.
I was hoping if anyone can point me to right direction of how to design it in a better way. Also, I think this can easily be done using interrupt handler but I didn't found any code about pyserial interrupts. Can anyone please suggest some good code for using interrupts in pyserial.
Thanks

Instead of using time.sleep(), its preferred to use serialport.in_waiting which help to check the number of bytes available in rcv buffer.
So once there is some data is rcv buffer then only read the data using read function.
so following code sequence can be followed without having any delay
while True:
bytesToRead = port.in_waiting
print(bytesToRead)
if(bytestoRead > 0):
res = port.read(bytesToRead)
print(res)
port.reset_input_buffer()
# put some check or filter then write data on serial port
port.write(("time\r".encode()))
res = port.readline()
print(res)

I am taking a stab here: Your time.sleep(5) might be too long. Have you tried making the sleep really short, for example time.sleep(.300)? If the time data gets written back between the sop returns you will catch it before it gets merged with sop, but I am making an assumption here that it will send time data back, else there is anyway nothing more you can do on the server side (the python) code. I do believe it won't hurt to make the sleep less, it is anyway just sitting there waiting (polling) for communication.
Not having the having the same environment on my side, makes it difficult to answer, because I can't test my answer, so I hope this might help.

python socketserver occasionally stops sending (and receiving?) messages

I've been experiencing a problem with a socketserver I wrote where the socketserver will seem to stop sending and receiving data on one of the ports it uses (while the other port continues to handle data just fine). Interestingly, after waiting a minute (or up to an hour or so), the socketserver will start sending and receiving messages again without any observable intervention.
I am using the Eventlet socketing framework, python 2.7, everything running on an ubuntu aws instance with external apps opening persistent connections to the socketserver.
From some reading I've been doing, it looks like I may not be implementing my socket server correctly.
According to http://docs.python.org/howto/sockets.html:
fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited > > (shrug), or indicate how long they are (much better), or end by shutting down the connection.
I am not entirely sure that I am using a fix length message here (or am I?)
This is how I am receiving my data:
def socket_handler(sock, socket_type):
logg(1,"socket_handler:initializing")
while True:
recv = sock.recv(1024)
if not recv:
logg(1,"didn't recieve anything")
break
if len(recv) > 5:
logg(1,"socket handler: %s" % recv )
plug_id, phone_sid, recv_json = parse_json(recv)
send = 1
if "success" in recv_json and recv_json["success"] == "true" and socket_type == "plug":
send = 0
if send == 1:
send_wrapper(sock, message_relayer(recv, socket_type))
else:
logg(2, 'socket_handler:Ignoring received input: ' + str(recv) )
logg(1, 'Closing socket handle: [%s]' % str(sock))
sock.shutdown(socket.SHUT_RDWR)
sock.close()
"sock" is a socket object returned by the listener.accept() function.
The socket_handler function is called like so:
new_connection, address = listener.accept()
...<code omitted>...
pool.spawn_n(socket_handler, new_connection, socket_type)
Does my implementation look incorrect to anyone? Am I basically implementing a fixed length conversation protocol? What can I do to help investigate the issue or make my code more robust?
Thanks in advance,
T

You might be having buffering related problems if you're requesting to receive more bytes at the server (1024) than you're actually sending from the client.
To fix the problem, what's is usually done is encode the length of the message first and then the message itself. This way, the receiver can get the length field (which is of known size) and then read the rest of the message based on the decoded length.
Note: The length field is usually as many bytes long as you need in your protocol. Some protocols are 4-byte aligned and use a 32 bit field for this, but if you find that you've got enough with 1 or 2 bytes, then you can use that. The point here is that both client and server know the size of this field.

Detect a closed connection in python's telnetlib

I'm using python's telnetlib to connect to a remote telnet server. I'm having a hard time detecting if the connection is still open, or if the remote server closed it on me.
I will notice the connection is closed the next time I try to read or write to it, but would like to have a way to detect it on demand.
Is there a way to send some sort of an 'Are You There' packet without affecting the actual connection? The telnet RFC supports an "are you there" and "NOP" commands - just not sure how to get telnetlib to send them!

You should be able to send a NOP this way:
from telnetlib import IAC, NOP
...
telnet_object.sock.sendall(IAC + NOP)

I've noticed that for some reason sending only once was not enough ... I've "discovered it" by accident, I had something like this:
def check_alive(telnet_obj):
try:
if telnet_obj.sock: # this way I've taken care of problem if the .close() was called
telnet_obj.sock.send(IAC+NOP) # notice the use of send instead of sendall
return True
except:
logger.info("telnet send failed - dead")
pass
# later on
logger.info("is alive %s", check_alive(my_telnet_obj))
if check_alive(my_telnet_obj):
# do whatever
after a few runs I've noticed that the log message was saying "is alive True", but the code didn't entered the "if", and that the log message "telnet send failed - dead" was printed, so in my last implementation, as I was saying here, I'm just calling the .send() method 3 times (just in case 2 were not enough).
That's my 2 cents, hope it helps

Following up on David's solution, after close() on the interface, the sock attribute changes from being a socket._socketobject to being the integer 0. The call to .sendall fails with an AttributeError if the socket is closed, so you may as well just check its type.
Tested with Linux and Windows 7.

The best way to detect if a connection is close it's by socket object. So,it's easier to check it this way,
def is_connected(telnet_obj):
return telnet_obj.get_socket().fileno()
If it is closed return -1
I took this code from this question.