We Keep Coding

Python pyserial one write delay

I'm having a weird issue with pyserial, using Python 3.6.9, running under WSL Ubuntu 18.4.2 LTS
I've set up a simple function to send GCODE commands to a serial port:
def gcode_send(data):
print("Sending: " + data.strip())
data = data.strip() + "\n" # Strip all EOL characters for consistency
s.write(data.encode()) # Send g-code block to grbl
grbl_out = s.readline().decode().strip()
print(grbl_out)
It sort of works, but every command I send is 'held' until the next is sent.
e.g.
I send G0 X0 > the device doesn't react
I send G0 X1 > the device reacts to G0 X0
I send G1 X0 > the device reacts to G0 X1
and so on...
My setup code is:
s = serial.Serial(com, 115200)
s.write("\r\n\r\n".encode()) # Wake up grbl
time.sleep(2) # Wait for grbl to initialize
s.flushInput() # Flush startup text in serial input
I can work around the delay for now, but it's quite annoying and I can't find anyone else experiencing the same. Any idea what could be causing this?

There might be a lot of problems here, but rest assured that the pyserial is not causing it. It uses the underlying OS's API to communicate with the UART driver. That being said you first have to test your code with real Linux to see whether WSL is causing it. I.e. whether a Linux and Windows UART buffers are correctly synced.
I am sorry that I cannot tell whether a problem is in your code or not because I do not know the device you are using, so I cannot guess what is happening on its end of communication channel. Have in mind that Windows alone can act weirdly in best of circumstances, so, prepare yourself for some frustrations here. Check your motherboard or USB2Serial converter drivers or whatever hw you are using.
Next thing, you should know that sometimes, communication gets confusing if timeouts aren't set. Why? Nobody really knows. So try setting timeouts. Check whether you need software Xon/Xoff turned on or not, and other RS232 parameters that might be required by the device you are communicating with.
Also, see what is going on with s.readline(), I wouldn't personally use it. Timeouts might help or you can use s.read(1024) with timeouts. I do not remember right now, but see whether pyserial supports asynchronous communication. If it does, you can try using it instead of standard blocking mode.
Also, check whether you have to forcefully flush the serial buffer after s.write() or add a sleep after it. It might happen that the device doesn't get the message but the read request is activated. As the device didn't receive the command it doesn't respond. After you send another command, IO buffer is flushed and the previous one is delivered and so forth. Serial communication is fun, but when it hits a snag it can be a real P in the A, believe me.
Ow, a P.S. Check whether the device sends "\r\n\r\n" or "\r\n" only, or "\r" or "\n" in response. s.readline() might get confused. For a start, try putting there two s.readline()s one after another and print out each output. If the device sends double EOL then the one s.readline() is stopping on the empty line and your program receives an empty response, when you send another command s.readline() goes through the buffer and returns a full line that is already there but not read before.

Here it goes. The code promissed in the comment. Big portions of it removed and error checks too.
It is a typing terminal for using PyS60 Python console on Nokia smartphones in the Symbian series via bluetooth. Works fantastically.
from serial import *
from thread import start_new_thread as thread
from time import sleep
import sys, os
# Original code works on Linux too
# The following code for gettin one character from stdin without echoing it on terminal
# has its Linux complement using tricks from Python's stdlib getpass.py module
# I.e. put the terminal in non-blocking mode, turn off echoing and use sys.stdin.read(1)
# Here is Win code only (for brevity):
import msvcrt
def getchar ():
return msvcrt.getch()
def pause ():
raw_input("\nPress enter to continue . . .")
port = raw_input("Portname: ")
if os.name=="nt":
nport = ""
for x in port:
if x.isdigit(): nport += x
port = int(nport)-1
try:
s = Serial(port, 9600)
except:
print >> sys.stderr, "Cannot open the port!\nThe program will be closed."
pause()
sys.exit(1)
print "Port ready!"
running = 1
def reader():
while running:
try:
msg = s.read()
# If timeout is set
while msg=="":
msg = s.read()
sys.stdout.write(msg)
except: sleep(0.001)
thread(reader,())
while 1:
try: c = getchar()
except Exception, e:
running = 0
print >> sys.stderr, e
s.write('\r\n\x04')
break
if c=='\003' or c=='\x04':
running = 0
s.write('\r\n\x04')
break
s.write(c)
s.close()
pause()

python socket programming for transferring a photo

I'm new to socket programming in python. Here is an example of opening a TCP socket in a Mininet host and sending a photo from one host to another. In fact I changed the code that I had used to send a simple message to another host (writing the received data to a text file) in order to meet my requirements. Although when I implement this revised code, there is no error and it seems to transfer correctly, I am not sure whether this is a correct way to do this transmission or not. Since I'm running both hosts on the same machine, I thought it may have an influence on the result. I wanted to ask you to check whether this is a correct way to transfer or I should add or remove something.
mininetSocketTest.py
#!/usr/bin/python
from mininet.topo import Topo, SingleSwitchTopo
from mininet.net import Mininet
from mininet.log import lg, info
from mininet.cli import CLI
def main():
lg.setLogLevel('info')
net = Mininet(SingleSwitchTopo(k=2))
net.start()
h1 = net.get('h1')
p1 = h1.popen('python myClient2.py')
h2 = net.get('h2')
h2.cmd('python myServer2.py')
CLI( net )
#p1.terminate()
net.stop()
if __name__ == '__main__':
main()
myServer2.py
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('10.0.0.1', 12345))
buf = 1024
f = open("2.jpg",'wb')
s.listen(1)
conn , addr = s.accept()
while 1:
data = conn.recv(buf)
print(data[:10])
#print "PACKAGE RECEIVED..."
f.write(data)
if not data: break
#conn.send(data)
conn.close()
s.close()
myClient2.py:
import socket
import sys
f=open ("1.jpg", "rb")
print sys.getsizeof(f)
buf = 1024
data = f.read(buf)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('10.0.0.1',12345))
while (data):
if(s.sendall(data)):
#print "sending ..."
data = f.read(buf)
print(f.tell(), data[:10])
else:
s.close()
s.close()

This loop in client2 is wrong:
while (data):
if(s.send(data)):
print "sending ..."
data = f.read(buf)
As the send
docs say:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this topic, consult the Socket Programming HOWTO.
You're not even attempting to do this. So, while it probably works on localhost, on a lightly-loaded machine, with smallish files, it's going to break as soon as you try to use it for real.
As the help says, you need to do something to deliver the rest of the buffer. Since there's probably no good reason you can't just block until it's all sent, the simplest thing to do is to call sendall:
Unlike send(), this method continues to send data from bytes until either all data has been sent or an error occurs. None is returned on success. On error, an exception is raised…
And this brings up the next problem: You're not doing any exception handling anywhere. Maybe that's OK, but usually it isn't. For example, if one of your sockets goes down, but the other one is still up, do you want to abort the whole program and hard-drop your connection, or do you maybe want to finish sending whatever you have first?
You should at least probably use a with clause of a finally, to make sure you close your sockets cleanly, so the other side will get a nice EOF instead of an exception.
Also, your server code just serves a single client and then quits. Is that actually what you wanted? Usually, even if you don't need concurrent clients, you at least want to loop around accepting and servicing them one by one.
Finally, a server almost always wants to do this:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Without this, if you try to run the server again within a few seconds after it finished (a platform-specific number of seconds, which may even depend whether it finished with an exception instead of a clean shutdown), the bind will fail, in the same way as if you tried to bind a socket that's actually in use by another program.

First of all, you should use TCP and not UDP. TCP will ensure that your client/server has received the whole photo properly. UDP is more used for content streaming.
Absolutely not your use case.

Python 2.7 Script works with breakpoint in Debug mode but not when Run

def mp_worker(row):
ip = row[0]
ip_address = ip
tcp_port = 2112
buffer_size = 1024
# Read the reset message sent from the sign when a new connection is established
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print('Connecting to terminal: {0}'.format(ip_address))
s.connect((ip_address, tcp_port))
#Putting a breakpoint on this call in debug makes the script work
s.send(":08a8RV;")
#data = recv_timeout(s)
data = s.recv(buffer_size)
strip = data.split("$", 1)[-1].rstrip()
strip = strip[:-1]
print(strip)
termStat = [ip_address, strip]
terminals.append(termStat)
except Exception as exc:
print("Exception connecting to: " + ip_address)
print(exc)
The above code is the section of the script that is causing the problem. It's a pretty simple function that connects to a socket based on a passed in IP from a DB query and receives a response that indicates the hardware's firmware version.
Now, the issue is that when I run it in debug with a breakpoint on the socket I get the entire expected response from the hardware, but if I don't have a breakpoint in there or I full on Run the script it only responds with part of the expected message. I tried both putting a time.sleep() in after the send to see if it would get the entire response and I tried using the commented out recv_timeout() method in there which uses a non-blocking socket and timeout to try to get an entire response, both with the exact same results.
As another note, this works in a script with everything in one main code block, but I need this part separated into a function so I can use it with the multiprocessing library. I've tried running it on both my local Windows 7 machine and on a Unix server with the same results.

I'll expands and reiterate on what I've put into a comment moment ago. I am still not entirely sure what is behind the different behavior in either scenario (apart from timing guess apparently disproved by an attempt to include sleep.
However, it's somewhat immaterial as stream sockets do not guarantee you get all the requested data at once and in chunks as requested. This is up for an application to deal with. If the server closes the socket after full response was sent, you could replace:
data = s.recv(buffer_size)
with recv() until zero bytes were received, this would be equivalent of getting 0 (EOF) from from the syscall:
data = ''
while True:
received = s.recv(buffer_size)
if len(received) == 0:
break
data += received
If that is not the case, you would have to rely on fixed or known (sent in the beginning) size you want to consider together. Or deal with this on protocol level (look for characters, sequences used to signal message boundaries.

I just recently found out a solution here, and thought I'd post it in case anyone else has issue, I just decided to try and call socket.recv() before calling socket.send() and then calling socket.recv() again afterwards and it seems to have fixed the issue; I couldn't really tell you why it works though.
data = s.recv(buffer_size)
s.send(":08a8RV;")
data = s.recv(buffer_size)

Can select() be used with files in Python under Windows?

I am trying to run the following python server under windows:
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
I get the error message (10038, 'An operation was attempted on something that is not a socket'). This probably relates back to the remark in the python documentation that "File objects on Windows are not acceptable, but sockets are. On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don’t originate from WinSock.". On internet there are quite some posts on this topic, but they are either too technical for me or simply not clear. So my question is: is there any way the select() statement in python can be used under windows? Please add a little example or modify my code above. Thanks!

Look like it does not like sys.stdin
If you change input to this
input = [server]
the exception will go away.
This is from the doc
Note:
File objects on Windows are not acceptable, but sockets are. On Windows, the
underlying select() function is provided by the WinSock library, and does not
handle file descriptors that don’t originate from WinSock.

I don't know if your code has other problems, but the error you're getting is because of passing input to select.select(), the problem is that it contains sys.stdin which is not a socket. Under Windows, select only works with sockets.
As a side note, input is a python function, it's not a good idea to use it as a variable.

Of course and the answers given are right...
you just have to remove the sys.stdin from the input but still use it in the iteration:
for s in inputready+[sys.stdin]:

Python: Non-blocking socket or Asynchronos I/O

I am new to Python and currently have to write a python socket to be run as a script that communicates with a device over TCP/IP (a weather station).
The device acts as the Server Side (listening over IP:PORT, accepting connection, receiving request, transferring data).
I only need to send one message, receive the answer and then peacefully and nicely shutdown and close the socket.
try:
comSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
sys.stderr.write("[ERROR] %s\n" % msg[1])
sys.exit(1)
try:
comSocket.connect((''))
except socket.error, msg:
sys.stderr.write("[ERROR] %s\n" % msg[1])
sys.exit(2)
comSocket.send('\r')
comSocket.recv(128)
comSocket.send('\r')
comSocket.recv(128)
comSocket.send('\r\r')
comSocket.recv(128)
comSocket.send('1I\r\r3I\r\r4I\r\r13I\r\r5I\r\r8I\r\r7I\r\r9I\r\r')
rawData = comSocket.recv(512)
comSocket.shutdown(1)
comSocket.close()
The problem I'm having is:
The communication channel is unreliable, the device is slow. So, sometimes the device response with message of length 0 (just an ACK), the my code will freeze and wait for response forever.
This piece of code contains the portion that involves SOCKET, the whole code will be run under CRON so freezing is not a desirable behavior.
My question is:
What would be the best way in Python to handle that behavior, so that the code doesn't freeze and wait forever but will attempt to move on to the next send (or such).

You can try a timeout approach, like Russel code or you can use a non-blocking socket, as shown in the code below. It will never block at socket.recv and you can use it inside a loop to retry as many times you want. This way your program will not hang at timeout. This way, you can test if data is available and if not, you can do other things and try again later.
socket.setblocking(0)
while (retry_condition):
try:
data = socket.recv(512)
except socket.error:
'''no data yet..'''

I'd recommend eventlet and green threads for this.
Twisted is a good library but a little steep learning curve for such a simple use case.
Check out some examples here.

Try, before receiving, putting a timeout on the socket:
comSocket.settimeout(5.0)
try:
rawData = comSocket.recv(512)
except socket.timeout:
print "No response from server"