We Keep Coding

How to do non blocking usb serial input in circuit python?

I am trying to interface a micropython board with python on my computer using serial read and write, however I can't find a way to read usb serial data in micropython that is non-blocking.
Basicly I want to call the input function without requiring an input to move on. (something like https://github.com/adafruit/circuitpython/pull/1186 but for usb)
I have tried using tasko, uselect (it failed to import the library and I can't find a download), and await functions. I'm not sure it there is a way to do this, but any help would be appreciated.

For CircuitPython there's supervisor.runtime.serial_bytes_available which may provide the building block for what you want to do.
This is discussed on Adafruit Forums: Receive commands from the computer via USB.

based on the relative new added usb_cdc buildin (>= 7.0.0) you can do something like this:
def read_serial(serial):
available = serial.in_waiting
while available:
raw = serial.read(available)
text = raw.decode("utf-8")
available = serial.in_waiting
return text
# main
buffer = ""
serial = usb_cdc.console
while True:
buffer += read_serial(serial)
if buffer.endswith("\n"):
# strip line end
input_line = buffer[:-1]
# clear buffer
buffer = ""
# handle input
handle_your_input(input_line)
do_your_other_stuff()
this is a very simple example.
the line-end handling could get very complex if you want to support universal line ends and multiple commands send at once...
i have created a library based on this: CircuitPython_nonblocking_serialinput

Paramiko capturing command output

I have an issue that has been giving me a headache for a few days. I am using the Paramiko module with Python 2.7.10 and I'd like to issue multiple commands to a Brocade router, but only return output from one of the given commands like so:
#!/usr/bin/env python
import paramiko, time
router = 'r1.test.example.com'
password = 'password'
username = 'testuser'
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(router, username=username, password=password)
print('Successfully connected to %s' % router)
remote_conn = ssh.invoke_shell()
output = remote_conn.recv(1000)
# Disable paging on Brocade.
remote_conn.send('terminal length 0\n')
# Check interface status.
remote_conn.send('show interfaces ethernet 0/1\n') # I only want output from this command.
time.sleep(2)
output = remote_conn.recv(5000)
print(output)
If I were to print the full output it would contain everything issued to the router, but I only want to see output from the show interfaces ethernet 0/1\n command.
Can anyone help with this issue?
One final thing I would like to ask. I want to filter through the output variable and check for occurrences of strings like "up" or "down", but I can't seem to get it to work because everything in the output appears to be on new lines?
For example:
If I iterate over the output variable in a for loop I get all of the characters in the variable like so:
for line in output:
print(line)
I get an output like this:
t
e
r
m
i
n
a
l
l
e
n
g
t
h
0
Any way around this?
Again,
Thanks in advance for any help.
Best regards,
Aaron C.

After reading all of the comment I have made the following changes:
#!/usr/bin/env python
import paramiko, time
router = 'r2.test.example.com'
password = 'password'
username = 'testuser'
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(router, username=username, password=password)
print('Successfully connected to %s' % router)
remote_conn = ssh.invoke_shell()
output = remote_conn.recv(1000)
# Disable paging on Brocade.
remote_conn.send('terminal length 0\n')
time.sleep(2)
# Clearing output.
if remote_conn.recv_ready():
output = remote_conn.recv(1000)
# Check interface status.
remote_conn.send('show interfaces ethernet 4/1\n') # I only want output from this command.
time.sleep(2)
# Getting output I want.
if remote_conn.recv_ready():
output = remote_conn.recv(5000)
print(output)
# Test: Check if interface is up.
for line in output.split('\n'):
if 'line protocol is up' in line:
print(line)
Everything works great now.
Thank you for all the help.
Best regards,
Aaron C.

For your second question: Though I am not specialist of paramiko, I see that function recv, according to the doc, returns a string. If you apply a for loop on a string, you will get characters (and not lines as one might perhaps expect). The newline is caused by your use of the print function as explained on this page, at paragraph 6.3.
I haven't studied what paramiko suggests to do. But why don't you treat the full string as a single entity? For example, you could check the presence of "up" as:
if "up" in output:
Or, if that suits your needs better, you could split the string into lines and then do whatever test you want to do:
for line in output.split('\n'):

If you can, the exec_command() call provides a simpler mechanism to invoke a command. I have seen Cisco switches abruptly drop connections that try exec_command(), so that may not be usable with Brocade devices.
If you must go the invoke_shell() route, be sure to clear all pending output after connecting and after send('terminal length 0\n'), checking recv_ready() before calling recv() to avoid blocking on reading data that might not ever arrive. Since you are controlling an interactive shell, sleep() calls might be needed to allow the server adequate time to process and send data, or it might be necessary to poll the output string to confirm that your last command completed by recognizing the shell prompt string.

Python serial port read delay

I'm playing around with this serial module in python. I have a little problem with it. I want my script to get a char from the console send it to an AVR board, and read back the response.
Everytime I read from the USB port, and print it out, I see the previous result. Why's that?
For example:
I write 5
I read nothing
I write 6
I read 5
import serial
import sys, time
port=serial.Serial(
port='/dev/ttyUSB0',\
baudrate=9600,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)
i=0
tmp = 0
while True:
tmp=raw_input('send: ')
port.write(tmp)
port.flushOutput()
print port.read(1)
port.flushInput()

From the documentation: "Writes are blocking by default, unless writeTimeout is set. For possible values refer to the list for timeout above." Try setting writeTimeout=0 as well in your constructor.

You are probably receiving a single unexpected byte on startup - either the microcontroller is sending it, or it might be noise from connecting a plug. As you are only reading a single byte for each string transmitted, you will always be off by one.
Instead of port.read(1), try:
while True:
tmp=raw_input('send: ')
port.write(tmp)
port.flushOutput()
print port.read(port.inWaiting())
port.flushInput()
This would also have happened if your typed in more than one character at the input prompt.

Determine if a socket can be read or used

Is there a way to determine if a socket can still be read? I tried isinstance and type but their type are the same as each other even if one has been closed and another is active so I'm trying to figure out how to see if a socket is alive and can be read or used or can not be so I can avoid this error: OSError: [WinError 10038] An operation was attempted on something that is not a socket

Based on your comment, ruler, I think I know what you're getting at.
What you want to do is break out of your while loop once your data stream reaches the end THEN close the socket after you're said and done. In the past, I have done the folowing:
while 1:
data = conn.recv(1024)
if not data: break
conn.send(data)
conn.close()
The while loop will remain so long as data streams in (based on the if loop). Otherwise, it will break out of the loop and, finally, close the socket. This way, you won't even have to worry about your socket closing on your mid-loop... just check for your stream and close it once all the data has been received. Hopefully, this helps.

On windows, you can do this by running the netstat command using the python subprocess module and then parsing the output.
import subprocess
output, error = subprocess.Popen(['netstat','-n'],stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
# then parse the port numbers / states out of the variable *output*

TCP Socket file transfer

I'm trying to write a secure transfer file program using Python and AES and i've got a problem i don't totally understand. I send my file by parsing it with 1024 bytes chunks and sending them over but the server side who receive the data crashes ( I use AES CBC therefore my data length must be a multiple of 16 bytes ) and the error i get says that it is not.
I tried to print the length of the data sent by the client on the client side and the length of the data received on the server and it shows that the client is sending exactly 1024 bytes each time like it's supposed to, but the server side shows that at some point in time, a received packet is not and so less than 1024 bytes ( for example 743 bytes ).
I tried to put a time.sleep(0.5) between each socket send on the client side and it seems to work. Is it possible that it is some kind of socket buffer failure on the server side ? That too much data is being send too fast by the client and that it breaks somehow the socket buffer on the server side so the data is corrupted or vanish and the recv(1024) only receive a broken chunk? That's the only thing i could think of, but this may also be completely false, if anyone has an idea of why this is not working properly it would be great ;)
Following my idea i tried :
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 32768000)
print socket.SO_RCVBUF
I tried to put a 32mbytes buffer on the server side but On Windows XP it shows 4098 on the print and on linux it shows only 8. I don't know how i must interpret this, the only thing i know is that it seems that it doesn't have a 32mbytes buffer so the code doesn't work.
Well it's been a really long post, i hope some of you had the courage to read it all to here ! i'm totally lost there so if anyone has any idea about this please share it :D
Thanks to Faisal my code is here :
Server Side: ( count is my filesize/1024 )
while 1:
txt=self.s.recv(1024)
if txt == " ":
break
txt = self.cipher.decrypt(txt)
if countbis == count:
txt = txt.rstrip()
tfile.write(txt)
countbis+=1
Client side :
while 1:
txt= tfile.read(1024)
if not txt:
self.s.send(" ")
break
txt += ' ' * (-len(txt) % 16)
txt = self.cipher.encrypt(txt)
self.s.send(txt)
Thanks in advance,
Nolhian

Welcome to network programming! You've just fallen into the same mistaken assumption that everyone makes the first time through in assuming that client sends & server recives should be symmetric. Unfortunately, this is not the case. The OS allows reception to occur in arbitrarily sized chunks. It's fairly easy to work around though, just buffer your data until the amount you've read in equals the amount you wish to receive. Something along the lines of this will do the trick:
buff=''
while len(buff) < 1024:
buff += s.recv( 1024 - len(buff) )

TCP is a stream protocol, it doesn't conserve message boundaries, as you have just discovered.

As others have pointed out you're probably processing an incomplete message. You need to either have fixed sized messages or have a delimiter (don't forget to escape your data!) so you know when a complete message has been received.

What TCP can guarantee is that all your data arrives, in the right order, at some point. (Unless something unexpected happens, by which it won't arrive.) But it's very possible that the data you send will still arrive in chunks. Much of it is because of limited send- and receive-buffers. What you should do is to continue doing your recv calls until you have enough data to process it. You might might have to call send multiple times; use its return value to keep track of how much data has been sent/buffered so far.
When you do print socket.SO_RCVBUF, you actually print the symbolic SO_RCVBUF contant (except that Python doesn't really have constants); the one used to tell setsockopt what you want to change. To get the current value, you should instead call getsockopt.

Not related to TCP (as that has been answered already), but appending to a string repeatedly will be rather inefficient if you're expecting to receive a lot. It might be better to append to a list and then turn the list into a string when you finished receiving by using ''.join(list).

For many applications, the complexities of TCP are neatly abstracted by Python's asynchat module.

Here is the nice snippet of code that I wrote some time ago, may be not the best , but it could be good example of big files transfer over the local network. http://setahost.com/sending-files-in-local-network-with-python/

As mentioned above
TCP is a stream protocol
You can try this code, where the data is your original data, you can read it from the file or user input
Sender
import socket as s
sock = s.socket(s.AF_INET, s.SOCK_STREAM)
sock.connect((addr,5000))
sock.sendall(data)
finish = t.time()
Receiver
import socket as s
sock = s.socket(s.AF_INET, s.SOCK_STREAM)
sock.setsockopt(s.SOL_SOCKET, s.SO_REUSEADDR, 1)
sock.bind(("", 5000))
sock.listen(1)
conn, _ = sock.accept()
pack = []
while True:
piece = conn.recv(8192)
if not piece:
break
pack.append(piece.decode())