Assume you try to interact with a shell via serial port. What happens when you hit the end of the line is that a <space><carriage return> gets inserted. This is already uncomfortable when using screen or minicom because it usually just continues writing on the same line (the linefeed is missing), but it results in buggy code when you need to parse the output stream. I am wondering how I can configure my serial connection to simply do nothing at the end of the line.
Example:
$ python -i -c "import serial; s=serial.Serial('/dev/ttyUSB3', 115200, timeout=1.5)"
>>> s.write("echo \"123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123213\""); print s.readall().encode('string-escape')
211
echo "12312312312312312312312312312312312312312312312312312 \r31231231231231231231231231231231231231231231231231231231231231231231231231231231 \r23123123123123123123123123123123123123123123123123123123123123123123213"
Solving this by changing the parser is not an option, because the parsing is done by a third party library in my case. Also just setting the line length to a very high number might work but is not what I like to do. I'd rather have control over the port's behaviour when the line is full.
It might be that the problem has nothing to do with pyserial. If you send data with the write command to a serial port, it just sends the data to the serial port regardless of what you send (even binary data). The same applies to read.
So, the extra space and newline are inserted by the shell in the other end of the connection. There is probably no reasonable way around this problem without configuring the other end. For example, in Linux you might want to try setterm -linewrap off or just simply change the terminal width to be large enough wtih stty.
I think this question will receive more complete answers in https://unix.stackexchange.com/ if your remote terminal is a Unix/Linux.
Related
I have programmed a device (Trinket M0 3V) to interface with Python over a virtual COM port. When I test it in a serial monitor it works just fine, but when I switch to PySerial I run into problems. Specifically, I do the following:
import serial
ser = serial.Serial(physicalAddress, timeout = 5)
command = 'bogus\r'.encode()
ser.write(command)
response = ser.readline()
The response that I get is just the command I've written on the line preceding it. However, when I execute ser.readline() again I do get the response that I need. Since everything works fine when I try communicating on my serial monitor it suggests that there's something I'm missing in my use of PySerial, but that's about as far as I'm able to get.
Did you find the infos of https://www.thecodingforums.com/threads/pyserial-unexpected-local-echo.711453/ ?
It had to be linked to the way the echo is handled, and the link gives steps to see if the echo is local or distant, and more...
I didn't find anything about a local echo done by PySerial!
But on the serial monitor (which one?), it seems to be possible to configure it, hence the reason you didn't see it!? (https://community.platformio.org/t/solved-enable-just-local-echo-in-serial-monitor/539)
I have a small loop that is supposed to read some text data of multiple lines and unspecified length (sometimes rather large) from the serial port, and write it to a file:
while True:
data = port.readline()
file.write(data)
if not data:
break
I have the port setup correctly and the whole program works beautifully on a windows machine. When I run it on a Linux box it still receives the data but the software flow control fails to work - instead the control codes for the XON and XOFF (\S and \Q) are actually being written to the file. Causing overrun errors and destroying my data.
I've read and tried everything I can think of - it seems most examples of this are only expecting a few bytes and aren't using any flow control. I have also tried the miniterm that is included with pySerial and it gives the same results.
Any ideas?
Added lines for file and port:
file = open('temp.txt', 'ab') #Open the temporary file to append the incoming data to.
port = serial.Serial('/dev/ttyAMA0', baudrate=9600, bytesize=8, parity='N', stopbits=1, timeout=1, xonxoff=1, rtscts=0, dsrdtr=0)
There isn't much more to it than that except that I write a string to the other device to start sending, which works since I'm seeing the response back with the data. I've also tried printing to the console rather than writing the file. Same thing - overruns and printed control codes.
Update: I have also tried reading a single byte at a time instead of the whole line and have tried other baud rates - all with no change.
Update#2: Have been doing some reading and testing. Tested with minicom. Same results. Manually verified that port was set properly with 'stty -a' (although I don't trust this as it may save and restore the settings according to something I read). It's possibly a problem in termios?
Try removing unprintables characters from your received data.
data = data.rstrip()
perhaps you can try using this serial capture code. it has a Pyserial approach also but it is simplified. There is a split formula for the newline that may help with the flow control if you change it to meet your needs.
https://github.com/DaDaDadeo/Batch-Capture/blob/master/serial_to_tcp.py
I want to pull connection tables from a firewall. In some cases it can be more than 200k lines of
"TCP outside 46.33.77.20:53415 inside 10.16.25.63:80, idle 0:00:04, bytes 3230, flags UIOB"
and the like.
I've tried to implement both pexpect and telnetlib in order to grab these tables. Unfortunately both timeout and/or die with anything greater than 40k.
pexpect implementation:
connect.send("sho conn\n")
connect.expect("<--- More --->", timeout=360)
tmp_txt = connect.before
telnetlib implementation:
telnet.write("sho conn\n")
tmp_text = telnet.read_until("<--- More --->")
Is there a more robust method of grabbing this information? I control the number of lines given at a time with a pager value (prior to running this). Also - I'm monitoring the cpu on the firewall, so I know it's displaying the connections. Either there are too many or it's too fast for pexpect or telnetlib to keep up.
Thanks.
It looks like your approach is fine to me. I would also page the output (to keep firewall CPU low) and then capture the output a screen full at a time.
If you are running into timeout errors then why not modify your expect to be a loop that expects each line or specific lines of output (I presume it has a regular format) and then only send space when it gets the "more" line for the next screen. I've used this pattern a lot to deal with long streams of output that may pause at different places.
You mention that the python process dies, we can't help you there - unless you are more detailed about what exception is being raised.
I'm trying to talk to a home made card over a serial port, and is therefor using pySerial. In Hyperterminal, everything works fine. I can write:
$ audio on
and the audio is enabled, but if I use
ser = serial.Serial("COM1", 38400)
ser.write("audio on\r\n")
nothing happens. I can read incoming data however, so it's not something wrong with the communication. I doesn't help if I change \r\n to just \n or \r either.
EDIT: Sometime I actually get the feedback: No such command when sending the exact same command as works from HyperTerminal. The setup is also the exact same as in HyperTerminal.
Solved:
To make it work, I had to send one and one character, and ending the transmission with \r.
Get an oscilloscope (you've got one, right?) and watch the serial line. Send one character per second through it and see what comes up on the scope (set it to trigger on the start bit). Serial port bits are in the order: start, LSB .. MSB, parity, stop.
See if there are characters that don't get through, or if there's a pattern. Another possibility is that everything is actually making it out the port, and your board is dropping characters.
Triple check that the baud rate of the device is indeed 38400
Triple check parity, stop bits, etc
Be aware of signal degradation for serial transmissions over long distances (probably not your issue)
If all the above checkout try putting the string into a byte array and sending that through the write command. Just a guess.
Sending characters via Hyperterminal deliver characters at the speed you type them. Sending characters through pyserial they are delivered as a continuous stream. The receiver (especially at high baud rates) could drop them, depending on the nature of the receiver.
Also, when you send commands to an interpreter, you only need the \r terminator (without the \n), (this is all that is sent by hyperterm, normally). HOWEVER, if you are just displaying the values, you may need the \n to generate the new line.
I'd like to write a simple command line proxy in Python to sit between a Telnet/SSH connection and a local serial interface. The application should simply bridge I/O between the two, but filter out certain unallowed strings (matched by regular expressions). (This for a router/switch lab in which the user is given remote serial access to the boxes.)
Basically, a client established a Telnet or SSH connection to the daemon. The daemon passes the client's input out (for example) /dev/ttyS0, and passes input from ttyS0 back out to the client. However, I want to be able to blacklist certain strings coming from the client. For instance, the command 'delete foo' should not be allowed.
I'm not sure how best to approach this. Communication must be asynchronous; I can't simply wait for a carriage return to allow the buffer to be fed out the serial interface. Matching regular expressions against the stream seems tricky too, as all of the following must be intercepted:
delete foo(enter)
del foo(enter)
el foo(ctrl+a)d(enter)
dl(left)e(right) foo(enter)
...and so forth. The only solid delimiter is the CR/LF.
I'm hoping someone can point me in the right direction. I've been looking through Python modules but so far haven't come up with anything.
Python is not my primary language, so I'll leave that part of the answer for others. I do alot of security work, though, and I would urge a "white list" approach, not a "black list" approach. In other words, pick a set of safe commands and forbid all others. This is much much easier than trying to think of all the malicious possibilities and guarding against all of them.
As all the examples you show finish with (enter), why is it that...:
Communication must be asynchronous; I
can't simply wait for a carriage
return to allow the buffer to be fed
out the serial interface
if you can collect incoming data until the "enter", and apply the "edit" requests (such as the ctrl-a, left, right in your examples) to the data you're collecting, then you're left with the "completed command about to be sent" in memory where it can be matched and rejected or sent on.
If you must do it character by character, .read(1) on the (unbuffered) input will allow you to, but the vetting becomes potentially more problematic; again you can keep an in-memory image of the edited command that you've sent so far (as you apply the edit requests even while sending them), but what happens when the "enter" arrives and your vetting shows you that the command thus composed must NOT be allowed -- can you e.g. send a number of "delete"s to the device to wipe away said command? Or is there a single "toss the complete line" edit request that would serve?
If you must send every character as you receive it (not allowed to accumulate them until decision point) AND there is no way to delete/erase characters already sent, then the task appears to be impossible (though I don't understand the "can't wait for the enter" condition AT ALL, so maybe there's hope).
After thinking about this for a while, it doesn't seem like there's any practical, reliable method to filter on client input. I'm going to attempt this from another angle: if I can identify persistent patterns in warning messages coming from the serial devices (e.g. confirmation prompts) I may be able to abort reliably. Thanks anyway for the input!
Fabric is doing a similar thing.
For SSH api you should check paramiko.