I have a small loop that is supposed to read some text data of multiple lines and unspecified length (sometimes rather large) from the serial port, and write it to a file:
while True:
data = port.readline()
file.write(data)
if not data:
break
I have the port setup correctly and the whole program works beautifully on a windows machine. When I run it on a Linux box it still receives the data but the software flow control fails to work - instead the control codes for the XON and XOFF (\S and \Q) are actually being written to the file. Causing overrun errors and destroying my data.
I've read and tried everything I can think of - it seems most examples of this are only expecting a few bytes and aren't using any flow control. I have also tried the miniterm that is included with pySerial and it gives the same results.
Any ideas?
Added lines for file and port:
file = open('temp.txt', 'ab') #Open the temporary file to append the incoming data to.
port = serial.Serial('/dev/ttyAMA0', baudrate=9600, bytesize=8, parity='N', stopbits=1, timeout=1, xonxoff=1, rtscts=0, dsrdtr=0)
There isn't much more to it than that except that I write a string to the other device to start sending, which works since I'm seeing the response back with the data. I've also tried printing to the console rather than writing the file. Same thing - overruns and printed control codes.
Update: I have also tried reading a single byte at a time instead of the whole line and have tried other baud rates - all with no change.
Update#2: Have been doing some reading and testing. Tested with minicom. Same results. Manually verified that port was set properly with 'stty -a' (although I don't trust this as it may save and restore the settings according to something I read). It's possibly a problem in termios?
Try removing unprintables characters from your received data.
data = data.rstrip()
perhaps you can try using this serial capture code. it has a Pyserial approach also but it is simplified. There is a split formula for the newline that may help with the flow control if you change it to meet your needs.
https://github.com/DaDaDadeo/Batch-Capture/blob/master/serial_to_tcp.py
Related
Is there a way to capture and write very fast serial data to a file?
I'm using a 32kSPS external ADC and a baud rate of 2000000 while printing in the following format: adc_value (32bits) \t millis()
This results in ~15 prints every 1 ms. Unfortunately every single soulution I have tried fails to capture and store real time data to a file. This includes: Processing sketches, TeraTerm, Serial Port Monitor, puTTY and some Python scripts. All of them are unable to log the data in real time.
Arduino Serial Monitor on the other hand is able to display real time serial data, but it's unable to log it in a file, as it lacks this function.
Here's a printscreen of the serial monitor in Arduino with the incoming data:
One problematic thing is probably that you try to do a write each time you receive a new record. That will waste a lot of time writing data.
Instead try to collect the data into buffers, and as a buffer is about to overflow write the whole buffer in a single and as low-level as possible write call.
And to not stop the receiving of the data to much, you could use threads and double-buffering: Receive data in one thread, write to a buffer. When the buffer is about to overflow signal a second thread and switch to a second buffer. The other thread takes the full buffer and writes it to disk, and waits for the next buffer to become full.
After trying more than 10 possible solutions for this problem including dedicated serial capture software, python scripts, Matlab scripts, and some C projects alternatives, the only one that kinda worked for me proved to be MegunoLink Pro.
It does not achieve the full 32kSPS potential of the ADC, rather around 12-15kSPS, but it is still much better than anything I've tried.
Not achieving the full 32kSPS might also be limited by the Serial.print() method that I'm using for printing values to the serial console. By the way, the platform I've been using is ESP32.
Later edit: don't forget to edit MegunoLinkPro.exe.config file in the MegunoLink Pro install directory in order to add further baud rates, like 1000000 or 2000000. By default it is limited to 500000.
I've got a custom script listening on port 161 for UDP packets to come in.
It listens fine, receives the string fine - and when I send a message from a test script (on another box), it displays fine in a log, etc.
I'm gathering the UDP data as follows:
data, addr = sock.recvfrom(1024)
data contains the string with the information I need.
When performing a tcpdump on the interface that the data is coming in, it looks normal, such as:
.1.3.6.1.4.1.3375.2.1.1.2.12.6 .1.3.6.1.4.1.3375.2.1.1.2.12.6 public "THIS IS THE TRAP" .1.3.6.1.4.1.3375.2.1.1.2.12.6 .1.3.6.1.4.1.3375.2.1.1.2.12.6
When I take that incoming data (in python) and print it, or output it to a file, I get a bunch of ESC sequences, or just otherwise unprintable data in the log file.
Everything inside of the Quote is preserved.
I've been able to strip out the ESC sequences and store the 'good stuff' inside of the quotes, but I'm losing my OID's. It's almost as if python thinks those ascii characters are something else.
I did notice that when trying to save the garbled data, if I change encoding to Latin-1 -- it becomes somewhat readable...but still there's some garbled characters in there.
I've tried to duplicate this matter here at home - but no matter what text I feed through my test.py to the listener on port 161, it comes out just fine and readable. This was implemented in a test environment at my work. PS I am not a programmer, but a network guy.
If it matters, the device sending SNMP traps out is an F5 LTM.
I know this is a pretty general question, so I appreciate anyone just taking the time to read my question in its entirety and spend a few minutes thinking about it.
I am working on a project for work and I am stuck on a part where I need to monitor a serial line and listen for certain words using python
so the setup is that we have a automated RAM testing machine that tests RAM one module at a time and interacts with software that came with the machine via serial. The software that came with the RAM tester is for monitoring/configuring the testing process, it also displays all of the information from the SPD chip from each module. while the RAM tester was running I ran a serial port monitoring program and I was able to see the same information that it displays in the software. The data I'm interested in is the speed of the RAM and the pass/fail result, both of which I was able to find in the data I monitored coming over the serial line. There are only 5 different speeds of RAM that we test, so I was hoping to have python monitor the serial line and wait for the speed of the RAM and the pass/fail results to come across. once python detects the speed of the RAM, and if it passes, I will have python write to an Arduino, and the Arduino will control a conveyor belt that will sort the ram by speed.
my idea is to have a variable for each of the RAM speeds and set the variables to 0. when python detects the RAM speed from the serial line it will set the corresponding variable to 1. then when the test is over the results, either pass or fail, will come over the serial line. this is where I am going to try to use a if statement. I imagine it would look something like this:
if PC-6400 == 1 and ser.read() == pass
ser.write(PC-6400) #serial write to the arduino
I know the use of the ser.read() == pass is incorrect and that's where I'm stuck. I do not know how to use a ser.read() function to look for certain words. I need it to look for the ram speed (in this case its PC-6400) and the word pass but I have not been successful in getting it to find either. I am currently suck in is getting python to detect the RAM speed so it can change the value of the variable. would it be something close to this?
if ser.read() == PC-6400
PC-6400 = 1
This whole thing is a bit difficult for me to explain and I hope it all makes sense, I thank you in advance if anyone can give me some advice on how to get this going. I am pretty new to python and this is the most adventurous project I have worked on using python so far.
I'm still a bit confused, but here's a very basic example to hopefully get you started.
This will read from a serial port. It will attempt to read 512 bytes (which just means 512 characters from a string). If 512 bytes aren't available then it will wait forever, so make sure you set a timeout when you made the serial connection.
return_str = ser.read(size = 512)
You can also see how many bytes are waiting to be read:
print "num bytes available = ", ser.inWaiting()
Once you have a string, you can check words within the string:
if "PASS" in return_str:
print "the module passed!"
if "PC-6400" in return_str:
print "module type is PC-6400"
To do something similar, but with variables:
# the name pass is reserved
pass_flag = "PASS"
PC6400 = 0
if pass_flag in return_str and "PC-6400" in return_str:
PC6400 = 1
Keep in mind that you it is possible to read part of a line if you are too quick. You can add delays by using timeouts or the time.sleep() function. You might also find you need to wait for a couple of seconds after initiating the connection as the arduino resets when you connect. This will give it time to reset before you try and read/write.
Assume you try to interact with a shell via serial port. What happens when you hit the end of the line is that a <space><carriage return> gets inserted. This is already uncomfortable when using screen or minicom because it usually just continues writing on the same line (the linefeed is missing), but it results in buggy code when you need to parse the output stream. I am wondering how I can configure my serial connection to simply do nothing at the end of the line.
Example:
$ python -i -c "import serial; s=serial.Serial('/dev/ttyUSB3', 115200, timeout=1.5)"
>>> s.write("echo \"123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123123213\""); print s.readall().encode('string-escape')
211
echo "12312312312312312312312312312312312312312312312312312 \r31231231231231231231231231231231231231231231231231231231231231231231231231231231 \r23123123123123123123123123123123123123123123123123123123123123123123213"
Solving this by changing the parser is not an option, because the parsing is done by a third party library in my case. Also just setting the line length to a very high number might work but is not what I like to do. I'd rather have control over the port's behaviour when the line is full.
It might be that the problem has nothing to do with pyserial. If you send data with the write command to a serial port, it just sends the data to the serial port regardless of what you send (even binary data). The same applies to read.
So, the extra space and newline are inserted by the shell in the other end of the connection. There is probably no reasonable way around this problem without configuring the other end. For example, in Linux you might want to try setterm -linewrap off or just simply change the terminal width to be large enough wtih stty.
I think this question will receive more complete answers in https://unix.stackexchange.com/ if your remote terminal is a Unix/Linux.
I have been attempting to have a Raspberry Pi interface with an embedded circuit using the UART interface. The UART interface on the Pi is in working order and I can receive messages from the circuit, though I am having trouble sending messages to the circuit.
I am using Python 3.3 with Pyserial 2.7. Sample code is available, though it uses Pyserial 2.6. When used with older versions of Python (<2.6), ser.write() accepts strings, but now it only accepts bytearrays.
The problem I am having is in sending carriage returns... The old code supposedly functioned with just:
ser.write("L1\r")
but now I am using the following:
ser.write(bytearray("L1\r", "ascii"))
The circuit does not respond to the command. I think the resultant message is sending \r as two individual characters rather than a carriage return. How would I make sure my code is outputting commands appended with carriage returns?
Notes: I can reasonably expect that the circuit is working well and that the Pi's UART interface is functional. The circuit is an Atlas Scientific Dissolved Oxygen Circuit. The circuit's documentation demands that commands be written in the form l1<cr> or L1<CR>.
Relevant links:
Old sample code (https://www.atlas-scientific.com/_files/code/pi_sample_code.pdf)
Documentation describing write method (http://pyserial.sourceforge.net/pyserial_api.html#classes)
Thanks in advance!
EDIT: Netch makes a strong point: ser.write(b'L1\r') works and is much cleaner. Both methods, however, ARE sending a correct '\r' sequence.. The problem is that the circuit still does not regard L1\r as a valid command. At this point, I think my issue may be some property of my serial port.
My port is declared as such:
ser = serial.Serial(
port = '/dev/ttyAMA0',
baudrate = 38400,
bytesize = serial.EIGHTBITS,
parity = serial.PARITY_NONE,
stopbits = serial.STOPBITS_ONE,
timeout = 1
)
This port declaration is done with accordance to the circuit's datasheet (I can only post two links unfortunately :( Google brings it up easily).
[EDIT] For future viewers, I just want to point out that for the newer EZO version of the circuit, the command is "L,1" or really "L,1\r"
[/EDIT]
This is a known issue with the circuit. The first read after power up will fail. instead of prepending a \r with every read, just send a bogus command with \r included and then reset input buffer
ser.write('bogus\r'.encode()) # EDIT: had to add .encode() to get it to work. see https://stackoverflow.com/questions/22275079/pyserial-write-wont-take-my-string
ser.flushInput() # or for pyserial 3+ ser.reset_input_buffer()
#now do your thing
EDIT: had to add .encode() to get it to work. see pySerial write() won't take my string
After powering up the EZO™ class circuit when it is in UART mode the
first command sent to it will comeback as an error. This is because
the UART buffer will show that it has received a character during
power up. Simply send a blank character to the pH circuit after it is
powered up, this will clear the buffer.
I have found a solution!! Unfortunately, I cannot explain how it works. Perhaps anyone reading this could elaborate on it and give a proper explanation!
The circuit's documentation demands commands be in the form CMD<CR>. Indeed, sample code provided by the manufacturer sends the L1 command through pyserial as ser.write("L1\r").
Now that ser.write() demands bytes however, I have found that ser.write(b'L1\r') does not work.. The command is received though it is somehow unknown to the circuit.
After toying around for a while, I have discovered that ser.write(b'\rL1\r') works! The debugging led flashes red once before processing the command. It seems like I just need to send a 'dummy command' to get the circuit's attention!
I am not sure if this is the fault of pyserial, the circuit, or my own ignorance. If anyone can shed some light on this, it would be much appreciated! :D
I have linked here the circuits documentation in case anyone is interested. https://www.atlas-scientific.com/_files/_datasheets/_circuit/DO_Circuit_5.0.pdf