We Keep Coding

Cannot get reset_input_buffer() function to work at all in Pyserial 3.5. Does anyone have any idea what may be happening?

I am trying to simulate a communication protocol where I am following a pattern, so I constantly loop though looking for the same set of characters to reply information. I'm using an RS-232 adapter and the protocol I am simulating is asynchronous and half-duplex where the rx/tx lines are tied together by design and that causes a sort of echo when reading after writing.
That said, I need to be able to clear the input buffer after every write I send out in order to avoid reading what I just wrote. So whenever I use reset_input_buffer() it does not clear the last message I sent out. I have tried to fix this using a couple of methods, such as: using reset_output_buffer() together with reset_input_buffer(), using reset_input_buffer() twice, and using flush(). None of these methods make any difference, the only other method that works to clear the buffer is closing and immediately opening the port but this causes a delay that messes with the timing as it is critical at certain times.
I'm open to any suggestions, please help!

pyserial 2.7, python 3.3, sending carriage returns

I have been attempting to have a Raspberry Pi interface with an embedded circuit using the UART interface. The UART interface on the Pi is in working order and I can receive messages from the circuit, though I am having trouble sending messages to the circuit.
I am using Python 3.3 with Pyserial 2.7. Sample code is available, though it uses Pyserial 2.6. When used with older versions of Python (<2.6), ser.write() accepts strings, but now it only accepts bytearrays.
The problem I am having is in sending carriage returns... The old code supposedly functioned with just:
ser.write("L1\r")
but now I am using the following:
ser.write(bytearray("L1\r", "ascii"))
The circuit does not respond to the command. I think the resultant message is sending \r as two individual characters rather than a carriage return. How would I make sure my code is outputting commands appended with carriage returns?
Notes: I can reasonably expect that the circuit is working well and that the Pi's UART interface is functional. The circuit is an Atlas Scientific Dissolved Oxygen Circuit. The circuit's documentation demands that commands be written in the form l1<cr> or L1<CR>.
Relevant links:
Old sample code (https://www.atlas-scientific.com/_files/code/pi_sample_code.pdf)
Documentation describing write method (http://pyserial.sourceforge.net/pyserial_api.html#classes)
Thanks in advance!
EDIT: Netch makes a strong point: ser.write(b'L1\r') works and is much cleaner. Both methods, however, ARE sending a correct '\r' sequence.. The problem is that the circuit still does not regard L1\r as a valid command. At this point, I think my issue may be some property of my serial port.
My port is declared as such:
ser = serial.Serial(
port = '/dev/ttyAMA0',
baudrate = 38400,
bytesize = serial.EIGHTBITS,
parity = serial.PARITY_NONE,
stopbits = serial.STOPBITS_ONE,
timeout = 1
)
This port declaration is done with accordance to the circuit's datasheet (I can only post two links unfortunately :( Google brings it up easily).

[EDIT] For future viewers, I just want to point out that for the newer EZO version of the circuit, the command is "L,1" or really "L,1\r"
[/EDIT]
This is a known issue with the circuit. The first read after power up will fail. instead of prepending a \r with every read, just send a bogus command with \r included and then reset input buffer
ser.write('bogus\r'.encode()) # EDIT: had to add .encode() to get it to work. see https://stackoverflow.com/questions/22275079/pyserial-write-wont-take-my-string
ser.flushInput() # or for pyserial 3+ ser.reset_input_buffer()
#now do your thing
EDIT: had to add .encode() to get it to work. see pySerial write() won't take my string
After powering up the EZO™ class circuit when it is in UART mode the
first command sent to it will comeback as an error. This is because
the UART buffer will show that it has received a character during
power up. Simply send a blank character to the pH circuit after it is
powered up, this will clear the buffer.

I have found a solution!! Unfortunately, I cannot explain how it works. Perhaps anyone reading this could elaborate on it and give a proper explanation!
The circuit's documentation demands commands be in the form CMD<CR>. Indeed, sample code provided by the manufacturer sends the L1 command through pyserial as ser.write("L1\r").
Now that ser.write() demands bytes however, I have found that ser.write(b'L1\r') does not work.. The command is received though it is somehow unknown to the circuit.
After toying around for a while, I have discovered that ser.write(b'\rL1\r') works! The debugging led flashes red once before processing the command. It seems like I just need to send a 'dummy command' to get the circuit's attention!
I am not sure if this is the fault of pyserial, the circuit, or my own ignorance. If anyone can shed some light on this, it would be much appreciated! :D
I have linked here the circuits documentation in case anyone is interested. https://www.atlas-scientific.com/_files/_datasheets/_circuit/DO_Circuit_5.0.pdf

How to get around the need for multiple reactors in twisted

I am running a Qt application on Linux using the qt4reactor
The application sends and receives bytes on the serial port.
This works very well on Linux with the QtReactor
However when I port the application to windows then I have a problem.
On windows I use the SerialPort class from _win32SerialPort.
The doc string in _win32SerialPort is quite clear:
Requires PySerial and win32all, and needs to be used with win32eventreactor.
I assume the need to use win32eventreactor is because the addReader, addWriter methods are written for windows.
When the QtReactor is used, as soon as loseConnection is called on the transport, this calls loseConnection in twisted.internet.abstract which eventually calls the qt4reactor addWriter method (to flush the output).
This then creates a qt4reactor.TwistedSocketNotifier which tries to get a file descriptor number for select(). The abstract.fileno method is not overwritten by _win32SerialPort, so -1 is always returned and I get a
QSocketNotifier: Invalid Socket specified
I've seen many posts about multiple reactors not allowed in twisted, however I think I am correct here to assume that I need QtReactor for the Qt application and the win32eventreactor for the windows serial port.
Or is there some other workaround I can use ?
NOTE 1: when using QtReactor on windows, the serial ports work fine i.e. they can send and receive data. It is only when I close the application that I get "Invalid Socket specified"
Note 2: Now I found a workaround. I use the QtReactor, but when closing my application I do
serial.connectionLost(failure.Failure(Exception))
where serial is an instance of _win32serialport.SerialPort
This way abstract.loseConnection is never called which means that QtReactor addWriter is never called to flush the output. I suspect though that the best solution involves calling loseConnection and getting the output flushed properly.

I recently added serial port supported to selectreactor, iocpreactor, and gtkreactor on Windows. This approach can be extended to support them in the Qt reactor as well, I expect.

Call the Qt message checking/handling functions in the idle event of the Win32 reactor.

Linux libnetfilter_queue delayed packet problem

I have to filter and modify network traffic using Linux kernel libnetfilter_queue (precisely the python binding) and dpkt, and i'm trying to implement delayed packet forward.
Normal filtering works really well, but if i try to delay packets with function like this
def setVerdict(pkt, nf_payload):
nf_payload.set_verdict_modified(nfqueue.NF_ACCEPT, str(pkt), len(pkt))
t = threading.Timer(10, setVerdict, [pkt, nf_payload])
t.start()
It crashs throwing no exception (surely is a low level crash). Can i implement delay using directly libnetfilter like this or I must copy pkt, drop it and send the copy using standard socket.socket.send()?
Thank you

Sorry for the late reply, but I needed to do something like this, although slightly more complicated. I used the C-version of the library and I copied packets to a buffer inside my program, and then issued a DROP verdict. After a timeout relating to your delay, I reinject the packet using a raw socket. This works fine, and seems quite efficient.
I think the reason for your crash was due to the fact that you didnt issue a verdict fast enough.

I can't answer your question, but why not use the "netem" traffic-queue module on the outgoing interface to delay the packet?
It is possible to configure tc queues to apply different policies to packets which are "marked" in some way; the normal way to mark such packets is with a netfilter module (e.g. iptables or nfqueue).

set timeout to http response read method in python

I'm building a download manager in python for fun, and sometimes the connection to the server is still on but the server doesn't send me data, so read method (of HTTPResponse) block me forever. This happens, for example, when I download from a server, which located outside of my country, that limit the bandwidth to other countries.
How can I set a timeout for the read method (2 minutes for example)?
Thanks, Nir.

If you're stuck on some Python version < 2.6, one (imperfect but usable) approach is to do
import socket
socket.setdefaulttimeout(10.0) # or whatever
before you start using httplib. The docs are here, and clearly state that setdefaulttimeout is available since Python 2.3 -- every socket made from the time you do this call, to the time you call the same function again, will use that timeout of 10 seconds. You can use getdefaulttimeout before setting a new timeout, if you want to save the previous timeout (including none) so that you can restore it later (with another setdefaulttimeout).
These functions and idioms are quite useful whenever you need to use some older higher-level library which uses Python sockets but doesn't give you a good way to set timeouts (of course it's better to use updated higher-level libraries, e.g. the httplib version that comes with 2.6 or the third-party httplib2 in this case, but that's not always feasible, and playing with the default timeout setting can be a good workaround).

You have to set it during HTTPConnection initialization.
Note: in case you are using an older version of Python, then you can install httplib2; by many, it is considered a superior alternative to httplib, and it does supports timeout.
I've never used it, though, and I'm just reporting what documentation and blogs are saying.

Setting the default timeout might abort a download early if it's large, as opposed to only aborting if it stops receiving data for the timeout value. HTTPlib2 is probably the way to go.

5 years later but hopefully this will help someone else...
I was wrecking my brain trying to figure this out. My problem was a server returning corrupt content and thus giving back less data than it thought it had.
I came up with a nasty solution that seems to be working properly. Here it goes:
# NOTE I directly disabling blocking is not necessary but it represents
# an important piece to the problem so I am leaving it here.
# http_response.fp._sock.socket.setblocking(0)
http_response.fp._sock.settimeout(read_timeout)
http_response.read(chunk_size)
NOTE This solution also works for the python requests ANY library that implements the normal python sockets (which should be all of them?). You just have to go a few levels deeper:
resp.raw._fp.fp._sock.socket.setblocking()
resp.raw._fp.fp._sock.settimeout(read_timeout)
resp.raw.read(chunk_size)
As of this writing, I have not tried the following but in theory it should work:
resp = requests.get(some_url, stream=True)
resp.raw._fp.fp._sock.socket.setblocking()
resp.raw._fp.fp._sock.settimeout(read_timeout)
for chunk in resp.iter_content(chunk_size):
# do stuff
Explanation
I stumbled upon this approach when reading this SO question for setting a timeout on socket.recv
At the end of the day, any http request has a socket. For the httplib that socket is located at resp.raw._fp.fp._sock.socket. The resp.raw._fp.fp._sock is a socket._fileobj (which I honestly didn't look far into) and I imagine it's settimeout method internally sets it on the socket attribute.