Objective: understand strange behavior regarding two classes initialized with identical arguments, and producing different results.
Background:
My project involves a Raspberry Pi 3 communicating with various sensors on an Arduino MEGA via serial (through the USB port) communication. The communication protocol is simple.
The Raspberry Pi sends two bytes:
The address of the sensor (e.g. '\x04')
A second empty byte that can contain additional commands (e.g. '\x00')
The MEGA waits for two bytes, and then replys with the requested information based on a large case/switch tree.
I have encapsulated the code for handling various sensors withing Python classes. The one that is giving me trouble is the Encoder() class. Here is a minimal example:
import serial
class Encoder(object):
def __init__(self, hex_name):
self.hex_name = hex_name
def value(self):
temp = self.hex_name + '\x00'
arduino.write(temp)
return int(arduino.readline())
if __name__ == '__main__':
arduino = serial.Serial('/dev/ttyACMO', 115200)
encoder_one = Encoder('\x03')
encoder_two = Encoder('\x04')
print encoder_one.value()
print encoder_two.value()
The Arduino handles the request as shown below:
if(Serial.available() >= 2){
temp1 = Serial.read();
temp2 = Serial.read();
switch(temp1){
case 1:
...
case 3:
Serial.println(positionLeft);
break;
case 4:
Serial.println(positionRight);
break;
case 5:
...
}
}
Problem:
I get nonsensical values from the encoders. Particularly concerning is that when I initialize both encoder with the same hex_name (i.e. '\x04'), I get different values from encoder_one.value() and encoder_two.value().
Hypothesis:
Something about the class structure is wrong.
I had this working alright before I encapsulated it in a class. I wondered if I was assigning encoder_one and encoder_two to the same object or something silly like that.
I added a line just before return in value() that would print the hex_name (i.e. print self.hex_name). When I set both encoders to '\x04' I get the same non-alphanumeric character printed. When I set one encoder to '\x03' and the other to '\x04' I get two different non-alphanumeric characters--one being the character from the previous test.
Frame-shift in the communication.
The MEGA is expecting two bytes. It waits for two bytes, sends the requested information, then clears the buffer. Is is possible that storing the hex values as string in my Python program, a extra byte is getting added when it performs arduino.write(). Something like a \n or other non-printable character.
Conclusion:
I've put about three hours into this bug, and I think solving it requires some information about how classes work that I do not understand.
It turned out to be the second hypothesis. There is apparently some non-printable character being included in the concatenation step of the hex bytes. Switching to the following order solved the problem.
EDIT: I discovered that performing a readline() immediately after write() was giving intermitent errors. I added a 10 ms delay between the two operations.
Old:
def value(self):
temp = self.hex_name + '\x00'
arduino.write(temp)
return int(arduino.readline())
New:
def value(self):
arduino.write(self.hex_name)
arduino.write('\x00')
time.sleep(0.010)
return int(arduino.readline())
The complete (now working) example is shown below.
import serial
class Encoder(object):
def __init__(self, hex_name):
self.hex_name = hex_name
def value(self):
arduino.write(self.hex_name)
arduino.write('\x00')
time.sleep(0.010)
return int(arduino.readline())
if __name__ == '__main__':
arduino = serial.Serial('/dev/ttyACMO', 115200)
encoder_one = Encoder('\x03')
encoder_two = Encoder('\x04')
print encoder_one.value()
print encoder_two.value()
Related
In PyQt5, I want to read my serial port after writing (requesting a value) to it. I've got it working using readyRead.connect(self.readingReady), but then I'm limited to outputting to only one text field.
The code for requesting parameters sends a string to the serial port. After that, I'm reading the serial port using the readingReady function and printing the result to a plainTextEdit form.
def read_configuration(self):
if self.serial.isOpen():
self.serial.write(f"?request1\n".encode())
self.label_massGainOutput.setText(f"{self.serial.readAll().data().decode()}"[:-2])
self.serial.write(f"?request2\n".encode())
self.serial.readyRead.connect(self.readingReady)
self.serial.write(f"?request3\n".encode())
self.serial.readyRead.connect(self.readingReady)
def readingReady(self):
data = self.serial.readAll()
if len(data) > 0:
self.plainTextEdit_commandOutput.appendPlainText(f"{data.data().decode()}"[:-2])
else: self.serial.flush()
The problem I have, is that I want every answer from the serial port to go to a different plainTextEdit form. The only solution I see now is to write a separate readingReady function for every request (and I have a lot! Only three are shown now). This must be possible in a better way. Maybe using arguments in the readingReady function? Or returning a value from the function that I can redirect to the correct form?
Without using the readyRead signal, all my values are one behind. So the first request prints nothing, the second prints the first etc. and the last is not printed out.
Does someone have a better way to implement this functionality?
QSerialPort has asyncronous (readyRead) and syncronous API (waitForReadyRead), if you only read configuration once on start and ui freezing during this process is not critical to you, you can use syncronous API.
serial.write(f"?request1\n".encode())
serial.waitForReadyRead()
res = serial.read(10)
serial.write(f"?request2\n".encode())
serial.waitForReadyRead()
res = serial.read(10)
This simplification assumes that responces comes in one chunk and message size is below or equal 10 bytes which is not guaranteed. Actual code should be something like this:
def isCompleteMessage(res):
# your code here
serial.write(f"?request2\n".encode())
res = b''
while not isCompleteMessage(res):
serial.waitForReadyRead()
res += serial.read(10)
Alternatively you can create worker or thread, open port and query requests in it syncronously and deliver responces to application using signals - no freezes, clear code, slightly more complicated system.
I want to write a command-line program that communicates with other interactive programs through a pseudo-terminal. In particular I want to be able to cause keystrokes received to conditionally be sent to the underlying process. Let's say for an example that I would like to silently ignore any "e" characters that are sent.
I know that Python has a pty module for working with pseudo-terminals and I have a basic version of my program working using it:
import os
import pty
def script_read(stdin):
data = os.read(stdin, 1024)
if data == b"e":
return ... # What goes here?
return data
pty.spawn(["bash"], script_read)
From experimenting, I know that returning an empty bytes object b"" causes the pty.spawn implementation to think that the underlying file descriptor has reached the end of file and should no longer be read from, which causes the terminal to become totally unresponsive (I had to kill my terminal emulator!).
For interactive use, the simplest way to do this is probably to just return a bytes object containing a single null byte: b"\0". The terminal emulator will not print anything for it and so it will look like that input is just completely ignored.
This probably isn't great for certain usages of pseudo-terminals. In particular, if the content written to the pseudo-terminal is going to be written again by the attached program this would probably cause random null bytes to appear in the file. Testing with cat as the attached program, the sequence ^# is printed to the terminal whenever a null byte is sent to it.
So, YMMV.
A more proper solution would be to create a wrapper type that can masquerade as an empty string for the purposes of os.write but that would evaluate as "truthy" in a boolean context to not trigger the end of file conditional. I did some experimenting with this and couldn't figure out what needs to be faked to make os.write fully accept the wrapper as a string type. I'm unclear if it's even possible. :(
Here's my initial attempt at creating such a wrapper type:
class EmptyBytes():
def __init__(self):
self.sliced = False
def __class__(self):
return type(b"")
def __getitem__(self, _key):
return b""
I'm writing a userspace driver for accessing FPGA registers in Python 3.5 that mmaps the FPGA's PCI address space, obtains a memoryview to provide direct access to the memory-mapped register space, and then uses struct.pack_into("<I", ...) to write a 32-bit value into the selected 32-bit aligned address.
def write_u32(address, data):
assert address % 4 == 0, "Address must be 32-bit aligned"
path = path.lib.Path("/dev/uio0")
file_size = path.stat().st_size
with path.open(mode='w+b') as f:
mv = memoryview(mmap.mmap(f.fileno(), file_size))
struct.pack_into("<I", mv, address, data)
Unfortunately, it appears that struct.pack_into does a memset(buf, 0, ...) that clears the register before the actual value is written. By examining write operations within the FPGA, I can see that the register is set to 0x00000000 before the true value is set, so there are at least two writes across the PCI bus (in fact for 32-bit access there are three, two zero writes, then the actual data. 64-bit involves six writes). This causes side-effects with some registers that count the number of write operations, or some that "clear on write" or trigger some event when written.
I'd like to use an alternative method to write the register data in a single write to the memory-mapped register space. I've looked into ctypes.memmove and it looks promising (not yet working), but I'm wondering if there are other ways to do this.
Note that a register read using struct.unpack_from works perfectly.
Note that I've also eliminated the FPGA from this by using a QEMU driver that logs all accesses - I see the same double zero-write access before data is written.
I revisited this in 2022 and the situation hasn't really changed. If you're considering using memoryview to write blocks of data at once, you may find this interesting.
Perhaps this would work as needed?
mv[address:address+4] = struct.pack("<I", data)
Update:
As seen from the comments, the code above does not solve the problem. The following variation of it does, however:
mv_as_int = mv.cast('I')
mv_as_int[address/4] = data
Unfortunately, precise understanding of what happens under the hood and why exactly memoryview behaves this way is beyond the capabilities of modern technology and will thus stay open for the researchers of the future to tackle.
You could try something like this:
def __init__(self,offset,size=0x10000):
self.offset = offset
self.size = size
mmap_file = os.open('/dev/mem', os.O_RDWR | os.O_SYNC)
mem = mmap.mmap(mmap_file, self.size,
mmap.MAP_SHARED,
mmap.PROT_READ | mmap.PROT_WRITE,
offset=self.offset)
os.close(mmap_file)
self.array = np.frombuffer(mem, np.uint32, self.size >> 2)
def wread(self,address):
idx = address >> 2
return_val = int(self.array[idx])
return return_val
def wwrite(self,address,data):
idx = address >> 2
self.array[idx] = np.uint32(data)
I have spent a while trying to interpret the solutions found here pySerial 2.6: specify end-of-line in readline() and here Line buffered serial input, but for the life of me I cannot seem to "quickly" communicate with one of my pressure monitors that replies with responses terminated with a carriage return character ('\r')
I am able to reliably communicate with the device but it is always limited by the timeout parameter when I define the serial port. I can see that the device is in fact responding with a \r character as displayed using the traditional method. When using the io.TextIOWrapper I get the desired functionality of clipping the termination character, but it still timeouts as the original method.
My python code is as follows:
import serial,io,timeit
cmd = b'#0002UAUX1\r'
ser = serial.Serial(port='COM4',baudrate=9600,timeout=1)
def readline1(ser):
return(ser.readline())
def readline2(ser):
sio = io.TextIOWrapper(io.BufferedRWPair(ser,ser,1),newline='\r')
return(sio.readline().encode())
def readline3(ser):
out = b''
while True:
ch = ser.read()
out += ch
if(ch == b'\r' or ch == b''):
break
return(out)
def timer(func):
tic = timeit.default_timer()
ser.write(cmd)
print(func())
toc = timeit.default_timer()
print('%0.2f seconds' % (toc-tic))
for func in (readline1,readline3,readline2):
timer(lambda : func(ser))
my shell response is:
b'>3.066E-02\r'
1.03 seconds
b'>3.066E-02\r'
0.02 seconds
b'>3.066E-02\r'
1.06 seconds
All methods return the same output, but the readline3 operates much quicker as it does not timeout. I don't understand why it isn't as easy as
ser.readline(eol='\r')
P.S. I'm quite new to python (heavily dependent on MATLAB) so any and all criticism is appreciated.
In Python3.x the prefix b'' in a string indicated that it is being displayed as an instance of bytes types, this means that it will ignore displaying things such as escape sequences for their purpose, '\n' will be displayed to terminal in place of a new line.
If you use the command print('Hello\r\nWorld') it will display
Hello
World
, not 'Hello\r\nWorld', however print(b'Hello\r\nWorld') or print(repr('Hello\r\nWorld')) will display 'Hello\r\nWorld'.
To see if this is the case in rwSerial1 replace print(out) with print(repr(out)) and it should display the string with all the characters.
it turns out that the answer is:
ser.read_until(b'\r')
Updates:
I wrote an asynchronous C version and it works as it should.
Turns out the speed issue was due to Python's GIL. There's a method to fine tune its behavior.
sys.setcheckinterval(interval)
Setting interval to zero (default is 100) fixes the slow speed issue. Now all that's left is to figure out is what's causing the other issue (not all pixels are filled). This one doesn't make any sense. usbmon shows all the communications are going through. libusb's debug messaging shows nothing out of the ordinary. I guess I need to take usbmon's output and compare sync vs async. The data that usbmon shows seems to look correct at a glance (The first byte should be 0x96 or 0x95).
As said below in the original question, S. Lott, it's for a USB LCD controller. There are three different versions of drv_send, which is the outgoing endpoint method. I've explained the differences below. Maybe it'll help if I outline the asynchronous USB operations. Note that syncrhonous USB operations work the same way, it's just that it's done synchronously.
We can view asynchronous I/O as a 5 step process:
Allocation: allocate a libusb_transfer (This is self.transfer)
Filling: populate the libusb_transfer instance with information about the transfer you wish to perform (libusb_fill_bulk_transfer)
Submission: ask libusb to submit the transfer (libusb_submit_transfer)
Completion handling: examine transfer results in the libusb_transfer structure (libusb_handle_events and libusb_handle_events_timeout)
Deallocation: clean up resources (Not shown below)
Original question:
I have three different versions. One's entirely synchronous, one's semi-asynchronous, and the last is fully asynchronous. The differences is that synchronous fully populates the LCD display I'm controlling with the expected pixels, and it's really fast. The semi-asynchronous version only populates a portion of the display, but it's still very fast. The asynchronous version is really slow and only fills a portion of the display. I'm baffled why the pixels aren't fully populated, and why the asynchronous version is really slow. Any clues?
Here's the fully synchronous version:
def drv_send(self, data):
if not self.Connected():
return
self.drv_locked = True
buffer = ''
for c in data:
buffer = buffer + chr(c)
length = len(buffer)
out_buffer = cast(buffer, POINTER(c_ubyte))
libusb_fill_bulk_transfer(self.transfer, self.handle, LIBUSB_ENDPOINT_OUT + 1, out_buffer, length, self.cb_send_transfer, None, 0)
lib.libusb_submit_transfer(self.transfer)
while self.drv_locked:
r = lib.libusb_handle_events(None)
if r < 0:
if r == LIBUSB_ERROR_INTERRUPTED:
continue
lib.libusb_cancel_transfer(transfer)
while self.drv_locked:
if lib.libusb_handle_events(None) < 0:
break
self.count += 1
Here's the semi-asynchronous version:
def drv_send(self, data):
if not self.Connected():
return
def f(d):
self.drv_locked = True
buffer = ''
for c in data:
buffer = buffer + chr(c)
length = len(buffer)
out_buffer = cast(buffer, POINTER(c_ubyte))
libusb_fill_bulk_transfer(self.transfer, self.handle, LIBUSB_ENDPOINT_OUT + 1, out_buffer, length, self.cb_send_transfer, None, 0)
lib.libusb_submit_transfer(self.transfer)
while self.drv_locked:
r = lib.libusb_handle_events(None)
if r < 0:
if r == LIBUSB_ERROR_INTERRUPTED:
continue
lib.libusb_cancel_transfer(transfer)
while self.drv_locked:
if lib.libusb_handle_events(None) < 0:
break
self.count += 1
self.command_queue.put(Command(f, data))
Here's the fully asynchronous version. device_poll is in a thread by itself.
def device_poll(self):
while self.Connected():
tv = TIMEVAL(1, 0)
r = lib.libusb_handle_events_timeout(None, byref(tv))
if r < 0:
break
def drv_send(self, data):
if not self.Connected():
return
def f(d):
self.drv_locked = True
buffer = ''
for c in data:
buffer = buffer + chr(c)
length = len(buffer)
out_buffer = cast(buffer, POINTER(c_ubyte))
libusb_fill_bulk_transfer(self.transfer, self.handle, LIBUSB_ENDPOINT_OUT + 1, out_buffer, length, self.cb_send_transfer, None, 0)
lib.libusb_submit_transfer(self.transfer)
self.count += 1
self.command_queue.put(Command(f, data))
And here's where the queue is emptied. It's the callback for a gobject timeout.
def command_worker(self):
if self.drv_locked: # or time.time() - self.command_time < self.command_rate:
return True
try:
tmp = self.command_queue.get_nowait()
except Queue.Empty:
return True
tmp.func(*tmp.args)
self.command_time = time.time()
return True
Here's the transfer's callback. It just changes the locked state back to false, indicating the operation's finished.
def cb_send_transfer(self, transfer):
if transfer[0].status.value != LIBUSB_TRANSFER_COMPLETED:
error("%s: transfer status %d" % (self.name, transfer.status))
print "cb_send_transfer", self.count
self.drv_locked = False
Ok I don't know if I get you right. You have some device with LCD, you have some firmware on it to handle USB requests. On PC side you are using PyUSB wich wraps libUsb.
Couple of suggestions if you are experiancing speed problems, try to limit data you are transfering. Do not transfer whole raw data, mayby only pixels that changed.
Second, have you measured speed of transfers by using some USB analuzer sofware, if you don't have money for hardvare usb analyzer maybe try software version. I never used that kind of analyzers but I think data provided by them is not very reiable.
Thirdly, see what device is realy doing, maybe that is bottleneck of your data transfers.
I have not much time today to exactly anwser your question so I will get back on this later.
I am watching this thread for some time, and there is dead silence around this, so I tried to spare some time and look deeper. Still not much time today maybe later today. Unfortunetly I am no python expert but I know some stuff about C,C++, windows and most of all USB. But I think this may be LCD device problem, what are you using, Because if the transfers works fine, and data was recived by the device it points that is device problem.
I looked at your code a little, could you do some testing, sending only 1 byte, 8 bytes, and Endpoint size byte length transfer. And see how it looks on USB mon ?
Endpoint size is size of Hardvare buffer used by PICO LCD USB controler. I am not sure what it is for your's but I am guessing that when you send ENdpoint size message next masage should be 0 bytes length. Maybe there is the problem.
Regarding the test I assume you have seen data wich you programed to send.
Second thing could be that the data gets overwriten, or not recived fast enough. Saying overwriten I mean LCD could not see data end, and mix one transfer with another.
I am not sure what USB mon is capable of showing, but according to USB standart after Endpoint size packet len, there should be 0 len packet data send, showing that is end of transfer.