We Keep Coding

How to properly read most recent value from serial?

I am reading values from a pressure sensing mat which has 32x32 individual pressure points. It outputs the readings on serial as 1024 bytes between 1 and 250 + 1 'end token' byte which is always 255 (or xFF).
I thought the function bellow would flush/reset the input buffer and then take a 'fresh' reading and return the max pressure value from that reading whenever I call it.
However, none of the ser.reset_input_buffer() and similar methods seem to actually empty the buffer. When I press down on the mat, run the program and immediately release the pressure, I don't see the max value drop immediately. Instead, it seems to be going through the buffer one by one.
import serial
import numpy as np
import time
def read_serial():
ser_bytes = bytearray([0])
# none of these seem to make a differece
ser.reset_input_buffer()
ser.flushInput()
ser.flush()
# 2050 bytes should always contain a whole chunk of 1025 bytes ending with 255 (xFF)
while len(ser_bytes) <= 2050:
ser_bytes = ser_bytes + ser.read_until(b'\xFF')
ser_ints = np.array(ser_bytes, dtype='int32') #bytes to ints
last_end_byte_index = np.max( np.where(ser_ints == 255) ) #find the last end byte
# get only the 1024 sensor readings as 32x32 np array
mat_reading = np.array( ser_ints[last_end_byte_index-1024: last_end_byte_index]).reshape(32,32)
return np.amax(mat_reading)
ser = serial.Serial('/dev/tty.usbmodem14201', 115200, timeout=1)
while True:
print(read_serial())
time.sleep(1)
The best solution I found so far is having a designated thread which keeps reading the buffer and updating a global variable. It works but seems a bit unresourceful if I only want to read the value about every 60 seconds. Is there a better way?
Also... is there a better way to read the 1025-byte chunk representing the entire mat? There are no line breaks, so ser.readline() won't work.
Thanks! Not sure how to make an MWE with serial, sorry about that ;)

how to read multiple values from com port using pyserial with python?

I have an STM32 connected to a thermal sensor which indicate the temperature of the room and i worte this program to read this temperatur from my com port. Now i need a program to read multiple value from my com port.
thank you for helping me to solve this problem.
import serial
serport = 'COM3'
serbaudrate = 38400
ser = serial.Serial(port = serport, baudrate = serbaudrate, bytesize=serial.EIGHTBITS, parity=serial.PARITY_NONE, timeout=2)
try:
ser.isOpen()
except:
print("Error")
exit()
if ser.isOpen():
try:
while 1:
if ser.inWaiting() > 0:
#print(ser.readline())
response = ser.readline()
print(response.decode("utf-8"))
else:
ser.write(b"1")
except Exception as e:
print(e)
#Tarmo thank you for answering my question. This is the code i wrote for programming th microcontroller:
while (1)
{
// Test: Set GPIO pin high
HAL_GPIO_WritePin(GPIOA, GPIO_PIN_10, GPIO_PIN_SET);
// Get ADC value
HAL_ADC_Start(&hadc1);
HAL_ADC_PollForConversion(&hadc1, HAL_MAX_DELAY);
raw = HAL_ADC_GetValue(&hadc1);
raw = ((raw*0.000244*3.3) - 0.25)/0.028;
HAL_GPIO_WritePin(GPIOA, GPIO_PIN_10, GPIO_PIN_RESET);
// Convert to string and print
sprintf(msg, "%hu\r\n", raw);
HAL_UART_Transmit(&huart2, (uint8_t*)msg, strlen(msg), HAL_MAX_DELAY);
// Pretend we have to do something else for a while
HAL_Delay(100);
}
in this case i am reading just one value because the thermal sensor gives only one value. if i have more than one value how can i delimit them with a semicolon?

If you've already gone the way of sending a numeric value as text, you can extend that. Assuming you want to send 3 integer values at the same time, just delimit them with a semicolon. E.g. 123;456;789\n.
Your microcontroller should then output data in this format and your Python program can read the entire line just like it does now. The you can parse this line into individual values using regex:
import re
...
response = ser.readline()
needles = re.match("(\d+);(\d+);(\d+)", response)
if needles:
print("Got: {} {} {}".format(needles[0], needles[1], needles[2]))

Let's assume you have 3 values from the ADC, in the variables raw1, raw2, and raw3. Then you can use sprintf() to format a message with all of them, and use Tarmo's suggestion to read them.
sprintf(msg, "%hu;%hu;%hu\r\n", raw1, raw2, raw3);
Please provide enough memory in the variable msg.

MinimalModbus Timing out on bulk Bit reading

hopping someone here can help please.
Im communicating with a modbus device and getting data back as expected, im pretty new to this so i need a bit of advice, im trying to read all of the bit registers on my device (Eaton SC200) for a custom HMI.
If i try a for next loop or even as in the code posted, the com port looses communications.
#!/usr/bin/env python
import minimalmodbus
import time
instrument = minimalmodbus.Instrument('COM11', 1)
minimalmodbus.TIMEOUT = 1
DO =[]
instrument.address # this is the slave address number
instrument.mode = minimalmodbus.MODE_RTU # rtu or ascii mode
#port name, slave address (in decimal)#0 = instrument.read_bit(1,1)
#digital = instrument.read_registers(1,32,4)
Version =instrument.read_registers(3001,3,4)
serialupper = instrument.read_registers(6001,1,4)
serialLower = instrument.read_registers(6002,1,4)
busvolt= instrument.read_registers(7001,4,4)
print type(serialupper[0])
print(serialupper, serialLower)
upper = serialupper[0] << 16 # shift upper left 16 bits
Snumber = upper + serialLower[0]
print ('Serial no: = ' + str(Snumber))
print(busvolt)
print ('Getting i-o')
print instrument.read_bit(1001,2)
print instrument.read_bit(1002,2)
print instrument.read_bit(1003,2)
print instrument.read_bit(1004,2)
print instrument.read_bit(1101,2)
print instrument.read_bit(1102,2)
print instrument.read_bit(1201,2)
print instrument.read_bit(1202,2)
print instrument.read_bit(1203,2)
print instrument.read_bit(1204,2)
print instrument.read_bit(1201,2)
print instrument.read_bit(1201,2)
The response i get is this:
<type 'int'>
([3754], [53255])
Serial no: = 246075399
[16989, 42144, 32704, 0]
Getting i-o
0
0
Traceback (most recent call last):
File "modbusRTU.py", line 29, in <module>
print instrument.read_bit(1003,2)
File "C:\Python27\lib\site-packages\minimalmodbus.py", line 193, in read_bit
return self._genericCommand(functioncode, registeraddress)
File "C:\Python27\lib\site-packages\minimalmodbus.py", line 697, in _genericCommand
payloadFromSlave = self._performCommand(functioncode, payloadToSlave)
File "C:\Python27\lib\site-packages\minimalmodbus.py", line 795, in _performCommand
response = self._communicate(request, number_of_bytes_to_read)
File "C:\Python27\lib\site-packages\minimalmodbus.py", line 930, in _communicate
raise IOError('No communication with the instrument (no answer)')
IOError: No communication with the instrument (no answer)
I have changed the Timeout to various values from 0.5 to 2 to see if that is the issue, sometimes i get a full data run other times i get nothing.
Any assistance would be gratefully received.

It's most likely a physical comms issue. Check your cable terminations, check the terminating resistors (for RS-485), check for RFI/EMI interference sources, and try lowering the baud rate. The longer the cable, the lower the baud rate you should use.
If this is RS-485, are the terminating resistors on both ends of the bus? And are they the correct size? This is more important on high speed networks. On slower (9600 and less) networks, I've seen RS-485 work fine with either no terminating resistors or with oversized terminators. Typically I'll set the RS-485 baud rate to 9600 if I'm not polling much data.

Allocate a file of particular size in Linux with python

I am writing an I/O intensive program in python and I need to allocate a specific amount of storage on hard disk. Since I need to be as fast as possible I do not want to make a file with zero (or dummy) content in a loop. Does python have any library or method to do so, or do I have to use a Linux command in python?
Actually, I am implementing an application that works like BitTorrent. In my code, the receiver stores every segment of the source file in a separate file (each segment of the source file comes from a random sender). At the end, all the separate files will be merged. It takes lots of time to do so.
Therefore, I want to allocate a file in advance and then write every received segment of the source file in its offset in the pre-allocated file.
def handler(self):
BUFFER_SIZE = 1024 # Normally 1024, but we want fast response
# self.request is the TCP socket connected to the client
data = self.request.recv(BUFFER_SIZE)
addr = ..... #Some address
details = str(data).split()
currentFileNum = int(details[0]) #Specifies the segment number of the received file.
totalFileNumber = int(details[1].rstrip('\0')) # Specifies the total number of the segments that should be received.
print '\tReceive: Connection address:', addr,'Current segment Number: ', currentFileNum, 'Total Number of file segments: ', totalFileNumber
f = open(ServerThreadHandler.fileOutputPrefix + '_Received.%s' % currentFileNum, 'wb')
data = self.request.recv(BUFFER_SIZE)
while (data and data != 'EOF'):
f.write(data)
data = self.request.recv(BUFFER_SIZE)
f.close()
print "Done Receiving." ," File Number: ", currentFileNum
self.request.sendall('\tThank you for data. File Number: ' + str(currentFileNum))
ServerThreadHandler.counterLock.acquire()
ServerThreadHandler.receivedFileCounter += 1
if ServerThreadHandler.receivedFileCounter == totalFileNumber:
infiles = []
for i in range(0, totalFileNumber):
infiles.append(ServerThreadHandler.fileOutputPrefix + '_Received.%s' % i)
File_manipulation.cat_files(infiles, ServerThreadHandler.fileOutputPrefix + ServerThreadHandler.fileOutputSuffix, BUFFER_SIZE) # It concatenates the files based on their segment numbers.
ServerThreadHandler.counterLock.release()

Generally (not only in Python but on the OS level) modern FS drivers support sparse files when you pre-create an apparently zero-filled file and then perform seek-and-write cycles to a point where you need to write a particular bit of data.
See How to create a file with file holes? to understand how to create such a file.

Python: ABP bit doesn't flip despite multiple attempted methods, how to fix?

I have revised my question in response to being put on hold. Hopefully this will better match SO standards.
The purpose of this program is to build and send UDP packets which use Alternating Bit Protocol as a simple resending mechanism. I have confirmed already that the packets can be sent and received correctly. The issue is with the ABP bit and its flipping.
The problem facing me now is that despite trying multiple different methods, I cannot flip the ABP bit used for confirming that a packet or ack received is the correct numbered one. I start out by sending a packet with ABP bit=0, and in response, the receiving process should see this and send back an ack with ABP bit=0. Upon receiving that, the sender program flips its ABP bit to 1 and sends a new packet with this ABP bit value. The receiver will get that, send a matching ack with ABP bit=1, and the sender will receive, flip its bit back to 0, and continue the cycle until the program has finished sending information.
Code below, sorry for length, but it is complete and ready to run. The program takes four command line arguments, here is the command I have been using:
% python ftpc.py 164.107.112.71 4000 8000 manygettysburgs.txt
where ftpc.py is the name of the sender program, 164.107.112.71 is an IP address, 4000 and 8000 are port numbers, and manygettysburgs.txt is a text file I have been sending. It should not make a difference if a different .txt is used, but for full accuracy use a file with a length of between 8000 and 9000 characters.
import sys
import socket
import struct
import os
import select
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
foo = 1 - val
return foo
def buildPacketHeader(IP, Port, Flag, ABP):
#pack IP for transport
#split into four 1-byte values
SplitIP = IP.split('.')
#create a 4-byte struct to pack IP, and pack it in remoteIP
GB = struct.Struct("4B")
remoteIP = GB.pack(int(SplitIP[0]),int(SplitIP[1]),int(SplitIP[2]),int(SplitIP[3]))
#remoteIP is now a 4-byte string of packed IP values
#pack Port for transport
#create a 2-byte struct to pack port, and pack it in remotePort
GBC = struct.Struct("H")
remotePort = GBC.pack(int(Port)) #needs another byte
#remotePort is now a 2-byte string
#add flag
flag = bytearray(1)
flag = str(Flag)
#add ABP
ABP = flipBit(ABP)
abp = str(ABP)
#create header and join the four parts together
Header = ''.join(remoteIP)
Header += remotePort
Header += flag
Header += abp
return Header
#assign arguments to local values
IP = sys.argv[1]
PORT = sys.argv[2]
TPORT = sys.argv[3]
localfile = sys.argv[4]
#declare the socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#create destination information arrays
remoteIP = bytearray(4)
remotePort = bytearray(2)
#create flag array
flag = bytearray(1)
#create ABP bit
bit = 1
print bit
#send file size packet
#get size of file from os command
filesize = bytearray(4)
#save as 4-byte string
filesize = str(os.stat(localfile).st_size)
#package the filesize string
filesizestr = buildPacketHeader(IP,PORT,1,bit) #build header
print bit
filesizestr += filesize #complete the packet
s.sendto(filesizestr, ('127.0.0.1', int(TPORT))) #send packet
# end of send file name packet
#begin listening for responses
read, write, err = select.select([s], [], [], 1) #timeout set to 1 seconds
if len(read) > 0:
#data received
data = read[0].recv(1200)
if data[7] != bit:
print "failed ack"
#resend packet
else:
print "Timeout."
#resend packet
#send next data packet
#get filename as string (from arg4 diredtly)
filename = bytearray(20)
#save as 20-byte string
filename = localfile
#package the filename string
filenamestr = buildPacketHeader(IP,PORT,2,bit) #build header
print bit
filenamestr += filename #complete the packet
s.sendto(filenamestr, ('127.0.0.1', int(TPORT))) #send packet
#end of send file name packet
#send file content packet
#reading while loop goes here
with open(localfile, 'rb', 0) as f: #open the file
while True:
fstr = f.read(1000)
if not fstr:
print "NOTHING"
break
#put together the main packet base
filecontentstr = buildPacketHeader(IP,PORT,3,bit)
print bit
filecontentbytearray = bytearray(1000) #create ytear array
filecontentbytearray = fstr #assign fstr to byte array
filecontentsend = ''.join(filecontentstr) #copy filecontentstr to new string since we will be using filecontentstr again in the future for other packets
filecontentsend += filecontentbytearray #append read data to be sent
s.sendto(filecontentsend, ('127.0.0.1', int(TPORT))) #send the file content packet
#end of send file content packet
s.close()
In this code, every time that buildPacketHeader is called, it performs flipBit as part of its operations. flipBit is supposed to flip the bit's value for ABP. I have prints set up to print out the new value of bit after all calls to buildPacketHeader, so as to track the value. Whenever I run the program, however, I always see the same value for the ABP bit.
I've tried several methods, including changing to a bool. Here are some changes to flipBit I have tried:
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
if val == 0:
val = 1
else:
val = 0
return val
and some with bools instead:
def flipBit(val):
val = not val
return val
def flipBit(val):
val = (True, False)[val]
return val
I figure that many of these methods are in fact working options due to past experience. That said, I am completely baffled as to why it is not working as expected in this program. I would assume that it is my inexperience with python that is at fault, for despite having now used it for a decent amount of time, there still are peculiarities which escape me. Any help is greatly appreciated.

I don't understand what your objection is to Python ints, but the ctypes module provides a world of low-level mutable objects; e.g.,
>>> import ctypes
>>> i = ctypes.c_ushort(12) # 2-byte unsigned integer, initial value 12
>>> i
c_ushort(12)
>>> i.value += 0xffff - 12
>>> hex(i.value)
'0xffff'
>>> i.value += 1
>>> i.value # silently overflowed to 0
0

This is my first time ever answering my own SO question, so hopefully this turns out right. If anyone has additions or further answers, then feel free to answer as well, or comment on this one.
I ended up solving the problem by adding another return value to the return statement of buildPacketHeader, so that in addition to returning a string I also return the new value of bit as well. I confirmed that it was working by setting up the following prints inside of buildPacketHeader:
#add ABP
print "before:",ABP #test line for flipBit
ABP = flipBit(ABP)
abp = str(ABP)
print "after:",ABP #test line for flipBit
The output of which is shown here (I ended it early but the proof of functionality is still visible)
% python ftpc.py 164.107.112.70 4000 8000 manygettysburgs.txt
before: 1
after: 0
Timeout, resending packet...
before: 0
after: 1
Timeout, resending packet...
before: 1
after: 0
As can be seen, the before of the second packet is the after of the first packet, and the before of the third packet is the after of the second packet. Through this, you can see that the program is now flipping bits correctly.
The change made to buildPacketHeader is shown below:
return Header
becomes
return Header, ABP
and calls to buildPacketHeader:
filesizestr = buildPacketHeader(IP,PORT,1,bit)
become
filesizestr, bit = buildPacketHeader(IP,PORT,1,bit)
Very simple for such a bother. Make sure you return values if you want to make them change.