We Keep Coding

Why does my long-running python script crash with "invalid pointer" after running for about 3 days?

I wrote a python 3 script which tests an SPI link to an FPGA. It runs on an Raspberry Pi 3. The test works like this: after putting the FPGA in test mode (a push switch), send the first byte, which can be any value. Then further bytes are sent indefinitely. Each one increments by the first value sent, truncated to 8 bits. Thus, if the first value is 37, the FPGA expects the following sequence:
37, 74, 111, 148, 185, 222, 4, 41 ...
Some additional IO pins are used to signal between the devices - RUN (RPi output) starts the test (necessary because the FPGA times out in about 15ms if it expects a byte) and ERR (FPGA output) signals an error. Errors can thus be counted at both ends.
In addition, the RPi script writes a one line summary of bytes sent and number of erros every million bytes.
All of this works just fine. But after running for about 3 days, I get the following error on the RPi:
free(): invalid pointer: 0x00405340
I get this exact same error on two identical test setups, even the same memory address. The last report says
"4294M bytes sent, 0 errors"
I seem to have proved the SPI link, but I am concerned that this long-running program crashes for no apparent reason.
Here is the important part of my test code:
def _report(self, msg):
now = datetime.datetime.now()
os.system("echo \"{} : {}\" > spitest_last.log".format(now, msg))
def spi_test(self):
global end_loop
input("Put the FPGA board into SPI test mode (SW1) and press any key")
self._set_run(True)
self.END_LOOP = False
print("SPI test is running, CTRL-C to end.")
# first byte is sent without LOAD, this is the seed
self._send_byte(self._val)
self._next_val()
end_loop = False
err_flag = False
err_cnt = 0
byte_count = 1
while not end_loop:
mb = byte_count % 1000000
if mb == 0:
msg = "{}M bytes sent, {} errors".format(int(byte_count/1000000), err_cnt)
print("\r" + msg, end="")
self._report(msg)
err_flag = True
else:
err_flag = False
#print("sending: {}".format(self._val))
self._set_load(True)
if self._errors and err_flag:
self._send_byte(self._val + 1)
else:
self._send_byte(self._val)
if self.is_error():
err_cnt += 1
msg = "{}M bytes sent, {} errors".format(int(byte_count/1000000), err_cnt)
print("\r{}".format(msg), end="")
self._report(msg)
self._set_load(False)
# increase the value by the seed and truncate to 8 bits
self._next_val()
byte_count += 1
# test is done
input("\nSPI test ended ({} bytes sent, {} errors). Press ENTER to end.".format(byte_count, err_cnt))
self._set_run(False)
(Note for clarification : there is a command line option to artifically create an error every million bytes. Hence the " err_flag" variable.)
I've tried using python3 in console mode, and there seems to be no issue with the size of the byte_count variable (there shouldn't be, according to what I have read about python integer size limits).
Anyone have an idea as to what might cause this?

This issue is connected to spidev versions older than 3.5 only. The comments below were done under assumption that I was using the upgraded version of spidev.
#############################################################################
I can confirm this problem. It is persistent with both RPi3B and RPi4B. Using python 3.7.3 at both RPi3 and RPi4. The version of spidev which I tried were 3.3, 3.4 and the latest 3.5. I was able to reproduce this error several times by simply looping through this single line.
spidevice2.xfer2([0x00, 0x00, 0x00, 0x00])
It takes up to 11 hours depending on the RPi version. After 1073014000 calls (rounded to 1000), the script crashes because of "invalid pointer". The total amount of bytes sent is the same as in danmcb's case. It seems as if 2^32 bytes represent a limit.
I tried different approaches. For example, calling close() from time to time followed by open(). This did not help.
Then, I tried to create the spiDev object locally, so it would re-created for every batch of data.
def spiLoop():
spidevice2 = spidev.SpiDev()
spidevice2.open(0, 1)
spidevice2.max_speed_hz = 15000000
spidevice2.mode = 1 # Data is clocked in on falling edge
for j in range(100000):
spidevice2.xfer2([0x00, 0x00, 0x00, 0x00])
spidevice2.close()
It still crashed at after approx. 2^30 calls of xfer2([0x00, 0x00, 0x00, 0x00]) which corresponds to approx. 2^32 bytes.
EDIT1
To speed up the process, I was sending in blocks of 4096 bytes using the code below. And I repeatedly created the SpiDev object locally. It took 2 hours to arrive at 2^32 bytes count.
def spiLoop():
spidevice2 = spidev.SpiDev()
spidevice2.open(0, 1)
spidevice2.max_speed_hz = 25000000
spidevice2.mode = 1 # Data is clocked in on falling edge
to_send = [0x00] * 2**12 # 4096 bytes
for j in range(100):
spidevice2.xfer2(to_send)
spidevice2.close()
del spidevice2
def runSPI():
for i in range(2**31 - 1):
spiLoop()
print((2**12 * 100 * (i + 1)) / 2**20, 'Mbytes')
EDIT2
Reloading the spidev on the fly does not help either. I tried this code on both RPi3 and RPi4 with the same result:
import importlib
def spiLoop():
importlib.reload(spidev)
spidevice2 = spidev.SpiDev()
spidevice2.open(0, 1)
spidevice2.max_speed_hz = 25000000
spidevice2.mode = 1 # Data is clocked in on falling edge
to_send = [0x00] * 2**12 # 4096 bytes
for j in range(100):
spidevice2.xfer2(to_send)
spidevice2.close()
del spidevice2
def runSPI():
for i in range(2**31 - 1):
spiLoop()
print((2**12 * 100 * (i + 1)) / 2**20, 'Mbytes')
EDIT3
Executing the code snippet did not isolate the problem either. It crashed after the 4th chuck of 1Gbyte-data was sent.
program = '''
import spidev
spidevice = None
def configSPI():
global spidevice
# We only have SPI bus 0 available to us on the Pi
bus = 0
#Device is the chip select pin. Set to 0 or 1, depending on the connections
device = 1
spidevice = spidev.SpiDev()
spidevice.open(bus, device)
spidevice.max_speed_hz = 250000000
spidevice.mode = 1 # Data is clocked in on falling edge
def spiLoop():
to_send = [0xAA] * 2**12
loops = 1024
for j in range(loops):
spidevice.xfer2(to_send)
return len(to_send) * loops
configSPI()
bytes_total = 0
while True:
bytes_sent = spiLoop()
bytes_total += bytes_sent
print(int(bytes_total / 2**20), "Mbytes", int(1000 * (bytes_total / 2**30)) / 10, "% finished")
if bytes_total > 2**30:
break
'''
for i in range(100):
exec(program)
print("program executed", i + 1, "times, bytes sent > ", (i + 1) * 2**30)

I belive the original asker's issue is a reference leak. Specifically py-spidev issue 91. Said reference leak has been fixed in the 3.5 release of spidev.
Python uses a shared pool of objects to represent small integer values*, rather than re-creating them each time. So when code leaks references to small numbers the result is not a memory leak but instead a reference count that keeps increasing. The python spidev library had an issue where it leaked references to small integers in this way.
On a 32-bit system** the eventual result is that the reference count overflows. Then something decrements the overflowed reference count and the reference counting system frees the object.
What I can't explain is the other answer that claims they can still reproduce the issue with 3.5. This issue was supposed to have been fixed in that version.
* Specifically numbers in the range -3 to 256 inclusive, so anything that can be represented in an unsigned byte plus a few negative values (presumably because they are commonly used as error returns) and 256 (presumably because it's often used as a multiplier).
** On a 64-bit system the reference count will not overflow within a human lifetime.

unable to unpack information between custom Preamble in Python and telnetlib

I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)

Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.

I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.

Python sockets stealing each other's packets

I'm trying to write a program to test data transfer speeds for various-sized packets in parallel. I noticed something odd, though, that the size of the packet seemed to have no effect on transfer time according to my program, whereas the Unix ping binary would time out on some of the packet sizes I'm using. I was sending 4 packets containing the string 'testquest' and one that was just 2000 bytes set to 0. However, when I printed the results, they all contained 'testquest' (and were far shorter than 2000 bytes). The only thing I can conclude is that these sockets are somehow all receiving the same packet, which would explain how they all had the same rtt.
I made this MCVE to illustrate the issue (you can ignore the 'checksum' function, it's included for completeness but I know from experience that it works):
#!/usr/bin/env python3
import socket
import struct
import time
from multiprocessing.pool import ThreadPool as Pool
from sys import argv, byteorder
def calculate_checksum(pkt):
"""
Implementation of the "Internet Checksum" specified in RFC 1071 (https://tools.ieft.org/html/rfc1071)
Ideally this would act on the string as a series of 16-bit ints (host
packed), but this works.
Network data is big-endian, hosts are typically little-endian,
which makes this much more tedious than it needs to be.
"""
countTo = len(pkt) // 2 * 2
total, count = 0, 0
# Handle bytes in pairs (decoding as short ints)
loByte, hiByte = 0, 0
while count < countTo:
if (byteorder == "little"):
loByte = pkt[count]
hiByte = pkt[count + 1]
else:
loByte = pkt[count + 1]
hiByte = pkt[count]
total += hiByte * 256 + loByte
count += 2
# Handle last byte if applicable (odd-number of bytes)
# Endianness should be irrelevant in this case
if countTo < len(pkt): # Check for odd length
total += pkt[len(pkt) - 1]
total &= 0xffffffff # Truncate sum to 32 bits (a variance from ping.c, which
# uses signed ints, but overflow is unlikely in ping)
total = (total >> 16) + (total & 0xffff) # Add high 16 bits to low 16 bits
total += (total >> 16) # Add carry from above (if any)
return socket.htons((~total) & 0xffff)
def ping(args):
sock, payload = args[0], args[1]
header = struct.pack("!BBH", 8, 0, 0)
checksum = calculate_checksum(header+payload)
header = struct.pack("!BBH", 8, 0, checksum)
timestamp = time.time()
sock.send(header+payload)
try:
response = sock.recv(20+len(payload))
except socket.timeout:
return 0
return (len(response), (time.time() - timestamp) * 1000)
host = argv[1] # A host that doesn't respond to ping packets > 1500B
# 1 is ICMP protocol number
sockets = [socket.socket(socket.AF_INET, socket.SOCK_RAW, proto=1) for i in range(12)]
for i, sock in enumerate(sockets):
sock.settimeout(0.1)
sock.bind(("0.0.0.0", i))
sock.connect((host, 1)) # Port number should never matter for ICMP
args = [(sockets[i], bytes(2**i)) for i in range(12)]
for arg in args:
print(ping(arg))
arg[0].close()
This actually shows me something more troubling - it seems that the rtt is actually decreasing with increasing packet size! Calling this program (as root, to get socket permissions) outputs:
0
0
(24, 15.784025192260742)
(28, 0.04601478576660156)
(28, 0.025033950805664062)
(28, 0.033855438232421875)
(28, 0.03528594970703125)
(28, 0.04887580871582031)
(28, 0.05316734313964844)
(28, 0.03790855407714844)
(28, 0.0209808349609375)
(28, 0.024080276489257812)
but now notice what happens when I try to send a packet of size 2048 using ping:
user#mycomputer ~/src/connvitals $ time ping -c1 -s2048 $box
PING <hostname redacted> (<IP address redacted>): 2048 data bytes
--- <hostname redacted> ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
real 0m11.018s
user 0m0.005s
sys 0m0.008s
Not only is the packet dropped, but it takes 11 seconds to do so! So why - if my timeout is set to 100ms - is this packet getting a "successful" response from my python script in only ~0.04ms??
Thank you in advance for any help you can provide.
Update:
I just checked again, and it seems that it's multiple sockets that are the problem, and the threading seems to have nothing to do with it. I get the same issue when I ping with each socket - then immediately close it - sequentially.

All your sockets are identical, and all bound to the same host. There simply isn't any information in the packet for the kernel to know which socket to go to, and raw(7) seems to imply all sockets will receive them.
You're probably getting all the responses in all the threads, meaning you're getting 12 times as many responses per thread as you're expecting.

Interpret serialdata in python

I'm currently struggling with an issue. I have an arduino sending serialdata to my raspberry pi. The raspberry pi reads the data and stores it in a database. What i'm struggling with is to get data in the correct order. If i start the script at the correct time, the values get read properly. If i don't they get mixed up.
I have a headerByte sent from the arduino, this value is 999 and it is the first value to be sent each time. Is there a way in python to make 999 the marker for the beginning of every read? My variables will never exceed 999 so this will not be a problem.
Python code:
import serial
import time
values = []
serialArduino = serial.Serial('/dev/ttyACM0', baudrate=9600, timeout=1)
voltageRead = serialArduino.readline()
currentRead = serialArduino.readline()
while True:
voltageRead = serialArduino.readline()
currentRead = serialArduino.readline()
print"V=", voltageRead, "A=", currentRead
Arduino Code:
void loop() {
float voltageRead = analogRead(A0);
float ampsRead = analogRead(A1);
float calculatedVoltage = voltageRead / 103;
float calculatedCurrent = ampsRead / 1;
int headerByte = 999;
Serial.println(headerByte);
Serial.println(calculatedVoltage);
Serial.println(calculatedCurrent);
delay(1000);
}

Your method isn't particularly efficient; you could send everything as a struct from the Arduino (header + data) and parse it on the RPi side with the struct module though your way does have the advantage of simplicity. But, if 999 is the highest value you expect in your readings, then it makes more sense to use a number greater than 999 like 1000. And 999 isn't really a byte.
That said, if "1000" is your header, you can simply check for the header's presence like this:
HEADER = "1000"
serialArduino = serial.Serial('/dev/ttyACM0', baudrate=9600, timeout=1)
while True:
if serialArduino.readline().rstrip() == HEADER: # check for header
voltageRead = serialArduino.readline() # read two lines
currentRead = serialArduino.readline()
print"V=", voltageRead, "A=", currentRead

Read latest character sent from Arduino in Python

I'm a beginner in both Arduino and Python, and I have an idea but I can't get it to work. Basically, when in Arduino a button is pressed, it sends "4" through the serial port. What I want in Python is as soon as it reads a 4, it should do something. This is what I got so far:
import serial
ser = serial.Serial('/dev/tty.usbserial-A900frF6', 9600)
var = 1
while var == 1:
if ser.inWaiting() > 0:
ser.readline(1)
print "hello"
But obviously this prints hello no matter what. What I would need is something like this:
import serial
ser = serial.Serial('/dev/tty.usbserial-A900frF6', 9600)
var = 1
while var == 1:
if ser.inWaiting() > 0:
ser.readline(1)
if last.read == "4":
print "hello"
But how can I define last.read?

I don't know a good way of synchronising the comms with readLine since it's not a blocking call. You can use ser.read(numBytes) which is a blocking call. You will need to know how many bytes Arduino is sending though to decode the byte stream correctly. Here is a simple example that reads 8 bytes and unpacks them into 2 unsigned shorts and a long (the <HHL part) in Python
try:
data = [struct.unpack('<HHL', handle.read(8)) for i in range(PACKETS_PER_TRANSMIT)]
except OSError:
self.emit(SIGNAL("connectionLost()"))
self.connected = False
Here's a reference to the struct.unpack()
The Arduino code that goes with that. It reads two analog sensor values and the micro timestamp and sends them over the serial.
unsigned int SensA, SensB;
byte out_buffer[64];
unsigned int buffer_head = 0;
unsigned int buffer_size = 64;
SensA = analogRead(SENSOR_A);
SensB = analogRead(SENSOR_B);
micr = micros();
out_buffer[buffer_head++] = (SensA & 0xFF);
out_buffer[buffer_head++] = (SensA >> 8) & 0xFF;
out_buffer[buffer_head++] = (SensB & 0xFF);
out_buffer[buffer_head++] = (SensB >> 8) & 0xFF;
out_buffer[buffer_head++] = (micr & 0xFF);
out_buffer[buffer_head++] = (micr >> 8) & 0xFF;
out_buffer[buffer_head++] = (micr >> 16) & 0xFF;
out_buffer[buffer_head++] = (micr >> 24) & 0xFF;
Serial.write(out_buffer, buffer_size);
The Arduino playground and Processing Forums are good places to look around for this sort of code as well.
UPDATE
I think I might have misled you with readLine not blocking. Either way, the above code should work. I also found this other thread on SO regarding the same subject.
UPDATE You don't need to use the analog sensors, that's just what the project I did happened to be using, you are of course free to pass what ever values over the serial. So what the Arduino code is doing is it has a buffer of type byte where the output is being stored before being sent. The sensor values and micros are then written to the buffer and the buffer sent over the serial. The (SensA & 0xFF) is a bit mask operator that takes the bit pattern of the SensA value and masks it with the bit pattern of 0xFF or 255 in decimal. Essetianlly this takes the first 8 bits from the 16 bit value of SensA which is an Arduino short. the next line does the same thing but shifts the bits right by 8 positions, thus taking the last 8 bits.
You'll need to understand bit patterns, bit masking and bit shifting for this. Then the buffer is written to the serial.
The Python code in turn does reads the bits from the serial port 8 bits at a time. Have a look at the struct.unpack docs. The for comprehension is just there to allow sending more than one set of values. Because the Arduino board and the Python code are running out of sync I added that to be able to send more than one "lines" per transmit. You can just replace that with struct.unpack('<HHL',handle.read(8)). Remember that the ´handle.read()´ takes a number of bytes where as the Arduino send code is dealing with bits.

I think it might work with this modifications:
import serial
ser = serial.Serial('/dev/tty.usbserial-A900frF6', 9600)
var = 1
while var == 1:
if (ser.inWaiting() > 0):
ser.readline(1)
print "hello"