Data saved to text file is inconsistent - python

My controller is receiving data from a radio module through a serial port, it's recording temperature and humidity to two decimal places every 1 second, using 'a' as a signal to time stamp it. For example:
a21.12 65.43
a21.13 65.40
Here is the code I'm using:
import serial
import datetime
connected = False
locations=['/dev/ttyUSB0','/dev/ttyUSB1','/dev/ttyUSB2','/dev/ttyUSB3']
for device in locations:
try:
print "Trying...",device
ser = serial.Serial(device, 9600)
break
except:
print "Failed to connect on",device
while not connected:
serin = ser.read()
connected = True
with open('tempdata.txt', 'w') as text_file:
while 1:
if ser.read() is 'a':
text_file.write(datetime.datetime.now().strftime('%d/%m/%Y %H:%M:%S'))
text_file.write(" ")
x = ser.read()
text_file.write(x)
text_file.flush()
ser.close()
When I check my text file afterwards the result seems to be different every time. If I let it run for just 2 or 3 seconds I sometimes get a correct result, sometimes I only get the humidity, sometimes I get the time stamp with a number that is half temperature, half humidity (like 2.16.3). If I let it run for more than a few seconds the file is just completely blank.
The basis for my code is from a question previously asked on here and it worked fine until I added the time stamp part. I tried changing the radio transfer rate from 9600 to 4800 but that just turned the numbers into garbage characters.
I'm running this on a Raspberry Pi 2 model B so I could be demanding too much from it in a short time.

You are calling read() twice and writing only the output of the second call. I don't imagine that is your intent.
You can change this section:
with open('tempdata.txt', 'a') as text_file:
while 1:
content = ser.read()
if content is 'a':
text_file.write(datetime.datetime.now().strftime('%d/%m/%Y %H:%M:%S'))
text_file.write(" ")
text_file.write(content)
text_file.flush()

Related

bind takes a long time

I am using over 20 raspberry pis to control heating and other functions in my house. For instance each room has a pi (client) with a temperature sensor, which sends a message to another pi (server) at one of three manifolds to open or close a valve on the heating system.
My problem is the amount of time the server takes to execute the bind instruction. A newly set up pi will typically take a few millisecond to execute the line, but soon the time becomes variable, sometimes milliseconds, usually seconds, but frequently tens of seconds or even hundreds of seconds.
typical code,
elapselp = default_timer() # timing start
port = 12571 # port number, many different ones used
while bound == 0:
try:
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.bind(('', port))
bound = 1
elapselp1 = default_timer() - elapselp
print ('bind time = ', elapselp1)
As stated above elapselp1 can be at the low millisecond level, or 200 seconds which causes significant problems.
I usually find I can get back to short times by changing the port number used, which means changing on all the clients, but after some time running happily the times will increase again. Rebooting or powering down does not help. I am using Raspberry 4s and Zero Ws, no difference.
I suspect that some register is not clearing, but outside that am clueless. Can somebody advise please.
I am running Rasparian 3.6 and Python3
Thank you for all your replies, here is the remainder of this section of the code.
As I’m not sure how to input his I’m breaking it down over a number of comments.
elapselp = default_timer()
port =12571
while bound == 0:
try:
s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s1.bind(('', port))
bound = 1
elapselp1 = default_timer()- elapselp
print (‘bind time = ‘, elapselp2)
except:
dummy = 1
s1.listen(5)
print ('waiting for input ', end='')
while True:
conn, addr = s1.accept()
print ('connected by ', addr)
conn.send('thank you'.encode())
data = conn.recv(1024).decode()
print ('data = ', data)
time.sleep(5)
conn.close()
s1.close()
if data == '10':
startdaw = default_timer()
GPIO.output(22, GPIO.LOW)
print ('switch dining area heating off')
h1 = 0
dawl = edaw
startdawr = default_timer()
elif data == '11':
GPIO.output(22, GPIO.HIGH)
## A lot of other actions depending on data received here
else:
print ('another code ', data)
if not data:
break
print ('yet another')
I have a number of Raspberry pi Zeros running the code for test along with 7 doing so as part of my control system. The performance is very erratic.
One unit running through the code every 30 seconds in response to a client has been running over 5 days, majority of the bind times have been under 1 millsec, I see some between 1 and 10, but it has recorded one time of 130 seconds.
Another unit which failed to bind for over two hours has now completed over 33,000 binds (this one just closes after the bind, no if statements), and the longest time for a bind is 710 msec.
If I take out the try/except I get no error message, it just drops through and loops back.
I hope there is some useful information in there.

How to extract data from Arduino only when it is available and save the data?

I am trying to extract sensor data from Arduino. Arduino code sends sensor data using serial communication every 1 second. My goal is to display this data and plot it in real-time in a GUI using Tkinter. But where I seem to be stuck is retrieving data from Arduino. In my code, I tried to extract the data and save them in multiple variables. But when Arduino is sending only 5 seconds worth of data which is 5 sensor measurements, my python code is saving more than 60 measurements in the variable. I don't have any idea how to solve this. I have been stuck here for some time. Here is my python code.
In the given pictures I ran the code for 5 seconds. That is 5 measurements will be sent from Arduino. I am recieving the data correctly, but when I process them and seperate them in individual variables I have analog_voltage containing:80 , current:70, voltage:75 and temperature:65 measurements as they should only contain 5. What should I do, is there any way to go around this problem?
This is the same issue when I am trying to save the file. For the 5 seconds of data the program saves a million rows of data in csv. Any help is much appreciated. Thank you for reading.
The image for arduino data is just for reference as it is not possible to run both the programs in a same time due to the serial communication protocols hence the data looks different.
data_to_sort =[]
analog_voltage = []
Voltage= []
Current=[]
Temperature=[]
def arduino_read_serialdata():
global ser
ser = serial.Serial('COM4', 9600, timeout=1)
time.sleep(1.0)
ser.flushInput()
while True:
global analog_voltage, Voltage, Temperature, Current, yp, cleaned_data, anvol
incoming_data = ser.inWaiting()
incoming_data = ser.readline()
incoming_data = incoming_data.decode()
cleaned_data = incoming_data.strip("/n")
data_to_sort.append(cleaned_data)
for everyrow in data_to_sort:
if everyrow.startswith("Measuring voltage and current with INA219 ..."):
pass
elif everyrow.startswith("MAX31855 test"):
pass
elif everyrow.startswith("Initializing sensor...DONE checking reading the values now."):
pass
elif everyrow.startswith("The Analog Voltage is:"):
anvol = everyrow[-8:]
anvol = float(anvol)
analog_voltage.append(anvol)
elif everyrow.startswith("Bus Voltage............:"):
vol = everyrow[-8:]
Voltage.append(float(vol))
elif everyrow.startswith("Current..............:"):
cur = everyrow[-8:]
Current.append(float(cur))
elif everyrow.startswith("Temperature in celcius:"):
temp = everyrow[-7:]
Temperature.append(float(temp))
else:
pass
with open("Sensor_data.csv", 'w+', newline='') as output_file:
output_file.flush()
output_file = csv.writer(output_file)
fields = ['Analog Voltage V', 'Voltage V', 'Current mA', 'Temperature °C']
output_file.writerow(fields)
for idx in range(0, len(analog_voltage)):
output_file.writerow([analog_voltage[idx]])
if keyboard.is_pressed("q"):
print("Manually interrupted by user")
ser.close()
# output_file.close()
print("Serial port closed")
file.close()
print("File closed, Writing done")
You have an array
data_to_sort =[]
and you append each incoming line to it
data_to_sort.append(cleaned_data)
And with each incoming data, you process the whole data
for everyrow in data_to_sort:
and add the values to these four lists:
analog_voltage = []
Voltage= []
Current=[]
Temperature=[]
and these lists are never cleared.
What should be done instead:
either clear these four lists with each incoming data or
process only the incoming line
My personal preference would be the latter. Instead of always writing the whole output file, append only the last line to the output file using "a" instead of "w+".

start reading incoming data from a serial port, using pyserial, when a specific character appears

I'm trying to read incoming data from a weight scale (Lexus Matrix One). I want the code to start reading the 8 characters after = appears.
The problem is that sometimes the code does that, and other times it starts reading the data at the middle of the measurement sent by the scale, making it impossible to read properly. I'm using the pyserial module on python 3 on windows.
import serial
ser=serial.Serial("COM4", baudrate=9600)
a=0
while a<10:
b=ser.read(8)
a=a+1
print(b)
the expected result is: b'= 0004.0'
but sometimes I get: b'4.0= 000'
I think we would need a little more information on the format of data coming from your weight scale to provide a complete answer. But your current code only reads the first 80 bytes from the stream, 8 bytes at a time.
If you want to read the next 8 bytes following any equals sign, you could try something like this:
import serial
ser = serial.Serial("COM4", baudrate=9600)
readings = []
done = False
while not done:
current_char = ser.read()
# check for equals sign
if current_char == b'=':
reading = ser.read(8)
readings.append(reading)
# this part will depend on your specific needs.
# in this example, we stop after 10 readings
# check for stopping condition and set done = True
if len(readings) >= 10:
done = True

Barometer data from Adruino to RPi. Python error with readline()

Been trying to fix this. The barometer data on the Adruino looks like this:
1008.94
1008.95
1008.94
Which I'm happy with. These get "read" every 10 second for now.
The problem is with my python script. I can read and write all the data to my mySQLdb. But the the data is not always good. It will go like something like:
1008.94
108.95
108.94
1008.96
The "lost zero" or other value always appears on the Adruino's serial monitor.
How can I get the python script to read the whole xxxx.xx line? Just thought even at times of low pressure xxx.xx.
Here is my python code:
import MySQLdb
import serial
import time
ser = serial.Serial('/dev/ttyACM0', 9600)
db = MySQLdb.connect("localhost", "root", "pword","weather")
cursor = db.cursor()
while 1:
# print ("Waiting for data...")
# print("")
x = ser.readline()
clock = (time.strftime("%H:%M:%S"))
print x
# print ("Collecting data...")
x = x
# print ("Inserting to database...")
# print ("")
sql = ("""INSERT INTO WeatherP (pres, Time) VALUES (%s, %s)""", (x, clock,))
cursor.execute(*sql)
db.commit()
#
Thank you.
Tinus
I found the following here:
ser.read(ser.inWaiting())
According to the pySerial documentation inWaiting() does return the size of the received data in the buffer. This could solve your problem because the read function has a size of data it takes given by a parameter (in your case 1 byte since you haven't specified) if the received data is larger it might cause such problems.

How to quickly read single byte serial data using Python

I am reading serial data in Python using the following code:
port = "COM11"
baud = 460800
timeout=1
ser = serial.Serial()
ser.port = port
ser.baudrate = baud
ser.timeout = timeout
while 1:
# Read from serial port, blocking
data =ser.read(1)
print data
# some further processing of data
I am sending data at very fast rate but when I use this code I am getting data at a very slow rate, maybe around 2 - 3 data per second. This is too slow because I want to do real time plotting.
So, instead of above code I tried:
while 1:
# Read from serial port, blocking
data =ser.read(1)
data1=(data)
# If there is more than 1 byte, read the rest
n = ser.inWaiting()
data1 = (data1 + ser.read(n))
print data1
Now the speed at which data is updated is the same but instead of a single byte I am checking a number of bytes in input queue and reading them. I am receiving around 3850 bytes per loop so this one appears much faster to me but in fact it is almost the same, the only change is that I am not reading a greater number of bytes.
I want to read a single byte and check for the real time it was received. To do so I cannot use second method where I use ser.inWaiting(). How can I read single byte data faster than using the approaches above?
Here's some test code I wrote for a project that you can try different baud settings with. Basically it sends out some data on Tx (which could be connected directly to Rx) and expects the data to be echoed back. It then compares the returned data with the sent data and lets you know if/when errors occur. Note that if there are no errors then the output will remain blank and at the end of test it will print "0 Comm Errors".
import serial, time
test_data = "hello this is so freakin cool!!!" + '\r' #Must always be terminated with '\r'
echo_timeout = 1 #the time allotted to read back the test_data string in seconds
cycleNum = 0
errors = 0
try:
ser = serial.Serial(port="COM1", baudrate=115200, timeout=1)
ser.flush()
print "starting test"
for x in xrange(100):
cycleNum += 1
d = ser.write(test_data)
ret_char = returned = ''
start_time = time.time()
while (ret_char <> '\r') and (time.time() - start_time < echo_timeout):
ret_char = ser.read(1)
returned += ret_char
if not returned == test_data:
errors += 1
print "Cycle: %d Sent: %s Received: %s" % (cycleNum, repr(test_data), repr(returned) )
except Exception as e:
print 'Python Error:', e
finally:
if 'ser' in locals():
print "%d Comm Errors" % errors
if ser.isOpen():
ser.close()
print "Port Was Successfully Closed"
else:
print "Port Already Closed"
else:
print "Serial Variable Was Never Initialized"

Categories