We Keep Coding

How to extract data from Arduino only when it is available and save the data?

I am trying to extract sensor data from Arduino. Arduino code sends sensor data using serial communication every 1 second. My goal is to display this data and plot it in real-time in a GUI using Tkinter. But where I seem to be stuck is retrieving data from Arduino. In my code, I tried to extract the data and save them in multiple variables. But when Arduino is sending only 5 seconds worth of data which is 5 sensor measurements, my python code is saving more than 60 measurements in the variable. I don't have any idea how to solve this. I have been stuck here for some time. Here is my python code.
In the given pictures I ran the code for 5 seconds. That is 5 measurements will be sent from Arduino. I am recieving the data correctly, but when I process them and seperate them in individual variables I have analog_voltage containing:80 , current:70, voltage:75 and temperature:65 measurements as they should only contain 5. What should I do, is there any way to go around this problem?
This is the same issue when I am trying to save the file. For the 5 seconds of data the program saves a million rows of data in csv. Any help is much appreciated. Thank you for reading.
The image for arduino data is just for reference as it is not possible to run both the programs in a same time due to the serial communication protocols hence the data looks different.
data_to_sort =[]
analog_voltage = []
Voltage= []
Current=[]
Temperature=[]
def arduino_read_serialdata():
global ser
ser = serial.Serial('COM4', 9600, timeout=1)
time.sleep(1.0)
ser.flushInput()
while True:
global analog_voltage, Voltage, Temperature, Current, yp, cleaned_data, anvol
incoming_data = ser.inWaiting()
incoming_data = ser.readline()
incoming_data = incoming_data.decode()
cleaned_data = incoming_data.strip("/n")
data_to_sort.append(cleaned_data)
for everyrow in data_to_sort:
if everyrow.startswith("Measuring voltage and current with INA219 ..."):
pass
elif everyrow.startswith("MAX31855 test"):
pass
elif everyrow.startswith("Initializing sensor...DONE checking reading the values now."):
pass
elif everyrow.startswith("The Analog Voltage is:"):
anvol = everyrow[-8:]
anvol = float(anvol)
analog_voltage.append(anvol)
elif everyrow.startswith("Bus Voltage............:"):
vol = everyrow[-8:]
Voltage.append(float(vol))
elif everyrow.startswith("Current..............:"):
cur = everyrow[-8:]
Current.append(float(cur))
elif everyrow.startswith("Temperature in celcius:"):
temp = everyrow[-7:]
Temperature.append(float(temp))
else:
pass
with open("Sensor_data.csv", 'w+', newline='') as output_file:
output_file.flush()
output_file = csv.writer(output_file)
fields = ['Analog Voltage V', 'Voltage V', 'Current mA', 'Temperature °C']
output_file.writerow(fields)
for idx in range(0, len(analog_voltage)):
output_file.writerow([analog_voltage[idx]])
if keyboard.is_pressed("q"):
print("Manually interrupted by user")
ser.close()
# output_file.close()
print("Serial port closed")
file.close()
print("File closed, Writing done")

You have an array
data_to_sort =[]
and you append each incoming line to it
data_to_sort.append(cleaned_data)
And with each incoming data, you process the whole data
for everyrow in data_to_sort:
and add the values to these four lists:
analog_voltage = []
Voltage= []
Current=[]
Temperature=[]
and these lists are never cleared.
What should be done instead:
either clear these four lists with each incoming data or
process only the incoming line
My personal preference would be the latter. Instead of always writing the whole output file, append only the last line to the output file using "a" instead of "w+".

Data saved to text file is inconsistent

My controller is receiving data from a radio module through a serial port, it's recording temperature and humidity to two decimal places every 1 second, using 'a' as a signal to time stamp it. For example:
a21.12 65.43
a21.13 65.40
Here is the code I'm using:
import serial
import datetime
connected = False
locations=['/dev/ttyUSB0','/dev/ttyUSB1','/dev/ttyUSB2','/dev/ttyUSB3']
for device in locations:
try:
print "Trying...",device
ser = serial.Serial(device, 9600)
break
except:
print "Failed to connect on",device
while not connected:
serin = ser.read()
connected = True
with open('tempdata.txt', 'w') as text_file:
while 1:
if ser.read() is 'a':
text_file.write(datetime.datetime.now().strftime('%d/%m/%Y %H:%M:%S'))
text_file.write(" ")
x = ser.read()
text_file.write(x)
text_file.flush()
ser.close()
When I check my text file afterwards the result seems to be different every time. If I let it run for just 2 or 3 seconds I sometimes get a correct result, sometimes I only get the humidity, sometimes I get the time stamp with a number that is half temperature, half humidity (like 2.16.3). If I let it run for more than a few seconds the file is just completely blank.
The basis for my code is from a question previously asked on here and it worked fine until I added the time stamp part. I tried changing the radio transfer rate from 9600 to 4800 but that just turned the numbers into garbage characters.
I'm running this on a Raspberry Pi 2 model B so I could be demanding too much from it in a short time.

You are calling read() twice and writing only the output of the second call. I don't imagine that is your intent.
You can change this section:
with open('tempdata.txt', 'a') as text_file:
while 1:
content = ser.read()
if content is 'a':
text_file.write(datetime.datetime.now().strftime('%d/%m/%Y %H:%M:%S'))
text_file.write(" ")
text_file.write(content)
text_file.flush()

How to quickly read single byte serial data using Python

I am reading serial data in Python using the following code:
port = "COM11"
baud = 460800
timeout=1
ser = serial.Serial()
ser.port = port
ser.baudrate = baud
ser.timeout = timeout
while 1:
# Read from serial port, blocking
data =ser.read(1)
print data
# some further processing of data
I am sending data at very fast rate but when I use this code I am getting data at a very slow rate, maybe around 2 - 3 data per second. This is too slow because I want to do real time plotting.
So, instead of above code I tried:
while 1:
# Read from serial port, blocking
data =ser.read(1)
data1=(data)
# If there is more than 1 byte, read the rest
n = ser.inWaiting()
data1 = (data1 + ser.read(n))
print data1
Now the speed at which data is updated is the same but instead of a single byte I am checking a number of bytes in input queue and reading them. I am receiving around 3850 bytes per loop so this one appears much faster to me but in fact it is almost the same, the only change is that I am not reading a greater number of bytes.
I want to read a single byte and check for the real time it was received. To do so I cannot use second method where I use ser.inWaiting(). How can I read single byte data faster than using the approaches above?

Here's some test code I wrote for a project that you can try different baud settings with. Basically it sends out some data on Tx (which could be connected directly to Rx) and expects the data to be echoed back. It then compares the returned data with the sent data and lets you know if/when errors occur. Note that if there are no errors then the output will remain blank and at the end of test it will print "0 Comm Errors".
import serial, time
test_data = "hello this is so freakin cool!!!" + '\r' #Must always be terminated with '\r'
echo_timeout = 1 #the time allotted to read back the test_data string in seconds
cycleNum = 0
errors = 0
try:
ser = serial.Serial(port="COM1", baudrate=115200, timeout=1)
ser.flush()
print "starting test"
for x in xrange(100):
cycleNum += 1
d = ser.write(test_data)
ret_char = returned = ''
start_time = time.time()
while (ret_char <> '\r') and (time.time() - start_time < echo_timeout):
ret_char = ser.read(1)
returned += ret_char
if not returned == test_data:
errors += 1
print "Cycle: %d Sent: %s Received: %s" % (cycleNum, repr(test_data), repr(returned) )
except Exception as e:
print 'Python Error:', e
finally:
if 'ser' in locals():
print "%d Comm Errors" % errors
if ser.isOpen():
ser.close()
print "Port Was Successfully Closed"
else:
print "Port Already Closed"
else:
print "Serial Variable Was Never Initialized"

sending and recieving through serial

I have got a simple program running in the uno that measures the distance using ping sensor and now i am trying to control some servos based on the distance in python but the conditional thingy is not working even in a simple code like this
import serial
data = serial.Serial('COM7',9600)
while(1):
if(data.inWaiting()>0):
dist = data.readline()
if(dist>100):
print("dist is greater than 100")
else:
print("this shit does not work")
It is always the if that works, I am a noob please help me!

Data from serial port type is str
You are trying to read the data and use if .... > 100 this will never work.
You need to read the data to a buffer and then check it, Also you need some sync frame to know you have got all the data(it's not a must but its much eaiser)
for example let say the read data is 100! and ! is your sync frame.
import serial
my_serial = serial.Serial('COM7',9600)
my_data = ""
while(1):
if(my_serial.inWaiting()>0):
my_data = my_serial.readline()
if '!' in my_data:
break
my_data = [:my_data.find("!")]
if int(my_data) > 100:
print("dist is greater than 100")
else:
print("this shit does not work")

Converting a bytearray from a serial port to a float

I am writing a python script that will communicate to a Fluke meter over a COM port. I am able to receive the data but want to parse it into a usable float. The code looks like this:
import serial
ser = serial.Serial('COM3', 115200, timeout=1)
#Decalring some variables
FlukeID = b'ID\r'
FlukeQM = b'QM\r'
#Requesting the meters ID to verify connection on terminal
ser.writelines(FlukeID)
line = ser.readline()
print(line)
#Declaring variables for my while loop
thermdata = 0
t=1
ser.writelines(FlukeQM)
thermdata = ser.readline()
while(t < 5):
ser.writelines(FlukeQM)
#thermdata = (thermdata + ser.readline()) /2
thermdata = ser.readline()
print(thermdata)
t+=1
The data returned by the device looks like this on the console:
8.597E3,OHM,NORMAL,NONE INCORRECT
EDIT: The data actually appears like this over the terminal:
b'0\r8.597E3,OHM,NORMAL,NONE\r'
I just want to be able to use the numerical value at the beginning so I can do some calculations over time. I also need to be able to use the scientific notion portion in my number as I will not know the range of my measurements before hand. I know there must be a simple way to do this and would greatly appreciate any help.
On a side note, I would also like to be able to graph these values or place them into some kind of .csv file. If you have any comments on where to look to learn how to do this also that would be great, but I am mostly concerned with the handling of the bytearray.

Use split() to break your string into the comma separated parts. Then the first part is the string '8.597E3', which you convert using the float() function.
s = '8.597E3,OHM,NORMAL,NONE'.split(',')
value = float(s[0])

How about something like:
def atof(text):
try:
return float(text)
except ValueError:
return text
thermdata = b'0\r8.597E3,OHM,NORMAL,NONE\r'
for line in thermdata.strip().split(b'\r'):
print(list(map(atof, line.split(b','))))
# [0.0]
# [8597.0, b'OHM', b'NORMAL', b'NONE']