We Keep Coding

How to extract data from Arduino only when it is available and save the data?

I am trying to extract sensor data from Arduino. Arduino code sends sensor data using serial communication every 1 second. My goal is to display this data and plot it in real-time in a GUI using Tkinter. But where I seem to be stuck is retrieving data from Arduino. In my code, I tried to extract the data and save them in multiple variables. But when Arduino is sending only 5 seconds worth of data which is 5 sensor measurements, my python code is saving more than 60 measurements in the variable. I don't have any idea how to solve this. I have been stuck here for some time. Here is my python code.
In the given pictures I ran the code for 5 seconds. That is 5 measurements will be sent from Arduino. I am recieving the data correctly, but when I process them and seperate them in individual variables I have analog_voltage containing:80 , current:70, voltage:75 and temperature:65 measurements as they should only contain 5. What should I do, is there any way to go around this problem?
This is the same issue when I am trying to save the file. For the 5 seconds of data the program saves a million rows of data in csv. Any help is much appreciated. Thank you for reading.
The image for arduino data is just for reference as it is not possible to run both the programs in a same time due to the serial communication protocols hence the data looks different.
data_to_sort =[]
analog_voltage = []
Voltage= []
Current=[]
Temperature=[]
def arduino_read_serialdata():
global ser
ser = serial.Serial('COM4', 9600, timeout=1)
time.sleep(1.0)
ser.flushInput()
while True:
global analog_voltage, Voltage, Temperature, Current, yp, cleaned_data, anvol
incoming_data = ser.inWaiting()
incoming_data = ser.readline()
incoming_data = incoming_data.decode()
cleaned_data = incoming_data.strip("/n")
data_to_sort.append(cleaned_data)
for everyrow in data_to_sort:
if everyrow.startswith("Measuring voltage and current with INA219 ..."):
pass
elif everyrow.startswith("MAX31855 test"):
pass
elif everyrow.startswith("Initializing sensor...DONE checking reading the values now."):
pass
elif everyrow.startswith("The Analog Voltage is:"):
anvol = everyrow[-8:]
anvol = float(anvol)
analog_voltage.append(anvol)
elif everyrow.startswith("Bus Voltage............:"):
vol = everyrow[-8:]
Voltage.append(float(vol))
elif everyrow.startswith("Current..............:"):
cur = everyrow[-8:]
Current.append(float(cur))
elif everyrow.startswith("Temperature in celcius:"):
temp = everyrow[-7:]
Temperature.append(float(temp))
else:
pass
with open("Sensor_data.csv", 'w+', newline='') as output_file:
output_file.flush()
output_file = csv.writer(output_file)
fields = ['Analog Voltage V', 'Voltage V', 'Current mA', 'Temperature °C']
output_file.writerow(fields)
for idx in range(0, len(analog_voltage)):
output_file.writerow([analog_voltage[idx]])
if keyboard.is_pressed("q"):
print("Manually interrupted by user")
ser.close()
# output_file.close()
print("Serial port closed")
file.close()
print("File closed, Writing done")

You have an array
data_to_sort =[]
and you append each incoming line to it
data_to_sort.append(cleaned_data)
And with each incoming data, you process the whole data
for everyrow in data_to_sort:
and add the values to these four lists:
analog_voltage = []
Voltage= []
Current=[]
Temperature=[]
and these lists are never cleared.
What should be done instead:
either clear these four lists with each incoming data or
process only the incoming line
My personal preference would be the latter. Instead of always writing the whole output file, append only the last line to the output file using "a" instead of "w+".

How can I put a Python function to sleep while still calling it (IFTTT)

I have a Python function set up to text me if my house gets above 30 degrees Celsius. The script also drives and LCD display that loops through various bits of weather info: house temp, humidity, outdoor conditions, and the times of streetcars.
Because the script is based on a loop, I get a text message every minute or so as long as the temperature is above 30 C. Ideally, I would like to find an elegant way to put the function to sleep while still calling it to check the temperature.
Below is an example of the code I'm using to trigger the IFTTT:
def send_event(api_key, event, value1=None, value2=None, value3=None):
"""Send an event to the IFTTT maker channel"""
url = "https://maker.ifttt.com/trigger/{e}/with/key/{k}/".format(e=event,
k=api_key)
payload = {'value1': value1, 'value2': value2, 'value3': value3}
return requests.post(url, data=payload)
Any and all help appreciated.
Thanks!

If I understand correctly, the problem is that you get too many texts. In that case, store some state about the previous events and use that to decide whether or not to send a text.
For example, a simple boolean (True/False) variable can be used as a simple flag. You can still use a loop, but only send an event on the first time it gets above 30 degrees and reset when it falls below:
temp_is_high = False
while True:
data = weather.get_data()
if data['temp'] > 30:
# only send if we are going from "cool" to "hot"
# not if we were already in "hot" mode
if not temp_is_high:
# we are not in "hot" mode: remember for next time
temp_is_high = True
send_event(...)
else:
# reset the condition flag
temp_is_high = False
There are variation on this theme. For example, you might want to add hysteresis in so that if your thermostat is set to 30 degrees, and the house temperature is hovering around that, measurements of [29, 30, 29, 30, 29, 30, ....] won't send a text every time. To do that, only reset the "hot mode" flag when the temperature, having passed 30, falls below (say) 26 degrees. Or 1 hour has passed, or any number of your own requirements.

Instead of trying to stop your loop from continuing, you should record when the last alert was sent and only resend if enough time has elapsed since the last alert.
Whenever a temperature alert gets sent, have it record the datetime to a variable. Each time the function is called, have it compare the current datetime to the variable to see if the difference is greater than a certain threshold. If it is, re-send the alert and replace the last alert veriable with the current datetime.
from datetime import datetime
alert_interval = 1800 # 30 minutes in seconds
last_alert = datetime(1, 1, 1) # set the initial time to very long ago
def send_event():
time_diff = datetime.now() - last_alert # get a timedelta
if time_diff.total_seconds() >= alert_interval:
last_alert = datetime.now()
# call the API to send another alert

Python 2.7 simple packet sniffer

I wrote simple packet sniffer in Python. I need to receive packets non-stop and send one packet every 10 seconds. I tried this:
current = time.time()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("xx.xx.xx.xx",xxxx))
time.sleep(0.5)
while True:
msg = str(s.recv(4096))
time.sleep(0.010)
print msg
if current==current+10:
s.send("<myPacket/>")
current = time.time()
but it doesn't work good. Anyone have better idea?

Your time handling is bad, use this instead:
While True:
...
time.sleep(10)
Your code doesn't work because:
'current' can never be equal to itself+10.
Also note that time.time() returns a float value e.g: 1440185304.78
which is very accurate to that exact moment, you should never assume you can find that exact float +10 back.
Using a larger/smaller statement is better in this case since the exact value might have passed while your loop is running whatever is in it, e.g:
t = time.time()
while True:
while time.time() < t + 10:
time.sleep(0.1)
t = time.time()
print ("hi")

How do I create this loop of sending large email and small email every 1min within a 5min interval

I am trying to formulate the logic for the following ....
1. Send a large e-mail (1MB or so) every 1 min for 5mins.
2. Wait for 5 mins
3. Send 5 small e-mails (few 100 bytes) every 1 min for 5mins
4. Then wait for 5 mins and start again 1 repeating the same cycle.
5. Start with 1 again
I created an email script that can accept 1MB and few 100 bytes attachment,
my confusion is how do I create this loop of sending large email and small email every 1min within a 5min interval
import time
def email(attachment)
.......
while true:
time.sleep(60) #Delay for 1 minute
email(attachment)

There are at least two ways to approach this.
The first is to use python completely.
def send_mail(large=True):
"""Send 5 emails, 1 minute apart."""
for _ in range(5):
# send an email here, large or small.
if large:
send_large_email()
else:
send_small_email()
wait() # 1 minute
def wait(mins=1):
time.sleep(mins * 60)
def main():
start = time.time()
day = 24 * 60 * 60
while time.time() - start < day:
send_mail()
wait(5) # wait 5 minutes
send_mail(large=False)
wait(5)
That's the basic outline you'd use.
The alternative approach is to use cron. Create two scripts, one to send a small email, and one to send a large email. Set cron to run those scripts at the proper intervals.

You could solve this in a lot of different ways, here is just one example. Hope it helps.
import time
def send_email(email):
print 'sending %s' % email
def get_large_email():
return 'large'
def get_small_email():
return 'small'
def get_email(size):
if size == 'large':
email_to_send = get_large_email()
elif size == 'small':
email_to_send = get_small_email()
return email_to_send
email_size = 'large'
while True:
for i in range(5):
email_to_send = get_email(email_size)
send_email(email_to_send)
time.sleep(60)
email_size = 'small' if email_size == 'large' else 'large'

How does paramiko Channel.recv() exactly work?

I'm having a hard time understanding how the recv() function works.
http://docs.paramiko.org/en/1.13/api/channel.html#paramiko.channel.Channel.recv
I understand this is receiving a chunk a data each time you call the function, but can someone elaborate on the structure or size of this data? Lets say I send a command date, I notice:
1st read gets: "date"
2nd read gets: actual response (Mon Jun 9 12:04:17 CDT 2014)
3rd read gets: prompt
But how does this handle debugging messages that appear randomly on the terminal?
Does the previous pattern hold true as long as the actual response is less than maximum bytes (nbytes)?
What happens if it exceeds nbytes?
As per request, I've added a snippet of the code below:
while reads<maxReads:
resp = self.__chan.recv(maxBytes)
print resp
self.__buffer += resp
if resp.endswith('$ ') or resp.endswith('# '):
break
reads += 1

Channel recv() corresponds to a socket.recv(), it does not have any specific structure or size, it just reads whatever data was sent from the remote server, not exceeding maxBytes.
You commonly use recv() in a loop until you get a piece of data that you are waiting for:
def _wait_for_data(self, options, verbose=False):
chan = self.chan
data = ""
while True:
x = chan.recv(1024)
if len(x) == 0:
self.log("*** Connection terminated\r")
sys.exit(3)
data += x
if verbose:
sys.stdout.write(x)
sys.stdout.flush()
for i in range(len(options)):
if re.search(options[i], data):
return i
return -1