We Keep Coding

PySerial: writing to microcontroller using a while loop adds 100ms to the loop time

I'm trying to do actuator control with values calculated from Python. I am at a point where I want to send the calculated data to a microcontroller in an infinite loop. It seems like ser.write() adds 100ms (total loop execution time of 120ms)
Right now I'm sending the value obtained by my Python code to a microcontroller using PySerial.
Python's data processing side isn't the problem because I checked that the data was being calculated every 20ms.
Below is the code I am using.
import serial as ser
COM = 'COM4'
brate = 115200
ser = ser.Serial()
ser.baudrate=brate
ser.port = COM
while(1):
##Data Calculation Code###
ser.open()
send = bytearray([int(calculated_data)])
ser.write(send)
if(i !=0):
ser.close()
Thank you for your time

I just figured it out after searching a little bit more.
Seems like the serial port doesn't have to be opened and closed everytime.
from serial import Serial
ser = ser.Serial(COM, brate)
Did the trick.

Serial communication between teensy bord and system using python

i am fairly new to the field of IOT. I am setting up a sensor with teensy for reading up its data and transmitting over using serial communication to system where using python I am reading the data and storing it into a database.
The problem I am facing is when i check my program using arduino serial monitor I am getting insane sample speed like 10k readings are done in 40 milli seconds but when i try to read the same program using python it is not even giving me more than 1000 readings per second and that too without the database code with it it only reads 200 samples per second. Is there any way i can increase this sample rate or do i have to set any extra parameters for communication through serial ?
Here is my code for teensy :
int i;
elapsedMillis sinceTest1;
void setup()
{
Serial.begin(2000000); // USB is always 12 Mbit/sec
i = 0;
delay(5000);
Serial.println("Setup Called");
Serial.flush();
}
void loop()
{
if (i == 0 || i == 500000)
{
Serial.println(sinceTest1);
}
Serial.println(i);
//Serial.println(Serial.baud());
i++;
}
For python :
import serial
import pymysql
from datetime import datetime
import time
import signal
import sys
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial(
port='COM5',\
baudrate=2000000,\
#baudrate=9600,\
#parity=serial.PARITY_NONE,\
#stopbits=serial.STOPBITS_ONE,\
#bytesize=serial.EIGHTBITS,\
#timeout=0
)
print("connected to: " + ser.portstr)
count=1
#this will store the line
line = []
#database connection
connection = pymysql.connect(host="localhost", user="root", passwd="", database="tempDatabase")
cursor = connection.cursor()
checker = 0
rl = ReadLine(ser)
while True:
time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print(time)
print(checker)
print(rl.readline())
insert1 = ("INSERT INTO tempinfo(value,test,counter) VALUES('{}','{}','{}');".format(33.5, time,checker)) #.format(data[0])
insert2 = ("INSERT INTO urlsync(textvalue,sync) VALUES('http://www.myname.com/value.php?&value={}&time={}',0);".format(33.5,time)) #.format(data[0])
cursor.execute(insert1)
cursor.execute(insert2)
connection.commit()
checker += 1
connection.close()
time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print(time )
ser.close()
P.S : 1000 samples per second is the rate I am getting when I am not using the commands for database, including them I am getting around 250 samples per second only.
Any help or suggestion is appreciated, thank you.

First off, great question. The issue you are facing is loaded with learning opportunities.
Let's go one by one:
-You are now in the position to understand the difference between a microcontroller and a computer. The microcontroller in its most basic form (if you are running bare-metal code, even if it's not very efficient code, like on an Arduino) will do just one thing, and particularly when it's hardware-related (like reading or writing to UARTs) it will do it very efficiently. On a desktop computer, on the other hand, you have layer upon layer of tasks running simultaneously (operating system background tasks, updating the screen and whatnot). With so many things happening at the same time and if you don't establish priorities, it will be very difficult to accurately predict what will exactly happen and when. So it's not only your Python code that is running, there will be many more things that will come up and interrupt the flow of your user task. If you are hoping to read data from the UART buffer at a stable (or at least predictable) speed, that will never happen with the architecture you are using at the moment.
-Even if you manage to strip down your OS to the bare minimum, kill all processes, go on a terminal with no graphics whatsoever... you still have to deal with the uncertainty of what you are doing on your own Python code (that's why you see better performance with the Arduino serial monitor, which is not doing anything other than removing data from the buffer). On your Python code, you are sequentially reading from the port, trying to find a particular character (line feed) and then attaching the data you read to a list. If you want to improve performance, you need to either just read data and store it for offline processing or look at multithreading (if you have a thread of your program dedicated to only reading from the buffer and you do further processing on a separate thread you could improve significantly the throughput, particularly if you set priorities right).
-Last, but actually, most importantly, you should ask yourself: Do I really need to read data from my sensor at 2 Mbps? If the answer is yes, and your sensor is not a video camera, I'm afraid you need to take a step back and look at the following concepts: sensor bandwidth and dynamic response. After you do that, the next question is: how fast is your sensor updating its output and why? is that update rate meaningful? I can give you a couple of references here. First, imagine you have a temperature sensor to read and record the temperature in an oven. Does it make sense to sample values from the sensor at 1 MHz (1 million readings per second) if the temperature in the oven is changing at a rate of 10 degrees C per minute or even 100 degrees per second? Is your sensor even able to react so fast (that where its dynamic response comes into play)? My guess: probably not. Many industrial devices integrate dozens of sensors to control critical processes and send all data through a 1.5 Mbps link (pretty standard for Profibus, for instance).

SMBus write_i2c_block_data() command

So I started a project on a microcontroler that uses I2C communication a month ago with no knowledge about anything but python.
I have to talk to a peristaltic pump that uses ASCII strings for communication.
So my setup currently consists of a Raspberry Pi, the I2C bus, Arduino and the peristaltic pump. The Arduino is only used as a powersupply. I thought a good starting point would be just to try and turn the pumps LED on and off. The code for LED on is "L,1" and for LED off is "L,0". (The "" indicates that whats inside it is the absolute code). [a link] https://www.atlas-scientific.com/_files/_datasheets/_peristaltic/EZO_PMP_Datasheet.pdf
By using smbus.SMBus in python I sent the data through the command through write_i2c_block_data. Documentation of smbus gives the following: write_i2c_block_data(int addr,char cmd,long vals[]) however i dont understand what is meant with 'char cmd'. I couldnt put a string command in there and it only worked when I put an integer in there.
Here is the code:
import smbus
import time
bus = smbus.SMBus(1)
slave_address = 0x67
led_on = 'L,1'
led_off = 'L,0'
def string_to_charlist(a_string):
stringlist = list(a_string)
return stringlist
def string_to_intlist(a_string):
lst = string_to_charlist(a_string)
intlist = []
for i in range(len(lst)):
an_int = string_to_charlist(lst[i])
intlist.append(an_int)
return intlist
ledon_intlist = string_to_intlist(led_on)
ledoff_intlist = string_to_intlist(led_off)
# this will result in ledon_intlist = [76,44,49]
# this will result in ledon_int =list [76,44,48]
command1_on = ledon_intlist.pop(0)
command1_off = ledoff_intlist.pop(0)
for i in range(1):
time.sleep(0.5)
bus.write_i2c_block_data(slave_address, command1_on, ledon_intlist)
time.sleep(0.5)
bus.write_i2c_block_data(slave_address, command1_on, ledon_intlist)
After running this code through the raspberry Pi command prompt the pumps LED started blinking on the given time frame. Unfortunately it never stopped blinking and also didt show up when I searched for it using i2ctools command i2cdetect -y 1 I assume the pump's chip is in an infinite loop now.
My questions are:
1. How should one use the write_i2c_block_data() command and what argument does it take. Currently I figured that the 1st argument is the slave address, 2nd is the initial byte of the stream and the 3rd argument is the rest of the stream integer values to be sent.
2. What possibly could have gone wrong that the pump is stuck in an infinite loop and how do I fix it

Multitasking Python-Arduino Communication

I am doing a project where I have to acquire data from an Arduino Uno connected to various sensors using Python on Windows (and eventually in Raspberry Pi 2 Model B).
As Python would constantly be listening to Arduino, I was thinking how should I let Python be always ready to read in a user input (any keyboard keys or even better, with a push-button if this is possible) after which Python will tell Arduino Uno to re-acquire data from all the sensors (i.e. refresh data from the sensors) and print it on the Python Console. I am currently using PyCharm with pySerial for communications between Arduino and Python. My code (without considering user input) is as of below:
import sys
import serial
import time
arduino = serial.Serial('COM3', 9600, timeout=1)
time.sleep(3) # wait for Arduino to initialize
def readData(): # reads inputs from Arduino
try:
datastring = arduino.readline()
print datastring
except:
pass
while True:
readData()
strin = 'p'
arduino.write(strin.encode()) # tell arduino a phase shifter setting has been finished
strin = 's'
arduino.write(strin.encode())
arduino.close()
time.sleep(0.5) # waits for 0.5 s
# print('Data to be transfered: %s'%ASCIIdata)
I understand that related topics have been posted before, however, I have tried the solutions online to no avail and I am finding it quite difficult to code as I am very new to Python. (I am reading documentation and examples and trying to code at the moment.)
Currently, I have done some research on this topic and have tried the following methods but to no avail:
Using msvcrt, getch and using nodelay(1) to allow my program to run even when no user input is received. I used "stdscr", and curses, to which I used "stdscr = curses.initscr()", however, in PyCharm, as it is an IDE, it returned an error "Redirection is not supported." My attempt at implementing this is as of below:
def readData(): # reads inputs from Arduino
try:
datastring = arduino.readline()
print datastring
except:
pass
while True:
stdscr = curses.initscr()
stdscr.nodelay(1)
user_input = stdscr.getch()
if user_input == -1:
readData()
I understand there is multithreading in Python and a number of leads lead me to think that it should be the way to go if I want to listen to user inputs continuously while still acquiring data from Arduino. May I ask if this is the correct direction in which I should pursue?
I have heard about select() and poll() though the former isn't supported on Windows. I will try select() when I port my program over to Raspberry Pi which runs on Linux.
May I ask if anyone can point me to a direction on how to implement a program that reads data continuously from Arduino in PyCharm, and but a user input will cause everything to stop (like an interrupt) and have Python (PyCharm) ask Arduino to pass it a set of fresh sensor data?
Thank you! :)

Two simultaneous Python loops with one result

I currently have a Python 2.6 piece of code that runs two loops simultaneously. The code uses the gps (gpsd) module and the scapy module. Basically the first function (gpsInfo) contains a continual while loop grabbing GPS data from a GPS device and writing the location to console. The second function (ClientDetect) runs in a continual loop also sniffs the air for wifi data and prints this data when specific packets are found. I've threaded these two loops with the GPS one running as a background thread. What I am looking to do (and have been struggling for 5 days to work out how) is for, when the ClientDetect function finds a match and prints the respective info, I want the respective GPS coordinates of when that hit was made also printed to console. At present my code doesn't seem to work.
observedclients = [] p = "" # Relate to wifi packet session =
gps.gps(mode=gps.WATCH_NEWSTYLE)
def gpsInfo():
while True:
session.poll()
time.sleep(5)
if gps.PACKET_SET:
session.stream
print session.fix.latitude + session.fix.longitude
time.sleep(0.1)
def WifiDetect(p):
if p.haslayer(Dot11):
if p.type == 0 and p.subtype in stamgmtstypes:
if p.addr2 not in observedclients:
print p.addr2
observedclients.append(p.addr2)
def ActivateWifiDetect():
sniff(iface="mon0", prn=WifiDetect)
if __name__ == '__main__':
t = threading.Thread(target=gpsInfo)
t.start()
WifiDetect()
Can anybody look at my code to see how best to grab the data simultaneously for when there is a wifi hit, for the GPS coordinates to be printed too? Somebody mentioned implementing queuing but I have researched this but to no avail with regards to how to implement it.
As said, the aim of this code is to scan for both GPS and specific wifi packets and when detected, print details relating to the packet and the location where is was detected.

A simple way of getting this is store the gps location in a global variable, and have the wifi sniffing thread read that global when it needs to print some data; The gotcha is that since two threads can be accessing the global variable at the same time, you'll want to wrap it with a mutex;
last_location = (None, None)
location_mutex = threading.Lock()
def gpsInfo():
global last_location
while True:
session.poll()
time.sleep(5)
if gps.PACKET_SET:
session.stream
with location_mutex:
# DON'T Print from inside thread!
last_location = session.fix.latitude, session.fix.longitude
time.sleep(0.1)
def WifiDetect(p):
if p.haslayer(Dot11):
if p.type == 0 and p.subtype in stamgmtstypes:
if p.addr2 not in observedclients:
with location_mutex:
print p.addr2, last_location
observedclients.append((p.addr2, last_location))

you need to tell python you are using an external variable when you use gps in a function. The code should look like this:
def gpsInfo():
global gps # new line
while True:
session.poll()
time.sleep(5)
if gps.PACKET_SET:
session.stream
print session.fix.latitude + session.fix.longitude
time.sleep(0.1)
def WifiDetect(p):
global p, observedclients # new line
if p.haslayer(Dot11):
if p.type == 0 and p.subtype in stamgmtstypes:
if p.addr2 not in observedclients:
print p.addr2
observedclients.append(p.addr2)

I think you should be more specific in your goal.
If all you want to do is get GPS coords when a Wifi network is sniffed, just do (psuedo-code):
while True:
if networkSniffed():
async_GetGPSCoords()
If you want a log of all GPS coords and want to match that up with Wifi network data, just print all the data out along with timestamps, and do post-processing do match up Wifi networks with GPS coords via timestamp.