We Keep Coding

Serial communication between teensy bord and system using python

i am fairly new to the field of IOT. I am setting up a sensor with teensy for reading up its data and transmitting over using serial communication to system where using python I am reading the data and storing it into a database.
The problem I am facing is when i check my program using arduino serial monitor I am getting insane sample speed like 10k readings are done in 40 milli seconds but when i try to read the same program using python it is not even giving me more than 1000 readings per second and that too without the database code with it it only reads 200 samples per second. Is there any way i can increase this sample rate or do i have to set any extra parameters for communication through serial ?
Here is my code for teensy :
int i;
elapsedMillis sinceTest1;
void setup()
{
Serial.begin(2000000); // USB is always 12 Mbit/sec
i = 0;
delay(5000);
Serial.println("Setup Called");
Serial.flush();
}
void loop()
{
if (i == 0 || i == 500000)
{
Serial.println(sinceTest1);
}
Serial.println(i);
//Serial.println(Serial.baud());
i++;
}
For python :
import serial
import pymysql
from datetime import datetime
import time
import signal
import sys
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial(
port='COM5',\
baudrate=2000000,\
#baudrate=9600,\
#parity=serial.PARITY_NONE,\
#stopbits=serial.STOPBITS_ONE,\
#bytesize=serial.EIGHTBITS,\
#timeout=0
)
print("connected to: " + ser.portstr)
count=1
#this will store the line
line = []
#database connection
connection = pymysql.connect(host="localhost", user="root", passwd="", database="tempDatabase")
cursor = connection.cursor()
checker = 0
rl = ReadLine(ser)
while True:
time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print(time)
print(checker)
print(rl.readline())
insert1 = ("INSERT INTO tempinfo(value,test,counter) VALUES('{}','{}','{}');".format(33.5, time,checker)) #.format(data[0])
insert2 = ("INSERT INTO urlsync(textvalue,sync) VALUES('http://www.myname.com/value.php?&value={}&time={}',0);".format(33.5,time)) #.format(data[0])
cursor.execute(insert1)
cursor.execute(insert2)
connection.commit()
checker += 1
connection.close()
time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print(time )
ser.close()
P.S : 1000 samples per second is the rate I am getting when I am not using the commands for database, including them I am getting around 250 samples per second only.
Any help or suggestion is appreciated, thank you.

First off, great question. The issue you are facing is loaded with learning opportunities.
Let's go one by one:
-You are now in the position to understand the difference between a microcontroller and a computer. The microcontroller in its most basic form (if you are running bare-metal code, even if it's not very efficient code, like on an Arduino) will do just one thing, and particularly when it's hardware-related (like reading or writing to UARTs) it will do it very efficiently. On a desktop computer, on the other hand, you have layer upon layer of tasks running simultaneously (operating system background tasks, updating the screen and whatnot). With so many things happening at the same time and if you don't establish priorities, it will be very difficult to accurately predict what will exactly happen and when. So it's not only your Python code that is running, there will be many more things that will come up and interrupt the flow of your user task. If you are hoping to read data from the UART buffer at a stable (or at least predictable) speed, that will never happen with the architecture you are using at the moment.
-Even if you manage to strip down your OS to the bare minimum, kill all processes, go on a terminal with no graphics whatsoever... you still have to deal with the uncertainty of what you are doing on your own Python code (that's why you see better performance with the Arduino serial monitor, which is not doing anything other than removing data from the buffer). On your Python code, you are sequentially reading from the port, trying to find a particular character (line feed) and then attaching the data you read to a list. If you want to improve performance, you need to either just read data and store it for offline processing or look at multithreading (if you have a thread of your program dedicated to only reading from the buffer and you do further processing on a separate thread you could improve significantly the throughput, particularly if you set priorities right).
-Last, but actually, most importantly, you should ask yourself: Do I really need to read data from my sensor at 2 Mbps? If the answer is yes, and your sensor is not a video camera, I'm afraid you need to take a step back and look at the following concepts: sensor bandwidth and dynamic response. After you do that, the next question is: how fast is your sensor updating its output and why? is that update rate meaningful? I can give you a couple of references here. First, imagine you have a temperature sensor to read and record the temperature in an oven. Does it make sense to sample values from the sensor at 1 MHz (1 million readings per second) if the temperature in the oven is changing at a rate of 10 degrees C per minute or even 100 degrees per second? Is your sensor even able to react so fast (that where its dynamic response comes into play)? My guess: probably not. Many industrial devices integrate dozens of sensors to control critical processes and send all data through a 1.5 Mbps link (pretty standard for Profibus, for instance).

How to obtain queue capacity and load in zero-mq

I have a publisher-subscriber architecture in ZeroMQ. I use Python.
I need to be able to tell when some queue is about to be too full, and preferably be able to do something about it.
I must be able to know if messages are lost.
I am, however, unable to find the relevant documentation on this subject.
Would love some help
Thanks!

Here is a snippet of what I did
def _flush_zmq_into_buffer(self):
# poll zmq for new messages. if any are found, put all possible items from zmq to self.buffer
# if many readers, need to use a read-write lock with writer priority
# http://code.activestate.com/recipes/577803-reader-writer-lock-with-priority-for-writers/
while True:
self._iteration += 1
self._flush_zmq_once()
sleep(0.005)
def _take_work_from_buffer(self):
while True:
try:
if self._buffer.qsize() > 0:
work_message = self._buffer.get(block=True)
# can't have any heavy operation here! this has to be as lean as possible!
else:
sleep(0.01)
continue
except queue.Empty as ex:
sleep(0.01)
continue
self._work_once(work_message)
def _flush_zmq_once(self):
self.__tick()
flushed_messages = 0
for i in range(self.max_flush_rate):
try:
message = self._parse_single_zmq_message()
self._buffer.put(message, block=True) # must block. can't lose messages.
except zmq.Again: # zmq empty
flushed_messages = i
break
self._log_load(flushed_messages)
self.__tock()
self.__print_flushed(flushed_messages)
This allows me to flush the zmq buffer into my own buffer much faster than I am parsing messages, thus not losing any message, and paying with latency.
This also allows me to know how many messages are flushed from zmq every flushing cycle, thus having an idea about the load.
The reason this is using polling over events, is that for a high rate of incoming messages, the event system would be more costly than the polling system. The last sentence is untested, but I believe it to be true.

pySerial Capturing a long response

Hi guys I'm working a on script that will get data from a host using the Data Communications Standard (Developed by: Data Communication Standard Committee Lens Processing Division of The Vision Council), by serial port and pass the data into ModBus Protocol for the device to perform it's operations.
Since I don't fiscally have access to the host machine I'm trying to develop a secondary script to emulate the host. I am currently on the stage where I need to read a lot of information from the serial port and I get only part of the data. I was hoping to get the whole string sent on the send_job() function on my host emulator script.
Guys also can any of you tell me if this would be a good approach? the only thing the machine is supposed to do is grab 2 values from the host response and assign them to two modbus holding registers.
NOTE: the initialization function is hard coded because it will always be the same and the actual response data will not matter except for status. Also the job request is hard coded i only pass the job # that i get from a modbus holding register, the exact logic on how the host resolved this should not matter i only need to send the job number scanned from the device in this format.
main script:
def request_job_modbus(job):
data = F'[06][1c]req=33[0d][0a]job={job}[0d][0a][1e][1d]'.encode('ascii')
writer(data)
def get_job_from_serial():
response = serial_client.read_all()
resp = response.decode()
return resp
# TODO : SEND INIT SEQUENCE ONCE AND VERIFY IF REQUEST status=0
initiation_request()
init_response_status = get_init_status()
print('init method being active')
print(get_init_status())
while True:
# TODO: get job request data
job_serial = get_job_from_serial()
print(job_serial)
host emulation script:
def send_job():
job_response = '''[06][1c]ans=33[0d]job=30925[0d]status=0;"ok"[0d]do=l[0d]add=;2.50[0d]ar=1[0d]
bcerin=;3.93[0d]bcerup=;-2.97[0d]crib=;64.00[0d]do=l[0d]ellh=;64.00[0d]engmask=;613l[0d]
erdrin=;0.00[0d]erdrup=;10.00[0d]ernrin=;2.00[0d]ernrup=;-8.00[0d]ersgin=;0.00[0d]
ersgup=;4.00[0d]gax=;0.00[0d]gbasex=;-5.30[0d]gcrosx=;-7.96[0d]kprva=;275[0d]kprvm=;0.55[0d]
ldpath=\\uscqx-tcpmain-at\lds\iot\do\800468.sdf[0d]lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]'''.encode('ascii')
writer(job_response)
def get_init_request():
req = p.readline()
print(req)
request = req.decode()[4:11]
# print(request)
if request == 'req=ini':
print('request == req=ini??? <<<<<<< cumple condicion y enviala respuesta')
send_init_response()
send_job()
while True:
# print(get_init_request())
get_init_request()
what I get in screen: main script
init method being active
bce
erd
condition was met init status=0
outside loop
ers
condition was met init status=0
inside while loop
trigger reset <<<--------------------
5782
`:lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]
outside loop
condition was met init status=0
outside loop
what I get in screen: host emulation script
b'[1c]req=ini[0d][0a][1e][1d]'
request == req=ini??? <<<<<<< cumple condicion y enviala respuesta
b''
b'[06][1c]req=33[0d][0a]job=5782[0d][0a][1e][1d]'
b''
b''
b''
b''
b''
b''

I'm suspect you're trying to write too much at once to a hardware buffer that is fairly small. Especially when dealing with low power hardware, assuming you can stuff an entire message into a buffer is not often correct. Even full modern PC's sometimes have very small buffers for legacy hardware like serial ports. You may find when you switch from development to actual hardware, that the RTS and DTR lines need to be used to determine when to send or receive data. This will be up to whoever designed the hardware unfortunately, as they are often also ignored.
I would try chunking your data transfer into smaller bits as a test to see if the whole message gets through. This is a quick and dirty first attempt that may have bugs, but it should get you down the right path:
def get_job_from_serial():
response = b'' #buffer for response
while True:
try:
response += serial_client.read() #read any available data or wait for timeout
#this technically could only be reading 1 char at a time, but any
#remotely modern pc should easily keep up with 9600 baud
except serial.SerialTimeoutException: #timeout probably means end of data
#you could also presumably check the length of the buffer if it's always
#a fixed length to determine if the entire message has been sent yet.
break
return response
def writer(command):
written = 0 #how many bytes have we actually written
chunksize = 128 #the smaller you go, the less likely to overflow
# a buffer, but the slower you go.
while written < len(command):
#you presumably might have to wait for p.dtr() == True or similar
#though it's just as likely to not have been implemented.
written += p.write(command[written:written+chunksize])
p.flush() #probably don't actually need this
P.S. I had to go to the source code for p.read_all (for some reason I couldn't find it online), and it does not do what I think you expect it does. The exact code for it is:
def read_all(self):
"""\
Read all bytes currently available in the buffer of the OS.
"""
return self.read(self.in_waiting)
There is no concept of waiting for a complete message, it just a shorthand for grab everything currently available.

blocking read(1) with timeout in pyserial

I use the following piece of code to read the serial port until i get a terminating character.
"""Read until you see a terminating character with a timeout"""
response=[]
byte_read=''
break_yes=0
time_now = time.clock()
while ((not (byte_read=='\r') ) and (break_yes==0)):
byte_read = self.ser.read(1)
if (not(len(byte_read)== 0) and (not (byte_read =='\r'))):
response.append(byte_read)
if ( time.clock() - time_now > 1 ):
if self.DEBUG_FLAG:
print "[animatics Motor class] time out occured. check code"
break_yes=1
if break_yes==0:
return ''.join(response)
else:
return 'FAIL'
This works well but because of the while loop, the cpu resources are taken up.
I think that having a blocking read(1) with a timeout will save some of the cpu.
The flag i am looking for C is "MIN == 0, TIME > 0 (read with timeout)" in termios
i am looking for a similar flag in Python.
I could also use the io.readline to read till i get '\r', but i want to stick to pyserial as much as possible without any other dependency.
Would greatly appreciate advice. Do let me know if i should do it in a completely different way either too.
Thanks,

You should read the documentation of Pyserial: it clearly states that a timeout of 0 as you pass it to the constructor will turn on non-blocking behaviour:
http://pyserial.sourceforge.net/pyserial_api.html#classes
Just get rid of the timeout parameter, and you should be set.

Aight, so I found out a way. Instead of polling with the no timeout, I use the select module in python, which is similar to the one in C.
It returns if any data is available immediately, or waits for the timeout period and exits, which is precisely what i wanted. I took deets comments for cleaning up the code and it looks like so now.
def readOnly(self):
"""Read until you see a terminating character with a timeout"""
response=[]
byte_read=''
while (not (byte_read=='\r')):
reading,_,_ = select.select([self.ser], [], [], 1) #returns immediately if there is data on serial port. waits 1 second to timeout if not.
if reading !=[]: #something is to be read on the file descriptor
byte_read = self.ser.read(1)
if (byte_read !='\r'):
response.append(byte_read)
else: #'\r' received
return ''.join(response)
break
else:
if self.DEBUG_FLAG:
print "[Motor class] time out occured. check code"
return 'FAIL'
break
`
This decreased the cpu usage from 50% to 5% so life is better now.
Thanks,

Can I avoid a threaded UDP socket in Python dropping data?

First off, I'm new to Python and learning on the job, so be gentle!
I'm trying to write a threaded Python app for Windows that reads data from a UDP socket (thread-1), writes it to file (thread-2), and displays the live data (thread-3) to a widget (gtk.Image using a gtk.gdk.pixbuf). I'm using queues for communicating data between threads.
My problem is that if I start only threads 1 and 3 (so skip the file writing for now), it seems that I lose some data after the first few samples. After this drop it looks fine. Even by letting thread 1 complete before running thread 3, this apparent drop is still there.
Apologies for the length of code snippet (I've removed the thread that writes to file), but I felt removing code would just prompt questions. Hope someone can shed some light :-)
import socket
import threading
import Queue
import numpy
import gtk
gtk.gdk.threads_init()
import gtk.glade
import pygtk
class readFromUDPSocket(threading.Thread):
def __init__(self, socketUDP, readDataQueue, packetSize, numScans):
threading.Thread.__init__(self)
self.socketUDP = socketUDP
self.readDataQueue = readDataQueue
self.packetSize = packetSize
self.numScans = numScans
def run(self):
for scan in range(1, self.numScans + 1):
buffer = self.socketUDP.recv(self.packetSize)
self.readDataQueue.put(buffer)
self.socketUDP.close()
print 'myServer finished!'
class displayWithGTK(threading.Thread):
def __init__(self, displayDataQueue, image, viewArea):
threading.Thread.__init__(self)
self.displayDataQueue = displayDataQueue
self.image = image
self.viewWidth = viewArea[0]
self.viewHeight = viewArea[1]
self.displayData = numpy.zeros((self.viewHeight, self.viewWidth, 3), dtype=numpy.uint16)
def run(self):
scan = 0
try:
while True:
if not scan % self.viewWidth: scan = 0
buffer = self.displayDataQueue.get(timeout=0.1)
self.displayData[:, scan, 0] = numpy.fromstring(buffer, dtype=numpy.uint16)
self.displayData[:, scan, 1] = numpy.fromstring(buffer, dtype=numpy.uint16)
self.displayData[:, scan, 2] = numpy.fromstring(buffer, dtype=numpy.uint16)
gtk.gdk.threads_enter()
self.myPixbuf = gtk.gdk.pixbuf_new_from_data(self.displayData.tostring(), gtk.gdk.COLORSPACE_RGB,
False, 8, self.viewWidth, self.viewHeight, self.viewWidth * 3)
self.image.set_from_pixbuf(self.myPixbuf)
self.image.show()
gtk.gdk.threads_leave()
scan += 1
except Queue.Empty:
print 'myDisplay finished!'
pass
def quitGUI(obj):
print 'Currently active threads: %s' % threading.enumerate()
gtk.main_quit()
if __name__ == '__main__':
# Create socket (IPv4 protocol, datagram (UDP)) and bind to address
socketUDP = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
host = '192.168.1.5'
port = 1024
socketUDP.bind((host, port))
# Data parameters
samplesPerScan = 256
packetsPerSecond = 1200
packetSize = 512
duration = 1 # For now, set a fixed duration to log data
numScans = int(packetsPerSecond * duration)
# Create array to store data
data = numpy.zeros((samplesPerScan, numScans), dtype=numpy.uint16)
# Create queue for displaying from
readDataQueue = Queue.Queue(numScans)
# Build GUI from Glade XML file
builder = gtk.Builder()
builder.add_from_file('GroundVue.glade')
window = builder.get_object('mainwindow')
window.connect('destroy', quitGUI)
view = builder.get_object('viewport')
image = gtk.Image()
view.add(image)
viewArea = (1200, samplesPerScan)
# Instantiate & start threads
myServer = readFromUDPSocket(socketUDP, readDataQueue, packetSize, numScans)
myDisplay = displayWithGTK(readDataQueue, image, viewArea)
myServer.start()
myDisplay.start()
gtk.gdk.threads_enter()
gtk.main()
gtk.gdk.threads_leave()
print 'gtk.main finished!'

UDP doesn't verify the target received it (like TCP does) - you must implement retransmission and such in your applications if you want to ensure all of the data arrives. Do you control the sending UDP source?

UDP is, by definition, unreliable. You must not write programs that expect UDP datagrams to always get through.
Packets are dropped all the time in TCP too, but your program does not need to care, because TCP applications cannot process packets; the TCP stack shows your application a stream of bytes. There is a lot of machinery there to make sure that if you send bytes 'ABCD', you will see 'A' 'B' 'C' 'D' on the end. You may get any possible collection of packets, of course: 'ABC', 'D', or 'AB', CD', etc. Or you may just see 'ABC', and then nothing.
TCP isn't "reliable" because it can magically make your network cables never fail or break; the guarantee that it provides is that up until the point where the stream breaks, you will see everything in order. And after the stream breaks, you'll see nothing.
In UDP there is no such guarantee. If you send four UDP datagrams, 'AB', 'CD', 'EF' 'GH', you may receive all of them, or none of them, or half of them, or just one of them. You may receive them in any order. The only guarantee that UDP tries to provide is that you won't see a message with 'ABCD' in it, because those bytes are in different datagrams.
To sum up: this has nothing to do with Python, or threads, or GTK. It's just a basic fact of life on networks based in physical reality: sometimes the electrical characteristics of your wires are not conducive to getting your messages all the way across them.
You may be able to reduce the complexity of your program by using Twisted, specifically, the listenUDP API, because then you won't be needing to juggle threads or their interaction with GTK: you can just call methods directly on the widget in question from your datagramReceived method. But this won't fix your underlying problem: UDP just drops data sometimes, period. The real solution is to convince your data source to use TCP instead.

Firstly; can you set the recv buffer size for the socket? If so, set it to something very large as this will let the UDP stack buffer more datagrams for you.
Secondly; if you can use asynchronous I/O then post multiple recv calls at once (again this allows the stack to service more datagrams before it starts to drop them).
Thirdly; you could try unrolling your loop a little and reading multiple datagrams before placing them in your queue; could the locking on the queue be causing the recv thread to run slowly??
Finally; the datagrams may be being dropped elsewhere on the network, there may be nothing that you can do, that the U in UDP...

Edit - Struck out listen/accept sentence, thanks Daniel, I was just coming to remove it when I saw your comment :)
I'd suggest that this is a network programming issue, rather than python per-se.
You've set a packet-per-second rate and a duration to define the number of recv calls you make to your UDP socket. I don't see a listen or accept call to the socket, I'll assume that recv handles that as you say you receive some data. You've not mentioned the generation of the data.
You've defined how many reads you're expecting to make, so I'd assume that the code makes that many receives before exiting, so my conclusion would be that your recv packetSize is insufficient and therefore one read isn't pulling an entire datagram, then the subsequent recv is pulling the next part of the previous datagram.
Can't you look at the data you have received and determine what is missing? What data are you "losing"? How do you know it's lost?
Furthermore, you could use wireshark to verify that your host is actually receiving the data at the same time as verifying the size of the datagrams. Match the capture against the data your recv thread is providing.
Update
You say that you're losing data, but not what it is. I see two possibilities for data-loss:
Truncating packets
Dropping packets
You've said that the payload size is the same size as that which you are passing to recv, so I'll take it that you're not truncating.
So the factors for dropping packets are a combination of rate of receipt, rate of read-from-receive-buffer and receive-buffer size.
Your calls to Queue.put may be slowing down your rate of read.
So, first determine that you can read 1200 packets per second by modifying readFromUDPSocket to not Queue.put, but count the number of receives and report time taken.
Once you've determined that you can call recv fast enough, the next step is working out what is slowing you down. I suspect it may be your use of Queue, I suggest batching payloads in N-sized groups for placing on the Queue so that you're not trying to call put at 12Hz.
Seeing as you want to sustain a rate of 1200 reads per second I don't think you'll get very far by increasing the receive buffer on the socket.

It seems that the problem is with the source. There are two issues:
Looking at Wireshark the source is not consistently transmitting 1200 packets-per-second. Possibly, as Len pointed out, a problem with the outbound stack dropping data. BTW the source is a programmable card with an ethernet port connected to my machine.
The other issue is the after the first 15 packets or so of data there is always a drop. What I discovered is that if I recv 20 packets in the initialisation part of the readFromUDPSocket thread, I can then read the data fine, e.g.
class readFromUDPSocket(threading.Thread):
def __init__(self, socketUDP, readDataQueue, packetSize, numScans):
threading.Thread.__init__(self)
self.socketUDP = socketUDP
self.readDataQueue = readDataQueue
self.packetSize = packetSize
self.numScans = numScans
for i in range(0, 20):
buffer = self.socketUDP.recv(self.packetSize)
def run(self):
for scan in range(1, self.numScans + 1):
buffer = self.socketUDP.recv(self.packetSize)
self.readDataQueue.put(buffer)
self.socketUDP.close()
print 'myServer finished!'
Not sure what this points to?! I think all of this rules out not being able to recv and put fast enough though.