We Keep Coding

Why does sending consecutive UDP messages cause messages to arrive late?

I've written a server python script in Windows 7 to send Ethernet UDP packets to a UNIX system running a C client receiving program that sends the message back to the server. However, sometimes (not always) a message in the last port (and always the last port) that python sends to won't arrive until the next batch of 4 messages is sent. This causes the timing of the message received for the last port incorrect to when it was sent, and I cannot have two messages back to back on the same port.
I have been able to verify this in Wireshark by locating two messages that arrived around the same time because the one that wasn't received was processed with the other. I have also checked the timing right after the recv() function and it shows a long delay and then a short delay because it basically had two packets received.
Things I have done to try to fix this, but has help me explain the problem or how to solve it: I can add a delay in between each sendto() and I will successfully send and receive all messages with correct timing but I want the test to work the way I've written it below; I've increased the priority of the receive thread thinking that my Ethernet receive was not getting signal to pick up the package or that some process was taking too long, but this didn't work and 20ms should be WAY more than necessary to process the data; I have removed ports C and D, then port B misses messages (Only having one port doesn't caause issues), I thought reducing the number of ports would improve timing; Sending to a dummy PORTE immediately after PORTD lets me receive all of the messages with correct timing (I assume the problem is transferred to PORTE); I have also reproduced the python script in a UNIX environment and C code and have had the same issue, pointing me to a receiving issue; I've also set my recv function to time out every 1ms hoping that it could recover somehow even though the timing would be off a bit, but I still saw messages back to back. I've also checked that no UDP packets have been dropped and that the buffer is large enough to hold those 4 messages. Any new ideas would help.
This is the core of the code, the python script will send 4 packets. One 20 byte message to a corresponding waiting thread in C and delay for 20ms
A representation of the python code looks something like
msg_cnt = 5000
while cnt < msg_cnt:
UDPsocket.sendto(data, (IP, PORTA))
UDPsocket.sendto(data, (IP, PORTB))
UDPsocket.sendto(data, (IP, PORTC))
UDPsocket.sendto(data, (IP, PORTD))
time.sleep(.02)
cnt++
The C code has 4 threads waiting to receive on their corresponding ports. Essentially each thread should receive its packet, process it, and send back to the server. This process should take less than 20ms before the next set of messages arrive
void * receiveEthernetThread(){
uint8_t ethRxBuff[1024];
if((byteCnt = recv(socketForPort, ethRxBuff, 1024, 0)) < 0){
perror("recv")
}else{
//Process Data, cannot have back to back messages on the same port
//Send back to the server
}
}

I found out the reason I was missing messages a while back and wanted to answer my question. I was running the program on a Zynq-7000 and didn't realize this would be an issue.
In the Xilinx Zynq-7000-TRM, there is a known issue describing:
" It is possible to have the last frame(s) stuck in the RX FIFO with the software having no way to get the last frame(s) out of there. The GEM only initiates a descriptor request when it receives another frame. Therefore, when a frame is in the FIFO and a descriptor only becomes available at a later time and no new frames arrive, there is no way to get that frame out or even know that it exists.
This issue does not occur under typical operating conditions. Typical operating conditions are when the system always has incoming Ethernet frames. The above mentioned issue occurs when the MAC stops receiving the Ethernet frames.
Workaround: There is no workaround in the software except to ensure a continual flow of Ethernet frames."
Was fixed basically by having continuous incoming Ethernet traffic, sorry for missing that crucial information.

This causes the timing of the message received for the last port
incorrect to when it was sent, and I cannot have two messages back to
back on the same port.
The short explanation is you are using UDP, and that protocol gives no guarantees about delivery or order.
That aside, what you are describing most definitely sounds like a buffering issue. Unfortunately, there is no real way to "flush" a socket.
You will either need to use a protocol that guarantees what you need (TCP), or implement your needs on top of UDP.
I believe your real problem though is how the data is being parsed on the server side. If your application is completely dependent on a 20ms interval of four separate packets from the network, that's just asking for trouble. If it's possible, I would fix that rather than attempt fixing (normal) socket buffering issues.
A hacky solution though, since I love hacky things:
Set up a fifth socket on the server. After sending your four time-sensitive packets, send "enough" packets to the fifth port to force any remaining time-sensitive packets through. What is "enough" packets is up to you. You can send a static number, assuming it will work, or have the fifth port send you a message back the moment it starts recv-ing.

How does PySerial work?

Say I have the following python script to read in serial data from my Arduino:
import serial
ser = serial.Serial("dev/ttyACM1", 9600)
ser.timeout = 2
ser.readlines()
On the other end I've flashed my Arduino with a program that sends 20 voltage readings every 0.5 seconds. The Arduino starts sending those readings from the moment it's hooked up, then after 20 seconds it stops and sends nothing.
Now what I've noticed is that I can read those 20 voltage values using the first script whenever I want. That is, I can hook up the Arduino, wait a couple of minutes then read in the values. This makes me think that the data is getting stored somewhere. I'm inclined to think that it's not being stored on the Arduino but on my laptop somewhere.
I've come up with a few questions that I hope the community could help me with:
Where is PySerial getting the data from (the Arduino or some buffer on my laptop)?
For how long is the data stored in this place?
Is there a way to clear this space before reading in values?
How much storage space is there?
When you set the baud rate in PySerial (second line in script), is this the rate that
PySerial reads data from the storage area (not the Arduino)?
I've noticed that if I set the baud rate in PySerial too high the first few lines of data are fragmented and sometimes completely wrong, why?
Not exactly related but when you set serial.Serial.timeout are the
units in seconds?
I appreciate your time.

Have you tried using a terminal program like TerraTerm (windows) or GTKTerm (linux) to open the same port to the arduino? I think this would be helpful to answer some of your questions.
Some quick answers to your questions that I can dump off the top of my head.
From the port specified, I'm guessing you are asking something deeper than this?
If you do a
x = ser.readlines()
then the data will be in x as long as you'd like.
There is a flush function defined in PySerial
Not sure. You can specify how many characters you would like to read though
example:
x = ser.read(number)
The pyserial documentation states the following
Read size bytes from the serial port. If a timeout is set it may return less characters as requested. With no timeout it will block until the requested number of bytes is read.
http://pyserial.sourceforge.net/pyserial_api.html
This is the clock rate of the port you are opening, ie /dev/ttyACM1, most serial comms are at 9600, if you happen to be using a USB to serial you'll need 115200
Clock rate mismatch. You're computer is sampling data at a rate higher than the arduino is providing it, causing it to be incorrectly displayed.
Seconds, quote from Pyserial documentation : "timeout = x: set timeout to x seconds (float allowed)" Same link as number 4
Hope that helps some!

Program execution stopping on atml atmega16 microcontroller upon terminal connection

I am attempting to receive data from the qu-bot at http://www.qu-bot.com. The robot has an ATML atmega16 microcontroller. I have written a program that runs on the robot which outputs data to its serial port. The program however stops whenever I connect to the controller. I tested this by adding a beep statement. The robot beeps as long as the program is running. The beeping stops when I connect to the robot. I tried qu-bo support and they suggested disabling the dtr flag on the serial port. I did that but no joy.
Is there anything else I can try?
[start of code running on the qu-bot]
Note:
This is written in some kind of proprietary variant of C which they call quick c.
// This code displays the uart functions.
int main(void)
{
INIT();
UART_INIT(57600);
UART_PRINT("HELLO!!\n");
DELAYS(1);
UART_PRINT("MY NAME IS QU-BOT.\n");
DELAYS(1);
UART_PRINT("HELLO!!\n");
UART_PRINT("YOU ARE USING UART SAMPLE CODES.\n");
while(1)
{
UART_PRINT("test\n");
BEEP();
DELAYS(60);
}
}
now for my python serial port reading program. I have tried this program both on raspbian and on windows 7 64bit. I am pasting the windows version. The raspbian version has a different name for the linux.
import serial
import time
ser=serial.Serial()
ser.port=8
ser.baudrate=57600
ser.setDsrDtr(False)
print 'initialized'
flag = ser.isOpen()
if flag:
print 'port already open.'
pass
else:
ser.open() # opening the port 'ser' that was just created to receive data
time.sleep(0.5)
print 'ready to read'
print ser
ser.write('a')
s=ser.read(4)
print s
ser.close()
Pranav
P.S. I have consulted the following links amongst others.
<https://learn.sparkfun.com/tutorials/terminal-basics/all>
<http://www.plainlystated.com/2013/06/raspberry-pi-serial-console/>
<http://elinux.org/RPi_Serial_Connection>
<https://learn.sparkfun.com/tutorials/terminal-basics/all>

Based on my experience in serial communications with microcontrollers the most likely cause of this problem is that when you connect the serial cable to the robot it causes a phantom byte (due to electrical noise that happens when you make the connection) to look like it's coming from the controller - either this or the controller is sending a legitimate byte to the robot. In either case it is likely that the arrival of a byte at the robot's serial port (even if it was only electrical noise mistaken to be a data byte - this is a very common occurrence) caused the robot's microcontroller to invoke the UART receive interrupt. If you don't have an appropriate UART receive handler (ISR - Interrupt Service Routine) written and linked into the correct interrupt vector then your robot's microcontroller can go off into "deep space" upon the detection of the first incoming serial data byte - and make your robot appear to "hang". If you intend to do "polled" serial communications (your code manually checks for received bytes in its main loop) instead of interrupt-driven (hardware detection of an incoming byte causes your UART Rx ISR to be invoked) then all you have to do is to disable UART interrupts and your problem should go away.

Echo Program in between Arduino and Python

I want to send some data to an Arduino through pyserial in Python. All I want to the Arduino to do is read the variable length string data from the serial port, and write it back so that Python can read it. Since I've been unable to do that, the code below only has Python sending on character. Here's the Python code:
import serial
import sys
import pywapi
import time
def main():
ser = serial.Serial(3, 9600, timeout=1)
print "Conn established"
print "Sending: %s" % "z".__repr__()
print ser.write('z'.encode("ascii"))
time.sleep(2)
print "Received: %s" % ser.read(10).__repr__()
ser.close()
Here's the Arduino code:
void setup(){
analogReference(DEFAULT);
Serial.begin(9600);
}
void loop(){
if(Serial.available() > 0)
Serial.println("x");
while(Serial.available() > 0){
Serial.print(Serial.read(), BYTE);
}
}
The output:
Conn established
Sending: 'z'
1
Received: ''
I know the code for the Arduino works because it works when data is being sent from the Arduino terminal. However, the moment I try to send anything from Python it fails. I've been struggling with this all day. Any help would be greatly appreciated.

Try increasing or removing the timeout, and set read's size to 1. You may also want to increase the sleep delay, or even implement a simple read loop.
Something like:
try:
while True:
data = ser.read(1).__repr__()
if data:
print "Received: %s." % data
else:
print "Looping."
except KeyboardInterrupt:
print "Done."
except:
raise
finally:
ser.close()
print "Closed port."
Then just use ctrl-c to stop it.

I would recommend verifying the two parts independently, using a separate serial port and serial comms software on the PC.
E.g. if your PC has two serial ports, then use a null-modem (loopback) cable to connect them. Or use com0com to make a pair of linked virtual serial ports. Run your Python software on one serial port, and a terminal program (Hyperterminal or RealTerm) on the other serial port. Manually verify the Python program's operation that way.
Then, connect your PC directly to the Arduino as usual, and use the terminal software to manually verify the Arduino software operation.
That process will allow you to narrow down the problem. Once you've verified them both, they should work well together.
Serial Port Monitor
Another method you can use is software that hooks into the PC's serial port driver, and allows you to monitor traffic on the serial port. I've used the Free Serial Port Monitor software from HHD Software in the past, and it worked well for our purposes. It allows you to monitor any of the PC's serial ports, and shows you a log (hex and text) of the serial data going over the port in both directions.

Do you need to flush the sent character out of any held serial buffer?
It may be that your character is not actually leaving the COM port and arriving at the Arduino. When you test this with the Arduino Terminal (I assume you mean the UI terminal in the development environment) you are actually sending your string + a carriage return I think, not just the character. (i.e. do you hit return after you type 'z' in your test?)
Try ser.flush() or perhaps also send a \r character. From your testing the Arduino works just fine, it's the python program that doesn't seem to be sending anything.

The reason you may have to send twice is that, if you're connecting via the USB, the first serial connection will reset the Arduino.

Pyserial problem with Arduino - works with the Python shell but not in a program

All right, so I am positive my Arduino circuit is correct and the code for it. I know this because when I use the serial monitor built into the Arduino IDE and send 'H' an LED lights up, when I send 'L' that LED turns off.
Now I made a Python program
import serial
ser = serial.Serial("COM4",9600)
ser.write("H")
When I run the code the LED blinks on for a second then goes back off.
However when I do each of these lines separately in the shell it works just like it is supposed to.
Any ideas?

When you open the serial port, this causes the Arduino to reset. Since the Arduino takes some time to bootup, all the input goes to the bitbucket (or probably to the bootloader which does god knows what with it). If you insert a sleep, you wait for the Arduino to come up so your serial code. This is why it works interactively; you were waiting the 1.5 seconds needed for the software to come up.
I confirmed that opening the serial port resets my Arduino Uno; I flashed a program which will blink the LED from the setup() routine -- calling open("/dev/ttyACM0") was sufficient to trigger the reset. This is IMHO a confusing and undocumented wrinkle in the serial support.

I had the same problem and it works if I add a delay of about 2 seconds from opening the serial connection to writing on it, 1 second was not enough.

Just to make it a bit more clear I'll modify the code so everyone can see what needs to be added!
import serial
import time
ser = serial.Serial("COM4",9600)
time.sleep(3)
ser.write("H")
Adding in a sleep statment helps to let the serial open up without any problems!

Assuming you are using an Arduino Uno
The USB port and the Uno serial bus exposed on pins 1 and 0 share the same RX/TX lines. I suggest getting a USB to TTL adapter like the one here so that you can communicate to the Arduino without using the USB port. The Arduino IDE has its own method for disengaging from the USB driver such that a virtual serial port can be created. Have your Ardunio use SoftwareSerial instead.
Here is an example I found on the internet where somebody had clashing bus issues.