Capturing RTP Timestamps - python

I was trying a little experiment in order to get the timestamps of the RTP packets using the VideoCapture class from Opencv's source code in python, also had to modify FFmpeg to accommodate the changes in Opencv.
Since I read about the RTP packet format.Wanted to fiddle around and see if I could manage to find a way to get the NTP timestamps. Was unable to find any reliable help in trying to get RTP timestamps. So tried out this little hack.
Credits to ryantheseer on github for the modified code.
Version of FFmpeg: 3.2.3
Version of Opencv: 3.2.0
In Opencv source code:
modules/videoio/include/opencv2/videoio.hpp:
Added two getters for the RTP timestamp:
.....
/** #brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
CV_WRAP virtual int64 getRTPTimeStampSeconds() const;
/** #brief Gets the lower bytes of the RTP time stamp in NTP format (fraction of seconds).
*/
CV_WRAP virtual int64 getRTPTimeStampFraction() const;
.....
modules/videoio/src/cap.cpp:
Added an import and added the implementation of the timestamp getter:
....
#include <cstdint>
....
....
static inline uint64_t icvGetRTPTimeStamp(const CvCapture* capture)
{
return capture ? capture->getRTPTimeStamp() : 0;
}
...
Added the C++ timestamp getters in the VideoCapture class:
....
/**#brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampSeconds() const
{
int64 seconds = 0;
uint64_t timestamp = 0;
//Get the time stamp from the capture object
if (!icap.empty())
timestamp = icap->getRTPTimeStamp();
else
timestamp = icvGetRTPTimeStamp(cap);
//Take the top 32 bytes of the time stamp
seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
return seconds;
}
/**#brief Gets the lower bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampFraction() const
{
int64 fraction = 0;
uint64_t timestamp = 0;
//Get the time stamp from the capture object
if (!icap.empty())
timestamp = icap->getRTPTimeStamp();
else
timestamp = icvGetRTPTimeStamp(cap);
//Take the bottom 32 bytes of the time stamp
fraction = (int64)((timestamp & 0xFFFFFFFF));
return fraction;
}
...
modules/videoio/src/cap_ffmpeg.cpp:
Added an import:
...
#include <cstdint>
...
Added a method reference definition:
...
static CvGetRTPTimeStamp_Plugin icvGetRTPTimeStamp_FFMPEG_p = 0;
...
Added the method to the module initializer method:
...
if( icvFFOpenCV )
...
...
icvGetRTPTimeStamp_FFMPEG_p =
(CvGetRTPTimeStamp_Plugin)GetProcAddress(icvFFOpenCV, "cvGetRTPTimeStamp_FFMPEG");
...
...
icvWriteFrame_FFMPEG_p != 0 &&
icvGetRTPTimeStamp_FFMPEG_p !=0)
...
icvGetRTPTimeStamp_FFMPEG_p = (CvGetRTPTimeStamp_Plugin)cvGetRTPTimeStamp_FFMPEG;
Implemented the getter interface:
...
virtual uint64_t getRTPTimeStamp() const
{
return ffmpegCapture ? icvGetRTPTimeStamp_FFMPEG_p(ffmpegCapture) : 0;
}
...
In FFmpeg's source code:
libavcodec/avcodec.h:
Added the NTP timestamp definition to the AVPacket struct:
typedef struct AVPacket {
...
...
uint64_t rtp_ntp_time_stamp;
}
libavformat/rtpdec.c:
Store the ntp time stamp in the struct in the finalize_packet method:
static void finalize_packet(RTPDemuxContext *s, AVPacket *pkt, uint32_t timestamp)
{
uint64_t offsetTime = 0;
uint64_t rtp_ntp_time_stamp = timestamp;
...
...
/*RM: Sets the RTP time stamp in the AVPacket */
if (!s->last_rtcp_ntp_time || !s->last_rtcp_timestamp)
offsetTime = 0;
else
offsetTime = s->last_rtcp_ntp_time - ((uint64_t)(s->last_rtcp_timestamp) * 65536);
rtp_ntp_time_stamp = ((uint64_t)(timestamp) * 65536) + offsetTime;
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp;
libavformat/utils.c:
Copy the ntp time stamp from the packet to the frame in the read_frame_internal method:
static int read_frame_internal(AVFormatContext *s, AVPacket *pkt)
{
...
uint64_t rtp_ntp_time_stamp = 0;
...
while (!got_packet && !s->internal->parse_queue) {
...
//COPY OVER the RTP time stamp TODO: just create a local copy
rtp_ntp_time_stamp = cur_pkt.rtp_ntp_time_stamp;
...
#if FF_API_LAVF_AVCTX
update_stream_avctx(s);
#endif
if (s->debug & FF_FDEBUG_TS)
av_log(s, AV_LOG_DEBUG,
"read_frame_internal stream=%d, pts=%s, dts=%s, "
"size=%d, duration=%"PRId64", flags=%d\n",
pkt->stream_index,
av_ts2str(pkt->pts),
av_ts2str(pkt->dts),
pkt->size, pkt->duration, pkt->flags);
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp; #Just added this line in the if statement.
return ret;
My python code to utilise these changes:
import cv2
uri = 'rtsp://admin:password#192.168.1.67:554'
cap = cv2.VideoCapture(uri)
while True:
frame_exists, curr_frame = cap.read()
# if frame_exists:
k = cap.getRTPTimeStampSeconds()
l = cap.getRTPTimeStampFraction()
time_shift = 0x100000000
#because in the getRTPTimeStampSeconds()
#function, seconds was multiplied by 0x10000000
seconds = time_shift * k
m = (time_shift * k) + l
print("Imagetimestamp: %i" % m)
cap.release()
What I am getting as my output:
Imagetimestamp: 0
Imagetimestamp: 212041451700224
Imagetimestamp: 212041687629824
Imagetimestamp: 212041923559424
Imagetimestamp: 212042159489024
Imagetimestamp: 212042395418624
Imagetimestamp: 212042631348224
...
What astounded me the most was that when i powered off the ip camera and powered it back on, timestamp would start from 0 then quickly increments. I read NTP time format is relative to January 1, 1900 00:00. Even when I tried calculating the offset, and accounting between now and 01-01-1900, I still ended up getting a crazy high number for the date.
Don't know if I calculated it wrong. I have a feeling it's very off or what I am getting is not the timestamp.

As I see it, you receive a timestamp of type uint64 which contains to values uint32 in the high and low bits. I see that in a part of the code you use:
seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
Which basically removes the lower bits and shifts the high bits to be in the lower bits. Then you cast it to int64. Here I only consider that it should be unsigned first of all, since it should not be negative in any case (seconds since epoch is always positive) and it should be uint32, since it is guarantee it is not bigger (you are taking only 32 bits). Also, this can be achieved (probably faster) with bitshifts like this:
auto seconds = static_cast<uint32>(timestamp >> 32);
Another error I spotted was in this part:
time_shift = 0x100000000
seconds = time_shift * k
m = (time_shift * k) + l
Here you are basically reconstructing the 64 bit timestamp, instead of creating the timestamp usable in other contexts. This means, you are shifting the lower bits in seconds to higher bits and adding the fraction part as the lower bits... This will end in a really big number which may not be useful always. You can still use it for comparison, but then all the conversions done in the C++ part are not needed. I think a more normal timestamp, which you can use with python datetime would be like this:
timestamp = float(str(k) + "." + str(l)) # don't know if there is a better way
date = datetime.fromtimestamp(timestamp)
If you don't care of the fractional part you can just use the seconds directly.
Another thing to consider is, that the timestamp of RTP protocols depends on the camera/server... They may use the clock timestamp or just some other clock like start of the streaming of start of the system. So it may or not be from epoch.

Related

Incorrect CRC calculation in protocol. One is implemented using zlib and the other one is calculated in function

I am implementing a protocol in an STM32F412 board. It's almost done, I just need to do a CRC check for the received data.
I tried using the internal CRC module for calculating the CRC but I could not match the result to any online CRC algorithm online, so I decided to do a simple implementation of the Ethernet CRC.
static const uint32_t crc32_tab[] =
{
0x00000000L, 0x77073096L, 0xee0e612cL, 0x990951baL, 0x076dc419L,
0x706af48fL, 0xe963a535L, 0x9e6495a3L, 0x0edb8832L, 0x79dcb8a4L,
0xe0d5e91eL, 0x97d2d988L, 0x09b64c2bL, 0x7eb17cbdL, 0xe7b82d07L,
0x90bf1d91L, 0x1db71064L, 0x6ab020f2L, 0xf3b97148L, 0x84be41deL,
0x1adad47dL, 0x6ddde4ebL, 0xf4d4b551L, 0x83d385c7L, 0x136c9856L,
0x646ba8c0L, 0xfd62f97aL, 0x8a65c9ecL, 0x14015c4fL, 0x63066cd9L,
0xfa0f3d63L, 0x8d080df5L, 0x3b6e20c8L, 0x4c69105eL, 0xd56041e4L,
0xa2677172L, 0x3c03e4d1L, 0x4b04d447L, 0xd20d85fdL, 0xa50ab56bL,
0x35b5a8faL, 0x42b2986cL, 0xdbbbc9d6L, 0xacbcf940L, 0x32d86ce3L,
0x45df5c75L, 0xdcd60dcfL, 0xabd13d59L, 0x26d930acL, 0x51de003aL,
0xc8d75180L, 0xbfd06116L, 0x21b4f4b5L, 0x56b3c423L, 0xcfba9599L,
0xb8bda50fL, 0x2802b89eL, 0x5f058808L, 0xc60cd9b2L, 0xb10be924L,
0x2f6f7c87L, 0x58684c11L, 0xc1611dabL, 0xb6662d3dL, 0x76dc4190L,
0x01db7106L, 0x98d220bcL, 0xefd5102aL, 0x71b18589L, 0x06b6b51fL,
0x9fbfe4a5L, 0xe8b8d433L, 0x7807c9a2L, 0x0f00f934L, 0x9609a88eL,
0xe10e9818L, 0x7f6a0dbbL, 0x086d3d2dL, 0x91646c97L, 0xe6635c01L,
0x6b6b51f4L, 0x1c6c6162L, 0x856530d8L, 0xf262004eL, 0x6c0695edL,
0x1b01a57bL, 0x8208f4c1L, 0xf50fc457L, 0x65b0d9c6L, 0x12b7e950L,
0x8bbeb8eaL, 0xfcb9887cL, 0x62dd1ddfL, 0x15da2d49L, 0x8cd37cf3L,
0xfbd44c65L, 0x4db26158L, 0x3ab551ceL, 0xa3bc0074L, 0xd4bb30e2L,
0x4adfa541L, 0x3dd895d7L, 0xa4d1c46dL, 0xd3d6f4fbL, 0x4369e96aL,
0x346ed9fcL, 0xad678846L, 0xda60b8d0L, 0x44042d73L, 0x33031de5L,
0xaa0a4c5fL, 0xdd0d7cc9L, 0x5005713cL, 0x270241aaL, 0xbe0b1010L,
0xc90c2086L, 0x5768b525L, 0x206f85b3L, 0xb966d409L, 0xce61e49fL,
0x5edef90eL, 0x29d9c998L, 0xb0d09822L, 0xc7d7a8b4L, 0x59b33d17L,
0x2eb40d81L, 0xb7bd5c3bL, 0xc0ba6cadL, 0xedb88320L, 0x9abfb3b6L,
0x03b6e20cL, 0x74b1d29aL, 0xead54739L, 0x9dd277afL, 0x04db2615L,
0x73dc1683L, 0xe3630b12L, 0x94643b84L, 0x0d6d6a3eL, 0x7a6a5aa8L,
0xe40ecf0bL, 0x9309ff9dL, 0x0a00ae27L, 0x7d079eb1L, 0xf00f9344L,
0x8708a3d2L, 0x1e01f268L, 0x6906c2feL, 0xf762575dL, 0x806567cbL,
0x196c3671L, 0x6e6b06e7L, 0xfed41b76L, 0x89d32be0L, 0x10da7a5aL,
0x67dd4accL, 0xf9b9df6fL, 0x8ebeeff9L, 0x17b7be43L, 0x60b08ed5L,
0xd6d6a3e8L, 0xa1d1937eL, 0x38d8c2c4L, 0x4fdff252L, 0xd1bb67f1L,
0xa6bc5767L, 0x3fb506ddL, 0x48b2364bL, 0xd80d2bdaL, 0xaf0a1b4cL,
0x36034af6L, 0x41047a60L, 0xdf60efc3L, 0xa867df55L, 0x316e8eefL,
0x4669be79L, 0xcb61b38cL, 0xbc66831aL, 0x256fd2a0L, 0x5268e236L,
0xcc0c7795L, 0xbb0b4703L, 0x220216b9L, 0x5505262fL, 0xc5ba3bbeL,
0xb2bd0b28L, 0x2bb45a92L, 0x5cb36a04L, 0xc2d7ffa7L, 0xb5d0cf31L,
0x2cd99e8bL, 0x5bdeae1dL, 0x9b64c2b0L, 0xec63f226L, 0x756aa39cL,
0x026d930aL, 0x9c0906a9L, 0xeb0e363fL, 0x72076785L, 0x05005713L,
0x95bf4a82L, 0xe2b87a14L, 0x7bb12baeL, 0x0cb61b38L, 0x92d28e9bL,
0xe5d5be0dL, 0x7cdcefb7L, 0x0bdbdf21L, 0x86d3d2d4L, 0xf1d4e242L,
0x68ddb3f8L, 0x1fda836eL, 0x81be16cdL, 0xf6b9265bL, 0x6fb077e1L,
0x18b74777L, 0x88085ae6L, 0xff0f6a70L, 0x66063bcaL, 0x11010b5cL,
0x8f659effL, 0xf862ae69L, 0x616bffd3L, 0x166ccf45L, 0xa00ae278L,
0xd70dd2eeL, 0x4e048354L, 0x3903b3c2L, 0xa7672661L, 0xd06016f7L,
0x4969474dL, 0x3e6e77dbL, 0xaed16a4aL, 0xd9d65adcL, 0x40df0b66L,
0x37d83bf0L, 0xa9bcae53L, 0xdebb9ec5L, 0x47b2cf7fL, 0x30b5ffe9L,
0xbdbdf21cL, 0xcabac28aL, 0x53b39330L, 0x24b4a3a6L, 0xbad03605L,
0xcdd70693L, 0x54de5729L, 0x23d967bfL, 0xb3667a2eL, 0xc4614ab8L,
0x5d681b02L, 0x2a6f2b94L, 0xb40bbe37L, 0xc30c8ea1L, 0x5a05df1bL,
0x2d02ef8dL
};
uint32_t calc_crc_calculate(uint8_t *pData, uint32_t uLen)
{
uint32_t val = 0xFFFFFFFFU;
int i;
for(i = 0; i < uLen; i++) {
val = crc32_tab[(val ^ pData[i]) & 0xFF] ^ ((val >> 8) & 0x00FFFFFF);
}
return val^0xFFFFFFFF;
}
I calculated the crc of 0x6F and compared the result to the online calculators and it apparently matches.
When I try to test the protocol with my python code I'm just unable to match the CRCs. On python I'm using the following code:
d = 0x6f
crc = zlib.crc32(bytes(d))&0xFFFFFFFF
I'm now unable to tell which is right. Apparently my algorithm is OK because it matches the online calculator. BUT those online calculators do not seem to be reliable sometimes and I doubt that python's zlib implementation is wrong .. I may be using it wrong at worst.
Actually you can compute the Ethernet CRC32 with the builtin module of the STM32. It took me quite a while to make it match up as well.
This code should match up for sizes divisible by 4 (I also used python zlib on the other end):
#include "stm32l4xx_hal.h"
uint32_t CRC32_Compute(const uint32_t *data, size_t sizeIn32BitWords)
{
CRC_HandleTypeDef hcrc = {
.Instance = CRC,
.Init.DefaultPolynomialUse = DEFAULT_POLYNOMIAL_ENABLE,
.Init.DefaultInitValueUse = DEFAULT_INIT_VALUE_ENABLE,
.Init.InputDataInversionMode = CRC_INPUTDATA_INVERSION_WORD,
.Init.OutputDataInversionMode = CRC_OUTPUTDATA_INVERSION_ENABLE,
.InputDataFormat = CRC_INPUTDATA_FORMAT_WORDS,
};
HAL_StatusTypeDef status = HAL_CRC_Init(&hcrc);
assert (status == HAL_OK)
uint32_t checksum = HAL_CRC_Calculate(&hcrc, data, sizeIn32BitWords);
uint32_t checksumInverted = ~checksum;
return checksumInverted;
}
The challenge with sizes not divisible by 4 is to get the "inversion/reversal" (changing the bit order) right. There is an example how the hardware handles this in the "RM0394 Reference manual STM32L43xxx STM32L44xxx STM32L45xxx STM32L46xxx advanced ARM®-based 32-bit MCUs Rev 3" on page 333.
The essence is that reversal reverses the bit order. For CRC32 this reversal must happen on the word level, i.e. over 32 bits.
Ok. It certainly was a bug on my part. But it was happening in my python code.
I suddenly realized that I was practically doing bytes(0x6F) which just creates an array with 111 positions.
What I actually needed to do was
import struct
d = pack('B', 0x6F)
crc = zlib.crc32(bytes(d))&0xFFFFFFFF
This question could have been avoided had I just done a little bit of rubber duck debugging. Hopefuly this will help someone else.

data gets corrupted between c and python

I am trying to use Cython and ctypes to call a c library function using Python.
But the data bytes get corrupted somehow. Could someone please help to locate the issue?
testCRC.c:
#include <stdio.h>
unsigned char GetCalculatedCrc(const unsigned char* stream){
printf("Stream is %x %x %x %x %x %x %x\n",stream[0],stream[1],stream[2],stream[3],stream[4],stream[5],stream[6]);
unsigned char dummy=0;
return dummy;
}
wrapped.pyx:
# Exposes a c function to python
def c_GetCalculatedCrc(const unsigned char* stream):
return GetCalculatedCrc(stream)
test.py:
x_ba=(ctypes.c_ubyte *7)(*[0xD3,0xFF,0xF7,0x7F,0x00,0x00,0x41])
x_ca=(ctypes.c_char * len(x_ba)).from_buffer(x_ba)
y=c_GetCalculatedCrc(x_ca.value)
output:
Stream is d3 ff f7 7f 0 0 5f # expected
0xD3,0xFF,0xF7,0x7F,0x00,0x00,0x41
Solution:
1.
I had to update the cython to 0.29 to have the fix for the bug which was not allowing to use the typed memory.(read only problem).
2.
It worked passing x_ca.raw. But when x_ca.value was passed it threw error 'out of bound access.'
After the suggestions from #ead & #DavidW:
´.pyx´:
def c_GetCalculatedCrc(const unsigned char[:] stream):
# Exposes a c function to python
print "received %s\n" %stream[6]
return GetCalculatedCrc(&stream[0])
´test.py´:
x_ba=(ctypes.c_ubyte *8)(*[0x47,0xD3,0xFF,0xF7,0x7F,0x00,0x00,0x41])
x_ca=(ctypes.c_char * len(x_ba)).from_buffer(x_ba)
y=c_GetCalculatedCrc(x_ca.raw)
output:
Stream is 47 d3 ff f7 7f 0 0 41
As pointed out by #DavidW the problem is your usage of x_ca.value: When x_ca.value is called, every time a new bytes-object is created (see documentation) and memory is copied:
x_ca.value is x_ca.value
#False -> every time a new object is created
However, when the memory is copied, it handles \0-character as end of string (which is typical for C-strings), as can be seen in source code:
static PyObject *
CharArray_get_value(CDataObject *self, void *Py_UNUSED(ignored))
{
Py_ssize_t i;
char *ptr = self->b_ptr;
for (i = 0; i < self->b_size; ++i)
if (*ptr++ == '\0')
break;
return PyBytes_FromStringAndSize(self->b_ptr, i);
}
Thus the result of x_ca.value is a bytes object of length 4, which doesn't share memory with x_ca - when you access stream[6] it leads to undefined behavior - anything could happen (also a crash).
So what can be done?
Normally, you just cannot have a pointer-argument in a def-function, but char * is an exception - a bytes object can be automatically converted to char *, which however doesn't happen via buffer protocol but via PyBytes_AsStringAndSize.
This is the reason, why you cannot pass x_ca to c_GetCalculatedCrc as it is: x_ca implements the buffer protocol, but is not a bytes-object and thus there is no PyBytes_AsStringAndSize.
An alternative is to use typed memory view, which utilizes the buffer protocol, i.e.
%%cython
def c_GetCalculatedCrc(const unsigned char[:] stream):
print(stream[6]);
and now passing x_ca directly, with original length/content:
c_GetCalculatedCrc(x_ca)
# 65 as expected
Another alternative would be to pass x_ca.raw to function expecting const unsigned char * as argument, as has been pointed out by #DavidW in comments, which shares memory with x_ca. However I would prefer the typed memory views - they are safer than raw pointers and you would not run into surprisingly undefined behavior.

c mmap write long to mmaped-area

i want to write a python-c-extension which should write a list of integers into a ram-area because my first version with python was a little bit to slow (30 ms).
In python this perfectly works with following code:
with open("/dev/mem", "r+b") as f: # open file
ddr_mem = mmap.mmap(f.fileno(), PRU_ICSS_LEN, offset=PRU_ICSS) # map pru shared-ram
while offset < ramSize:
ddr_mem[(sharedRam+offset):(sharedRam+offset+4)] = struct.pack('i', self.data[self.i])
offset += 4
self.i += 1
Because in the list are only long integer values (4 Bytes) the offset will be increased by 4 for every new value in the list.
As the mmap-area is 12 kB big it is possible to write 3072 values into it, because 12288 Byte / 4 Byte = 3072 or i am wrong?
Now within my c extension i tried the same with this piece of code:
if(ddrMem == NULL) {
//printf("\nopen: shared-ram...");
mem_fd = open("/dev/mem", O_RDWR | O_SYNC);
if (mem_fd < 0) {
printf("Failed to open /dev/mem (%s)\n", strerror(errno));
return NULL;
}
/* map the DDR memory */
ddrMem = mmap(0, 0x0FFFFFFF, PROT_WRITE | PROT_READ, MAP_SHARED, mem_fd, PRU_ICSS + OFFSET_DDR + 0xE000); //TODO: weird offset - 0xE000
if (ddrMem == NULL) {
printf("Failed to map the device (%s)\n", strerror(errno));
close(mem_fd);
return NULL;
}
for (i = 0; i < d_len; i++) {
PyObject* temp = PySequence_Fast_GET_ITEM(seq, i);
elem = PyInt_AsLong(temp);
*((long*) DDR_regaddr+offset) = elem; // write to shared ram
offset = offset + 4;
if(offset >= (ramSize)){
offset = 0;
}
But now only every fourth address in the ram-area will get a new value. If i increase the offset with one it works but then i am able to write twice the data --> 6144 elements.
What is the trick ? What i am doing wrong ? Are theses calulations right ? I am not sure if i am right with my thougts.
Your offset is incorrect because you are performing pointer arithmetic, which already accounts for the size of the long in question.
Try incrementing your pointer by 1 instead of 4.
long *ptr = 0xff;
long *ptrOffset = ptr + 1; // Will access the next 'long' space in memory
long *ptrOffset2 = ptr + 4; // Will access the fourth next 'long' space in memory
Also, the size of the long is actually architecture and compiler dependent. Assuming that it is 4 bytes is not safe.
Ok sorry.. :(
Wrong memory mapping !
-> missed an offset of 0xF000
But many thanks #kevr - this was also very helpful :)

Arduino to Raspberry crc32 check

I'm trying to send messages through the serial USB interface of my Arduino (C++) to a Raspberry Pi (Python).
On the Arduino side I define a struct which I then copy into a char[]. The last part of the struct contains a checksum that I want to calculate using CRC32. I copy the struct into a temporary char array -4 bytes to strip the checksum field. The checksum is then calculated using the temporary array and the result is added to the struct. The struct is then copied into byteMsg which gets send over the serial connection.
On the raspberry end I do the reverse, I receive the bytestring and calculate the checksum over the message - 4 bytes. Then unpack the bytestring and compare the received and calculated checksum but this fails unfortunately.
For debugging I compared the crc32 check on both the python and arduino for the string "Hello World" and they generated the same checksum so doesn't seem to be a problem with the library. The raspberry is also able to decode the rest of the message just fine so the unpacking of the data into variables seem to be ok as well.
Any help would be much appreciated.
The Python Code:
def unpackMessage(self, message):
""" Processes a received byte string from the arduino """
# Unpack the received message into struct
(messageID, acknowledgeID, module, commandType,
data, recvChecksum) = struct.unpack('<LLBBLL', message)
# Calculate the checksum of the recv message minus the last 4
# bytes that contain the sent checksum
calcChecksum = crc32(message[:-4])
if recvChecksum == calcChecksum:
print "Checksum checks out"
The Aruino crc32 library taken from http://excamera.com/sphinx/article-crc.html
crc32.h
#include <avr/pgmspace.h>
static PROGMEM prog_uint32_t crc_table[16] = {
0x00000000, 0x1db71064, 0x3b6e20c8, 0x26d930ac,
0x76dc4190, 0x6b6b51f4, 0x4db26158, 0x5005713c,
0xedb88320, 0xf00f9344, 0xd6d6a3e8, 0xcb61b38c,
0x9b64c2b0, 0x86d3d2d4, 0xa00ae278, 0xbdbdf21c
};
unsigned long crc_update(unsigned long crc, byte data)
{
byte tbl_idx;
tbl_idx = crc ^ (data >> (0 * 4));
crc = pgm_read_dword_near(crc_table + (tbl_idx & 0x0f)) ^ (crc >> 4);
tbl_idx = crc ^ (data >> (1 * 4));
crc = pgm_read_dword_near(crc_table + (tbl_idx & 0x0f)) ^ (crc >> 4);
return crc;
}
unsigned long crc_string(char *s)
{
unsigned long crc = ~0L;
while (*s)
crc = crc_update(crc, *s++);
crc = ~crc;
return crc;
}
Main Arduino Sketch
struct message_t {
unsigned long messageID;
unsigned long acknowledgeID;
byte module;
byte commandType;
unsigned long data;
unsigned long checksum;
};
void sendMessage(message_t &msg)
{
// Set the messageID
msg.messageID = 10;
msg.checksum = 0;
// Copy the message minus the checksum into a char*
// Then perform the checksum on the message and copy
// the full msg into byteMsg
char byteMsgForCrc32[sizeof(msg)-4];
memcpy(byteMsgForCrc32, &msg, sizeof(msg)-4);
msg.checksum = crc_string(byteMsgForCrc32);
char byteMsg[sizeof(msg)];
memcpy(byteMsg, &msg, sizeof(msg));
Serial.write(byteMsg, sizeof(byteMsg));
void loop() {
message_t msg;
msg.module = 0x31;
msg.commandType = 0x64;
msg.acknowledgeID = 0;
msg.data = 10;
sendMessage(msg);
Kind Regards,
Thiezn
You are making the classic struct-to-network/serial/insert communication layer mistake. Structs have hidden padding in order to align the members onto suitable memory boundaries. This is not guaranteed to be the same across different computers, let alone different CPUs/microcontrollers.
Take this struct as an example:
struct Byte_Int
{
int x;
char y;
int z;
}
Now on a basic 32-bit x86 CPU you have a 4-byte memory boundary. Meaning that variables are aligned to either 4 bytes, 2 bytes or not at all according to the type of variable. The example would look like this in memory: int x on bytes 0,1,2,3, char y on byte 4, int z on bytes 8,9,10,11. Why not use the three bytes on the second line? Because then the memory controller would have to do two fetches to get a single number! A controller can only read one line at a time. So, structs (and all other kinds of data) have hidden padding in order to get variables aligned in memory. The example struct would have a sizeof() of 12, and not 9!
Now, how does that relate to your problem? You are memcpy()ing a struct directly into a buffer, including the padding. The computer on the other end doesn't know about this padding and misinterprets the data. What you need a serialization function that takes the members of your structs and pasts them into a buffer one at a time, that way you lose the padding and end up with something like this:
[0,1,2,3: int x][4: char y][5,6,7,8: int z]. All as one lengthy bytearray/string which can be safely sent using Serial(). Of course on the other end you would have to parse this string into intelligible data. Python's unpack() does this for you as long as you give the right format string.
Lastly, an int on an Arduino is 16 bits long. On a pc generally 4! So assemble your unpack format string with care.
The char array I was passing to the crc_string function contained '\0' characters. The crc_string was iterating through the array until it found a '\0' which shouldn't happen in this case since I was using the char array as a stream of bytes to be sent over a serial connection.
I've changed the crc_string function to take the array size as argument and iterate through the array using that value. This solved the issue.
Here's the new function
unsigned long crc_string(char *s, size_t arraySize)
{
unsigned long crc = ~0L;
for (int i=0; i < arraySize; i++) {
crc = crc_update(crc, s[i]);
}
crc = ~crc;
return crc;
}

RaspberryPi to Arduino -send and receive string

This is the code I am currently using to send and receive int values from a RaspberryPi to an Arduino using i2C. It works fine for values 0-255, but because of the 1 byte limit, anything larger fails.
To circumvent this, I'd like to send and receive string values instead, and then convert back to int if necessary.
What changes would I need to make in the following?
Here is my RPi Python code
import smbus
import time
# for RPI version 1, use "bus = smbus.SMBus(0)"
bus = smbus.SMBus(1)
# This is the address we setup in the Arduino Program
address = 0x04
def writeNumber(value):
bus.write_byte(address, value)
# bus.write_byte_data(address, 0, value)
return -1
def readNumber():
number = bus.read_byte(address)
# number = bus.read_byte_data(address, 1)
return number
while True:
try:
var = int(raw_input("Enter 1 - 9: "))
except ValueError:
print "Could you at least give me an actual number?"
continue
writeNumber(var)
print "RPI: Hi Arduino, I sent you ", var
# sleep one second
#time.sleep(1)
number = readNumber()
print "Arduino: Hey RPI, I received a digit ", number
print
And here is my Arduino code
#include <Wire.h>
#define SLAVE_ADDRESS 0x04
int number = 0;
int state = 0;
void setup() {
pinMode(13, OUTPUT);
Serial.begin(9600); // start serial for output
// initialize i2c as slave
Wire.begin(SLAVE_ADDRESS);
// define callbacks for i2c communication
Wire.onReceive(receiveData);
Wire.onRequest(sendData);
Serial.println("Ready!");
}
void loop() {
delay(100);
}
// callback for received data
void receiveData(int byteCount){
while(Wire.available()) {
number = Wire.read();
if (Wire.available() > 1) // at least 2 bytes
{
number = Wire.read() * 256 + Wire.read();
}
Serial.print("data received: ");
Serial.println(number);
//sendData();
if (number == 1){
if (state == 0){
digitalWrite(13, HIGH); // set the LED on
state = 1;
}
else{
digitalWrite(13, LOW); // set the LED off
state = 0;
}
}
}
}
// callback for sending data
void sendData(){
Wire.write(number);
}
This problem essentially has two parts: splitting an integer into its bytes and reassembling an integer from bytes. These parts must be replicated on both the Pi and Arduino. I'll address the Pi side first, in Python:
Splitting an integer:
def writeNumber(value):
# assuming we have an arbitrary size integer passed in value
for character in str(val): # convert into a string and iterate over it
bus.write_byte(address, ord(character)) # send each char's ASCII encoding
return -1
Reassembling an integer from bytes:
def readNumber():
# I'm not familiar with the SMbus library, so you'll have to figure out how to
# tell if any more bytes are available and when a transmission of integer bytes
# is complete. For now, I'll use the boolean variable "bytes_available" to mean
# "we are still transmitting a single value, one byte at a time"
byte_list = []
while bytes_available:
# build list of bytes in an integer - assuming bytes are sent in the same
# order they would be written. Ex: integer '123' is sent as '1', '2', '3'
byte_list.append(bus.read_byte(address))
# now recombine the list of bytes into a single string, then convert back into int
number = int("".join([chr(byte) for byte in byte_list]))
return number
Arduino Side, in C
Split an Integer:
void sendData(){
int i = 0;
String outString = String(number); /* convert integer to string */
int len = outString.length()+1 /* obtain length of string w/ terminator */
char ascii_num[len]; /* create character array */
outString.toCharArray(ascii_num, len); /* copy string to character array */
for (i=0; i<len); ++i){
Wire.write(ascii_num[i]);
}
}
Reassembling a received Integer:
Note: I'm having some trouble understanding what your other code in this routine is doing, so I'm going to reduce it to just assembling the integer.
void receiveData(int byteCount){
int inChar;
String inString = "";
/* As with the Python receive routine, it will be up to you to identify the
terminating condition for this loop - "bytes_available" means the same thing
as before */
while(bytes_available){
inChar = Wire.read();
inString += char(inChar);
}
number = inString.toInt();
}
I don't have the materials on hand to test this, so it's possible I've gotten the byte order flipped in one routine or another. If you find stuff coming in or out backwards, the easiest place to fix it is in the Python script by using the built-in function reversed() on the strings or lists.
References (I used some code from the Arduino Examples):
Arduino String objects
Arduino String Constructors
Python Built-ins chr() and ord()
Check the following Link:
[http://www.i2c-bus.org/][1]
When I was sending data back and forward using I2C I was converting the string characters to bytearrays and viceversa. So since you are always sending bytes. It will always work since you are sending numbers between 0-255.
Not sure this helps but at least may give you an idea.
You could convert the number to a string of digits like you said. But you could also send the raw bytes.
String of digits
Advantages
Number can have infinite digits. Note that when Arduino reads the number as a string, it is infinite, but you can't convert it all to integer if it overflows the 16-bit range (or 32-bit for Due).
Disadvantages
Variable size, thus requiring more effort in reading.
Waste of bytes, because each decimal digit would be a byte, plus the null-terminator totalizing (digits + 1) size.
Having to use decimal arithmetic (which really is only useful for human counting), note that a "number to string" operation also uses decimal arithmetic.
You can't send/receive negative numbers (unless you send the minus signal, wasting more time and bytes).
Raw bytes
Advantages
Number of bytes sent for each integer is always 4.
You can send/receive negative numbers.
The bitwise arithmetic in C++ for extracting each byte from the number is really fast.
Python already has the struct library which packs/unpacks each byte in a number to a string to send/receive, so you don't need to do the arithmetic like in C++.
Disadvantages
Number has a limited range (signed 32-bit integer in our case, which ranges from -2147483648 to 2147483647). But it doesn't matter because no Arduino can handle more than 32-bit anyways.
So I would use the raw bytes method, which I can provide some untested functions here:
import struct
# '<i' stands for litle-endian signed integer
def writeNumber(value):
strout = struct.pack('<i', value)
for i in range(4):
bus.write_byte(address, strout[i])
return -1
def readNumber():
strin = ""
for _ in range(4):
strin += bus.read_byte(address)
return struct.unpack('<i', strin)[0]
And the Arduino part:
void receiveData(int byteCount)
{
// Check if we have a 32-bit number (4 bytes) in queue
while(Wire.available() >= 4)
{
number = 0;
for(int i = 0; i < 32; i += 8)
{
// This is merging the bytes into a single integer
number |= ((int)Wire.read() << i);
}
Serial.print("data received: ");
Serial.println(number);
// ...
}
}
void sendData()
{
for(int i = 0; i < 32; i += 8)
{
// This is extracting each byte from the number
Wire.write((number >> i) & 0xFF);
}
}
I don't have any experience with I2C, but if its queue is a FIFO, then the code should work.

Categories