How could I work with binary data in Python? - python

I'm trying to work with robotics binary data but I am very stuck. I don't understand even how to work with it. I want to create a dataframe from it and use pandas to get some statistics.
When I open the file I get this:
struct Time
{
long long time;
unsigned short millitm;
short timezone;
short dstflag;
};
struct wxp1
{
float x;
float y;
};
struct wxp2
struct position
{
wxp1 position; // based on Basic Full(679x382)
wxp1 position2; // based on Basic Full(679x382)
wxp1 position3; // based on Basic Full(679x382)
wxp1 estimatedposition;
wxp1 estimatedposition2;
wxp1 estimatedposition3;
float score;
};
Followed by binary
00\x00\x00\x00\x00\x00\x00\x00\x etc
Not familiar at all with it I tried to open it with struct or methods I found on stackoverflow without success.
from numpy import fromfile, dtype
from pandas import DataFrame
records = fromfile('/content/my_file.blog')
df=DataFrame(records)
But I don't get something relevant with it...

Related

Get address of member of struct using pyelftools

I am using the pyelftools to read an elf file. How can I get an offset value or address of a member in a struct? For example, say I have the following struct in C.
typedef struct
{
int valA;
} TsA;
typedef struct
{
int valB;
} TsB;
typedef struct
{
int valC;
TsB b;
} TsC;
typedef struct
{
TsA a;
TsC c;
} TsStruct;
TsStrcut myStruct;
How can I get an address of myStruct.c.b.valB? I found a similar question here but did not find any good answer.
Find the DIE for the structure, the one with tag DW_TAG_structure_type and DW_AT_name equal to structure names.
Enumerate the DW_TAG_member subdies under it. While there, look at the DW_AT_member_location, it's the offset of the corresponding structure element.
It might help if you take a look at the DIE structure visually first. DWARF Explorer might help (disclaimer: I wrote it).

GDB Python: Parsing field name for structure array

Can anyone please explain how can i parse array of structure for getting fields of structure.
Using script mentioned in this
gdb python : Can anyone explain me how to use this script written in this post?
Trying to parse below structure:
typedef struct Q
{
int a;
int b;
}Q;
typedef struct T
{
int x,
int y;
Q q;
}T;
struct S
{
T t[10];
char b;
};

C send struct over UART to Python

I'm trying to send a struct over UART (from an ESP32) to be processed by Python by using this guide.
// we send this to the host, to be processed by python script
struct package {
uint8_t modifier;
uint8_t keyboard_keys[6];
};
// instantiate struct
package to_send = {};
// send the contents of keyboard_keys and keyboard_modifier_keys
// https://folk.uio.no/jeanra/Microelectronics/TransmitStructArduinoPython.html
void usb_keyboard_send(void)
{
to_send.modifier = keyboard_modifier_keys;
for(uint8_t i = 0; i < 6; i++) {
to_send.keyboard_keys[i] = keyboard_keys[i];
}
printf("S");
printf((uint8_t *)&to_send, sizeof(to_send));
printf("E");
}
However I get the error: invalid conversion from 'uint8_t* {aka unsigned char*}' to 'const char*' [-fpermissive]
I'm pretty new to C++, and I've tried all sorts of casting, but I just can't get it to work. Could someone offer guidance please?
Setting aside that it's generally a bad idea to mix ASCII and raw binary, your code is almost right.
You have 2 major errors:
// instantiate struct
package to_send = {};
should be:
// instantiate struct
struct package to_send = {};
Also, to write directly (not formatted text) to STDOUT you want to use fwrite()
i.e.
printf("S");
fwrite((uint8_t *)&to_send, sizeof(uint8_t), sizeof(struct_package), STDOUT);
printf("E");
As an aside, after fixing these 2 errors you may be surprised to find that your struct isn't the number of bytes you expect. The compiler may optimize it to make memory accesses faster by padding fields to word sized boundaries (32 bits on ESP32). sizeof() will return the correct value taking in to account whatever optimizations are done, but your Python code may not expect that. To fix this you probably wan to use a compiler hint, e.g. __attribute__((__packed__)). See here for a general guide to structure packing.

Python generically parse data into object structure

What I want to know is if I have a defined structured object with known parameters and a known order. I want to parse a binary blob into this structure in a generic way.
For example, I know that my file is a binary file of this structure
typedef struct {
uint frCompressedSize;
uint frUncompressedSize;
ushort frFileNameLength;
ushort frExtraFieldLength;
char frFileName[ frFileNameLength ];
uchar frExtraField[ frExtraFieldLength ];
uchar frData[ frCompressedSize ];
} ZIPFILERECORD;
Is there a better way to do this than reading in individual fields at a time in a hard coded manner? In my real code the structure has almost 100 parameters so the hardcoded method is not my first choice.
Any Ideas?
thanks!
You are looking for the python struct library

Python serial communication

I'm working on an Arduino project, and I am interfacing it with a Python script due to memory limitations. On the Python side I have a 2 dimensional matrix containing respective x, y values for coordinates, and in this list is 26000 coordinate pairs. So, in interest of clarifying the data structure for all of you, pathlist[0][0], would return the X value of the first coordinate of my list. Performing different operations, etc. on this list in Python is posing no problems. Where I am running into trouble however is sending these values to Arduino over serial, in a way that is useful.
Due to the nature of serial communication (at least I think this is the case) I must send each each integer as a string, and only one digit at a time. So, a number like 345 would be sent over as 3 individual characters, those being of course, 3, 4, then 5.
What I am struggling with is finding a way to rebuild those integers on the Arduino.
Whenever I send a value over, it's receiving the data and outputting it like so:
//Python is sending over the number '25'
2ÿÿ52
//Python is sending the number 431.
4ÿÿ321ÿÿÿ2
The Arduino code is:
String str;
int ds = 4;
void setup() {
Serial.begin(9600);
}
void loop(){
if (Serial.available()>0) {
for (int i=0; i<4; i=i+1) {
char d= Serial.read();
str.concat(d);
}
char t[str.length()+1];
str.toCharArray(t, (sizeof(t)));
int intdata = atoi(t);
Serial.print(intdata);
}
}
And the Python code looks like this:
import serial
s = serial.Serial(port='/dev/tty.usbmodemfd131', baudrate=9600)
s.write(str(25))
I'm almost certain that the problem isn't stemming from the output method (Serial.print), seeing as when I declare another int, it formats fine on output, so I am assuming the problem lies in how the intdata variable is constructed.
One thing of note that may help diagnose this problem is that if I change Serial.print(intdata) to Serial.print(intdata+5) my result is 2ÿÿ57, where I would expect 30 (25+5). This 7 is present regardless of the input. For instance I could write 271 to the serial and my result would look as follows:
//For input 271.
2ÿÿ771ÿÿÿ7
It appears to me that Arduino is chunking the values into pairs of two and appending the length to the end. I can't understand why that would happen though.
It also seems to me that the ÿ are being added in the for loop. Meaning that they are added because nothing is being sent at that current moment. But even fixing that by adding yet another if(Serial.available()>0) conditional, the result is still not treated like an integer.
Also, would using Pickle be appropriate here?
What am I doing wrong?
You should wait a bit for the serial data to arrive.
The Arduino code should be:
if (Serial.available()){
delay(100); // Wait for all data.
while (Serial.available()) {
char d = Serial.read();
str.concat(d);
}
}
Also you have to clear your string before re-using it.
[Edit]
I forgot to mention ÿ == -1 == 255 which means Serial.read() it is saying it can't read anything.
I would change the communication so python sends newlines between numbers, so you're not as dependent on the timing:
s.write(str(25)+'\n')
and then on the receiving side:
void loop(){
while (Serial.available() > 0) {
char d = Serial.read();
if (d == '\n') {
char t[str.length()+1];
str.toCharArray(t, (sizeof(t)));
int intdata = atoi(t);
Serial.print(intdata);
str = String();
}
else {
str.concat(d);
}
}
}

Categories