I have a Numpy vector of bools and I'm trying to use the C API to get a bytes object as quickly as possible from it. (Ideally, I want to map the binary value of the vector to the bytes object.)
I can read in the vector successfully and I have the data in bool_vec_arr. I thought of creating an int and setting its bits in this way:
PyBytesObject * pbo;
int byte = 0;
int i = 0;
while ( i < vec->dimensions[0] )
{
if ( bool_vec_arr[i] )
{
byte |= 1UL << i % 8;
}
i++;
if (i % 8 == 0)
{
/* do something here? */
byte = 0;
}
}
return PyBuildValue("S", pbo);
But I'm not sure how to use the value of byte in pbo. Does anyone have any suggestions?
You need to store the byte you've just completed off. Your problem is you haven't made an actual bytes object to populate, so do that. You know how long the result must be (one-eighth the size of the bool vector, rounded up), so use PyBytes_FromStringAndSize to get a bytes object of the correct size, then populate it as you go.
You'd just allocate with:
// Preallocate enough bytes
PyBytesObject *pbo = PyBytes_FromStringAndSize(NULL, (vec->dimensions[0] + 7) / 8);
// Put check for NULL here
// Extract pointer to underlying buffer
char *bytebuffer = PyBytes_AsString(pbo);
where adding 7 then dividing by 8 rounds up to ensure you have enough bytes for all the bits, then assign to the appropriate index when you've finished a byte, e.g.:
if (i % 8 == 0)
{
bytebuffer[i / 8 - 1] = byte; // Store completed byte to next index
byte = 0;
}
If the final byte might be incomplete, you'll need to decide how to handle this (do the pad bits appear on the left or right, is the final byte omitted and therefore you shouldn't round up the allocation, etc.).
Related
I'm trying to translate this c code into python, but Im having problems with the char* to ushort* conversion:
void sendAsciiCommand(string command) {
unsigned int nchars = command.length() + 1; // Char count of command string
unsigned int nshorts = ceil(nchars / 2); // Number of shorts to store the string
std::vector<unsigned short> regs(nshorts); // Vector of short registers
// Transform char array to short array with endianness conversion
unsigned short *ascii_short_ptr = (unsigned short *)(command.c_str());
for (unsigned int i = 0; i < nshorts; i++)
regs[i] = htons(ascii_short_ptr[i]);
return std::string((char *)regs.data());
}
As long I have tried with this code in Python 2.7:
from math import ceil
from array import array
command = "hello"
nchars = len(command) + 1
nshorts = ceil(nchars/2)
regs = array("H", command)
But it gives me the error:
ValueError: string length not a multiple of item size
Any help?
The exception text:
ValueError: string length not a multiple of item size
means what is says, i.e., the length of the string from which you are trying to create an array must be a multiple of the item size. In this case the item size is that of an unsigned short, which is 2 bytes. Therefore the length of the string must be a multiple of 2. hello has length 5 which is not a multiple of 2, so you can't create an array of 2 byte integers from it. It will work if the string is 6 bytes long, e.g. hello!.
>>> array("H", 'hello!')
array('H', [25960, 27756, 8559])
You might still need to convert to network byte order. array uses the native byte order on your machine, so if your native byte order is little endian you will need to convert it to big endian (network byte order). Use sys.byteorder to check and array.byteswap() to swap the byte order if required:
import sys
from array import array
s = 'hello!'
regs = array('H', s)
print(regs)
# array('H', [25960, 27756, 8559])
if sys.byteorder != 'big':
regs.byteswap()
print(regs)
# array('H', [26725, 27756, 28449])
However, it's easier to use struct.unpack() to convert straight to network byte order if necessary:
import struct
s = 'hello!'
n = len(s)/struct.calcsize('H')
regs = struct.unpack('!{}H'.format(n), s)
print(regs)
#(26725, 27756, 28449)
If you really need an array:
regs = array('H', struct.unpack('!{}H'.format(n), s))
It's also worth pointing out that your C++ code contains an error. If the string length is odd an extra byte will be read at the end of the string and this will be included in the converted data. That extra byte will be \0 as the C string should be null terminated, but the last unsigned short should either be ignored, or you should check that the length of the string is multiple of an unsigned short, just as Python does.
I'm trying to parse a proprietary binary-format (Wintec NAL) with python. There's existing and working C-code that does the same (Author: Dennis Heynlein), which i'm trying to port to Python.
I'm struggling to understand parts of the C-code. Here's the definition of the binary format in C:
/*
* File extension:. NAL
* File format: binary, 32 byte fixed block length
*/
/*
* For now we will read raw structs direct from the data file, ignoring byte
* order issues (since the data is in little-endian form compatible with i386)
*
* XXX TODO: write marshalling functions to read records in the proper
* byte-order agnostic way.
*/
#pragma pack (1)
typedef struct nal_data32 {
unsigned char point_type; /* 0 - normal, 1 - start, 2 - marked */
unsigned char padding_1;
unsigned int second: 6, minute: 6, hour: 5;
unsigned int day: 5, month: 4, year: 6; /* add 2000 to year */
signed int latitude; /* divide by 1E7 for degrees */
signed int longitude; /* divide by 1E7 for degrees */
unsigned short height; /* meters */
signed char temperature; /* °C */
unsigned short pressure; /* mbar */
unsigned char cadence; /* RPM */
unsigned char pulse; /* BPM */
signed char slope; /* degrees */
signed short compass; /* °Z axis */
signed short roll; /* °X axis */
signed short yaw; /* °Y axis */
unsigned char speed; /* km/h */
unsigned char bike; /* ID# 0-3 */
unsigned char padding_2;
unsigned char padding_3;
} nal_t;
I'm using python-bitstring to replicate this functionality in Python, but i have difficulties in understanding the time-format given above and adopting it to Python.
from bitstring import ConstBitStream
nal_format=('''
uint:8,
uint:8,
bin:32,
intle:32,
intle:32,
uint:16,
uint:8,
uint:16,
uint:8,
uint:8,
uint:8,
uint:16,
uint:16,
uint:16,
uint:8,
uint:8,
uint:8,
uint:8
''')
f = ConstBitStream('0x01009f5a06379ae1cb13f7a6b62bca010dc703000000c300fefff9ff00000000')
f.pos=0
#type,padding1,second,minute,hour,day,month,year,lat,lon,height,temp,press,cad,pulse,slope,compass,roll,yaw,speed,bike,padding2,padding3=f.peeklist(nal_format)
type,padding1,time,lat,lon,height,temp,press,cad,pulse,slope,compass,roll,yaw,speed,bike,padding2,padding3=f.readlist(nal_format)
print type
print padding1
#print second
#print minute
#print hour
#print day
#print month
#print year
print time
print lat
print lon
While i've figured out that latitude and longitude has to be defined as little-endian, i have no idea how to adapt the 32bit wide timestamp so it fits the format given in the C-definition (And i also couldn't figure out a matching mask for "height" - correspondingly i didn't try the fields after it).
These are the values for the hex-string above:
date: 2013/12/03-T05:42:31
position: 73.3390583° E, 33.2128666° N
compass: 195°, roll -2°, yaw -7°
alt: 458 meters
temp: 13 °C
pres: 967 mb
I'm not familiar with bitstring, so I'll convert your input into packed binary data and then use struct to handle it. Skip to the break if you're uninterested in that part.
import binascii
packed = binascii.unhexlify('01009f5a06379ae1cb13f7a6b62bca010dc703000000c300fefff9ff00000000')
I can go over this part in more detail if you want. It's just turning '0100...' into b'\x01\x00...'.
Now, the only "gotcha" in unpacking this is figuring out that you only want to unpack ONE unsigned int, since that bit field fits into 32 bits (the width of a single unsigned int):
format = '<ccIiiHbHBBbhhhBBBB'
import struct
struct.unpack(format,packed)
Out[49]:
('\x01',
'\x00',
923163295,
...
)
That converts the output into an output we can use. You can unpack that into your long list of variables, like you were doing before.
Now, your question seemed to be centered around how to mask time (above: 923163295) to get the proper values out of the bit field. That's just a little bit of math:
second_mask = 2**6 - 1
minute_mask = second_mask << 6
hour_mask = (2**5 - 1) << (6+6)
day_mask = hour_mask << 5
month_mask = (2**4 - 1) << (6+6+5+5)
year_mask = (2**6 - 1) << (6+6+5+5+4)
time & second_mask
Out[59]: 31
(time & minute_mask) >> 6
Out[63]: 42
(time & hour_mask) >> (6+6)
Out[64]: 5
(time & day_mask) >> (6+6+5)
Out[65]: 3
(time & month_mask) >> (6+6+5+5)
Out[66]: 12
(time & year_mask) >> (6+6+5+5+4)
Out[67]: 13L
In function form, the whole thing is a bit more natural:
def unmask(num, width, offset):
return (num & (2**width - 1) << offset) >> offset
Which (now that I think about it) rearranges into:
def unmask(num, width, offset):
return (num >> offset) & (2**width - 1)
unmask(time, 6, 0)
Out[77]: 31
unmask(time, 6, 6)
Out[78]: 42
#etc
And if you want to get fancy,
from itertools import starmap
from functools import partial
width_offsets = [(6,0),(6,6),(5,12),(5,17),(4,22),(6,26)]
list(starmap(partial(unmask,time), width_offsets))
Out[166]: [31, 42, 5, 3, 12, 13L]
Format all those numbers correctly and finally out comes the expected date/time:
'20{:02d}/{:02d}/{:02d}-T{:02d}:{:02d}:{:02d}'.format(*reversed(_))
Out[167]: '2013/12/03-T05:42:31'
(There is likely a way to do all of this bitwise math elegantly with that bitstring module, but I just find it satisfying to solve things from first principles.)
The time stamp in the 'C' structure is a 'C' bitfield. The compiler uses the number after the colon to allocate a number of bits within the larger field definition. In this case, an unsigned int (4 bytes). Look here for a better explanation. The big gotcha, for bit fields, is that the bits are assigned based on the endian type of the computer so they aren't very portable.
There appears to be an error in your Python format declaration. It probably should have an additional 4 byte unsigned int allocated for the date. Something like:
nal_format=('''
uint:8,
uint:8,
bin:32,
bin:32,
intle:32,
intle:32,
''')
To represent the bit field in Python, use a Python Bit Array to represent the bits. Check out this.
One other thing to be aware of, the pack(1) on the structure. It tells the compiler to align on one byte boundaries. In other words, don't add any padding between fields. typically the alignment is 4 bytes causing the compiler to start each field on a 4 byte boundary. Check here for more information.
I'm working on talking to a library that handles strings as wchar_t arrays. I need to convert these to char arrays so that I can hand them over to Python (using SWIG and Python's PyString_FromString function). Obviously not all wide characters can be converted to chars. According to the documentation for wcstombs, I ought to be able to do something like
wcstombs(NULL, wideString, wcslen(wideString))
to test the string for unconvertable characters -- it's supposed to return -1 if there are any. However, in my test case it's always returning -1. Here's my test function:
void getString(wchar_t* target, int size) {
int i;
for(i = 0; i < size; ++i) {
target[i] = L'a' + i;
}
printf("Generated %d characters, nominal length %d, compare %d\n", size,
wcslen(target), wcstombs(NULL, target, size));
}
This is generating output like this:
Generated 32 characters, nominal length 39, compare -1
Generated 16 characters, nominal length 20, compare -1
Generated 4 characters, nominal length 6, compare -1
Any idea what I'm doing wrong?
On a related note, if you know of a way to convert directly from wchar_t*s to Python unicode strings, that'd be welcome. :) Thanks!
Clearly, as you found, it's essential to zero-terminate your input data.
Regarding the final paragraph, I would convert from wide to UTF8 and call PyUnicode_FromString.
Note that I am assuming you are using Python 2.x, it's presumably all different in Python 3.x.
I have a checksum function in Python:
def checksum(data):
a = b = 0
l = len(data)
for i in range(l):
a += ord(data[i])
b += (l - i)*ord(data[i])
return (b << 16) | a, a, b
that I am trying to port to a C module for speed. Here's the C function:
static PyObject *
checksum(PyObject *self, PyObject *args)
{
int i, length;
unsigned long long a = 0, b = 0;
unsigned long long checksum = 0;
char *data;
if (!PyArg_ParseTuple(args, "s#", &data, &length)) {
return NULL;
}
for (i = 0; i < length; i++) {
a += (int)data[i];
b += (length - i) * (int)data[i];
}
checksum = (b << 16) | a;
return Py_BuildValue("(Kii)", checksum, (int)a, (int)b);
}
I use it by opening a file and feeding it a 4096 block of data. They both return the same values for small strings, but when I feed it binary data straight from a file, the C version returns wildly different values. Any help would be appreciated.
I would guess that you have some kind of overflow in your local variables. Probably b gets to large. Just dump the values for debugging purposes and you should see if it's the problem. As you mention, that you are porting the method for performance reasons. Have you checked psyco? Might be fast enough and much easier. There are more other tools which compile parts of python code on the fly to C, but I don't have the names in my head.
I'd suggest that the original checksum function is "incorrect". The value returned for checksum is of unlimited size (for any given size in MB, you could construct an input for which the checksum will be at least of this size). If my calculations are correct, the value can fit in 64 bits for inputs of less than 260 MB, and b can fit in an integer for anything less than 4096 bytes. Now, I might be off with the number, but it means that for larger inputs the two functions are guaranteed to work differently.
To translate the first function to C, you'd need to keep b and c in Python integers, and to perform the last calculation as a Python expression. This can be improved, though:
You could use C long long variables to store an intermediate sum and add it to the Python integers after a certain number of iterations. If the number of iterations is n, the maximum value for a is n * 255, and for b is len(data) * n * 255. Try to keep those under 2**63-1 when storing them in C long long variables.
You can use long long instead of unsigned long long, and raise a RuntimeError every time it gets negative in debug mode.
Another solution would be to limit the Python equivalent to 64 bits by using a & 0xffffffffffffffff and b & 0xffffffffffffffff.
The best solution would be to use another kind of checksum, like binascii.crc32.
Python says I need 4 bytes for a format code of "BH":
struct.error: unpack requires a string argument of length 4
Here is the code, I am putting in 3 bytes as I think is needed:
major, minor = struct.unpack("BH", self.fp.read(3))
"B" Unsigned char (1 byte) + "H" Unsigned short (2 bytes) = 3 bytes (!?)
struct.calcsize("BH") says 4 bytes.
EDIT: The file is ~800 MB and this is in the first few bytes of the file so I'm fairly certain there's data left to be read.
The struct module mimics C structures. It takes more CPU cycles for a processor to read a 16-bit word on an odd address or a 32-bit dword on an address not divisible by 4, so structures add "pad bytes" to make structure members fall on natural boundaries. Consider:
struct { 11
char a; 012345678901
short b; ------------
char c; axbbcxxxdddd
int d;
};
This structure will occupy 12 bytes of memory (x being pad bytes).
Python works similarly (see the struct documentation):
>>> import struct
>>> struct.pack('BHBL',1,2,3,4)
'\x01\x00\x02\x00\x03\x00\x00\x00\x04\x00\x00\x00'
>>> struct.calcsize('BHBL')
12
Compilers usually have a way of eliminating padding. In Python, any of =<>! will eliminate padding:
>>> struct.calcsize('=BHBL')
8
>>> struct.pack('=BHBL',1,2,3,4)
'\x01\x02\x00\x03\x04\x00\x00\x00'
Beware of letting struct handle padding. In C, these structures:
struct A { struct B {
short a; int a;
char b; char b;
}; };
are typically 4 and 8 bytes, respectively. The padding occurs at the end of the structure in case the structures are used in an array. This keeps the 'a' members aligned on correct boundaries for structures later in the array. Python's struct module does not pad at the end:
>>> struct.pack('LB',1,2)
'\x01\x00\x00\x00\x02'
>>> struct.pack('LBLB',1,2,3,4)
'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04'
By default, on many platforms the short will be aligned to an offset at a multiple of 2, so there will be a padding byte added after the char.
To disable this, use: struct.unpack("=BH", data). This will use standard alignment, which doesn't add padding:
>>> struct.calcsize('=BH')
3
The = character will use native byte ordering. You can also use < or > instead of = to force little-endian or big-endian byte ordering, respectively.