Is it possible with ctypes to make pointer arithmetic?
First, let me show you what I'm trying to do in C
#include <stdio.h>
struct Foo {
short *Bar;
short *end_Bar;
};
int main() {
short tab[3] = {1,2,3};
struct Foo foo;
foo.Bar = tab;
foo.end_Bar = foo.Bar + 2; // Pointer arithmetic
short *temp = foo.Bar;
while(temp != foo.end_Bar)
printf("%hi", *(temp++));
printf("%hi", *(foo.end_Bar));
return 0;
}
Now you understand that what I'm doing is creating an array of integer, and keeping in reference two pointers in a structure. One pointer at the begining and one at the end, instead of keeping the first pointer and the length of the array.
Now in Python I have an object that inherit from ctypes.Structure and as two members which are ctypes.POINTER(ctypes.c_short) type.
import ctypes
class c_Foo(ctypes.Structure):
_fields_ = [
("Bar", ctypes.POINTER(ctypes.c_short)),
("end_Bar", ctypes.POINTER(ctypes.c_short))
]
if __name__ == "__main__":
tab = [1,2,3]
foo = c_Foo()
foo.Bar = (c_short * len(tab))(*tab)
foo.end_Bar = foo.Bar + 2 # TypeError, unsupported operand
So now the question. Is it possible to do pointer arithmetic with ctypes? I know that you can access value of the array by it index, but I don't want that, because I don't want a length reference in my structure.
It's convoluted, but this computes a c_short object at a byte offset in tab that shares its buffer, then gets a pointer to it:
from ctypes import *
class c_Foo(Structure):
_fields_ = [
("Bar", POINTER(c_short)),
("end_Bar", POINTER(c_short))
]
tab = (c_short*3)(1,2,3)
foo = c_Foo()
foo.Bar = tab
foo.end_Bar = pointer(c_short.from_buffer(tab,sizeof(c_short)*2))
print(tab[2])
print(foo.Bar[2])
print(foo.end_Bar[0])
tab[2] = 4
print(tab[2])
print(foo.Bar[2])
print(foo.end_Bar[0])
3
3
3
4
4
4
Related
I want to create a Python-datatype using ctypes that matches the C-datatype "const char**", which resembles an array of pointers. However, I'm not able to code this in Python.
The simplified C-function header looks like this:
int foo(int numOfProp, const char** propName, const char** propValue);
In C, the correct function call would look like this:
const char *PropName[2];
PropName[0] = "Prop_Index_1";
PropName[1] = "Prop_Index_2";
const char *PropValue[2];
PropValue[0] = "10";
PropValue[1] = "20";
stream_id = (*foo)(2, PropName, PropValue);
Basically, the function takes two arrays (pair of name and value) as well as the length of both arrays, and returns a stream ID. When the DLL is loaded, I can see that the function expects this ctypes datatype for the property arrays:
"LP_c_char_p"
However, I am really struggling to create this datatype based on lists of strings.
My first attempt (based on How do I create a Python ctypes pointer to an array of pointers) looks like this:
# set some dummy values
dummy_prop_values = [
"10",
"20"
]
# create property dict
properties = {
f"Prop_Index_{i}": dummy_prop_values[i] for i in range(len(dummy_prop_values))
}
def first_try():
# create a dummy ctype string
ctypes_array = ctypes.c_char_p * 2
# create empty c-type arrays for the stream properties
prop_names = ctypes_array()
prop_values = ctypes_array()
# fill the empty arrays with their corresponding values
for i, (prop_name, prop_value) in enumerate(properties.items()):
prop_names[i] = prop_name.encode()
prop_values[i] = prop_value.encode()
# get pointer to properties
ptr_prop_names = ctypes.pointer(prop_names)
ptr_prop_values = ctypes.pointer(prop_values)
return ptr_prop_names, ptr_prop_values
It throws this kind of error when I hand over the returned values to the function foo (which actually makes sense, since I explicitly created an array of length 2... I don't know how/why this worked for the other guy asking the question):
ctypes.ArgumentError: argument 2: <class 'TypeError'>: expected LP_c_char_p instance instead of LP_c_char_p_Array_2
My second attempt (based more or less on my own thoughts) looks like this:
def second_try():
# convert properties to lists
prop_names = [x for x in properties.keys()]
prop_values = [x for x in properties.values()]
# concat list elements, zero terminated
# but I guess this is wrong anyway because it leads to an early string-termination (on byte-level)...?
prop_names = ctypes.c_char_p("\0".join(prop_names).encode())
prop_values = ctypes.c_char_p("\0".join(prop_values).encode())
# get pointer to properties
ptr_prop_names = ctypes.pointer(prop_names)
ptr_prop_values = ctypes.pointer(prop_values)
return ptr_prop_names, ptr_prop_values
This actually doesn't throw an error, but returns -1 as the stream ID, which denotes that "creating the stream wasn't successfull". I double checked all the other arguments of the function call, and these two properties are the only ones that can be wrong somehow.
For whatever reason I just can't figure out exactly where I make a mistake, but hopefully someone here can point me in the right direction.
To convert a list some type into a ctypes array of that type, the straightforward idiom is:
(element_type * num_elements)(*list_of_elements)
In this case:
(c_char_p * len(array))(*array)
Note that (*array) expands the array as if each individual element was passed as a parameter, which is required to initialize the array.
Full example:
test.c - To verify the parameters are passed as expected.
#include <stdio.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
API int foo(int numOfProp, const char** propName, const char** propValue) {
for(int i = 0; i < numOfProp; i++)
printf("name = %s value = %s\n", propName[i], propValue[i]);
return 1;
}
test.py
import ctypes as ct
dll = ct.CDLL('./test')
# Always define .argtypes and .restype to help ctypes error checking
dll.foo.argtypes = ct.c_int, ct.POINTER(ct.c_char_p), ct.POINTER(ct.c_char_p)
dll.foo.restype = ct.c_int
# helper function to build ctypes arrays
def make_charpp(arr):
return (ct.c_char_p * len(arr))(*(s.encode() for s in arr))
def foo(arr1, arr2):
if len(arr1) != len(arr2):
raise ValueError('arrays must be same length')
return dll.foo(len(arr1) ,make_charpp(arr1), make_charpp(arr2))
foo(['PropName1', 'PropName2'], ['10', '20'])
Output:
name = PropName1 value = 10
name = PropName2 value = 20
So I have a function imported from a .dll library that takes in a pointer to a struct of the form:
struct D {
DWORD A;
BYTE * B;
};
The idea of function is that if A is NULL, then function(D*) will update A with the size of the required buffer. Hence, if on the other hand B is an array of size A, then function(D*) will return the filled array with the A bytes.
In trying to import the C function with ctypes, I mimic the code:
class D(Structure):
_fields_ = [("A",DWORD),("B",POINTER(BYTE))]
#function is imported from .dll
function.argtypes = [POINTER(D)]
function.restype = BOOL
But when I attempt in python to run the function twice, I get a type error that LP_c_byte_p_250 doesn't work with LP_c_byte (sorry, I am on mobile and may not have the names quite right).
data = D()
function(pointer(data))
size = data.A #returns 250
buf = ARRAY(BYTE,size)
data.B = pointer(buf)
function(pointer(data))
How do I set struct up so that ctypes doesn't prevent any sized array from occupying variable B?
Just to point out, if I skip the first function call and redefine my struct and argtypes explicitly, then I do get it to work.
class D(Structure):
_fields_ = [("A",DWORD),("B",POINTER(ARRAY(BYTE,250)))]
#function is imported from .dll
function.argtypes = [POINTER(D)]
function.restype = BOOL
data = D()
data.A = 250
function(pointer(data)) #returns 250 bytes to B
So clearly I can recreate my struct and reimport my function every time I have a different size, but that doesn't seem right. What am I doing wrong?
Here's an example. Just make sure the types agree:
test.c (C example to demonstrate)
#include <windows.h>
struct D {
DWORD A;
BYTE* B;
};
__declspec(dllexport)
BOOL function(struct D* data)
{
if(data->A)
memset(data->B,'x',data->A); // Return data if size non-zero
else
data->A = 10; // return a size
return TRUE;
}
test.py
from ctypes import *
from ctypes import wintypes as w
class D(Structure):
_fields_ = [("A",w.DWORD),("B",POINTER(w.BYTE))]
# A pointer doesn't know the size of data it points to, so passing through
# bytes() and slicing to known size makes a nice display.
def __repr__(self):
return f'D(A={self.A}, B={bytes(self.B[:data.A])})'
dll = CDLL('./test')
dll.function.argtypes = POINTER(D),
dll.function.restype = w.BOOL
data = D()
print('pre ',data)
if dll.function(byref(data)):
data.B = (w.BYTE * data.A)() # array of w.BYTE instantiated this way
# can be directly assigned to a POINTER(BYTE)
print('before',data)
if dll.function(byref(data)):
print('after ',data)
output:
pre D(A=0, B=b'')
before D(A=10, B=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
after D(A=10, B=b'xxxxxxxxxx')
It appears the missing ingredient was to use the function cast in ctypes. It seems as though ctypes does some generic casting itself automatically in simple cases, but when presented something more difficult (like a struct built of two other stucts), it doesn't seem to automatically cast.
data = D()
function(pointer(data))
size = data.A #returns 250
buf = ARRAY(BYTE,size)
data.B = pointer(buf)
buf = cast(buf,POINTER(D))
function(pointer(data))
This method even appears to work through a c_void_p; in which case you can forcibly use C functions without needing to explicitly define the exact structure they need in ctypes.
I'm having difficulty passing string from Python to C by ctypes:
My C code(compiled into a hello.so)
...
typedef struct test{
unsigned int a;
unsigned char* b;
} my_struct;
int hello(my_struct *in) {
FILE *fp;
...
fprintf(fp, "%d\t%s\n", in->a, in->b);
fclose(fp);
return 0;
}
My Python code:
...
from ctypes import *
...
class TEST_STRUCT(Structure):
_fields_ = [("a", c_int),
("b", c_char_p)]
...
hello_lib = ctypes.cdll.LoadLibrary("hello.so")
hello = hello_lib.hello
hello.argtypes = [POINTER(TEST_STRUCT)]
name = create_string_buffer(b"test")
hello_args = TEST_STRUCT(1, name)
hello(ctypes.byref(hello_args))
...
I get the error:
hello_args = TEST_STRUCT(1, name)
TypeError: expected string, c_char_Array_5 found
I tried to change c_char_p to c_wchar_p or c_char*5 or c_wchar*5 etc. Sometimes it can run without error, the first int parameter of the struct can be printed correctly, but not the second string pointer, the best I can get is just the first character 't' instead of the whole word "test".
BTW, my python3 version is 3.3.0
The solution depends on whether you need the data type to be mutable (from Python's perspective).
If don't need the data to be mutable, then don't bother with the buffer constructor. Just pass the bytes object direct to the TEST_STRUCT constructor. eg.
from ctypes import *
class TEST_STRUCT(Structure):
_fields_ = [("a", c_int),
("b", c_char_p)]
hello_args = TEST_STRUCT(1, b"test")
If you need a mutable buffer then you need to specify your char* type slightly differently in the TEST_STRUCT class. eg.
from ctypes import *
class TEST_STRUCT(Structure):
_fields_ = [("a", c_int),
("b", POINTER(c_char))] # rather than c_char_p
name = create_string_buffer(b"test")
hello_args = TEST_STRUCT(1, name)
c_char_p versus POINTER(c_char)
Ostensibly, whilst these two appear similar they are subtly different. POINTER(c_char) is for any buffer of char data. c_char_p is specifically for NULL terminated strings. As a side effect of this condition all c_char_p objects are immutable (on the python side) so as to make this guarantee easier to enforce. eg. {65, 66, 67, 0} is a string ("ABC"), whereas {1, 2, 3} would not be.
I have the following struct in my C program:
typedef struct
{
void* str;
DWORD str_length;
DWORD count;
} mystruct;
I would like to create a buffer in Python, write it to a file, and then read it from my C program, and reference this buffer as a "mystruct".
what I tried in python was:
from struct import *
str = raw_input("str: ")
count = raw_input("count: ")
s = Struct ( str(len(str)) + 'sLL' )
s.pack( str , len(str) , int(count))
it returns a binary buffer , but not with my data.
where have I got it wrong? is there a better way to do it?
I'm preferring using ctypes for this kind of job. It gives more options in handling pointers. Here is an example how your issue could be resolved using ctypes (in Python 3.x but you can easily convert it to Python 2.x).
Your structure still doesn't contains data. Only pointer to buffer that is accessible from C code. This buffer is created using create_string_buffer that convert python string to C null terminated one.
from ctypes import *
class mystruct(Structure):
_fields_ = [("str", c_char_p),
("str_length", c_long),
("count", c_long)]
s = b"ABC" * 10
c = 44
#convert python string to null terminated c char array
c_str = create_string_buffer(s)
#cast char array to char*
c_str_pointer = cast(c_str, c_char_p)
#create structure
a = mystruct(c_str_pointer, len(s), c)
# print byte representation of structure
print(bytes(a))
print(a.str)
print(a.str_length)
print(a.count)
If you don't need pointers and constant buffer is enough you can also do this using ctypes
from ctypes import *
class mystruct(Structure):
_fields_ = [("str", c_char*100),
("str_length", c_long),
("count", c_long)]
s = b"ABC" * 10
c = 44
a = mystruct(s, len(s), c)
# print byte representation of structure
print(bytes(a))
print(a.str)
print(a.str_length)
print(a.count)
I'm trying to interface a shared C library to some python code.
The interface with the library is something like
typedef struct{
int v1;
double* v2} input;
There are two other types like this one: for configuration and the output type.
I set up these structs in python using ctypes Structures like this:
class input(Structure):
_fields_ = [("v1",c_int),("v2",POINTER(c_double)]
The C code have some functions that receive a pointer to that structure and the argtypes are defined as in:
fun.argtypes = [constraints,input,POINTER(input)]
constraints is another structure with some int fields for configuration.
First I update the v2 field in the input struct
input.v2 = generated_array.ctypes.data_as(POINTER(c_double))
Then I call it:
fun(constraints,input,byref(output))
The function prototype asks for struct and * to struct (output struct type is assumed to be the same as the input struct type).
Then I want to access the results of fun stored in the v2 field of output. But I'm getting unexpected results. Is there a better/correct way of do this?
I searched a lot here and read the documentation but I can't find what's wrong. I don't have any error messages, but the warnings that I receive from the shared lib seems to show that there are mistakes on these interfaces.
I guess I have found the problem:
When I call the method a numpy array of complex is called. Then I create 4 vectors:
out_real = ascontiguousarray(zeros(din.size,dtype=c_double))
out_imag = ascontiguousarray(zeros(din.size,dtype=c_double))
in_real = ascontiguousarray(din.real,dtype = c_double)
in_imag = ascontiguousarray(din.imag,dtype = c_double)
where din is the input vector. I had tested the method this way:
print in_real.ctypes.data_as(POINTER(c_double))
print in_imag.ctypes.data_as(POINTER(c_double))
print out_real.ctypes.data_as(POINTER(c_double))
print out_imag.ctypes.data_as(POINTER(c_double))
and the result was:
<model.LP_c_double object at 0x1d81f80>
<model.LP_c_double object at 0x1d81f80>
<model.LP_c_double object at 0x1d81f80>
<model.LP_c_double object at 0x1d81f80>
It seems that all of them are pointing to the same place.
After some changes, it works as expected...
After several tests I found that the code was nearly correct the first time. I was creating the Structure instance once and updating its fields. I changed to creating a new instance at each call of fun. I also changed all arrays types to be the equivalent ctypes type; this seemed to make the function work as expected.
The print behavior still holds as in the test above, but the function seems to work even with this strange behavior. It is correct as pointed by #ericsun comment below.
The struct has an int field that's possibly for the array length, though I'm just guessing without the complete function prototype. If this is indeed the case, here's an example that may help.
First, I need to compile a test function in a shared library. I'll simply multiply the input array by 2:
import os
import numpy as np
from ctypes import *
open('tmp.c', 'w').write('''\
typedef struct {
int v1; double *v2;
} darray;
int test(darray *input, darray *output) {
int i;
/* note: this should first test for compatible size */
for (i=0; i < input->v1; i++)
*(output->v2 + i) = *(input->v2 + i) * 2;
return 0;
}
''')
os.system('gcc -shared -o tmp.so tmp.c')
Next create the ctypes definitions. I added a classmethod to make a darray from a numpy.ndarray:
c_double_p = POINTER(c_double)
class darray(Structure):
_fields_ = [
('v1', c_int),
('v2', c_double_p),
]
#classmethod
def fromnp(cls, a):
return cls(len(a), a.ctypes.data_as(c_double_p))
lib = CDLL('./tmp.so')
lib.test.argtypes = POINTER(darray), POINTER(darray)
Test:
a1 = np.arange(3) + 1.0
a2 = np.zeros(3)
print 'before:', '\na1 =', a1, '\na2 =', a2
lib.test(darray.fromnp(a1), darray.fromnp(a2))
print 'after:', '\na1 =', a1, '\na2 =', a2
Output:
before:
a1 = [ 1. 2. 3.]
a2 = [ 0. 0. 0.]
after:
a1 = [ 1. 2. 3.]
a2 = [ 2. 4. 6.]