We Keep Coding

How to set arbitrary size array in ctypes?

So I have a function imported from a .dll library that takes in a pointer to a struct of the form:
struct D {
DWORD A;
BYTE * B;
};
The idea of function is that if A is NULL, then function(D*) will update A with the size of the required buffer. Hence, if on the other hand B is an array of size A, then function(D*) will return the filled array with the A bytes.
In trying to import the C function with ctypes, I mimic the code:
class D(Structure):
_fields_ = [("A",DWORD),("B",POINTER(BYTE))]
#function is imported from .dll
function.argtypes = [POINTER(D)]
function.restype = BOOL
But when I attempt in python to run the function twice, I get a type error that LP_c_byte_p_250 doesn't work with LP_c_byte (sorry, I am on mobile and may not have the names quite right).
data = D()
function(pointer(data))
size = data.A #returns 250
buf = ARRAY(BYTE,size)
data.B = pointer(buf)
function(pointer(data))
How do I set struct up so that ctypes doesn't prevent any sized array from occupying variable B?
Just to point out, if I skip the first function call and redefine my struct and argtypes explicitly, then I do get it to work.
class D(Structure):
_fields_ = [("A",DWORD),("B",POINTER(ARRAY(BYTE,250)))]
#function is imported from .dll
function.argtypes = [POINTER(D)]
function.restype = BOOL
data = D()
data.A = 250
function(pointer(data)) #returns 250 bytes to B
So clearly I can recreate my struct and reimport my function every time I have a different size, but that doesn't seem right. What am I doing wrong?

Here's an example. Just make sure the types agree:
test.c (C example to demonstrate)
#include <windows.h>
struct D {
DWORD A;
BYTE* B;
};
__declspec(dllexport)
BOOL function(struct D* data)
{
if(data->A)
memset(data->B,'x',data->A); // Return data if size non-zero
else
data->A = 10; // return a size
return TRUE;
}
test.py
from ctypes import *
from ctypes import wintypes as w
class D(Structure):
_fields_ = [("A",w.DWORD),("B",POINTER(w.BYTE))]
# A pointer doesn't know the size of data it points to, so passing through
# bytes() and slicing to known size makes a nice display.
def __repr__(self):
return f'D(A={self.A}, B={bytes(self.B[:data.A])})'
dll = CDLL('./test')
dll.function.argtypes = POINTER(D),
dll.function.restype = w.BOOL
data = D()
print('pre ',data)
if dll.function(byref(data)):
data.B = (w.BYTE * data.A)() # array of w.BYTE instantiated this way
# can be directly assigned to a POINTER(BYTE)
print('before',data)
if dll.function(byref(data)):
print('after ',data)
output:
pre D(A=0, B=b'')
before D(A=10, B=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
after D(A=10, B=b'xxxxxxxxxx')

It appears the missing ingredient was to use the function cast in ctypes. It seems as though ctypes does some generic casting itself automatically in simple cases, but when presented something more difficult (like a struct built of two other stucts), it doesn't seem to automatically cast.
data = D()
function(pointer(data))
size = data.A #returns 250
buf = ARRAY(BYTE,size)
data.B = pointer(buf)
buf = cast(buf,POINTER(D))
function(pointer(data))
This method even appears to work through a c_void_p; in which case you can forcibly use C functions without needing to explicitly define the exact structure they need in ctypes.

Python ctypes: how to use pointer passed to DLL?

Using ctypes, I create an array in Python to pass to a ctypes.WinDLL:
arrA = (ctypes.c_float * len(varA))(*varA)
Then I pass a pointer to the array into the DLL:
retvar = SimpleTest(ctypes.byref(pvarr),ctypes.byref(pvarr2))
Where the first element of pvarr is a pointer to the array arrA.
But when I try to write to values to the array arrA (using the pointer that was passed), I get "invalid access to memory location."
So my question is: how can I now use the pointer to a ctypes.c_float array created in Python and passed to a DLL? Is that not possible?
After working on this for awhile, I have refined the problem. The array shown above is immutable; to pass a mutable array, we do this:
OutputArrayType = ctypes.c_float * 1000
arrNew = OutputArrayType()
The ctypes array type is a class, so to use it we must create a class instance (arrayNew).
But when I pass its pointer by reference:
retvar = SimpleTest(ctypes.byref(PVarrNew),ctypes.byref(arrNew))
the pointer value is a different number from id(arrayNew). Even so, the DLL still cannot access any values in either pointer, as I get "invalid access to memory."
So my question now is: why does the pointer passed by ctypes.byref(arrNew) differ from id(arrNew).
Mark, thanks for your reply. Here is the concise summary:
The first array is not mutable in place, it creates a new object with each assignment; I confirmed that by checking its id() before and after an assignment. However, the second array, instantiated as a class, is mutable in place so that's why I use it.
So I create the array:
OutputArrayType = ctypes.c_int64 * 1000
arrNew = OutputArrayType()
and call the DLL:
retvar = SimpleTest(ctypes.byref(PVarrNew),ctypes.byref(arrNew))
In 64-bit assembler, the second parameter (the pointer to arrNew) is passed in rdx. So in NASM, I can write to this array in two different ways, but both return "invalid access to memory location."
mov rdi,rdx
mov rax,1534
mov [rdi+8],rax
push qword [rax]
pop qword [rdx+32]
If I reverse the order of the arrays in the call to the DLL (where the pointer to the array is in rcx), the same happens.
However, I can read from the array, but I can't write to it.
Thanks very much for your help.

Your two examples are both mutable. You mentioned an assignment, that always creates a new object but without an example... :^) Here's what I mean about a reproducible example. Works compiled 32- or 64-bit using appropriate Python:
test.c (Windows example with Microsoft compiler)
#include <stdio.h>
__declspec(dllexport) void __stdcall SimpleTest(float* array, int length)
{
int i;
for(i = 0; i < length; ++i)
{
printf("array[%d] = %f\n",i,array[i]);
array[i] *= 2;
}
}
test.py (portable between Python 2 and 3)
from __future__ import print_function
from ctypes import *
varA = [1.2,2.5,3.5]
arr = (c_float * len(varA))(*varA) # 1st example
print(id(arr))
arr[0] = 1.5 # mutate array
print(id(arr))
arrType = c_float * 10 # 2nd example
arrInstance = arrType()
print(id(arrInstance))
arrInstance[0] = 1.5 # mutate array
print(id(arrInstance))
dll = WinDLL('test')
print(list(arr))
dll.SimpleTest(arr,len(arr)) # Passing the array to C and changing it
print(list(arr))
Output (same on 32-and 64- except the IDs are different, note the ID doesn't change after mutation)
2610964912456
2610964912456
2610964912968
2610964912968
[1.5, 2.5, 3.5]
array[0] = 1.500000
array[1] = 2.500000
array[2] = 3.500000
[3.0, 5.0, 7.0]

python ctypes pointer arithmetic

Is it possible with ctypes to make pointer arithmetic?
First, let me show you what I'm trying to do in C
#include <stdio.h>
struct Foo {
short *Bar;
short *end_Bar;
};
int main() {
short tab[3] = {1,2,3};
struct Foo foo;
foo.Bar = tab;
foo.end_Bar = foo.Bar + 2; // Pointer arithmetic
short *temp = foo.Bar;
while(temp != foo.end_Bar)
printf("%hi", *(temp++));
printf("%hi", *(foo.end_Bar));
return 0;
}
Now you understand that what I'm doing is creating an array of integer, and keeping in reference two pointers in a structure. One pointer at the begining and one at the end, instead of keeping the first pointer and the length of the array.
Now in Python I have an object that inherit from ctypes.Structure and as two members which are ctypes.POINTER(ctypes.c_short) type.
import ctypes
class c_Foo(ctypes.Structure):
_fields_ = [
("Bar", ctypes.POINTER(ctypes.c_short)),
("end_Bar", ctypes.POINTER(ctypes.c_short))
]
if __name__ == "__main__":
tab = [1,2,3]
foo = c_Foo()
foo.Bar = (c_short * len(tab))(*tab)
foo.end_Bar = foo.Bar + 2 # TypeError, unsupported operand
So now the question. Is it possible to do pointer arithmetic with ctypes? I know that you can access value of the array by it index, but I don't want that, because I don't want a length reference in my structure.

It's convoluted, but this computes a c_short object at a byte offset in tab that shares its buffer, then gets a pointer to it:
from ctypes import *
class c_Foo(Structure):
_fields_ = [
("Bar", POINTER(c_short)),
("end_Bar", POINTER(c_short))
]
tab = (c_short*3)(1,2,3)
foo = c_Foo()
foo.Bar = tab
foo.end_Bar = pointer(c_short.from_buffer(tab,sizeof(c_short)*2))
print(tab[2])
print(foo.Bar[2])
print(foo.end_Bar[0])
tab[2] = 4
print(tab[2])
print(foo.Bar[2])
print(foo.end_Bar[0])
3
3
3
4
4
4

Why does setting ctypes dll.function.restype=c_void_p return long?

It seems odd that even after setting restype, python returns long rather than c_void_p.
For example;
# python code
from ctypes import *
dll = windll.LoadLibrary("my.dll")
dll.my_object_create.restype = c_void_p
x = dll.my_object_create()
print type(x) # prints <type 'long'>
//c++ code
my_object *my_object_create() { return new my_object(); }
void my_object_destroy(my_object *obj) { delete obj; }
I recently had to fix a bug where, feeding x back to another ctypes function, the pointer got trampled. This was fixed by changing the initial dll call to
x = c_void_p(dll.my_object_create())
...I'm guessing somewhere along the line ctypes treated x as 4 bytes long not 8 (64 bit architecture).
So I am wondering if there is a reason why the existing behaviour leads you into this trap?

P_get for the 'P' pointer type uses PyLong_FromVoidPtr. If the address fits in a platform long, it returns a Python int; otherwise it returns a Python long, which has variable precision. That's fine, but when passing this integer value as an argument, the default behavior is to convert to a C int, which is 32-bit on all supported platforms.
I think the best solution is to set argtypes to properly convert an argument to a pointer type. Another option is to set restype to a subclass of c_void_p. Using a subclass disables the conversion to a Python integer. GetResult checks this by calling _ctypes_simple_instance, which actually returns the opposite of what its name and the source comment suggest. (In 2.5 this function was named IsSimpleSubType, and the source comment was wrong back then too. The "simple" in question was never the metaclass PyCSimpleType, but the base type _SimpleCData.)
POSIX:
# Configure the interpreter to load visible extension-
# module symbols, such as _ctypes_simple_instance,
# into the global symbol table.
import sys, DLFCN
sys.setdlopenflags((sys.getdlopenflags() & ~DLFCN.RTLD_LOCAL) |
DLFCN.RTLD_GLOBAL)
from ctypes import *
_ctypes_simple_instance = PyDLL(None)._ctypes_simple_instance
_ctypes_simple_instance.argtypes = py_object,
malloc = CDLL(None).malloc
class my_void_p(c_void_p):
pass
>>> _ctypes_simple_instance(c_void_p)
0
>>> _ctypes_simple_instance(my_void_p)
1
>>> malloc.restype = c_void_p
>>> type(malloc(100))
<type 'int'>
>>> malloc.restype = my_void_p
>>> type(malloc(100))
<class '__main__.my_void_p'>
Windows:
_ctypes_simple_instance isn't exported by _ctypes.pyd.
from ctypes import *
malloc = cdll.msvcrt.malloc
class my_void_p(c_void_p):
pass
>>> malloc.restype = c_void_p
>>> type(malloc(100))
<class 'int'>
>>> malloc.restype = my_void_p
>>> type(malloc(100))
<class '__main__.my_void_p'>

How to pack and unpack using ctypes (Structure <-> str)

This might be a silly question but I couldn't find a good answer in the docs or anywhere.
If I use struct to define a binary structure, the struct has 2 symmetrical methods for serialization and deserialization (pack and unpack) but it seems ctypes doesn't have a straightforward way to do this. Here's my solution, which feels wrong:
from ctypes import *
class Example(Structure):
_fields_ = [
("index", c_int),
("counter", c_int),
]
def Pack(ctype_instance):
buf = string_at(byref(ctype_instance), sizeof(ctype_instance))
return buf
def Unpack(ctype, buf):
cstring = create_string_buffer(buf)
ctype_instance = cast(pointer(cstring), POINTER(ctype)).contents
return ctype_instance
if __name__ == "__main__":
e = Example(12, 13)
buf = Pack(e)
e2 = Unpack(Example, buf)
assert(e.index == e2.index)
assert(e.counter == e2.counter)
# note: for some reason e == e2 is False...

The PythonInfo wiki has a solution for this.
FAQ: How do I copy bytes to Python from a ctypes.Structure?
def send(self):
return buffer(self)[:]
FAQ: How do I copy bytes to a ctypes.Structure from Python?
def receiveSome(self, bytes):
fit = min(len(bytes), ctypes.sizeof(self))
ctypes.memmove(ctypes.addressof(self), bytes, fit)
Their send is the (more-or-less) equivalent of pack, and receiveSome is sort of a pack_into. If you have a "safe" situation where you're unpacking into a struct of the same type as the original, you can one-line it like memmove(addressof(y), buffer(x)[:], sizeof(y)) to copy x into y. Of course, you'll probably have a variable as the second argument, rather than a literal packing of x.

Have a look at this link on binary i/o in python:
http://www.dabeaz.com/blog/2009/08/python-binary-io-handling.html
Based on this you can simply write the following to read from a buffer (not just files):
g = open("foo","rb")
q = Example()
g.readinto(q)
To write is simply:
g.write(q)
The same for using sockets:
s.send(q)
and
s.recv_into(q)
I did some testing with pack/unpack and ctypes and this approach is the fastest except for writing straight in C

Tested on Python3
e = Example(12, 13)
serialized = bytes(e)
deserialized = Example.from_buffer_copy(serialized)