We Keep Coding

How to pass array of ctypes structures to pyOpenCL?

I have been following along with a tutorial online using OpenCL, where I am doing everything using python and pyOpenCL. As a stripped down example of my problem, I need to pass an array of C structs as an argument to an OpenCL kernel.
Here is an artificial example of OpenCL code:
typedef struct Test{
float a;
float3 b;
} Test;
__kernel void render_kernel(__constant Test *tests, const int width, const int height, const int num_structs, __global float3* output)
{
unsigned int work_item_id = get_global_id(0);
unsigned int x_coord = work_item_id % width;
unsigned int y_coord = work_item_id / width;
Test test = tests[0];
output[work_item_id] = test.b;
}
This will do something silly which is just give one of the float3 arrays as the output, but I just need to know I am actually getting the data through to the kernel properly.
I am trying to mimic an array of these structures on the python side with the following code:
class Test(ctypes.Structure):
_fields_ = [
("a", ctypes.c_float),
("b", (ctypes.c_float * 4))
]
class Test_Array(ctypes.Structure):
_fields_ = [("TEST_ARRAY", ctypes.POINTER(Test))]
def __init__(self, num_structs):
elems = (Test * num_structs)()
self.TEST_ARRAY = ctypes.cast(elems, ctypes.POINTER(Test))
self.elements = num_structs
for num in range(0, num_structs):
self.TEST_ARRAY[num].a = 1.0
self.TEST_ARRAY[num].b = (1.0, 0.0, 0.0, 1.0)
num_structs = 2
test_arr = Test_Array(num_structs)
#host buffer
color_out = np.empty((win.width * win.height, 4), dtype=np.float32)
cl_prog = CL()
cl_prog.load_program("shaders/struct.cl")
#device buffers
cl_structs = cl_prog.create_input_buffer(num_structs * ctypes.sizeof(Test))
cl_output = cl_prog.create_output_buffer(color_out.nbytes)
cl.enqueue_fill_buffer(cl_prog.queue, cl_structs, test_arr.TEST_ARRAY,
0, num_structs * ctypes.sizeof(Test))
global_work_size = (win.width * win.height,)
cl_prog.program.render_kernel(cl_prog.queue, global_work_size, None,
cl_structs, np.int32(win.width), np.int32(win.height),
np.int32(num_structs), cl_output)
cl_prog.retrieve_data(color_out, cl_output)
print(color_out)
This isn't really relevant as the functions in this class are just wrappers around pyOpenCL functions, but here is the CL class which is instantiated.
class CL:
def __init__(self):
self.platform = cl.get_platforms()[0]
self.device = self.platform.get_devices()[0]
self.ctx = cl.Context([self.device])
self.queue = cl.CommandQueue(self.ctx)
def load_program(self, file_path):
with open(file_path) as f:
src = f.read()
self.program = cl.Program(self.ctx, src).build()
def create_output_buffer(self, size):
"""
creates and returns a write only cl.Buffer of size bytes.
"""
mf = cl.mem_flags
return cl.Buffer(self.ctx, mf.WRITE_ONLY, size)
def create_input_buffer(self, size):
"""
returns a read only cl.Buffer of size bytes.
"""
mf = cl.mem_flags
return cl.Buffer(self.ctx, mf.READ_ONLY, size)
def retrieve_data(self, host_buffer, device_buffer):
"""
retrieves data from a buffer on the device, device_buffer, and copies it
over to host_buffer
"""
cl.enqueue_copy(self.queue, host_buffer, device_buffer)
def fill_buffer(self, memory, pattern, offset, size, wait_for=None):
"""
A wrapper around cl.enqueue_fill_buffer which uses self.queue
"""
cl.enqueue_fill_buffer(self.queue, memory, pattern, offset, size, wait_for)
def enqueue_copy(self, device_buffer, host_buffer):
cl.enqueue_copy(self.queue, device_buffer, host_buffer)
When I run the above code, it compiles and runs fine, but the information I get back from the buffer is just garbage that was already in memory. I can't tell if my problem is with alignment of the data, the way I am creating the array of ctypes structs, or what?
I am not attached to using a C array of C structs. I suspect there is a way to do this with numpy arrays, but I can't figure it out. Any way to properly get the data from the host to the device would be greatly appreciated.

Some suggest back in 2014, this could perhaps be done like so:
__kernel void render_kernel(struct Params Test, ...){
}
You can see this post.
Otherwise, something called Attributes of Variables may be an option?
Hope you got this sorted out and share the experience. I would love to see how this is done as I may want to try pass SQL query over to kernel to process.

Python ctypes: how to allocate output buffer for C function in callback

I have next callback as one of arguments in function in c-code:
typedef unsigned char* (*my_callback)(int size);
//for example:
unsigned char * tmp_buff = nullptr;
tmp_buff = i_alloc_fn(10);
printf("Tmp buff addr = %d.\n", tmp_buff);
*tmp_buff = 111;
printf("I am still alive");
I am trying to provide this callback from python (C-code is loaded as .so lib). I tried 2 ways.
ALLOC_CALLBACK_FUNC = ctypes.CFUNCTYPE(ctypes.c_char_p, ctypes.c_int)
#...
def py_alloc_callback(size):
libc = ctypes.CDLL("libc.so.6")
mem_ptr = libc.malloc(ctypes.c_uint(size))
return mem_ptr
And
ALLOC_CALLBACK_FUNC = ctypes.CFUNCTYPE(ctypes.c_char_p, ctypes.c_int)
stringbuffer = ''
#...
def py_alloc_callback(size):
global stringbuffer
stringbuffer=ctypes.create_string_buffer(size)
return ctypes.POINTER(ctypes.c_ubyte)(stringbuffer)
But both variants led to segmentation fault in C-code when it tried to write to allocated memory.
Please, help me fix it

mem_ptr = libc.malloc(ctypes.c_uint(size))
is clearly wrong. The parameter to malloc is of type size_t.

Now it works:
def py_alloc_callback(size):
libc = ctypes.CDLL("libc.so.6")
alloc_f = libc.malloc
alloc_f.restype = ctypes.c_void_p
alloc_f.argtypes = [ ctypes.c_uint ]
return alloc_f(ctypes.c_uint(size))

Creating a python callback for a C function from a DLL with a char buffer.

I am trying to create a python wrapper for a C DLL. The C DLL uses a callbacks to "send" messages via UDP from the DLL.
What I expected to happen was that python code would load the DLL library and the two functions. Then it would register the call back with the DLL using the RegisterCallbackSendMessage function. The pointer to the callback would be stored in memory in the DLL memory space. Then the python code would call the SendWhoIs DLL function. This function would create a buffer and use the SendMessage function pointer to send the message from the DLL to the python code.
The issue I am having is that the message parameter in the SendMessage function in python doesn't know how to interpreter a void * and I can not pass this buffer to sock.sendto function.
I get the following error.
sock.sendto(message, (UDP_IP, UDP_PORT))
TypeError: a bytes-like object is required, not 'int'
My question is: How do I convert a c_void_p to a byte array that sock.sendto can accept.?
I have tried to reduce my code as much as possible and still make it understandable.
This is my C code
// C DLL code
//
#define DllExport extern "C" __declspec( dllexport )
typedef uint16_t(*FPCallbackSendMessage)(const uint8_t * message, const uint32_t length);
FPCallbackSendMessage g_CallbackSendMessage;
DllExport bool RegisterCallbackSendMessage(uint16_t(*p_CallbackSendMessage)(const uint8_t * message, const uint32_t length)) {
g_CallbackSendMessage = p_CallbackSendMessage ;
return true;
}
void SendWhoIs(unsigned int dd) {
printf("dd=%d", dd);
char buffer[500];
g_CallbackSendMessage( buffer, 500 );
}
And this is my python code
# Python code
# =================
print("Start")
customDLL = cdll.LoadLibrary ("customeDLL.dll")
def SendMessage( message, length ):
print("SendMessage...")
UDP_IP = "192.168.1.1"
UDP_PORT = 47808
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(message, (UDP_IP, UDP_PORT))
return True
print("Registering Callback SendMessage...")
SendMessageFUNC = CFUNCTYPE(c_bool, c_void_p, c_uint)
SendMessage_func = SendMessageFUNC( SendMessage )
RegisterCallbackSendMessage = customDLL.RegisterCallbackSendMessage
RegisterCallbackSendMessage.argtypes = [ SendMessageFUNC ]
RegisterCallbackSendMessage( SendMessage_func )
print("Do message...")
SendWhoIs = BACnetStackDLL.SendWhoIs
SendWhoIs.argtypes = [c_uint]
SendWhoIs( 47 )
print("End")

I figured it out.
in the python SendMessage function I figured out a way to convert the c_void_p to a byte array.
# convert the pointer to a buffer.
buffer = c_char * length
messageToSend = buffer.from_address(message)

PySerial read() returning incorrect values

I have an Arduino with a standard analog temp sensor powered up by the USB port of a Raspberry Pi, and a Python script that gets all incoming values.
However, I find that the incoming values are off by about 10deg.
When I hook it up to my computer, the serial monitor returns the correct value. That narrows it down to an error on the Raspberry Pi I think.
Python code:
import serial
ser = serial.Serial('/dev/ttyACM0', 9600)
while True:
temp = float(ser.readline())
tempstring = "{:.1f}".format(temp)
print(tempstring)
f = open("/var/www/temp", "w")
f.write(tempstring)
f.close()
Arduino code (for good measure)
const int thermometer = A0;
void setup(){
Serial.begin(9600);
}
void loop(){
int sensorReading = analogRead(thermometer);
float volts = (sensorReading/1024.0) * 5.0;
float temp = (volts - 0.5) * 100.0;
int largeTemp = temp * 100.0;
Serial.println(largeTemp);
delay(10000);
}
I didn't want to bother with sending a float, hence the int largeTemp = temp * 100.0; line.

Convert string data received from network packet to Structure

I am a newbiew to python and ctypes. what I have is:-
C program:
struct query
{
uint16_t req_no;
uint32_t req_len;
uint64_t req;
};
struct response
{
uint16_t req_no;
uint16_t status;
uint32_t value_len;
uint64_t value;
};
// functions for creating query and response packets using
// above structs respectively, returning char buffer.
char* create_query(//some args);
char* create_response(//some args);
I have Created a libquery.so for the above C code. My TCP Server is a C program.
I am trying to create a TCP python client (my project needs it!) for the same.
I can successfully send query and receive data(using functions in libquery.so) from python client.
But when i get response data, I want to convert it to "struct response" type.
I have create a similar "Structure" class in python, but can't get anything out of it.
Please help.
Some code snippet of my Python code:-
// some ctypes imports
lib = cdll.LoadLibrary('./libquery.so')
class Info1(Structure):
_fields_ = [("req_no",c_int),
("status",c_int),
("value_len",c_int),
("value",c_int)]
header = Info1()
// Did some TCP connection code here and send data to server by calling
// create_query() method, data confirmed to be correct on server side...
# Receive response data
data = sock.recv(512)
header = str_to_class('Info1')
header.req_no = int(ord(data[0])) // Works; but I don't want to go this ways..
header.status = int(ord(data[1]))
header.value_len = int(ord(data[2]))
header.value = int(ord(data[3]))
print above header values..
I tried using :-
def str_to_class(Info1):
return getattr(sys.modules[__name__], Info1)
But don't know how to make it work.
Anybody know how to make it work OR is there any other way??

Your 'Info1' does not match C 'struct response'. So I changed in following code.
You can use ctypes.memmove.
from ctypes import *
class Info1(Structure):
_fields_ = [("req_no", c_uint16),
("status", c_uint16),
("value_len", c_uint32),
("value", c_uint64)]
data = (
'\x01\x00'
'\x02\x00'
'\x03\x00\x00\x00'
'\x04\x00\x00\x00\x00\x00\x00\x00'
)
# Assumed data was received. I assumed both server, clients are little-endian.
# else, use socket.ntoh{s|l}, socket.hton{s|l} ....
header = Info1()
memmove(addressof(header), data, sizeof(header))
assert header.req_no == 1
assert header.status == 2
assert header.value_len == 3
assert header.value == 4
You can also use struct.
import struct
data = '....' # same as above
struct.unpack('HHLQ', data) == (1, 2, 3, 4) # '>HHLQ' if data is htonl/htons-ed in sencding part.