We Keep Coding

How to pass a string pointer-to-pointer from C to Python?

I need to pass a string pointer-to-pointer from C to Python, so Python can update the pointer and C can read it later.
Steps
C set a char**
C call Python
Python allocate memory
Python update the char**
C read the string
C Code:
#include <stdio.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef void (*CALLBACK)(char**);
CALLBACK g_cb;
// Expose API to register the callback
API void set_callback(CALLBACK cb) {
g_cb = cb;
}
// Expose API to call the Python callback with a char**
API void call_python_function(char** pp) {
if(g_cb) {
g_cb(pp);
printf("Python response: %s\n", *pp);
}
}
Python Code:
import ctypes as ct
CALLBACK = ct.CFUNCTYPE(None, PPCHAR)
dll = ct.CDLL('./test')
dll.set_callback.argtypes = CALLBACK,
dll.set_callback.restype = None
dll.call_python_function.argtypes = POINTER(POINTER(ctypes.c_char)),
dll.call_python_function.restype = None
dll.set_callback(my_function)
def my_function(pp):
buffer = ct.create_string_buffer(128)
pp = buffer
Output:
Python response: (null)
No errors or warnings while building, C can call the Python function no issue, but Python can't update the char**. My question is How can I pass a string pointer-to-pointer from C to Python ?

Here's a working example of passing a char** from C to Python.
test.c
#include <stdio.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
typedef void (*CALLBACK)(char**);
CALLBACK g_cb;
// Expose API to register the callback
API void set_callback(CALLBACK cb) {
g_cb = cb;
}
// Expose API to call the Python callback with a char**
API void call_python_function(char** pp) {
if(g_cb) {
g_cb(pp);
printf("%s\n", *pp);
}
}
test.py
import ctypes as ct
# Set up some types.
# Note that `c_char_p` can't be used as ctypes has special handling
# to convert it to a Python bytes object that inteferes with the
# callback working properly.
PCHAR = ct.POINTER(ct.c_char)
PPCHAR = ct.POINTER(PCHAR)
CALLBACK = ct.CFUNCTYPE(None, PPCHAR) # Note first parameter is return value
dll = ct.CDLL('./test')
# Declare function arguments and return values
dll.set_callback.argtypes = CALLBACK,
dll.set_callback.restype = None
dll.call_python_function.argtypes = PPCHAR,
dll.call_python_function.restype = None
# Set up callback function. Note that the buffer can't go out-of-scope
# once the function returns or undefined behavior occurs, so the buffer
# is stored as an attribute of the function object so it will continue
# to exist. A global variable would work, too.
#CALLBACK
def my_function(pp):
my_function.buffer = ct.create_string_buffer(b'Hi From Python')
pp[0] = my_function.buffer # [0] dereferences char** so can assign char*
dll.set_callback(my_function)
p = PCHAR()
dll.call_python_function(ct.byref(p))
# Cast to a `c_char_p` to access `.value` and get a bytes object
# up to the terminating null.
print(ct.cast(p, ct.c_char_p).value)
Output:
Hi From Python
b'Hi From Python'

How to insert a C struct dynamic link lib into Python using ctypes?

I wrote a python code embedded with C code by using ctypes.
the C code is as follows:
test.h
#include<Python.h>
PyObject *getFeature(wchar_t *text);
// where the unigram is a Set Object with type 'PySetObject'
test.c
#include<test.h>
PyObject *getFeature(wchar_t *text)
{
int ret = -1;
// the below three line is to initialize for the following statements.
// they are not being used because this func is a simple version.
const wchar_t *startFeature = L"[START]";
const wchar_t *endFeature = L"[END]";
const wchar_t *delim = L".";
PyObject *featureList = PyList_New(0);
PyObject *curString = PyUnicode_FromWideChar(text, 2);
ret = PyList_Append(featureList, curString);
Py_DECREF(curString);
return featureList;
}
the above C code is a simple version, and the whole C code is at thw whole C code and there exists about 20 const wchar_t * variables initialized in the beginning of the C function
and then I compiled it and get a shared lib called libtest.so. So I can import this C .so file into the python code with ctypes like below:
test.py
import ctypes
dir_path = 'path/to/the/libtest.so'
feature_extractor = ctypes.PyDLL(
os.path.join(dir_path, 'libtest.so'))
get_feature_c = feature_extractor.getFeature
get_feature_c.argtypes = [
ctypes.c_wchar_p, ctypes.py_object]
get_feature_c.restype = ctypes.py_object
def get_feature(text):
return [text[:2]]
times = 100000
for i in range(times):
res = get_feature_c('ncd') # this func will always initialized const wchar_t * variables first
and in this example, those three initialization statements const wchar_t *startFeature and etc. is being initialized every time in the loop when calling the C func.
the question is :
I wanna first initialize those const wchar_t* variables in the python code and then call the get_feature_c func, just like a c++ class, thus avoiding executing the initialization in every loop.
So, how to use ctypes to realize my needs? and I only find that ctypes wraps a C func, and can not wrap a C struct to form a class like c++.
Thank you very much, outperforming programming master~

Loading c DLL with python, problems with pointer

Currently I have C++ loading DLL. I need replace C++ code with python. My problems are:
in callback function device_ID1_callback, all values seems empty, i'm guessing i did not use pointer correctly.
after call device_get_info, all values are 0, I suppose get some values none zero back.
I have tried anything I can think of for weeks but with very little luck.
To simplified the problem, here's partial of my code. Thanks for your time and help!!
in my lib.h file, i have
typedef unsigned int DeviceHandler;
typedef struct {
unsigned int fpga_version;
}DeviceInfo_t;
typedef struct {
unsigned int check_id;
float distance[256];
}MeasureResult_t;
DLLEPXORT int EXCALL device_add(DeviceHandler* outHandler, char* device_ip, MeasureModeCallback callback);
DLLEPXORT void EXCALL device_get_info(DeviceHandler handler, DeviceInfo_t* p_device_info);
in sample C++ file:
"""
void device_ID1_callback(const void *out,unsigned int out_num){
MeasureResult_t *ptr = (MeasureResult_t *)out;
printf("[ChechID:0x%x] %d pack's data\n",ptr[0].check_id,out_num);
}
void demo_callback_mode(){
int ret;
DeviceHandler device_handler;
DeviceInfo_t device_info;
ret = device_add(&device_handler,"192.168.1.2",&device_ID1_callback);
device_get_info(device_handler,&device_info);
printf("[FPGA] version : %d\n", device_info.fpga_version);
}
"""
*end of c++ *
Here's my python code:
"""
import ctypes as c
class MeasureResult_t(c.Structure):
_fields_ = [
('check_id', c.c_int),
('distance[256]', c.c_float)]
class DeviceInfo_t(c.Structure):
_fields_ = [
('fpga_version', c.c_int)
]
def device_ID1_callback(out, out_num):
print("---enter device call back function---")
print(dir(out))
print("out: ",out.contents)
#print(dir(out))
print("out_num:",out_num)
print("---exit device call back function---\n\n")
return 0
_dev = c.CDLL("./OPSensor/osp_lidar")
T_device_handler = c.c_int
T_device_handler_ptr = c.POINTER(T_device_handler)
_dev.device_add.argtypes = [T_device_handler_ptr, c.c_char_p]
_dev.device_add.restype = c.c_int
device_handler = c.c_int()
ip_val = c.c_char_p("192.168.1.2".encode('utf-8'))
out = MeasureResult_t()
out_num = c.c_int()
CMPFUNC_t = c.CFUNCTYPE(None, c.POINTER(MeasureResult_t), c.c_int)
MeasureModeCallback = CMPFUNC_t(device_ID1_callback)
ret = _dev.device_add(c.byref(device_handler), (ip_val), MeasureModeCallback(c.byref(out), out_num))
_dev.device_get_info.argtypes = [T_device_handler_ptr, c.POINTER(DeviceInfo_t)]
_dev.device_get_info.restype = c.c_void_p # assume it returns C int
p_device_info = DeviceInfo_t()
#_dev.device_get_info(c.byref(device_handler), c.byref(p_device_info)) # does not work
_dev.device_get_info((device_handler), c.byref(p_device_info)) #does not work either
print(device_handler) # I have correct device_handler value
print(p_device_info.fpga_version) # the value i got is 0, does seem right
"""

Here's my attempt at a minimal reproducible example. I implemented dummy functions that demonstrate the callback you described:
// lib.c
#define DLLEXPORT __declspec(dllexport)
#define EXCALL
typedef unsigned int DeviceHandler;
typedef struct {
unsigned int fpga_version;
} DeviceInfo_t;
typedef struct {
unsigned int check_id;
float distance[256];
} MeasureResult_t;
typedef void (*MeasureModeCallback)(MeasureResult_t*, unsigned int);
DLLEXPORT int EXCALL device_add(DeviceHandler* outHandler, char* device_ip, MeasureModeCallback callback) {
*outHandler = 123; // dummy device ID
MeasureResult_t m; // some fake measurement results
m.check_id = 456;
for(int i = 0; i < 256; ++i)
m.distance[i] = (float)(i * .25);
callback(&m, 789); // call the callback
return 1;
}
DLLEXPORT void EXCALL device_get_info(DeviceHandler handler, DeviceInfo_t* p_device_info) {
p_device_info->fpga_version = handler * 2; // fake fpga version
}
To create a callback in Python, assign the CFUNCTYPE prototype to the callback type, decorate the callback function with that type, and use that type in the callback argument definition, and the actual function name when passing it as an argument.
Also note the that float distance[256] is declared in Python as c.c_float * 256 to create an array and the differences in the .argtypes/.restype attributes for the functions. device_get_info takes a DeviceHandler, not a c.POINTER(DeviceHandler) for example.
# test.py
import ctypes as c
class MeasureResult_t(c.Structure):
_fields_ = (('check_id', c.c_uint),
('distance', c.c_float * 256))
def __repr__(self): # defines how to display this class
return f'MeasureResult_t(check_id={self.check_id}, distance=[{self.distance[0]}, ..., {self.distance[255]}])'
class DeviceInfo_t(c.Structure):
_fields_ = ('fpga_version', c.c_uint),
def __repr__(self): # defines how to display this class
return f'DeviceInfo_t(fpga_version={self.fpga_version})'
# Declare the callback type
MeasureModeCallback = c.CFUNCTYPE(None, c.POINTER(MeasureResult_t), c.c_uint)
DeviceHandler = c.c_uint
# apply the decorator so this function can be called from C
#MeasureModeCallback
def device_ID1_callback(out, out_num):
print('---enter device call back function---')
print('out: ',out.contents)
print('out_num:',out_num)
print('---exit device call back function---')
_dev = c.CDLL('./lib')
# Use the argument type
_dev.device_add.argtypes = c.POINTER(DeviceHandler), c.c_char_p, MeasureModeCallback
_dev.device_add.restype = c.c_int
_dev.device_get_info.argtypes = DeviceHandler, c.POINTER(DeviceInfo_t)
_dev.device_get_info.restype = None
device_handler = DeviceHandler()
ip_val = b'192.168.1.2'
# Use the callback function name when calling the function
ret = _dev.device_add(c.byref(device_handler), ip_val, device_ID1_callback)
device_info = DeviceInfo_t()
_dev.device_get_info(device_handler, c.byref(device_info))
print(f'{device_handler.value=}')
print(f'{device_info=}')
Output. Note the classes know how to display themselves and the fake data agrees with my implementation:
---enter device call back function---
out: MeasureResult_t(check_id=456, distance=[0.0, ..., 63.75])
out_num: 789
---exit device call back function---
device_handler.value=123
device_info=DeviceInfo_t(fpga_version=246)

how to pass float* array to C method from python script whose memory allocation takes place in C layer

I am trying to call C methods from Python script, C method calls inturn the C++ method. I am allocating array inside the getResults() method using malloc(). Now the issue is how to pass the arguments to float* oresults in python script whose memory allocation takes place inside the C layer.
This is io.c
int getResults(char* iFilename, char* iStagename, int iStateidCnt,
int* Stateids, int iEntityIdCount, int* iEntityids, char* iEntityType,
char* iVariablegroup, char* ivariable, int *oRescount,
float* oResults)
{
int Status, i;
EString etype(iEntityType), stagename(iStagename);
EString vargroup(iVariablegroup);
std::vector<ERF_INT> entity_ids;
std::vector<ERF_INT> stateids;
std::vector<ERF_FLOAT> results;
_CopyIntArrayIntoVector(iStateidCnt, Stateids, stateids);
_CopyIntArrayIntoVector(iEntityIdCount, iEntityids, entity_ids);
CreateIoInstance(iFilename, iStagename);
ioData pIodata = CreateIoDataInstance();
if (iEntityIdCount <= 0)
pIodata.setWholeSection(true);
else
{
pIodata.setWholeSection(false);
pIodata.setEntityList(entity_ids);
}
pIodata.setStateList(stateids);
pIodata.setType(etype);
pIodata.setVariableGroup(iVariablegroup);
pIodata.setVariable(ivariable);
//This is C++ method
Status = pIo->get_results(pIodata, results);
*oRescount = results.size();
//allocation for oresults whose size > 2
oResults = (float*)malloc(results.size() * sizeof(float));
_CopyVectorIntoDoubleArray(results, oResults);
return Status;
}
test.py
from ctypes import *
import os, sys
dll = CDLL('D:\\erf_utils_python\\erf_utils_io.dll')
dll.getresults.argtypes = (c_char_p,c_char_p,c_int,POINTER(c_int),c_int,POINTER(c_int),c_char_p,
c_char_p,c_char_p,POINTER(c_int),POINTER(c_float))
dll.getresults.restype = c_int
def make_array(ctype,arr):
return len(arr),(ctype * len(arr))(*arr)
def getresults(filename,stagename,sids,eids,entitytype,groups,variables):
if(len(sids)>0):
stateidcount,stateids = make_array(c_int,sids)
if(len(eids)>0):
entityidcount,entityid = make_array(c_int,eids)
oresultlen = c_int()
float_values = POINTER(c_float)
err = dll.getresults(filename,stagename,stateidcount,stateids,entityidcount,entityid,
entitytype,groups,variables,byref(oresultlen), byref(float_values))
return err,oresultlen.value, float_values
filename = b'D:\\inputfile.h5'
stagename = b"post"
stateids = [2]
stateidcount = 1
entityidcount = 1
entityid = [1]
entitytype = b"test"
variablecount = 1
variablegroup = b"testdata"
variable = b"next"
err,oreslen,ores = getresults(filename,stagename,stateids,entityid,entitytype,variablegroup,variable)
TypeError: byref() argument must be a ctypes instance, not
'_ctypes.PyCPointerType' this is the error I get when I run the
script. I am little confused on how to send argument for float
*oresults in script.

In the C++ code, the signature int getResults(..., float* oResults) isn't able to communicate the allocated pointer back to the caller. The line
oResults = (float*)malloc(results.size() * sizeof(float));
sets the oResults pointer locally within getResults, without affecting the caller. In order to output a pointer, you must either return it or use a pointer-to-pointer argument: int getResults(..., float** oResults).
In the Python code, I'm not familiar with ctypes but it looks like float_values = POINTER(c_float) is a problem. POINTER(c_float) creates a Python type for a pointer to float. You'd want POINTER(c_float)() to make an instance of such a pointer (which is initially null).
ctypes documentation on pointers:
https://docs.python.org/3/library/ctypes.html#pointers

The float* oResults parameter is passed by value, so it is impossible to return an allocated pointer in that parameter. Instead, use a float** oResults.
Also, float_values = POINTER(c_float) is a type, not an instance of a type. So byref(float_values) is equivalent to the invalid C &(float*). Instead, you want an instance of a pointer POINTER(c_float)() (note parentheses) and pass that by reference, similar to C float *p; func(&p). This will pass the pointer by address to the C function, which can then modify it as an output parameter.
Here's a simplified example focusing on just the int *oRescount and float** oResults parameters. Also needed is a function to free the allocation:
test.cpp
#include <vector>
#define API __declspec(dllexport)
extern "C" {
API int getResults(size_t *oRescount, float** oResults) {
std::vector<float> results {1.25,2.5,3.75,5.0}; // Simulated results
*oRescount = results.size(); // Return size of results
auto tmp = new float[results.size()]; // allocate
for(size_t i = 0; i < results.size(); ++i) // copy vector to allocation
tmp[i] = results[i];
*oResults = tmp; // return allocation
return 0;
}
API void freeResults(float* oResults) {
delete [] oResults;
}
}
test.py
from ctypes import *
dll = CDLL('./test')
dll.getResults.argtypes = POINTER(c_size_t),POINTER(POINTER(c_float))
dll.getResults.restype = c_int
def getresults():
oRescount = c_size_t() # instance to hold the returned size
oResults = POINTER(c_float)() # instance of a float* to hold the returned allocation.
err = dll.getResults(byref(oRescount), byref(oResults))
# oResults is a float* and it is possible to index past the end.
# Make a copy into a Python list slicing to the correct size,
# then free it so there is no memory leak.
results = oResults[:oRescount.value]
dll.freeResults(oResults)
return err,results
err,ores = getresults()
print(err,ores)
Output:
0 [1.25, 2.5, 3.75, 5.0]

How to get a string from C++ to python when using ctypes and wchar_t?

I can:
Get an integer from C++ and use it in python
Send a python string (as a wchar_t) to C++ and do some logic with it
I cannot
Step 2 in opposite direction.
Here is my C++ code (compiled with clion and cygwin as a shared library using C++14).
#include <iostream>
wchar_t aa[2];
extern "C" {
int DoA()
{
return 10;
}
int DoB(wchar_t * in)
{
if (in[1] == 'a')
{
return 25;
}
return 30;
}
wchar_t * DoC()
{
aa[0] = 'a';
aa[1] = 'b';
return aa;
}
}
Here is my python 3.6.1 code that shows what I can and what I cannot do. So how should I get my string and do things with it in python? I expect to use the address with wstring_at to get the value, but it is not working.
from ctypes import *
import os.path
print('Hello')
itExist = os.path.exists('C:/Users/Daan/CLionProjects/stringproblem/cmake-build-release/cygstringproblem.dll')
print(itExist)
lib = cdll.LoadLibrary('C:/Users/Daan/CLionProjects/stringproblem/cmake-build-release/cygstringproblem.dll')
print('dll loaded')
A = lib.DoA()
print(A)
Bx = lib.DoB(c_wchar_p('aaa'))
print(Bx)
By = lib.DoB(c_wchar_p('bbb'))
print(By)
Ca = lib.DoC()
print(Ca)
print('Issue is coming')
Cb = wstring_at(Ca,2)
print(Cb)
Here is the output with error.
Hello
True
dll loaded
10
25
30
-1659080704
Issue is coming
Traceback (most recent call last):
File "ShowProblem.py", line 19, in <module>
Cb = wstring_at(Ca,2)
File "C:\Users\Daan\AppData\Local\Programs\Python\Python36\lib\ctypes\__init__.py", line 504, in wstring_at
return _wstring_at(ptr, size)
OSError: exception: access violation reading 0xFFFFFFFF9D1C7000

I reproduced your problem on Linux and corrected it by defining the return type from your DoC function:
from ctypes import *
print('Hello')
lib = cdll.LoadLibrary(PATH_TO_TOUR_LIB)
print('dll loaded')
# this line solved the issue for me
lib.DoC.restype = c_wchar_p
A = lib.DoA()
print(A)
Bx = lib.DoB(c_wchar_p('aaa'))
print(Bx)
By = lib.DoB(c_wchar_p('bbb'))
print(By)
Ca = lib.DoC()
print(Ca)
print('Issue is coming')
Cb = wstring_at(Ca,2)
print(Cb)
I also allocated the memory dynamically (some Python expert might comment on this, I guess that this causes a memory leak):
extern "C" {
int DoA()
{
return 10;
}
int DoB(wchar_t * in)
{
if (in[1] == 'a')
{
return 25;
}
return 30;
}
wchar_t * DoC()
{
wchar_t* aa = new wchar_t[2];
aa[0] = 'a';
aa[1] = 'b';
return aa;
}
}
Let me know if it works on Windows.

If you set the .argtypes and .restype of your wrapped functions, you can call them more naturally. To handle an output string, it will be thread safe if you allocate the buffer in Python instead of using a global variable, or just return a wide string constant. Here's an example coded for the Microsoft compiler:
test.c
#include <wchar.h>
#include <string.h>
__declspec(dllexport) int DoA(void) {
return 10;
}
__declspec(dllexport) int DoB(const wchar_t* in) {
if(wcslen(in) > 1 && in[1] == 'a') // Make sure not indexing past the end.
return 25;
return 30;
}
// This version good if variable data is returned.
// Need to pass a buffer of sufficient length.
__declspec(dllexport) int DoC(wchar_t* aa, size_t length) {
if(length < 3)
return 0;
aa[0] = 'a';
aa[1] = 'b';
aa[2] = '\0';
return 1;
}
// Safe to return a constant. No memory leak.
__declspec(dllexport) wchar_t* DoD(void) {
return L"abcdefg";
}
test.py
from ctypes import *
# Set up the arguments and return type
lib = CDLL('test')
lib.DoA.argtypes = None
lib.DoA.restype = c_int # default, but just to be thorough.
lib.DoB.argtypes = [c_wchar_p]
lib.DoB.restype = c_int
lib.DoC.argtypes = [c_wchar_p,c_size_t]
lib.DoC.restype = c_int
lib.DoD.argtypes = None
lib.DoD.restype = c_wchar_p
# Map to local namespace functions
DoA = lib.DoA
DoB = lib.DoB
DoD = lib.DoD
# Do some pre- and post-processing to hide the memory details.
def DoC():
tmp = create_unicode_buffer(3) # Writable array of wchar_t.
lib.DoC(tmp,sizeof(tmp))
return tmp.value # return a Python string instead of the ctypes array.
print(DoA())
print(DoB('aaa'))
print(DoB('bbb'))
print(DoC())
print(DoD())
Output:
10
25
30
ab
abcdefg