We Keep Coding

CPPYY/CTYPES passing array of strings as char* args[]

I only recently started using cppyy and ctypes, so this may be a bit of a silly question. I have the following C++ function:
float method(const char* args[]) {
...
}
and from Python I want to pass args as a list of strings, i.e.:
args = *magic*
x = cppyy.gbl.method(args)
I have previously found this, so I used
def setParameters(strParamList):
numParams = len(strParamList)
strArrayType = ct.c_char_p * numParams
strArray = strArrayType()
for i, param in enumerate(strParamList):
strArray[i] = param
lib.SetParams(numParams, strArray)
and from Python:
args = setParameters([b'hello', b'world'])
c_types.c_char_p expects a bytes array. However, when calling x = cppyy.gbl.method(args) I get
TypeError: could not convert argument 1 (could not convert argument to buffer or nullptr)
I'm not entirely sure why this would be wrong since the args is a <__main__.c_char_p_Array_2> object, which I believe should be converted to a const char* args[].

For the sake of having a concrete example, I'll use this as the .cpp file:
#include <cstdlib>
extern "C"
float method(const char* args[]) {
float sum = 0.0f;
const char **p = args;
while(*p) {
sum += std::atof(*p++);
}
return sum;
}
And I'll assume it was compiled with g++ method.cpp -fPIC -shared -o method.so. Given those assumptions, here's an example of how you could use it from Python:
#!/usr/bin/env python3
from ctypes import *
lib = CDLL("./method.so")
lib.method.restype = c_float
lib.method.argtypes = (POINTER(c_char_p),)
def method(args):
return lib.method((c_char_p * (len(args) + 1))(*args))
print(method([b'1.23', b'45.6']))
We make a C array to hold the Python arguments. len(args) + 1 makes sure there's room for the null pointer sentinel.

ctypes does not have a public API that is usable from C/C++ for extension writers, so the handling of ctypes by cppyy is by necessity somewhat clunky. What's going wrong, is that the generated ctypes array of const char* is of type const char*[2] not const char*[] and since cppyy does a direct type match for ctypes types, that fails.
As-is, some code somewhere needs to do a conversion of the Python strings to low-level C ones, and hold on to that memory for the duration of the call. Me, personally, I'd use a little C++ wrapper, rather than having to think things through on the Python side. The point being that an std::vector<std::string> can deal with the necessary conversions (so no bytes type needed, for example, but of course allowed if you want to) and it can hold the temporary memory.
So, if you're given some 3rd party interface like this (putting it inline for cppyy only for the sake of the example):
import cppyy
cppyy.cppdef("""
float method(const char* args[], int len) {
for (int i = 0; i < len; ++i)
std::cerr << args[i] << " ";
std::cerr << std::endl;
return 42.f;
}
""")
Then I'd generate a wrapper:
# write a C++ wrapper to hide C code
cppyy.cppdef("""
namespace MyCppAPI {
float method(const std::vector<std::string>& args) {
std::vector<const char*> v;
v.reserve(args.size());
for (auto& s : args) v.push_back(s.c_str());
return ::method(v.data(), v.size());
}
}
""")
Then replace the original C API with the C++ version:
# replace C version with C++ one for all Python users
cppyy.gbl.method = cppyy.gbl.MyCppAPI.method
and things will be as expected for any other person downstream:
# now use it as expected
cppyy.gbl.method(["aap", "noot", "mies"])
All that said, obviously there is no reason why cppyy couldn't do this bit of wrapping automatically. I created this issue: https://bitbucket.org/wlav/cppyy/issues/235/automatically-convert-python-tuple-of

Cython - Passing variable number of arguments from Python to C ellipsis [duplicate]

Cython equivalent of c define
#define myfunc(Node x,...) SetNode(x.getattributeNode(),__VA_ARGS__)
I have a c api SetNode which takes first argument a node of struct type node and N variables (N is variable number from 0-N)
here is a c example to solve such problum
exampleAPI.c
#include<stdarg.h>
float sumN(int len,...){
va_list argp;
int i;
float s=0;
va_start(argp,len);
for(i=0;i<len;i++){
s+=va_arg(argp,int);
}
va_end(argp);
}
exampleAPI.h
#include<stdarg.h>
float sumN(int len,...)
examplecode.c
#include<stdarg.h>
#include"exampleAPI.h"
int len(float first,...){
va_list argp;
int i=1;
va_start(argp,first);
while(1){
if(va_arg(argp,float)==NULL){
return i
}
else{
i++;
}
}
va_end(argp);
}
#define sum(...) sumN(len(__VA_ARGS__),__VA_ARGS__)
Now calling
sum(1,2,3,4);
will return 10.000000
sum(1.5,6.5);
will return 8.00000
I need a cython alternative for bellow c definition and not above example
because I have a C-API which has SetNode function which takes variable number of arguments and I want to wrap it in cython and call from python
#define myfunc(Node x,...) SetNode(x.getattributeNode(),__VA_ARGS__)
here Node is a class defined in cython which holds a c stuct as attribute and getattributeNode() is a function of Node class which returns c struct that needs to be passed into C-API.
cdef extern "Network.h":
ctypedef struct node_bn:
pass
node_bn* SetNode(node_bn* node,...)
cdef class Node:
cdef node_bn *node
cdef getattributeNode(self):
return self.node
def setNode(self,*arg):
self.node=SetNode(self.node,*arg) # Error cannot convert python objects to c type
Alternative thing I tried
cdef extern from "stdarg.h":
ctypedef struct va_list:
pass
ctypedef struct fake_type:
pass
void va_start(va_list, void* arg)
void* va_arg(va_list, fake_type)
void va_end(va_list)
fake_type int_type "int"
cdef extern "Network.h":
ctypedef struct node_bn:
pass
node_bn* VSetNode(node_bn* node,va_list argp)
cdef class Node:
cdef node_bn *node
cdef getattributeNode(self):
return self.node
cpdef _setNode(self,node_bn *node,...):
cdef va_list agrp
va_start(va_list, node)
self.node=VSetNode(node,argp)
va_end(va_list)
def setNode(self,*arg):
self._setNode(self.node,*arg)
works fine when argument list is empty
n = Node()
n.setNode() #This works
n.SetNode("top",1) # error takes exactly one argument given 3 in self._setNode(self.node,*arg)
If anyone could suggest cython equivalent to it, it would be great.

I don't think it's easily done though Cython (the problem is telling Cython what type conversions to do for an arbitrary number of arguments). The best I can suggest is to use the standard library ctypes library for this specific case and wrap the rest in Cython.
For the sake of an example, I've used a very simple sum function. va_sum.h contains:
typedef struct { double val; } node_bn;
node_bn* sum_va(node_bn* node,int len, ...);
/* on windows this must be:
__declspec(dllexport) node_bn* sum_va(node_bn* node,int len, ...);
*/
and va_sum.c contains:
#include <stdarg.h>
#include "va_sum.h"
node_bn* sum_va(node_bn* node,int len, ...) {
int i;
va_list vl;
va_start(vl,len);
for (i=0; i<len; ++i) {
node->val += va_arg(vl,double);
}
va_end(vl);
return node;
}
I've written it so it adds everything to a field in a structure just to demonstrate that you can pass pointers to structures without too much trouble.
The Cython file is:
# definition of a structure
cdef extern from "va_sum.h":
ctypedef struct node_bn:
double val;
# obviously you'll want to wrap things in terms of Python accessible classes, but this atleast demonstrates how it works
def test_sum(*args):
cdef node_bn input_node;
cdef node_bn* output_node_p;
input_node.val = 5.0 # create a node, and set an initial value
from ctypes import CDLL, c_double,c_void_p
import os.path
# load the Cython library with ctypes to gain access to the "sum_va" function
# Assuming you've linked it in when you build the Cython module
full_path = os.path.realpath(__file__)
this_file_library = CDLL(full_path)
# I treat all my arguments as doubles - you may need to do
# something more sophisticated, but the idea is the same:
# convert them to the c-type the function is expecting
args = [ c_double(arg) for arg in args ]
sum_va = this_file_library.sum_va
sum_va.restype = c_void_p # it returns a pointer
# pass the pointers as a void pointer
# my c compiler warns me if I used int instead of long
# but which integer type you have to use is likely system dependent
# and somewhere you have to be careful
output_node_p_as_integer = sum_va(c_void_p(<long>&input_node),len(args),
*args)
# unfortunately getting the output needs a two stage typecast - first to a long, then to a pointer
output_node_p = <node_bn*>(<long>(output_node_p_as_integer))
return output_node_p.val
You need to compile your va_sum.c together with your Cython file (e.g. by adding sources = ['cython_file.pyx','va_sum.c'] in setup.py)
Ctypes is probably a bit slower than Cython (I think there's a reasonable overhead on each call), and it's odd to mix them, but this should at least let you write the main wrapper in Cython, and use ctypes to get round the specific limitation.

This is probably not the proper answer, since I am not sure I understand the question fully. I would have replied in a comment, but the code formatting is too poor.
In Python the functions sum and len are available:
def my_len(*args):
return len(args)
def my_sum(*args):
return sum(args)
print "len =", my_len("hello", 123, "there")
print "sum =", my_sum(6.5, 1.5, 2.0)
outputs:
len = 3
sum = 10.0

Convert Python object to C void type

How can I convert Python object to C void type using Cython?
Currently I am getting this message when I try to cast
Casting temporary Python object to non-numeric non-Python type

This can be done like this :
1. Cast from Python to C
If you really meant void * this would be :
some_pyobj = "abc"
cdef void *ptr
ptr = <void *>some_pyobj
If you meant PyObject * this would be :
cdef PyObject *ptr
ptr = <PyObject *>some_pyobj # Cast from Python object to C pointer
Then, from C side, the PyObject struct is available by including Python.h.
Here is the reference (from object.h Python include file) :
/* Nothing is actually declared to be a PyObject, but every pointer to
* a Python object can be cast to a PyObject*. This is inheritance built
* by hand. Similarly every pointer to a variable-size Python object can,
* in addition, be cast to PyVarObject*.
*/
typedef struct _object {
PyObject_HEAD
} PyObject;
2. Cast from C to Python
It works in both ways, meaning that the following is also possible :
cdef PyObject *ptr
ptr = <PyObject *>some_pyobj
cdef object some_other_pyobj
some_other_pyobj = <object>ptr # Cast from C pointer to Python object

You can:
def run():
pyobj = 'abc'
pyobj_void_star = <void *>pyobj

How to register a destructor for a C-allocated numpy array?

I want to allocate numbers for a numpy array in C/C++, and pass them to python as a numpy array. That I can do with PyArray_SimpleNewFromData.
The problem is that I also want to register a function that should be invoked from Python when the numpy array reference counter reaches zero, and that would invoke some destructor semantics on the C side... Here is a pseudo-example of what I need:
float* arr; PyObject* np_arr; void (*destructor)(float* arr);
// ... C-allocate array on arr, ...
// ...
// ... initialize destructor with some suitable value, and then:
np_arr = /* ... create the array to wrap arr,
and to use destructor on some meaningful way ... */
is there a simple way of doing so?

The idea is to create a Python object that knows how to free your memory when destroyed and make it the base of the returned C-allocated numpy array. This sounds tricky but it can be easily achieved via something known as capsules in python. Let me give an example,
Suppose that you have the following code,
PyObject *arr;
int nd = 2;
npy_intp dims[] = {5, 10};
double *data = some_function_that_returns_a_double_star(x, y, z);
arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
return arr;
There is an obvious memory leak here since you cannot free data until arr is deleted as it says here in the red warning box. Fixing this, on the other hand, is easy. Define a function which is basically a destructor function that knows how to do garbage collection.
void capsule_cleanup(PyObject *capsule) {
void *memory = PyCapsule_GetPointer(capsule, NULL);
// Use your specific gc implementation in place of free if you have to
free(memory);
}
Now augment your code as,
PyObject *arr;
int nd = 2;
npy_intp dims[] = {5, 10};
double *data = some_function_that_returns_a_double_star(x, y, z);
arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, (void *)data);
PyObject *capsule = PyCapsule_New(data, NULL, capsule_cleanup);
// NULL can be a string but use the same string while calling PyCapsule_GetPointer inside capsule_cleanup
PyArray_SetBaseObject((PyArrayObject *) arr, capsule);
return arr;
There is no need to Py_DECREF the capsule. The function PyArray_SetBaseObject steals reference.
Hope this helps!

Writing a Python C extension: how to correctly load a PyListObject?

While attempting to read a Python list filled with float numbers and to populate real channels[7] with their values (I'm using F2C, so real is just a typedef for float), all I am able to retrieve from it are zero values. Can you point out the error in the code below?
static PyObject *orbital_spectra(PyObject *self, PyObject *args) {
PyListObject *input = (PyListObject*)PyList_New(0);
real channels[7], coefficients[7], values[240];
int i;
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &input)) {
return NULL;
}
for (i = 0; i < PyList_Size(input); i++) {
printf("%f\n", PyList_GetItem(input, (Py_ssize_t)i)); // <--- Prints zeros
}
//....
}

PyList_GetItem will return a PyObject*. You need to convert that to a number C understands. Try changing your code to this:
printf("%f\n", PyFloat_AsDouble(PyList_GetItem(input, (Py_ssize_t)i)));

Few things I see in this code.
You leak a reference, don't create that empty list at the beginning, it's not needed.
You don't need to cast to PyListObject.
PyList_GetItem returns a PyObject, not a float. Use PyFloat_AsDouble to extract the value.
If PyList_GetItem returns NULL, then an exception has been thrown, and you should check for it.