We Keep Coding

Python C++ API function with multiple arguments

I'm trying to create a python module using my C++ code and I want to declare a function with multiple arguments. (3 in this case) I've read the docs and it says that I must declare METH_VARARGS which I did, but I think I also must change something inside my function to actually receive the arguments. Otherwise it gives me "too many arguments" error when I use my function in python.
Here is the code snippet I'm using:
...
// This function can be called inside a python file.
static PyObject *
call_opencl(PyObject *self, PyObject *args)
{
const char *command;
int sts;
// We except at least one argument to this function
// Not sure how to accept more than one.
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
OpenCL kernel = OpenCL();
kernel.init();
std::cout << "This message is called from our C code: " << std::string(command) << std::endl;
sts = 21;
return PyLong_FromLong(sts);
}
static PyMethodDef NervebloxMethods[] = {
{"call_kernel", call_opencl, METH_VARARGS, "Creates an opencv instance."},
{NULL, NULL, 0, NULL} /* Sentinel */
};
...

You are still expecting one argument.
if (!PyArg_ParseTuple(args, "s", &command))
the documentation defines how you can expect optional or additional arguments, for example "s|dd" will expect a string and two optional numbers, you still have to pass two doubles to the function for when the numbers are available.
double a = 0; // initial value
double b = 0;
if (!PyArg_ParseTuple(args, "s|dd", &command, &a, &b))

pybind how can I operate over a py::list object

For a better understanding of how to pass arguments from Python to C++ functions with the pybind library, I wanted to build a small dummy/demo code where I could receive a Python list on the C++ side, cast it to a float pointer object, and then print it.
Though I know I can use the py::list class I haven't figured out the methods available of this class. I looked in the documentation reference, and then in the code (list.h, stl.h) and couldn't figure out which methods where available.
What is the equivalent of __getitem__? Do I have every python method available for py::list?

The code you are looking for is here:
class list : public object {
public:
PYBIND11_OBJECT_CVT(list, object, PyList_Check, PySequence_List)
explicit list(size_t size = 0) : object(PyList_New((ssize_t) size), stolen_t{}) {
if (!m_ptr) pybind11_fail("Could not allocate list object!");
}
size_t size() const { return (size_t) PyList_Size(m_ptr); }
detail::list_accessor operator[](size_t index) const { return {*this, index}; }
detail::list_iterator begin() const { return {*this, 0}; }
detail::list_iterator end() const { return {*this, PyList_GET_SIZE(m_ptr)}; }
template <typename T> void append(T &&val) const {
PyList_Append(m_ptr, detail::object_or_cast(std::forward<T>(val)).ptr());
}
};
Also keep in mind that py::list inherits from py::object, which in turn inherits from py::handle (this also means that you are passing by reference). In my experience, there is very little documentation for this kind of usage, and reading the code is your best bet.
We can see from the class definition that we can use the member functions size, operator[], begin, end (C++ iterators!) and append (templated!). If this is not enough, you can use attr to access python attributes (including methods). Here is an example:
Python code (some_python.py):
import cppimport
cpp = cppimport.imp("some_cpp")
l = [1,2,3,4,5]
cpp.test(l)
print('after C++', l)
cpp.float_cast(l)
C++ code (some_cpp.cpp):
/* <%
setup_pybind11(cfg)
%> */
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
namespace py = pybind11;
void test(py::list l) {
l.attr("pop")();
std::cout << "List has length " << l.size() << std::endl;
for (py::handle obj : l) { // iterators!
std::cout << " - " << obj.attr("__str__")().cast<std::string>() << std::endl;
}
l.append(10); // automatic casting (through templating)!
}
void float_cast(py::list l) {
float f = l.cast<float>();
}
PYBIND11_MODULE(some_cpp, m) {
m.def("test", &test);
m.def("float_cast", &float_cast);
}
Output:
List has length 4
- 1
- 2
- 3
- 4
after C++ [1, 2, 3, 4, 10]
Traceback (most recent call last):
File "some_python.py", line 9, in <module>
cpp.float_cast(l)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
As you can see I've also included your specific question of casting to a float. Here I've used the cast method of py::handle, which gives a nice exception. You may try to "directly" cast the object (something like float* f = (float*) &l;) but this will give you garbage and I guess this is not what you're looking for.
One more remark: pybind/stl.h enables conversion between Python's standard types and C++ versions. For example, a list may be converted to a std::vector<int>, including typechecks. An important effect of this is that the data is passed as a copy instead of as a reference.

How can one use PyArg_ParseTupleAndKeywords to parse a tuple with optional arguments as well as keywords?

I've looked everywhere, but I can't find an example of PyArg_ParseTupleAndKeywords() being used with a tuple — containing optional arguments — and keywords. The closest I've found is this question, but the answer isn't particularly helpful. Most examples seem to have the keywords as the optional arguments, but it seems like the tuple should be able to contain optional arguments as well.
Suppose I'm trying to parse the following parameters:
numpy array of doubles (mandatory)
numpy array of doubles (optional, no keyword)
optional keyword arguments:
k1 => numpy array of doubles
k2 => integer
k3 => double
k4 => Python class instance
It seems like I should be doing something like
static PyObject* pymod_func(PyObject* self, PyObject* args, PyObject* kwargs) {
static char* keywords[] = {"k1", "k2", "k3", "k4", NULL};
PyObject *arg1, *arg2, *k1, *k4
PyObject *arr1, *arr2, *karr1;
double *k3;
int *k2;
PyArg_ParseTupleAndKeywords(args, kwargs, "O!|O!OidO", keywords, &arg1, &PyArray_Type, &arg2, &PyArray_Type, &k1, &PyArray_Type, &k2, &k3, &k4);
arr1 = PyArray_FROM_OTF(arg1, NPY_FLOAT64, NPY_ARRAY_INOUT_ARRAY);
if (arr1 == NULL) return NULL;
arr2 = PyArray_FROM_OTF(arg1, NPY_FLOAT64, NPY_ARRAY_INOUT_ARRAY);
// no null check, because optional
karr1 = PyArray_FROM_OTF(k1, NPY_FLOAT64, NPY_ARRAY_INOUT_ARRAY);
// again, no null check, because this is optional
// do things with k3, k2, and k4 also
return NULL;
}
Other places I've looked, but without finding much help:
https://docs.python.org/2/extending/extending.html
https://docs.python.org/2/c-api/arg.html
http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.how-to-extend.html
What is the appropriate way to use PyArg_ParseTupleAndKeywords()?

As of Python 3.3, you can use $ in the format string to indicate that the rest of the arguments are keyword-only, and as of Python 3.6, you can indicate a positional-only parameter by using an empty name in the keywords argument.
So, in a sufficiently high version of Python, you can use code like:
static char* keywords[] = {"", "", "k1", "k2", "k3", "k4", NULL};
// [...]
PyArg_ParseTupleAndKeywords(args, kwargs,
"O!|O!$O!idO", keywords,
&PyArray_Type, &arg1, &PyArray_Type, &arg2,
&PyArray_Type, &k1, &k2, &k3, &k4);

I think something similar to the below simplified solution should work for your scenario but it can get nasty if you have many optional args or more interesting arg types.
I'm not sure if there is a better solution but I haven't been able to find one. Hopefully one day someone will post a cleaner solution.
You'll have to be clever to produce useful arg parsing error messages
in more complex parsing scenarios.
static PyObject* nasty_func(PyObject* self, PyObject* args, PyObject* kwargs) {
static char* keywords[] = {"one", "optional", "two", NULL};
static char* keywords_alt[] = {"one", "two", NULL};
int ok = 0;
PyObject *result = NULL;
int *one;
char *two;
int *optional;
ok = PyArg_ParseTupleAndKeywords(args, kwargs, "iis", keywords, &one, &optional, &two);
if (!ok) {
PyErr_Clear();
ok = PyArg_ParseTupleAndKeywords(args, kwargs, "is", keywords_alt, &one, &two);
if (!ok) {
PyErr_SetString(PyExc_TypeError, "Invalid args. allowed formats: 'one:i, two:s' or 'one:i, optional:i, two:s'");
return NULL;
}
}
// do stuff with your parsed variables
return result;
}

What's the difference between tp_clear, tp_dealloc and tp_free?

I have a custom python module for fuzzy string search, implementing Levenshtein distance calculation, it contains a python type, called levtree which has two members a pointer to a wlevtree C type (called tree) which does all the calculations and a PyObject* pointing to a python-list of python-strings, called wordlist. Here is what I need:
-when I create a new instance of levtree I use a constructor which takes a tuple of strings as its only input (and it is the dictionary in which the instance will perform all the searches), this constructor will have to create a new instance of wordlist into the new instance of levtree and copy the content of the input tuple into the new instance of wordlist. Here is my first code snippet and my first question:
static int
wlevtree_python_init(wlevtree_wlevtree_obj *self, PyObject *args, PyObject *kwds)
{
int numLines; /* how many lines we passed for parsing */
wchar_t** carg; /* argument to pass to the C function*/
unsigned i;
PyObject * strObj; /* one string in the list */
PyObject* intuple;
/* the O! parses for a Python object (listObj) checked
to be of type PyList_Type */
if (!(PyArg_ParseTuple(args, "O!", &PyTuple_Type, &intuple)))
{
return -1;
}
/* get the number of lines passed to us */
numLines = PyTuple_Size(intuple);
carg = malloc(sizeof(char*)*numLines);
/* should raise an error here. */
if (numLines < 0)
{
return -1; /* Not a list */
}
self->wordlist = PyList_New(numLines);
Py_IncRef(self->wordlist);
for(i=0; i<numLines; i++)
{
strObj = PyTuple_GetItem(intuple, i);
//PyList_Append(self->wordlist, string);
PyList_SetItem(self->wordlist, i, strObj);
Py_IncRef(strObj);
}
/* iterate over items of the list, grabbing strings, and parsing
for numbers */
for (i=0; i<numLines; i++)
{
/* grab the string object from the next element of the list */
strObj = PyList_GetItem(self->wordlist, i); /* Can't fail */
/* make it a string */
if(PyUnicode_Check(strObj))
{
carg[i] = PyUnicode_AsUnicode( strObj );
if(PyErr_Occurred())
{
return -1;
}
}
else
{
strObj = PyUnicode_FromEncodedObject(strObj,NULL,NULL);
if(PyErr_Occurred())
{
return -1;
}
carg[i] = PyUnicode_AsUnicode( strObj );
}
}
self->tree = (wlevtree*) malloc(sizeof(wlevtree));
wlevtree_init(self->tree,carg,numLines);
free(carg);
return 0;
}
Do I have to call Py_IncRef(self->wordlist); after self->wordlist = PyList_New(numLines); or it is redundant because references are already incremented in PyList_new?
Then I have the same doubt on PyList_SetItem(self->wordlist, i, strObj); and Py_IncRef(strObj);..
-when I destroy an instance of levtree i want to call the C function that frees the space occupied by tree, destroy wordlist and decrement all reference count on all the strings contained into wordlist.. Here is my tp_dealloc:
static void
wlevtree_dealloc(wlevtree_wlevtree_obj* self)
{
//wlevtree_clear(self);
if(self->tree!=NULL)
{
wlevtree_free(self->tree);
}
free(self->tree);
PyObject *tmp, *strObj;
unsigned i;
int size = PyList_Size(self->wordlist);
for(i=0; i<size; i++)
{
strObj = PyList_GetItem(self->wordlist, i);
Py_CLEAR(strObj);
}
Py_CLEAR(self->wordlist);
Py_TYPE(self)->tp_free((PyObject *)self);
}
Is it correct to make all the deallocation work here?
At the moment I don't have a tp_clear and a tp_free, do I need them?
My code at the moment works on allocation but not on deallocation because even though I can call init on the same python variable more than once, at the end of every python script (which works correctly) I get a "Segmentation Fault" which makes me think that something in the deallocation process goes wrong..

tp_clear is only needed if you implement cyclic garbage collection. It appears that this is not needed because you only maintain references to Python unicode objects.
tp_dealloc is called when the reference count of the object goes down to zero. This is where you destroy the object and its members. It should then free the memory occupied by the object by calling tp_free.
tp_free is where the memory for the object is freed. Implement this only if you implement tp_alloc yourself.
The reason for the separation between tp_dealloc and tp_free is that if your type is subclassed, then only the subclass knows how the memory was allocated and how to properly free the memory.
If your type is a subclass of an exisiting type, your tp_dealloc may need to call the tp_dealloc of the derived class, but that depends on the details of the case.
To summarize, it seems that you are handling object destruction correctly (except that you leak carg when exiting the function with an error).

Writing a Python C extension: how to correctly load a PyListObject?

While attempting to read a Python list filled with float numbers and to populate real channels[7] with their values (I'm using F2C, so real is just a typedef for float), all I am able to retrieve from it are zero values. Can you point out the error in the code below?
static PyObject *orbital_spectra(PyObject *self, PyObject *args) {
PyListObject *input = (PyListObject*)PyList_New(0);
real channels[7], coefficients[7], values[240];
int i;
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &input)) {
return NULL;
}
for (i = 0; i < PyList_Size(input); i++) {
printf("%f\n", PyList_GetItem(input, (Py_ssize_t)i)); // <--- Prints zeros
}
//....
}

PyList_GetItem will return a PyObject*. You need to convert that to a number C understands. Try changing your code to this:
printf("%f\n", PyFloat_AsDouble(PyList_GetItem(input, (Py_ssize_t)i)));

Few things I see in this code.
You leak a reference, don't create that empty list at the beginning, it's not needed.
You don't need to cast to PyListObject.
PyList_GetItem returns a PyObject, not a float. Use PyFloat_AsDouble to extract the value.
If PyList_GetItem returns NULL, then an exception has been thrown, and you should check for it.