We Keep Coding

Python C api - function overloading

I have a number of C functions that accept different arguments, e.g.
foo_i(int a)
foo_c(char c)
Is it possible to overload these functions in python C api?
I tried to use the following methods table:
static PyMethodDef test_methods[] = {
{"foo", (PyCFunction)foo_i, METH_VARARGS, "int"},
{"foo", (PyCFunction)foo_c, METH_VARARGS, "char"},
{NULL, NULL, 0, NULL}
};
But when I invoke foo from python I always end up using the function at the bottom of the table.
Any ideas on how to invoke both foo_i() and foo_c() using foo() in python C-api?
Thanks!

Either give them different Python level names, or write a single wrapper function that type checks the argument provided and dispatches to the correct "real" function. Python itself has no direct support for overloading functions based on argument types.
If you want the wrapper written for you, you might take a look at pybind11, which does allow overloading in the sense you're attempting (it does so via a type checking wrapper under the hood, so it's just syntactic sugar, not a change in behavior).
Untested example code:
static PyObject*
foo_wrapper(PyObject *self, PyObject *arg)
{
Py_buffer view;
Py_ssize_t ival;
// Check for/handle length 1 bytes-like object (bytes, bytearray, small mmap, etc.)
if (PyObject_GetBuffer(arg, &view, PyBUF_SIMPLE) == 0) {
if (view.len != 1) {
PyErr_Format(PyExc_ValueError, "Must receive exactly one byte, got %zd", view.len);
PyBuffer_Release(&view);
return NULL;
}
foo_c(((char*)view.buf)[0]);
Py_RETURN_NONE; // Or convert return from foo_c if it exists
}
// Check for/handle integer-like object that fits in C int
PyErr_Clear(); // Ignore error for objects not supporting buffer protocol
ival = PyNumber_AsSsize_t(arg, PyExc_ValueError);
if (PyErr_Occurred()) {
if (PyErr_ExceptionMatches(PyExc_TypeError)) {
// Replace with general error message about both accepted argument types,
// since only reporting error from int conversion might confuse folks
PyErr_Format(PyExc_TypeError, "Argument must be length 1 bytes-like object or integer; received %R", Py_TYPE(arg));
}
return NULL;
}
// Check valid range (Py_ssize_t often larger than int)
if (ival < INT_MIN or ival > INT_MAX) {
return PyErr_Format(PyExc_ValueError, "Integer must be in range [%d-%d]; received %zd", INT_MIN, INT_MAX, ival);
}
foo_i((int)ival);
Py_RETURN_NONE; // Or convert return from foo_i if it exists
}
static PyMethodDef test_methods[] = {
{"foo", (PyCFunction)foo_wrapper, METH_O, "Wrapper for foo_c and foo_i"},
{NULL, NULL, 0, NULL}
};

How to write auto-populate the parameter list of a class member method into a Variadic arguments?

I recently wrote an extension to Python 3 in C++, but I encountered some trouble when I called C++ in python.
I don't know how to encapsulate the code below, without the need to write callback function repeated every time?
I am considering binding a callback function and parameter list in some form, but I don't know how to do this.
Here is my main code:
class TestClass
{
PyObject_HEAD
public:
int add(int a, int b) {
return (a + b);
}
};
// ... Ignore some details here ...
static PyObject* function1(TestClass *self, PyObject* args) {
// How to changed the following code to take the value from PyArg_ParseTuple
// through the binding function and parameter list?
int a, b;
if (!PyArg_ParseTuple(args, "ii", &a, &b))
{
return nullptr;
}
// How to changed the following code to the return type of the binding function?
return Py_BuildValue("i", self->add(a, b));
}
Is it possible to implement a call like BINDING_FUNCTION(TestClass::add); in some way?

Yes, this is possible, but it takes a nontrivial amount of code. Too much, I think, to easily fit in an answer here.
The good news is that others have already written the code you need and put them in libraries. The ones I know are pybind11 (modern) and boost.python (somewhat older)
You could either look how these libraries do this, or actually use these libraries.
Approximate steps required if you really want to do this yourself:
Create generic functions to create format strings from function argument types and return type
Create template meta function to define tuple type to hold the arguments of a given (member-) function
Implement generic function that can fill such a tuple using PyArg_ParseTuple(). This would use the format-string-generating function and the tuple definition
implement a generic function that takes an object, a (member-) function pointer and a tuple with arguments and that will call the member function with the given arguments. SO has answers on how to do this.
Typically, you'd want to specialize that last function for void return type functions and you might want to write some extra code if you want to support free functions in addition to member functions.
The code below is for educational purposes only because it's a very much simplified implementation of the steps described above. The libraries mentioned before implement the many corner cases that this code does not address.
#include <iostream>
#include <sstream>
#include <type_traits>
#include <tuple>
// some object we want to wrap
struct Duck
{
int MemberFunc( int x, int y, float w)
{
std::cout << "Member function called\n";
return x+ w * y;
}
};
// PART 1: create format strings for function argument- and return types
// "excercise for the reader": implement these overloads for all supported types
template<typename T> struct Tag{};
const char *GetTypeFormat( const Tag<int>&)
{
return "i";
}
const char *GetTypeFormat( const Tag<float>&)
{
return "f";
}
// create a format string from a list of argument types
template< typename... Args>
void GetTypeFormats( std::ostream &strm)
{
(void)(int[]){0, ((strm << GetTypeFormat(Tag<Args>{})),0)...};
}
// this is quite inefficient because it creates the format string at
// run-time. Doing this as constexpr is an interesting challenge
// ("...for the reader")
template< typename R, typename Class, typename... Args>
std::string GetArgumentFormats( R (Class::*f)(Args...))
{
std::stringstream strm;
GetTypeFormats<Args...>( strm);
return strm.str();
}
template< typename R, typename Class, typename... Args>
std::string GetReturnFormat( R (Class::*f)(Args...))
{
std::stringstream strm;
GetTypeFormats<R>( strm);
return strm.str();
}
// PART 2: declare std::tuple-type to hold function arguments
// given a list of types that could be function parameter types, define a
// tuple type that can hold all argument values to such a function
// THIS IS VERY MUCH A SIMPLIFIED IMPLEMENTATION
// This doesn't take pointer types into account for instance.
template< typename F>
struct ArgumentTuple {};
template< typename R, typename Class, typename... Args>
struct ArgumentTuple<R (Class::*)( Args...)>
{
using type = std::tuple<
typename std::remove_cv<
typename std::remove_reference<Args>::type>::type...>;
};
// for demo purposes. emulate python binding functions
using PyObject = void;
bool PyArg_ParseTuple( PyObject *, const char *, ...) {}
template< typename T>
PyObject *Py_BuildValue( const char*, T ){}
// PART 3: given some function pointer, obtain arguments from a PyObject
template<typename F, size_t... Indexes>
auto FillTuple( PyObject *obj, F f, std::index_sequence<Indexes...>) -> typename ArgumentTuple<F>::type
{
using std::get;
typename ArgumentTuple<F>::type arguments;
// no error checking whatsoever: "exercise for the reader"
PyArg_ParseTuple( obj, GetArgumentFormats( f).c_str(), &get<Indexes>( arguments)...);
return arguments;
}
template< typename R, typename Class, typename... Args>
auto FillTuple( PyObject *obj, R (Class::*f)(Args...))
{
return FillTuple( obj, f, std::index_sequence_for<Args...>{});
}
// PART 4, call a member function given a tuple of arguments
// helper function
template<
typename R,
typename Class,
typename MF,
typename ArgumentTuple,
size_t... Indexes>
R Apply( Class &obj, MF f, ArgumentTuple &args, std::index_sequence<Indexes...>)
{
using std::get;
return (obj.*f)( get<Indexes>( args)...);
}
// Apply a (member-) function to a tuple of arguments.
template<
typename R,
typename Class,
typename ArgumentTuple,
typename... Args>
R Apply( Class &obj, R (Class::*f)( Args...), ArgumentTuple &args)
{
return Apply<R>( obj, f, args, std::index_sequence_for<Args...>{});
}
// LAST PART: glue everything together in a single function.
#define BIND_MEMBER_FUNCTION( class_, memberfunc_) \
PyObject *Call##class_##memberfunc_( class_ *self, PyObject *args)\
{ \
/* no error checking whatsoever: "exercise for the reader"*/\
auto arguments = FillTuple( args, &class_::memberfunc_); \
/* deal with void-returning functions: yet another EFTR */ \
return Py_BuildValue( \
GetReturnFormat( &class_::memberfunc_).c_str(), \
Apply( *self, &class_::memberfunc_, arguments)); \
} \
/**/
BIND_MEMBER_FUNCTION( Duck, MemberFunc);

pybind how can I operate over a py::list object

For a better understanding of how to pass arguments from Python to C++ functions with the pybind library, I wanted to build a small dummy/demo code where I could receive a Python list on the C++ side, cast it to a float pointer object, and then print it.
Though I know I can use the py::list class I haven't figured out the methods available of this class. I looked in the documentation reference, and then in the code (list.h, stl.h) and couldn't figure out which methods where available.
What is the equivalent of __getitem__? Do I have every python method available for py::list?

The code you are looking for is here:
class list : public object {
public:
PYBIND11_OBJECT_CVT(list, object, PyList_Check, PySequence_List)
explicit list(size_t size = 0) : object(PyList_New((ssize_t) size), stolen_t{}) {
if (!m_ptr) pybind11_fail("Could not allocate list object!");
}
size_t size() const { return (size_t) PyList_Size(m_ptr); }
detail::list_accessor operator[](size_t index) const { return {*this, index}; }
detail::list_iterator begin() const { return {*this, 0}; }
detail::list_iterator end() const { return {*this, PyList_GET_SIZE(m_ptr)}; }
template <typename T> void append(T &&val) const {
PyList_Append(m_ptr, detail::object_or_cast(std::forward<T>(val)).ptr());
}
};
Also keep in mind that py::list inherits from py::object, which in turn inherits from py::handle (this also means that you are passing by reference). In my experience, there is very little documentation for this kind of usage, and reading the code is your best bet.
We can see from the class definition that we can use the member functions size, operator[], begin, end (C++ iterators!) and append (templated!). If this is not enough, you can use attr to access python attributes (including methods). Here is an example:
Python code (some_python.py):
import cppimport
cpp = cppimport.imp("some_cpp")
l = [1,2,3,4,5]
cpp.test(l)
print('after C++', l)
cpp.float_cast(l)
C++ code (some_cpp.cpp):
/* <%
setup_pybind11(cfg)
%> */
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
namespace py = pybind11;
void test(py::list l) {
l.attr("pop")();
std::cout << "List has length " << l.size() << std::endl;
for (py::handle obj : l) { // iterators!
std::cout << " - " << obj.attr("__str__")().cast<std::string>() << std::endl;
}
l.append(10); // automatic casting (through templating)!
}
void float_cast(py::list l) {
float f = l.cast<float>();
}
PYBIND11_MODULE(some_cpp, m) {
m.def("test", &test);
m.def("float_cast", &float_cast);
}
Output:
List has length 4
- 1
- 2
- 3
- 4
after C++ [1, 2, 3, 4, 10]
Traceback (most recent call last):
File "some_python.py", line 9, in <module>
cpp.float_cast(l)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
As you can see I've also included your specific question of casting to a float. Here I've used the cast method of py::handle, which gives a nice exception. You may try to "directly" cast the object (something like float* f = (float*) &l;) but this will give you garbage and I guess this is not what you're looking for.
One more remark: pybind/stl.h enables conversion between Python's standard types and C++ versions. For example, a list may be converted to a std::vector<int>, including typechecks. An important effect of this is that the data is passed as a copy instead of as a reference.

OUT argument internally allocated to return an array of structs

I'm new to swig and I have the following function which i cant fix:
int get_list(IN const char * string, OUT struct entry ** results);
where struct entry is defined:
struct flux_entry
{
char * addr_str;
char cc[2];
};
the entry struct is properly converted to a python class.
I googled but couldn't find any explanation i could use.
I want to make it return a tuple of: (original get_list int return value, python list of entry python objects, based on the results buffer), but don't know how to convert the C entry to a python object in the argout code snippet.
I've managed to get thus far:
%typemap(argout) struct entry **
{
PyObject *o = PyList_New(0);
int i;
for(i=0; $1[i] ; i++)
{
PyList_Append(o, SWIG_HOW_TO_CONVERT_TO_PYOBJECT($1[i]));
}
$result = o;
}
what should i replace SWIG_HOW_TO_CONVERT_TO_PYOBJECT with?
passed results is supposed to be a pointer to a (struct entry *) type, set to NULL before calling get_list and should be set to an allocated array of struct entry * pointers. maybe a small wrapper function could make that easier?
the struct entry array is allocated within the C function using malloc, after calculating (inside get_list) how many elements are needed, and ends with a NULL pointer to indicate the end of the array.
i'd also like to make sure it's freed somewhere :)
thanks!

This should at least give you a starting point that works. I still wasn't sure how the data was returned, since to return an array of pointers so that the final one was NULL I'd think you'd need a struct entry ***, so I just set addr_str = NULL on the last one as a sentinel, and just put some dummy data partially based on the input string into the fields. Modify as needed to suit your needs:
%module example
// Insert the structure definition and function to wrap into the wrapper code.
%{
struct entry {
char* addr_str;
char cc[2];
};
int get_list(const char* string, struct entry** results)
{
*results = malloc(3 * sizeof(struct entry));
(*results)[0].addr_str = malloc(10);
strcpy((*results)[0].addr_str,"hello");
(*results)[0].cc[0] = string[0];
(*results)[0].cc[1] = string[1];
(*results)[1].addr_str = malloc(10);
strcpy((*results)[1].addr_str,"there");
(*results)[1].cc[0] = string[2];
(*results)[1].cc[1] = string[3];
(*results)[2].addr_str = NULL;
return 0;
}
%}
#include <typemaps.i>
// Define the structure for SWIG
struct entry {
char* addr_str;
char cc[2];
};
// Define a set of typemaps to be used for an output parameter.
// This typemap suppresses requiring the parameter as an input.
// A temp variable is created and passed instead.
%typemap(in,numinputs=0) struct entry **OUTPUT (struct entry* temp) %{
$1 = &temp;
%}
// Build a list of tuples containing the two entries from the struct.
// Append the new Python list object to the existing "int" result.
%typemap(argout) struct entry **OUTPUT {
int i = 0;
PyObject* out = PyList_New(0);
while((*$1)[i].addr_str != NULL)
{
//PyObject* t = PyTuple_New(2);
//PyTuple_SET_ITEM(t,0,PyBytes_FromString((*$1)[i].addr_str));
//PyTuple_SET_ITEM(t,1,PyBytes_FromStringAndSize((*$1)[i].cc,2));
//PyList_Append(out,t);
//Py_DECREF(t);
PyObject* s = SWIG_NewPointerObj(*$1+i,$descriptor(struct entry*),0);
PyList_Append(out,s);
Py_DECREF(s);
++i;
}
$result = SWIG_AppendOutput($result,out);
}
// Since a Python object was created and the data copied for each entry struct,
// free the memory returned in the structure.
//%typemap(freearg) struct entry **OUTPUT {
// int i=0;
// while((*$1)[i].addr_str != NULL) {
// free((*$1)[i].addr_str);
// ++i;
// }
// free(*$1);
//}
// Apply the OUTPUT typemap set to the "results" parameter.
%apply struct entry **OUTPUT {struct entry** results};
// Finally, define the function for SWIG
int get_list(const char* string, struct entry** results);
Demo (Python 3.3):
>>> import example
>>> example.get_list('abcd')
[0, [(b'hello', b'ab'), (b'there', b'cd')]]
Hope that helps.
Edit:
I commented out the tuple creation and just save the entry* proxy instead. This doesn't leak Python objects, but the memory malloced for use by an entry* is not freed. I'm not sure where to put that, although I'm experimenting with %extend.

Writing a Python C extension: how to correctly load a PyListObject?

While attempting to read a Python list filled with float numbers and to populate real channels[7] with their values (I'm using F2C, so real is just a typedef for float), all I am able to retrieve from it are zero values. Can you point out the error in the code below?
static PyObject *orbital_spectra(PyObject *self, PyObject *args) {
PyListObject *input = (PyListObject*)PyList_New(0);
real channels[7], coefficients[7], values[240];
int i;
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &input)) {
return NULL;
}
for (i = 0; i < PyList_Size(input); i++) {
printf("%f\n", PyList_GetItem(input, (Py_ssize_t)i)); // <--- Prints zeros
}
//....
}

PyList_GetItem will return a PyObject*. You need to convert that to a number C understands. Try changing your code to this:
printf("%f\n", PyFloat_AsDouble(PyList_GetItem(input, (Py_ssize_t)i)));

Few things I see in this code.
You leak a reference, don't create that empty list at the beginning, it's not needed.
You don't need to cast to PyListObject.
PyList_GetItem returns a PyObject, not a float. Use PyFloat_AsDouble to extract the value.
If PyList_GetItem returns NULL, then an exception has been thrown, and you should check for it.