We Keep Coding

Pass Python bytes to C/C++ function using swig?

How do you pass a bytes value from Python (like data loaded from a file with open('my file.dat', 'rb').read()) to a C/C++ function using swig?
When I try using char * or uint8_t * and then a size parameter it gives me an error like this:
TypeError: in method 'processData', argument 3 of type 'char *'
I've tried using %pybuffer_mutable_binary and %pybuffer_binary but they don't seem to change the definition of the wrapper and I still get the same error.

Without code can't diagnose what is wrong, but likely you didn't declare %pybuffer lines before the function definitions. If declared after the generated wrappers won't use them when processing the functions, which would explain "they don't seem to change the definition of the wrapper".
Here's a working example. Note that passing an immutable item to a function that modifies the string will crash Python. It would be nice if the commands from pybuffer.i type-checked the Python object for mutability. If you want that don't use pybuffer.i.
test.i
%module test
%{
#include <stdlib.h>
#include <string.h>
%}
%include <pybuffer.i>
%pybuffer_mutable_string(char* str1)
%pybuffer_string(const char* str2)
%pybuffer_mutable_binary(char* str3, size_t size)
%pybuffer_binary(const char* str4, size_t size)
%inline %{
void funcms(char *str1) {
strupr(str1);
}
size_t funcs(const char *str2) {
return strlen(str2);
}
void funcmb(char* str3, size_t size) {
memset(str3,'A',size);
}
size_t funcb(const char* str4, size_t size) {
size_t tmp = 0;
for(size_t i = 0; i < size; ++i)
tmp += str4[i];
return tmp % 256;
}
%}
Demo:
>>> import test
>>> b=bytearray(b'abc') # mutable string (nul-terminated)
>>> test.funcms(b)
>>> b
bytearray(b'ABC')
>>> test.funcs(b'abc') # immutable string (nul-terminated)
3
>>> b=bytearray(b'ab\0cd\0ef') # mutable data (includes nulls)
>>> test.funcmb(b)
>>> b
bytearray(b'AAAAAAAA')
>>> test.funcb(b'ab\0cd\0ef') # immutable data (compute byte checksum)
85
>>> sum(b'ab\0cd\0ef')%256 # verify result
85

I think the best way to do this is a type map using the Python buffer interface. This passes a pointer to your data to the C/C++ function without any copying of data. For example:
%typemap(in, numinputs=1) (const char *data, unsigned long int size) {
Py_buffer view;
if (PyObject_CheckBuffer($input) != 1) {
PyErr_SetString(
PyExc_TypeError,
"in method '$symname', argument $argnum does not support the buffer interface");
SWIG_fail;
}
if (PyObject_GetBuffer($input, &view, PyBUF_SIMPLE) != 0) {
PyErr_SetString(
PyExc_TypeError,
"in method '$symname', argument $argnum does not export a simple buffer");
SWIG_fail;
}
$1 = view.buf;
$2 = view.len;
PyBuffer_Release(&view);
}
%typemap(doc) const char *data, unsigned long int size "$1_name: readable buffer (e.g. bytes)"

pybind how can I operate over a py::list object

For a better understanding of how to pass arguments from Python to C++ functions with the pybind library, I wanted to build a small dummy/demo code where I could receive a Python list on the C++ side, cast it to a float pointer object, and then print it.
Though I know I can use the py::list class I haven't figured out the methods available of this class. I looked in the documentation reference, and then in the code (list.h, stl.h) and couldn't figure out which methods where available.
What is the equivalent of __getitem__? Do I have every python method available for py::list?

The code you are looking for is here:
class list : public object {
public:
PYBIND11_OBJECT_CVT(list, object, PyList_Check, PySequence_List)
explicit list(size_t size = 0) : object(PyList_New((ssize_t) size), stolen_t{}) {
if (!m_ptr) pybind11_fail("Could not allocate list object!");
}
size_t size() const { return (size_t) PyList_Size(m_ptr); }
detail::list_accessor operator[](size_t index) const { return {*this, index}; }
detail::list_iterator begin() const { return {*this, 0}; }
detail::list_iterator end() const { return {*this, PyList_GET_SIZE(m_ptr)}; }
template <typename T> void append(T &&val) const {
PyList_Append(m_ptr, detail::object_or_cast(std::forward<T>(val)).ptr());
}
};
Also keep in mind that py::list inherits from py::object, which in turn inherits from py::handle (this also means that you are passing by reference). In my experience, there is very little documentation for this kind of usage, and reading the code is your best bet.
We can see from the class definition that we can use the member functions size, operator[], begin, end (C++ iterators!) and append (templated!). If this is not enough, you can use attr to access python attributes (including methods). Here is an example:
Python code (some_python.py):
import cppimport
cpp = cppimport.imp("some_cpp")
l = [1,2,3,4,5]
cpp.test(l)
print('after C++', l)
cpp.float_cast(l)
C++ code (some_cpp.cpp):
/* <%
setup_pybind11(cfg)
%> */
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
namespace py = pybind11;
void test(py::list l) {
l.attr("pop")();
std::cout << "List has length " << l.size() << std::endl;
for (py::handle obj : l) { // iterators!
std::cout << " - " << obj.attr("__str__")().cast<std::string>() << std::endl;
}
l.append(10); // automatic casting (through templating)!
}
void float_cast(py::list l) {
float f = l.cast<float>();
}
PYBIND11_MODULE(some_cpp, m) {
m.def("test", &test);
m.def("float_cast", &float_cast);
}
Output:
List has length 4
- 1
- 2
- 3
- 4
after C++ [1, 2, 3, 4, 10]
Traceback (most recent call last):
File "some_python.py", line 9, in <module>
cpp.float_cast(l)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
As you can see I've also included your specific question of casting to a float. Here I've used the cast method of py::handle, which gives a nice exception. You may try to "directly" cast the object (something like float* f = (float*) &l;) but this will give you garbage and I guess this is not what you're looking for.
One more remark: pybind/stl.h enables conversion between Python's standard types and C++ versions. For example, a list may be converted to a std::vector<int>, including typechecks. An important effect of this is that the data is passed as a copy instead of as a reference.

Swig and pointer problems (python)

I'm using Swig to generate a python wrapper for a DLL file. What I do is:
Generate the wrapper and loader file using swig -c++ -python
myfile.i
Create a new DLL file, include myfile_wrapper.cxx and
compile to _myfile.pyd.
Load the module myfile.py created by Swig in Idle and try to use it.
The interface file looks like:
%module myfile
/* Make a test with cpointer - needed?? */
%include cpointer.i
%pointer_functions(MyHandle, my_handle_p);
%pointer_class(int, intp);
%{
#define SWIG_FILE_WITH_INIT
#include "MyFile.h"
}%
%include "MyFile.h"
The function looks like
typedef struct tagMyHandle
{
void* reserved;
} *MyHandle;
int OpenFile(const char *szPath, MyHandle* pFile); // pFile is an out parameter
int GetNumberOfItems(MyHandle hFile, int *pnNrOfItems); // pnNrOfItems is an out parameter
If I try to use this from Python I have to do like this:
import myfile
handle_p = myfile.new_my_handle_p()
myfile.OpenFile("Path", handle_p)
handle = myfile.my_file_p_value(handle_p)
num_items_p = myfile.new_intp()
myfile.GetNumberOfItems(handle, num_items_p)
num_items = num_items_p.value()
Am I using Swig incorrectly? It feels that it's a very cumbersome way to call the functions that are supposed to be wrapped for Python.
I would like to do something like:
result, handle = OpenFile("path")
result, items = GetNumberIfItems(handle)
I can't change the source code for myfile.h.
I'm looked at input/output parameters, but do I have to define them for each output type? MyFile.h have hundreds of functions with different output types. And it only supports primitive data types, but most types in MyFile.h are not primitive types, but like the struct MyHandle.
I have looked at SWIG function with pointer struct and http://www.swig.org/Doc3.0/Python.html#Python_nn18 as well, but without any good solution.
Update 1
After a lot of help, I've solved most problems, but I still have a few left that I don't understand.
Problem 1:
// For the out parameter, shouldn't be needed?
%typemap(in,numinputs=0) MyHandle* pOutParam (MyHandle h) %{
$1 = &h;
%}
// For all others
%typemap(in,numinputs=0) MyHandle* (MyHandle h) %{
$1 = &h;
%}
// For the return type
%typemap(argout) MyHandle* pOutParam (PyObject* o) %{
o = PyLong_FromVoidPtr(*$1);
$result = SWIG_Python_AppendOutput($result,o);
%}
%typemap(in) MyHandle %{
$1 = reinterpret_cast<MyHandle>(PyLong_AsVoidPtr($input));
%}
and the code
int OpenFile(const char *szPath, MyHandle* pOutParam);
int DoSomething(MyHandle* pInParam);
OpenFile works like charm, but DoSomething still tries to return MyHandle instead of taking it as an in parameter, and I don't understand why. %typemap(argout) MyHandle* is only defined for pOutParam.
Problem 2:
I don't understand how to make the type map for something like
int GetFileName(char *szPathBuffer, int iLength);
how to create a char buffer and send that in, like I C:
char szBuffer[MAX_PATH]; GetFileName(szBuffer, MAX_PATH);
Maybe something together with cstring_bounded_output or should I do something like
%typemap(in) (char*, int) {
$2 = PyString_Size($input);
$1 = (char*) malloc($2 * sizeof(char*));
}
but where is it deallocated?
Problem 3:
What is the correct mapping for enum values. If I have
typedef enum tagMyEnum {
MyTrue = 1,
MyFalse = 0 } MyEnum;
and the function
int IsCorrect(MyEnum* pOutValue);
#Mark Tolonen:
Thanks for all help! I really appreciate it! I've learned so much new things about Swig!

Here's an example with a interface similar to what you want to illustrate using typemaps to redefine an interface:
myfile.h
typedef struct tagMyHandle
{
void* reserved;
} *MyHandle;
int OpenFile(const char *szPath, MyHandle* pFile);
int GetNumberOfItems(MyHandle hFile, int *pnNrOfItems);
// Added this to free the allocated handle.
void CloseFile(MyHandle hFile);
myfile.cpp
A hack implementation of the header...
#include "myfile.h"
int OpenFile(const char *szPath, MyHandle* pFile)
{
*pFile = new tagMyHandle;
(*pFile)->reserved = new int(7);
return 1;
}
int GetNumberOfItems(MyHandle hFile, int *pnNrOfItems)
{
*pnNrOfItems = *reinterpret_cast<int*>(hFile->reserved) + 5;
return 1;
}
// mirrors OpenFile to free the allocated handle.
void CloseFile(MyHandle hFile)
{
delete reinterpret_cast<int*>(hFile->reserved);
delete hFile;
}
myfile.i
%module myfile
%{
#include "MyFile.h"
%}
// An input typemap for the an output parameter, called before the C++ function is called.
// It suppresses requiring the parameter from Python, and uses a temporary
// variable to hold the output value.
%typemap(in,numinputs=0) MyHandle* (MyHandle h) %{
$1 = &h;
%}
// An output argument typemap, called after the C++ function is called.
// It retrieves the output value and converts it to a Python int,
// then appends it to the existing return value. Python will get a tuple of
// (return_value,handle).
%typemap(argout) MyHandle* (PyObject* o) %{
o = PyLong_FromVoidPtr(*$1);
$result = SWIG_Python_AppendOutput($result,o);
%}
// An input typemap that converts a Python int to a MyHandle*.
%typemap(in) MyHandle %{
$1 = reinterpret_cast<MyHandle>(PyLong_AsVoidPtr($input));
%}
// This applies a pre-defined int* output typemap to all int* parameters.
%apply int *OUTPUT {int *};
%include "MyFile.h"
Output
>>> import myfile
>>> s,h = myfile.OpenFile('path')
>>> s,h
(1, 7706832)
>>> s,v = myfile.GetNumberOfItems(h)
>>> s,v
(1, 12)
>>> myfile.CloseFile(h)

How to deal with Python slice objects in boost::python?

Suppose there is a class MyArray in C++. It implements an array of SomeType In order to make a __getitem__ function for it in Python, I do something like this
const SomeType& getitem(const MyArray *arr, PyObject *slice) {
// ???
}
BOOST_PYTHON_MODULE(mymodule)
{
class_<MyArray>("MyArray")
.def("__getitem__", &getitem)
// probably some other methods...
;
}
It is possible to get indices in slice by using these functions. However, "Boost::Python is designed with the idea in mind that users never touch a PyObject*".
Is there a better 'boost way' to do this?

Boost.Python is designed to minimize the need to interact with PyObject, and it often accomplishes this by:
Providing higher-level type wrappers.
Allowing access to the Python object's interface through the associated boost::python::object.
For example, one can access the Python object's interface through C++ in a similar manner as one would do in Python. The following demonstrates accessing the start attribute of a boost::python::object that refers to a Python slice instance:
namespace python = boost::python;
python::object slice = get_slice_object();
python::object start = slice.attr("start");
std::size_t start_index = !start.is_none()
? python::extract<std::size_t>(start) // Extract index.
: 0; // Default.
While this approach works, it tends to result in much boilerplate code: creating defaults when None is provided, handling zero-length slices, and converting negative indexes to positive index. In this case, Boost.Python provides a higher-level type wrapper boost::python::slice that has a get_indices() member-function that will remove much of the boilerplate code. Here is a complete minimal example:
#include <vector>
#include <boost/range/algorithm.hpp>
#include <boost/range/irange.hpp>
#include <boost/python.hpp>
#include <boost/python/slice.hpp>
/// #brief Mockup class that creates a range from 0 to N.
struct counter
{
counter(std::size_t n)
{
data.reserve(n);
boost::copy(boost::irange(std::size_t(0), n), std::back_inserter(data));
}
std::vector<int> data;
};
/// #brief Handle slicing for counter object.
boost::python::list spam_getitem(
const counter& self,
boost::python::slice slice)
{
namespace python = boost::python;
python::list result;
// Boost.Python will throw std::invalid_argument if the range would be
// empty.
python::slice::range<std::vector<int>::const_iterator> range;
try
{
range = slice.get_indices(self.data.begin(), self.data.end());
}
catch (std::invalid_argument)
{
return result;
}
// Iterate over fully-closed range.
for (; range.start != range.stop; std::advance(range.start, range.step))
{
result.append(*range.start);
}
result.append(*range.start); // Handle last item.
return result;
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<counter>("Counter", python::init<int>())
.def("__getitem__", &spam_getitem)
;
}
Interactive usage:
>>> from example import Counter
>>> counter = Counter(5)
>>> assert(counter[:] == [0,1,2,3,4])
>>> assert(counter[:-2] == [0,1,2])
>>> assert(counter[-2:] == [3,4])
>>> assert(counter[::2] == [0,2,4])
>>> assert(counter[1::2] == [1,3])
>>> assert(counter[100:] == [])

OUT argument internally allocated to return an array of structs

I'm new to swig and I have the following function which i cant fix:
int get_list(IN const char * string, OUT struct entry ** results);
where struct entry is defined:
struct flux_entry
{
char * addr_str;
char cc[2];
};
the entry struct is properly converted to a python class.
I googled but couldn't find any explanation i could use.
I want to make it return a tuple of: (original get_list int return value, python list of entry python objects, based on the results buffer), but don't know how to convert the C entry to a python object in the argout code snippet.
I've managed to get thus far:
%typemap(argout) struct entry **
{
PyObject *o = PyList_New(0);
int i;
for(i=0; $1[i] ; i++)
{
PyList_Append(o, SWIG_HOW_TO_CONVERT_TO_PYOBJECT($1[i]));
}
$result = o;
}
what should i replace SWIG_HOW_TO_CONVERT_TO_PYOBJECT with?
passed results is supposed to be a pointer to a (struct entry *) type, set to NULL before calling get_list and should be set to an allocated array of struct entry * pointers. maybe a small wrapper function could make that easier?
the struct entry array is allocated within the C function using malloc, after calculating (inside get_list) how many elements are needed, and ends with a NULL pointer to indicate the end of the array.
i'd also like to make sure it's freed somewhere :)
thanks!

This should at least give you a starting point that works. I still wasn't sure how the data was returned, since to return an array of pointers so that the final one was NULL I'd think you'd need a struct entry ***, so I just set addr_str = NULL on the last one as a sentinel, and just put some dummy data partially based on the input string into the fields. Modify as needed to suit your needs:
%module example
// Insert the structure definition and function to wrap into the wrapper code.
%{
struct entry {
char* addr_str;
char cc[2];
};
int get_list(const char* string, struct entry** results)
{
*results = malloc(3 * sizeof(struct entry));
(*results)[0].addr_str = malloc(10);
strcpy((*results)[0].addr_str,"hello");
(*results)[0].cc[0] = string[0];
(*results)[0].cc[1] = string[1];
(*results)[1].addr_str = malloc(10);
strcpy((*results)[1].addr_str,"there");
(*results)[1].cc[0] = string[2];
(*results)[1].cc[1] = string[3];
(*results)[2].addr_str = NULL;
return 0;
}
%}
#include <typemaps.i>
// Define the structure for SWIG
struct entry {
char* addr_str;
char cc[2];
};
// Define a set of typemaps to be used for an output parameter.
// This typemap suppresses requiring the parameter as an input.
// A temp variable is created and passed instead.
%typemap(in,numinputs=0) struct entry **OUTPUT (struct entry* temp) %{
$1 = &temp;
%}
// Build a list of tuples containing the two entries from the struct.
// Append the new Python list object to the existing "int" result.
%typemap(argout) struct entry **OUTPUT {
int i = 0;
PyObject* out = PyList_New(0);
while((*$1)[i].addr_str != NULL)
{
//PyObject* t = PyTuple_New(2);
//PyTuple_SET_ITEM(t,0,PyBytes_FromString((*$1)[i].addr_str));
//PyTuple_SET_ITEM(t,1,PyBytes_FromStringAndSize((*$1)[i].cc,2));
//PyList_Append(out,t);
//Py_DECREF(t);
PyObject* s = SWIG_NewPointerObj(*$1+i,$descriptor(struct entry*),0);
PyList_Append(out,s);
Py_DECREF(s);
++i;
}
$result = SWIG_AppendOutput($result,out);
}
// Since a Python object was created and the data copied for each entry struct,
// free the memory returned in the structure.
//%typemap(freearg) struct entry **OUTPUT {
// int i=0;
// while((*$1)[i].addr_str != NULL) {
// free((*$1)[i].addr_str);
// ++i;
// }
// free(*$1);
//}
// Apply the OUTPUT typemap set to the "results" parameter.
%apply struct entry **OUTPUT {struct entry** results};
// Finally, define the function for SWIG
int get_list(const char* string, struct entry** results);
Demo (Python 3.3):
>>> import example
>>> example.get_list('abcd')
[0, [(b'hello', b'ab'), (b'there', b'cd')]]
Hope that helps.
Edit:
I commented out the tuple creation and just save the entry* proxy instead. This doesn't leak Python objects, but the memory malloced for use by an entry* is not freed. I'm not sure where to put that, although I'm experimenting with %extend.