We Keep Coding

Pass Python bytes to C/C++ function using swig?

How do you pass a bytes value from Python (like data loaded from a file with open('my file.dat', 'rb').read()) to a C/C++ function using swig?
When I try using char * or uint8_t * and then a size parameter it gives me an error like this:
TypeError: in method 'processData', argument 3 of type 'char *'
I've tried using %pybuffer_mutable_binary and %pybuffer_binary but they don't seem to change the definition of the wrapper and I still get the same error.

Without code can't diagnose what is wrong, but likely you didn't declare %pybuffer lines before the function definitions. If declared after the generated wrappers won't use them when processing the functions, which would explain "they don't seem to change the definition of the wrapper".
Here's a working example. Note that passing an immutable item to a function that modifies the string will crash Python. It would be nice if the commands from pybuffer.i type-checked the Python object for mutability. If you want that don't use pybuffer.i.
test.i
%module test
%{
#include <stdlib.h>
#include <string.h>
%}
%include <pybuffer.i>
%pybuffer_mutable_string(char* str1)
%pybuffer_string(const char* str2)
%pybuffer_mutable_binary(char* str3, size_t size)
%pybuffer_binary(const char* str4, size_t size)
%inline %{
void funcms(char *str1) {
strupr(str1);
}
size_t funcs(const char *str2) {
return strlen(str2);
}
void funcmb(char* str3, size_t size) {
memset(str3,'A',size);
}
size_t funcb(const char* str4, size_t size) {
size_t tmp = 0;
for(size_t i = 0; i < size; ++i)
tmp += str4[i];
return tmp % 256;
}
%}
Demo:
>>> import test
>>> b=bytearray(b'abc') # mutable string (nul-terminated)
>>> test.funcms(b)
>>> b
bytearray(b'ABC')
>>> test.funcs(b'abc') # immutable string (nul-terminated)
3
>>> b=bytearray(b'ab\0cd\0ef') # mutable data (includes nulls)
>>> test.funcmb(b)
>>> b
bytearray(b'AAAAAAAA')
>>> test.funcb(b'ab\0cd\0ef') # immutable data (compute byte checksum)
85
>>> sum(b'ab\0cd\0ef')%256 # verify result
85

I think the best way to do this is a type map using the Python buffer interface. This passes a pointer to your data to the C/C++ function without any copying of data. For example:
%typemap(in, numinputs=1) (const char *data, unsigned long int size) {
Py_buffer view;
if (PyObject_CheckBuffer($input) != 1) {
PyErr_SetString(
PyExc_TypeError,
"in method '$symname', argument $argnum does not support the buffer interface");
SWIG_fail;
}
if (PyObject_GetBuffer($input, &view, PyBUF_SIMPLE) != 0) {
PyErr_SetString(
PyExc_TypeError,
"in method '$symname', argument $argnum does not export a simple buffer");
SWIG_fail;
}
$1 = view.buf;
$2 = view.len;
PyBuffer_Release(&view);
}
%typemap(doc) const char *data, unsigned long int size "$1_name: readable buffer (e.g. bytes)"

Expressing a Python class as a type in std::variant in C++ using pybind11

The working example below returns, a vector of type variant consisting of float and int64_t, to Python. The intent (illustrated by the commented lines of code **) is to add further functionality to this by enabling a Decimal (python class), constructed in C++, to be passed back to Python in the same structure.
#include <pybind11/pybind11.h>
#include <vector>
#include <variant>
#include <string>
#include <pybind11/stl.h>
#include <pybind11/complex.h>
#include <pybind11/functional.h>
#include <pybind11/chrono.h>
namespace py = pybind11;
// Importing Decimal class, as shown here
// https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html?highlight=Decimal(#accessing-python-libraries-from-c
py::object Decimal = py::module_::import("decimal").attr("Decimal");
//typedef std::variant<float, int64_t, Decimal> Listtypes; // **
typedef std::variant<float, int64_t> Listtypes;
std::vector<ListTypes> returnList() {
std::vector<ListTypes> list(3);
int64_t integerVal = 987654321;
float floatVal = 1.01;
// Constructing python object, as shown here
//https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html#callingpythonfunctions
py::object pi = Decimal("3.14159");
list[0] = integerVal;
list[1] = floatVal;
//list[2] = pi; // **
return list;
}
PYBIND11_MODULE(pBind, m) {
m.def("returnList", &returnList, "Function to return list of differing types");
}
So to go about this, can the Decimal be expressed as part of the std::variant so that it can be passed back to Python with the vector, or is the solution not so simple?

you cannot just add the pi variable as is to your std::variant vector because the type of it is py::object. You could add it to your ListTypes so just changing the line
typedef std::variant<float, int64_t, py::object> Listtypes;
but then your list on the Python side would be [987654321, 1.0099999904632568, Decimal(3.14159)]
I think you'd rather use the casting offered by pybind to convert your pi variable to float so that your code becomes
#include <pybind11/pybind11.h>
#include <vector>
#include <variant>
#include <string>
#include <pybind11/stl.h>
#include <pybind11/complex.h>
#include <pybind11/functional.h>
#include <pybind11/chrono.h>
namespace py = pybind11;
// Importing Decimal class, as shown here
// https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html?highlight=Decimal(#accessing-python-libraries-from-c
py::object Decimal = py::module_::import("decimal").attr("Decimal");
//typedef std::variant<float, int64_t, Decimal> Listtypes; // **
typedef std::variant<float, int64_t> Listtypes;
std::vector<ListTypes> returnList() {
std::vector<ListTypes> list(3);
int64_t integerVal = 987654321;
float floatVal = 1.01;
// Constructing python object, as shown here
//https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html#callingpythonfunctions
py::object pi = Decimal("3.14159");
list[0] = integerVal;
list[1] = floatVal;
list[2] = pi.cast<float>(); // we cast to float
return list;
}
PYBIND11_MODULE(pBind, m) {
m.def("returnList", &returnList, "Function to return list of differing types");
}

SWIG "out" typemap returns address instead of data

I am using SWIG to accept a list that is variable in size from Python, send it to C++ to do something with it, and then send it back to Python to be printed out.
I am new to Python, C++ and Swig. Currently the sent list will be handled as a vector argument in my C++ function. After that what is returned from the function is a pointer, which is handled by an "out" typemap.
The list is able to be shown from Python, however only if given a set size in the out typemap. Currently I need to make it handle lists of various sizes.
When trying to achieve this, I end up returning the address rather than a list in Python.
Below showcases the code that works when given a fixed size.
customvector.cc
#include "customvector.h"
#include <algorithm>
#include <functional>
float * summy(std::vector<float> a)
{
float * p = a.data();
return p;
}
customvector.h
#include <stdio.h>
#include <iostream>
#include <vector>
float * summy(std::vector<float> a);
customvector.i
/* File: customvector.i */
%module customvector
%{
#define SWIG_FILE_WITH_INIT
#include "customvector.h"
%}
%include "std_vector.i"
%include <stdint.i>
namespace std {
%template(Line) vector < float >;
}
%typemap(out) float* summy{
int i;
$result = PyList_New(3);
for (i = 0; i < 3; i++) {
PyObject *o = PyFloat_FromDouble((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
float * summy(std::vector<float> a);
My python result:
>>> import customvector
>>> a = [1,2,3]
>>> customvector.summy(a)
[1.0, 2.0, 3.0]
I then edited my interface file so that the out typemap now uses [ANY] instead of just 3 to allow the length to vary.
Edited customvector.i
/* File: customvector.i */
%module customvector
%{
#define SWIG_FILE_WITH_INIT
#include "customvector.h"
%}
%include "std_vector.i"
%include <stdint.i>
namespace std {
%template(Line) vector < float >;
}
%typemap(out) float* summy [ANY]{ //changed from 3 to [ANY]
int i;
$result = PyList_New($1_dim0); //changed from 3 to $1_dim0
for (i = 0; i < $1_dim0; i++) {
PyObject *o = PyFloat_FromDouble((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
float * summy(std::vector<float> a);
Result from Python:
>>> import customvector
>>> a = [1,2,3]
>>> customvector.summy(a)
<Swig Object of type 'float *' at 0x000001E4E32E6420>
Which is not what I want, and it should display what was shown earlier on.
I tried following the documentation listed somewhere here: http://www.swig.org/Doc2.0/Typemaps.html#Typemaps_nn40 for SWIG to get the value than the output, but it doesn't seem to work.
I also came across this solution to allow length to vary: Python/SWIG: Output an array
but I am not sure how it works as I tried using it but the code doesn't compile (saying that templen is not defined).
How can I output from C++ into python such a data with variable size?

You have undefined behavior in your implementation. Why not use std::vector for the return value as well?
%module test
%include <std_vector.i>
%template() std::vector<float>;
%inline %{
std::vector<float> summy(std::vector<float> a)
{
for(auto& i: a)
i += 1;
return a;
}
%}
Demo:
>>> import test
>>> test.summy([1,2,3,4])
(2.0, 3.0, 4.0, 5.0)
The default behavior for returned vectors is to use a tuple. That can be overridden with a typemap:
%module test
%include <std_vector.i>
%template() std::vector<float>;
%typemap(out) std::vector<float> %{
$result = PyList_New($1.size());
for (int i = 0; i < $1.size(); ++i)
PyList_SET_ITEM($result,i,PyFloat_FromDouble($1[i]));
%}
%inline %{
#include <vector>
std::vector<float> summy(std::vector<float> a)
{
for(auto& i: a)
i += 1;
return a;
}
%}
Demo:
>>> import test
>>> test.summy([1,2,3,4])
[2.0, 3.0, 4.0, 5.0]

Swig and pointer problems (python)

I'm using Swig to generate a python wrapper for a DLL file. What I do is:
Generate the wrapper and loader file using swig -c++ -python
myfile.i
Create a new DLL file, include myfile_wrapper.cxx and
compile to _myfile.pyd.
Load the module myfile.py created by Swig in Idle and try to use it.
The interface file looks like:
%module myfile
/* Make a test with cpointer - needed?? */
%include cpointer.i
%pointer_functions(MyHandle, my_handle_p);
%pointer_class(int, intp);
%{
#define SWIG_FILE_WITH_INIT
#include "MyFile.h"
}%
%include "MyFile.h"
The function looks like
typedef struct tagMyHandle
{
void* reserved;
} *MyHandle;
int OpenFile(const char *szPath, MyHandle* pFile); // pFile is an out parameter
int GetNumberOfItems(MyHandle hFile, int *pnNrOfItems); // pnNrOfItems is an out parameter
If I try to use this from Python I have to do like this:
import myfile
handle_p = myfile.new_my_handle_p()
myfile.OpenFile("Path", handle_p)
handle = myfile.my_file_p_value(handle_p)
num_items_p = myfile.new_intp()
myfile.GetNumberOfItems(handle, num_items_p)
num_items = num_items_p.value()
Am I using Swig incorrectly? It feels that it's a very cumbersome way to call the functions that are supposed to be wrapped for Python.
I would like to do something like:
result, handle = OpenFile("path")
result, items = GetNumberIfItems(handle)
I can't change the source code for myfile.h.
I'm looked at input/output parameters, but do I have to define them for each output type? MyFile.h have hundreds of functions with different output types. And it only supports primitive data types, but most types in MyFile.h are not primitive types, but like the struct MyHandle.
I have looked at SWIG function with pointer struct and http://www.swig.org/Doc3.0/Python.html#Python_nn18 as well, but without any good solution.
Update 1
After a lot of help, I've solved most problems, but I still have a few left that I don't understand.
Problem 1:
// For the out parameter, shouldn't be needed?
%typemap(in,numinputs=0) MyHandle* pOutParam (MyHandle h) %{
$1 = &h;
%}
// For all others
%typemap(in,numinputs=0) MyHandle* (MyHandle h) %{
$1 = &h;
%}
// For the return type
%typemap(argout) MyHandle* pOutParam (PyObject* o) %{
o = PyLong_FromVoidPtr(*$1);
$result = SWIG_Python_AppendOutput($result,o);
%}
%typemap(in) MyHandle %{
$1 = reinterpret_cast<MyHandle>(PyLong_AsVoidPtr($input));
%}
and the code
int OpenFile(const char *szPath, MyHandle* pOutParam);
int DoSomething(MyHandle* pInParam);
OpenFile works like charm, but DoSomething still tries to return MyHandle instead of taking it as an in parameter, and I don't understand why. %typemap(argout) MyHandle* is only defined for pOutParam.
Problem 2:
I don't understand how to make the type map for something like
int GetFileName(char *szPathBuffer, int iLength);
how to create a char buffer and send that in, like I C:
char szBuffer[MAX_PATH]; GetFileName(szBuffer, MAX_PATH);
Maybe something together with cstring_bounded_output or should I do something like
%typemap(in) (char*, int) {
$2 = PyString_Size($input);
$1 = (char*) malloc($2 * sizeof(char*));
}
but where is it deallocated?
Problem 3:
What is the correct mapping for enum values. If I have
typedef enum tagMyEnum {
MyTrue = 1,
MyFalse = 0 } MyEnum;
and the function
int IsCorrect(MyEnum* pOutValue);
#Mark Tolonen:
Thanks for all help! I really appreciate it! I've learned so much new things about Swig!

Here's an example with a interface similar to what you want to illustrate using typemaps to redefine an interface:
myfile.h
typedef struct tagMyHandle
{
void* reserved;
} *MyHandle;
int OpenFile(const char *szPath, MyHandle* pFile);
int GetNumberOfItems(MyHandle hFile, int *pnNrOfItems);
// Added this to free the allocated handle.
void CloseFile(MyHandle hFile);
myfile.cpp
A hack implementation of the header...
#include "myfile.h"
int OpenFile(const char *szPath, MyHandle* pFile)
{
*pFile = new tagMyHandle;
(*pFile)->reserved = new int(7);
return 1;
}
int GetNumberOfItems(MyHandle hFile, int *pnNrOfItems)
{
*pnNrOfItems = *reinterpret_cast<int*>(hFile->reserved) + 5;
return 1;
}
// mirrors OpenFile to free the allocated handle.
void CloseFile(MyHandle hFile)
{
delete reinterpret_cast<int*>(hFile->reserved);
delete hFile;
}
myfile.i
%module myfile
%{
#include "MyFile.h"
%}
// An input typemap for the an output parameter, called before the C++ function is called.
// It suppresses requiring the parameter from Python, and uses a temporary
// variable to hold the output value.
%typemap(in,numinputs=0) MyHandle* (MyHandle h) %{
$1 = &h;
%}
// An output argument typemap, called after the C++ function is called.
// It retrieves the output value and converts it to a Python int,
// then appends it to the existing return value. Python will get a tuple of
// (return_value,handle).
%typemap(argout) MyHandle* (PyObject* o) %{
o = PyLong_FromVoidPtr(*$1);
$result = SWIG_Python_AppendOutput($result,o);
%}
// An input typemap that converts a Python int to a MyHandle*.
%typemap(in) MyHandle %{
$1 = reinterpret_cast<MyHandle>(PyLong_AsVoidPtr($input));
%}
// This applies a pre-defined int* output typemap to all int* parameters.
%apply int *OUTPUT {int *};
%include "MyFile.h"
Output
>>> import myfile
>>> s,h = myfile.OpenFile('path')
>>> s,h
(1, 7706832)
>>> s,v = myfile.GetNumberOfItems(h)
>>> s,v
(1, 12)
>>> myfile.CloseFile(h)

Overwrite char (or std::string) array positions with SWIG?

I was able to write a void function in C/C++, and wrap to Python/Numpy with SWIG (int* INPLACE_ARRAY1, int DIM1), that receives a int* vector as parameter, do some math on this vector, and overwrite the results on the same vector, and this result was available inside Python's object. Like follows:
extern "C" void soma_aloc(int* vetor, int tamanho)
{
int m = 0;
int* ponteiro = new int[tamanho];
for(m = 0; m < tamanho; m++)
{
ponteiro[m] = vetor[m];
};
for(m = 0; m < tamanho; m++)
{
ponteiro[m] = ponteiro[m] * 10;
};
for(m = 0; m < tamanho; m++)
{
vetor[m] = ponteiro[m];
};
delete [] ponteiro;
};
This was a test to learn how to wrap pointers to int and double arrays with SWIG using the typemaps (DATA_TYPE* INPLACE_ARRAY1, int DIM1) and (DATA_TYPE* INPLACE_ARRAY2, int DIM1, int DIM2), and worked well.
But the problem is, I've tried the same idea with char/string Numpy vectors (like a vector vec1 = numpy.array(['a','a','a']) or numpy.array(['a','a','a'],dtype=str), and change each position to be like (['b','b','b']), but Python shows in method 'vector_char2D', argument 1 of type 'char *'. It's possible to do the same with char/string?
.cpp:
extern "C" void vetor_char2D(char* vetorchar, int tamanho_vetor)
{
for(int i = 0; i < tamanho_vetor; i++)
{
vetorchar[i] = 'b';
};
};
.i:
%module testestring
%include stl.i
%include std_string.i
%{
#include <stdio.h>
#include <stdlib.h>
//#include <string.h>
#include <string>
#include <iostream>
#define SWIG_FILE_WITH_INIT
#include "testestring.hpp"
%}
%include "numpy.i"
%init %{
import_array();
%}
%apply (char* INPLACE_ARRAY1, int DIM1) {(char* vetorchar, int tamanho_vetor)}
%include "testestring.hpp" (just the header of the above function vetor_char2D)
%clear (char* vetorchar, int tamanho_vetor);
I have very litthe experience with SWIG. It's possible to do this with char*, char** and/or std::string*/std::string** ? Thanks in advance!

Use std::vector:
void vetor_char2D(std::vector<std::string>& vetorchar)
{
for (int i = 0; i < vetorchar.size(); i++)
vetorchar[i] = "b";
};
which indicates clearly that the vector can be modified, and strings within it can be modified, and the SWIG typemaps for STL vectors and strings will work nicely. Note the double rather than single quote for string; Python only has strings no chars so it doesn't matter. You can also make it work with char* etc but it is rarely worth the effort, lot easier to use above. If you don't want to change the source, you can include the above as a wrapper in the .i file via an %inline directive.
Note you should not need the extern C qualifier. And you should use the #include <string> not string.h.