We Keep Coding

Pass Python bytes to C/C++ function using swig?

How do you pass a bytes value from Python (like data loaded from a file with open('my file.dat', 'rb').read()) to a C/C++ function using swig?
When I try using char * or uint8_t * and then a size parameter it gives me an error like this:
TypeError: in method 'processData', argument 3 of type 'char *'
I've tried using %pybuffer_mutable_binary and %pybuffer_binary but they don't seem to change the definition of the wrapper and I still get the same error.

Without code can't diagnose what is wrong, but likely you didn't declare %pybuffer lines before the function definitions. If declared after the generated wrappers won't use them when processing the functions, which would explain "they don't seem to change the definition of the wrapper".
Here's a working example. Note that passing an immutable item to a function that modifies the string will crash Python. It would be nice if the commands from pybuffer.i type-checked the Python object for mutability. If you want that don't use pybuffer.i.
test.i
%module test
%{
#include <stdlib.h>
#include <string.h>
%}
%include <pybuffer.i>
%pybuffer_mutable_string(char* str1)
%pybuffer_string(const char* str2)
%pybuffer_mutable_binary(char* str3, size_t size)
%pybuffer_binary(const char* str4, size_t size)
%inline %{
void funcms(char *str1) {
strupr(str1);
}
size_t funcs(const char *str2) {
return strlen(str2);
}
void funcmb(char* str3, size_t size) {
memset(str3,'A',size);
}
size_t funcb(const char* str4, size_t size) {
size_t tmp = 0;
for(size_t i = 0; i < size; ++i)
tmp += str4[i];
return tmp % 256;
}
%}
Demo:
>>> import test
>>> b=bytearray(b'abc') # mutable string (nul-terminated)
>>> test.funcms(b)
>>> b
bytearray(b'ABC')
>>> test.funcs(b'abc') # immutable string (nul-terminated)
3
>>> b=bytearray(b'ab\0cd\0ef') # mutable data (includes nulls)
>>> test.funcmb(b)
>>> b
bytearray(b'AAAAAAAA')
>>> test.funcb(b'ab\0cd\0ef') # immutable data (compute byte checksum)
85
>>> sum(b'ab\0cd\0ef')%256 # verify result
85

I think the best way to do this is a type map using the Python buffer interface. This passes a pointer to your data to the C/C++ function without any copying of data. For example:
%typemap(in, numinputs=1) (const char *data, unsigned long int size) {
Py_buffer view;
if (PyObject_CheckBuffer($input) != 1) {
PyErr_SetString(
PyExc_TypeError,
"in method '$symname', argument $argnum does not support the buffer interface");
SWIG_fail;
}
if (PyObject_GetBuffer($input, &view, PyBUF_SIMPLE) != 0) {
PyErr_SetString(
PyExc_TypeError,
"in method '$symname', argument $argnum does not export a simple buffer");
SWIG_fail;
}
$1 = view.buf;
$2 = view.len;
PyBuffer_Release(&view);
}
%typemap(doc) const char *data, unsigned long int size "$1_name: readable buffer (e.g. bytes)"

How to initialize C struct members at creation in SWIG for Python?

Let's say we have a following struct in C:
typedef struct buffer
{
uint8_t* const data;
const size_t maxSize;
size_t size;
} buffer_t;
How can I make a SWIG wrapper for this so that when buffer_t is created in Python it allocates given number of bytes to data pointer and sets maxSize accordingly?
Details
So basically the problem here is related to constant struct members. By default SWIG initializes structs with default values. This leads into problems:
StructWithConsts_t struct;
struct.constant = 5; // Error. This should've been set when creating the struct.
The accepted answer with constructors and destructors provide solution for this problem.

You can treat C structs as C++ classes in SWIG and extend them with creators and destroyers methods.
In your case your .i should have this
typedef struct
{
uint8_t* const data;
const size_t maxSize;
size_t size;
} buffer_t;
%extend buffer_t {
buffer_t(size_t size)
{
buffer_t* pBuffer = malloc(sizeof *pBuffer);
pBuffer->data = malloc(size * sizeof (*pBuffer->data));
pBuffer->maxSize = size;
return pBuffer;
}
void ~buffer_t()
{
buffer_t *pBuffer = ($self);
free(pBuffer->data);
free(pBuffer);
return;
}
};
In python it will be used like this
from yourmodule import buffer_t
buffer = buffer_t(10)
The garbage collector will take care of freeing the memory.
There are other ways of doing this but you don't have a reproducing example to test it out.

I found one solution, but it is rather hackish due to pointer typecasts. I'm not convinced that this is the proper way to do this, but it works:
In the .i file I created a wrapper using %inline:
%inline %{
buffer_t* new_buffer(size_t size)
{
buffer_t* pBuffer = (buffer_t*)malloc(sizeof(buffer_t));
*(uint8_t**)&pBuffer->data = (uint8_t*)malloc(size * sizeof(uint8_t));
*(size_t*)&pBuffer->maxSize = size;
return (buffer_t*)pBuffer;
}
void free_buffer(buffer_t* pBuffer)
{
free(pBuffer->data);
free(pBuffer);
return;
}
%}

SWIG "out" typemap returns address instead of data

I am using SWIG to accept a list that is variable in size from Python, send it to C++ to do something with it, and then send it back to Python to be printed out.
I am new to Python, C++ and Swig. Currently the sent list will be handled as a vector argument in my C++ function. After that what is returned from the function is a pointer, which is handled by an "out" typemap.
The list is able to be shown from Python, however only if given a set size in the out typemap. Currently I need to make it handle lists of various sizes.
When trying to achieve this, I end up returning the address rather than a list in Python.
Below showcases the code that works when given a fixed size.
customvector.cc
#include "customvector.h"
#include <algorithm>
#include <functional>
float * summy(std::vector<float> a)
{
float * p = a.data();
return p;
}
customvector.h
#include <stdio.h>
#include <iostream>
#include <vector>
float * summy(std::vector<float> a);
customvector.i
/* File: customvector.i */
%module customvector
%{
#define SWIG_FILE_WITH_INIT
#include "customvector.h"
%}
%include "std_vector.i"
%include <stdint.i>
namespace std {
%template(Line) vector < float >;
}
%typemap(out) float* summy{
int i;
$result = PyList_New(3);
for (i = 0; i < 3; i++) {
PyObject *o = PyFloat_FromDouble((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
float * summy(std::vector<float> a);
My python result:
>>> import customvector
>>> a = [1,2,3]
>>> customvector.summy(a)
[1.0, 2.0, 3.0]
I then edited my interface file so that the out typemap now uses [ANY] instead of just 3 to allow the length to vary.
Edited customvector.i
/* File: customvector.i */
%module customvector
%{
#define SWIG_FILE_WITH_INIT
#include "customvector.h"
%}
%include "std_vector.i"
%include <stdint.i>
namespace std {
%template(Line) vector < float >;
}
%typemap(out) float* summy [ANY]{ //changed from 3 to [ANY]
int i;
$result = PyList_New($1_dim0); //changed from 3 to $1_dim0
for (i = 0; i < $1_dim0; i++) {
PyObject *o = PyFloat_FromDouble((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
float * summy(std::vector<float> a);
Result from Python:
>>> import customvector
>>> a = [1,2,3]
>>> customvector.summy(a)
<Swig Object of type 'float *' at 0x000001E4E32E6420>
Which is not what I want, and it should display what was shown earlier on.
I tried following the documentation listed somewhere here: http://www.swig.org/Doc2.0/Typemaps.html#Typemaps_nn40 for SWIG to get the value than the output, but it doesn't seem to work.
I also came across this solution to allow length to vary: Python/SWIG: Output an array
but I am not sure how it works as I tried using it but the code doesn't compile (saying that templen is not defined).
How can I output from C++ into python such a data with variable size?

You have undefined behavior in your implementation. Why not use std::vector for the return value as well?
%module test
%include <std_vector.i>
%template() std::vector<float>;
%inline %{
std::vector<float> summy(std::vector<float> a)
{
for(auto& i: a)
i += 1;
return a;
}
%}
Demo:
>>> import test
>>> test.summy([1,2,3,4])
(2.0, 3.0, 4.0, 5.0)
The default behavior for returned vectors is to use a tuple. That can be overridden with a typemap:
%module test
%include <std_vector.i>
%template() std::vector<float>;
%typemap(out) std::vector<float> %{
$result = PyList_New($1.size());
for (int i = 0; i < $1.size(); ++i)
PyList_SET_ITEM($result,i,PyFloat_FromDouble($1[i]));
%}
%inline %{
#include <vector>
std::vector<float> summy(std::vector<float> a)
{
for(auto& i: a)
i += 1;
return a;
}
%}
Demo:
>>> import test
>>> test.summy([1,2,3,4])
[2.0, 3.0, 4.0, 5.0]

Overwrite char (or std::string) array positions with SWIG?

I was able to write a void function in C/C++, and wrap to Python/Numpy with SWIG (int* INPLACE_ARRAY1, int DIM1), that receives a int* vector as parameter, do some math on this vector, and overwrite the results on the same vector, and this result was available inside Python's object. Like follows:
extern "C" void soma_aloc(int* vetor, int tamanho)
{
int m = 0;
int* ponteiro = new int[tamanho];
for(m = 0; m < tamanho; m++)
{
ponteiro[m] = vetor[m];
};
for(m = 0; m < tamanho; m++)
{
ponteiro[m] = ponteiro[m] * 10;
};
for(m = 0; m < tamanho; m++)
{
vetor[m] = ponteiro[m];
};
delete [] ponteiro;
};
This was a test to learn how to wrap pointers to int and double arrays with SWIG using the typemaps (DATA_TYPE* INPLACE_ARRAY1, int DIM1) and (DATA_TYPE* INPLACE_ARRAY2, int DIM1, int DIM2), and worked well.
But the problem is, I've tried the same idea with char/string Numpy vectors (like a vector vec1 = numpy.array(['a','a','a']) or numpy.array(['a','a','a'],dtype=str), and change each position to be like (['b','b','b']), but Python shows in method 'vector_char2D', argument 1 of type 'char *'. It's possible to do the same with char/string?
.cpp:
extern "C" void vetor_char2D(char* vetorchar, int tamanho_vetor)
{
for(int i = 0; i < tamanho_vetor; i++)
{
vetorchar[i] = 'b';
};
};
.i:
%module testestring
%include stl.i
%include std_string.i
%{
#include <stdio.h>
#include <stdlib.h>
//#include <string.h>
#include <string>
#include <iostream>
#define SWIG_FILE_WITH_INIT
#include "testestring.hpp"
%}
%include "numpy.i"
%init %{
import_array();
%}
%apply (char* INPLACE_ARRAY1, int DIM1) {(char* vetorchar, int tamanho_vetor)}
%include "testestring.hpp" (just the header of the above function vetor_char2D)
%clear (char* vetorchar, int tamanho_vetor);
I have very litthe experience with SWIG. It's possible to do this with char*, char** and/or std::string*/std::string** ? Thanks in advance!

Use std::vector:
void vetor_char2D(std::vector<std::string>& vetorchar)
{
for (int i = 0; i < vetorchar.size(); i++)
vetorchar[i] = "b";
};
which indicates clearly that the vector can be modified, and strings within it can be modified, and the SWIG typemaps for STL vectors and strings will work nicely. Note the double rather than single quote for string; Python only has strings no chars so it doesn't matter. You can also make it work with char* etc but it is rarely worth the effort, lot easier to use above. If you don't want to change the source, you can include the above as a wrapper in the .i file via an %inline directive.
Note you should not need the extern C qualifier. And you should use the #include <string> not string.h.

OUT argument internally allocated to return an array of structs

I'm new to swig and I have the following function which i cant fix:
int get_list(IN const char * string, OUT struct entry ** results);
where struct entry is defined:
struct flux_entry
{
char * addr_str;
char cc[2];
};
the entry struct is properly converted to a python class.
I googled but couldn't find any explanation i could use.
I want to make it return a tuple of: (original get_list int return value, python list of entry python objects, based on the results buffer), but don't know how to convert the C entry to a python object in the argout code snippet.
I've managed to get thus far:
%typemap(argout) struct entry **
{
PyObject *o = PyList_New(0);
int i;
for(i=0; $1[i] ; i++)
{
PyList_Append(o, SWIG_HOW_TO_CONVERT_TO_PYOBJECT($1[i]));
}
$result = o;
}
what should i replace SWIG_HOW_TO_CONVERT_TO_PYOBJECT with?
passed results is supposed to be a pointer to a (struct entry *) type, set to NULL before calling get_list and should be set to an allocated array of struct entry * pointers. maybe a small wrapper function could make that easier?
the struct entry array is allocated within the C function using malloc, after calculating (inside get_list) how many elements are needed, and ends with a NULL pointer to indicate the end of the array.
i'd also like to make sure it's freed somewhere :)
thanks!

This should at least give you a starting point that works. I still wasn't sure how the data was returned, since to return an array of pointers so that the final one was NULL I'd think you'd need a struct entry ***, so I just set addr_str = NULL on the last one as a sentinel, and just put some dummy data partially based on the input string into the fields. Modify as needed to suit your needs:
%module example
// Insert the structure definition and function to wrap into the wrapper code.
%{
struct entry {
char* addr_str;
char cc[2];
};
int get_list(const char* string, struct entry** results)
{
*results = malloc(3 * sizeof(struct entry));
(*results)[0].addr_str = malloc(10);
strcpy((*results)[0].addr_str,"hello");
(*results)[0].cc[0] = string[0];
(*results)[0].cc[1] = string[1];
(*results)[1].addr_str = malloc(10);
strcpy((*results)[1].addr_str,"there");
(*results)[1].cc[0] = string[2];
(*results)[1].cc[1] = string[3];
(*results)[2].addr_str = NULL;
return 0;
}
%}
#include <typemaps.i>
// Define the structure for SWIG
struct entry {
char* addr_str;
char cc[2];
};
// Define a set of typemaps to be used for an output parameter.
// This typemap suppresses requiring the parameter as an input.
// A temp variable is created and passed instead.
%typemap(in,numinputs=0) struct entry **OUTPUT (struct entry* temp) %{
$1 = &temp;
%}
// Build a list of tuples containing the two entries from the struct.
// Append the new Python list object to the existing "int" result.
%typemap(argout) struct entry **OUTPUT {
int i = 0;
PyObject* out = PyList_New(0);
while((*$1)[i].addr_str != NULL)
{
//PyObject* t = PyTuple_New(2);
//PyTuple_SET_ITEM(t,0,PyBytes_FromString((*$1)[i].addr_str));
//PyTuple_SET_ITEM(t,1,PyBytes_FromStringAndSize((*$1)[i].cc,2));
//PyList_Append(out,t);
//Py_DECREF(t);
PyObject* s = SWIG_NewPointerObj(*$1+i,$descriptor(struct entry*),0);
PyList_Append(out,s);
Py_DECREF(s);
++i;
}
$result = SWIG_AppendOutput($result,out);
}
// Since a Python object was created and the data copied for each entry struct,
// free the memory returned in the structure.
//%typemap(freearg) struct entry **OUTPUT {
// int i=0;
// while((*$1)[i].addr_str != NULL) {
// free((*$1)[i].addr_str);
// ++i;
// }
// free(*$1);
//}
// Apply the OUTPUT typemap set to the "results" parameter.
%apply struct entry **OUTPUT {struct entry** results};
// Finally, define the function for SWIG
int get_list(const char* string, struct entry** results);
Demo (Python 3.3):
>>> import example
>>> example.get_list('abcd')
[0, [(b'hello', b'ab'), (b'there', b'cd')]]
Hope that helps.
Edit:
I commented out the tuple creation and just save the entry* proxy instead. This doesn't leak Python objects, but the memory malloced for use by an entry* is not freed. I'm not sure where to put that, although I'm experimenting with %extend.