We Keep Coding

Object created in Python via Pybind11 not recognized as a derived class by C++ functions

The minimum code to reproduce the issue is as follows:
aaa.hpp
#include <string>
#include <vector>
#include <iostream>
#include <stdexcept>
#include <cmath>
#include <cassert>
#include <utility>
template <typename D>
class BaseClass{
/* The base class. */
protected:
bool skip_nan = true;
};
template <typename D>
class DerivedClass : public BaseClass<D>{
public:
explicit DerivedClass(bool skip_nan_){ this->skip_nan = skip_nan_; }
};
aaa_py.cpp
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <pybind11/stl.h>
#include "aaa.hpp"
namespace py = pybind11;
template <typename D>
void fff(BaseClass<D>& rs){
}
PYBIND11_MODULE(aaa_py, m) {
m.def("fff_float", &fff<float>);
py::class_<DerivedClass<float>>(m, "DerivedClass_float").def(py::init<bool>(), py::arg("skip_nan")=true);
}
test.py
import aaa_py as rsp
ds = rsp.DerivedClass_float()
rsp.fff_float(ds)
Error:
Traceback (most recent call last):
File "usage_example_python.py", line 8, in <module>
rsp.fff_float(ds)
TypeError: fff_float(): incompatible function arguments. The following argument types are supported:
1. (arg0: BaseClass<float>) -> None
Invoked with: <aaa_py.DerivedClass_float object at 0x000002C9EAF57308>
Basically the error is saying that function fff_float expects somebody from BaseClass<float>, but receives DerivedClass<float>. If I change the function to accept DerivedClass<D>&, then there would be no errors. Or, if I create an instantiation of DerivedClass<float> directly in c++ and pass it to the float_fff, there would also be no issues. So it seems that the issue is that for an object created in Python, somehow the information about its base class is lost. How can I deal with this?

You need to declare BaseClass<float> in Python and tell pybind11 that DerivedClass_float extends it.
PYBIND11_MODULE(MyModule, m)
{
m.def("fff_float", &fff<float>);
// declare base class - this simply expose to Python, it's impossible to
// construct a BaseClass_float in Python since no constructor is provided
py::class_<BaseClass<float>>(m, "BaseClass_float");
// tell pybind11 that DerivedClass<float> extends BaseClass<float>
py::class_<DerivedClass<float>, BaseClass<float>>(m, "DerivedClass_float")
// ^^^^^^^^^^^^^^^^
.def(py::init<bool>(), py::arg("skip_nan") = true);
}

How to properly wrap a two dimensional vector using %template

I'm trying to make a basic csv parser in c++ for a particular csv schema, and I'm trying to wrap the function for Python, but I keep getting a "StdVectorTraits not found" warning after wrapper generation. The wrapper is still able to be compiled using g++, but when I try to import the underlying shared object using the object.py script, I get "ImportError: undefined symbol: _Z8myVectorRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"
This is the swig interface file:
%module parser;
%include "/usr/share/swig4.0/std/std_vector.i";
%include "/usr/share/swig4.0/std/std_iostream.i";
%include "/usr/share/swig4.0/std/std_sstream.i";
%include "/usr/share/swig4.0/std/std_string.i";
%include "/usr/share/swig4.0/std/std_basic_string.i";
%{
#include <vector>
#include <fstream>
#include <sstream>
#include <iostream>
std::vector<std::vector<double>> myVector(std::string&);
%}
%template(doubleVector) std::vector<double>;
%template(doubleVecVector) std::vector<std::vector<double>>;
std::vector<std::vector<double>> myVector(std::string& path)
{
std::ifstream file;
std::string read;
std::vector<std::vector<double>> data;
file.open(path);
for (int i = 0; i < 21; i++)
{
std::getline(file, read);
}
for (int i = 0; i < 32; i++)
{
std::vector<double> line;
std::getline(file, read);
std::stringstream ss(read);
for (int j = 0; j < 48; j++)
{
std::getline(ss, read, ',');
line.push_back(std::stof(read));
}
data.push_back(line);
}
file.close();
return data;
}
Error:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/home/../test.ipynb Cell 1 in <cell line: 1>()
----> 1 import parser
File ~/../parser.py:15, in <module>
13 from . import _parser
14 else:
---> 15 import _parser
17 try:
18 import builtins as __builtin__
ImportError: /home/../_parser.so: undefined symbol: _Z8myVectorRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

The function definition should be between %{ and %}. Everything between %{/%} is included directly in the generated wrapper. The function prototype should be at the end of the file after the %template declarations to direct SWIG to generate a wrapper for that function.
Since the function body is in the wrong place it isn't defined hence the undefined symbol error.
Stripped down example:
test.i
%module test
%{
#include <vector>
std::vector<std::vector<double>> myVector()
{
return {{1.0,1.5},{2.0,2.5}};
}
%}
%include <std_vector.i>
%template(doubleVector) std::vector<double>;
%template(doubleVecVector) std::vector<std::vector<double>>;
std::vector<std::vector<double>> myVector();
Demo:
>>> import test
>>> test.myVector()
((1.0, 1.5), (2.0, 2.5))

How to pass bytes object in Python to a C++ function wrapped with Swig?

I have a bytes variable in Python that I want to pass to a C++ function, and then want to return the bytes later on, also via a C++ function. The C++ code is wrapped in Swig. Does anyone know how I can do this or have a basic example? I have tried using void* as the type in C++, but I can't get it to work.
C++ example.hpp code:
class MyTestClass {
public:
void set_data(void* data);
void* get_data();
private:
void* data;
};
C++ example.cpp code:
void MyTestClass::set_data(void* data)
{
this->data = data;
}
void* MyTestClass::get_data()
{
return data;
}
Swig interface file:
/* File : example.i */
%module example
%include <std_string.i>
%{
#include "example.hpp"
%}
%include "example.hpp"
Test Python code:
from example import MyTestClass
test_class = MyTestClass()
data1 = b"test"
test_class.set_data(data1)
data2 = test_class.get_data()
print(data2)
Compilation:
swig -c++ -python -modern -py3 example.i
export CC=g++
python setup.py build_ext --inplace
Error:
Traceback (most recent call last):
File "./test.py", line 6, in <module>
test_class.set_data(data_1)
File "example.py", line 119, in set_data
return _example.MyTestClass_set_data(self, data)
TypeError: in method 'MyTestClass_set_data', argument 2 of type 'void *'

You should use char* instead of void*.
In addition, you should not use the
%include <std_string.i>
in the example.i file, as that has the typemap definition converting char * to a string.
Minor issue: you example.cpp shall #include "example.hpp" to have a definition for MyTestCLass.

I managed to get it working with a cstring.i template:
%cstring_output_allocate_size(char** data_out, int* maxdata, NULL)
When I set the data I also set the size, which I use to set maxdata. The third parameter is NULL and not free because the data is meant to be a reference only.

Using a DLL in python - cannot locate functions or classes

I'm currently struggling to get some python code to locate functions within a DLL file.
I have looked at several posts on here about this and the various methods don't seem to work for me.
Please ignore the fact that the code could be implements using the GUI library in Python, this is not an option for me.
So my header file is as follows:
#pragma once
#ifdef MOBILEFUNC_EXPORTS
#define MOBILEFRONTEND_API __declspec(dllexport)
#else
#define MOBILEFRONTEND_API __declspec(dllimport)
#endif
#ifdef __cplusplus
class mobileFrontEnd{
public:
char* getPath();
};
#else typedef struct _frontEnd frontEnd;
#endif
#ifdef __cplusplus
extern "C" {
#endif
MOBILEFRONTEND_API mobileFrontEnd *frontend_create();
MOBILEFRONTEND_API char* getPath();
#ifdef __cplusplus
}
#endif
using namespace System;
using namespace System::Windows::Forms;
namespace frontEnd {
class MOBILEFRONTEND_API mobileFrontEnd
{
public:
static char* getPath();
};
}
And my main C++ file is:
#include "stdafx.h"
#include "mobileFrontEnd.h"
using namespace::Runtime::InteropServices;
namespace frontEnd
{
mobileFrontEnd *frontend_create()
{
mobileFrontEnd *self = new mobileFrontEnd;
return self;
}
char* mobileFrontEnd::getPath()
{
FolderBrowserDialog^ openDialog1 = gcnew FolderBrowserDialog();
if (openDialog1->ShowDialog() == DialogResult::OK)
{
String^ path = openDialog1->SelectedPath;
return (char*)(void*)Marshal::StringToHGlobalAnsi(path);
}
else
{
return 0;
}
}
}
The DLL imports using the CDLL or WinDLL functions in python, however any attempts to access functions or class result in an error stating the class/function does not exists.
I don't have any real python code for this so fa as I've been attempting to check it in python command prompt.
Am I missing something to ensure it exports functions correctly?
Edited with some python code:
So something similar to this (from http://eli.thegreenplace.net/2008/08/31/ctypes-calling-cc-code-from-python)
import ctypes
>>> test_dll = ctypes.CDLL("C:\\Users\\xxxx\\Documents\\Visual Studio 2012\\Projects\\mobileFrontEnd\\Release\\mobilefrontend.dll")
>>> test_cb = test_dll.getPath();
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
test_cb = test_dll.getPath();
File "C:\Python27\lib\ctypes\__init__.py", line 378, in __getattr__
func = self.__getitem__(name)
File "C:\Python27\lib\ctypes\__init__.py", line 383, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'getPath' not found
>>>
Edit 2:
Also just in case it isn't clear from the code (Windows form being used), the DLL is compiled in Visual Studio 2012 express and include 'common library runtime' support

Using the following code seems to work for accessing the functions:
import ctypes
test_dll = ctypes.CDLL("**path to dll**")
test1 = ctypes.WINFUNCTYPE(None)
test2 = test1(("getPath",test_dll))
test2()
No idea why the functions cannot be seen in the attributes of test_dll

Wrapping C code including Python API using SWIG and distutils fails on Mac 10.8 64bits

I have been trying to wrap an existing C code into a Python module for some time now and I keep running into recurrent errors and a failed build... After an extensive look at the available documentation and posts on this forum I would like to know if someone has an idea why it is not working.
I would like to use the stsci.imagemanip.interp2d.expand2d() function which consists of a python script calling a C function which uses the Python.h library. This under Mac 10.8 64bits with Python 2.7.
I am new to wrapping C functions to Python and I decided to use SWIG and distutils.
My input files are the following:
bilinearinterp.i
/* bilinearinterp.i */
%module bilinearinterp
%{
#define SWIG_FILE_WITH_INIT
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <stdio.h>
#include <Python.h>
#include <math.h>
#include <numpy/arrayobject.h>
#include "bilinearinterp.h"
%}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
bilinearinterp.c
/* File: bilinearinterp.c */
#include <Python.h>
#include <stdio.h>
#include <math.h>
#include <numpy/arrayobject.h>
#include "bilinearinterp.h"
static void InterpInfo (float ai, int npts, int *i, float *p, float *q) {
/* arguments:
float ai i: independent variable in same units as i
int npts i: size of array within which *i is an index
int *i o: array index close to ai
float *p, *q o: weights for linear interpolation
*/
DO SOME CALCULATIONS
}
int unbin2d (float *a, float *b, int inx, int iny, int onx, int ony) {
/* arguments:
PyArrayObject *a i: input data
PyArrayObject *b o: output data
*/
DO SOME CALCULATIONS
CALL InterpInfo
return (1);
}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args)
{
PyObject *input, *output;
PyArrayObject *dataa, *datab;
int inx, iny, onx, ony;
int status=0;
if (!PyArg_ParseTuple(args,"OO:bilinearinterp",&input,&output))
return NULL;
dataa = (PyArrayObject *)PyArray_ContiguousFromAny(input, PyArray_FLOAT, 1, 2);
datab = (PyArrayObject *)PyArray_ContiguousFromAny(output, PyArray_FLOAT, 1, 2);
inx = PyArray_DIM(input,0);
iny = PyArray_DIM(input,1);
onx = PyArray_DIM(output,0);
ony = PyArray_DIM(output,1);
status = unbin2d((float *)dataa->data,(float *)datab->data, inx, iny, onx, ony);
Py_XDECREF(dataa);
Py_XDECREF(datab);
return Py_BuildValue("i",status);
}
static PyMethodDef bilinearinterp_methods[] =
{
{"bilinearinterp", bilinearinterp, METH_VARARGS,
"bilinearinterp(input, output)"},
{0, 0} /* sentinel */
};
void initbilinearinterp(void) {
Py_InitModule("bilinearinterp", bilinearinterp_methods);
import_array();
}
bilinearinterp.h
/* File: bilinearinterp.H */
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
setup.py
#!/usr/bin/env python
from distutils.core import setup, Extension
bilinearinterp_module = Extension('_bilinearinterp',
sources = ['bilinearinterp_wrap.c','bilinearinterp.c'],
# Path to locate numpy/arrayobject.h
include_dirs=['/Library/Python/2.7/site-packages/numpy-override/numpy/core/include'])
setup (name = 'bilinearinterp',
version = '1.1',
author = "Space Telescope Science Institute - stsci_python",
description = """bilinear interpretation for 2D array extrapolation""",
ext_modules = [bilinearinterp_module],
py_modules = ["bilinearinterp"],
)
Then I run the following in Terminal:
>>>swig -python bilinearinterp.i
>>>python setup.py build_ext --inplace
The appropriate files are being created:
_bilinearinterp.so bilinearinterp.py bilinearinterp.pyc bilinearinterp_wrap.c
As well as a build directory containing some files in it.
I get warnings when setup.py is running but it seems to complete.
Over the various tests I did two recurrent errors keep coming back:
warning: "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
Which may induces several other warnings associated to numpy
warning: function 'bilinearinterp' has internal linkage but is not defined [-Wundefined-internal] static PyObject * bilinearinterp(PyObject *obj, PyObject *args); bilinearinterp_wrap.c:2971:24: note: used here result = (PyObject *)bilinearinterp(arg1,arg2);
Then when calling the module in a python script I get the following:
>>> from bilinearinterp import bilinearinterp as lininterp
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "bilinearinterp.py", line 28, in <module>
_bilinearinterp = swig_import_helper()
File "bilinearinterp.py", line 24, in swig_import_helper
_mod = imp.load_module('_bilinearinterp', fp, pathname, description)
ImportError: dlopen(./_bilinearinterp.so, 2): Symbol not found: _bilinearinterp
Referenced from: ./_bilinearinterp.so
Expected in: flat namespace
in ./_bilinearinterp.so
Would someone have an idea from where the error could be coming from?
Is it from my usage of SWIG or from setup.py?
Another approach with only a swig interface file and the C code (without including the *.h file) does not work. Based on the swig introduction chapter
bilinearinterp.i
/* bilinearinterp.i */
%module bilinearinterp
%{
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
%}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
And I compile using:
>>>swig -python -o bilinearinterp_wrap.c bilinearinterp.i
>>>gcc -c -fpic bilinearinterp.c bilinearinterp_wrap.c
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-I/System//Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include
bilinearinterp_wrap.c:2955:19: warning: function 'bilinearinterp' has internal
linkage but is not defined [-Wundefined-internal]
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
^
bilinearinterp_wrap.c:2971:24: note: used here
result = (PyObject *)bilinearinterp(arg1,arg2);
^
1 warning generated.
>>>gcc -bundle -flat_namespace -undefined suppress bilinearinterp.o bilinearinterp_wrap.o -o _bilinearinterp.so
Then I still get the same import error in the python interpreter.
Could it be due to my numpy installation or my gcc arguments?
Thanks

Well, I don't quite understand what you are trying to achieve, but I see some bad practices. For example, you make bilinearinterp everywhere static, and static functions are not exported and only can be used in the same module where implemented. I think it would be helpful to read a bit about SWIG design ideas. For sure it will take a some time, but you will get a broader perspective of what SWIG is and isn't capable of.