We Keep Coding

Why does this trivial usage of PyArray_SimpleNewFromData segfault?

I'm trying to use NumPy in a C extension for python.
I'm getting a segfault I can't explain, I've narrowed it down to this simple example.
#include "numpy/arrayobject.h"
int main()
{
int dims[] = {1};
double d[] = {1};
PyArray_SimpleNewFromData(1, dims, NPY_DOUBLE, &d);
}

https://docs.scipy.org/doc/numpy/user/c-info.how-to-extend.html
This function must be declared so that it is visible to code outside
of the routine. Besides adding the methods and constants you desire,
this subroutine must also contain calls like import_array() and/or
import_ufunc() depending on which C-API is needed. Forgetting to place
these commands will show itself as an ugly segmentation fault (crash)
as soon as any C-API subroutine is actually called.
I also had to include Py_Initialize() beforehand since this was a standalone example not being run through a Python vm as it would normally be.

Cython: std::function callbacks with custom parameter types

Please read this post before answering: Pass a closure from Cython to C++
In the accepted answers, it is neatly shown how a python function is converted into a std::function using Boost Python.
Following this example I'm able to wrap functions taking an std::function as an argument and call them using a python function as input. However, this only works when the std::function parameters are primitives like int, double, string etc.
Any guidance on how to make this work for custom types as well will be highly appreciated.

This won't be a complete answer - it assumes you can fill in the gaps from my previous answer that the question was based on. Unfortunately it is a little bit more complicated than that case.
Just to define the problem - assume you have a parameter of a custom C++ class, like:
class cpp_class {
// some non-trivial contents
};
and thus your C++ interface looks like this:
void call_some_std_func(std::function<void(cpp_class&)> callback) {
callback(5,std::string("hello"));
}
The first thing to do is to write a Cython wrapper for your C++ class (in principle you could make a Boost Python wrapper instead). Here you need to make a choice of about "ownership" of the C++ object. The first choice is to make a copy:
cdef extern from "cpp_file.hpp":
cppclass cpp_class:
pass # details
cdef class CyWrapper:
cdef cpp_class* ptr
def __dealloc__(self):
del self.ptr
# other details following standard wrapper pattern
cdef public make_CyWrapper(cpp_class& x):
obj = CyWrapper()
obj.ptr = new cpp_class(x)
return obj
I've created a wrapper class with a destructor that handles the memory and a publicly accessible constructor function that can be called from external code. This version is safe because your wrapper owns the object it holds and so there can be no writes to invalid memory. However, because it makes a copy, you can't make changes to the original C++ object.
A second option is to hold a pointer to an object you don't own. The code is basically identical except you remove the __dealloc__ and avoid making a copy in make_CyWrapper:
obj.ptr = &x // instead of new cpp_class(x)
This is unsafe - you need to ensure your C++ object outlives the Cython wrapper - but allows you to modify the object.
You could also imagine a few other options: you could take ownership of an existing object with your Cython wrapper (Such a scheme would have to pass by pointer rather than reference, or it could use move constructors); you could deconstruct your C++ class into a representation expressed in basic types and pass those to Python; you could use shared pointers to split the ownership; or you have a more elaborate way of marking your Cython wrapper as "invalid" once your held C++ instance is destructed.
What you do next depends on whether you're using Boost Python (for it's convenient, callable wrapping of Python objects) or if you're making your own version. (I showed both possibilities in the previous answer).
Assuming Boost Python, you need to do two things - tell it about the conversion and make sure that it imports the module that your wrapper is defined in (if you don't do this you get exciting segmentation faults)
struct convert_to_PyWrapper {
static PyObject* convert(const cpp_class& rhs) {
// the const_cast here is a bit dodgy, but was needed to make it work
return make_CyWrapper(const_cast<cpp_class&>(rhs));
}
};
inline void setup_boost_python() {
PyInit_your_module_name(); // named inityour_module_name in Python 2
boost::python::to_python_converter<
cpp_class,
convert_to_PyWrapper>();
}
You need to make sure that your Python/Cython code calls "setup_boost_python" before attempting to use the callback (if you put it at module level it's done on import, which is ideal).
If you're following my "manual" scheme (avoiding the dependency on Boost Python) then you need to modify the call_obj Cython function that does the C++ to Cython type conversion.
cdef public void call_obj(obj, cpp_class& c):
obj(make_CyWrapper(c))
You also need to ensure the wrapper Cython module is imported before use (otherwise you get segmentation faults). I did this in "py_object_wrapper.hpp" but providing it's done once somewhere you can place it where you like.
void operator()(cpp_class& a) {
PyInit_your_module_name();
if (held) {
call_obj(held,a);
}
}

Running python code in a python object in boost::python

I'm writing a library for python in C++ using boost::python and I've hit a snag.
I'd like to take an object (in this case, an np.array) and run some python code on it (in this case, the code that gets the data pointer from an array). The problem is that the boost tutorial doesn't specifically say how to get a python object that is in the C++ code and pass it to the python code you want to run.
Here is a sketch of what I want to do:
#include <boost/python.hpp>
#include <cctypes>
void* get_data_ptr(boost::python::object& array) {
boost::python::object main_module = import("__main__");
boost::python::object main_namespace = main_module.attr("__dict__");
boost::python::object ignored = exec(
"import ctypes\n"
"temp = ctypes.c_void_p.from_buffer(array)\n"
"py_ptr = ctypes.addressof(temp)\n",
main_namespace); // array would have to go to python somehow
uintptr_t ptr = extract<uintptr_t>(main_namespace["py_ptr"]);
// Some operation on the np array
}
How would I do that?
PS: It is important that my method does not have collateral effects, I.E. does not add anything such as a variable to the global namespace that would alter the python code that is running on top of it.

SWIG argument error when using "using std::vector" in python

This is very related to this question
Regardless of whether or not this is coding practice, I have come across code that looks like this
test.hh
#include <vector>
using std::vector;
class Test
{
public:
vector<double> data;
};
I am trying to swig this using swig3.0 using the following interface file
test.i
%module test_swig
%include "std_vector.i"
namespace std {
%template(VectorDouble) vector<double>;
};
%{
#include "test.hh"
%}
%naturalvar Test::data;
%include "test.hh"
And the following test code
test.py
t = test.Test()
jprint(t)
a = [1, 2, 3]
t.data = a # fails
doing so gives me the following error
in method 'Test_data_set', argument 2 of type 'vector< double >'
This can be fixed by either changing the using std::vector in test.hh to using namespace std or by removing using std::vector and changing vector<double> to std::vector<double>. This is not what I want.
The problem is that I was given this code as is. I am not allowed to make changes, but I am supposed to still make everything available in python via SWIG. What's going on here?
Thanks in advance.

To me, this looks like SWIG does not support the using std::vector; statement correctly. I think it's a SWIG bug. I can think of the following workarounds:
Add using namespace std; to the SWIG interface file (this will only affect the way wrappers are created; the using statement will not enter C++ code)
Add #define vector std::vector to the SWIG interface file (this will only work if vector is never used as std::vector)
Copy the declarations from the header file to the SWIG interface file, and change vector to std::vector. This will cause SWIG to generate correct wrappers, and again will not affect the C++ library code.

Python NumPy arrays to boost::multi_array numpy_boost setup to interface between C++ and NumPy

I have a C/C++ DLL with libraries I need to call from NumPy, where the format inside the DLL is all boost::multi_array format. While I have seen some posts involving this project: https://github.com/mdboom/numpy-boost
there really is very little documentation and examples from users out there on the steps required to wrap the boost::multi_array to NumPy interface with this library. My questions: while the examples are enough to guess the C++ part of the interface (other than strings, how is this handled? Q1), what do you have to do on the Python side to use the compiled DLL (Q2)? It seems Ctypes wouldn't work (correct me if I'm wrong) so does this have to be rewritten in Cython?
For reference, this is the C part with all the datatypes I'm trying to wrap:
extern "C"
{
DECLDIR void Cfunction(
boost::multi_array<double, 2>& p_result,
const vector<string>& p_calcType,
const string& p_optionType,
unsigned long p_nTimeStep = 50,
const vector<double>& p_premium = vector<double>());

With ctypes you can load DLL dynamically and call functions from it. From my point of view, it's easiest way for using compiled code within Python code. Here is how: How can I use a DLL file from Python?
numpy allows you to get pointer to memory location. Look at that: http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.ndarray.ctypes.html
If you don't have source code of your DLL and cannot change its interface, you create C++ wrapper (your own separate DLL) that accepts raw pointer from Python instead of boost::multi_array.