I have a C++ code which execute python script with boost_python package. Everything is fine, as longa as I extract int, string, or other not-array variables from python. However I have to extract a numpy::ndarray and convert it to cpp vector. I tried as follow:
main.cpp
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/numpy.hpp>
using namespace boost::python;
int main()
double t_end=7
try
{
Py_Initialize();
object module = import("__main__");
object name_space = module.attr("__dict__");
exec_file("MyModule.py", name_space, name_space);
object MyFunc = name_space["MyFunc"];
object result = MyFunc(t_end);
auto result_array = extract<numpy::ndarray>(result);
const numpy::ndarray& ret = result_array();
int input_size = ret.shape(0);
double* input_ptr = reinterpret_cast<double*>(ret.get_data());
std::vector<double> v(input_size);
for (int i = 0; i < input_size; ++i)
v[i] = *(input_ptr + i);
}
catch (error_already_set)
{
PyErr_Print();
}
Py_Finalize();
And example py script:
MyModule.py
import numpy as np
def MyFunc(t_end):
result = np.array([2,3,1,t_end])
return results
However it ends with error:
read access violation BOOST_NUMPY_ARRAY_API was nullptr
I also was trying to declare numpy::ndarray directly like numpy::ndarray result_array = extract<numpy::ndarray>(result); But the error is exactly the same. I've checked if my ndarray is not empty by printing it directly from python, and it is not. At the python step all seems to be correct. So what is causing the violation and how to fix it?
That error occurs since you're using the numpy module without first initializing it.
Notice the beginning of the official tutorial:
Initialise the Python runtime, and the numpy module. Failure to call these results in segmentation errors:
namespace np = boost::python::numpy;
int main(int argc, char **argv)
{
Py_Initialize();
np::initialize();
Your code is lacking the call to np::initialize();.
Related
Background
I have some functions which are written in C++, which require high real-time performance. I want to quickly export these functions as dynamic link library to be exposed to Python so that I could do some high level programming.
In these functions, in order to simply usage, I use PyList_New in <Python.h> to collect some intermedia data. But I met some errors.
Code Example
I found the core problem is that I CAN'T event export a python object. After compiling the source to dll and use ctypes to load it, result shows
OSError: exception: access violation reading 0x0000000000000008
C++ code:
#include <Python.h>
#ifdef _MSC_VER
#define DLL_EXPORT __declspec( dllexport )
#else
#define DLL_EXPORT
#endif
#ifdef __cplusplus
extern "C"{
#endif
DLL_EXPORT PyObject *test3() {
PyObject* ptr = PyList_New(10);
return ptr;
}
#ifdef __cplusplus
}
#endif
Python test code:
if __name__ == "__main__":
import ctypes
lib = ctypes.cdll.LoadLibrary(LIB_DLL)
test3 = lib.test3
test3.argtypes = None
test3.restype = ctypes.py_object
print(test3())
Environment Config
Clion with Microsoft Visual Studio 2019 Community, and the arch is amd64.
I know that, the right way is to use the recommanded method to wrap C++ source using Python/C Api to a module, but it seems that I have to code a lot. Anyone can help?
ctypes is normally for calling "regular" C functions, not Python C API functions, but it can be done. You must use PyDLL to load a function that uses Python, as it won't release the GIL (global intepreter lock) required to be held when using Python functions. Your code as shown is invalid, however, because it doesn't populate the list it creates (using OP code as test.c):
>>> from ctypes import *
>>> lib = PyDLL('./test')
>>> lib.test3.restype=py_object
>>> lib.test3()
[<NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>]
Instead, write a C or C++ function normally:
test.cpp
#ifdef _MSC_VER
#define DLL_EXPORT __declspec( dllexport )
#else
#define DLL_EXPORT
#endif
#ifdef __cplusplus
extern "C"{
#endif
DLL_EXPORT int* create(int n) {
auto p = new int[n];
for(int i = 0; i < n; ++i)
p[i] = i;
return p;
}
DLL_EXPORT void destroy(int* p) {
delete [] p;
}
#ifdef __cplusplus
}
#endif
test.py
from ctypes import *
lib = CDLL('./test')
lib.create.argtypes = c_int,
lib.create.restype = POINTER(c_int)
lib.destroy.argtypes = POINTER(c_int),
lib.destroy.restype = None
p = lib.create(5)
print(p) # pointer to int
print(p[:5]) # convert to list...pointer doesn't have length so slice.
lib.destroy(p) # free memory
Output:
<__main__.LP_c_long object at 0x000001E094CD9DC0>
[0, 1, 2, 3, 4]
I solved it by myself. Just change to Release and all the problems are solved.
I am creating a wrapper to a code in c for Python. The c code basically runs in terminal and has the following main function prototype:
void main(int argc, char *argv[]){
f=fopen(argv[1],"r");
f2=fopen(argv[2],"r");
So basically arguments read are strings in terminal. I created following python ctype wrapper, but it appears I am using wrong type. I know the arguments passed from the terminal is read as characters but an equivalent python side wrapper is giving following error:
import ctypes
_test=ctypes.CDLL('test.so')
def ctypes_test(a,b):
_test.main(ctypes.c_char(a),ctypes.c_char(b))
ctypes_test("323","as21")
TypeError: one character string expected
I have tried adding one character, just to check if shared object gets executed, it does as print commands work but momentarily till the section of the code in shared object needs file name. I also tried
ctypes.c_char_p but get.
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Updated as per the suggestion in the comments to the following:
def ctypes_test(a,b):
_test.main(ctypes.c_int(a),ctypes.c_char_p(b))
ctypes_test(2, "323 as21")
Yet getting the same error.
Using this test DLL for Windows:
#include <stdio.h>
__declspec(dllexport)
void main(int argc, char* argv[])
{
for(int i = 0; i < argc; ++i)
printf("%s\n", argv[i]);
}
This code will call it. argv is basically a char** in C, so the ctypes type is POINTER(c_char_p). You also have to pass bytes strings and it can't be a Python list. It has to be an array of ctypes pointers.
>>> from ctypes import *
>>> dll = CDLL('./test')
>>> dll.main.restype = None
>>> dll.main.argtypes = c_int, POINTER(c_char_p)
>>> args = (c_char_p * 3)(b'abc', b'def', b'ghi')
>>> dll.main(len(args), args)
abc
def
ghi
I'm trying to figure out why I can't simply get and set the python path through its C API. I am using Python3.6, on Ubuntu 17.10 with gcc version 7.2.0. Compiling with:
gcc pytest.c `python3-config --libs` `python3-config --includes`
#include <Python.h>
int main()
{
Py_Initialize(); // removes error if put after Py_SetPath
printf("setting path\n"); // prints
Py_SetPath(L"/usr/lib/python3.6"); // Error in `./a.out': free(): invalid size: 0x00007fd5a8365030 ***
printf("success\n"); // doesn't print
return 0;
}
Setting the path works fine, unless I also try to get the path prior to doing so. If I get the path at all, even just to print without modifying the returned value or anything, I get a "double free or corruption" error.
Very confused. Am I doing something wrong or is this a bug? Anyone know a workaround if so?
Edit: Also errors after calling Py_Initialize();. Updated code. Now errors even if I don't call Py_GetPath() first.
From alk it seems related to this bug: https://bugs.python.org/issue31532
Here is the workaround I am using. Since you can't call Py_GetPath() before Py_Initialize(), and also seemingly you can't call Py_SetPath() after Py_Initialize(), you can add to or get the path like this after calling Py_Initialize():
#include <Python.h>
int main()
{
Py_Initialize();
// get handle to python sys.path object
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
// make a list of paths to add to sys.path
PyObject *newPaths = PyUnicode_Split(PyUnicode_FromWideChar(L"a:b:c", -1), PyUnicode_FromWideChar(L":", 1), -1);
// iterate through list and add all paths
for(int i=0; i<PyList_Size(newPaths); i++) {
PyList_Append(path, PyList_GetItem(newPaths, i));
}
// print out sys.path after appends
PyObject *newlist = PyUnicode_Join(PyUnicode_FromWideChar(L":", -1), path);
printf("newlist = %ls\n", PyUnicode_AsWideCharString(newlist, NULL));
return 0;
}
[the below answer refers to this version of the question.]
From the docs:
void Py_Initialize()
Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the exception of Py_SetProgramName(), Py_SetPythonHome() and Py_SetPath().
But the code you show does call Py_GetPath() before it calls Py_Initialize();, which it per the above paragraph implicitly should not.
I checked already a lot of posts and the subprocess documentation but non of them provided a solution to my problem. At least, i can't find one.
Anyway, here is my problem description:
I would like to call a .exe from a .py file. The .exe needs a integer input argument and returns also an integer value, which i would like to use for further calculations in python.
In order to keep things simple, i would like to use a minimun working example of my "problem"-code (see below). If i run this code, then .exe crashes and i don't know why. Maybe i just missed something but i don't know what!? So here is what i did:
c++ code which i use to generate: MyExe.exe
#include <iostream>
using namespace std;
#include <stdlib.h>
#include <string>
int main(int argc, char* argv[])
{
int x = atoi(argv[1]);
return x;
}
My python code:
from subprocess import Popen, PIPE
path = 'Path to my MyExe.exe'
def callmyexe(value):
p = Popen([path], stdout=PIPE, stdin=PIPE)
p.stdin.write(bytes(value))
return p.stdout.read
a = callmyexe(5)
b = a + 1
print(b)
I use MSVC 2015 and Python 3.6.
You have to use cout for output:
#include <iostream>
using namespace std;
#include <stdlib.h>
#include <string>
int main(int argc, char* argv[])
{
int x = atoi(argv[1]);
cout << x;
}
And command line parameters for the input:
from subprocess import check_output
path = 'Path to my MyExe.exe'
def callmyexe(value):
return int(check_output([path, str(value)]))
a = callmyexe(5)
b = a + 1
print(b)
I am trying to create a python library from a class which uses opencv 2.3. I want to be able to pass numpy array's into the class where they will be converted into cv::Mat's processed then converted back to numpy array's and returned.
Here is a simple test class I am working on to get this working before wrapping my own class. Currently I am just trying to receive a numpy array concert to a cv::Mat, process it and then write it to file. After this is working I will work on returning the processed array to python.
Here is the simple class:
foo.h :
#include <opencv2/core/core.hpp>
class Foo {
public:
Foo();
~Foo();
cv::Mat image;
void bar( cv::Mat in );
};
foo.cpp :
#include "foo.h"
Foo::Foo(){}
Foo::~Foo(){}
void Foo::bar( cv::Mat in) {
image = in;
cv::Canny( image, image, 50, 100 );
cv::imwrite("image.png", image);
}
And here is where I have attempted to wrap this class using boost::python (I am using code from the opencv source for the the numpy to mat conversion)
wrap_foo.cpp
#include <boost/python.hpp>
#include <numpy/arrayobject.h>
#include <opencv2/core/core.hpp>
#include "foo.h"
using namespace cv;
namespace bp = boost::python;
//// Wrapper Functions
void bar(Foo& f, bp::object np);
//// Converter Functions
cv::Mat convertNumpy2Mat(bp::object np);
//// Wrapper Functions
void bar(Foo& f, bp::object np)
{
Mat img = convertNumpy2Mat(np);
f.bar(img);
return;
}
//// Boost Python Class
BOOST_PYTHON_MODULE(lib)
{
bp::class_<Foo>("Foo")
.def("bar", bar)
;
}
//// Converters
cv::Mat convertNumpy2Mat(bp::object np)
{
Mat m;
numpy_to_mat(np.ptr(),m);
return m;
}
The numpy_to_mat function is from the opencv source (modules/python/src2/cv2.cpp). The full file has the function below what I wrote above. This code compiles with bjam just fine but the when I import into python it crashes. The error is this: libFoo.so: undefined symbol: _ZN2cv3Mat10deallocateEv. I have tried a number of different things but I can't get this to work.
Help is most appreciated.
I think this is probably a bit late but it may be useful to others who experienced the same problem...
I think you need to add the path to the newly created library to your LD_LIBRARY_PATH for your program to locate it.
Assuming the current directory '.' is where your library is at, type the following in your terminal before running your program:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
NOTE: The above is temporary export is temporary. You might want to copy your libs to standard library paths such as /usr/local/lib or add the path permanently by including the above command in your .profile (or any shell startup configuration file).