Now, I have called python to C++. Using ctype to connect between both of them. And I have a problem about core dump when in running time.
I have a library which is called "libfst.so"
This is my code.
NGramFST.h
#include <iostream>
class NGramFST{
private:
static NGramFST* m_Instace;
public:
NGramFST(){
}
static NGramFST* getInstance() {
if (m_Instace == NULL){
m_Instace = new NGramFST();
}
return m_Instace;
}
double getProbabilityOfWord(std::string word, std::string context) {
std::cout << "reloading..." << std::endl;
return 1;
}
};
NGramFST.cpp
#include "NGramFST.h"
NGramFST* NGramFST::m_Instace = NULL;
extern "C" {
double FST_getProbability(std::string word, std::string context){
return NGramFST::getInstance()->getProbabilityOfWord(word, context);
}
}
And this is my python code.
from ctypes import cdll
lib = cdll.LoadLibrary('./libfst.so')
#-------------------------main code------------------------
class FST(object):
def __init__(self):
print 'Initializing'
def getProbabilityOfWord(self, word, context):
lib.FST_getProbability(word, context)
fst = FST()
print fst.getProbabilityOfWord(c_wchar_p('jack london'), c_wchar_p('my name is'))
This is error
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
I reviewed again but I can not detect where is my problem.
ctypes does not understand C++ types (it's not called c++types). It cannot handle std::string. It wouldn't know that your function expects std::string arguments anyway.
In order to work with ctypes, your library needs a C-compatible interface. extern "C" is necessary but not sufficient. The functions need to be actually callable from C.
Better yet, use a modern C++/Python binding library such as pybind11.
It work when I change python code below
string1 = "my string 1"
string2 = "my string 2"
# create byte objects from the strings
b_string1 = string1.encode('utf-8')
b_string2 = string2.encode('utf-8')
print fst.getProbabilityOfWord(b_string1, b_string2)
and c++ code change type of param bellow
FST_getProbability(const char* word, const char* context)
Related
I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.
To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.
C++ code for the bindings
VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
std::array<double, 3> pyP{p[0], p[1], p[2]};
PYBIND11_OVERLOAD_PURE(VNCS::Real, /* Return type */
VNCS::BlendingField, /* Parent class */
blending, /* Name of function in C++ (must match Python name) */
pyP /* Argument(s) */
);
}
void VNCS::py::module::blendingField(pybind11::module &m)
{
pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
m, "BlendingField")
.def(pybind11::init(
[](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
.def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}
Python code
class BlendingField(PyVNCS.BlendingField):
def __init__(self, *args, **kwargs):
PyVNCS.BlendingField.__init__(self, *args, **kwargs)
def __del__(self):
print("dtor from python")
def blending(self, point):
return point[0]
blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
name="simCreator",
coarseMesh="#lowResTopology",
detailMesh="#highResTopology")
simCreator.setBlendingField(blendingField)
This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?
I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.
Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:
The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.
It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.
Here's an example lifted from the tests:
// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
MyObject2(const MyObject2 &) = default;
MyObject2(int value) : value(value) { print_created(this, toString()); }
std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
virtual ~MyObject2() { print_destroyed(this); }
private:
int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
.def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });
If you do go that route, which I would recommend, this question would be a duplicate of this one.
Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:
class Child { };
class Parent {
public:
Parent() : child(std::make_shared<Child>()) { }
Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
private:
std::shared_ptr<Child> child;
};
PYBIND11_MODULE(example, m) {
py::class_<Child, std::shared_ptr<Child>>(m, "Child");
py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
.def(py::init<>())
.def("get_child", &Parent::get_child);
}
from example import Parent
print(Parent().get_child()) // UB or Segfault
Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.
Motivation
I am currently trying to use a custom class both in python and in c++ with pybind11. The motivation behind this is to use a python-trained classifier within c++.
There are some working examples, e.g. in the official documentation https://pybind11.readthedocs.io/en/stable/advanced/cast/custom.html or by the nice examples from tdegeus https://github.com/tdegeus/pybind11_examples/blob/master/09_numpy_cpp-custom-matrix/pybind_matrix.h
However, i still have problems to transfer this to a simple example with a custom class.
Below is a minimal working example with a wrapper function for pybind11 in c++ and the use of it in python.
#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;
//A custom data class
class class_DATA{
public:
std::vector<int> a;
};
//the actual function
class_DATA func_test(class_DATA data){
std::vector<int> a = {1,2};
data.a = a;
return data;
}
//the wrapper function that communicates with python
py::object wrapper(py::object data_py){
class_DATA data_cpp = class_DATA();
py::object a = data_py.attr("a");
data_cpp.a = a.cast<std::vector<int>>();
class_DATA data_result_cpp = func_test(data_cpp);
py::object data_result_py = data_py;
data_result_py.attr("a") = py::cast(data_result_cpp.a);
return data_result_py;
}
//defining the modules
PYBIND11_MODULE(TEST,m){
m.doc() = "pybind11 example plugin";
//the test function
m.def("func_test", &wrapper);
//the custom class
py::class_<class_DATA>(m, "class_DATA", py::dynamic_attr())
.def(py::init<>())
.def_readwrite("a", &class_DATA::a);
}
from build.Debug.TEST import func_test
//the custom data class in python
class class_DATA_py:
def __init__(self):
self.a = [0]
test_py = class_DATA_py()
data_return = func_test(test_py)
The output input is data.a = 0 and the output data_return.a = [1,2] as desired.
Question
How can I replace the wrapper function call with a custom caster? Most likely with the general shape outlined below (?)
namespace pybind11 { namespace detail {
template <> struct type_caster<class_DATA> : public type_caster_base<class_DATA> {
using base = type_caster_base<class_DATA>;
public:
PYBIND11_TYPE_CASTER(class_DATA, _("class_DATA"));
// Conversion part 1 (Python->C++):
bool load(py::handle src, bool convert){
PyObject *source = src.ptr();
//what to do here?
return true;
}
// Conversion part 2 (C++ -> Python):
static py::handle cast(class_DATA src, py::return_value_policy policy, py::handle parent){
//what to do here?
return base::cast(src, policy, parent);
}
};
}}
c++ is not my forte and python crashes repeatedly without any error messages when I meddle with this part, so any help would be really appreciated.
I'm trying to embed a Python function in C using PyPy and cffi. I'm following this guide from the PyPy documentation.
The problem is, all the examples I've found operate on ints, and my function takes a string and returns a string. I can't seem to figure out how to embed this function in C, as C doesn't seem to really have strings, rather making do with arrays of chars.
Here's what I've tried:
# interface.py
import cffi
ffi = cffi.FFI()
ffi.cdef('''
struct API {
char (*generate_cool_page)(char url[]);
};
''')
...
#ffi.callback("char[] (char[])")
def generate_cool_page(url):
# do some processing with BS4
return str(soup)
def fill_api(ptr):
global api
api = ffi.cast("struct API*", ptr)
api.generate_cool_page = generate_cool_page
--
// c_tests.c
#include "PyPy.h"
#include <stdio.h>
#include <stdlib.h>
struct API {
char (*generate_cool_page)(char url[]);
};
struct API api; /* global var */
int initialize_api(void)
{
static char source[] =
"import sys; sys.path.insert(0, '.'); "
"import interface; interface.fill_api(c_argument)";
int res;
rpython_startup_code();
res = pypy_setup_home(NULL, 1);
if (res) {
fprintf(stderr, "Error setting pypy home!\n");
return -1;
}
res = pypy_execute_source_ptr(source, &api);
if (res) {
fprintf(stderr, "Error calling pypy_execute_source_ptr!\n");
return -1;
}
return 0;
}
int main(void)
{
if (initialize_api() < 0)
return 1;
printf(api.generate_cool_page("https://example.com"));
return 0;
}
When I run gcc -I/opt/pypy3/include -Wno-write-strings c_tests.c -L/opt/pypy3/bin -lpypy3-c -g -o c_tests and then run ./c_tests, I get this error:
debug: OperationError:
debug: operror-type: CDefError
debug: operror-value: cannot render the type <char()(char *)>: it is a function type, not a pointer-to-function type
Error calling pypy_execute_source_ptr!
I don't have a ton of experience with C and I feel like I'm misrepresenting the string argument/return value. How do I do this properly?
Thanks for your help!
Note that you should not be using pypy's deprecated interface to embedding; instead, see http://cffi.readthedocs.io/en/latest/embedding.html.
The C language doesn't have "strings", but only arrays of chars. In C, a function that wants to return a "string" is usually written
differently: it accepts as first argument a pointer to a pre-existing buffer (of type char[]), and as a second argument the length of that buffer; and when called, it fills the buffer. This can be messy because you ideally need to handle buffer-too-small situations in the caller, e.g. allocate a bigger array and call the function again.
Alternatively, some functions give up and return a freshly malloc()-ed char *. Then the caller must remember to free() it, otherwise a leak occurs. I would recommend that approach in this case because guessing the maximum length of the string before the call might be difficult.
So, something like that. Assuming you start with
http://cffi.readthedocs.io/en/latest/embedding.html, change
plugin.h to contain::
// return type is "char *"
extern char *generate_cool_page(char url[]);
And change this bit of plugin_build.py::
ffibuilder.embedding_init_code("""
from my_plugin import ffi, lib
#ffi.def_extern()
def generate_cool_page(url):
url = ffi.string(url)
# do some processing
return lib.strdup(str(soup)) # calls malloc()
""")
ffibuilder.cdef("""
#include <string.h>
char *strdup(const char *);
""")
From the C code, you don't need initialize_api() at all in the
new embedding mode; instead, you just say #include "plugin.h"
and call the function directly::
char *data = generate_cool_page("https://example.com");
if (data == NULL) { handle_errors... }
printf("Got this: '%s'\n", data);
free(data); // important!
I'm using pybind11 to access some functions which I wrote in C++ from python. I wish to return an initialized struct instance to python(from a class member function) so that I can access the instance values in python. I have provided a simplified version of my source file. One can reproduce the error with the following.
This is the c++ wrapper
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "sum.h"
namespace py = pybind11;
PYBIND11_PLUGIN(GetSum) {
py::module mod("GetSum", "wrapping caltri");
mod.def("get_instance", &GetSum::get_instance, py::return_value_policy::copy, "get instance of out");
py::class_<GetSum, std::shared_ptr<GetSum>>(mod, "GetSum")
.def(py::init<>())
.def("input", &GetSum::input);
return mod.ptr();
}
This is a part of the class header sum.h
extern "C" {
#include "mesh.h"
}
#include<iostream>
#include<vector>
class GetSum {
struct A out;
public:
GetSum();
void input(std::vector<float>);
struct A get_instance() {
return out; };
~GetSum();
};
This is class definition sum.cpp
#include "sum.h"
GetSum::GetSum() {
out.pointlist = nullptr;
}
void GetSum::input(std::vector<float> num){
out.pointlist = new float[num.size()];
for(size_t i = 0; i < num.size(); i++)
*(out.pointlist+i) = num[i];
}
GetSum::~GetSum(){
delete [] out.pointlist;
}
Struct definition: mesh.h
#include <stdio.h>
struct A {
float *pointlist;
};
and this is how I was calling it in python.
import GetSum
m = GetSum.GetSum()
m.input([1.0,2.0])
GetSum.s = GetSum.get_instance()
After which I get the error: get_instance(): incompatible function arguments. The following argument types are supported:
1. (arg0: GetSum) -> A
Can someone help me figure out where I might be going wrong?
Thanks!
You need to declare struct A to pybind11 before it can be used as a return type by adding at minimum:
py::class_<A>(mod, "A")
.def(py::init<>());
to your wrapper code. Further, since get_instance is not static, you need to call it as follows in your example python code:
GetSum.s = GetSum.get_instance(m)
I'm using Boost.Python to create a wrapper for my C++ library, and I'm having some troubles, googling all the day did not produc any results. For example, I have the following code:
class Base
{
public:
virtual void func() = 0;
};
class Derived : public Base
{
public:
virtual void func()
{
cout << "Derived::func()"<< endl;
}
};
// wrapper for Base
struct BaseWrapper : Base, python::wrapper<Base>
{
virtual void func()
{
this->get_override("func");
}
};
Base* makeDerived()
{
return new Derived;
}
vector<Base*>* makeDerivedVec()
{
vector<Base*> *v = new vector<Base*>;
v->push_back(new Derived);
v->push_back(new Derived);
v->push_back(new Derived);
return v;
}
BOOST_PYTHON_MODULE(mylib)
{
// export Base
class_<BaseWrapper, noncopyable>("Base")
.def("func", pure_virtual(&Base::func));
class_<vector<Base*> >("BasePtrVec")
.def(vector_indexing_suite<vector<Base*> >());
// export Derived
class_<Derived, bases<Base> >("Derived")
.def("func", &Derived::func);
// export makeDerived()
def("makeDerived", &makeDerived, return_value_policy<manage_new_object>());
// export makeDerivedVec()
def("makeDerivedVec", &makeDerivedVec, return_value_policy<manage_new_object>());
}
So, I compile it, import in python and try this:
b = mylib.Base() b.func()
d = mylib.makeDerived() d.func()
The first line, as expected, throws an exception saying that b.func() is pure virtual, and the second line prints out
Derived::func()
And that's ok.
But the code
dlist = mylib.makeDerivedVec()
for d in dlist:
d.func()
does not work, and Python throws an exception:
TypeError: No to_python (by-value) converter found for C++ type: Base*
Why it handled correctly the Base* returned by makeDerived() and refuses to work with Base* contained in std::vector? How can I make it work?
You can fix this by registering Base* as a type that can be used to point to a BaseWrapper*:
class_<BaseWrapper, noncopyable, Base*>("Base")
.def("func", pure_virtual(&Base::func));
But it seems that this means that Base can't have a pure virtual function...