We Keep Coding

Increase ref_count of pybind11 handle in C++

I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.
To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.
C++ code for the bindings
VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
std::array<double, 3> pyP{p[0], p[1], p[2]};
PYBIND11_OVERLOAD_PURE(VNCS::Real, /* Return type */
VNCS::BlendingField, /* Parent class */
blending, /* Name of function in C++ (must match Python name) */
pyP /* Argument(s) */
);
}
void VNCS::py::module::blendingField(pybind11::module &m)
{
pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
m, "BlendingField")
.def(pybind11::init(
[](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
.def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}
Python code
class BlendingField(PyVNCS.BlendingField):
def __init__(self, *args, **kwargs):
PyVNCS.BlendingField.__init__(self, *args, **kwargs)
def __del__(self):
print("dtor from python")
def blending(self, point):
return point[0]
blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
name="simCreator",
coarseMesh="#lowResTopology",
detailMesh="#highResTopology")
simCreator.setBlendingField(blendingField)
This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?

I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.
Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:
The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.
It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.
Here's an example lifted from the tests:
// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
MyObject2(const MyObject2 &) = default;
MyObject2(int value) : value(value) { print_created(this, toString()); }
std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
virtual ~MyObject2() { print_destroyed(this); }
private:
int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
.def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });
If you do go that route, which I would recommend, this question would be a duplicate of this one.
Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:
class Child { };
class Parent {
public:
Parent() : child(std::make_shared<Child>()) { }
Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
private:
std::shared_ptr<Child> child;
};
PYBIND11_MODULE(example, m) {
py::class_<Child, std::shared_ptr<Child>>(m, "Child");
py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
.def(py::init<>())
.def("get_child", &Parent::get_child);
}
from example import Parent
print(Parent().get_child()) // UB or Segfault
Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.

How can you bind exceptions with custom fields and constructors in pybind11 and still have them function as python exception?

This appears to be a known limitation in pybind11. I read through all the docs, whatever bug reports seemed applicable, and everything I could find in the pybind11 gitter. I have a custom exception class in c++ that contains custom constructors and fields. A very basic example of such a class, trimmed for space is here:
class BadData : public std::exception
{
public:
// Constructors
BadData()
: msg(),
stack(),
_name("BadData")
{}
BadData(std::string _msg, std::string _stack)
: msg(_msg),
stack(_stack),
_name("BadData")
{}
const std::string&
getMsg() const
{
return msg;
}
void
setMsg(const std::string& arg)
{
msg = arg;
}
// Member stack
const std::string&
getStack() const
{
return stack;
}
void
setStack(const std::string& arg)
{
stack = arg;
}
private:
std::string msg;
std::string stack;
std::string _name;
I currently have python binding code that binds this into python, but it is custom generated and we'd much rather use pybind11 due to its simplicity and compile speed.
The default mechanism for binding an exception into pybind11 would look like
py::register_exception<BadData>(module, "BadData");
That will create an automatic translation between the C++ exception and the python exception, with the what() value of the c++ exception translating into the message of the python exception. However, all the extra data from the c++ exception is lost and if you're trying to throw the exception in python and catch it in c++, you cannot throw it with any of the extra data.
You can bind extra data onto the python object using the attr and I even went somewhat down the path of trying to extend the pybind11:exception class to make it easier to add custom fields to exceptions.
template <typename type>
class exception11 : public ::py::exception<type>
{
public:
exception11(::py::handle scope, const char *name, PyObject *base = PyExc_Exception)
: ::py::exception<type>(scope, name, base)
{}
template <typename Func, typename... Extra>
exception11 &def(const char *name_, Func&& f, const Extra&... extra) {
::py::cpp_function cf(::py::method_adaptor<type>(std::forward<Func>(f)),
::py::name(name_),
::py::is_method(*this),
::py::sibling(getattr(*this, name_, ::py::none())),
extra...);
this->attr(cf.name()) = cf;
return *this;
}
};
This adds a def function to exceptions similar to what is done with class_. The naive approach to using this doesn't work
exception11< ::example::data::BadData>(module, "BadData")
.def("getStack", &::example::data::BadData::getStack);
Because there is no automatic translation between BadData in c++ and in python. You can try to work around this by binding in a lambda:
.def("getStack", [](py::object& obj) {
::example::data::BadData *cls = obj.cast< ::example::data::BadData* >();
return cls->getStack();
});
The obj.cast there also fails because there is no automatic conversion. Basically, with no place to store the c++ instance, there isn't really a workable solution for this approach that I could find. In addition I couldn't find a way to bind in custom constructors at all, which made usability on python very weak.
The next attempt was based on a suggestion in the pybind11 that you could use the python exception type as a metaclass a normal class_ and have python recognize it as a valid exception. I tried a plethora of variations on this approach.
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::reinterpret_borrow<py::object>(PyExc_Exception))
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::cast(PyExc_Exception))
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::cast(PyExc_Exception->ob_type))
py::class_< ::example::data::BadData>(module, "BadData", py::metaclass((PyObject *) &PyExc_Exception->ob_type))
There were more that I don't have saved. But the overall results was either 1) It was ignored completely or 2) it failed to compile or 3) It compiled and then immediately segfaulted or ImportError'd when trying to make an instance. There might have been one that segfaulted on module import too. It all blurs together. Maybe there is some magic formula that would make such a thing work, but I was unable to find it. From my reading of the pybind11 internals, I do not believe that such a thing is actually possible. Inheriting from a raw python type does not seem to be something it is setup to let you do.
The last thing I tried seemed really clever. I made a python exception type
static py::exception<::example::data::BadData> exc_BadData(module, "BadDataBase");
and then had my pybind11 class_ inherit from that.
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), exc_BadData)
But that also segfaulted on import too. So I'm basically back to square one with this.

So I figured out a way to actually do this but it involves 1) doing some hacking of the pybind11 code itself and 2) introducing some size inefficiencies to the bound python types. From my point of view, the size issues are fairly immaterial. Yes it would be better to have everything perfectly sized but I'll take some extra bytes of memory for ease of use. Given this inefficiency, though, I'm not submitting this as a PR to the pybind11 project. While I think the trade-off is worth it, I doubt that making this the default for most people would be desired. It would be possible, I guess to hide this functionality behind a #define in c++ but that seems like it would be super messy long-term. There is probably a better long-term answer that would involve a degree of template meta-programming (parameterizing on the python container type for class_) that I'm just not up to.
I'm providing my changes here as diffs against the current master branch in git when this was written (hash a54eab92d265337996b8e4b4149d9176c2d428a6).
The basic approach was
Modify pybind11 to allow the specification of an exception base class for a class_ instance.
Modify pybind11's internal container to have the extra fields needed for a python exception type
Write a small amount of custom binding code to handle setting the error correctly in python.
For the first part, I added a new attribute to type_record to specify if a class is an exception and added the associated process_attribute call for parsing it.
diff --git a/src/pybind11/include/pybind11/attr.h b/src/pybind11/include/pybind11/attr.h
index 58390239..b5535558 100644
--- a/src/pybind11/include/pybind11/attr.h
+++ b/src/pybind11/include/pybind11/attr.h
## -73,6 +73,9 ## struct module_local { const bool value; constexpr module_local(bool v = true) :
/// Annotation to mark enums as an arithmetic type
struct arithmetic { };
+// Annotation that marks a class as needing an exception base type.
+struct is_except {};
+
/** \rst
A call policy which places one or more guard variables (``Ts...``) around the function call.
## -211,7 +214,8 ## struct function_record {
struct type_record {
PYBIND11_NOINLINE type_record()
: multiple_inheritance(false), dynamic_attr(false), buffer_protocol(false),
- default_holder(true), module_local(false), is_final(false) { }
- default_holder(true), module_local(false), is_final(false),
- is_except(false) { }
/// Handle to the parent scope
handle scope;
## -267,6 +271,9 ## struct type_record {
/// Is the class inheritable from python classes?
bool is_final : 1;
- // Does the class need an exception base type?
- bool is_except : 1;
- PYBIND11_NOINLINE void add_base(const std::type_info &base, void *(*caster)(void *)) {
auto base_info = detail::get_type_info(base, false);
if (!base_info) {
## -451,6 +458,11 ## struct process_attribute<is_final> : process_attribute_default<is_final> {
static void init(const is_final &, type_record *r) { r->is_final = true; }
};
+template <>
+struct process_attribute<is_except> : process_attribute_default<is_except> {
- static void init(const is_except &, type_record *r) { r->is_except = true; }
+};
I modified the internals.h file to add a separate base class for exception types. I also added an extra bool argument to make_object_base_type.
diff --git a/src/pybind11/include/pybind11/detail/internals.h b/src/pybind11/include/pybind11/detail/internals.h
index 6224dfb2..d84df4f5 100644
--- a/src/pybind11/include/pybind11/detail/internals.h
+++ b/src/pybind11/include/pybind11/detail/internals.h
## -16,7 +16,7 ## NAMESPACE_BEGIN(detail)
// Forward declarations
inline PyTypeObject *make_static_property_type();
inline PyTypeObject *make_default_metaclass();
-inline PyObject *make_object_base_type(PyTypeObject *metaclass);
+inline PyObject *make_object_base_type(PyTypeObject *metaclass, bool is_except);
// The old Python Thread Local Storage (TLS) API is deprecated in Python 3.7 in favor of the new
// Thread Specific Storage (TSS) API.
## -107,6 +107,7 ## struct internals {
PyTypeObject *static_property_type;
PyTypeObject *default_metaclass;
PyObject *instance_base;
+ PyObject *exception_base;
#if defined(WITH_THREAD)
PYBIND11_TLS_KEY_INIT(tstate);
PyInterpreterState *istate = nullptr;
## -292,7 +293,8 ## PYBIND11_NOINLINE inline internals &get_internals() {
internals_ptr->registered_exception_translators.push_front(&translate_exception);
internals_ptr->static_property_type = make_static_property_type();
internals_ptr->default_metaclass = make_default_metaclass();
- internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass);
+ internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass, false);
+ internals_ptr->exception_base = make_object_base_type(internals_ptr->default_metaclass, true);
And then in class.h I added the necessary code to generate the exception base type. The first caveat is here. Since PyExc_Exception is a garbage collected type, I had to scope the assert call that checked the GC flag on the type. I have not currently seen any bad behavior from this change, but this is definitely voiding the warranty right here. I would highly, highly recommend always passing the py:dynamic_attr() flag to any classes you are using py:except on, since that turns on all the necessary bells and whistles to handle GC correctly (I think). A better solution might be to turn all those things on in make_object_base_type without having to invoke py::dynamic_attr.
diff --git a/src/pybind11/include/pybind11/detail/class.h b/src/pybind11/include/pybind11/detail/class.h
index a05edeb4..bbb9e772 100644
--- a/src/pybind11/include/pybind11/detail/class.h
+++ b/src/pybind11/include/pybind11/detail/class.h
## -368,7 +368,7 ## extern "C" inline void pybind11_object_dealloc(PyObject *self) {
/** Create the type which can be used as a common base for all classes. This is
needed in order to satisfy Python's requirements for multiple inheritance.
Return value: New reference. */
-inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
+inline PyObject *make_object_base_type(PyTypeObject *metaclass, bool is_except=false) {
constexpr auto *name = "pybind11_object";
auto name_obj = reinterpret_steal<object>(PYBIND11_FROM_STRING(name));
## -387,7 +387,12 ## inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
auto type = &heap_type->ht_type;
type->tp_name = name;
- type->tp_base = type_incref(&PyBaseObject_Type);
+ if (is_except) {
+ type->tp_base = type_incref(reinterpret_cast<PyTypeObject*>(PyExc_Exception));
+ }
+ else {
+ type->tp_base = type_incref(&PyBaseObject_Type);
+ }
type->tp_basicsize = static_cast<ssize_t>(sizeof(instance));
type->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HEAPTYPE;
## -404,7 +409,9 ## inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
setattr((PyObject *) type, "__module__", str("pybind11_builtins"));
PYBIND11_SET_OLDPY_QUALNAME(type, name_obj);
- assert(!PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
+ if (!is_except) {
+ assert(!PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
+ }
return (PyObject *) heap_type;
}
## -565,7 +572,8 ## inline PyObject* make_new_python_type(const type_record &rec) {
auto &internals = get_internals();
auto bases = tuple(rec.bases);
- auto base = (bases.size() == 0) ? internals.instance_base
+ auto base = (bases.size() == 0) ? (rec.is_except ? internals.exception_base
+ : internals.instance_base)
And then the final change, which is the inefficiency part. In Python, everything is a PyObject, but that is really only two fields (setup with the PyObject_HEAD macro) and the actual object struct may have a lot of extra fields. And having a very precise layout is important because python uses offsetof to seek into these things some times. From the Python 2.7 source code (Include/pyerrord.h) you can see the struct that is used for base exceptions
typedef struct {
PyObject_HEAD
PyObject *dict;
PyObject *args;
PyObject *message;
} PyBaseExceptionObject;
Any pybind11 type that extends PyExc_Exception has to have a instance struct that contains the same initial layout. And in pybind11 currently, the instance struct just has PyObject_HEAD. That means if you don't change the instance struct, this will all compile, but when python seeks into this object, it will do with the assumption that hose extra fields exist and then it will seek right off the end of viable memory and you'll get all sorts of fun segfaults. So this change adds those extra fields to every class_ in pybind11. It does not seem to break normal classes to have these extra fields and it definitely seems to make exceptions work correctly. If we broke the warranty before, we just tore it up and lit it on fire.
diff --git a/src/pybind11/include/pybind11/detail/common.h b/src/pybind11/include/pybind11/detail/common.h
index dd626793..b32e0c70 100644
--- a/src/pybind11/include/pybind11/detail/common.h
+++ b/src/pybind11/include/pybind11/detail/common.h
## -392,6 +392,10 ## struct nonsimple_values_and_holders {
/// The 'instance' type which needs to be standard layout (need to be able to use 'offsetof')
struct instance {
PyObject_HEAD
+ // Necessary to support exceptions.
+ PyObject *dict;
+ PyObject *args;
+ PyObject *message;
/// Storage for pointers and holder; see simple_layout, below, for a description
However, once these changes are all made, here is what you can do.
Bind in the class
auto PyBadData = py::class_< ::example::data::BadData>(module, "BadData", py::is_except(), py::dynamic_attr())
.def(py::init<>())
.def(py::init< std::string, std::string >())
.def("__str__", &::example::data::BadData::toString)
.def("getStack", &::example::data::BadData::getStack)
.def_property("message", &::example::data::BadData::getMsg, &::example::data::BadData::setMsg)
.def("getMsg", &::example::data::BadData::getMsg);
And take a function in c++ that throws the exception
void raiseMe()
{
throw ::example::data::BadData("this is an error", "");
}
and bind that in too
module.def("raiseMe", &raiseMe, "A function throws");
Add an exception translator to put the entire python type into the exception
py::register_exception_translator([](std::exception_ptr p) {
try {
if (p) {
std::rethrow_exception(p);
}
} catch (const ::example::data::BadData &e) {
auto err = py::cast(e);
auto errType = err.get_type().ptr();
PyErr_SetObject(errType, err.ptr());
}
});
And then you get all the things you could want!
>>> import example
>>> example.raiseMe()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
example.BadData: BadData(msg=this is an error, stack=)
You can, of course, also instantiate and raise the exception from python as well
>>> import example
>>> raise example.BadData("this is my error", "no stack")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
example.BadData: BadData(msg=this is my error, stack=no stack)

Pass Python object to C and back again

I'm trying to get to grips with using embedded Python from a C++ application. Specifically I want my C++ to launch some PyTorch code.
I am making some initialization function in Python to perform the device (CPU or GPU) discovery and would like to pass this back to the C++ code. The C++ will call another Python function for inference which is when the C++ will pass the device to Python.
pFunc_init = PyObject_GetAttrString(pModule, "torch_init");
if (pFunc_init && PyCallable_Check(pFunc_init)) {
pValue_device = PyObject_CallObject(pFunc_init, pArgs);
if (pArgs != NULL)
Py_DECREF(pArgs);
if (pValue_device != NULL) {
pFunc_infer = PyObject_GetAttrString(pModule, "torch_infer");
if (pFunc_infer && PyCallable_Check(pFunc_infer)) {
//
// TODO put object pValue_device into pArgs_device.
//
pValue_infer = PyObject_CallObject(pFunc_infer, pArgs_device);
if (pValue_infer != NULL) {
printf("Result pValue_infer: %ld\n", PyLong_AsLong(pValue_infer));
Py_DECREF(pValue_infer);
}
}
Py_DECREF(pValue_device);
}
else {
Py_DECREF(pFunc_init);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr, "Call failed\n");
return 1;
}
}
The TODO marks where I would like to put this code. With simple Python objects I think I know what I need but how to deal with this custom Python object?

You can define a Python function "detect_device" which returns a string say "cuda" or "cpu". After that in your C++ code, you can do something like this.
PyObject *detect_device, *pArgsDevice;
detect_device = PyObject_GetAttrString(pModule, "detect_device");
deviceObject = PyObject_CallObject(detect_device, NULL);
pArgsDevice = NULL;
pArgsDevice = PyTuple_New(1);
PyTuple_SetItem(pArgsDevice, 0, deviceObject);
PS: Wrote the answer in hurry due to some urgency. Will add explanation soon, but I think if you understand the code that you have written, you would be able to understand this. Letme know in comments about your progress.

Memory Core Dump C++

Now, I have called python to C++. Using ctype to connect between both of them. And I have a problem about core dump when in running time.
I have a library which is called "libfst.so"
This is my code.
NGramFST.h
#include <iostream>
class NGramFST{
private:
static NGramFST* m_Instace;
public:
NGramFST(){
}
static NGramFST* getInstance() {
if (m_Instace == NULL){
m_Instace = new NGramFST();
}
return m_Instace;
}
double getProbabilityOfWord(std::string word, std::string context) {
std::cout << "reloading..." << std::endl;
return 1;
}
};
NGramFST.cpp
#include "NGramFST.h"
NGramFST* NGramFST::m_Instace = NULL;
extern "C" {
double FST_getProbability(std::string word, std::string context){
return NGramFST::getInstance()->getProbabilityOfWord(word, context);
}
}
And this is my python code.
from ctypes import cdll
lib = cdll.LoadLibrary('./libfst.so')
#-------------------------main code------------------------
class FST(object):
def __init__(self):
print 'Initializing'
def getProbabilityOfWord(self, word, context):
lib.FST_getProbability(word, context)
fst = FST()
print fst.getProbabilityOfWord(c_wchar_p('jack london'), c_wchar_p('my name is'))
This is error
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
I reviewed again but I can not detect where is my problem.

ctypes does not understand C++ types (it's not called c++types). It cannot handle std::string. It wouldn't know that your function expects std::string arguments anyway.
In order to work with ctypes, your library needs a C-compatible interface. extern "C" is necessary but not sufficient. The functions need to be actually callable from C.
Better yet, use a modern C++/Python binding library such as pybind11.

It work when I change python code below
string1 = "my string 1"
string2 = "my string 2"
# create byte objects from the strings
b_string1 = string1.encode('utf-8')
b_string2 = string2.encode('utf-8')
print fst.getProbabilityOfWord(b_string1, b_string2)
and c++ code change type of param bellow
FST_getProbability(const char* word, const char* context)

How can I embed a Python function that returns a string in C using cffi?

I'm trying to embed a Python function in C using PyPy and cffi. I'm following this guide from the PyPy documentation.
The problem is, all the examples I've found operate on ints, and my function takes a string and returns a string. I can't seem to figure out how to embed this function in C, as C doesn't seem to really have strings, rather making do with arrays of chars.
Here's what I've tried:
# interface.py
import cffi
ffi = cffi.FFI()
ffi.cdef('''
struct API {
char (*generate_cool_page)(char url[]);
};
''')
...
#ffi.callback("char[] (char[])")
def generate_cool_page(url):
# do some processing with BS4
return str(soup)
def fill_api(ptr):
global api
api = ffi.cast("struct API*", ptr)
api.generate_cool_page = generate_cool_page
--
// c_tests.c
#include "PyPy.h"
#include <stdio.h>
#include <stdlib.h>
struct API {
char (*generate_cool_page)(char url[]);
};
struct API api; /* global var */
int initialize_api(void)
{
static char source[] =
"import sys; sys.path.insert(0, '.'); "
"import interface; interface.fill_api(c_argument)";
int res;
rpython_startup_code();
res = pypy_setup_home(NULL, 1);
if (res) {
fprintf(stderr, "Error setting pypy home!\n");
return -1;
}
res = pypy_execute_source_ptr(source, &api);
if (res) {
fprintf(stderr, "Error calling pypy_execute_source_ptr!\n");
return -1;
}
return 0;
}
int main(void)
{
if (initialize_api() < 0)
return 1;
printf(api.generate_cool_page("https://example.com"));
return 0;
}
When I run gcc -I/opt/pypy3/include -Wno-write-strings c_tests.c -L/opt/pypy3/bin -lpypy3-c -g -o c_tests and then run ./c_tests, I get this error:
debug: OperationError:
debug: operror-type: CDefError
debug: operror-value: cannot render the type <char()(char *)>: it is a function type, not a pointer-to-function type
Error calling pypy_execute_source_ptr!
I don't have a ton of experience with C and I feel like I'm misrepresenting the string argument/return value. How do I do this properly?
Thanks for your help!

Note that you should not be using pypy's deprecated interface to embedding; instead, see http://cffi.readthedocs.io/en/latest/embedding.html.
The C language doesn't have "strings", but only arrays of chars. In C, a function that wants to return a "string" is usually written
differently: it accepts as first argument a pointer to a pre-existing buffer (of type char[]), and as a second argument the length of that buffer; and when called, it fills the buffer. This can be messy because you ideally need to handle buffer-too-small situations in the caller, e.g. allocate a bigger array and call the function again.
Alternatively, some functions give up and return a freshly malloc()-ed char *. Then the caller must remember to free() it, otherwise a leak occurs. I would recommend that approach in this case because guessing the maximum length of the string before the call might be difficult.
So, something like that. Assuming you start with
http://cffi.readthedocs.io/en/latest/embedding.html, change
plugin.h to contain::
// return type is "char *"
extern char *generate_cool_page(char url[]);
And change this bit of plugin_build.py::
ffibuilder.embedding_init_code("""
from my_plugin import ffi, lib
#ffi.def_extern()
def generate_cool_page(url):
url = ffi.string(url)
# do some processing
return lib.strdup(str(soup)) # calls malloc()
""")
ffibuilder.cdef("""
#include <string.h>
char *strdup(const char *);
""")
From the C code, you don't need initialize_api() at all in the
new embedding mode; instead, you just say #include "plugin.h"
and call the function directly::
char *data = generate_cool_page("https://example.com");
if (data == NULL) { handle_errors... }
printf("Got this: '%s'\n", data);
free(data); // important!