I am trying to overload a python extension function that would take either a object or a string.
typedef struct
{
PyObject_HEAD
} CustomObject;
PyObject* customFunction(CustomObject* self, PyObject* args);
PyMethodDef methods[] =
{
{"customFunction", (PyCFunction) customFunction, METH_VARAGS, "A custom function"},
{NULL}
}
PyTypeObject TypeObj =
{
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "customModule.CustomObject",
.tp_doc = "Custom Object",
.tp_basicsize = sizeof(CustomObject),
.tp_itemsize = 0,
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_methods = methods,
}
// Area of problem
PyObject* customFunction(CustomObject* self, PyObject* args)
{
const char* string;
PyObject* object;
if (PyArg_ParseTuple(args, "O!", &TypeObj, &object)) // TypeObj is the PyTypeObject fpr CustomObject
{
std::cout << "Object function\n"
// Do whatever and return PyObject*
}
else if (PyArg_ParseTuple(args, "s", &string))
{
std::cout << "String function\n"
// Do whatever and return PyObject*
}
return PyLong_FromLong(0); // In case nothing above works
}
In python I have a try except for the function and I get this error Error: <built-in method customFunction of CustomModule.CustomObject object at 0xmemoryadress> returned a result with an error set
Here are the Python docs for this PyArg_ParseTuple:
int PyArg_ParseTuple(PyObject *args, const char *format, ...)
Parse the parameters of a function that takes only positional parameters into local variables. Returns true on success; on failure, it returns false and raises the appropriate exception
I am guessing that PyArg_ParseTuple is setting an error, which is causing the entire function not to work (I do have customFunction in my method table for the module, I am just omitting that code). If I have the following Python:
import CustomModule
try:
CustomModule.customFunction("foo")
except Exception as e:
print("Error:", e)
String function does get outputted, so the code in the string if statement does work, but I assume the error occurs because PyArg_ParseTuple for the object failed, so it returns an error (not 100% sure if this is correct).
Is there a way I can prevent PyArg_ParseTuple() from raising an error, is there another function, or is there a better way to 'overload' my custom functions?
I'd probably just use PyArg_ParseTuple to get a generic unspecified object, and then handle the object types later with Py*_Check:
if (!PyArg_ParseTuple(args, "O", &object)) {
return NULL;
}
if (PyObject_IsInstance(object, (PyObject*)&PyType)) { // or a more specific function if one exists
std::cout << "Object function\n";
} else if (PyUnicode_Check(object)) {
std::cout << "String function\n";
} else {
// set an error, return NULL
}
The reason for this is that the Python "ask forgiveness, not permission" pattern of
try:
something()
except SomeException:
somethingElse()
doesn't translate very well into C, and involves quite a bit of code to handle the exceptions. If you really want to do it that way then you need to call PyErr_Clear before the second PyArg_ParseTuple, and ideally you should check it's the exception you think, and not something else entirely.
Related
I'm trying to get a simple C++ 20 based generator pattern work with PyBind11. This is the code:
#include <pybind11/pybind11.h>
#include <coroutine>
#include <iostream>
struct Generator2 {
Generator2(){}
struct Promise;
using promise_type=Promise;
std::coroutine_handle<Promise> coro;
Generator2(std::coroutine_handle<Promise> h): coro(h) {}
~Generator2() {
if(coro)
coro.destroy();
}
int value() {
return coro.promise().val;
}
bool next() {
std::cout<<"calling coro.resume()";
coro.resume();
std::cout<<"coro.resume() called";
return !coro.done();
}
struct Promise {
void unhandled_exception() {std::rethrow_exception(std::move(std::current_exception()));}
int val;
Generator2 get_return_object() {
return Generator2{std::coroutine_handle<Promise>::from_promise(*this)};
}
std::suspend_always initial_suspend() {
return {};
}
std::suspend_always yield_value(int x) {
val=x;
return {};
}
std::suspend_never return_void() {
return {};
}
std::suspend_always final_suspend() noexcept {
return {};
}
};
};
Generator2 myCoroutineFunction() {
for(int i = 0; i < 100; ++i) {
co_yield i;
}
}
class Gen{
private:
Generator2 myCoroutineResult;
public:
Gen(){
myCoroutineResult = myCoroutineFunction();
}
int next(){
return (myCoroutineResult.next());
}
};
PYBIND11_MODULE(cmake_example, m) {
pybind11::class_<Gen>(m, "Gen")
.def(pybind11::init())
.def("next", &Gen::next);
}
However I'm getting an error:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Could c++ coroutines, coroutine_handles, co_yield etc. be a low-level thing that is not supported by PyBind11 yet?
Even though PyBind11 does not support coroutines directly, your problem does not mix coroutine and pybind code since you are hiding the coroutine behind Gen anyway.
The problem is that your Generator2 type uses the compiler provided copy and move constructors.
This line:
myCoroutineResult = myCoroutineFunction();
Creates a coroutine handle when you call myCoroutineFunction, and puts it in the temporary Generator2 in the right hand side. Then, you initialize myCoroutineResult from the right hand side generator. All is well, but then the temporary gets destroyed. Your destructor checks whether the handle is valid or not:
~Generator2() {
if(coro)
coro.destroy();
}
But in your implementation, the coro member of the member generator gets copied from the temporary without resetting the temporary's coro member. So the coroutine itself gets destroyed once you initialize myCoroutineResult, and you are holding onto a dangling coroutine handle. Remember that std::coroutine_handles behave like a raw pointer.
Essentially, you have a violation of the rule of 5. You have a custom destructor, but no copy/move constructors or assignment operators. Since you cannot copy construct a coroutine, you can ignore the copy constructors but you need to provide move constructors/assigment operators:
Generator2(Generator2&& rhs) : coro{std::exchange(rhs.coro, nullptr)} {
// rhs will not delete our coroutine,
// since we put nullptr to its coro member
}
Generator2& operator=(Generator2&& rhs) {
if (&rhs == this) {
return *this;
}
if (coro) {
coro.destroy();
}
coro = std::exchange(rhs.coro, nullptr);
return *this;
}
Also, use member initialization list to initialize members instead of assigning them within the constructor body. So instead of this:
Gen(){
myCoroutineResult = myCoroutineFunction();
}
Use this:
Gen() : myCoroutineResult{myCoroutineFunction()} {}
The reasoning can be seen even in this answer. The first one calls the assignment operator, which performs a bunch of additional work, whereas the second one calls the move constructor, which is as lean as it gets.
I would not like to keep converting every Python String Object from PyObject* to std::string or char* with PyUnicode_DecodeUTF8 and PyUnicode_AsUTF8 because it is an expensive operation.
On my last question How to extend/reuse Python C Extensions/API implementation?, I managed to use the Python open function, to directly give me a PyObject* string. Once doing that, it is very simple to send the string back to the Python program because I can just pass its PyObject* pointer back, instead of doing a full char-by-char copy as PyUnicode_DecodeUTF8 or PyUnicode_AsUTF8 do.
On the regex implementation of CPython API, I can find a function like this:
static void* getstring(PyObject* string, Py_ssize_t* p_length,
int* p_isbytes, int* p_charsize,
Py_buffer *view)
{
/* given a python object, return a data pointer, a length (in
characters), and a character size. return NULL if the object
is not a string (or not compatible) */
/* Unicode objects do not support the buffer API. So, get the data directly. */
if (PyUnicode_Check(string)) {
if (PyUnicode_READY(string) == -1)
return NULL;
*p_length = PyUnicode_GET_LENGTH(string);
*p_charsize = PyUnicode_KIND(string);
*p_isbytes = 0;
return PyUnicode_DATA(string);
}
/* get pointer to byte string buffer */
if (PyObject_GetBuffer(string, view, PyBUF_SIMPLE) != 0) {
PyErr_SetString(PyExc_TypeError, "expected string or bytes-like object");
return NULL;
}
*p_length = view->len;
*p_charsize = 1;
*p_isbytes = 1;
if (view->buf == NULL) {
PyErr_SetString(PyExc_ValueError, "Buffer is NULL");
PyBuffer_Release(view);
view->buf = NULL;
return NULL;
}
return view->buf;
}
It does not seem to be using PyUnicode_DecodeUTF8 or PyUnicode_AsUTF8 to work with the PyObject* coming from the Python Interpreter.
How can I use basic string operations with PyObject* strings without conversion then to std::string or char*?
I would call basic operations the following examples: (Just for exemplifying, I am using Py_BuildValue to build a PyObject* string from a string as a char* or std::string)
static PyObject* PyFastFile_do_concatenation(PyFastFile* self)
{
PyObject* hello = Py_BuildValue( "s", "Hello" );
PyObject* word = Py_BuildValue( "s", "word" );
// I am just guessing the `->value` property
PyObject* hello_world = hello->value + word->value;
hello_world; // return the `PyObject*` string `Hello word`
}
static PyObject* PyFastFile_do_substring(PyFastFile* self)
{
PyObject* hello = Py_BuildValue( "s", "Hello word" );
PyObject* hello_world = hello->value[5:];
hello_world; // return the `PyObject*` string `word`
}
static PyObject* PyFastFile_do_contains(PyFastFile* self)
{
PyObject* hello = Py_BuildValue( "s", "Hello word" );
if( "word" in hello->value ) {
Py_BuildValue( "p", true ); // return the `PyObject*` boolean `true`
}
Py_BuildValue( "p", false ); // return the `PyObject*` boolean `false`
}
I want to check if an object is an instance of a certain class. In Python I can do this with instanceof. In C/C++, I found a function named PyObject_IsInstance. But it seems not to work like isinstance.
In detail (also described as sample codes below):
In C++ I defined my custom type My. The type definition is MyType and the object definition is MyObject.
Add MyType to the exported module with name My.
In Python, create a new instance my = My(), and isinstance(my, My) returns True.
While in C++ we use PyObject_IsInstance(my, (PyObject*)&MyType) to check my, and this returns 0, which means my is not an instance of the class defined by MyType.
Full C++ code:
#define PY_SSIZE_T_CLEAN
#include <python3.6/Python.h>
#include <python3.6/structmember.h>
#include <stddef.h>
typedef struct {
PyObject_HEAD
int num;
} MyObject;
static PyTypeObject MyType = []{
PyTypeObject ret = {
PyVarObject_HEAD_INIT(NULL, 0)
};
ret.tp_name = "cpp.My";
ret.tp_doc = NULL;
ret.tp_basicsize = sizeof(MyObject);
ret.tp_itemsize = 0;
ret.tp_flags = Py_TPFLAGS_DEFAULT;
ret.tp_new = PyType_GenericNew;
return ret;
}();
// check if obj is an instance of MyType
static PyObject *Py_fn_checkMy(PyObject *obj) {
if (PyObject_IsInstance(obj, (PyObject *)&MyType)) Py_RETURN_TRUE;
else Py_RETURN_FALSE;
}
static PyMethodDef modmethodsdef[] = {
{ "checkMy", (PyCFunction)Py_fn_checkMy, METH_VARARGS, NULL },
{ NULL }
};
static PyModuleDef moddef = []{
PyModuleDef ret = {
PyModuleDef_HEAD_INIT
};
ret.m_name = "cpp";
ret.m_doc = NULL;
ret.m_size = -1;
return ret;
}();
PyMODINIT_FUNC
PyInit_cpp(void)
{
PyObject *mod;
if (PyType_Ready(&MyType) < 0)
return NULL;
mod = PyModule_Create(&moddef);
if (mod == NULL)
return NULL;
Py_INCREF(&MyType);
PyModule_AddObject(mod, "My", (PyObject *)&MyType);
PyModule_AddFunctions(mod, modmethodsdef);
return mod;
}
Compile this into cpp.so, and test it in Python:
>>> import cpp
>>> isinstance(cpp.My(), cpp.My)
True
>>> cpp.checkMy(cpp.My())
False
METH_VARARGS
This is the typical calling convention, where the methods have the type PyCFunction. The function expects two PyObject* values. The first one is the self object for methods; for module functions, it is the module object. The second parameter (often called args) is a tuple object representing all arguments. This parameter is typically processed using PyArg_ParseTuple() or PyArg_UnpackTuple().
The function signature of Py_fn_checkMy does not match this. It should take two arguments. The first is the module, and this is what you are checking against MyType. The second argument (which you don't actually accept) is a tuple containing the object you passed. You should extract the argument from the tuple and check the type of this.
You'd probably be better using METH_O to specify a single argument instead of extracting arguments from a tuple:
static PyObject *Py_fn_checkMy(PyObject *self, PyObject *obj) {
if (PyObject_IsInstance(obj, (PyObject *)&MyType)) Py_RETURN_TRUE;
else Py_RETURN_FALSE;
}
static PyMethodDef modmethodsdef[] = {
{ "checkMy", (PyCFunction)Py_fn_checkMy, METH_O, NULL },
{ NULL }
};
if I call a SWIG-wrapped C/C++ function from Python, is it possible to obtain the current call stack? I would like something similar to the result of ''.join(traceback.format_stack()), but I don't want to pass this from Python to my C/C++ functions because I don't always need it. So I would like to obtain it on the fly and print it if something wrong happens on my C/C++ side.
I figured out a solution following this post, although I still prefer more natural ways of getting the same thing if there is any.
// This is similar to the python code:
// def GetScriptingLanguageCallStack():
// import traceback
// return ''.join(traceback.format_stack())
string GetScriptingLanguageCallStack() {
string result;
PyObject* module_name = PyString_FromString("traceback");
PyObject* pyth_module = PyImport_Import(module_name);
Py_DECREF(module_name);
if (pyth_module != nullptr) {
PyObject* pyth_func = PyObject_GetAttrString(pyth_module, "format_stack");
if (pyth_func != nullptr) {
if (PyCallable_Check(pyth_func)) {
PyObject* pyth_val = PyObject_CallFunctionObjArgs(pyth_func, 0);
if (pyth_val != nullptr) {
if (PyList_Check(pyth_val)) {
const int size = PyList_GET_SIZE(pyth_val);
for (int i = 0; i < size; ++i) {
PyObject* pyth_line = PyList_GET_ITEM(pyth_val, i);
result += PyString_AsString(pyth_line);
}
}
Py_DECREF(pyth_val);
}
}
Py_DECREF(pyth_func);
}
Py_DECREF(pyth_module);
}
return result;
}
By the way, I do not prefer the approach in the linked post which uses frame object, because the line number given is not pointing to the exact line which makes the further function calls, but only on the line containing the function name.
I'm using Python 3.4.3 embded in my c++ project, When I Call Py_RunString it always returns None object.
here is my code
#include <Python.h>
#include <string>
int main(){
//Initialize the python interpreter
Py_Initialize();
//create new dictionary containing both global and local definitions
PyObject *globals = PyDict_New();
PyObject *locals = PyDict_New();
//Set the build in definitions to the global dictionary: eg:len, str ,.. funtions
PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins());
//evaluate some python code and get the result, here is my issue, always None
PyObject *string_result = PyRun_StringFlags(
"1 + 1" /*or what ever python code*/
,
Py_file_input, globals, locals, NULL);
//check whether the python code caused any Exception and print it
if (PyErr_Occurred()) {
PyErr_Print(); PyErr_Clear(); return 1;
}
else {
//if no Exceptions then try To Get string represiation of the python object, But it always None
auto str = PyToStr(string_result);
printf("python result is %s",str);
}
return 0;
}
and here is my Function that convert any python object to c++ std::wstring
std::wstring PyToStr(PyObject* Object)
{
//it is equivalent to python code : str(Object)
PyObject* objectsRepresentation = PyObject_Str(Object);
//convert Python String Object to C++ wchar_t*
const wchar_t* ws = PyUnicode_AsUnicode(objectsRepresentation);
if (ws)
return ws;
//if ws is NULL it could not be converted implicitly to std::wstring
return L"";
}
If you just want to evaluate a single line you can use Py_eval_input instead of Py_file_input. It should then return the evaluated value.