We Keep Coding

Getting result of PyRun_String when python code returns an object

i have a problem with my code.
i have a python file for the capturing of mavlink messages(i'm using pymavlink library) and i need to create a library for interfacing python results with c/c++ code.
this is my python code from .py file
from pymavlink import mavutil
the_connection = mavutil.mavlink_connection('udpin:localhost:14550')
the_connection.wait_heartbeat()
print("Heartbeat from system (system %u component %u)" % (the_connection.target_system, the_connection.target_component))
while 1:
attitude=the_connection.messages['ATTITUDE']
print("attitude: ",attitude)
i need to recover the attitude object as PyObject, the result of the last print is:
attitude: ATTITUDE {time_boot_ms : 1351674, roll : -0.006938610225915909, pitch : -0.009435104206204414, yaw : 1.8100472688674927, rollspeed : 0.0005244240164756775, pitchspeed : -0.0023000920191407204, yawspeed : 0.0002169199287891388}
i have a streaming of messages, so i need to call the connection and the to evaluate the result in a loop. so i tried to call the simple python commands as string, to open the connection and then access to the data. My C code is:
Py_Initialize();
PyRun_SimpleString("from pymavlink import mavutil\n"
"the_connection = mavutil.mavlink_connection('udpin:localhost:14550')\n"
"the_connection.wait_heartbeat()\n"
"print(\"Heartbeat from system (system %u component %u)\" % (the_connection.target_system, the_connection.target_component), flush=True)" );
PyObject* main_module=PyImport_AddModule("__main__");
PyObject* pdict = PyModule_GetDict(main_module);
PyObject* pdict_new = PyDict_New();
while (1) {
PyObject* pval = PyRun_String("the_connection.messages['ATTITUDE']", Py_single_input, pdict, pdict_new);
PyObject* repr = PyObject_Str(pval);
PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
const char* bytes = PyBytes_AS_STRING(str);
PyObject_Print(pval, stdout, 0);
printf(" end\n");
Py_XDECREF(repr);
}
Py_Finalize();
the result of this code is:
<pymavlink.dialects.v20.ardupilotmega.MAVLink_attitude_message object at 0x7fba218220>
None end
<pymavlink.dialects.v20.ardupilotmega.MAVLink_attitude_message object at 0x7fba218220>
None end
<pymavlink.dialects.v20.ardupilotmega.MAVLink_attitude_message object at 0x7fba218220>
None end
<pymavlink.dialects.v20.ardupilotmega.MAVLink_attitude_message object at 0x7fba218220>
None end
i've tried using a return of the object, but it didn't work
PyObject* pval = PyRun_String("return(the_connection.messages['ATTITUDE'])", Py_single_input, pdict, pdict_new);
i'm not expert of C/C++, is there a way to obtain the result in the right way?i'm not interested in a string format, i only need a way to use the result as c object
i'm using python 3.9, on a raspberry pi, gcc version is 10.2.1.
thank you

You want
PyRun_String("the_connection.messages['ATTITUDE']", Py_eval_input, pdict, pdict_new);
Py_eval_input treats it like the Python builtin eval (so what you're running must be an expression rather than a statement, which it is...).
In contrast, Py_single_input evaluates a single statement, but just returns None because a statement doesn't necessary returns anything. (In Python all expressions are statements, but not all statements are expressions). It's more akin to exec (but only deals with a single line).
Using "return(the_connection.messages['ATTITUDE'])" doesn't work because return is specifically designed to appear in a Python function.

Pybind11 - Capture output in realtime from C++ to Python

Using Pybind11, I am able to call a C++ native function from my Python code.
My C++ programme has a long-running function that keeps on going until explicitly stopped and this function generates some output using std::cout. As this long-running function never returns due to its nature, I am trying to get the output of this C++ code in my Python code for further processing.
I am aware of this: https://pybind11.readthedocs.io/en/stable/advanced/pycpp/utilities.html#capturing-standard-output-from-ostream however I really do not see how to reflect the C++ generated output into my Python code.
Here is the code:
int myFunc()
{
...
for(;;) { // Can be only stopped if requested by user
std::cout << capturedEvent;
}
...
return 0;
}
namespace py = pybind11;
PYBIND11_MODULE(Handler, m) {
// Add a scoped redirect for your noisy code
m.def("myFunc", []() {
py::scoped_ostream_redirect stream(
std::cout, // std::ostream&
py::module_::import("sys").attr("stdout") // Python output
);
myFunc();
});
...
}
And Python:
from Handler import myFunc
from random import random
DATA = [(random() - 0.5) * 3 for _ in range(999999)]
def test(fn, name):
result = fn()
print('capturedEvent is {} {} \n\n'.format(result, name))
if __name__ == "__main__":
test(myFunc, '(PyBind11 C++ extension)')
I would like to retrieve, in realtime, the content of the C++ capturedEvent variable.
If there is another approach than capturing stdout, (maybe sharing a variable in realtime?) please, let me know, maybe my strategy is wrong.
Thank you.

pybind11 memory leak and crashes

I am facing memory leak and crash issues in pybind11.
I am calling a python function "myfunc" from a python file "mydl.py" that uses Tensorflow Keras deep learning functions, Numpy, and Redis modules using pybind11 in a repeatitive C++ code. The code structure is as follows.
class myclass {
public:
myclass() {
py::initialize_interpreter();
{
py::module sys = py::module::import("sys");
py::module os = py::module::import("os");
py::str cwd = os.attr("getcwd")();
py::print("os.cwd: ", cwd);
py::str bin = cwd + py::str("/../bin");
// Add bin to sys.path
py::module site = py::module::import("site");
site.attr("addsitedir")(bin);
}
}
~myclass() {
py::finalize_interpreter();
}
int callpyfunc(string a1, string a2) {
int retval;
{
py::module mydl = py::module::import("mydl");
py::object result = mydl.attr("myfunc")(a1, a2);
retval = result.cast<int>();
}
return retval;
}
}
myclass *mcobj1;
int main() {
mcobj1 = new myclass();
int retval;
while (/* some deep learning condition is not met */) {
retval = mcobj1->callpyfunc(a1, a2);
}
del mcobj1;
}
The memory size of this program goes on increasing consistently to the point of it consuming entire 62 GB RAM and crashing. It seems like Python interpreter is not releasing memory allocated for different objects inside each call to "myfunc" of "mydl.py" even after the call gets done.
Here's what all I have tried with no luck of fixing the issue:
Using scoped interpreter inside callpyfunc instead of doing initialize_interpreter and finalize_interpreter. But in that case the code crashes quietly in the second call to "callpyfunc", the first call goes fine. This is exactly what is mentioned here
Moving initialize_interpreter along with import of modules like "sys", "os" and finalize_interpreter inside callpyfunc. But in that case the code crashes in the second call to "callpyfunc" at line py::module mydl = py::module::import("mydl"); and never reaches finalizing of interpreter.

Calling C++ code via embedded Python

I have successfully created a Python module that appears to work in isolation, but doesn't affect the program that is running it.
I have the following module:
BOOST_PYTHON_MODULE(mandala)
{
class_<state_mgr_t, state_mgr_t*, noncopyable>("states", no_init)
.def("push", &state_mgr_t::push)
.def("pop", &state_mgr_t::pop)
.def("count", &state_mgr_t::count);
scope().attr("states") = boost::python::object(boost::python::ptr(&states));
}
The states object is referencing a global value, states:
extern state_mgr_t states;
I can run the following script lines from within my program:
from mandala import states
states.count()
> 0
All of that is fine and whatnot, but I was expecting that running this python script would affect the actual state of the program that is running it. It appears as though Python is actually just dealing with it's own instance of states and not affecting the parent program.
Now I'm wondering if I've completely misunderstood what Boost.Python is capable of; I was expecting something similar to Lua, where I could modify the C++ program via scripts.
Is this not possible? Or am I doing something very wrong?
Thank you in advance!

If you are embedding Python into your C++ program, it should be possible to access the instance from your script. Obviously, I don't have your full code, but have you tried something like this?
PyImport_AppendInittab("mandala", &initmandala);
Py_Initialize();
try {
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
main_namespace.attr("states") = ptr(&states);
object ignored = exec("print states.count()", main_namespace);
} catch(const error_already_set&) {
PyErr_Print();
}

Segfault on accessing a C initialised ctypes structure element

I am trying to access elements of a structure from ctypes. The structure is created in an init function in the C code and pointer to it is returned to Python. The problem I am having is that I get a segfault when trying to access elements of the returned structure. Here is the code:
The C code (which I've called ctypes_struct_test.c):
#include <stdio.h>
#include <stdbool.h>
typedef struct {
bool flag;
} simple_structure;
simple_structure * init()
{
static simple_structure test_struct = {.flag = true};
if (test_struct.flag) {
printf("flag is set in C\n");
}
return &test_struct;
}
The Python code (which I've called ctypes_struct_test.py):
#!/usr/bin/env python
import ctypes
import os
class SimpleStructure(ctypes.Structure):
_fields_ = [('flag', ctypes.c_bool)]
class CtypesWrapperClass(object):
def __init__(self):
cwd = os.path.dirname(os.path.abspath(__file__))
library_file = os.path.join(cwd,'libctypes_struct_test.so')
self._c_ctypes_test = ctypes.CDLL(library_file)
self._c_ctypes_test.init.restypes = ctypes.POINTER(SimpleStructure)
self._c_ctypes_test.init.argtypes = []
self.simple_structure = ctypes.cast(\
self._c_ctypes_test.init(),\
ctypes.POINTER(SimpleStructure))
a = CtypesWrapperClass()
print 'Python initialised fine'
print a.simple_structure.contents
print a.simple_structure.contents.flag
The C is compiled under Linux as follows:
gcc -o ctypes_struct_test.os -c --std=c99 -fPIC ctypes_struct_test.c
gcc -o libctypes_struct_test.so -shared ctypes_struct_test.os
On running python ctypes_struct_test.py, I get the following output:
flag is set in C
Python initialised fine
<__main__.SimpleStructure object at 0x166c680>
Segmentation fault
Is there some problem with what I am trying to do, or the way I am trying to do it?

To make this work, replace the wrong line
self._c_ctypes_test.init.restypes = ctypes.POINTER(SimpleStructure)
by the correct line
self._c_ctypes_test.init.restype = ctypes.POINTER(SimpleStructure)
Also consider to remove the pointless cast
self.simple_structure = ctypes.cast(
self._c_ctypes_test.init(), ctypes.POINTER(SimpleStructure))
It's casting from ctypes.POINTER(SimpleStructure) to ctypes.POINTER(SimpleStructure)!

You declare test_struct as a static local variable inside your init routine -- I'm suspicious of that at first blush. (Bear with me; my C is a bit rusty.)
Static local variables in C are supposed to persist across multiple calls to the same function, but their scope is the same as an automatic local variable -- and returning a pointer to an automatic local variable would certainly give you a segfault. Even if this would ordinarily work if you were calling that function from the same translation unit in C, the fact that you're calling it from Python (and that you're segfaulting as soon as you try to access the struct) is a red flag.
Instead of returning a pointer to a static variable, try using malloc() in your init routine and returning the pointer you get from that.
[[[ WARNING: this will result in a memory leak. ]]]
But that's OK; if the segfault goes away, you'll know that that was the problem. Then you can fix the memory leak by providing a second routine in your C code that calls free(), and making sure you invoke it from your python code when you're done with the struct.