Using Pybind11, I am able to call a C++ native function from my Python code.
My C++ programme has a long-running function that keeps on going until explicitly stopped and this function generates some output using std::cout. As this long-running function never returns due to its nature, I am trying to get the output of this C++ code in my Python code for further processing.
I am aware of this: https://pybind11.readthedocs.io/en/stable/advanced/pycpp/utilities.html#capturing-standard-output-from-ostream however I really do not see how to reflect the C++ generated output into my Python code.
Here is the code:
int myFunc()
{
...
for(;;) { // Can be only stopped if requested by user
std::cout << capturedEvent;
}
...
return 0;
}
namespace py = pybind11;
PYBIND11_MODULE(Handler, m) {
// Add a scoped redirect for your noisy code
m.def("myFunc", []() {
py::scoped_ostream_redirect stream(
std::cout, // std::ostream&
py::module_::import("sys").attr("stdout") // Python output
);
myFunc();
});
...
}
And Python:
from Handler import myFunc
from random import random
DATA = [(random() - 0.5) * 3 for _ in range(999999)]
def test(fn, name):
result = fn()
print('capturedEvent is {} {} \n\n'.format(result, name))
if __name__ == "__main__":
test(myFunc, '(PyBind11 C++ extension)')
I would like to retrieve, in realtime, the content of the C++ capturedEvent variable.
If there is another approach than capturing stdout, (maybe sharing a variable in realtime?) please, let me know, maybe my strategy is wrong.
Thank you.
Related
I want to execute the same Python function several times and I want each execution to be completely isolated from the previous one.
But the problem is that the first execution of the method changes a global variable and when I execute the method the second time the global variable has the value left by the first execution.
I want that from one call to another the environment is reset and is like the first time.
var1=1
def Met(n) :
global var1
if n!=1 : var1=n
print(f"Py: {var1}")
return var1
if __name__ == "__main__":
args = sys.argv
globals()[args[1]](*args[2:])
If I execute it from command line:
python .\Test1.py Met 2
python .\Test1.py Met 1
The output is (correct):
Py: 2
Py: 1
What I want, OK.
But if I run it from Pythonet I get:
Py: 2
Py: 2
Incorrect!
That is, the second call, the value of var1 is the value of first call.
The code (simplifying):
public string Met2(int n) {
dynamic txtResul="X";
using ( Py.GIL() ) {
using (var scope=gbl.NewScope() ) { //Witout this, it doesn't work too
dynamic f=Py.Import(NOM_FICH_Py);
txtResul=f.Met(n);
}
}
return txtResul.ToString();
}
I have tried initialising with and without this:
PythonEngine.Initialize();
gbl = Py.CreateScope(nombre);
pyPtr=PythonEngine.BeginAllowThreads();
I have also tried to finalize everything and run it, with and without this:
public void Fin() {
PythonEngine.EndAllowThreads(pyPtr);
PythonEngine.Shutdown();
}
I have also tried running each function call as a separate script:
using (Py.GIL()) {
dynamic sys = Py.Import("sys");
string script ="import sys\n"+ "from io import StringIO\n"+
$"from {NOM_FICH_Py} import {funcion.Split('(')[0]}\n"+
"sys.stdout=StringIO()\n"+ "sys.stdout.flush()\n"+
"sys.stderr=StringIO()\n"+ "sys.stderr.flush()\n"+
funcion;
var scope = Py.CreateScope();
var vars=scope.Variables();
scope.Exec(script, new PyDict());
salida=sys.stdout.getvalue();
scope.Dispose();
}
I use Python 3.10 and Python .Net 3.0.0-rc 4 and Python.Net 3.0.0-preview2022-03-03 (in other computer) and I cann't get the 'reset' as I executed it like command line.
Thanks in advance.
Extra question, is there any wiki/documentacion to read the detail of functions like CreateScope, NewScope, Exec...? In https://github.com/pythonnet/pythonnet/wiki I haven't found the function's detail.
The only supported way to fully reset Python environment with Python.NET is to restart the process.
At the end, the only posible solution it's to exectue like a script in cmd.
It's not the best solution (it's not posible to debug) but it's the only one I know.
public string ExecCmd(string funcion, string args) {
Process p = new Process();
p.StartInfo=new ProcessStartInfo(PYTHON_EXE, $"{NombreFichApp} {funcion} {args}") {
RedirectStandardOutput=true,
UseShellExecute=false,
CreateNoWindow=true
};
p.Start();
string salida = p.StandardOutput.ReadToEnd();
p.WaitForExit();
return salida;
}
I am facing memory leak and crash issues in pybind11.
I am calling a python function "myfunc" from a python file "mydl.py" that uses Tensorflow Keras deep learning functions, Numpy, and Redis modules using pybind11 in a repeatitive C++ code. The code structure is as follows.
class myclass {
public:
myclass() {
py::initialize_interpreter();
{
py::module sys = py::module::import("sys");
py::module os = py::module::import("os");
py::str cwd = os.attr("getcwd")();
py::print("os.cwd: ", cwd);
py::str bin = cwd + py::str("/../bin");
// Add bin to sys.path
py::module site = py::module::import("site");
site.attr("addsitedir")(bin);
}
}
~myclass() {
py::finalize_interpreter();
}
int callpyfunc(string a1, string a2) {
int retval;
{
py::module mydl = py::module::import("mydl");
py::object result = mydl.attr("myfunc")(a1, a2);
retval = result.cast<int>();
}
return retval;
}
}
myclass *mcobj1;
int main() {
mcobj1 = new myclass();
int retval;
while (/* some deep learning condition is not met */) {
retval = mcobj1->callpyfunc(a1, a2);
}
del mcobj1;
}
The memory size of this program goes on increasing consistently to the point of it consuming entire 62 GB RAM and crashing. It seems like Python interpreter is not releasing memory allocated for different objects inside each call to "myfunc" of "mydl.py" even after the call gets done.
Here's what all I have tried with no luck of fixing the issue:
Using scoped interpreter inside callpyfunc instead of doing initialize_interpreter and finalize_interpreter. But in that case the code crashes quietly in the second call to "callpyfunc", the first call goes fine. This is exactly what is mentioned here
Moving initialize_interpreter along with import of modules like "sys", "os" and finalize_interpreter inside callpyfunc. But in that case the code crashes in the second call to "callpyfunc" at line py::module mydl = py::module::import("mydl"); and never reaches finalizing of interpreter.
I am trying to use python bindings to interface a simple python wrapper around my c++ code. I am currently wanting to return a map of values. When I try to create a dictionary entry my application segfaults when the key > size 1. Even ignoring the returning of the object I still get the error. Only adding "ke" segfaults as well. I have successfully returned a dict with {"k": 10} but that is it.
C++:
extern "C" void Test() {
signal(SIGSEGV, handler); // install our handler
PyObject* results = PyDict_New();
printf("Adding k\n");
PyDict_SetItemString(results, "k", PyLong_FromLong(3000));
printf("Adding ke\n");
PyDict_SetItemString(results, "ke", PyLong_FromLong(3000));
printf("Adding key\n");
PyDict_SetItemString(results, "key", PyLong_FromLong(3000));
}
Python:
import ctypes
_test_bench = ctypes.CDLL('<path_to_so>')
_test_bench.Test.argtypes = None
_test_bench.Test.restype = None
def test() -> None:
global _test_bench
_test_bench.Test()
test()
Output:
Adding k
Adding ke
Error: signal 11:
You can't use the Python API from a library loaded with CDLL. You need to use PyDLL.
(Also, don't forget to do your refcount management. That's not the cause of the crash, but it is still a problem.)
I am currently trying to wrap a c-dll to controll a camera connected via USB. To grab image data off the camera, the SDK-Provider (Thorlabs) describes two possibilities:
Poll the camera in a loop
Use callbacks
I wrapped the DLL and got the polling method to work, which basically consists of calling a method which returns either NULL or the pending frame.
The DLL also provides methods to assign callback functions to certain events, such as camera connected/disconnected and frame available. The C-code from the example looks like this:
void camera_connect_callback(char* cameraSerialNumber, enum USB_PORT_TYPE usb_bus_speed, void* context)
{
printf("camera %s connected with bus speed = %d!\n", cameraSerialNumber, usb_bus_speed);
}
int main(void)
{
[...]
// Set the camera connect event callback. This is used to register for run time camera connect events.
if (tl_camera_set_camera_connect_callback(camera_connect_callback, 0))
{
printf("Failed to set camera connect callback!\n");
close_sdk_dll();
return 1;
}
[...]
}
My goal is now to get the callback working with python, so that python methods are called from the dll. Therefore, I used CFFI in API-mode to compile the wrapper, then tried to pass a callback method to the DLL in the same fashion:
#ffi.callback("void(char*, void*)")
def disconnect_callback(serialNumber, context):
print("Disconnect callback was called!")
[...]
print("Set disconnect callback: {}".format(lib.tl_camera_set_camera_disconnect_callback(disconnect_callback, ffi.NULL)))
[...]
For testing purposes I only included the simplest callback in the working polling example. The setter method returns 0 but the method is never executed.
Is this even the right way of accomplishing my goal? Do I need to introduce some kind of threading so that the callback can interrupt the sequential execution of the remaining program?
There is not much documentation or examples on this topic so Im really hoping you guys can help me out.
EDIT
So I tried a very basic example which works and technically answeres my question, but does not solve my problem.
if I define a C callback method as follows:
#include <math.h>
int quadcalc(int input, int (*getval)(int)){
return (int) pow(getval(input), 2);
}
I can assign a handler as expected: (After compiling of course)
#ffi.callback("int(int)")
def callback_func(value):
return int(rnd.random() * value)
if __name__ == '__main__':
print(lib.quadcalc(10, callback_func))
And everything works as expected. Unfortunately, this doesn't work on the actual Problem. So callback works in general, but not in the specific case, which is why the question is still open.
Do I need to introduce some kind of threading so that the callback can interrupt the sequential execution of the remaining program?
Invoking the Callback method takes care of setting up alternate program flows to capture the event, so it can be routed to a handler to be processed. When the defined event occurs, it does need handler code. For example, here is a very generic, but complete example of code that uses a callback in C (including the handler function.):
void populate_array(int *array, size_t arraySize, int (*getNextValue)(void))
{
for (size_t i=0; i<arraySize; i++)
array[i] = getNextValue();
}
// handler function
int getNextRandomValue(void)
{
return rand();
}
int main(void)
{
int myarray[10];
populate_array(myarray, 10, getNextRandomValue);
...
}
There is an equally good example just below the first one, here.
Beyond these, there is a good tutorial in the CFFI documentation, with the first section dedicated to addressing your title question How to make a C-DLL wrapped with CFFI to callback python:
The first section presents a simple working example of using CFFI to call a C function in a compiled shared object (DLL) from Python.
Steps are: (see link above for details of each step.)
- Create the file piapprox_build.py:
- Execute this script:
- At runtime, you use the extension module like this:
...In the rest of this page, we describe some more
advanced examples and other CFFI modes...
I am totally new to boost.python.
I reviewed a lot of recommending of using boost.python to apply with python, however still not easy to understand and find a solution for me.
What I want is to import a function or class that directly from a python "SourceFile"
Example File:
Main.cpp
MyPythonClass.py
Let's says if there is a "Dog" class in "MyPythonClass.py" with "bark()" function, how do I get callback and send argument in cpp?
I have no idea what I should do!
Please help me!
When one needs to call Python from C++, and C++ owns the main function, then one must embed the Python interrupter within the C++ program. The Boost.Python API is not a complete wrapper around the Python/C API, so one may find the need to directly invoke parts of the Python/C API. Nevertheless, Boost.Python's API can make interoperability easier. Consider reading the official Boost.Python embedding tutorial for more information.
Here is a basic skeleton for a C++ program that embeds Python:
int main()
{
// Initialize Python.
Py_Initialize();
namespace python = boost::python;
try
{
... Boost.Python calls ...
}
catch (const python::error_already_set&)
{
PyErr_Print();
return 1;
}
// Do not call Py_Finalize() with Boost.Python.
}
When embedding Python, it may be necessary to augment the module search path via PYTHONPATH so that modules can be imported from custom locations.
// Allow Python to load modules from the current directory.
setenv("PYTHONPATH", ".", 1);
// Initialize Python.
Py_Initialize();
Often times, the Boost.Python API provides a way to write C++ code in a Python-ish manner. The following example demonstrates embedding a Python interpreter in C++, and having C++ import a MyPythonClass Python module from disk, instantiate an instance of MyPythonClass.Dog, and then invoking bark() on the Dog instance:
#include <boost/python.hpp>
#include <cstdlib> // setenv
int main()
{
// Allow Python to load modules from the current directory.
setenv("PYTHONPATH", ".", 1);
// Initialize Python.
Py_Initialize();
namespace python = boost::python;
try
{
// >>> import MyPythonClass
python::object my_python_class_module = python::import("MyPythonClass");
// >>> dog = MyPythonClass.Dog()
python::object dog = my_python_class_module.attr("Dog")();
// >>> dog.bark("woof");
dog.attr("bark")("woof");
}
catch (const python::error_already_set&)
{
PyErr_Print();
return 1;
}
// Do not call Py_Finalize() with Boost.Python.
}
Given a MyPythonClass module that contains:
class Dog():
def bark(self, message):
print "The dog barks: {}".format(message)
The above program outputs:
The dog barks: woof
Boost python is used to call cplusplus functions from a python source. Pretty much like the Perl xs module.
If you have a function say bark() in main.cpp, you can use boost python to convert this main.cpp into a python callable module.
Then from python script(assuming link output file is main.so):
import main
main.bark()