So, StackOverflow, I'm stumped.
The code as I have it is a C++ function with embedded Python. I generate a message on the C++ side, send it to the python, get a different message back. I got it to work, I got it tested, so far, so good.
The next step is that I need Python to generate messages on their own and send them into C++. This is where I'm starting to get stuck. After spending a few hours puzzling over the documentation, it seemed like the best way would be to define a module to hold my functions. So I wrote up the following stub:
static PyMethodDef mailbox_methods[] = {
{ "send_external_message",
[](PyObject *caller, PyObject *args) -> PyObject *
{
classname *interface = (classname *)
PyCapsule_GetPointer(PyTuple_GetItem(args, 0), "turkey");
class_field_type toReturn;
toReturn = class_field_type.python2cpp(PyTuple_GetItem(args, 1));
interface ->send_message(toReturn);
Py_INCREF(Py_None);
return Py_None;
},
METH_VARARGS,
"documentation" },
{ NULL, NULL, 0, NULL }
};
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT,
"turkey",
"documentation",
-1,
mailbox_methods
};
//defined in header file, just noting its existence here
PyObject *Module;
PyMODINIT_FUNC PyInit_turkey()
{
Module = PyModule_Create(&moduledef);
return Module;
}
And on the Python side, I had the following receiver code:
import turkey
I get the following response:
ImportError: No module named 'turkey'
Now, here's the part where I get really confused. Here's the code in my initialization function:
PyInit_turkey();
PyObject *interface = PyCapsule_New(this, "instance", NULL);
char *repr = PyUnicode_AsUTF8(PyObject_Repr(Module));
cout << "REPR: " << repr << "\n";
if (PyErr_Occurred())
PyErr_Print();
It prints out
REPR: <module 'turkey'>
Traceback (most recent call last):
<snipped>
import turkey
ImportError: No module named 'turkey'
So the module exists, but it's not ever being passed to Python anywhere. I can't find documentation on how to pass it in and get it initialized on the Python side. I realize that I'm probably just missing a trivial step, but I can't for the life off me figure out what it is. Can anyone help?
The answer was, in the end, a single function that I was missing. Included at the start of my initialization function, before I call Py_Initialize:
PyImport_AppendInittab("facade", initfunc);
Py_Initialize();
PyEval_InitThreads();
The Python documentation does not mention PyImport_AppendInittab except in passing, and this is why I was having such a difficult time making the jump.
To anyone else who finds this in the future: You do not need to create a DLL to extend python or use a pre-built library to bring a module into Python. It can be far easier than that.
Related
I have the following code in the Qt Quick Application together with Boost.
In this Cpp there is a personal module created using BOOST_PYTHON_MODULE(hello). The main goal is to be able to import hello in Python and call the methods of hello struct. My Python script only contains very simple structure as i just want to see no errors when importing hello.
import hello
print("Import was successful!")
Most of the codes below are copied from a different question in stackoverflow but not entirely so i had to repost the parts.
Main.cpp
#include <cstdlib> // setenv, atoi
#include <iostream> // cerr, cout, endl
#include <boost/python.hpp>
#include <QGuiApplication>
#include <QQmlApplicationEngine>
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
//---------------------------------------------------------------------------------------------------------------------
/// Staticly linking a Python extension for embedded Python.
BOOST_PYTHON_MODULE(hello)
{
namespace python = boost::python;
python::class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
;
}
//---------------------------------------------------------------------------------------------------------------------
int main(int argc, char *argv[])
{
QCoreApplication::setAttribute(Qt::AA_EnableHighDpiScaling);
QGuiApplication app(argc, argv);
namespace python = boost::python;
try
{
int uploaded = PyImport_AppendInittab("hello", &PyInit_hello);
//This executes else part
if(uploaded == -1)
std::cout<< "Module table was not extended: " << uploaded << std::endl;
else
std::cout<< "Module Table was extended" << std::endl;
Py_Initialize();
} catch (...)
{
PyErr_Print();
return 1;
}
return app.exec();
}
Finally, I run my QT application and the return app.exec(); keeps it running while i try and run my python script as mentioned above from the terminal. The python script is in the same directory as the currently running application, not sure if that makes any difference.
Then the error i get is:
Traceback (most recent call last):
File "test_hilton.py", line 1, in <module>
import hello
ModuleNotFoundError: No module named 'hello'
Not sure what i am missing here. According to the Python API:
PyImport_AppendInittab - Add a single module to the existing table of
built-in modules. This is a convenience wrapper around
PyImport_ExtendInittab(), returning -1 if the table could not be
extended. The new module can be imported by the name name, and uses
the function initfunc as the initialization function called on the
first attempted import.
And the If-else part inside the try-catch block inside the main proves that the hello module is being added to the table. Out of ideas on what to do, looked in different places. But still stuck with this part of the problem.
Since the hello module is defined in that Qt program it is available only in that program. Executing the program doesn't make it available to python interpreter that expects to find hello.py or hello.so (the file extension may vary depending on the operating system) when importing hello by import hello.
You need to build a python module, answer might help.
I'm trying to figure out why I can't simply get and set the python path through its C API. I am using Python3.6, on Ubuntu 17.10 with gcc version 7.2.0. Compiling with:
gcc pytest.c `python3-config --libs` `python3-config --includes`
#include <Python.h>
int main()
{
Py_Initialize(); // removes error if put after Py_SetPath
printf("setting path\n"); // prints
Py_SetPath(L"/usr/lib/python3.6"); // Error in `./a.out': free(): invalid size: 0x00007fd5a8365030 ***
printf("success\n"); // doesn't print
return 0;
}
Setting the path works fine, unless I also try to get the path prior to doing so. If I get the path at all, even just to print without modifying the returned value or anything, I get a "double free or corruption" error.
Very confused. Am I doing something wrong or is this a bug? Anyone know a workaround if so?
Edit: Also errors after calling Py_Initialize();. Updated code. Now errors even if I don't call Py_GetPath() first.
From alk it seems related to this bug: https://bugs.python.org/issue31532
Here is the workaround I am using. Since you can't call Py_GetPath() before Py_Initialize(), and also seemingly you can't call Py_SetPath() after Py_Initialize(), you can add to or get the path like this after calling Py_Initialize():
#include <Python.h>
int main()
{
Py_Initialize();
// get handle to python sys.path object
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
// make a list of paths to add to sys.path
PyObject *newPaths = PyUnicode_Split(PyUnicode_FromWideChar(L"a:b:c", -1), PyUnicode_FromWideChar(L":", 1), -1);
// iterate through list and add all paths
for(int i=0; i<PyList_Size(newPaths); i++) {
PyList_Append(path, PyList_GetItem(newPaths, i));
}
// print out sys.path after appends
PyObject *newlist = PyUnicode_Join(PyUnicode_FromWideChar(L":", -1), path);
printf("newlist = %ls\n", PyUnicode_AsWideCharString(newlist, NULL));
return 0;
}
[the below answer refers to this version of the question.]
From the docs:
void Py_Initialize()
Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the exception of Py_SetProgramName(), Py_SetPythonHome() and Py_SetPath().
But the code you show does call Py_GetPath() before it calls Py_Initialize();, which it per the above paragraph implicitly should not.
I am totally new to boost.python.
I reviewed a lot of recommending of using boost.python to apply with python, however still not easy to understand and find a solution for me.
What I want is to import a function or class that directly from a python "SourceFile"
Example File:
Main.cpp
MyPythonClass.py
Let's says if there is a "Dog" class in "MyPythonClass.py" with "bark()" function, how do I get callback and send argument in cpp?
I have no idea what I should do!
Please help me!
When one needs to call Python from C++, and C++ owns the main function, then one must embed the Python interrupter within the C++ program. The Boost.Python API is not a complete wrapper around the Python/C API, so one may find the need to directly invoke parts of the Python/C API. Nevertheless, Boost.Python's API can make interoperability easier. Consider reading the official Boost.Python embedding tutorial for more information.
Here is a basic skeleton for a C++ program that embeds Python:
int main()
{
// Initialize Python.
Py_Initialize();
namespace python = boost::python;
try
{
... Boost.Python calls ...
}
catch (const python::error_already_set&)
{
PyErr_Print();
return 1;
}
// Do not call Py_Finalize() with Boost.Python.
}
When embedding Python, it may be necessary to augment the module search path via PYTHONPATH so that modules can be imported from custom locations.
// Allow Python to load modules from the current directory.
setenv("PYTHONPATH", ".", 1);
// Initialize Python.
Py_Initialize();
Often times, the Boost.Python API provides a way to write C++ code in a Python-ish manner. The following example demonstrates embedding a Python interpreter in C++, and having C++ import a MyPythonClass Python module from disk, instantiate an instance of MyPythonClass.Dog, and then invoking bark() on the Dog instance:
#include <boost/python.hpp>
#include <cstdlib> // setenv
int main()
{
// Allow Python to load modules from the current directory.
setenv("PYTHONPATH", ".", 1);
// Initialize Python.
Py_Initialize();
namespace python = boost::python;
try
{
// >>> import MyPythonClass
python::object my_python_class_module = python::import("MyPythonClass");
// >>> dog = MyPythonClass.Dog()
python::object dog = my_python_class_module.attr("Dog")();
// >>> dog.bark("woof");
dog.attr("bark")("woof");
}
catch (const python::error_already_set&)
{
PyErr_Print();
return 1;
}
// Do not call Py_Finalize() with Boost.Python.
}
Given a MyPythonClass module that contains:
class Dog():
def bark(self, message):
print "The dog barks: {}".format(message)
The above program outputs:
The dog barks: woof
Boost python is used to call cplusplus functions from a python source. Pretty much like the Perl xs module.
If you have a function say bark() in main.cpp, you can use boost python to convert this main.cpp into a python callable module.
Then from python script(assuming link output file is main.so):
import main
main.bark()
I am attempting to write a C extension for python. With the code (below) I get the compiler warning:
implicit declaration of function ‘Py_InitModule’
And it fails at run time with this error:
undefined symbol: Py_InitModule
I have spent literally hours searching for a solution with no joy. I have tried multiple minor changes to syntax, I even found a post suggesting the method has been deprecated. However I find no replacement.
Here is the code:
#include <Python.h>
//a func to calc fib numbers
int cFib(int n)
{
if (n<2) return n;
return cFib(n-1) + cFib(n-2);
}
static PyObject* fib(PyObject* self,PyObject* args)
{
int n;
if (!PyArg_ParseTuple(args,"i",&n))
return NULL;
return Py_BuildValue("i",cFib(n));
}
static PyMethodDef module_methods[] = {
{"fib",(PyCFunction) fib, METH_VARARGS,"calculates the fibonachi number"},
{NULL,NULL,0,NULL}
};
PyMODINIT_FUNC initcModPyDem(void)
{
Py_InitModule("cModPyDem",module_methods,"a module");
}
If it helps here is my setup.py :
from distutils.core import setup, Extension
module = Extension('cModPyDem', sources=['cModPyDem.c'])
setup(name = 'packagename',
version='1.0',
description = 'a test package',
ext_modules = [module])
And the test code in test.py :
import cModPyDem
if __name__ == '__main__' :
print(cModPyDem.fib(200))
Any help would be much, much appreciated.
The code you have would work fine in Python 2.x, but Py_InitModule is no longer used in Python 3.x. Nowadays, you create a PyModuleDef structure and then pass a reference to it to PyModule_Create.
The structure would look like:
static struct PyModuleDef cModPyDem =
{
PyModuleDef_HEAD_INIT,
"cModPyDem", /* name of module */
"", /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
module_methods
};
And then your PyMODINIT_FUNC function would look like:
PyMODINIT_FUNC PyInit_cModPyDem(void)
{
return PyModule_Create(&cModPyDem);
}
Note that the name of the PyMODINIT_FUNC function must be of the form PyInit_<name> where <name> is the name of your module.
I think it would be worthwhile if you read Extending in the Python 3.x documentation. It has a detailed description of how to build extension modules in modern Python.
I ran into the same problem with Py_InitModule(). I started with the aforementioned Python 3 docs, specifically the "Extending and Embedding the Python Interpreter" doc. But that doc's chapter entitled "A Simple Example" leaves out details. So. I googled this scipy lecture:
http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html
which is in many ways more suitable for someone new to Python-C API extensions ... except it has not been updated for Python v3. So ... consult the scipy lecture, and the Python 3 docs, and this StackOverflow discussion, culling the pertinent information from each for your needs.
I am trying to use Python in C++ and have the following code. I intended to parse and do sys.path.append on a user input path. It looks like the call to PyRun_SimpleString caused some sort of spillage into a private class var of the class. How did this happen? I have tried various buffer size 50, 150, 200, and it did not change the output.
class Myclass
{
...
private:
char *_modName;
char *_modDir;
};
Myclass::Myclass()
{
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString((char *)"sys.path.append('/home/userA/Python')");
}
Myclass::init()
{
// this function is called before Myclass::test()
// a couple other python funcitons are called as listed below.
// PyString_FromString, PyImport_Import, PyObject_GetAttrString, PyTuple_New, PyTuple_SetItem, PyObject_CallObject, PyDict_GetItemString
}
Myclass::test()
{
char buffer[150];
char *strP1 = (char *)"sys.path.append('";
char *strP2 = (char *)"')";
strcpy (buffer, strP1);
strcat (buffer, _modDir);
strcat (buffer, strP2);
printf("Before %s\n", _modDir);
printf("Before %s\n", _modName);
PyRun_SimpleString(buffer);
printf("After %s\n", _modName);
}
Here is the output. FYI I'm using a,b,c,d,f for illustration purpose only. It almost fills like PyRun_SimpleString(buffer) stick the end of buffer into _modName.
Before /aaa/bbb/ccc/ddd
Before ffffff
After cc/ddd'
Thanks for Klamer Schutte for hinting in the correct direction.
The DECRF in my code was the culprit. Being unfamilar with how reference works, I guess. The DECREF call released pValue together with it the content pointed by _modName. A guess a more beginner quesiton would be, should I have added a Py_INCREF(pValue) after _modName assignment?
_modName = PyString_AsString (PyDict_GetItemString(pValue, (char*)"modName"));
Py_DECREF(pValue);