I'm trying to embed Python in my C++ application, somewhat akin to the method found here in section 1.4:
https://docs.python.org/3.5/extending/embedding.html
The synopsis of the problem I'm having is that I can't get the C++ application to work with .py files that import the 'emb' module, ie, the Python extension module that's written into the C++ code.
I have a Python file, testmod.py:
import emb
# define some functions
def printhello(input):
emb.numargs()
return 2
def timesfour(input):
print(input * 4)
In my C++ application, I have this code which works:
PyImport_AppendInittab("emb", &(mynamespace::PyInit_emb) );
Py_Initialize();
PyObject *globals = PyModule_GetDict(PyImport_AddModule("__main__"));
PyObject *testModule = PyImport_ImportModule("emb");
PyObject* pFunc = PyObject_GetAttrString(testModule, "numargs");
After this, pFunc is non-NULL; things look good. So I think the 'embedded module' is fine.
If I change the last two lines from above to:
PyObject* testModule = PyImport_ImportModule("testmod");
PyObject* pFunc = PyObject_GetAttrString(testModule, "printhello");
This also works fine, provided the line emb.numargs() is removed from testmod.py Once I add that line, and re-run the C++ application, testModule becomes NULL, which means something has gone wrong.
Any ideas?
Is this the way this capability is supposed to be used?
Related
I'm writing a C++ program that requires Python (3.11) code to be embedded into it and am using Python.h to try and accomplish this. The general idea is that my a python script, which will be stored by the C++ program as a string, as I'll be performing operations on the source at runtime, will contain a "main()" function which returns an array of known size.
I'm aware I can do it via:
...
PyObject *pName = PyString_FromString("main");
PyObject *pModule = PyImport_Import(pName)
...
However, in order to actually execute the script, I would need to write it to a file just so that python could read it again. This adds extra time to execution that I'd prefer to avoid. Isn't there some way in which I can pass python the source code directly as a string and work from there? Or am I just screwed?
EDIT: BTW, PyRun_SimpleString does not do what I want, as it doesn't return anything from the executed code.
Found the answer thanks to nick in the comments.
An example of usage of PyRun_String: https://schneide.blog/2011/10/10/embedding-python-into-cpp/, and extracting list variables from python script https://docs.python.org/3/c-api/list.html
The final frankenstein:
PyObject *main = PyImport_AddModule("__main__");
PyObject *globalDictionary = PyModule_GetDict(main);
PyObject *localDictionary = PyDict_New();
PyRun_String("a=[0, 1, 2, 3, 4, 5]", Py_file_input, globalDictionary, localDictionary);
PyObject *result = PyDict_GetItemString(localDictionary, "a");
double a[6];
for (int i = 0; i < PyList_Size(result); i++) {
a[i] = PyFloat_AsDouble(PyList_GetItem(result, i));
}
I have a C++ program and it has sort of plugin structure: when program starts up, it's looking for dll in the plugin folder with certain exported function signatures, such as:
void InitPlugin(FuncTable* funcTable);
Then the program will call the function in the dll to initialize and pass function pointers to the dll. From that time on, the dll can talk to the program.
I know Cython let you call C function in Python, but I'm not sure can I write a Cython code and compile it to a dll so my C++ program can initialize with it. An example code would be great.
Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.
The first step would be to mark cdef-function which should be used from external C-code with public, for example:
#cyfun.pyx:
#doesn't need python interpreter
cdef public int double_me(int me):
return 2*me;
#needs initialized python interpreter
cdef public void print_me(int me):
print("I'm", me);
cyfun.c and cyfun.h can be generated with
cython -3 cyfun.pyx
These files will be used for building of the dll.
The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail). Note: the initialization/clean-up could be put also in DllMain - see such a version further bellow.
The header-file for the dll would look like following:
//cyfun_dll.h
#ifdef BUILDING_DLL
#define DLL_PUBLIC __declspec(dllexport)
#else
#define DLL_PUBLIC __declspec(dllimport)
#endif
//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();
DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);
So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.
The implementation follows in cyfun_dll.c-file:
//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"
DLL_PUBLIC int cyfun_init(){
int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
if(status==-1){
return -1;//error
}
Py_Initialize();
PyObject *module = PyImport_ImportModule("cyfun");
if(module==NULL){
Py_Finalize();
return -1;//error
}
return 0;
}
DLL_PUBLIC void cyfun_finalize(){
Py_Finalize();
}
DLL_PUBLIC int cyfun_double_me(int me){
return double_me(me);
}
DLL_PUBLIC void cyfun_print_me(int me){
print_me(me);
}
Noteworthy details:
we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
we use cyfun.h generated by cython from cyfun.pyx.
cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29, PEP-489 is default. More information can be found in this SO-post. If the Python-interpreter isn't initialized or if the module cyfun is not imported, the chances are high, that calling the functionality from cyfun.h will end in a segmentation fault.
cyfun_double_me just wraps double_me so it becomes visible outside of the dll.
Now we can build the dll!
:: set up tool chain
call "<path_to_vcvarsall>\vcvarsall.bat" x64
:: build cyfun.c generated by cython
cl /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include>
:: build dll-wrapper
cl /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>
:: link both obj-files into a dll
link cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>
The dll is now built, but the following details are noteworthy:
<other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to run cythonize some_file.pyx` and to inspect the log.
we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
we have the dependency on the python-dll, so later on it must be somewhere where it can be found.
Were you go from here depends on your task, we test our dll with a simple main:
//test.c
#include "cyfun_dll.h"
int main(){
if(0!=cyfun_init()){
return -1;
}
cyfun_print_me(cyfun_double_me(2));
cyfun_finalize();
return 0;
}
which can be build via
...
:: build main-program
cl /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>
:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>
And now calling test_prog.exe leads to the expected output "I'm 4".
Depending on your installation, following things must be considered:
test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
The embeded python interpreter needs an installation, see this and/or this SO-posts.
IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.
Thus, it may make sense to put initialization/clean-up code into DllMain (and make cyfun_init() and cyfun_finalize() private), e.g.
BOOL WINAPI DllMain(
HINSTANCE hinstDLL, // handle to DLL module
DWORD fdwReason, // reason for calling function
LPVOID lpReserved ) // reserved
{
// Perform actions based on the reason for calling.
switch( fdwReason )
{
case DLL_PROCESS_ATTACH:
return cyfun_init()==0;
case DLL_PROCESS_DETACH:
cyfun_finalize();
break;
case DLL_THREAD_ATTACH:
// Do thread-specific initialization.
break;
case DLL_THREAD_DETACH:
// Do thread-specific cleanup.
break;
}
return TRUE;
}
If your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize the python-interpreter. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.
I'd imagine it'd be difficult to call it directly, with Cython depending so much on the Python runtime.
Your best bet is to embed a Python interpreter directly inside your app, e.g. as described in this answer, and call your Cython code from the interpreter. That's what I would do.
In 2014 I have written a PyQt4 app that used a non PyQt module within and it worked as it should for nearly two years. Now, the non PyQt module had stopped working as expected within PyQt application (I have qt4 and qt5 ports, both share the problem). Below is a sample code to illustrate what is going on:
#! /usr/bin/env python3
import sys, getopt
from PyQt5.QtWidgets import QApplication # or PyQt4
# non PyQt module:
import mymodule
# demo of the mymodule use. It returns nested list
# with info about objects saved in fname
fname = "/my/file/name"
res0 = mymodule.getnames(fname)
# res0 = [[...],...,[...]] the way it should
#
app = QApplication(sys.argv)
...
# with or without actual run of the application via
# app.exec_()
...
res2 = mymodule.getnames(fname)
# this time the result is:
# res2 = None
Originally, I called the method of mymodule from within PyQt app and it returned the nested list, now it returns None. I have tried to find any explanation or clarification, why the observed behaviour has changed but failed to do so.
I have isolated the code responsible for rendering the mymodule behaviour to the call QApplication([sys.argv]) and wondered if anyone has an idea what is going on. What bugs me the most is, that the module is functional before the application start, but after that not any more.
Some further details about mymodule:
it is a wrapper of a C library written in mymodule.c and installed via the python setup.py install call.
Thank you for any comments in advance!
[Edit] Further informations about mymodule:
mymodule.c code snippet
#include <Python.h>
#include <extlib.h>
#include <math.h>
...
static PyObject * mymodule_getnames(PyObject *self, PyObject *args)
{
char *filename;
if (!PyArg_ParseTuple(args, "s", &filename))
return NULL;
FILE *fp = extlib_func(filename);
...
PyObject *names = PyList_New(0);
...
PyList_Append(names, Py_BuildValue("s", title));
...
return names;
}
...
int main(int argc, char *argv[])
{
Py_SetProgramName(
#if PY_MAJOR_VERSION >= 3
(wchar_t *)
#endif
argv[0]);
Py_Initialize();
return 0;
}
I have resolved the problem by not using anaconda as my python package manager. By installing everything in the system it worked as it should again. But the question still remains since the reason why is it not working within anaconda (but did for such a long time) is still beyond me.
I have posted this answer but will not accept it, if someone comes with an explanation of the phenomena. If after some time there will be no other reply I will mark it accepted to unclutter the questions pool.
Thank you all for your time and suggestions.
I am totally new to boost.python.
I reviewed a lot of recommending of using boost.python to apply with python, however still not easy to understand and find a solution for me.
What I want is to import a function or class that directly from a python "SourceFile"
Example File:
Main.cpp
MyPythonClass.py
Let's says if there is a "Dog" class in "MyPythonClass.py" with "bark()" function, how do I get callback and send argument in cpp?
I have no idea what I should do!
Please help me!
When one needs to call Python from C++, and C++ owns the main function, then one must embed the Python interrupter within the C++ program. The Boost.Python API is not a complete wrapper around the Python/C API, so one may find the need to directly invoke parts of the Python/C API. Nevertheless, Boost.Python's API can make interoperability easier. Consider reading the official Boost.Python embedding tutorial for more information.
Here is a basic skeleton for a C++ program that embeds Python:
int main()
{
// Initialize Python.
Py_Initialize();
namespace python = boost::python;
try
{
... Boost.Python calls ...
}
catch (const python::error_already_set&)
{
PyErr_Print();
return 1;
}
// Do not call Py_Finalize() with Boost.Python.
}
When embedding Python, it may be necessary to augment the module search path via PYTHONPATH so that modules can be imported from custom locations.
// Allow Python to load modules from the current directory.
setenv("PYTHONPATH", ".", 1);
// Initialize Python.
Py_Initialize();
Often times, the Boost.Python API provides a way to write C++ code in a Python-ish manner. The following example demonstrates embedding a Python interpreter in C++, and having C++ import a MyPythonClass Python module from disk, instantiate an instance of MyPythonClass.Dog, and then invoking bark() on the Dog instance:
#include <boost/python.hpp>
#include <cstdlib> // setenv
int main()
{
// Allow Python to load modules from the current directory.
setenv("PYTHONPATH", ".", 1);
// Initialize Python.
Py_Initialize();
namespace python = boost::python;
try
{
// >>> import MyPythonClass
python::object my_python_class_module = python::import("MyPythonClass");
// >>> dog = MyPythonClass.Dog()
python::object dog = my_python_class_module.attr("Dog")();
// >>> dog.bark("woof");
dog.attr("bark")("woof");
}
catch (const python::error_already_set&)
{
PyErr_Print();
return 1;
}
// Do not call Py_Finalize() with Boost.Python.
}
Given a MyPythonClass module that contains:
class Dog():
def bark(self, message):
print "The dog barks: {}".format(message)
The above program outputs:
The dog barks: woof
Boost python is used to call cplusplus functions from a python source. Pretty much like the Perl xs module.
If you have a function say bark() in main.cpp, you can use boost python to convert this main.cpp into a python callable module.
Then from python script(assuming link output file is main.so):
import main
main.bark()
I'm trying to call some function from Kernel32.dll in my Python script running on Linux. As Johannes Weiß pointed How to call Wine dll from python on Linux? I'm loading kernel32.dll.so library via ctypes.cdll.LoadLibrary() and it loads fine. I can see kernel32 loaded and even has GetLastError() function inside. However whenever I'm trying to call the function i'm gettings segfault.
import ctypes
kernel32 = ctypes.cdll.LoadLibrary('/usr/lib/i386-linux-gnu/wine/kernel32.dll.so')
print kernel32
# <CDLL '/usr/lib/i386-linux-gnu/wine/kernel32.dll.so', handle 8843c10 at b7412e8c>
print kernel32.GetLastError
# <_FuncPtr object at 0xb740b094>
gle = kernel32.GetLastError
# OK
gle_result = gle()
# fails with
# Segmentation fault (core dumped)
print gle_result
First I was thinking about calling convention differences but it seems to be okay after all. I'm ending with testing simple function GetLastError function without any params but I'm still getting Segmentation fault anyway.
My testing system is Ubuntu 12.10, Python 2.7.3 and wine-1.4.1 (everything is 32bit)
UPD
I proceed with my testing and find several functions that I can call via ctypes without segfault. For instance I can name Beep() and GetCurrentThread() functions, many other functions still give me segfault. I created a small C application to test kernel32.dll.so library without python but i've got essentially the same results.
int main(int argc, char **argv)
{
void *lib_handle;
#define LOAD_LIBRARY_AS_DATAFILE 0x00000002
long (*GetCurrentThread)(void);
long (*beep)(long,long);
void (*sleep)(long);
long (*LoadLibraryExA)(char*, long, long);
long x;
char *error;
lib_handle = dlopen("/usr/local/lib/wine/kernel32.dll.so", RTLD_LAZY);
if (!lib_handle)
{
fprintf(stderr, "%s\n", dlerror());
exit(1);
}
// All the functions are loaded e.g. sleep != NULL
GetCurrentThread = dlsym(lib_handle, "GetCurrentThread");
beep = dlsym(lib_handle, "Beep");
LoadLibraryExA = dlsym(lib_handle, "LoadLibraryExA");
sleep = dlsym(lib_handle, "Sleep");
if ((error = dlerror()) != NULL)
{
fprintf(stderr, "%s\n", error);
exit(1);
}
// Works
x = (*GetCurrentThread)();
printf("Val x=%d\n",x);
// Works (no beeping, but no segfault too)
(*beep)(500,500);
// Segfault
(*sleep)(5000);
// Segfault
(*LoadLibraryExA)("/home/ubuntu/test.dll",0,LOAD_LIBRARY_AS_DATAFILE);
printf("The End\n");
dlclose(lib_handle);
return 0;
}
I was trying to use different calling conventions for Sleep() function but got no luck with it too. When I comparing function declarations\implementation in Wine sources they are essentially the same
Declarations
HANDLE WINAPI GetCurrentThread(void) // http://source.winehq.org/source/dlls/kernel32/thread.c#L573
BOOL WINAPI Beep( DWORD dwFreq, DWORD dwDur ) // http://source.winehq.org/source/dlls/kernel32/console.c#L354
HMODULE WINAPI DECLSPEC_HOTPATCH LoadLibraryExA(LPCSTR libname, HANDLE hfile, DWORD flags) // http://source.winehq.org/source/dlls/kernel32/module.c#L928
VOID WINAPI DECLSPEC_HOTPATCH Sleep( DWORD timeout ) // http://source.winehq.org/source/dlls/kernel32/sync.c#L95
WINAPI is defined to be __stdcall
However some of them works and some don't. As I can understand this sources are for kernel32.dll file and kernel32.dll.so file is a some kind of proxy that supposed to provide access to kernel32.dll for linux code. Probably I need to find exact sources of kernel32.dll.so file and take a look on declarations.
Is there any tool I can use to take a look inside .so file and find out what functions and what calling conventions are used?
The simplest way to examine a DLL is to use the nm command, i.e.
$ nm kernel32.dll.so | grep GetLastError
7b86aae0 T _GetLastError
As others have pointed out, the default calling convention for Windows C DLLs is stdcall. It has nothing to do with using Python. On the Windows platform, ctypes.windll is available.
However, I am not even sure what you are trying to do is at all possible. Wine is a full-blown Windows emulator and it is safe to guess that at least you would have to start it with wine_init before loading any other functions. The Windows API probably have some state (set when Windows boots).
The easiest way to continue is probably to install a Windows version of Python under Wine and run your script from there.