I have this code in a C application that's embedding Python (2.7.1):
{
PyObject *user_dict;
PyObject *user_func;
PyObject *result;
PyObject *header_tuple;
PyObject *original_recipients;
PyObject *working_recipients;
if (!Py_IsInitialized())
{
Py_Initialize();
}
if (!expy_exim_dict)
{
PyObject *module = Py_InitModule(expy_exim_module, expy_exim_methods); /* Borrowed reference */
Py_INCREF(module); /* convert to New reference */
expy_exim_dict = PyModule_GetDict(module); /* Borrowed reference */
Py_INCREF(expy_exim_dict); /* convert to New reference */
}
if (!expy_user_module)
{
if (expy_path_add)
{
PyObject *sys_module;
PyObject *sys_dict;
PyObject *sys_path;
PyObject *add_value;
sys_module = PyImport_ImportModule("sys"); /* New Reference */
if (!sys_module)
{
PyErr_Clear();
*return_text = "Internal error, can't import Python sys module";
log_write(0, LOG_REJECT, "Couldn't import Python 'sys' module");
return PYTHON_FAILURE_RETURN;
}
sys_dict = PyModule_GetDict(sys_module); /* Borrowed Reference, never fails */
sys_path = PyMapping_GetItemString(sys_dict, "path"); /* New reference */
if (!sys_path || (!PyList_Check(sys_path)))
{
PyErr_Clear(); /* in case sys_path was NULL, harmless otherwise */
*return_text = "Internal error, sys.path doesn't exist or isn't a list";
log_write(0, LOG_REJECT, "expy: Python sys.path doesn't exist or isn't a list");
return PYTHON_FAILURE_RETURN;
}
add_value = PyString_FromString(expy_path_add); /* New reference */
if (!add_value)
{
PyErr_Clear();
log_write(0, LOG_PANIC, "expy: Failed to create Python string from [%s]", expy_path_add);
return PYTHON_FAILURE_RETURN;
}
if (PyList_Append(sys_path, add_value))
{
PyErr_Clear();
log_write(0, LOG_PANIC, "expy: Failed to append [%s] to Python sys.path", expy_path_add);
}
Py_DECREF(add_value);
Py_DECREF(sys_path);
Py_DECREF(sys_module);
}
expy_user_module = PyImport_ImportModule(expy_scan_module); /* New Reference */
if (!expy_user_module)
{
PyErr_Clear();
/* Handle error */
}
}
When PyImport_ImportModule fails, it returns NULL. How can I find out why it failed to import? (e.g. when importing the module works find outside of the embedding).
(The code is part of py-exim-localscan, and I'm wanting to add more information about failures in the rare cases when they occur).
You do this by looking at the exception that was raised. Currently you wipe the exception (that's what PyErr_Clear() does.) Don't do that, and instead print the traceback or inspect the exception object. See http://docs.python.org/c-api/exceptions.html for information on how to do that from C code, but usually the best idea is to just let the exception propagate.
Related
I'm trying to embed Python in a C++ application, and I need it to run as a release build, as I am only interested in debugging the C++ code. Also, I do not have the _d debug versions of all libraries I need. I am using Python 3.7.0, in MSVC 2017 with C++11, the library I don't have debug symbols for is VTK, which I installed through a wheel file supplied by my employer. If I try to build the Python wrapper myself I get another load of issues, so I am unable to build the debug files. If I run in Debug mode, I am unable to import the library, whereas if I run in Release, I get further issues:
Exception thrown at 0x00007FF90841DBC9 (python37_d.dll) in PythonEmbedding.exe: 0xC0000005: Access violation reading location 0x0000000000000025.
The C++ code:
int main(int argc, char *argv[])
{
PyObject *pIntrospector = NULL;
if (PyVtk_InitIntrospector(pIntrospector) == OK)
{
printf("Initialization succeeded\n");
}
vtkObjectBase *pConeSource = NULL;
if (PyVtk_CreateVtkObject(pIntrospector, "vtkConeSource", pConeSource) == OK)
{
printf("Object creation succeeded\n");
}
return 0;
}
int PyVtk_InitIntrospector(
PyObject *pIntrospector)
{
/* Activating virtual environment */
#ifdef _DEBUG
// For Visual Studio debug builds
const wchar_t *sPyHome = L"venv-dbg";
#else
// For release builds
const wchar_t *sPyHome = L"venv";
#endif
Py_SetPythonHome(sPyHome);
/* Initializing Python environment and setting PYTHONPATH. */
Py_Initialize();
PyRun_SimpleString("import sys\nimport os");
PyRun_SimpleString("sys.path.append( os.path.dirname(os.getcwd()) )");
PyRun_SimpleString("sys.path.append(\".\")");
PyRun_SimpleString("import importlib.machinery as m");
PyRun_SimpleString("print(m.all_suffixes())");
/* Decode module from its name. Returns error if the name is not decodable. */
PyObject *pIntrospectorModuleName = PyUnicode_DecodeFSDefault("Introspector");
if (pIntrospectorModuleName == NULL)
{
fprintf(stderr, "Fatal error: cannot decode module name\n");
return PYTHON_INTROSPECTION_STRING_DECODE_ERROR;
}
/* Imports the module previously decoded. Returns error if the module is not found. */
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
if (pIntrospectorModule == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Failed to load \"Introspector\"\n");
Py_DECREF(pIntrospectorModuleName);
return PYTHON_INTROSPECTION_MODULE_LOAD_ERROR;
}
/* Looks for the Introspector class in the module. If it does not find it, returns and error. */
PyObject* pIntrospectorClass = PyObject_GetAttrString(pIntrospectorModule, "Introspector");
if (pIntrospectorClass == NULL || !PyCallable_Check(pIntrospectorClass))
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Cannot find class \"Introspector\"\n");
if (pIntrospectorClass != NULL)
{
Py_DECREF(pIntrospectorClass);
}
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
return PYTHON_INTROSPECTION_CLASS_NOT_FOUND_ERROR;
}
/* Instantiates an Introspector object. If the call returns NULL there was an error
creating the object, and thus it returns error. */
pIntrospector = PyObject_CallObject(pIntrospectorClass, NULL);
if (pIntrospector == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Introspector instantiation failed\n");
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return PYTHON_INTROSPECTION_OBJECT_CREATION_ERROR;
}
/* Decreasing reference to local data. */
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return OK;
}
I have not added the code to the PyVtk_CreateVtkObject function as it won't enter it, but if I do not add the calls after PyVtk_InitIntroepsctor it won't give the aforementioned error. Finally, if I import Introspector in the Python interpreter myself, it works fine.
Is there a solution to either run it in Debug or Release? I cannot wrap my head around it...
P.S.: I already tried to use Boost::Python, I have two issues open on it as it is giving me problems as well.
Update 1: In particular, the excpetion is thrown when I do this:
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
Update 2: I have further scoped down the issue to this: whenever I import the vtk package from within the embedded interpreter, it throws the Access Violation on this code:
// Add special attribute __vtkname__
PyObject *s = PyString_FromString(classname);
PyDict_SetItemString(pytype->tp_dict, "__vtkname__", s);
Py_DECREF(s); // <-- In particular on this Py_DECREF
If I try to import anything else, there is no issue, it seems.
I'm trying to write a c extension for python to speed up some number crunching I'm doing in my project with out having to port the whole thing to C. Unfortunately when I try to return numpy arrays by using my extension function in python it causes segmentation fault 11. Here's a minimal example below.
#include "Python.h"
#include "numpy/arrayobject.h"
#include <math.h>
static PyObject *myFunc(PyObject *self, PyObject *args)
{
PyArrayObject *out_array;
int dims[1];
dims[0] = 2;
out_array = (PyArrayObject *) PyArray_FromDims(1,dims,NPY_DOUBLE);
// return Py_BuildValue("i", 1); // if I swap this return value it works
return PyArray_Return(out_array);
}
static PyMethodDef minExMethiods[] = {
{"myFunc", myFunc, METH_VARARGS},
{NULL, NULL} /* Sentinel - marks the end of this structure */
};
static struct PyModuleDef minExModule = {
PyModuleDef_HEAD_INIT,
"minEx", /* name of module */
NULL, /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module,
or -1 if the module keeps state in global variables. */
minExMethiods
};
PyMODINIT_FUNC PyInit_minEx(void)
{
return PyModule_Create(&minExModule);
}
Can anyone suggest what I might be doing wrong? I'm using conda with a python 3.6 environment on OS X 10.13.6
thanks
well showing my inexperience here but hopefully useful to others.
I needed to call. import_array() in the modules initialisation function to use numpy's c helping functions properly. I also then got errors that PyArray_FromDims was depreciated. the fixed code bellow.
#include "Python.h"
#include "numpy/arrayobject.h"
#include <math.h>
static PyObject *myFunc(PyObject *self, PyObject *args)
{
PyArrayObject *out_array;
npy_intp dims[1];
dims[0] = 2;
out_array = (PyArrayObject *) PyArray_SimpleNew(1,dims,PyArray_DOUBLE);
return PyArray_Return(out_array);
}
static PyMethodDef minExMethiods[] = {
{"myFunc", myFunc, METH_VARARGS},
{NULL, NULL} /* Sentinel - marks the end of this structure */
};
static struct PyModuleDef minExModule = {
PyModuleDef_HEAD_INIT,
"minEx", /* name of module */
NULL, /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module,
or -1 if the module keeps state in global variables. */
minExMethiods
};
PyMODINIT_FUNC PyInit_minEx(void)
{
PyObject *module = PyModule_Create(&minExModule);
import_array();
return module;
}
I'm building C application which will be using Python plugins. When trying to call the method from another Python module, the function PyImport_ImportModule() seems to imports the module properly, then i try to get the function from this module using PyObject_GetAttrString() and all that I get is null.
I already tried using PyModule_GetDict() and PyDict_GetItemString() to get the method from the module, but the effect was the same.
main.c:
#include <stdio.h>
#include <python3.6/Python.h>
int main()
{
PyObject *arg, *pModule, *ret, *pFunc, *pValue, *pMethod, *pDict;
Py_Initialize();
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Append(path, PyUnicode_FromString("."));
pModule = PyImport_ImportModule("test");
if(pModule == NULL)
{
perror("Can't open module");
}
pMethod = PyObject_GetAttrString(pModule, "myfun");
if(pMethod == NULL)
{
perror("Can't find method");
}
ret = PyEval_CallObject(pMethod, NULL);
if(ret == NULL)
{
perror("Couldn't call method");
}
PyArg_Parse(ret, "&d", pValue);
printf("&d \n", pValue);
Py_Finalize();
return 0;
}
test.py:
def myfun():
c = 123 + 123
print('the result is: ', c)
myfun()
The result i got is:
Can't find method: Success
Segmentation fault (core dumped)
When I used the gdb debugger the output was:
pModule = (PyObject *) 0x7ffff5a96f48
pMethod = (PyObject *) 0x0
Your program is not wroking because the module being imported is the test built-in module, rather than your test.py script. This is because you are appending the current directory to sys.path, so it is checked after every other already existing path in the list. You should insert it at the beginning of the list instead, so that it is checked first.
This will work:
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Insert(path, 0, PyUnicode_FromString("."));
By the way, you should #include the Python header before anything else, as stated in the documentation:
Note: Since Python may define some pre-processor definitions which affect the standard headers on some systems, you must include Python.h before any standard headers are included.
I am trying to achieve the following: passing a python object to a c++ callback chain (which are typical in many popular c++ libraries). In the c++ code, callbacks pass on objects that have necessary information for consecutive callbacks in the cascade/chain.
Here is a small test code I wrote: we pass a python object to a c routine (case 1) and call it's method. That works ok. But when I pass the python object to a c++ object and try to call it "inside" the c++ object, I get segfault.. :(
Here it goes:
c++ module ("some.cpp"):
#include <stdint.h>
#include <iostream>
#include <Python.h>
/* objective:
* create c++ objects / routines that accept python objects
* then call methods of the python objects inside c++
*
* python objects (including its variables and methods) could be passed along, for example in c++ callback chains ..
* .. and in the end we could call a python callback
*
* Compile and test as follows:
* python setup.py build_ext
* [copy/link some.so where test.py is]
* python test.py
*
*/
class testclass {
public:
testclass(int* i, PyObject* po) {
std::cerr << "testclass constructor! \n";
i=i; po=po;
}
~testclass() {}
void runpo() {
PyObject* name;
const char* mname="testmethod";
name=PyString_FromString(mname);
std::cerr << "about to run the python method .. \n";
PyObject_CallMethodObjArgs(po, name, NULL);
std::cerr << ".. you did it - i will buy you a beer!\n";
}
public:
int* i;
PyObject* po;
};
/* Docstrings */
static char module_docstring[] = "hand-made python module";
/* Available functions */
static PyObject* regi_wrapper(PyObject * self, PyObject * args);
void regi(int* i, PyObject* po);
/* Module specification */
static PyMethodDef module_methods[] = {
{"regi_wrapper",regi_wrapper, METH_VARARGS, "lets see if we can wrap this sucker"},
{NULL, NULL, 0, NULL}
};
/* Initialize the module */
PyMODINIT_FUNC initsome(void)
{
PyObject *m = Py_InitModule3("some", module_methods, module_docstring);
if (m == NULL)
return;
// import_array(); // numpy not required here ..
}
static PyObject* regi_wrapper(PyObject * self, PyObject * args)
{
int* input_i; // whatever input variable
PyObject* input_po; // python object
PyObject* ret; // return variable
// parse arguments
if (!PyArg_ParseTuple(args, "iO", &input_i, &input_po)) {
return NULL;
}
// https://stackoverflow.com/questions/16606872/calling-python-method-from-c-or-c-callback
// Py_INCREF(input_po); // need this, right? .. makes no difference
/* // seems not to make any difference ..
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
*/
regi(input_i, input_po);
// PyGILState_Release(gstate); // .. makes no difference
// Py_DECREF(input_po); // .. makes no difference
Py_RETURN_TRUE;
}
void regi(int* i, PyObject* po) {
// search variable and methods from PyObject "po" and call its methods?
PyObject* name;
const char* mname="testmethod";
testclass* testobj;
testobj=new testclass(i,po);
/* [insert // in front of this line to test case 1]
// ***** this one works! *********
name=PyString_FromString(mname);
PyObject_CallMethodObjArgs(po, name, NULL);
*/ // [insert // in front of this line to test case 1]
// *** I WOULD LIKE THIS TO WORK *** but it gives segfault.. :(
testobj->runpo(); // [uncomment to test case 2]
}
setup.py:
from distutils.core import setup, Extension
# the c++ extension module
extension_mod = Extension("some", ["some.cpp"])
setup(name = "some", ext_modules=[extension_mod])
test.py:
import some
class sentinel:
def __init__(self):
pass
def testmethod(self):
print "hello from sentinel.testmethod"
pass
se=sentinel()
some.regi_wrapper(1,se)
This question seems relevant:
Calling python method from C++ (or C) callback
.. however the answer did not help me.
What am I missing/misunderstanding here (my c++ sucks big time, so I might have missed something obvious) .. ?
Also, some bonus questions:
a) I am familiar with swig and swig "directors".. however, I would like to use swig for general wrapping of the code, but my custom wrapping for the sort of things described in this question (i.e. without directors). Is there any way to achieve this?
b) Any other suggestions to achieve what I am trying to achieve here, are highly appreciated.. is this possible or just pure insanity?
Using in the constructor
po=this->po
solves the "issue". Sorry for the spam! I will leave here this thing as an example.. maybe someone finds it useful.
I am getting a segmentation fault when decrefing a PyObject* in my C++ code using the Python/C API, and I can't figure out why. I am using C++ and Python 2.7. I am using new-style classes for future Python 3 compatibility.
My goal is to create a C++ class MyClass to serve as a wrapper for a class defined in a Python module. In the MyClass constructor, I pass in the name of the Python module, import the module, locate the class (which always has a pre-defined name PyClass), and call that class to create an instance of it. I then store the resulting PyObject* in MyClass for future use. In the MyClass destructor, I decref that stored PyObject* to avoid memory leaks.
I have already verified that everything is working correctly as far as locating the class and creating an instance of it. I have even verified that I can use the stored PyObject* in other MyClass methods, for example, to access methods in the PyClass. However, when the destructor does the decref, it causes a segfault.
Here is a sample of my code. I also call Py_Initialize() and Py_Finalize() elsewhere at appropriate times, and I have left out some of my error-checking code for brevity:
MyPythonModule.py
class PyClass:
pass
MyClass.h
class MyClass {
public:
MyClass(const char* modulename);
~MyClass();
private:
void* _StoredPtr;
};
MyClass.cpp
#include <Python.h>
#include <iostream>
#include "MyClass.h"
MyClass::MyClass(const char* modulename) {
_StoredPtr = NULL;
PyObject *pName = NULL, *pModule = NULL, *pAttr = NULL;
// Import the Python module.
pName = PyString_FromString(modulename);
if (pName == NULL) {goto error;}
pModule = PyImport_Import(pName);
if (pModule == NULL) {goto error;}
// Create a PyClass instance and store a pointer to it.
pAttr = PyObject_GetAttrString(pModule, "PyClass");
if (pAttr == NULL) {goto error;}
_StoredPtr = (void*) PyObject_CallObject(pAttr, NULL);
Py_DECREF(pAttr);
if (_StoredPtr == NULL) {goto error;}
error:
if (PyErr_Occurred()) {PyErr_Print();}
Py_XDECREF(pName);
Py_XDECREF(pModule);
return;
}
MyClass::~MyClass() {
std::cout << "Starting destructor..." << std::endl;
Py_XDECREF((PyObject*)(_StoredPtr));
std::cout << "Destructor complete." << std::endl;
}
I know that I could avoid the segfault by leaving out the Py_XDECREF() in the destructor, but I am afraid of causing a memory leak because I do not understand exactly why this is happening. It seems especially strange that I can use _StoredPtr successfully in other MyClass methods, yet I can't decref it.
I have also tried storing the PyObject* of the imported module in MyClass and holding on to it until after _StoredPtr is decrefed, but the _StoredPtr decref still segfaults. I tried commenting out the Py_DECREF(pAttr); line, but that doesn't help.
As I mentioned, I can retrieve methods in the PyClass using _StoredPtr, and I have also tried storing these in MyClass and decrefing them in the destructor. When I do this, I can decref _StoredPtr, but then it segfaults when I try to decref the method's PyObject*. If I do this with several methods, it is always the last decref that causes the segfault, no matter what order I put them in.
Any insights as to what's happening here?
This works for me
#include <Python.h>
#include <iostream>
#include "MyClass.h"
MyClass::MyClass(const char* modulename) {
_StoredPtr = NULL;
PyObject *pName = NULL, *pModule = NULL, *pAttr = NULL;
// Import the Python module.
pName = PyString_FromString(modulename);
if (pName == NULL) {goto error;}
pModule = PyImport_Import(pName);
if (pModule == NULL) {goto error;}
// Create a PyClass instance and store a pointer to it.
pAttr = PyObject_GetAttrString(pModule, "PyClass");
if (pAttr == NULL) {goto error;}
_StoredPtr = (void*) PyObject_CallObject(pAttr, NULL);
Py_DECREF(pAttr);
if (_StoredPtr == NULL) {goto error;}
else{
// do something with _StoredPtr
Py_XDECREF((*PyObject)_StoredPtr)
}
error:
if (PyErr_Occurred()) {PyErr_Print();}
Py_XDECREF(pName);
Py_XDECREF(pModule);
return;
}
MyClass::~MyClass() {}
I basically moved the XDECREF outside the destructor into the function that is using the PyObject.