I am getting a segmentation fault when decrefing a PyObject* in my C++ code using the Python/C API, and I can't figure out why. I am using C++ and Python 2.7. I am using new-style classes for future Python 3 compatibility.
My goal is to create a C++ class MyClass to serve as a wrapper for a class defined in a Python module. In the MyClass constructor, I pass in the name of the Python module, import the module, locate the class (which always has a pre-defined name PyClass), and call that class to create an instance of it. I then store the resulting PyObject* in MyClass for future use. In the MyClass destructor, I decref that stored PyObject* to avoid memory leaks.
I have already verified that everything is working correctly as far as locating the class and creating an instance of it. I have even verified that I can use the stored PyObject* in other MyClass methods, for example, to access methods in the PyClass. However, when the destructor does the decref, it causes a segfault.
Here is a sample of my code. I also call Py_Initialize() and Py_Finalize() elsewhere at appropriate times, and I have left out some of my error-checking code for brevity:
MyPythonModule.py
class PyClass:
pass
MyClass.h
class MyClass {
public:
MyClass(const char* modulename);
~MyClass();
private:
void* _StoredPtr;
};
MyClass.cpp
#include <Python.h>
#include <iostream>
#include "MyClass.h"
MyClass::MyClass(const char* modulename) {
_StoredPtr = NULL;
PyObject *pName = NULL, *pModule = NULL, *pAttr = NULL;
// Import the Python module.
pName = PyString_FromString(modulename);
if (pName == NULL) {goto error;}
pModule = PyImport_Import(pName);
if (pModule == NULL) {goto error;}
// Create a PyClass instance and store a pointer to it.
pAttr = PyObject_GetAttrString(pModule, "PyClass");
if (pAttr == NULL) {goto error;}
_StoredPtr = (void*) PyObject_CallObject(pAttr, NULL);
Py_DECREF(pAttr);
if (_StoredPtr == NULL) {goto error;}
error:
if (PyErr_Occurred()) {PyErr_Print();}
Py_XDECREF(pName);
Py_XDECREF(pModule);
return;
}
MyClass::~MyClass() {
std::cout << "Starting destructor..." << std::endl;
Py_XDECREF((PyObject*)(_StoredPtr));
std::cout << "Destructor complete." << std::endl;
}
I know that I could avoid the segfault by leaving out the Py_XDECREF() in the destructor, but I am afraid of causing a memory leak because I do not understand exactly why this is happening. It seems especially strange that I can use _StoredPtr successfully in other MyClass methods, yet I can't decref it.
I have also tried storing the PyObject* of the imported module in MyClass and holding on to it until after _StoredPtr is decrefed, but the _StoredPtr decref still segfaults. I tried commenting out the Py_DECREF(pAttr); line, but that doesn't help.
As I mentioned, I can retrieve methods in the PyClass using _StoredPtr, and I have also tried storing these in MyClass and decrefing them in the destructor. When I do this, I can decref _StoredPtr, but then it segfaults when I try to decref the method's PyObject*. If I do this with several methods, it is always the last decref that causes the segfault, no matter what order I put them in.
Any insights as to what's happening here?
This works for me
#include <Python.h>
#include <iostream>
#include "MyClass.h"
MyClass::MyClass(const char* modulename) {
_StoredPtr = NULL;
PyObject *pName = NULL, *pModule = NULL, *pAttr = NULL;
// Import the Python module.
pName = PyString_FromString(modulename);
if (pName == NULL) {goto error;}
pModule = PyImport_Import(pName);
if (pModule == NULL) {goto error;}
// Create a PyClass instance and store a pointer to it.
pAttr = PyObject_GetAttrString(pModule, "PyClass");
if (pAttr == NULL) {goto error;}
_StoredPtr = (void*) PyObject_CallObject(pAttr, NULL);
Py_DECREF(pAttr);
if (_StoredPtr == NULL) {goto error;}
else{
// do something with _StoredPtr
Py_XDECREF((*PyObject)_StoredPtr)
}
error:
if (PyErr_Occurred()) {PyErr_Print();}
Py_XDECREF(pName);
Py_XDECREF(pModule);
return;
}
MyClass::~MyClass() {}
I basically moved the XDECREF outside the destructor into the function that is using the PyObject.
Related
I'm trying to embed Python in a C++ application, and I need it to run as a release build, as I am only interested in debugging the C++ code. Also, I do not have the _d debug versions of all libraries I need. I am using Python 3.7.0, in MSVC 2017 with C++11, the library I don't have debug symbols for is VTK, which I installed through a wheel file supplied by my employer. If I try to build the Python wrapper myself I get another load of issues, so I am unable to build the debug files. If I run in Debug mode, I am unable to import the library, whereas if I run in Release, I get further issues:
Exception thrown at 0x00007FF90841DBC9 (python37_d.dll) in PythonEmbedding.exe: 0xC0000005: Access violation reading location 0x0000000000000025.
The C++ code:
int main(int argc, char *argv[])
{
PyObject *pIntrospector = NULL;
if (PyVtk_InitIntrospector(pIntrospector) == OK)
{
printf("Initialization succeeded\n");
}
vtkObjectBase *pConeSource = NULL;
if (PyVtk_CreateVtkObject(pIntrospector, "vtkConeSource", pConeSource) == OK)
{
printf("Object creation succeeded\n");
}
return 0;
}
int PyVtk_InitIntrospector(
PyObject *pIntrospector)
{
/* Activating virtual environment */
#ifdef _DEBUG
// For Visual Studio debug builds
const wchar_t *sPyHome = L"venv-dbg";
#else
// For release builds
const wchar_t *sPyHome = L"venv";
#endif
Py_SetPythonHome(sPyHome);
/* Initializing Python environment and setting PYTHONPATH. */
Py_Initialize();
PyRun_SimpleString("import sys\nimport os");
PyRun_SimpleString("sys.path.append( os.path.dirname(os.getcwd()) )");
PyRun_SimpleString("sys.path.append(\".\")");
PyRun_SimpleString("import importlib.machinery as m");
PyRun_SimpleString("print(m.all_suffixes())");
/* Decode module from its name. Returns error if the name is not decodable. */
PyObject *pIntrospectorModuleName = PyUnicode_DecodeFSDefault("Introspector");
if (pIntrospectorModuleName == NULL)
{
fprintf(stderr, "Fatal error: cannot decode module name\n");
return PYTHON_INTROSPECTION_STRING_DECODE_ERROR;
}
/* Imports the module previously decoded. Returns error if the module is not found. */
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
if (pIntrospectorModule == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Failed to load \"Introspector\"\n");
Py_DECREF(pIntrospectorModuleName);
return PYTHON_INTROSPECTION_MODULE_LOAD_ERROR;
}
/* Looks for the Introspector class in the module. If it does not find it, returns and error. */
PyObject* pIntrospectorClass = PyObject_GetAttrString(pIntrospectorModule, "Introspector");
if (pIntrospectorClass == NULL || !PyCallable_Check(pIntrospectorClass))
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Cannot find class \"Introspector\"\n");
if (pIntrospectorClass != NULL)
{
Py_DECREF(pIntrospectorClass);
}
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
return PYTHON_INTROSPECTION_CLASS_NOT_FOUND_ERROR;
}
/* Instantiates an Introspector object. If the call returns NULL there was an error
creating the object, and thus it returns error. */
pIntrospector = PyObject_CallObject(pIntrospectorClass, NULL);
if (pIntrospector == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Introspector instantiation failed\n");
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return PYTHON_INTROSPECTION_OBJECT_CREATION_ERROR;
}
/* Decreasing reference to local data. */
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return OK;
}
I have not added the code to the PyVtk_CreateVtkObject function as it won't enter it, but if I do not add the calls after PyVtk_InitIntroepsctor it won't give the aforementioned error. Finally, if I import Introspector in the Python interpreter myself, it works fine.
Is there a solution to either run it in Debug or Release? I cannot wrap my head around it...
P.S.: I already tried to use Boost::Python, I have two issues open on it as it is giving me problems as well.
Update 1: In particular, the excpetion is thrown when I do this:
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
Update 2: I have further scoped down the issue to this: whenever I import the vtk package from within the embedded interpreter, it throws the Access Violation on this code:
// Add special attribute __vtkname__
PyObject *s = PyString_FromString(classname);
PyDict_SetItemString(pytype->tp_dict, "__vtkname__", s);
Py_DECREF(s); // <-- In particular on this Py_DECREF
If I try to import anything else, there is no issue, it seems.
I am embedding Python using C embedding API. The main thread does
Py_Initialize();
PyEval_InitThreads();
Then I have more threads created by native code, whose lifetime I do not control. They need to call Python too. So far they seemed to be working fine with
void* gil = PyGILState_Ensure();
calls to Python go here
PyGILState_Release(gil);
Problem is given this simple setup I faced issues with Python code, that uses threading.local. Imagine secondary thread S, that periodically executes increase_counter:
// initialized once at the beginning of program to
// an instance of threading.local()
PyObject* threadLocal;
...
void increase_counter()
{
void* gil = PyGILState_Ensure();
// this really does C API calls, but for simplicity I'll write Python
if hasattr(threadLocal, "counter"):
threadLocal.counter += 1
else:
threadLocal.counter = 1
// end of Python
PyGILState_Release(gil);
}
Well, the problem is that in thread S multiple calls to increase_counter don't actually increase anything - hasattr always returns False, and the value of counter for this thread is discarded as soon as PyGILState_Release is called.
It only works correctly in S if the whole body of S is wrapped into:
void* gilForS = PyGILState_Ensure();
void* sPythonThreadState = PyEval_SaveThread();
// rest of the S body, that sometimes calls increase_counter
PyEval_RestoreThread(sPythonThreadState);
PyGILState_Release(gilForS);
Which I can do for the purpose of this question, but in the actual product the lifetime of S is not controlled by me (it is a thread pool thread), only increase_counter, so I can't make it run PyEval_SaveThread in the beginning, and I can't ensure PyEval_RestoreThread will be called in the end.
What is the proper way to initialize threads like S so that threading.local would correctly work there?
Full example, that reproduces the issue as requested. Prints "set counter" twice instead of "set counter" + "Found counter!". It works as expected if I uncomment code in async_thread which I can't do in real product:
#include <Python.h>
#include <pthread.h>
PyObject *threadLocal;
void *async_thread(void *arg);
void increase_counter();
int main(int argc, char *argv[])
{
Py_Initialize();
PyEval_InitThreads();
PyObject *threading = PyImport_ImportModule("threading");
PyObject *tlocal = PyObject_GetAttrString(threading, "local");
threadLocal = PyObject_Call(tlocal, PyTuple_New(0), NULL);
pthread_t async;
int err = pthread_create(&async, NULL, async_thread, NULL);
if (err)
{
printf("unable to create thread\n");
exit(-1);
}
PyThreadState* ts = PyEval_SaveThread();
pthread_join(async, NULL);
PyEval_RestoreThread(ts);
Py_Finalize();
pthread_exit(NULL);
}
void *async_thread(void *arg)
{
//PyGILState_STATE gil = PyGILState_Ensure();
for (int i = 0; i < 2; i++)
{
increase_counter();
}
//PyGILState_Release(gil);
pthread_exit(NULL);
return NULL;
}
void increase_counter()
{
PyGILState_STATE gil = PyGILState_Ensure();
if (PyObject_HasAttrString(threadLocal, "counter"))
{
printf("Found counter!\n");
}
else
{
PyObject *val = PyLong_FromLong(1);
PyObject_SetAttrString(threadLocal, "counter", val);
printf("set counter\n");
}
PyGILState_Release(gil);
}
I'm building C application which will be using Python plugins. When trying to call the method from another Python module, the function PyImport_ImportModule() seems to imports the module properly, then i try to get the function from this module using PyObject_GetAttrString() and all that I get is null.
I already tried using PyModule_GetDict() and PyDict_GetItemString() to get the method from the module, but the effect was the same.
main.c:
#include <stdio.h>
#include <python3.6/Python.h>
int main()
{
PyObject *arg, *pModule, *ret, *pFunc, *pValue, *pMethod, *pDict;
Py_Initialize();
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Append(path, PyUnicode_FromString("."));
pModule = PyImport_ImportModule("test");
if(pModule == NULL)
{
perror("Can't open module");
}
pMethod = PyObject_GetAttrString(pModule, "myfun");
if(pMethod == NULL)
{
perror("Can't find method");
}
ret = PyEval_CallObject(pMethod, NULL);
if(ret == NULL)
{
perror("Couldn't call method");
}
PyArg_Parse(ret, "&d", pValue);
printf("&d \n", pValue);
Py_Finalize();
return 0;
}
test.py:
def myfun():
c = 123 + 123
print('the result is: ', c)
myfun()
The result i got is:
Can't find method: Success
Segmentation fault (core dumped)
When I used the gdb debugger the output was:
pModule = (PyObject *) 0x7ffff5a96f48
pMethod = (PyObject *) 0x0
Your program is not wroking because the module being imported is the test built-in module, rather than your test.py script. This is because you are appending the current directory to sys.path, so it is checked after every other already existing path in the list. You should insert it at the beginning of the list instead, so that it is checked first.
This will work:
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Insert(path, 0, PyUnicode_FromString("."));
By the way, you should #include the Python header before anything else, as stated in the documentation:
Note: Since Python may define some pre-processor definitions which affect the standard headers on some systems, you must include Python.h before any standard headers are included.
I am trying to achieve the following: passing a python object to a c++ callback chain (which are typical in many popular c++ libraries). In the c++ code, callbacks pass on objects that have necessary information for consecutive callbacks in the cascade/chain.
Here is a small test code I wrote: we pass a python object to a c routine (case 1) and call it's method. That works ok. But when I pass the python object to a c++ object and try to call it "inside" the c++ object, I get segfault.. :(
Here it goes:
c++ module ("some.cpp"):
#include <stdint.h>
#include <iostream>
#include <Python.h>
/* objective:
* create c++ objects / routines that accept python objects
* then call methods of the python objects inside c++
*
* python objects (including its variables and methods) could be passed along, for example in c++ callback chains ..
* .. and in the end we could call a python callback
*
* Compile and test as follows:
* python setup.py build_ext
* [copy/link some.so where test.py is]
* python test.py
*
*/
class testclass {
public:
testclass(int* i, PyObject* po) {
std::cerr << "testclass constructor! \n";
i=i; po=po;
}
~testclass() {}
void runpo() {
PyObject* name;
const char* mname="testmethod";
name=PyString_FromString(mname);
std::cerr << "about to run the python method .. \n";
PyObject_CallMethodObjArgs(po, name, NULL);
std::cerr << ".. you did it - i will buy you a beer!\n";
}
public:
int* i;
PyObject* po;
};
/* Docstrings */
static char module_docstring[] = "hand-made python module";
/* Available functions */
static PyObject* regi_wrapper(PyObject * self, PyObject * args);
void regi(int* i, PyObject* po);
/* Module specification */
static PyMethodDef module_methods[] = {
{"regi_wrapper",regi_wrapper, METH_VARARGS, "lets see if we can wrap this sucker"},
{NULL, NULL, 0, NULL}
};
/* Initialize the module */
PyMODINIT_FUNC initsome(void)
{
PyObject *m = Py_InitModule3("some", module_methods, module_docstring);
if (m == NULL)
return;
// import_array(); // numpy not required here ..
}
static PyObject* regi_wrapper(PyObject * self, PyObject * args)
{
int* input_i; // whatever input variable
PyObject* input_po; // python object
PyObject* ret; // return variable
// parse arguments
if (!PyArg_ParseTuple(args, "iO", &input_i, &input_po)) {
return NULL;
}
// https://stackoverflow.com/questions/16606872/calling-python-method-from-c-or-c-callback
// Py_INCREF(input_po); // need this, right? .. makes no difference
/* // seems not to make any difference ..
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
*/
regi(input_i, input_po);
// PyGILState_Release(gstate); // .. makes no difference
// Py_DECREF(input_po); // .. makes no difference
Py_RETURN_TRUE;
}
void regi(int* i, PyObject* po) {
// search variable and methods from PyObject "po" and call its methods?
PyObject* name;
const char* mname="testmethod";
testclass* testobj;
testobj=new testclass(i,po);
/* [insert // in front of this line to test case 1]
// ***** this one works! *********
name=PyString_FromString(mname);
PyObject_CallMethodObjArgs(po, name, NULL);
*/ // [insert // in front of this line to test case 1]
// *** I WOULD LIKE THIS TO WORK *** but it gives segfault.. :(
testobj->runpo(); // [uncomment to test case 2]
}
setup.py:
from distutils.core import setup, Extension
# the c++ extension module
extension_mod = Extension("some", ["some.cpp"])
setup(name = "some", ext_modules=[extension_mod])
test.py:
import some
class sentinel:
def __init__(self):
pass
def testmethod(self):
print "hello from sentinel.testmethod"
pass
se=sentinel()
some.regi_wrapper(1,se)
This question seems relevant:
Calling python method from C++ (or C) callback
.. however the answer did not help me.
What am I missing/misunderstanding here (my c++ sucks big time, so I might have missed something obvious) .. ?
Also, some bonus questions:
a) I am familiar with swig and swig "directors".. however, I would like to use swig for general wrapping of the code, but my custom wrapping for the sort of things described in this question (i.e. without directors). Is there any way to achieve this?
b) Any other suggestions to achieve what I am trying to achieve here, are highly appreciated.. is this possible or just pure insanity?
Using in the constructor
po=this->po
solves the "issue". Sorry for the spam! I will leave here this thing as an example.. maybe someone finds it useful.
I am trying to make a call to a python module function from my cpp file.
The call i have made is as follows:
#include <iostream>
#include "Python.h"
int
main(int argc, char** argv)
{
Py_Initialize();
PyObject *pName = PyString_FromString("tmpPyth");
PyObject *pModule = PyImport_Import(pName);
std::cout<< "Works fine till here";
PyObject *pDict = PyModule_GetDict(pModule);
if (pModule != NULL) {
PyObject *pFunc = PyObject_GetAttrString(pDict, "pyFunc");
if(pFunc != NULL){
PyObject_CallObject(pFunc, NULL);
}
}
else
std::cout << "Python Module not found";
return 0;
}
My python module is defined as follows:
import numpy
import scipy
import matplotlib
from scipy import stats
def blah():
baseline = [9.74219, 10.2226, 8.7469, 8.69791, 9.96442, 9.96472, 9.37913, 9.75004]
follow_up = [9.94227,9.46763,8.53081,9.43679,9.97695,10.4285,10.159,8.86134]
paired_sample = stats.ttest_rel(baseline , follow_up )
print "The t-statistic is %.3f and the p-value is %.3f." % paired_sample
The code in the cpp file runs fine till the 1st "std::cout" but then ends up giving me a "seg fault". Running the python code separately works fine and gives the desired output.
I cant figure out what is going wrong. Any help will be appreciated.
(Note the program is compiling correctly and running correctly till the 1st "cout")
So there are a couple of things that you were not doing right. See the comments inline. Assuming that both your CPP file and Python file lives at the following path: /home/shanil/project.
test.cpp:
#include <iostream>
#include "Python.h"
int
main(int argc, char** argv)
{
Py_Initialize();
// First set in path where to find your custom python module.
// You have to tell the path otherwise the next line will try to load
// your module from the path where Python's system modules/packages are
// found.
PyObject* sysPath = PySys_GetObject("path");
PyList_Append(sysPath, PyString_FromString("/home/shanil/project"));
// Load the module
PyObject *pName = PyString_FromString("my_mod");
PyObject *pModule = PyImport_Import(pName);
// Random use-less check
std::cout<< "Works fine till here\n";
if (pModule != NULL) {
std::cout << "Python module found\n";
// Load all module level attributes as a dictionary
PyObject *pDict = PyModule_GetDict(pModule);
// Remember that you are loading the module as a dictionary, the lookup you were
// doing on pDict would fail as you were trying to find something as an attribute
// which existed as a key in the dictionary
PyObject *pFunc = PyDict_GetItem(pDict, PyString_FromString("my_func"));
if(pFunc != NULL){
PyObject_CallObject(pFunc, NULL);
} else {
std::cout << "Couldn't find func\n";
}
}
else
std::cout << "Python Module not found\n";
return 0;
}
my_mod.py:
def my_func():
print 'got called'