We Keep Coding

Debugging Python Extensions written in C

I'm pretty familiar with writing C code and I'm comfortable in writing python code. I'm trying to learn how to write modules in C that can be called from Python-3.9.X on OSX 10.15.7. I've gotten a couple 'hello world' type of examples to work, but for complex examples I'm struggling to figure out how I would debug the C-extensions that I write.
MWE:
src/add.c
// The C function that actually does the work
static PyObject * add_c_func(PyObject *self, PyObject *args)
{
int a=0;
int b=0;
int c=0;
// This determines the number arguments used by add_c_func
if (!PyArg_ParseTuple(args, "iii", &a, &b, &c))
{
return NULL;
}
printf("%i\n", a+b+c);
Py_RETURN_NONE;
}
// This defines the function used by
static PyMethodDef AddMethods[] = {
{"add_py_func", add_c_func, METH_VARARGS, "Add three numbers."},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef addpymod =
{
PyModuleDef_HEAD_INIT,
"addpymod", /* name of module */
"", /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
AddMethods
};
PyMODINIT_FUNC PyInit_addpymod(void)
{
return PyModule_Create(&addpymod);
}
setup.py :
from setuptools import setup, Extension
setup(
name='addpymod',
version='1.0',
description='Python Package with Hello World C Extension',
ext_modules=[
Extension(
'addpymod',
sources=['src/add.c'],
py_limited_api=True)
],
)
Compiling / installing (by default it uses clang):
python setup.py install
Trying to debug :
(py-ext-test) local: understand-python-c-ext $ gdb
GNU gdb (GDB) 10.1
.
.
.
(gdb) b add.c : 20
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (add.c : 20) pending.
(gdb) python
>import addpymod
>addpymod.add_py_func(10,10,10) # Why didn't my breakpoint get hit?
>Quit
# This clearly failed, I'm not even sure where my stdout is
There are multiple levels of complexity here and I'm sure that I'm being tripped by more than one.
Question :
How do I debug my add.c using gdb (preferred) or possibly lldb since it was compiled by default with clang's -g option?

Embedded Python still looking for debug symbols even with _DEBUG undefined

I'm trying to embed Python in a C++ application, and I need it to run as a release build, as I am only interested in debugging the C++ code. Also, I do not have the _d debug versions of all libraries I need. I am using Python 3.7.0, in MSVC 2017 with C++11, the library I don't have debug symbols for is VTK, which I installed through a wheel file supplied by my employer. If I try to build the Python wrapper myself I get another load of issues, so I am unable to build the debug files. If I run in Debug mode, I am unable to import the library, whereas if I run in Release, I get further issues:
Exception thrown at 0x00007FF90841DBC9 (python37_d.dll) in PythonEmbedding.exe: 0xC0000005: Access violation reading location 0x0000000000000025.
The C++ code:
int main(int argc, char *argv[])
{
PyObject *pIntrospector = NULL;
if (PyVtk_InitIntrospector(pIntrospector) == OK)
{
printf("Initialization succeeded\n");
}
vtkObjectBase *pConeSource = NULL;
if (PyVtk_CreateVtkObject(pIntrospector, "vtkConeSource", pConeSource) == OK)
{
printf("Object creation succeeded\n");
}
return 0;
}
int PyVtk_InitIntrospector(
PyObject *pIntrospector)
{
/* Activating virtual environment */
#ifdef _DEBUG
// For Visual Studio debug builds
const wchar_t *sPyHome = L"venv-dbg";
#else
// For release builds
const wchar_t *sPyHome = L"venv";
#endif
Py_SetPythonHome(sPyHome);
/* Initializing Python environment and setting PYTHONPATH. */
Py_Initialize();
PyRun_SimpleString("import sys\nimport os");
PyRun_SimpleString("sys.path.append( os.path.dirname(os.getcwd()) )");
PyRun_SimpleString("sys.path.append(\".\")");
PyRun_SimpleString("import importlib.machinery as m");
PyRun_SimpleString("print(m.all_suffixes())");
/* Decode module from its name. Returns error if the name is not decodable. */
PyObject *pIntrospectorModuleName = PyUnicode_DecodeFSDefault("Introspector");
if (pIntrospectorModuleName == NULL)
{
fprintf(stderr, "Fatal error: cannot decode module name\n");
return PYTHON_INTROSPECTION_STRING_DECODE_ERROR;
}
/* Imports the module previously decoded. Returns error if the module is not found. */
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
if (pIntrospectorModule == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Failed to load \"Introspector\"\n");
Py_DECREF(pIntrospectorModuleName);
return PYTHON_INTROSPECTION_MODULE_LOAD_ERROR;
}
/* Looks for the Introspector class in the module. If it does not find it, returns and error. */
PyObject* pIntrospectorClass = PyObject_GetAttrString(pIntrospectorModule, "Introspector");
if (pIntrospectorClass == NULL || !PyCallable_Check(pIntrospectorClass))
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Cannot find class \"Introspector\"\n");
if (pIntrospectorClass != NULL)
{
Py_DECREF(pIntrospectorClass);
}
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
return PYTHON_INTROSPECTION_CLASS_NOT_FOUND_ERROR;
}
/* Instantiates an Introspector object. If the call returns NULL there was an error
creating the object, and thus it returns error. */
pIntrospector = PyObject_CallObject(pIntrospectorClass, NULL);
if (pIntrospector == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Introspector instantiation failed\n");
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return PYTHON_INTROSPECTION_OBJECT_CREATION_ERROR;
}
/* Decreasing reference to local data. */
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return OK;
}
I have not added the code to the PyVtk_CreateVtkObject function as it won't enter it, but if I do not add the calls after PyVtk_InitIntroepsctor it won't give the aforementioned error. Finally, if I import Introspector in the Python interpreter myself, it works fine.
Is there a solution to either run it in Debug or Release? I cannot wrap my head around it...
P.S.: I already tried to use Boost::Python, I have two issues open on it as it is giving me problems as well.
Update 1: In particular, the excpetion is thrown when I do this:
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
Update 2: I have further scoped down the issue to this: whenever I import the vtk package from within the embedded interpreter, it throws the Access Violation on this code:
// Add special attribute __vtkname__
PyObject *s = PyString_FromString(classname);
PyDict_SetItemString(pytype->tp_dict, "__vtkname__", s);
Py_DECREF(s); // <-- In particular on this Py_DECREF
If I try to import anything else, there is no issue, it seems.

Return vector from C++ to Python without copying using SWIG

I am trying to build a python module in C++ using SWIG. I have the following code:
struct ContainerElement
{
uint8_t i;
double d;
double d2;
};
class Container
{
private:
std::vector<uint8_t> ints;
std::vector<double> doubles;
std::vector<double> doubles2;
public:
std::vector<uint8_t>& getInts() { return ints; }
std::vector<double>& getDoubles() { return doubles; }
std::vector<double>& getDoubles2() { return doubles2; }
void addElement(ContainerElement element)
{
ints.emplace_back(element.i);
doubles.emplace_back(element.d);
doubles2.emplace_back(element.d2);
}
};
void fillContainer(Container& container)
{
for (int i = 0; i < 1e6; ++i)
{
container.addElement({(uint8_t)i, (double)i,(double)i });
}
}
And an interface file container.i:
%module container
%{
#include "container.h"
%}
%include container.h
%include "std_vector.i"
%include "stdint.i"
%template(DoubleVector) std::vector<double>;
%template(IntVector) std::vector<uint8_t>;
When I call this code in python:
import psutil
import os
import container
import numpy as np
print(psutil.Process(os.getpid()).memory_info().rss)
cont = container.Container()
container.fillContainer(cont)
print(psutil.Process(os.getpid()).memory_info().rss)
a = np.array(cont.getInts(), copy=False)
b = np.array(cont.getDoubles(), copy=False)
c = np.array(cont.getDoubles2(), copy=False)
print(psutil.Process(os.getpid()).memory_info().rss)
This works, however when I check the memory usage of the program (using psutil.Process(os.getpid()).memory_info().rss) it seems to make a copy when I call the functions getInts, getDoubles and getDoubles2. Is there a way to avoid this?

Numpy can't do anything with a vector object: it needs the underlying memory. The easiest would have been if you could call the vector's data(), but SWIG does not expose that function. Just for reference, here's cppyy (http://cppyy.org) which does expose data() to show that that simply works, i.e. that the problem isn't with numpy:
import cppyy
cppyy.include("container.h")
import psutil
import os
import numpy as np
print(psutil.Process(os.getpid()).memory_info().rss)
cont = cppyy.gbl.Container()
cppyy.gbl.fillContainer(cont)
print(psutil.Process(os.getpid()).memory_info().rss)
a = np.array(cont.getInts().data(), copy=False)
b = np.array(cont.getDoubles().data(), copy=False)
c = np.array(cont.getDoubles2().data(), copy=False)
print(psutil.Process(os.getpid()).memory_info().rss)
(Of course cppyy takes more memory straight of the bat, b/c it loads LLVM; incrementally it will win however.)
So first you need to expose that data() call through SWIG and then I think it's easiest to add a python layering on top of that, to make the usage clean to client code (with the caveat that you may run into memory issues if the original vectors get deleted or resized, but that's always a problem for any code that does not copy).
Start by modifying your container code:
// include the numpy header; find its location with:
// python -c 'import numpy; print(numpy.get_include())'
#include "numpy/arrayobject.h"
// ... rest of your container code here ...
// data to numpy helpers
PyObject* vec_data_uint8_t(std::vector<uint8_t>& d) {
import_array();
npy_intp dims[] = {(npy_intp)d.size()};
npy_intp strides[] = {sizeof(std::vector<uint8_t>::value_type)};
return PyArray_NewFromDescr(&PyArray_Type, PyArray_DescrFromType(NPY_UINT8),
1, dims, strides, d.data(), NPY_ARRAY_C_CONTIGUOUS, nullptr);
}
PyObject* vec_data_double(std::vector<double>& d) {
import_array();
npy_intp dims[] = {(npy_intp)d.size()};
npy_intp strides[] = {sizeof(std::vector<double>::value_type)};
return PyArray_NewFromDescr(&PyArray_Type, PyArray_DescrFromType(NPY_DOUBLE),
1, dims, strides, d.data(), NPY_ARRAY_C_CONTIGUOUS, nullptr);
}
SWIG will automatically pick those up, so container.i is fine. Now add a mycontainer.py helper module like so:
import numpy
from container import *
IntVector.__array__ = vec_data_uint8_t
DoubleVector.__array__ = vec_data_double
and in your original python code, replace import container with:
import mycontainer as container
Now you can use your vectors as always but when handed to a numpy array, the custom __array__ function will be called, given it access to the internal data and will not copy if copy=False.

Python C Extension - Error in `python3.6': free(): invalid pointer: 0x

I've been learning about extending Python with C and I finally got it to work in some way, but I'm getting this really weird error. I made a test module with a single function to test if a number is prime. Here is the code:
cmodule.c
#include <stdbool.h>
#include <math.h>
#include <Python.h>
bool isprime(int n) {
if (n == 0 || n == 1)
return false;
for (int i = 2; i < (int)sqrt(n) + 1; i++) {
if (n % i == 0)
return false;
}
return true;
}
static PyObject * isprime_wrapper(PyObject * self, PyObject * args) {
int n;
if (!PyArg_ParseTuple(args, "i", &n))
return NULL;
bool retval = isprime(n);
if (retval)
return Py_True;
else
return Py_False;
}
static PyMethodDef methods[] = {
{"isprime", isprime_wrapper, METH_VARARGS, "Tests if a number is prime."},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef cmodule = {
PyModuleDef_HEAD_INIT,
"cmodule",
"Test module.",
-1,
methods
};
PyMODINIT_FUNC PyInit_cmodule(void) {
return PyModule_Create(&cmodule);
}
setup.py
from distutils.core import setup, Extension
cmodule = Extension("cmodule", sources = ["cmodule.c"])
setup(name = "CModuleTest",
ext_modules = [cmodule])
At first glance it works fine. I run python3.6 setup.py build and it builds with no errors, and I can go into the directory with the .so file, start up a Python session, and call cmodule.isprime() fine. However, the weird part is when I use the code in a loop, where I get this error:
>>> import cmodule
>>> for i in range(1, 1000001):
... n = cmodule.isprime(i)
...
*** Error in `python3.6': free(): invalid pointer: 0x0000561cc1faf620 ***
I can add the full backtrace if anyone would find it helpful. I printed out all the values of i while doing this, and it seemed to always be getting this error when it tries to run isprime(154). However, when I just open up Python in the terminal and try running isprime(154) or any other number by itself, it works fine. The only time I've been able to replicate this is in the loop, and I have no clue where it could be coming from. I've looked at other people that have had similar problems and it seemed to stem from them using free() in their C code improperly, but I never call free() from my code. Does anyone have an idea what could be causing this? Thanks for any help.

CPython objects are referenced counted and returning Py_True and Py_False could be the issue. Use Py_RETURN_TRUE, etc. instead. They properly increment the reference count of and return their respective singleton objects.

Calling python object's method from c++

I am trying to achieve the following: passing a python object to a c++ callback chain (which are typical in many popular c++ libraries). In the c++ code, callbacks pass on objects that have necessary information for consecutive callbacks in the cascade/chain.
Here is a small test code I wrote: we pass a python object to a c routine (case 1) and call it's method. That works ok. But when I pass the python object to a c++ object and try to call it "inside" the c++ object, I get segfault.. :(
Here it goes:
c++ module ("some.cpp"):
#include <stdint.h>
#include <iostream>
#include <Python.h>
/* objective:
* create c++ objects / routines that accept python objects
* then call methods of the python objects inside c++
*
* python objects (including its variables and methods) could be passed along, for example in c++ callback chains ..
* .. and in the end we could call a python callback
*
* Compile and test as follows:
* python setup.py build_ext
* [copy/link some.so where test.py is]
* python test.py
*
*/
class testclass {
public:
testclass(int* i, PyObject* po) {
std::cerr << "testclass constructor! \n";
i=i; po=po;
}
~testclass() {}
void runpo() {
PyObject* name;
const char* mname="testmethod";
name=PyString_FromString(mname);
std::cerr << "about to run the python method .. \n";
PyObject_CallMethodObjArgs(po, name, NULL);
std::cerr << ".. you did it - i will buy you a beer!\n";
}
public:
int* i;
PyObject* po;
};
/* Docstrings */
static char module_docstring[] = "hand-made python module";
/* Available functions */
static PyObject* regi_wrapper(PyObject * self, PyObject * args);
void regi(int* i, PyObject* po);
/* Module specification */
static PyMethodDef module_methods[] = {
{"regi_wrapper",regi_wrapper, METH_VARARGS, "lets see if we can wrap this sucker"},
{NULL, NULL, 0, NULL}
};
/* Initialize the module */
PyMODINIT_FUNC initsome(void)
{
PyObject *m = Py_InitModule3("some", module_methods, module_docstring);
if (m == NULL)
return;
// import_array(); // numpy not required here ..
}
static PyObject* regi_wrapper(PyObject * self, PyObject * args)
{
int* input_i; // whatever input variable
PyObject* input_po; // python object
PyObject* ret; // return variable
// parse arguments
if (!PyArg_ParseTuple(args, "iO", &input_i, &input_po)) {
return NULL;
}
// https://stackoverflow.com/questions/16606872/calling-python-method-from-c-or-c-callback
// Py_INCREF(input_po); // need this, right? .. makes no difference
/* // seems not to make any difference ..
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
*/
regi(input_i, input_po);
// PyGILState_Release(gstate); // .. makes no difference
// Py_DECREF(input_po); // .. makes no difference
Py_RETURN_TRUE;
}
void regi(int* i, PyObject* po) {
// search variable and methods from PyObject "po" and call its methods?
PyObject* name;
const char* mname="testmethod";
testclass* testobj;
testobj=new testclass(i,po);
/* [insert // in front of this line to test case 1]
// ***** this one works! *********
name=PyString_FromString(mname);
PyObject_CallMethodObjArgs(po, name, NULL);
*/ // [insert // in front of this line to test case 1]
// *** I WOULD LIKE THIS TO WORK *** but it gives segfault.. :(
testobj->runpo(); // [uncomment to test case 2]
}
setup.py:
from distutils.core import setup, Extension
# the c++ extension module
extension_mod = Extension("some", ["some.cpp"])
setup(name = "some", ext_modules=[extension_mod])
test.py:
import some
class sentinel:
def __init__(self):
pass
def testmethod(self):
print "hello from sentinel.testmethod"
pass
se=sentinel()
some.regi_wrapper(1,se)
This question seems relevant:
Calling python method from C++ (or C) callback
.. however the answer did not help me.
What am I missing/misunderstanding here (my c++ sucks big time, so I might have missed something obvious) .. ?
Also, some bonus questions:
a) I am familiar with swig and swig "directors".. however, I would like to use swig for general wrapping of the code, but my custom wrapping for the sort of things described in this question (i.e. without directors). Is there any way to achieve this?
b) Any other suggestions to achieve what I am trying to achieve here, are highly appreciated.. is this possible or just pure insanity?

Using in the constructor
po=this->po
solves the "issue". Sorry for the spam! I will leave here this thing as an example.. maybe someone finds it useful.