We Keep Coding

Debugging Python Extensions written in C

I'm pretty familiar with writing C code and I'm comfortable in writing python code. I'm trying to learn how to write modules in C that can be called from Python-3.9.X on OSX 10.15.7. I've gotten a couple 'hello world' type of examples to work, but for complex examples I'm struggling to figure out how I would debug the C-extensions that I write.
MWE:
src/add.c
// The C function that actually does the work
static PyObject * add_c_func(PyObject *self, PyObject *args)
{
int a=0;
int b=0;
int c=0;
// This determines the number arguments used by add_c_func
if (!PyArg_ParseTuple(args, "iii", &a, &b, &c))
{
return NULL;
}
printf("%i\n", a+b+c);
Py_RETURN_NONE;
}
// This defines the function used by
static PyMethodDef AddMethods[] = {
{"add_py_func", add_c_func, METH_VARARGS, "Add three numbers."},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef addpymod =
{
PyModuleDef_HEAD_INIT,
"addpymod", /* name of module */
"", /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
AddMethods
};
PyMODINIT_FUNC PyInit_addpymod(void)
{
return PyModule_Create(&addpymod);
}
setup.py :
from setuptools import setup, Extension
setup(
name='addpymod',
version='1.0',
description='Python Package with Hello World C Extension',
ext_modules=[
Extension(
'addpymod',
sources=['src/add.c'],
py_limited_api=True)
],
)
Compiling / installing (by default it uses clang):
python setup.py install
Trying to debug :
(py-ext-test) local: understand-python-c-ext $ gdb
GNU gdb (GDB) 10.1
.
.
.
(gdb) b add.c : 20
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (add.c : 20) pending.
(gdb) python
>import addpymod
>addpymod.add_py_func(10,10,10) # Why didn't my breakpoint get hit?
>Quit
# This clearly failed, I'm not even sure where my stdout is
There are multiple levels of complexity here and I'm sure that I'm being tripped by more than one.
Question :
How do I debug my add.c using gdb (preferred) or possibly lldb since it was compiled by default with clang's -g option?

Calling a C function with gmp library included from a Python code

I am trying to send a string from a function implemented in Python to a function implemented in C, perform some operations using GMP library and return a string to the function implemented in python. Details of code given below:
myModule.c file:
#include<Python.h>
#include<gmp.h>
char* Cnumfunc(char* n)
{
printf("This is the string %s\n",n);
mpz_t p1;
mpz_init(p1);
mpz_set_str(p1,n,10);
gmp_printf("Value of the gmp number is : %Zd \n", p1);
mpz_add_ui(p1,p1,1);//so now p1=11
char retstr[100];//create a buffer of large size
mpz_get_str(retstr,10,p1);
return retstr;//Should return "11"
}
static PyObject* numfunc(PyObject* self, PyObject* args)
{
char* n;
if (!PyArg_ParseTuple(args, "s", &n))
return NULL;
return Py_BuildValue("s", Cnumfunc(n));
}
static PyMethodDef myMethods[] = {
{"numfunc", numfunc, METH_VARARGS, "Calculate the numfunc"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef myModule = {
PyModuleDef_HEAD_INIT,
"myModule", //#name of module.
"numfunc Module",
-1,
myMethods
};
PyMODINIT_FUNC PyInit_myModule(void)
{
return PyModule_Create(&myModule);
}
Now setup.py file:
from distutils.core import setup, Extension
module = Extension('myModule', sources = ['myModule.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a package for myModule',
ext_modules = [module])
And finally the test.py file:
import myModule
a=myModule.numfunc("10")
print(a)
Now I run
python3 setup build
And then I copy the myModule.cpython-38-x86_64-linux-gnu.so file in the build directory to the directory which contains test.py and then do
python3 test.py
This gives the error:
Traceback (most recent call last):
File "test.py", line 1, in <module>
import myModule
ImportError: /home/aghosh/pybind/myModule.cpython-38-x86_64-linux-gnu.so: undefined symbol: __gmpz_set_str
I am really new to this kind of coding involving multiple languages and it will be very helpful if anyone has a solution to this problem

Including external shared intel's mkl library in c extension for python

I wrote a python c extension that uses routine from Intel's math kernel library (mkl). This is the first time that I write a c extension. I just learned that today.
The c extension compiled. But when I import it in python, it says undefined symbol, and can't find a function that is defined in the mkl.
How to include any external c library in a python c extension?
Thank you for your help.
mkl_helper.c:
#include "Python.h"
#include "numpy/arrayobject.h"
#include "mkl.h"
static PyObject* test4 (PyObject *self, PyObject *args)
{
// test4 (m, n,
// a, ja, ia,
// c, jc, ic)
PyArrayObject *shape_array;
PyArrayObject *a_array; // csr_matrix.data
PyArrayObject *ja_array; // csr_matrix.indices
PyArrayObject *ia_array; // csr_matrix.indptr
PyArrayObject *c_array;
PyArrayObject *jc_array;
PyArrayObject *ic_array;
if (!PyArg_ParseTuple(args, "O!O!O!O!O!O!O!",
&PyArray_Type, &shape_array,
&PyArray_Type, &a_array,
&PyArray_Type, &ja_array,
&PyArray_Type, &ia_array,
&PyArray_Type, &c_array,
&PyArray_Type, &jc_array,
&PyArray_Type, &ic_array))
{
return NULL;
}
long * ptr_int = shape_array->data;
int m = ptr_int[0];
int n = ptr_int[1];
int k = n;
float * a_data_ptr = a_array->data;
float * ja_data_ptr = ja_array->data;
float * ia_data_ptr = ia_array->data;
float * c_data_ptr = c_array->data;
float * jc_data_ptr = jc_array->data;
float * ic_data_ptr = ic_array->data;
char trans = 'T';
int sort = 0;
int nzmax = n*n;
int info = -3;
int request = 0;
mkl_scsrmultcsr(&trans, &request, &sort,
&m, &n, &k,
a_data_ptr, ja_data_ptr, ia_data_ptr,
a_data_ptr, ja_data_ptr, ia_data_ptr,
c_data_ptr, jc_data_ptr, ic_data_ptr,
&nzmax, &info);
return PyInt_FromLong(info);
}
static struct PyMethodDef methods[] = {
{"test4", test4, METH_VARARGS, "test2(arr1)\n take a numpy array and return its shape as a tuple"},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC
initmkl_helper (void)
{
(void)Py_InitModule("mkl_helper", methods);
import_array();
}
setup.py:
from distutils.core import setup, Extension
import numpy as np
ext_modules = [ Extension('mkl_helper', sources = ['mkl_helper.c']) ]
setup(
name = 'mkl_helper',
version = '1.0',
include_dirs = [np.get_include()], #Add Include path of numpy
ext_modules = ext_modules
)
test.py:
import mkl_helper
result of running test.py:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: /home/rxu/local/lib/python2.7/site-packages/mkl_helper.so: undefined symbol: mkl_scsrmultcsr
Update 6/16/2016:
this seems to be useful:
1.12. Providing a C API for an Extension Module in https://docs.python.org/2/extending/extending.html says even including one c extension in another c extension can have problems if linked as share library. so, i guess I have to link the mkl as a static library? or add inlcude mkl.h to the python.h?
But then, in python (without c), I can use ctypes.cdll.LoadLibrary("./mkl_rt.so") to load the mkl's shared library and then use c function from the shared library without problem (as in here). Yet Python/C api cannot do the same thing in c?
For linking external c library staticly, the setup.py might need:
extra objects in class distutils.core.Extensions at https://docs.python.org/2/distutils/apiref.html?highlight=include#distutils.ccompiler.CCompiler.add_include_dir
Related question about cython with no answer: Combining Cython with MKL
This one seems more helpful: Python, ImportError: undefined symbol: g_utf8_skip
This one use dlopen which is deprecated: Undefined Symbol in C++ When Loading a Python Shared Library

oopcode's answer in Python, ImportError: undefined symbol: g_utf8_skip works.
The situation improved with the following.
importing the c extension into python has no error.
Calling the c extension from python give the following error:
Intel MKL FATAL ERROR: Cannot load libmkl_mc.so or libmkl_def.so.
I remember when I manually compiled numpy with mkl, the site.cfg file asked for library path and include path for the intel's mkl. Guess I need to add the library path to the extra_link_args as well... But that didn't work.
Someone with anaconda has that error too as in here.
Similar case at intel's forum here.
This stackoverflow question says extra_compile_args is also needed: How to pass flag to gcc in Python setup.py script
setup.py
from distutils.core import setup, Extension
import numpy as np
extra_link_args=["-I", "(intel's dir)/intel/compilers_and_libraries_2016.3.210/linux/mkl/include", "-L", "(intel's dir)/intel/mkl/lib/intel64/libmkl_mc.so", "-mkl"]
ext_modules = [ Extension('mkl_helper', sources = ['mkl_helper.c'], extra_link_args=extra_link_args) ]
setup(
name = 'mkl_helper',
version = '1.0',
include_dirs = [np.get_include()], #Add Include path of numpy
ext_modules = ext_modules
)
Update:
I finally got it working as in here
But mkl stillll just use only one of the 12 cpu.

Wrapping C code including Python API using SWIG and distutils fails on Mac 10.8 64bits

I have been trying to wrap an existing C code into a Python module for some time now and I keep running into recurrent errors and a failed build... After an extensive look at the available documentation and posts on this forum I would like to know if someone has an idea why it is not working.
I would like to use the stsci.imagemanip.interp2d.expand2d() function which consists of a python script calling a C function which uses the Python.h library. This under Mac 10.8 64bits with Python 2.7.
I am new to wrapping C functions to Python and I decided to use SWIG and distutils.
My input files are the following:
bilinearinterp.i
/* bilinearinterp.i */
%module bilinearinterp
%{
#define SWIG_FILE_WITH_INIT
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <stdio.h>
#include <Python.h>
#include <math.h>
#include <numpy/arrayobject.h>
#include "bilinearinterp.h"
%}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
bilinearinterp.c
/* File: bilinearinterp.c */
#include <Python.h>
#include <stdio.h>
#include <math.h>
#include <numpy/arrayobject.h>
#include "bilinearinterp.h"
static void InterpInfo (float ai, int npts, int *i, float *p, float *q) {
/* arguments:
float ai i: independent variable in same units as i
int npts i: size of array within which *i is an index
int *i o: array index close to ai
float *p, *q o: weights for linear interpolation
*/
DO SOME CALCULATIONS
}
int unbin2d (float *a, float *b, int inx, int iny, int onx, int ony) {
/* arguments:
PyArrayObject *a i: input data
PyArrayObject *b o: output data
*/
DO SOME CALCULATIONS
CALL InterpInfo
return (1);
}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args)
{
PyObject *input, *output;
PyArrayObject *dataa, *datab;
int inx, iny, onx, ony;
int status=0;
if (!PyArg_ParseTuple(args,"OO:bilinearinterp",&input,&output))
return NULL;
dataa = (PyArrayObject *)PyArray_ContiguousFromAny(input, PyArray_FLOAT, 1, 2);
datab = (PyArrayObject *)PyArray_ContiguousFromAny(output, PyArray_FLOAT, 1, 2);
inx = PyArray_DIM(input,0);
iny = PyArray_DIM(input,1);
onx = PyArray_DIM(output,0);
ony = PyArray_DIM(output,1);
status = unbin2d((float *)dataa->data,(float *)datab->data, inx, iny, onx, ony);
Py_XDECREF(dataa);
Py_XDECREF(datab);
return Py_BuildValue("i",status);
}
static PyMethodDef bilinearinterp_methods[] =
{
{"bilinearinterp", bilinearinterp, METH_VARARGS,
"bilinearinterp(input, output)"},
{0, 0} /* sentinel */
};
void initbilinearinterp(void) {
Py_InitModule("bilinearinterp", bilinearinterp_methods);
import_array();
}
bilinearinterp.h
/* File: bilinearinterp.H */
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
setup.py
#!/usr/bin/env python
from distutils.core import setup, Extension
bilinearinterp_module = Extension('_bilinearinterp',
sources = ['bilinearinterp_wrap.c','bilinearinterp.c'],
# Path to locate numpy/arrayobject.h
include_dirs=['/Library/Python/2.7/site-packages/numpy-override/numpy/core/include'])
setup (name = 'bilinearinterp',
version = '1.1',
author = "Space Telescope Science Institute - stsci_python",
description = """bilinear interpretation for 2D array extrapolation""",
ext_modules = [bilinearinterp_module],
py_modules = ["bilinearinterp"],
)
Then I run the following in Terminal:
>>>swig -python bilinearinterp.i
>>>python setup.py build_ext --inplace
The appropriate files are being created:
_bilinearinterp.so bilinearinterp.py bilinearinterp.pyc bilinearinterp_wrap.c
As well as a build directory containing some files in it.
I get warnings when setup.py is running but it seems to complete.
Over the various tests I did two recurrent errors keep coming back:
warning: "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
Which may induces several other warnings associated to numpy
warning: function 'bilinearinterp' has internal linkage but is not defined [-Wundefined-internal] static PyObject * bilinearinterp(PyObject *obj, PyObject *args); bilinearinterp_wrap.c:2971:24: note: used here result = (PyObject *)bilinearinterp(arg1,arg2);
Then when calling the module in a python script I get the following:
>>> from bilinearinterp import bilinearinterp as lininterp
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "bilinearinterp.py", line 28, in <module>
_bilinearinterp = swig_import_helper()
File "bilinearinterp.py", line 24, in swig_import_helper
_mod = imp.load_module('_bilinearinterp', fp, pathname, description)
ImportError: dlopen(./_bilinearinterp.so, 2): Symbol not found: _bilinearinterp
Referenced from: ./_bilinearinterp.so
Expected in: flat namespace
in ./_bilinearinterp.so
Would someone have an idea from where the error could be coming from?
Is it from my usage of SWIG or from setup.py?
Another approach with only a swig interface file and the C code (without including the *.h file) does not work. Based on the swig introduction chapter
bilinearinterp.i
/* bilinearinterp.i */
%module bilinearinterp
%{
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
%}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
And I compile using:
>>>swig -python -o bilinearinterp_wrap.c bilinearinterp.i
>>>gcc -c -fpic bilinearinterp.c bilinearinterp_wrap.c
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-I/System//Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include
bilinearinterp_wrap.c:2955:19: warning: function 'bilinearinterp' has internal
linkage but is not defined [-Wundefined-internal]
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
^
bilinearinterp_wrap.c:2971:24: note: used here
result = (PyObject *)bilinearinterp(arg1,arg2);
^
1 warning generated.
>>>gcc -bundle -flat_namespace -undefined suppress bilinearinterp.o bilinearinterp_wrap.o -o _bilinearinterp.so
Then I still get the same import error in the python interpreter.
Could it be due to my numpy installation or my gcc arguments?
Thanks

Well, I don't quite understand what you are trying to achieve, but I see some bad practices. For example, you make bilinearinterp everywhere static, and static functions are not exported and only can be used in the same module where implemented. I think it would be helpful to read a bit about SWIG design ideas. For sure it will take a some time, but you will get a broader perspective of what SWIG is and isn't capable of.

Import C++ dll failed in Python

Following document in "Extending and Embedding the Python Interpreter", I created a VC project, and a dll file was successfully created with name "spam_d.dll"
Main code was
static PyObject *
spam_system(PyObject *self, PyObject *args)
{
const char *command;
int sts;
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
sts = system(command);
return Py_BuildValue("i", sts);
}
static PyMethodDef SpamMethods[] = {
{"system", spam_system, METH_VARARGS, "Execute a shell command."},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC
initspam(void)
{
(void) Py_InitModule("spam", SpamMethods);
}
Then I typed following command in python:
import spam
[39003 refs]
spam.system("pwd")
/SVN/Python/PCbuild
0
[39005 refs]
It looks working correctly.
But when I rename the dll name from spam_d.pyd to spam.pyd. Python can't find the module.
>>> import spam
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named spam
[39005 refs]
From the first case, it looks python could setup relationship between "import spam" and "spam_d.pyd" correctly.
How did python know "spam" module is "spam_d.pyd", but not "spam.pyd"?
And is there any documents mention it.

The python tries to link against a debug library with suffix _d.pyd, since it's a Debug build. To link aganist spam.pyd, you need a Release build.