We Keep Coding

Debugging Python Extensions written in C

I'm pretty familiar with writing C code and I'm comfortable in writing python code. I'm trying to learn how to write modules in C that can be called from Python-3.9.X on OSX 10.15.7. I've gotten a couple 'hello world' type of examples to work, but for complex examples I'm struggling to figure out how I would debug the C-extensions that I write.
MWE:
src/add.c
// The C function that actually does the work
static PyObject * add_c_func(PyObject *self, PyObject *args)
{
int a=0;
int b=0;
int c=0;
// This determines the number arguments used by add_c_func
if (!PyArg_ParseTuple(args, "iii", &a, &b, &c))
{
return NULL;
}
printf("%i\n", a+b+c);
Py_RETURN_NONE;
}
// This defines the function used by
static PyMethodDef AddMethods[] = {
{"add_py_func", add_c_func, METH_VARARGS, "Add three numbers."},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef addpymod =
{
PyModuleDef_HEAD_INIT,
"addpymod", /* name of module */
"", /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
AddMethods
};
PyMODINIT_FUNC PyInit_addpymod(void)
{
return PyModule_Create(&addpymod);
}
setup.py :
from setuptools import setup, Extension
setup(
name='addpymod',
version='1.0',
description='Python Package with Hello World C Extension',
ext_modules=[
Extension(
'addpymod',
sources=['src/add.c'],
py_limited_api=True)
],
)
Compiling / installing (by default it uses clang):
python setup.py install
Trying to debug :
(py-ext-test) local: understand-python-c-ext $ gdb
GNU gdb (GDB) 10.1
.
.
.
(gdb) b add.c : 20
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (add.c : 20) pending.
(gdb) python
>import addpymod
>addpymod.add_py_func(10,10,10) # Why didn't my breakpoint get hit?
>Quit
# This clearly failed, I'm not even sure where my stdout is
There are multiple levels of complexity here and I'm sure that I'm being tripped by more than one.
Question :
How do I debug my add.c using gdb (preferred) or possibly lldb since it was compiled by default with clang's -g option?

Custom Python extension - Import error: undefined symbol

Learning how to write a C extension in Python. Running into the following error when executing ptest.py
Traceback (most recent call last):
File "ptest.py", line 1, in <module>
import euler_py as eul
ImportError: /usr/local/lib/python3.6/site-packages/euler_py.cpython-
36m-x86_64-linux-gnu.so: undefined symbol: problem_one
I'm assuming this is some type of linking issue.
setup.py
sources = glob.glob('ext/*.c')
euler = Extension(
'euler_py',
include_dirs=['src'],
sources=sources,
extra_compile_args=['-std=c99']
)
setup(
name='euler_py',
version='0.1',
description='Project Euler Solutions',
ext_modules=[euler]
)
ptest.py
import euler_py as eul
print(eul.problem_one(10))
The underlying functions are in /src and I wrote test.c to test purely in C. My extension is in /ext/euler_py.c shown below
ext/euler_py.c
#include <Python.h>
#include "../src/euler.h"
static char module_docstring[] = "Provides interface to Project Euler problems";
/*
* Function implementations
*/
static PyObject* euler_py_problem_one(PyObject *self, PyObject *args)
{
int max, result;
if (!PyArg_ParseTuple(args, "i", &max))
return NULL;
result = problem_one(max);
return Py_BuildValue("i", result);
}
// END function implementations
// Wire in functions to module
static PyMethodDef module_methods[] = {
{"problem_one", euler_py_problem_one, METH_VARARGS, "Solution to problem 1"},
{NULL, NULL, 0, NULL}
};
// Module definition
static struct PyModuleDef euler_py_module = {
PyModuleDef_HEAD_INIT,
"euler_py",
module_docstring,
-1,
module_methods
};
// Module initialization function
PyMODINIT_FUNC PyInit_euler_py(void)
{
return PyModule_Create(&euler_py_module);
}
Repo is here. I've played around with library_dirs & include_dirs on the Extension() initiation and no luck. Python version 3.6. Need a second look.
EDIT
Repo linked to has changed since original ask. Added linking for other 3 functions in the same manor as previous.

You forget to include all source files:
sources = glob.glob('ext/*.c') + glob.glob('src/*.c')
you could see c extension build detail with setup.py build -fv:
$ python setup.py build -fv
...
clang -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.6/ext/euler_py.o build/temp.macosx-10.13-x86_64-3.6/src/euler.o build/temp.macosx-10.13-x86_64-3.6/src/helpers.o -o build/lib.macosx-10.13-x86_64-3.6/euler_py.cpython-36m-darwin.so
now helpers.o and euler.o properly linked.

How to use opencv functions in C++ file and bind it with Python?

I want to pass an image from python script to c++ code for opencv computations. To bind the two I have followed this. The binding is working fine But when I use any opencv in-built functions it gives me error.
Traceback (most recent call last):
File "/home/prashant/Desktop/example.py", line 1, in
import pbcvt # your module, also the name of your compiled dynamic library file w/o the extension
ImportError: /usr/local/lib/python2.7/dist-packages/pbcvt.so: undefined symbol: _ZN2cv8cvtColorERKNS_11_InputArrayERKNS_12_OutputArrayEii
I am using opencv 3.1 and python 2.7.Any help/guidance is much appreciated.`
Code for reference.
Python file.
import pbcvt
import numpy as np
import cv2
a = cv2.imread("/home/prashant/Documents/opencv-practice/screenshot.png")
c = pbcvt.dot(a)
cv2.imshow("gray image",c)
cv2.waitKey(0)
cv2.destroyAllWindows()
C++ code:
#include <pyboostcvconverter/pyboostcvconverter.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
namespace pbcvt {
using namespace std;
using namespace boost::python;
cv::Mat dot(PyObject *image) {
cv::Mat matImage, greyMat;
matImage = pbcvt::fromNDArrayToMat(image);
cv::cvtColor(matImage, greyMat, CV_BGR2GRAY);
return greyMat;
}
cv::Mat dot2(cv::Mat leftMat, cv::Mat rightMat) {
auto c1 = leftMat.cols, r2 = rightMat.rows;
if (c1 != r2) {
PyErr_SetString(PyExc_TypeError,
"Incompatible sizes for matrix multiplication.");
throw_error_already_set();
}
cv::Mat result = leftMat * rightMat;
return result;
}
#if (PY_VERSION_HEX >= 0x03000000)
static void *init_ar() {
#else
static void init_ar(){
#endif
Py_Initialize();
import_array();
return NUMPY_IMPORT_ARRAY_RETVAL;
}
BOOST_PYTHON_MODULE (pbcvt) {
//using namespace XM;
init_ar();
//initialize converters
to_python_converter<cv::Mat,
pbcvt::matToNDArrayBoostConverter>();
pbcvt::matFromNDArrayBoostConverter();
//expose module-level functions
def("dot", dot);
def("dot2", dot2);
}
} //end namespace pbcvt

It could be that you have to edit the CMakeLists.txt in line 27 find_package(OpenCV COMPONENTS core REQUIRED) and add more components like find_package(OpenCV COMPONENTS core imgproc highgui REQUIRED)

Wrapping C code including Python API using SWIG and distutils fails on Mac 10.8 64bits

I have been trying to wrap an existing C code into a Python module for some time now and I keep running into recurrent errors and a failed build... After an extensive look at the available documentation and posts on this forum I would like to know if someone has an idea why it is not working.
I would like to use the stsci.imagemanip.interp2d.expand2d() function which consists of a python script calling a C function which uses the Python.h library. This under Mac 10.8 64bits with Python 2.7.
I am new to wrapping C functions to Python and I decided to use SWIG and distutils.
My input files are the following:
bilinearinterp.i
/* bilinearinterp.i */
%module bilinearinterp
%{
#define SWIG_FILE_WITH_INIT
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <stdio.h>
#include <Python.h>
#include <math.h>
#include <numpy/arrayobject.h>
#include "bilinearinterp.h"
%}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
bilinearinterp.c
/* File: bilinearinterp.c */
#include <Python.h>
#include <stdio.h>
#include <math.h>
#include <numpy/arrayobject.h>
#include "bilinearinterp.h"
static void InterpInfo (float ai, int npts, int *i, float *p, float *q) {
/* arguments:
float ai i: independent variable in same units as i
int npts i: size of array within which *i is an index
int *i o: array index close to ai
float *p, *q o: weights for linear interpolation
*/
DO SOME CALCULATIONS
}
int unbin2d (float *a, float *b, int inx, int iny, int onx, int ony) {
/* arguments:
PyArrayObject *a i: input data
PyArrayObject *b o: output data
*/
DO SOME CALCULATIONS
CALL InterpInfo
return (1);
}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args)
{
PyObject *input, *output;
PyArrayObject *dataa, *datab;
int inx, iny, onx, ony;
int status=0;
if (!PyArg_ParseTuple(args,"OO:bilinearinterp",&input,&output))
return NULL;
dataa = (PyArrayObject *)PyArray_ContiguousFromAny(input, PyArray_FLOAT, 1, 2);
datab = (PyArrayObject *)PyArray_ContiguousFromAny(output, PyArray_FLOAT, 1, 2);
inx = PyArray_DIM(input,0);
iny = PyArray_DIM(input,1);
onx = PyArray_DIM(output,0);
ony = PyArray_DIM(output,1);
status = unbin2d((float *)dataa->data,(float *)datab->data, inx, iny, onx, ony);
Py_XDECREF(dataa);
Py_XDECREF(datab);
return Py_BuildValue("i",status);
}
static PyMethodDef bilinearinterp_methods[] =
{
{"bilinearinterp", bilinearinterp, METH_VARARGS,
"bilinearinterp(input, output)"},
{0, 0} /* sentinel */
};
void initbilinearinterp(void) {
Py_InitModule("bilinearinterp", bilinearinterp_methods);
import_array();
}
bilinearinterp.h
/* File: bilinearinterp.H */
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
setup.py
#!/usr/bin/env python
from distutils.core import setup, Extension
bilinearinterp_module = Extension('_bilinearinterp',
sources = ['bilinearinterp_wrap.c','bilinearinterp.c'],
# Path to locate numpy/arrayobject.h
include_dirs=['/Library/Python/2.7/site-packages/numpy-override/numpy/core/include'])
setup (name = 'bilinearinterp',
version = '1.1',
author = "Space Telescope Science Institute - stsci_python",
description = """bilinear interpretation for 2D array extrapolation""",
ext_modules = [bilinearinterp_module],
py_modules = ["bilinearinterp"],
)
Then I run the following in Terminal:
>>>swig -python bilinearinterp.i
>>>python setup.py build_ext --inplace
The appropriate files are being created:
_bilinearinterp.so bilinearinterp.py bilinearinterp.pyc bilinearinterp_wrap.c
As well as a build directory containing some files in it.
I get warnings when setup.py is running but it seems to complete.
Over the various tests I did two recurrent errors keep coming back:
warning: "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
Which may induces several other warnings associated to numpy
warning: function 'bilinearinterp' has internal linkage but is not defined [-Wundefined-internal] static PyObject * bilinearinterp(PyObject *obj, PyObject *args); bilinearinterp_wrap.c:2971:24: note: used here result = (PyObject *)bilinearinterp(arg1,arg2);
Then when calling the module in a python script I get the following:
>>> from bilinearinterp import bilinearinterp as lininterp
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "bilinearinterp.py", line 28, in <module>
_bilinearinterp = swig_import_helper()
File "bilinearinterp.py", line 24, in swig_import_helper
_mod = imp.load_module('_bilinearinterp', fp, pathname, description)
ImportError: dlopen(./_bilinearinterp.so, 2): Symbol not found: _bilinearinterp
Referenced from: ./_bilinearinterp.so
Expected in: flat namespace
in ./_bilinearinterp.so
Would someone have an idea from where the error could be coming from?
Is it from my usage of SWIG or from setup.py?
Another approach with only a swig interface file and the C code (without including the *.h file) does not work. Based on the swig introduction chapter
bilinearinterp.i
/* bilinearinterp.i */
%module bilinearinterp
%{
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
%}
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
And I compile using:
>>>swig -python -o bilinearinterp_wrap.c bilinearinterp.i
>>>gcc -c -fpic bilinearinterp.c bilinearinterp_wrap.c
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-I/System//Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include
bilinearinterp_wrap.c:2955:19: warning: function 'bilinearinterp' has internal
linkage but is not defined [-Wundefined-internal]
static PyObject * bilinearinterp(PyObject *obj, PyObject *args);
^
bilinearinterp_wrap.c:2971:24: note: used here
result = (PyObject *)bilinearinterp(arg1,arg2);
^
1 warning generated.
>>>gcc -bundle -flat_namespace -undefined suppress bilinearinterp.o bilinearinterp_wrap.o -o _bilinearinterp.so
Then I still get the same import error in the python interpreter.
Could it be due to my numpy installation or my gcc arguments?
Thanks

Well, I don't quite understand what you are trying to achieve, but I see some bad practices. For example, you make bilinearinterp everywhere static, and static functions are not exported and only can be used in the same module where implemented. I think it would be helpful to read a bit about SWIG design ideas. For sure it will take a some time, but you will get a broader perspective of what SWIG is and isn't capable of.

Accessing functions from a third party .dll shared library through SWIG and Python

I have from a third party a DLL delivered together with his .lib and .h files (let's say the files are: "test.dll", "test.lib" and "test.h")
This delivered DLL contains some functions that I should access them from a Python script.
For this, I must build an extension (.pyd) using SWIG and MSVC2010.
(I copy the third party files into the MSVC project's directory)
To have an overview about "test.h" file, this is how it looks like (for simplicity, I put only one function, "CreateFile()", which returns a file handle):
/* File : test.h */
#if !defined ( TEST_H )
#define TEST_H
#if defined ( _MSC_VER )
#pragma warning( disable: 4103)
#pragma pack( push, 8)
#elif defined ( __BORLANDC__ )
#pragma option push -a8
#pragma nopushoptwarn
#pragma nopackwarning
#endif
#include <wtypes.h>
/*----------------------------------------------------------------------------
| BL API
-----------------------------------------------------------------------------*/
#if defined ( DLL_EXPORTS )
#define BLAPI( ret) ret __stdcall
#else
#define BLAPI( ret) __declspec( dllimport) ret __stdcall
#endif
/*----------------------------------------------------------------------------
| API
-----------------------------------------------------------------------------*/
#if defined ( __cplusplus )
extern "C" {
#endif
BLAPI( HANDLE) CreateFile( LPCTSTR lpFileName, DWORD dwDesiredAccess);
#if defined ( __cplusplus )
}
#endif
/*----------------------------------------------------------------------------
|
-----------------------------------------------------------------------------*/
#if defined ( _MSC_VER )
#pragma pack( pop)
#elif defined ( __BORLANDC__ )
#pragma option pop
#pragma nopushoptwarn
#pragma nopackwarning
#endif
#endif // TEST_H
I intend to make a class to wrap those third party functions (implemented in "test.dll" library). The "myInterface.h" file looks like this:
/* File : myInterface.h */
#include <windows.h>
#include "test.h" // <--- third party header
class myInterfaceClass {
public:
myInterfaceClass() {
}
virtual ~myInterfaceClass() {
};
HANDLE hFile;
BOOL errorCode;
BOOL OpenFile( LPCTSTR lpFileName ); // <-- function wrapper
};
...and the implementation of the class, I put it into "myInterface.cxx" file:
/* File : myInterface.cxx */
#include "myInterface.h"
#include "test.h" // <--- third party header
#include <windows.h>
BOOL myInterfaceClass::OpenFile( LPCTSTR lpFileName )
{
errorCode = TRUE;
// open file
hFile = CreateFile(lpFileName, GENERIC_READ); // <--- third party function call
errorCode = ( INVALID_HANDLE_VALUE == hFile);
return errorCode;
}
To use SWIG, I must add into the MSVC project the following SWIG interface file .i:
/* File : myInterface.i */
%module myInterface
%{
#include "myInterface.h"
#include "test.h"
%}
/* Let's just grab the original header file here */
%include "myInterface.h"
%include "test.h"
In the MSVC project, in the "Properties" of this .i file, I put on "Custom Build Tool" -> "Command Line", the following:
echo PYTHON_INCLUDE: %PYTHON_INCLUDE%
echo PYTHON_LIB: %PYTHON_LIB%
rem
rem WARNING!: Use quotes (" ") on path names to avoid errors !
rem
echo on
echo.
echo. "%(FullPath)"
echo.
"%SWIG_PATH%\swig.exe" -c++ -python -I%SWIG_PATH%\lib -Itest "%(FullPath)"
OK!
When I try to build the PYD extension, I've got this error:
Error 1 error : Syntax error in input(1). D:\ADXWorkZone\testSwig\test.h 33 1 myInterface
...but there is nothing wrong with "test.h" file. I use the same file (without any modifications) to implement the same C++ wrapper class as a classic DLL and it works well.
Project specs:
Properties -> C/C++ -> General -> Additional Include Directories: $(PYTHON_INCLUDE)
Properties -> Linker -> General -> Output File: _myInterface.pyd
Properties -> Linker -> Input -> Additional Dependencies: $(PYTHON_LIB);test.lib
Could anyone help me, please?
Any idea will be appreciated!
Thank you!

Try adding the following before your other %includes in your interface file.
%include <windows.i>
SWIG doesn't recurse into nested includes, and this provides definitions such as BOOL and _declspec that otherwise confuse SWIG.