I'm pretty familiar with writing C code and I'm comfortable in writing python code. I'm trying to learn how to write modules in C that can be called from Python-3.9.X on OSX 10.15.7. I've gotten a couple 'hello world' type of examples to work, but for complex examples I'm struggling to figure out how I would debug the C-extensions that I write.
MWE:
src/add.c
// The C function that actually does the work
static PyObject * add_c_func(PyObject *self, PyObject *args)
{
int a=0;
int b=0;
int c=0;
// This determines the number arguments used by add_c_func
if (!PyArg_ParseTuple(args, "iii", &a, &b, &c))
{
return NULL;
}
printf("%i\n", a+b+c);
Py_RETURN_NONE;
}
// This defines the function used by
static PyMethodDef AddMethods[] = {
{"add_py_func", add_c_func, METH_VARARGS, "Add three numbers."},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef addpymod =
{
PyModuleDef_HEAD_INIT,
"addpymod", /* name of module */
"", /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
AddMethods
};
PyMODINIT_FUNC PyInit_addpymod(void)
{
return PyModule_Create(&addpymod);
}
setup.py :
from setuptools import setup, Extension
setup(
name='addpymod',
version='1.0',
description='Python Package with Hello World C Extension',
ext_modules=[
Extension(
'addpymod',
sources=['src/add.c'],
py_limited_api=True)
],
)
Compiling / installing (by default it uses clang):
python setup.py install
Trying to debug :
(py-ext-test) local: understand-python-c-ext $ gdb
GNU gdb (GDB) 10.1
.
.
.
(gdb) b add.c : 20
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (add.c : 20) pending.
(gdb) python
>import addpymod
>addpymod.add_py_func(10,10,10) # Why didn't my breakpoint get hit?
>Quit
# This clearly failed, I'm not even sure where my stdout is
There are multiple levels of complexity here and I'm sure that I'm being tripped by more than one.
Question :
How do I debug my add.c using gdb (preferred) or possibly lldb since it was compiled by default with clang's -g option?
Related
I'm trying to embed Python in a C++ application, and I need it to run as a release build, as I am only interested in debugging the C++ code. Also, I do not have the _d debug versions of all libraries I need. I am using Python 3.7.0, in MSVC 2017 with C++11, the library I don't have debug symbols for is VTK, which I installed through a wheel file supplied by my employer. If I try to build the Python wrapper myself I get another load of issues, so I am unable to build the debug files. If I run in Debug mode, I am unable to import the library, whereas if I run in Release, I get further issues:
Exception thrown at 0x00007FF90841DBC9 (python37_d.dll) in PythonEmbedding.exe: 0xC0000005: Access violation reading location 0x0000000000000025.
The C++ code:
int main(int argc, char *argv[])
{
PyObject *pIntrospector = NULL;
if (PyVtk_InitIntrospector(pIntrospector) == OK)
{
printf("Initialization succeeded\n");
}
vtkObjectBase *pConeSource = NULL;
if (PyVtk_CreateVtkObject(pIntrospector, "vtkConeSource", pConeSource) == OK)
{
printf("Object creation succeeded\n");
}
return 0;
}
int PyVtk_InitIntrospector(
PyObject *pIntrospector)
{
/* Activating virtual environment */
#ifdef _DEBUG
// For Visual Studio debug builds
const wchar_t *sPyHome = L"venv-dbg";
#else
// For release builds
const wchar_t *sPyHome = L"venv";
#endif
Py_SetPythonHome(sPyHome);
/* Initializing Python environment and setting PYTHONPATH. */
Py_Initialize();
PyRun_SimpleString("import sys\nimport os");
PyRun_SimpleString("sys.path.append( os.path.dirname(os.getcwd()) )");
PyRun_SimpleString("sys.path.append(\".\")");
PyRun_SimpleString("import importlib.machinery as m");
PyRun_SimpleString("print(m.all_suffixes())");
/* Decode module from its name. Returns error if the name is not decodable. */
PyObject *pIntrospectorModuleName = PyUnicode_DecodeFSDefault("Introspector");
if (pIntrospectorModuleName == NULL)
{
fprintf(stderr, "Fatal error: cannot decode module name\n");
return PYTHON_INTROSPECTION_STRING_DECODE_ERROR;
}
/* Imports the module previously decoded. Returns error if the module is not found. */
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
if (pIntrospectorModule == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Failed to load \"Introspector\"\n");
Py_DECREF(pIntrospectorModuleName);
return PYTHON_INTROSPECTION_MODULE_LOAD_ERROR;
}
/* Looks for the Introspector class in the module. If it does not find it, returns and error. */
PyObject* pIntrospectorClass = PyObject_GetAttrString(pIntrospectorModule, "Introspector");
if (pIntrospectorClass == NULL || !PyCallable_Check(pIntrospectorClass))
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Cannot find class \"Introspector\"\n");
if (pIntrospectorClass != NULL)
{
Py_DECREF(pIntrospectorClass);
}
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
return PYTHON_INTROSPECTION_CLASS_NOT_FOUND_ERROR;
}
/* Instantiates an Introspector object. If the call returns NULL there was an error
creating the object, and thus it returns error. */
pIntrospector = PyObject_CallObject(pIntrospectorClass, NULL);
if (pIntrospector == NULL)
{
if (PyErr_Occurred())
{
PyErr_Print();
}
fprintf(stderr, "Introspector instantiation failed\n");
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return PYTHON_INTROSPECTION_OBJECT_CREATION_ERROR;
}
/* Decreasing reference to local data. */
Py_DECREF(pIntrospectorModuleName);
Py_DECREF(pIntrospectorModule);
Py_DECREF(pIntrospectorClass);
return OK;
}
I have not added the code to the PyVtk_CreateVtkObject function as it won't enter it, but if I do not add the calls after PyVtk_InitIntroepsctor it won't give the aforementioned error. Finally, if I import Introspector in the Python interpreter myself, it works fine.
Is there a solution to either run it in Debug or Release? I cannot wrap my head around it...
P.S.: I already tried to use Boost::Python, I have two issues open on it as it is giving me problems as well.
Update 1: In particular, the excpetion is thrown when I do this:
PyObject *pIntrospectorModule = PyImport_Import(pIntrospectorModuleName);
Update 2: I have further scoped down the issue to this: whenever I import the vtk package from within the embedded interpreter, it throws the Access Violation on this code:
// Add special attribute __vtkname__
PyObject *s = PyString_FromString(classname);
PyDict_SetItemString(pytype->tp_dict, "__vtkname__", s);
Py_DECREF(s); // <-- In particular on this Py_DECREF
If I try to import anything else, there is no issue, it seems.
Learning how to write a C extension in Python. Running into the following error when executing ptest.py
Traceback (most recent call last):
File "ptest.py", line 1, in <module>
import euler_py as eul
ImportError: /usr/local/lib/python3.6/site-packages/euler_py.cpython-
36m-x86_64-linux-gnu.so: undefined symbol: problem_one
I'm assuming this is some type of linking issue.
setup.py
sources = glob.glob('ext/*.c')
euler = Extension(
'euler_py',
include_dirs=['src'],
sources=sources,
extra_compile_args=['-std=c99']
)
setup(
name='euler_py',
version='0.1',
description='Project Euler Solutions',
ext_modules=[euler]
)
ptest.py
import euler_py as eul
print(eul.problem_one(10))
The underlying functions are in /src and I wrote test.c to test purely in C. My extension is in /ext/euler_py.c shown below
ext/euler_py.c
#include <Python.h>
#include "../src/euler.h"
static char module_docstring[] = "Provides interface to Project Euler problems";
/*
* Function implementations
*/
static PyObject* euler_py_problem_one(PyObject *self, PyObject *args)
{
int max, result;
if (!PyArg_ParseTuple(args, "i", &max))
return NULL;
result = problem_one(max);
return Py_BuildValue("i", result);
}
// END function implementations
// Wire in functions to module
static PyMethodDef module_methods[] = {
{"problem_one", euler_py_problem_one, METH_VARARGS, "Solution to problem 1"},
{NULL, NULL, 0, NULL}
};
// Module definition
static struct PyModuleDef euler_py_module = {
PyModuleDef_HEAD_INIT,
"euler_py",
module_docstring,
-1,
module_methods
};
// Module initialization function
PyMODINIT_FUNC PyInit_euler_py(void)
{
return PyModule_Create(&euler_py_module);
}
Repo is here. I've played around with library_dirs & include_dirs on the Extension() initiation and no luck. Python version 3.6. Need a second look.
EDIT
Repo linked to has changed since original ask. Added linking for other 3 functions in the same manor as previous.
You forget to include all source files:
sources = glob.glob('ext/*.c') + glob.glob('src/*.c')
you could see c extension build detail with setup.py build -fv:
$ python setup.py build -fv
...
clang -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.6/ext/euler_py.o build/temp.macosx-10.13-x86_64-3.6/src/euler.o build/temp.macosx-10.13-x86_64-3.6/src/helpers.o -o build/lib.macosx-10.13-x86_64-3.6/euler_py.cpython-36m-darwin.so
now helpers.o and euler.o properly linked.
I embedded the python 3.6 interpreter successfully in a C++ program, but I have a problem.
I'd like to embed two interpreters in the same program:
One which can use my C++ defined module (MyModule)
One which can not use this module.
Regarding the documentation I should call PyImport_AppendInittab before Py_Initialize function, so the module will be available in the whole program, but I'd like to create separate interpreters with separate built-in modules.
Calling Py_Initialize and Py_Finalize doesn't help, the module will be available in the second interpreter, too. By the way calling init and finalize function multiple times causes huge memory leaks, so I think this wouldn't be a good solution even if it would work.
Do you have any idea how to solve this issue?
Full code:
#include <iostream>
#pragma push_macro("_DEBUG")
#undef _DEBUG
#include "Python.h"
#pragma pop_macro("_DEBUG")
static PyObject* Addition (PyObject *self, PyObject *args)
{
double a = 0.0;
double b = 0.0;
if (!PyArg_ParseTuple (args, "dd", &a, &b)) {
return nullptr;
}
double result = a + b;
return PyFloat_FromDouble (result);
}
static PyMethodDef ModuleMethods[] =
{
{"Add", Addition, METH_VARARGS, "Adds numbers."},
{nullptr, nullptr, 0, nullptr}
};
static PyModuleDef ModuleDef = {
PyModuleDef_HEAD_INIT,
"MyModule",
NULL,
-1,
ModuleMethods,
NULL,
NULL,
NULL,
NULL
};
static PyObject* ModuleInitializer (void)
{
return PyModule_Create (&ModuleDef);
}
int main ()
{
Py_SetPythonHome (L".");
Py_SetPath (L"python36.zip\\Lib");
{ // first interpreter
PyImport_AppendInittab ("MyModule", ModuleInitializer);
Py_Initialize ();
PyRun_SimpleString (
"import MyModule\n"
"print (MyModule.Add (1, 2))"
);
Py_Finalize ();
}
{ // second interpreter without AppendInittab (should not find MyModule, but it does)
Py_Initialize ();
PyRun_SimpleString (
"import MyModule\n"
"print (MyModule.Add (1, 2))"
);
Py_Finalize ();
}
system ("pause");
return 0;
}
#include <Python.h>
int isCodeValid() {
char *base = calloc(512, 1);
// free(base);
// base = calloc(512,1);
base = realloc(512, 1);
free(base);
return 1;
}
static PyMethodDef CodecMethods[] = {
{ NULL, NULL, 0, NULL } };
PyMODINIT_FUNC inittest(void) {
//check for the machine code
//Py_FatalError
if (isCodeValid() != 0)
printf("nothing\n");
else {
printf("starting ... \n");
}
(void) Py_InitModule("test", CodecMethods);
}
above is a simple c extension using realloc
here is the setup.py
# coding=utf-8
from distutils.core import setup, Extension
import os
cfd = os.path.dirname(os.path.abspath(__file__))
module1 = Extension('test', sources=["test.c"])
setup(name='test', version='0.2', description='codec for test',
ext_modules=[module1],)
import test
after compile with:
python2.7 setup.py build_ext --inplace --force
I get the error :
Python(30439) malloc: *** error for object 0x200: pointer being realloc'd was not allocated
*** set a breakpoint in malloc_error_break to debug
but using
free(base);
base = calloc(512,1);
works fine without error
Anything I messed up here?
The first argument to realloc() must be a pointer, not an int literal, to a previously allocated memory (or NULL). The 512 is being cast to a pointer and the complaint is correct that the memory was not previously allocated.
To correct:
/* Don't do this:
base = realloc(base, 512);
because if realloc() fails it returns NULL
and does not free(base), resulting in memory
remaining allocated and the code having no way
to free it: a memory leak.
*/
char* tmp = realloc(base, 512);
if (tmp)
{
base = tmp;
}
Compile with warning level at maximum as the compiler will emit a warning makes pointer from integer or similar. And don't ignore the warnings, preferably treat as errors.
I am a newbie in python, so may be this is a silly question. I want to write simple c program with embedded python script. I have two files:
call-function.c:
#include <Python.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict, *pFunc, *pValue;
if (argc < 3)
{
printf("Usage: exe_name python_source function_name\n");
return 1;
}
// Initialize the Python Interpreter
Py_Initialize();
// Build the name object
if ((pName = PyString_FromString(argv[1])) == NULL) {
printf("Error: PyString_FromString\n");
return -1;
}
// Load the module object
if ((pModule = PyImport_Import(pName)) == NULL) {
printf("Error: PyImport_Import\n");
return -1;
}
// pDict is a borrowed reference
if ((pDict = PyModule_GetDict(pModule))==NULL) {
printf("Error: PyModule_GetDict\n");
return -1;
}
...
and
hello.py:
def hello():
print ("Hello, World!")
I compile and run this as follows:
gcc -g -o call-function call-function.c -I/usr/include/python2.6 -lpython2.6
./call-function hello.py hello
and have this:
Error: PyImport_Import
i.e. PyImport_Import returns NULL. Could you help me with this issue? Any help will be appreciated.
Best wishes,
Alex
I have resolved this issue by setting PYTHONPATH to pwd. Also module name (without .py) should be set for argv[1].
Thank you!
I ran into this issue also after struggling for a while.After searching the web I found that is was a system path issue. After adding the two lines after Py_Initialize(); it worked.
OS: Windows 7, Compiler: Embarcadero C++ Builder XE6, Python: Version 2.7
Reference: C++ With Python
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\"C:\\Python27\")");
If the python source file is located in the working directory (i.e. where the *.cpp files of the project reside), you can use...
PyRun_SimpleString("sys.path.append(os.getcwd())");
...to add the working directory to the Python path.
This is an obscure case but my python function was importing code that required argv to be set. In order to fix that I had to add:
PySys_SetArgv(argc, argv);
after the Py_Initialize() call and it started working.