We Keep Coding

Calling a C function with gmp library included from a Python code

I am trying to send a string from a function implemented in Python to a function implemented in C, perform some operations using GMP library and return a string to the function implemented in python. Details of code given below:
myModule.c file:
#include<Python.h>
#include<gmp.h>
char* Cnumfunc(char* n)
{
printf("This is the string %s\n",n);
mpz_t p1;
mpz_init(p1);
mpz_set_str(p1,n,10);
gmp_printf("Value of the gmp number is : %Zd \n", p1);
mpz_add_ui(p1,p1,1);//so now p1=11
char retstr[100];//create a buffer of large size
mpz_get_str(retstr,10,p1);
return retstr;//Should return "11"
}
static PyObject* numfunc(PyObject* self, PyObject* args)
{
char* n;
if (!PyArg_ParseTuple(args, "s", &n))
return NULL;
return Py_BuildValue("s", Cnumfunc(n));
}
static PyMethodDef myMethods[] = {
{"numfunc", numfunc, METH_VARARGS, "Calculate the numfunc"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef myModule = {
PyModuleDef_HEAD_INIT,
"myModule", //#name of module.
"numfunc Module",
-1,
myMethods
};
PyMODINIT_FUNC PyInit_myModule(void)
{
return PyModule_Create(&myModule);
}
Now setup.py file:
from distutils.core import setup, Extension
module = Extension('myModule', sources = ['myModule.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a package for myModule',
ext_modules = [module])
And finally the test.py file:
import myModule
a=myModule.numfunc("10")
print(a)
Now I run
python3 setup build
And then I copy the myModule.cpython-38-x86_64-linux-gnu.so file in the build directory to the directory which contains test.py and then do
python3 test.py
This gives the error:
Traceback (most recent call last):
File "test.py", line 1, in <module>
import myModule
ImportError: /home/aghosh/pybind/myModule.cpython-38-x86_64-linux-gnu.so: undefined symbol: __gmpz_set_str
I am really new to this kind of coding involving multiple languages and it will be very helpful if anyone has a solution to this problem

python import path for sub modules if put in namespace package

I have a python modules written in C, it has a main module and a submodule(name with a dot, not sure this can be called real submodule):
PyMODINIT_FUNC initsysipc(void) {
PyObject *module = Py_InitModule3("sysipc", ...);
...
init_sysipc_light();
}
static PyTypeObject FooType = { ... };
PyMODINIT_FUNC init_sysipc_light(void) {
PyObject *module = Py_InitModule3("sysipc.light", ...);
...
PyType_Ready(&FooType);
PyModule_AddObject(module, "FooType", &FooType);
}
The module is compiled as sysipc.so, and when I put it in current directory, following import works without problem:
import sysipc
import sysipc.light
from sysipc.light import FooType
The problem is I want to put this module inside a namespace package, the folder structure is like this:
company/
company/__init__.py
company/dept/
company/dept/__init__.py
company/dept/sys/
company/dept/sys/__init__.py
company/dept/sys/sysipc.so
all the three __init__.py just includes the standard setuptool import line:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
in current directory, following imports does not work:
from company.dept.sys import sysipc;
from company.dept.sys.sysipc.light import FooType;
How should I import the types and methods defined in module sysipc.light in this case?
===================================
Update with the actual error:
I have sysipc.so built, if I run python in current directory as this module, import will work as expected:
[root#08649fea17ef 2]# python2
Python 2.7.18 (default, Jul 20 2020, 00:00:00)
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysipc
>>> import sysipc.light
>>>
If however if I put it into a namespace folder, like this:
company/
company/__init__.py
company/dept
company/dept/__init__.py
company/dept/sys
company/dept/sys/sysipc.so
company/dept/sys/__init__.py
import the submodule will not work:
>>> from company.dept.sys import sysipc
>>> from company.dept.sys import sysipc.light
File "<stdin>", line 1
from company.dept.sys import sysipc.light
^
SyntaxError: invalid syntax
>>> from company.dept.sys.sysipc import light
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name light
>>>
The module is built with this simple code, it is for python2. I also have same example for python3.

Quoting from https://www.python.org/dev/peps/pep-0489/#multiple-modules-in-one-library :
To support multiple Python modules in one shared library, the library can export additional PyInit* symbols besides the one that corresponds to the library's filename.
Note that this mechanism can currently only be used to load extra modules, but not to find them. (This is a limitation of the loader mechanism, which this PEP does not try to modify.) ...
In other words, you need to restructure the project as follows for importlib to be able to find the submodule light in the sysipc package:
company/__init__.py
company/dept/__init__.py
company/dept/sys/__init__.py
company/dept/sys/sysipc/__init__.py
company/dept/sys/sysipc/sysipc.so
company/dept/sys/sysipc/light.so -> sysipc.so # hardlink
The hardlink between light.so and sysipc.so can be created with:
ln company/dept/sys/sysipc/sysipc.so company/dept/sys/sysipc/light.so
Then in company/dept/sys/sysipc/__init__.py you import all symbols from sysipc.so using:
from .sysipc import *
In addition, you need to change the name of the submodule C extension init function from init_sysipc_light to init_light for Python2, or from PyInit_sysipc_light to PyInit_light for Python3, since importlib loads modules by looking for an exported PyInit_<module name> from the dynamic module and the module name here is only light, i.e., the parent package prefix is not part of the (sub)module name.
Here is the extension code (Python3) and a couple of functions for testing:
#include <Python.h>
PyObject *sysipc_light_foo(PyObject *self, PyObject *args) {
printf("[*] sysipc.light.foo\n");
return PyLong_FromLong(0);
}
static PyMethodDef sysipc_light_methods[] = {
{"foo", (PyCFunction)sysipc_light_foo, METH_VARARGS, "sysipc.light.foo function"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef sysipc_light_module = {
PyModuleDef_HEAD_INIT,
"sysipc.light",
"sysipc child module",
-1,
sysipc_light_methods
};
PyMODINIT_FUNC PyInit_light(void)
{
PyObject *module = NULL;
module = PyModule_Create(&sysipc_light_module);
return module;
}
PyObject *sysipc_bar(PyObject *self, PyObject *args) {
printf("[*] sysipc.bar\n");
return PyLong_FromLong(0);
}
static PyMethodDef sysipc_methods[] = {
{"bar", (PyCFunction)sysipc_bar, METH_VARARGS, "sysipc.bar function"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef sysipc_module = {
PyModuleDef_HEAD_INIT,
"sysipc",
"sysipc parent module",
-1,
sysipc_methods
};
PyMODINIT_FUNC PyInit_sysipc(void)
{
PyObject *module = NULL;
module = PyModule_Create(&sysipc_module);
PyInit_light();
return module;
}
test.py:
#!/usr/bin/env python3
from company.dept.sys import sysipc
from company.dept.sys.sysipc import light
sysipc.bar()
light.foo()
Output:
[*] sysipc.bar
[*] sysipc.light.foo

There are two issues here: first, Py_InitModule and friends expect to create the module being imported. The hint is that the string you pass it is not the fully-qualified name of the module: Python uses the name it already knows to determine where in sys.modules to put the new object. You can, however, use the fully qualified name; other magic attributes like __file__ will have the correct values.
The second issue is that the attribute light needs to be set on the containing module for from imports to work.
Meanwhile, there's no reason to have a separate initialization function (that the interpreter will never call), and combining them avoids the need to recover a pointer to the module later:
static PyTypeObject FooType = { ... };
PyMODINIT_FUNC initsysipc(void) {
PyObject *module = Py_InitModule3("sysipc", ...);
...
PyObject *const sub = Py_InitModule3("company.dept.sys.sysipc.light", ...);
...
PyType_Ready(&FooType);
// PyModule_AddObject steals a reference:
Py_INCREF(FooType);
PyModule_AddObject(sub, "FooType", &FooType);
Py_INCREF(sub);
PyModule_AddObject(module, "light", sub);
}
That said, sysipc will still not be a proper package: at the least, it lacks __path__. If that matters, you might prefer MEE's answer that uses a real (if more complicated) package architecture.

Can't import C extension file in Python

My extension C function, in pyc.c
#include<Python.h>
static PyObject *add(PyObject* self,PyObject* args) {
int a=9;
int b=10;
int c=a+b;
return Py_BuildValue(c);
//return Py_BuildValue( "s","hiiiii");
}
static char prin_docs[] = "prin( ): Any message you want to put here!!\n";
static PyMethodDef consume_methods[] = {
{"add", (PyCFunction)add,
METH_NOARGS, prin_docs},
{NULL}
};
PyMODINIT_FUNC initadd(void) {
Py_InitModule3(add, consume_methods,
"Extension module example!");
}
My setup file:
from distutils.core import setup, Extension
setup(name='add', version='1.0',
ext_modules=[Extension('add', ['pyc.c'])])
My Python program that uses that C file:
import add
print(add.add());
The above code successfully created a .so object but I can't import in the Python program by executing that. I get the following error:
Traceback (most recent call last):
File "pycall.py", line 1, in <module>
import add
ImportError: No module named add

Did you actually setup.py install your module? If you only built it, it won't be on sys.path.

Custom Python extension - Import error: undefined symbol

Learning how to write a C extension in Python. Running into the following error when executing ptest.py
Traceback (most recent call last):
File "ptest.py", line 1, in <module>
import euler_py as eul
ImportError: /usr/local/lib/python3.6/site-packages/euler_py.cpython-
36m-x86_64-linux-gnu.so: undefined symbol: problem_one
I'm assuming this is some type of linking issue.
setup.py
sources = glob.glob('ext/*.c')
euler = Extension(
'euler_py',
include_dirs=['src'],
sources=sources,
extra_compile_args=['-std=c99']
)
setup(
name='euler_py',
version='0.1',
description='Project Euler Solutions',
ext_modules=[euler]
)
ptest.py
import euler_py as eul
print(eul.problem_one(10))
The underlying functions are in /src and I wrote test.c to test purely in C. My extension is in /ext/euler_py.c shown below
ext/euler_py.c
#include <Python.h>
#include "../src/euler.h"
static char module_docstring[] = "Provides interface to Project Euler problems";
/*
* Function implementations
*/
static PyObject* euler_py_problem_one(PyObject *self, PyObject *args)
{
int max, result;
if (!PyArg_ParseTuple(args, "i", &max))
return NULL;
result = problem_one(max);
return Py_BuildValue("i", result);
}
// END function implementations
// Wire in functions to module
static PyMethodDef module_methods[] = {
{"problem_one", euler_py_problem_one, METH_VARARGS, "Solution to problem 1"},
{NULL, NULL, 0, NULL}
};
// Module definition
static struct PyModuleDef euler_py_module = {
PyModuleDef_HEAD_INIT,
"euler_py",
module_docstring,
-1,
module_methods
};
// Module initialization function
PyMODINIT_FUNC PyInit_euler_py(void)
{
return PyModule_Create(&euler_py_module);
}
Repo is here. I've played around with library_dirs & include_dirs on the Extension() initiation and no luck. Python version 3.6. Need a second look.
EDIT
Repo linked to has changed since original ask. Added linking for other 3 functions in the same manor as previous.

You forget to include all source files:
sources = glob.glob('ext/*.c') + glob.glob('src/*.c')
you could see c extension build detail with setup.py build -fv:
$ python setup.py build -fv
...
clang -bundle -undefined dynamic_lookup build/temp.macosx-10.13-x86_64-3.6/ext/euler_py.o build/temp.macosx-10.13-x86_64-3.6/src/euler.o build/temp.macosx-10.13-x86_64-3.6/src/helpers.o -o build/lib.macosx-10.13-x86_64-3.6/euler_py.cpython-36m-darwin.so
now helpers.o and euler.o properly linked.

No module named ctypes using Python/C API

I have PythonCode.py:
import os, ctypes
print "ctypes are imported"
And CCode.c:
...
PySys_SetPath(path_to_PythonCode);
PyObject *pModule = PyImport_ImportModule("PythonCode");
if (!pModule) {
PyErr_Print();
return;
}
...
PyErr_Print() prints:
Traceback (most recent call last): File ".../PythonCode.py", line 1,
in <module> import os, ctypes
ImportError: No module named ctypes
Executing Python code in terminal shows that ctypes do exist:
$ python --version
Python 2.7.1
$ python PythonCode.py
ctypes are imported
Why Python/C API cannot import ctypes?

You replaced the default sys.path list with a single path. Instead you can insert the new path:
PyObject *sys_path, *path;
sys_path = PySys_GetObject("path");
if (sys_path == NULL || !PyList_Check(sys_path)) {
/* ZOMG! That's so wrong... */
return;
}
path = PyString_FromString(path_to_PythonCode);
PyList_Insert(sys_path, 0, path);
Py_DECREF(path);