We Keep Coding

Call a function from a cython generated .so file inside a c++ code

My goal is to call python functions from C++. These python function must be compiled with cython in a .so file. The .so file must be the one that communicate with the c++ program.
Before all:
I am on Ubuntu, I am working with miniconda with python3.9.
I am in a folder (~/exp) made like this:
exp
exp
__ init __.py
main.py
setup.py
run.cpp
run.py
I translate the main.py to main.pyx, the file contains this code:
def add(a, b):
return a+b
def entry_point():
print(add(21,21))
I compiled this script with cython and obtained a .so file, with this setup.py script:
from setuptools import setup
from Cython.Build import cythonize
setup(
name='exp',
ext_modules=cythonize("exp/main.pyx"),
libraries=[('python3.9', {'include_dirs': ["~/miniconda3/include/python3.9"]})],
library_dirs=['~/miniconda3/lib']
)
And this command:
python3 setup.py build_ext --inplace
Now I have a main.cpython-39-x86_64-linux-gnu.so file in ~/exp/exp.
When I launch (run.py) which contains:
from exp.main import entry_point
if __name__ == "__main__":
entry_point()
I have a normal behavior : It returns 42.
Now, here come the problems
I compile my run.cpp source, which contains :
#include <iostream>
#include <dlfcn.h>
#include "Python.h"
int main(int argc, char *argv[]) {
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyObject *pName, *pModule, *pDict, *pFunc, *pValue, *presult;
// Initialize the Python Interpreter
Py_Initialize();
// Build the name object
pName = PyUnicode_FromString((char*)"exp/main");
// Load the module object
pModule = PyImport_Import(pName);
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");
Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
with the command :
g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run.o -ldl
And then execute : ./run.o
To end with a beautiful :
Segmentation fault (core dumped)
I tried with dlopen without success either.
Maybe I miss something, any help would be welcome.
Thank you :)

Firstly, thank you to Craig Estey and DavidW for their comments.
So I finally was able to make it work, two things was wrong:
pValue was not used, so the Py_DECREF raised an Error
the module path "exp/main" was indeed not valid, but "exp.main" was valid.
A very last thing. Something I omitted was the PyObject_CallObject that allows to call my PyObject pFunc.
I've finally got my '42' answer.
Here the final code:
#include <iostream>
#include <dlfcn.h>
#include "Python.h"
int main(int argc, char *argv[]) {
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyObject *pName, *pModule, *pDict, *pFunc, *presult;
// Initialize the Python Interpreter
// Build the name object
pName = PyUnicode_FromString((char*)"exp.main");
// Load the module object
pModule = PyImport_Import(pName);
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");
presult = PyObject_CallObject(pFunc, NULL);
// Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
(Craig pointed out that executable file might not finish by '.o', learn more: What is *.o file)
So, the new compile command is:
g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run

Import error "undefined symbol: _ZNK9FastNoise8GetNoiseEff" when calling C++ extension in Python 3.6

I am currently trying to make some C++ extensions for a python script. In the C++ side of the story, it seems to be compiled just fine and it generates my .so shared library, but when I call it inside my python script it raises an error of undefined symbol. The current code is as follow:
#include <iostream>
#include "FastNoise.h"
#include <string>
#include <time.h>
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/highgui.hpp>
#include <boost/python/def.hpp>
#include <boost/python/module.hpp>
using namespace std;
using namespace cv;
namespace bp = boost::python;
int gen(int _size)
{
FastNoise myNoise;
myNoise.SetNoiseType(FastNoise::Simplex);
myNoise.SetSeed((int)(rand() * time(NULL)));
Size img_size(_size, _size);
Mat noise_map(img_size, CV_32FC3);
for (int y = 0; y < _size; y++) {
for (int x = 0; x < _size; x++) {
Vec3f &color = noise_map.at<Vec3f>(Point(x, y));
color.val[0] = (myNoise.GetNoise(x, y) + 1) / 2;
color.val[1] = (myNoise.GetNoise(x, y) + 1) / 2;
color.val[2] = (myNoise.GetNoise(x, y) + 1) / 2;
}
}
imshow("test", noise_map);
waitKey(0);
return 0;
}
BOOST_PYTHON_MODULE(gen) {
bp::def("gen", gen);
}
And here is how I compiled it:
g++ main.cpp -I/opt/opencv/include/opencv -I/usr/include/python3.6m -I/usr/local/include/boost -L/opt/opencv/release/lib -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/local/lib -lopencv_core -lopencv_highgui -lopencv_imgcodecs -lpython3.6m -lboost_python36 -o NoiseModule.so -shared -fPI
When I import it within python it gives me this error:
>>> import NoiseModule
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/matheus/PycharmProjects/TerrainGenerator/NoiseGenerator/NoiseModule.so: undefined symbol: _ZNK9FastNoise8GetNoiseEff
>>>
Any kind of help in regards of this problem will be really appreciated.

Your shared object doesn't have access to every function you use. You probably have a file like FastNoise.cpp which implements your FastNoise object. Yet you only use main.cpp to compile your dynamic library (.so) file. So make sure all .cpp files are included in the build of your python c++ extension.
Another option might be to make to implement your FastNoise object entirely inside of the header.

python pointer to C structure in SWIG -- accessing the struct members

I'm trying to get a very basic Python-to-C interface working with SWIG where I can pass a pointer to a structure to a C function and the members get populated, and I can then access the members in python.
In the below example, everything works except when I try to print the structure members:
print swig_test.diags_mem0_get(var)
Results in:
$ ./runpython.py
Traceback (most recent call last):
File "./runpython.py", line 11, in <module>
print swig_test.diags_mem0_get(var)
AttributeError: 'module' object has no attribute 'diags_mem0_get'
Whereas this:
print var.mem0
Results in:
$ ./runpython.py
<Swig Object of type 'uint16_t *' at 0x7f8261e15b40>swig/python detected a memory leak of type 'uint16_t *', no destructor found.
I am following the SWIG 3.0 Documentation, specifically section "5.5 Structures and unions" here: http://swig.org/Doc3.0/SWIGDocumentation.html#SWIG_nn31
What am I doing wrong?
I have distilled the example down to bare bones:
swig_test.h
typedef struct diags_t {
uint16_t mem0;
uint16_t mem1;
} diags;
diags *malloc_diags(void);
void free_diags(diags *pdiag);
int get_diags(diags *pdiags);
swig_test.c
#include <stdlib.h> // malloc()
#include <stdint.h> // uint16_t
#include "swig_test.h"
int main(int argc, char **argv) {
return 0;
}
int get_diags(diags *pdiags) {
pdiags->mem0 = 0xdead;
pdiags->mem1 = 0xbeef;
return 0;
}
diags *malloc_diags(void) {
diags *dptr = malloc(sizeof(diags));
return dptr;
}
void free_diags(diags *pdiag) {
if (pdiag != NULL)
free(pdiag);
}
swig_test.i
%module swig_test
%{
#include "swig_test.h"
%}
%include "swig_test.h"
Makefile
CXX = gcc
INCLUDES = -I./
COMPFLAGS = -c -Wall -fPIC
PYINC = /usr/include/python2.7
SWIG = /usr/bin/swig
all: swig_test _swig_test.so
swig_test: swig_test.o
$(CXX) -Wall $^ -o $#
swig_test.o: swig_test.c
$(CXX) $(COMPFLAGS) $(INCLUDES) $^
_swig_test.so: swig_test_wrap.o swig_test.o
$(CXX) -shared $^ -L$(PYLIB) -lpython2.7 -o $#
swig_test_wrap.o: swig_test_wrap.c
$(CXX) $(COMPFLAGS) $(INCLUDES) -I$(PYINC) $^
swig_test_wrap.c: swig_test.i
$(SWIG) -python $(INCLUDES) $^
And finally the simple python example:
runpython.py
#!/usr/bin/python2
import swig_test
var = swig_test.malloc_diags()
if var == 'NULL':
print "Error, no memory left"
else:
ret = swig_test.get_diags(var)
if ret == 0:
print swig_test.diags_mem0_get(var)
print var.mem0
swig_test.free_diags(var)

The functionality you're looking for comes in typemaps. The documentation freely admits that "At first glance, this code will look a little confusing." Here's what worked for me.
In essence, a typemap is a few lines of code that SWIG swaps in when it needs to convert between Python and C. They're separately defined for Python to C (%typemap(in)) and C to Python (%typemap(out)). SWIG's documentation also defines a few magic variables:
$input refers to an input object that needs to be converted to C/C++.
$result refers to an object that is going to be returned by a wrapper function.
$1 refers to a C/C++ variable that has the same type as specified in the typemap declaration (an int in this example).
For unsigned integer support, you just need in and out maps for uint8_t, uint16_t, and uint32_t
The lines below provide that functionality. They can go into SWIG's .i file, or the main header (with an ifdef SWIG guard around them).
/* uintXX_t mapping: Python -> C */
%typemap(in) uint8_t {
$1 = (uint8_t) PyInt_AsLong($input);
}
%typemap(in) uint16_t {
$1 = (uint16_t) PyInt_AsLong($input);
}
%typemap(in) uint32_t {
$1 = (uint32_t) PyInt_AsLong($input);
}
/* uintXX_t mapping: C -> Python */
%typemap(out) uint8_t {
$result = PyInt_FromLong((long) $1);
}
%typemap(out) uint16_t {
$result = PyInt_FromLong((long) $1);
}
%typemap(out) uint32_t {
$result = PyInt_FromLong((long) $1);
}
Resources I found useful:
http://www.swig.org/Doc3.0/Typemaps.html#Typemaps_nn3
https://docs.python.org/2/c-api/int.html?highlight=pyint_aslong

One "fix" is to change the uint16_t members to int types. Then this syntax works:
print var.mem0
Clearly SWIG has some problems with non-integer types.
I'd love it if somebody proposed an alternative solution that lets me keep my uint16_t types.

Error while creating a python binding

This program in C runs and compiles well :
#ifdef HAVE_CONFIG_H
#include <config.h>
#endif
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <getopt.h>
#include <atasmart.h>
int main(){
const char *device = "/dev/sda";
int ret;
uint64_t ms;
SkDisk *d;
if ((ret = sk_disk_open(device, &d)) < 0) {
printf("Failed to open disk\n");
return 1;
}
if ((ret = sk_disk_smart_read_data(d)) < 0) {
printf("Failed to read SMART data: \n");
}
if ((ret = sk_disk_smart_get_power_on(d, &ms)) < 0) {
printf("Failed to get power on time:\n");
}
printf("%llu\n", (unsigned long long) ms);
return 0;
}
using:
gcc atatest.c `pkg-config --cflags --libs libatasmart`
However while trying to create python bindings based on that program:
#ifdef HAVE_CONFIG_H
#include <config.h>
#endif
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <getopt.h>
#include <atasmart.h>
#include <Python.h>
static PyObject *pySmart_powerOn(PyObject *self, PyObject *args)
{
const char *device = "/dev/sda";
int ret;
uint64_t ms;
SkDisk *d;
if (!PyArg_ParseTuple(args, "s", &device))
{
return NULL;
}
if ((ret = sk_disk_smart_get_power_on(d, &ms)) < 0) {
return Py_BuildValue("s", "Failed to get power on time");
}
return Py_BuildValue("K", (unsigned long long) ms);
}
static PyMethodDef pySmart_methods[] = {
{ "powerOn", (PyCFunction)pySmart_powerOn, METH_VARARGS, NULL },
{ NULL, NULL, 0, NULL }
};
PyMODINIT_FUNC initpySmart()
{
Py_InitModule3("pySmart", pySmart_methods, "Trial module");
}
I create a shared library using
gcc -shared -I/usr/include/python2.7 `pkg-config --cflags --libs libatasmart` atabind.c -o pySmart.so -fPIC
then I get a warning as follows :, but the file compiles
In file included from /usr/include/python2.7/Python.h:8:0,
from atabind.c:12:
/usr/include/python2.7/pyconfig.h:1158:0: warning: "_POSIX_C_SOURCE" redefined [enabled by default]
/usr/include/features.h:214:0: note: this is the location of the previous definition
when in Python i run
import pySmart
I get
ImportError: ./pySmart.so: undefined symbol: sk_disk_smart_get_power_on
My guess is that the error is caused because I have compiled the pySmart.so shared library with incorrect flags/options.. but I'm unable to figure it out!

You need to specify linker flags (-lfoo) after your source files. That's because of the way how linker works: when you specify a library to it, it checks the library for symbols needed so far. If no symbols needed (as if you didn't get to any source objects yet), it just skips the library.
Try the following commandline:
gcc -shared -I/usr/include/python2.7 \
`pkg-config --cflags libatasmart` \
atabind.c \
`pkg-config --libs libatasmart` \
-o pySmart.so -fPIC

You should include your Python.h first then any std header.
All function, type and macro definitions needed to use the Python/C API are included in your code by
#include "Python.h"
This implies inclusion of the following standard headers:
<stdio.h>, <string.h>, <errno.h>, <limits.h>, <assert.h> and <stdlib.h> (if available).
Since Python may define some pre-processor definitions which affect the standard headers on some systems, you must include Python.h before any standard headers are included.
Alternatively:
just _GNU_SOURCE , and it will be ignored by GNU libc's /usr/include/features.h

swig, python and wchar_t problem

I am new to the Python C binding swig and have been trying to solve this problem for a while now. I have an external C library (Example.c) that I would like to call from Python. I read Swig tutorial and able to generate the wrapper in no time. The problem now is that when I invoke the API and I got this:
>>> import Example
>>> dir(Example)
['Example_CreateConnection', 'trimmed to fit the screen']
>>> Example.Example_CreateConnection("")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: in method 'Example_CreateConnection', argument 1 of type 'ExampleChar const *'
It seemed like it cannot find the type ExampleChar. The following is my swig file:
%module Example
%{
#include "ExampleSDK.h"
%}
%include "ExampleTypes.h"
%include "ExampleSDK.h"
ExampleTypes.h looks like this:
#ifndef ExampleTypes_H
#define ExampleTypes_H
typedef wchar_t ExampleChar;
#endif /* ExampleTypes_H */
ExampleSDK.h looks like this:
#ifndef ExampleSDK_H
#define ExampleSDK_H
#include "ExampleTypes.h"
void Example_CreateConnection(const ExampleChar *temp);
#endif /* ExampleSDK_H */
The following are the command lines being invoked to generate the wrapper:
swig -python -I. Example.i
gcc -c Example.c -I/Developer/SDKs/MacOSX10.6.sdk/usr/include/
gcc -c Example_wrap.c -I/usr/include/python2.6 -I.
gcc -bundle -flat_namespace -undefined suppress -o _Example.so Example_wrap.o Example.o -L/usr/lib/python2.6/config/ -lpython2.6
Here is how the Example.c looks like:
#include "runetype.h" // for Mac wchar_t definition
#include "ExampleSDK.h"
void Example_CreateConnection(const ExampleChar *temp)
{
//do nothing
}
I am not sure what is wrong with it. I hope someone will be able to point out the mistake(s) I have done over here. Thank you.
Regards,
Chuan Lim

Last time I used wchat_t with SWIG+Python I ended up needing to add something like:
%include "pywstrings.swg"
%include "pystrings.swg"
%include "std_string.i"
%include "typemaps.i"
%fragment("SWIG_AsVal_wchar_t", "header", fragment="<wchar.h>") {
SWIGINTERN int SWIG_AsVal_wchar_t(PyObject* p, wchar_t* c) {
return SWIG_OK;
}
}
%fragment("SWIG_From_wchar_t", "header", fragment="<wchar.h>") {
SWIGINTERNINLINE PyObject* SWIG_From_wchar_t(wchar_t c) {
return SWIG_Py_Void();
}
}
// Python -> C
%typemap(in) wchar_t const * {
$1 = PyString_to_wchar_t($input);
}
// C -> Python
%typemap(out) wchar_t * {
$result = wchar_t_to_PyObject($1);
}
in my Swig interface file.