We Keep Coding

Call a function from a cython generated .so file inside a c++ code

My goal is to call python functions from C++. These python function must be compiled with cython in a .so file. The .so file must be the one that communicate with the c++ program.
Before all:
I am on Ubuntu, I am working with miniconda with python3.9.
I am in a folder (~/exp) made like this:
exp
exp
__ init __.py
main.py
setup.py
run.cpp
run.py
I translate the main.py to main.pyx, the file contains this code:
def add(a, b):
return a+b
def entry_point():
print(add(21,21))
I compiled this script with cython and obtained a .so file, with this setup.py script:
from setuptools import setup
from Cython.Build import cythonize
setup(
name='exp',
ext_modules=cythonize("exp/main.pyx"),
libraries=[('python3.9', {'include_dirs': ["~/miniconda3/include/python3.9"]})],
library_dirs=['~/miniconda3/lib']
)
And this command:
python3 setup.py build_ext --inplace
Now I have a main.cpython-39-x86_64-linux-gnu.so file in ~/exp/exp.
When I launch (run.py) which contains:
from exp.main import entry_point
if __name__ == "__main__":
entry_point()
I have a normal behavior : It returns 42.
Now, here come the problems
I compile my run.cpp source, which contains :
#include <iostream>
#include <dlfcn.h>
#include "Python.h"
int main(int argc, char *argv[]) {
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyObject *pName, *pModule, *pDict, *pFunc, *pValue, *presult;
// Initialize the Python Interpreter
Py_Initialize();
// Build the name object
pName = PyUnicode_FromString((char*)"exp/main");
// Load the module object
pModule = PyImport_Import(pName);
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");
Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
with the command :
g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run.o -ldl
And then execute : ./run.o
To end with a beautiful :
Segmentation fault (core dumped)
I tried with dlopen without success either.
Maybe I miss something, any help would be welcome.
Thank you :)

Firstly, thank you to Craig Estey and DavidW for their comments.
So I finally was able to make it work, two things was wrong:
pValue was not used, so the Py_DECREF raised an Error
the module path "exp/main" was indeed not valid, but "exp.main" was valid.
A very last thing. Something I omitted was the PyObject_CallObject that allows to call my PyObject pFunc.
I've finally got my '42' answer.
Here the final code:
#include <iostream>
#include <dlfcn.h>
#include "Python.h"
int main(int argc, char *argv[]) {
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyObject *pName, *pModule, *pDict, *pFunc, *presult;
// Initialize the Python Interpreter
// Build the name object
pName = PyUnicode_FromString((char*)"exp.main");
// Load the module object
pModule = PyImport_Import(pName);
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");
presult = PyObject_CallObject(pFunc, NULL);
// Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
(Craig pointed out that executable file might not finish by '.o', learn more: What is *.o file)
So, the new compile command is:
g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run

fixing warning message related to mpd_setminalloc

I am constantly seeing
context.c:55: warning: mpd_setminalloc: ignoring request to set
MPD_MINALLOC a second time
after every call to the below function in runtime that calls a python function in c++
This function uses Python.h as explained in https://docs.python.org/2/extending/extending.html
void process_string(string text)
{
//cout<<text<<endl;
PyObject *pName, *pModule, *pDict, *pFunc;
PyObject *pArgs, *pValue;
Py_Initialize();
PyRun_SimpleString("import sys");
PyRun_SimpleString("import os");
PyRun_SimpleString("sys.path.append( os.path.dirname(os.getcwd()) )");
pName = PyUnicode_FromString("python_files.strings");
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != nullptr) {
pFunc = PyObject_GetAttrString(pModule, "process_string");
if (pFunc && PyCallable_Check(pFunc)) {
pArgs = PyTuple_New(1);
pValue = PyUnicode_FromString(text.c_str());
cout<<_PyUnicode_AsString(pValue)<<endl;
if (!pValue) {
Py_DECREF(pArgs);
Py_DECREF(pModule);
fprintf(stderr, "Cannot convert argument\n");
}
PyTuple_SetItem(pArgs, 0, pValue);
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pValue != NULL)
{
//cout<<_PyUnicode_AsString(pValue)<<endl;
Py_DECREF(pValue);
} else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr, "Call failed\n");
}
} else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find function \"%s\"\n", "process_string");
}
Py_XDECREF(pFunc);
Py_DECREF(pModule);
} else {
PyErr_Print();
fprintf(stderr, "Failed to load \"%s\"\n", "python_files.strings");
}
Py_Finalize();
}
The problem is in the c++ side of the code as even if I change the python function to simply return the input, I see the warning on the console.
The string.py file(a example) :
import os
import sys
import warnings
def process_string(text):
if not sys.warnoptions:
warnings.simplefilter("ignore")
return text
I tried disabling warning print on the python side with no advantage.

I cannot reproduce the behaviour you describe but this is very likely due to the fact that you call Py_Initialize several times.
Each time after the first one, the decimal module is initialized and calls mpd_setminalloc with minalloc_is_set == 1, hence the warning message.
void
mpd_setminalloc(mpd_ssize_t n)
{
static int minalloc_is_set = 0;
if (minalloc_is_set) {
mpd_err_warn("mpd_setminalloc: ignoring request to set "
"MPD_MINALLOC a second time\n");
return;
}
if (n < MPD_MINALLOC_MIN || n > MPD_MINALLOC_MAX) {
mpd_err_fatal("illegal value for MPD_MINALLOC"); /* GCOV_NOT_REACHED */
}
MPD_MINALLOC = n;
minalloc_is_set = 1;
}
You should have the initialization of the Python interpreter in a separate function, and call it once only. For example, if you want to put it in main:
int
main()
{
int i;
Py_Initialize();
for( i=0; i < 5; i++)
process_string("A nice string");
process_string("Another nice string");
Py_Finalize();
return 0;
}
Note you could also put the imports there, as the Python interpreter will remain alive until your program finishes.
For reference the compiling command I use is:
g++ -c -I/usr/include/python3.6m -Wno-unused-result -Wsign-compare -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -DNDEBUG -g -fwrapv -O3 -Wall <yourprogram.cpp> -lpython3.6m
And the linking command is:
g++ -g -L/usr/lib -lpython3.6m -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -o <yourexecutable> <yourobjectfile.o>

Undeclared identifier: PyUnicode_DecodeFSDefault

I try to compile the example code from Python website https://docs.python.org/3/extending/embedding.html. Everything works fine except the following line:
pName = PyUnicode_DecodeFSDefault(argv[1]);
I have Python 3.6 installed on my MacOS El Captain. My make file looks like the following:
call_function: call_function.o
gcc -o call_function call_function.o -export-dynamic -L/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin -lpython3.6m -lpthread -lm -ldl -lutil
call_function.o:call_function.cpp
gcc -c call_function.cpp -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m
When I compile the C++ code, I get the following error:
gcc -c call_function.cpp -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m
call_function.cpp:16:13: error: use of undeclared identifier 'PyUnicode_DecodeFSDefault'
pName = PyUnicode_DecodeFSDefault(argv[1]);
^
1 error generated.
Does anyone know how to fix the above error? I would greatly appreciated for your help.
Here is the complete example code:
#include <Python/Python.h>
#include <Python/unicodeobject.h>
int main(int argc, char *argv[])
{
PyObject *pName, *pModule, *pDict, *pFunc;
PyObject *pArgs, *pValue;
int i;
if (argc < 3) {
fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
return 1;
}
Py_Initialize();
pName = PyUnicode_DecodeFSDefault(argv[1]);
// pName = PyUnicode_FromString(argv[1]); <-- also gives me an error
/* Error checking of pName left out */
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != NULL) {
pFunc = PyObject_GetAttrString(pModule, argv[2]);
/* pFunc is a new reference */
if (pFunc && PyCallable_Check(pFunc)) {
pArgs = PyTuple_New(argc - 3);
for (i = 0; i < argc - 3; ++i) {
pValue = PyLong_FromLong(atoi(argv[i + 3]));
if (!pValue) {
Py_DECREF(pArgs);
Py_DECREF(pModule);
fprintf(stderr, "Cannot convert argument\n");
return 1;
}
/* pValue reference stolen here: */
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pValue != NULL) {
printf("Result of call: %ld\n", PyLong_AsLong(pValue));
Py_DECREF(pValue);
}
else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr,"Call failed\n");
return 1;
}
}
else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
}
Py_XDECREF(pFunc);
Py_DECREF(pModule);
}
else {
PyErr_Print();
fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
return 1;
}
Py_Finalize();
return 0;
}

I figure out what the problem is. The include <Python/Python.h> statement is actually linked to /System/Library/Frameworks/Python.framework/Versions/Current which currently is Python2.7. I have installed Python 3.6 on my Mac and even I include /Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m in my make file, it's still referred to the Python.h file in MacOS's Python 2.7 folder. I tried to change the symbolic link of the Current to the new 3.6 folder and it's not letting me make the changes. Does anybody know how to change the symbolic link of the Python2.7 current folder so my gcc would look into the Python 3.6 folder for Python.h instead?

It is been a long time since this question got posted, but the solution that I came up is as follows,
char * full_cls_name = argv[1];
Py_Initialize();
pName = PyString_FromString(full_cls_name);
You should also check whether argv[1] exists or not?

This is what worked for me.
gcc -o call $(python3-config --cflags --embed) multiply_embed.c $(python3-config --ldflags --embed)
I got the answer from this post - answer by Ciro Santilli Путлер.
I also got module not found error when running the code, which was overcome by setting the PYTHONPATH variable in the zshrc file as described here.

Calling a Python function from C++

I am trying to make a call to a python module function from my cpp file.
The call i have made is as follows:
#include <iostream>
#include "Python.h"
int
main(int argc, char** argv)
{
Py_Initialize();
PyObject *pName = PyString_FromString("tmpPyth");
PyObject *pModule = PyImport_Import(pName);
std::cout<< "Works fine till here";
PyObject *pDict = PyModule_GetDict(pModule);
if (pModule != NULL) {
PyObject *pFunc = PyObject_GetAttrString(pDict, "pyFunc");
if(pFunc != NULL){
PyObject_CallObject(pFunc, NULL);
}
}
else
std::cout << "Python Module not found";
return 0;
}
My python module is defined as follows:
import numpy
import scipy
import matplotlib
from scipy import stats
def blah():
baseline = [9.74219, 10.2226, 8.7469, 8.69791, 9.96442, 9.96472, 9.37913, 9.75004]
follow_up = [9.94227,9.46763,8.53081,9.43679,9.97695,10.4285,10.159,8.86134]
paired_sample = stats.ttest_rel(baseline , follow_up )
print "The t-statistic is %.3f and the p-value is %.3f." % paired_sample
The code in the cpp file runs fine till the 1st "std::cout" but then ends up giving me a "seg fault". Running the python code separately works fine and gives the desired output.
I cant figure out what is going wrong. Any help will be appreciated.
(Note the program is compiling correctly and running correctly till the 1st "cout")

So there are a couple of things that you were not doing right. See the comments inline. Assuming that both your CPP file and Python file lives at the following path: /home/shanil/project.
test.cpp:
#include <iostream>
#include "Python.h"
int
main(int argc, char** argv)
{
Py_Initialize();
// First set in path where to find your custom python module.
// You have to tell the path otherwise the next line will try to load
// your module from the path where Python's system modules/packages are
// found.
PyObject* sysPath = PySys_GetObject("path");
PyList_Append(sysPath, PyString_FromString("/home/shanil/project"));
// Load the module
PyObject *pName = PyString_FromString("my_mod");
PyObject *pModule = PyImport_Import(pName);
// Random use-less check
std::cout<< "Works fine till here\n";
if (pModule != NULL) {
std::cout << "Python module found\n";
// Load all module level attributes as a dictionary
PyObject *pDict = PyModule_GetDict(pModule);
// Remember that you are loading the module as a dictionary, the lookup you were
// doing on pDict would fail as you were trying to find something as an attribute
// which existed as a key in the dictionary
PyObject *pFunc = PyDict_GetItem(pDict, PyString_FromString("my_func"));
if(pFunc != NULL){
PyObject_CallObject(pFunc, NULL);
} else {
std::cout << "Couldn't find func\n";
}
}
else
std::cout << "Python Module not found\n";
return 0;
}
my_mod.py:
def my_func():
print 'got called'

How to solve the 'Segmentation fault' when hybrid programming of C & Python?

Under my Ubuntu:
$ cat test.py
#Filename test.py
def Hello():
print "Hello, world!"
$ cat tom.cpp
#include <Python.h>
int main()
{
Py_Initialize();
PyObject * pModule = NULL;
PyObject * pFunc = NULL;
pModule = PyImport_ImportModule("test");
pFunc = PyObject_GetAttrString(pModule, "Hello");
PyEval_CallObject(pFunc, NULL);
Py_Finalize();
return 0;
}
And then compile it:
g++ tom.cpp -I/usr/include/python2.7 -L/usr/lib/python2.7 -lpython2.7
Run:
$ ./a.out
Segmentation fault
Why?
Could anyone help?
Thanks!
BR,
Tom

The previous poster is probably right, so my comment is more "generic"...but in C/C++, you should NEVER accept a pointer back from a function without confirming it's not NULL before attempting to de-refence it. The above code should more rightly be:
pModule = PyImport_ImportModule("test");
if (pModule == NULL) {
printf("ERROR importing module");
exit(-1);
}
pFunc = PyObject_GetAttrString(pModule, "Hello");
if (pFunc == NULL) {
printf("ERROR getting Hello attribute");
exit(-1);
}
PyEval_CallObject(pFunc, NULL);

The issue is caused by PyObject_GetAttrString returning NULL. I have also added the directory path using PyRun_SimpleString as my dev dir was not under python path
#include <Python.h>
int main() {
Py_Initialize();
PyRun_SimpleString("import sys; sys.path.insert(0, 'add the directory path here')");
PyObject * pModule = NULL;
PyObject * pFunc = NULL;
pModule = PyImport_ImportModule("test");
pFunc = PyObject_GetAttrString(pModule, "Hello");
if (pFunc != NULL) {
PyEval_CallObject(pFunc, NULL);
Py_Finalize();
}
else {
printf("pFunc returned NULL\n");
}
return 0;
}