We Keep Coding

Call a function from a cython generated .so file inside a c++ code

My goal is to call python functions from C++. These python function must be compiled with cython in a .so file. The .so file must be the one that communicate with the c++ program.
Before all:
I am on Ubuntu, I am working with miniconda with python3.9.
I am in a folder (~/exp) made like this:
exp
exp
__ init __.py
main.py
setup.py
run.cpp
run.py
I translate the main.py to main.pyx, the file contains this code:
def add(a, b):
return a+b
def entry_point():
print(add(21,21))
I compiled this script with cython and obtained a .so file, with this setup.py script:
from setuptools import setup
from Cython.Build import cythonize
setup(
name='exp',
ext_modules=cythonize("exp/main.pyx"),
libraries=[('python3.9', {'include_dirs': ["~/miniconda3/include/python3.9"]})],
library_dirs=['~/miniconda3/lib']
)
And this command:
python3 setup.py build_ext --inplace
Now I have a main.cpython-39-x86_64-linux-gnu.so file in ~/exp/exp.
When I launch (run.py) which contains:
from exp.main import entry_point
if __name__ == "__main__":
entry_point()
I have a normal behavior : It returns 42.
Now, here come the problems
I compile my run.cpp source, which contains :
#include <iostream>
#include <dlfcn.h>
#include "Python.h"
int main(int argc, char *argv[]) {
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyObject *pName, *pModule, *pDict, *pFunc, *pValue, *presult;
// Initialize the Python Interpreter
Py_Initialize();
// Build the name object
pName = PyUnicode_FromString((char*)"exp/main");
// Load the module object
pModule = PyImport_Import(pName);
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");
Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
with the command :
g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run.o -ldl
And then execute : ./run.o
To end with a beautiful :
Segmentation fault (core dumped)
I tried with dlopen without success either.
Maybe I miss something, any help would be welcome.
Thank you :)

Firstly, thank you to Craig Estey and DavidW for their comments.
So I finally was able to make it work, two things was wrong:
pValue was not used, so the Py_DECREF raised an Error
the module path "exp/main" was indeed not valid, but "exp.main" was valid.
A very last thing. Something I omitted was the PyObject_CallObject that allows to call my PyObject pFunc.
I've finally got my '42' answer.
Here the final code:
#include <iostream>
#include <dlfcn.h>
#include "Python.h"
int main(int argc, char *argv[]) {
setenv("PYTHONPATH",".",1);
Py_Initialize();
PyObject *pName, *pModule, *pDict, *pFunc, *presult;
// Initialize the Python Interpreter
// Build the name object
pName = PyUnicode_FromString((char*)"exp.main");
// Load the module object
pModule = PyImport_Import(pName);
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");
presult = PyObject_CallObject(pFunc, NULL);
// Py_DECREF(pValue);
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
Py_Finalize();
}
(Craig pointed out that executable file might not finish by '.o', learn more: What is *.o file)
So, the new compile command is:
g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run

embedding python in c++: python script not recognized

I am trying to embed python script into c++ project.
Below is what I have tried so far.
#include<iostream>
#include <Python.h>
int
main()
{
Py_Initialize();
PyObject* sysPath = PySys_GetObject("path");
PyObject* modPath = PyBytes_FromString("C:\\Users\\naal\\Documents\\Visual Studio 2017\\Projects\\Project1\pyscripts");
int result = PyList_Insert(sysPath,0, modPath);
PyObject *pModule = PyImport_ImportModule("myscript2");
printf("%p\n", pModule);
return 0;
}
below is the python script "myscript2.py"
def find_me():
print("hey you found me")
The problem is, the main module is not able to find the python script i.e object pyModule is always NULL, no matter how I change python script path.
What am I doing wrong ?

I ended up implementing this in another way.
#include<iostream>
#include <Python.h>
int main() {
std::string location = "C:\\Users\\myscript.py";
const char* file_location = location.c_str(); FILE* file_pointer;
Py_Initialize();
file_pointer = _Py_fopen(file_location, "r");
PyRun_SimpleFile(file_pointer, file_location);
Py_Finalize();
return 1;
}
The above seemed to work. I still don't know why the SYSPATH idea originially mentioned in the question didn't work.

C++ application crashes with embedded Python code

I'm trying to run a Python code within a C++ application, based of example from this tutorial. Everything works fine with just running a script:
char filename[] = "D:\\Docs\\Embedding\\bin\\Debug\\load_continiuous_snl_data.py";
FILE* fp;
Py_Initialize();
fp = _Py_fopen(filename, "r");
PyRun_SimpleFile(fp, filename);
Py_Finalize();
But when I try to import script as a module, application crashes
std::vector<int> ExampleData(5);
for(int i=0;i<5;i++)
{
ExampleData[i] = i+1;
}
PyObject *pName,*pModule,*pFunc,*pname;
PyObject *pValue,*pArgs,*pDict;
Py_Initialize();
pName = PyUnicode_FromString("D:\\Docs\\Embedding\\bin\\Debug\\load_continiuous_snl_data.py");
//pModule = PyImport_Import(pname); //running without this line doesn't lead to crashing
Py_Finalize();
I'm using Python 3.7 32bit on Windows. Thanks in advance for any suggestion.

PyImport_ImportModule always return NULL even though system path is set

I am trying to write a simple C/C++ program in Ubuntu 17.10 to call python function using embedding. However, it always complained that it is unable to import the module using PyImport_ImportModule.
I have included Python.h in the cpp file and compile it like this:
g++ -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/ -g -o main main.cpp -I/usr/include/python3.6 -lpython3.6
I have tried the following things but none of the approach works:
add the current path to path using PySys_GetObject and PyList_Insert (as per this link)
set PYTHONPATH=/tmp then run the main program (as per this link)
/tmp/main.cpp
#include <iostream>
#include <python3.6/Python.h>
using namespace std;
int main()
{
Py_Initialize();
PyObject* sysPath = PySys_GetObject("path");
PyObject *path = PyBytes_FromString("/tmp");
int result = PyList_Append(sysPath, path);
PyObject* module = PyImport_ImportModule("hello");
if (0 != module)
{
PyObject* helloFunc = PyObject_GetAttrString(module, "hello");
if (helloFunc)
{
PyEval_CallObject(helloFunc, 0);
}
else
{
fprintf(stderr, "hello function is NULL");
}
}
else
{
fprintf(stderr, "hello module is NULL -- somehow");
}
Py_Finalize();
return 0;
}
/tmp/hello.py
class hello(object):
def __init__(self):
print("Hello world: init")
def hello(self):
print("Hello world: hello")
Please can someone tell me why PyImport_ImportModule always return NULL in my simple program?
Many thanks!

Why do I need to explicitly add os.getcwd() to sys.path when importing Python scripts from a C application?

I was going through the Python documentation regarding Extending Python with C/C++, and I found a very interesting scenario. Consider that I have the following, trivial Python script named test_script.py in my working directory:
import time
def My_Python_Func():
print "My_Python_Func() Called: " + str(time.time())
In the same directory, I have a file called another_test_script.py, which has:
import test_script
test_script.My_Python_Func()
Which works fine, when called as python ./another_test_script.py. Now, I'm trying to call said function from a C environment linked with the Python libraries as follows:
#include <stdio.h>
#include <Python.h>
PyObject *pName, *pModule, *pDict, *pFunc, *pVal;
int main()
{
const char* script_name = "test_script";
const char* func_name = "My_Python_Func";
Py_Initialize();
pName = PyString_FromString(script_name);
PyRun_SimpleString("import os");
PyRun_SimpleString("import sys");
/* * * * IF I COMMENT THE FOLLOWING LINE OUT, pModule IS ALWAYS SET TO NULL * * * */
PyRun_SimpleString("sys.path.insert(0, os.getcwd())");
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != NULL)
{
pFunc = PyObject_GetAttrString(pModule, func_name);
}
pDict = PyModule_GetDict(pModule);
pFunc = PyDict_GetItemString(pDict, func_name);
if (PyCallable_Check(pFunc))
{
PyObject_CallObject(pFunc, NULL);
Py_DECREF(pModule);
Py_DECREF(pName);
}
else
{
PyErr_Print();
}
Py_Finalize();
return 0;
}
As noted in the comments in the above C program, commenting out the line containing PyRun_SimpleString("sys.path.insert(0, os.getcwd())") causes the call to PyImport_Import to fail (returning NULL). Furthermore, calling PyRun_SimpleString("import test_script") appears to behave the same way.
Why do I need to manually add the current working directory to Python's sys.path list of strings, when simply importing it from another Python script in the same working directory? Furthermore, why does Python not search the current working directory in the first place (as os.getcwd() returns the correct path)? Is this an appropriate solution, if I wish to import functions from scripts bundled with my C application?

Adding the script directory to sys.path is a peculiarity of the python executable; it is not done by default when the interpreter is initialized, since it is not always appropriate to do so in embedded applications.