I'm trying to call a python function inside my c++
When i import my python module it gives this error:
ModuleNotFoundError: No module named 'test'
Here is how i'm importing the module:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int main()
{
Py_Initialize();
PyObject *pName = PyUnicode_FromString("test");
PyObject *pModule = PyImport_Import(pName);
if (pModule == nullptr)
{
PyErr_Print();
std::exit(1);
}
Py_Finalize();
return 0;
}
I feel like this must have to do with my project structure. It currently looks like this:
project
|-- main.cpp
|-- test.py
I'm not sure if it's worth mentioning, but the executable is in the same directory, and it's being run from that directory aswell.
How can i fix this?
Use Py_SetPath before Py_Initialize() to set the sys.path.
Here is a longer list of what can be done before you initialize the interpreter: pre-init-safe
Isolate the embedded Python interpreter and set the path properly to avoid problems with partially using the modules of the installed Python version.
This was solved by setting the "PYTHONPATH" environment variable:
setenv("PYTHONPATH", ".", 1);
Thanks #Botje
As dwib mentioned I was able to overcome by the error by setting the PYTHONPATH variable in my zshrc (may be bashrc for you) file.
export PYTHONPATH=$PATH:/home/jk/jk/c
where /home/jk/jk/c is the folder where my file was residing.
Also sourced it
. ~/.zshrc
Related
Following this thread
Assume I have the following files in the directory prog:
main.cpp (located in the directory prog)
int main(int argc, char *argv[]){
Py_Initialize();
//Adding current path
PyObject* sys = PyImport_ImportModule("sys");
PyObject* path = PyObject_GetAttrString(sys, "path");
PyObject* cur_dir = PyUnicode_FromString("");
PyList_Append(path, cur_dir);
PyObject* myModuleString = PyUnicode_DecodeFSDefault((char*)"multiset_bell_numbers.py");
PyObject *pModule = PyImport_Import(myModuleString);
if (pModule == nullptr) {
PyErr_Print();
throw std::runtime_error("pModule is empty");
}
return 0;
}
multiset_bell_numbers.py (located in the directory prog)
from sympy.utilities.iterables import multiset_partitions
def calc_partitions(arr):
res = list(multiset_partitions(arr))
res.sort(key=lambda part:len(part))
return res
CMakeLists.txt
cmake_minimum_required(VERSION 3.24)
project(prog)
set(CMAKE_CXX_STANDARD 17)
find_package(Python REQUIRED COMPONENTS Interpreter Development)
include_directories(${Python_INCLUDE_DIRS})
add_executable(prog main.cpp)
target_link_libraries(prog ${Python_LIBRARIES})
But still got an error:
ModuleNotFoundError: No module named 'multiset_bell_numbers'
I have also tried to remove ".py" from the suffix of the string
What should I do?
In order for Python to find the multiset_bell_numbers module, it needs to be located in a directory that is in the Python Module Search Path.
The following directories will be checked:
The current folder from which the program executes.
A list of folders specified in the PYTHONPATH environment variable, if you set it before.
An installation-dependent list of folders that you configured when you installed Python.
If you ensure that the multiset_bell_numbers is in the same directory that the program is executed from, it should import successfully.
If that is not feasible, you can try updating PYTHONPATH to include the directory containing your module.
I am trying to call a C function from Python (Extending Python with C).
I built a hello.c file and a setup.py file. When I run the setup file it's supposed to build a Python extension so that I can import it in Python.
But, when I try to run python setup.py install I am getting the error
error: [WinError 2] The system cannot find the file specified
I tried setting up the environmental variable, insert path in to sys, but still the error persists.
Can someone identify where I'm going wrong?
setup.py:
from distutils.core import setup, Extension
module1 = Extension('helloworld', sources = ['hello.c'])
setup(name='helloworld', version='1.0', ext_modules=[module1])
hello.c:
#include <Python.h>
static PyObject* helloworld(PyObject* self, PyObject* args) {
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] =
"helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld,
METH_NOARGS, helloworld_docs},
{NULL}
};
void inithelloworld(void) {
Py_InitModule3("helloworld", helloworld_funcs,
"Extension module example!");
}
error:
I am trying to execute a python interpreter from a python environment, called env_tensorflow, using pybind11 library.
Even though, I include and link the cpp file to the pybind11 library which is included from that environment (env_tensorflow), the interpreter prints its binary path as:
/usr/bin/python3
However, I expect the output to be:
/home/user/miniconda3/envs/env_tensorflow/bin/python3
What am I doing wrong? How can I run the python interpreter from a specific environment?
main.cpp
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/Python.h"
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/pybind11/pybind11.h"
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/pybind11/embed.h"
namespace py = pybind11;
int main() {
py::scoped_interpreter guard{};
py::module sys = py::module::import("sys");
py::print(sys.attr("executable"));
//prints: /usr/bin/python3
//expected print: /home/user/miniconda3/envs/env_tensorflow/bin/python3
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.12)
project(my_proj)
set(CMAKE_CXX_STANDARD 11)
add_library(my_python3.6m SHARED IMPORTED)
set_target_properties(my_python3.6m PROPERTIES
IMPORTED_LOCATION "/home/user/miniconda3/envs/env_tensorflow/lib/libpython3.6m.so"
INTERFACE_INCLUDE_DIRECTORIES "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/"
)
add_executable(my_proj main.cpp)
target_link_libraries(my_proj my_python3.6m)
You need to activate the Conda environment:
source activate env_tensorflow
Once you've done that, you should be able to run cmake, make, and your application with the correct Python interpreter. You probably don't need the IMPORTED target in CMake, you can just link against python3.6m as normal, so long as the environment is active during the build process.
I'm writing a C++ extension for python to speed up a raster image viewer created in-house.
I've got working code, but noticed that the speed hadn't increased that much and after profiling a bit deeper realised that it was due to the gdal.ReadAsArray calls, which were python callbacks from the C extension.
To get around the overhead of Python C-API when calling python objects I decided I would use the C++ libraries for gdal rather than using the Python callback to the existing gdalDataset. (space isn't a problem).
However after implementing the code for this I ran into an error at runtime(the extension compiled fine)
which was
import getRasterImage_new
ImportError: /local1/data/scratch/lib/python2.7/site-packages
/getRasterImage_new.so: undefined symbol: _ZN11GDALDataset14GetRasterYSizeEv
The code below replicates the error(some edits may be needed to run on your machines(ignore the uninitialised variables, it's just whats needed to replicate the error).
Python:
#!/usr/bin/env python
import numpy
from osgeo import gdal
import PythonCTest
print("test starting")
PythonCTest.testFunction(1)
print("test complete")
C++:
#include "Python.h"
#include "numpy/arrayobject.h"
#include "gdal_priv.h"
#include <iostream>
extern "C"{
static PyObject* PythonCTest_testFunction(PyObject* args);
static PyMethodDef PythonCTest_newMethods[] = {
{"testFunction", (PyCFunction)PythonCTest_testFunction, METH_VARARGS,
"test function"},
{NULL,NULL,0,NULL}};
PyMODINIT_FUNC initPythonCTest(void){
(void)Py_InitModule("PythonCTest",PythonCTest_newMethods);
import_array();
}
}
GDALDataset* y;
static PyObject* PythonCTest_testFunction(PyObject* args){
std::cout << "in testFunction\n";
y->GetRasterYSize();
std::cout << "doing stuff" << "\n";
return Py_None;
}
Any suggestions would be very welcome.
EDIT
You can also remove the from osgeo import gdal and the error stull occurs(only just noticed that).
EDIT 2
I forgot to say that I'm compiling my extension using distutils
current setup.py is
#!/usr/bin/env python
from distutils.core import setup, Extension
import os
os.environ["CC"] = "g++"
setup(name='PythonCTest', version='1.0', \
ext_modules=[Extension('PythonCTest', ['PythonCTest.cpp'],
extra_compile_args=['--std=c++14','-l/usr/include/gdal', '-I/usr/include/gdal'])])
A Python extension module is a dynamically loadable (shared) library. When linking a shared library, you need to specify its library dependencies, such as -lgdal and, for that matter, -lpython2.7. Failing to do so results in a library with unresolved symbols, and if those symbols are not provided by the time it is loaded, the loading will fail, as reported by Python.
To resolve the error, you need to add libraries=['gdal'] to Extension constructor. Specifying -lgdal in extra_compile_args won't work because compile arguments, as the name implies, are used for compilation and not for linking.
Note that an unresolved symbol would not go undetected when linking an executable, where the build would fail with a linker error. To get the same diagnostics when linking shared libraries, include -Wl,-zdefs in link arguments.
I created extensions for my Python and created a abcPython.dll. How can I import this dll into my Python scripts?
I get an error message when I try to import it usin the following command
>>>import abcPython
>>>Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named abcPython
>>>
I manually created a system environment variable named PYTHONPATH which stores the path to the abcPython.dll, but still the error remains.
How can I fix this?
Follow Building C and C++ Extensions on Windows carefully - in sub-section 7, it says:
The output file should be called spam.pyd (in Release mode) or spam_d.pyd (in Debug mode). The extension .pyd was chosen to avoid confusion with a system library spam.dll to which your module could be a Python interface
...
Changed in version 2.5: Previously, file names like spam.dll (in release mode) or spam_d.dll (in debug mode) were also recognized.
Try the renaming your DLL to use a .pyd extension instead of .dll.
(thanks, Peter Hansen)
The reference points to a C example, which explicitly includes an INIT function,
PyMODINIT_FUNC initexample(void). The resulting DLL should be renamed example.pyd :
#include "Python.h"
static PyObject *
ex_foo(PyObject *self, PyObject *args)
{
printf("Hello, world\n");
Py_INCREF(Py_None);
return Py_None;
}
static PyMethodDef example_methods[] = {
{"foo", ex_foo, METH_VARARGS, "foo() doc string"},
{NULL, NULL}
};
PyMODINIT_FUNC
initexample(void)
{
Py_InitModule("example", example_methods);
}
Aarrgghh! Yet another 2.X/3.X gotcha. RTFErrorMessage:
ImportError: dynamic module does not define init function (PyInit_abcPython)
Note the prefix: it's not init, it's PyInit_
See the 3.1 docs ... "The initialization function must be named PyInit_name(), where name is the name of the module"
Simple renaming of .dll to .pyd did not help. I was using SWIG to create the extension module. I created a .pyd instead of creating a .dll module and that solved the issue.
As an example: Imagine you have compiled OpenCV and have several *.dll and the cv2.pyd file.
You need to copy those files to 'DLLs' folder within the python directory.
Then import the module to check wether it is ok.