I am trying to execute a python interpreter from a python environment, called env_tensorflow, using pybind11 library.
Even though, I include and link the cpp file to the pybind11 library which is included from that environment (env_tensorflow), the interpreter prints its binary path as:
/usr/bin/python3
However, I expect the output to be:
/home/user/miniconda3/envs/env_tensorflow/bin/python3
What am I doing wrong? How can I run the python interpreter from a specific environment?
main.cpp
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/Python.h"
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/pybind11/pybind11.h"
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/pybind11/embed.h"
namespace py = pybind11;
int main() {
py::scoped_interpreter guard{};
py::module sys = py::module::import("sys");
py::print(sys.attr("executable"));
//prints: /usr/bin/python3
//expected print: /home/user/miniconda3/envs/env_tensorflow/bin/python3
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.12)
project(my_proj)
set(CMAKE_CXX_STANDARD 11)
add_library(my_python3.6m SHARED IMPORTED)
set_target_properties(my_python3.6m PROPERTIES
IMPORTED_LOCATION "/home/user/miniconda3/envs/env_tensorflow/lib/libpython3.6m.so"
INTERFACE_INCLUDE_DIRECTORIES "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/"
)
add_executable(my_proj main.cpp)
target_link_libraries(my_proj my_python3.6m)
You need to activate the Conda environment:
source activate env_tensorflow
Once you've done that, you should be able to run cmake, make, and your application with the correct Python interpreter. You probably don't need the IMPORTED target in CMake, you can just link against python3.6m as normal, so long as the environment is active during the build process.
Related
I want to use python in my android application. But I do not want to use any framework like django or flask. So to achieve this functionality I decided to use c++ as a bridge between my android application and python. Because I have already used python with c++ in my past projects. Now I was trying the same method to use python in a c++ file which is a part of my android application but it is not working this time. When I try to build my app it throws an error.
The Error Is:-
config.py:- "io.h" file not found
Now let me describe what technique I am using to connect with python:-
I am using pybind11 to make a connection between my c++ and python module for that I have created a named
"methodchannel.cpp" inside app->src->main->cpp directory
In this cpp directory I have:-
One CMakeLlists.txt file
One cpp file named="methodchannel.cpp"
A directory named = "modules"
A directory named = "externals"
In this "externals" directory at number 4 I have my resurces of python and pybind11 and In my "modules" directory I have my python module or python script which will be used in my android project.
Here is my CPP or JNI code:-
#include "jni.h"
#include "iostream"
#include "externals\python\include/Python.h"
#include "externals\pybind11\include\pybind11\embed.h"
namespace py = pybind11;
extern "C"
JNIEXPORT jint JNICALL
Java_com_example_methodchannel_MainActivity_00024Companion_getNum(JNIEnv *env, jobject thiz, jint num)
{
// TODO: implement getNum()
py::scoped_interpreter guard{};
auto module = py::module::import("modules.module");
auto func = module.attr("myfunc");
int result = func(num).cast<int>();
return 12 + result;
}
I want to use python in my android application. But I do not want to use any framework like django or flask. So to achieve this functionality I decided to use c++ as a bridge between my android application and python. Because I have already used python with c++ in my past projects. Now I was trying the same method to use python in a c++ file which is a part of my android application but it is not working this time. When I try to build my app it throws an error.
The Error Is:-
config.py:- "io.h" file not found
Now let me describe what technique I am using to connect with python:-
I am using pybind11 to make a connection between my c++ and python module for that I have created a named
"methodchannel.cpp" inside app->src->main->cpp directory
In this cpp directory I have:-
One CMakeLlists.txt file
One cpp file named="methodchannel.cpp"
A directory named = "modules"
A directory named = "externals"
In this "externals" directory at number 4 I have my resurces of python and pybind11 and In my "modules" directory I have my python module or python script which will be used in my android project.
Here is my CPP or JNI code:-
#include "jni.h"
#include "iostream"
#include "externals\python\include/Python.h"
#include "externals\pybind11\include\pybind11\embed.h"
namespace py = pybind11;
extern "C"
JNIEXPORT jint JNICALL
Java_com_example_methodchannel_MainActivity_00024Companion_getNum(JNIEnv *env, jobject thiz, jint num)
{
// TODO: implement getNum()
py::scoped_interpreter guard{};
auto module = py::module::import("modules.module");
auto func = module.attr("myfunc");
int result = func(num).cast<int>();
return 12 + result;
}
Here is an Image of my file system
I'm trying to call a python function inside my c++
When i import my python module it gives this error:
ModuleNotFoundError: No module named 'test'
Here is how i'm importing the module:
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int main()
{
Py_Initialize();
PyObject *pName = PyUnicode_FromString("test");
PyObject *pModule = PyImport_Import(pName);
if (pModule == nullptr)
{
PyErr_Print();
std::exit(1);
}
Py_Finalize();
return 0;
}
I feel like this must have to do with my project structure. It currently looks like this:
project
|-- main.cpp
|-- test.py
I'm not sure if it's worth mentioning, but the executable is in the same directory, and it's being run from that directory aswell.
How can i fix this?
Use Py_SetPath before Py_Initialize() to set the sys.path.
Here is a longer list of what can be done before you initialize the interpreter: pre-init-safe
Isolate the embedded Python interpreter and set the path properly to avoid problems with partially using the modules of the installed Python version.
This was solved by setting the "PYTHONPATH" environment variable:
setenv("PYTHONPATH", ".", 1);
Thanks #Botje
As dwib mentioned I was able to overcome by the error by setting the PYTHONPATH variable in my zshrc (may be bashrc for you) file.
export PYTHONPATH=$PATH:/home/jk/jk/c
where /home/jk/jk/c is the folder where my file was residing.
Also sourced it
. ~/.zshrc
If I build CPython from source on Windows I encounter problems when I want to pip install a package that contains a C-Extension. It seems like the error happens while linking the libraries.
For example when installing cython (but it also crashes with the same error on other C extension packages):
LINK : fatal error LNK1104: cannot open file 'python38.lib'
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.23.28105\bin\HostX86\x86\link.exe' failed with exit status 1104
The reason why it cannot open the "python38.lib" is because the ".lib" file in debug mode is called "python38_d.lib".
A minimal reproducible example would be (on the command-line) based on the Quick Reference of the CPython developer guide:
git clone --branch v3.8.0 https://github.com/python/cpython.git
cd cpython
git checkout v3.8.0
.\PCbuild\build.bat -e -d
.\PCbuild\win32\python_d.exe -m ensurepip
.\PCbuild\win32\python_d.exe -m pip install pip --upgrade -vv
.\PCbuild\win32\python_d.exe -m pip install setuptools --upgrade -vv
.\PCbuild\win32\python_d.exe -m pip install cython -vv
The resulting distutils.sysconfig.get_config_vars() is:
{'BINDIR': '...\\cpython\\PCbuild\\win32',
'BINLIBDEST': ...\\cpython\\Lib',
'EXE': '.exe',
'EXT_SUFFIX': '_d.cp38-win32.pyd',
'INCLUDEPY': '...\\cpython\\include;...\\cpython\\PC',
'LIBDEST': '...\\cpython\\Lib',
'SO': '_d.cp38-win32.pyd',
'VERSION': '38',
'exec_prefix': '...\\cpython',
'prefix': '...\\cpython',
'srcdir': '...\\cpython'}
Is there something I'm missing? Is building C-Extensions on Python-debug builds on Windows simply not supported? If it is supported: how would I do it?
This code is a little bit hacky, but works for me on MSVC19, allowing to debug application without building debug python libraries.
#ifdef _DEBUG
#define _DEBUG_WAS_DEFINED
#undef _DEBUG
#endif
#include "Python.h"
#ifdef _DEBUG_WAS_DEFINED
#define _DEBUG
#undef _DEBUG_WAS_DEFINED
#endif
Linking against pythonXY.lib is a little bit sneaky on Windows. When you look at the command line for linking, you will see that no python-library is passed to the linker, i.e. 'link.exe`. Note: This is also the case for Linux, but on Linux one doesn't have to because the needed symbols will be provided by the python-executable.
However, it is easy to check via dumpbin /dependents resulting.pyd, that there is a dependency on pythonXY.dll, also adding extra_link_args = ["/VERBOSE:LIB"] to extension-definition and triggering verbose-mode of the linker will show that the linker uses pythonXY.lib.
The sneaky part: Microsoft Compler has a convinience-pragma #pragma comment(lib, ...) to automatically trigger linking of a library, which is also used in Python-headers:
# if defined(_MSC_VER)
/* So MSVC users need not specify the .lib
file in their Makefile (other compilers are
generally taken care of by distutils.) */
# if defined(_DEBUG)
# pragma comment(lib,"python39_d.lib")
# elif defined(Py_LIMITED_API)
# pragma comment(lib,"python3.lib")
# else
# pragma comment(lib,"python39.lib")
# endif /* _DEBUG */
# endif /* _MSC_VER */
As you can see, to link against the debug version, one needs to define _DEBUG.
_DEBUG is automatically defined by distutils on Windows, if build_ext is called with options --debug, e.g.
python setup.py build_ext -i --debug
That can be translated to pip as
pip install --global-option build --global-option --debug XXXXX
which can be interpreted roughly as: trigger build command (which also includes build_ext-command) with option --debug prior to installing.
Another subtility when building debug C-extensions, there is more to it on Windows:
#ifdef _DEBUG
# define Py_DEBUG
#endif
Having defined Py_DEBUG macro meant incompartible ABIs until Python3.8, because it also assumed Py_TRACE_REFS which leads to different memory layout of PyObject and some additional functionality missing in the release-mode.
However, since Python3.8, one probably can get away with it by providing the missing pythonXY_d.lib/pythonYX.lib as a symlink linking to another version.
I'm trying to use xtensor-python example found here.
I have xtensor-python, pybind11, and xtensor installed and also created a CMakeLists.txt.
from /build I ran.
$ cmake ..
$ make
and it builds without errors.
My CMakeLists.txt looks like this.
cmake_minimum_required(VERSION 3.15)
project(P3)
find_package(xtensor-python REQUIRED)
find_package(pybind11 REQUIRED)
find_package(xtensor REQUIRED)
My example.cpp file.
#include <numeric> // Standard library import for std::accumulate
#include "pybind11/pybind11.h" // Pybind11 import to define Python bindings
#include "xtensor/xmath.hpp" // xtensor import for the C++ universal functions
#define FORCE_IMPORT_ARRAY // numpy C api loading
#include "xtensor-python/pyarray.hpp" // Numpy bindings
double sum_of_sines(xt::pyarray<double>& m)
{
auto sines = xt::sin(m); // sines does not actually hold values.
return std::accumulate(sines.cbegin(), sines.cend(), 0.0);
}
PYBIND11_MODULE(ex3, m)
{
xt::import_numpy();
m.doc() = "Test module for xtensor python bindings";
m.def("sum_of_sines", sum_of_sines, "Sum the sines of the input values");
}
My python file.
import numpy as np
import example as ext
a = np.arange(15).reshape(3, 5)
s = ext.sum_of_sines(v)
s
But my python file can't import my example.cpp file.
File "examplepyth.py", line 2, in <module>
import example as ext
ImportError: No module named 'example'
I am new to cmake. I would like to know how to set this project up properly with CMakeLists.txt
The recommended way is to build and install with a setup.py file instead of cmake. You can use the cookie-cutter to get the boilerplate generated for you.
Hey I am not sure about xtensor-python since I have not used it, but I might give you some pointers for building pybind11 with cmake within an Anaconda environment. Your Cmake.txt looks a bit incomplete. For me the following set-up works:
In my Anaconda-shell I use the following commands:
cmake -S <folder where Cmake.txt is> B <folder where Cmake.txt is\build> -G"Visual Studio 15 2017 Win64"
which puts all the linking into the subfolder build, so the actual building can be done via
cmake --build build
The necessary Cmake.txt looks something like below. The created library TEST is then located in the subfolder debug\Build
#minimum version of cmake
cmake_minimum_required(VERSION 2.8.12)
#setup project
project(TEST)
#load the libraries
find_package(pybind11 REQUIRED)
set(EXTERNAL_LIBRARIES_ROOT_PATH <Filepath where my external libraries are at>)
set(EIGEN3_INCLUDE_DIR ${EXTERNAL_LIBRARIES_ROOT_PATH}/eigen-eigen-c753b80c5aa6)
#get all the files in the folder
file(GLOB SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/*.h
${CMAKE_CURRENT_SOURCE_DIR}/*.cpp
)
#include the directories
include_directories(${PYTHON_INCLUDE_DIRS} ${pybind11_INCLUDE_DIRS} ${EIGEN3_INCLUDE_DIR})
pybind11_add_module(TEST MODULE ${SOURCES})
#in some cases need to link the libraries
#target_link_libraries(TEST PUBLIC ${OpenCV_LIBRARIES} ${Boost_LIBRARIES})
If you want a working minimal example for which I use exactly this Cmake.txt file, it happens to be in another question I posted here on stackoverflow: pybind11 how to use custom type caster for simple example class
Hope this helps as a first starting point (I left the EIGEN3 inside as to give you an idea how it is done with a header-only library. For actual libraries like OpenCV you need the target_link_libraries command in addition).