We Keep Coding

linking C++ and Python using Boost.Python, on Remote Hosting

First, pythonanywhere is an amazing remote hosting site, and it would be great if it weren't just for python.
I've read loads of solutions, trying to get this simple example to work, in an attempt to use c++ code, using python code hosted in pythonanywhere.
c++ code
char const* greet()
{
return "hello, world";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello)
{
using namespace boost::python;
def("greet", greet);
}
In my account in pythonanywhere I did the following:
Create hello.cpp file with C++ code and upload it.
Run the following two cmds:
gcc -o middle -c -fPIC -I /usr/include/python3.8/ -L /usr/include/python3.8/ hello.cpp
gcc -o hello.so -shared middle -lboost_python -lpython3.8
The second command terminates unexpectedly showing the error:
/usr/bin/ld: cannot find -lboost_python
If I run the second command without the -lboost_python argument, it succeeds, but when I use python an error is thrown.
$ python3.8
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/Ottoman/hello.so: undefined symbol: _ZNK5boost6python7objects21py_function_impl_base9max_arityEv
I need two kinds of answers. First, what am I doing wrong and how can I solve my problem? Second, What I'm trying to do is good practice?, since my ultimate goal is to execute much more complex C++ code. If not, what alternatives do I have? Thank you!!!

Instead of -lboost_python you should use -lboost_python38, since you are using python3.8.

What wrong for I failing call Visual c++ class's method from python

My environment and Version:
Windows 10,
Microsoft Visual Studio Professional 2017:
Visual Project: 'hello2' / Debug / x86,
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:04:37) [MSC v.1929 32 bit (Intel)],
Boost 1.77.0:
Boost Builds command:
> b2 --build-type=complete address-model=32 --toolset=msvc-14.1 stage >build.log 2>&1
Remarks:
I also download and try Boost prebuild (binary) from:
https://sourceforge.net/projects/boost/files/boost-binaries/1.77.0/
but having same problem (as describe below) while I includes and link with this prebuild Boost.
The project's code copied from Boost Example/document:
https://www.boost.org/doc/libs/1_77_0/libs/python/doc/html/tutorial/tutorial/exposing.html
Steps summary:
Build Visual DLL project hello2
DLL includes:
C++ class,
Boost.Python wrapper
Copied 'hello2.dll' to hello2.pyd
python interpreter:
import hello2 (actual hello2.pyd) as a python module
(try to access the class method - but faile - see below)
…
….
Detailing:
Visual Project: 'hello2', File: hello2.cpp:
#include "pch.h"
#include <boost/python.hpp>
using namespace boost::python;
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
BOOST_PYTHON_MODULE(hello2)
{
class_<World>("World")
.def("set", &World::set)
.def("greet", &World::greet)
;
}
On Windows Commands line:
C:\Users\haimh\AppData\Local\Programs\Python\OnGointHH\SlnForTestProjects\Debug> copy hello2.dll hello2.pyd
1 file(s) copied.
Try-1 (On Windows Commands line):
C:\Users\haimh\AppData\Local\Programs\Python\OnGointHH\SlnForTestProjects\Debug> py
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:04:37) [MSC v.1929 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import hello2
>>> planet = hello2.World()
[Error message thrown in popup window: Mrcrosoft Visual C++ Runtime Library Assertion failed]
[1]: https://i.stack.imgur.com/Fc1wo.jpg
Try-2 (On Windows Commands line):
C:\Users\haimh\AppData\Local\Programs\Python\OnGointHH\SlnForTestProjects\Debug> py
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:04:37) [MSC v.1929 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import hello2
>>> planet = hello2.World
>>> planet.set('howdy')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
World.set(str)
did not match C++ signature:
set(struct World {lvalue}, class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> >)
>>>

I'm not sure why your first try (planet = hello2.World()) didn't work. My best (not-so-good) guess is a version mismatch between the python headers and the python interpreter you're using. You can check this using the python version macros.
As for the second version - this doesn't work because set is not a static method. Only static methods can be called directly on the class World. Regular methods need to be called on instances of the class. Trying to do that in pure Python will get you a similar error:
>>> class Klass:
... def f(self):
... pass
...
>>> Klass.f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f() missing 1 required positional argument: 'self'
In the C++ case, you're missing the argument struct World {lvalue} which is the equivalent of self.

I found - at least how to make settings create a working code/project...
The answer is: Change Visual C++ settings from 'Debug' to 'Release'...
Then the program run fine without any problem.
Thanks to everyone who tried to help.

How to run the python interpreter from a virtualenv using pybind11?

I am trying to execute a python interpreter from a python environment, called env_tensorflow, using pybind11 library.
Even though, I include and link the cpp file to the pybind11 library which is included from that environment (env_tensorflow), the interpreter prints its binary path as:
/usr/bin/python3
However, I expect the output to be:
/home/user/miniconda3/envs/env_tensorflow/bin/python3
What am I doing wrong? How can I run the python interpreter from a specific environment?
main.cpp
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/Python.h"
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/pybind11/pybind11.h"
#include "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/pybind11/embed.h"
namespace py = pybind11;
int main() {
py::scoped_interpreter guard{};
py::module sys = py::module::import("sys");
py::print(sys.attr("executable"));
//prints: /usr/bin/python3
//expected print: /home/user/miniconda3/envs/env_tensorflow/bin/python3
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.12)
project(my_proj)
set(CMAKE_CXX_STANDARD 11)
add_library(my_python3.6m SHARED IMPORTED)
set_target_properties(my_python3.6m PROPERTIES
IMPORTED_LOCATION "/home/user/miniconda3/envs/env_tensorflow/lib/libpython3.6m.so"
INTERFACE_INCLUDE_DIRECTORIES "/home/user/miniconda3/envs/env_tensorflow/include/python3.6m/"
)
add_executable(my_proj main.cpp)
target_link_libraries(my_proj my_python3.6m)

You need to activate the Conda environment:
source activate env_tensorflow
Once you've done that, you should be able to run cmake, make, and your application with the correct Python interpreter. You probably don't need the IMPORTED target in CMake, you can just link against python3.6m as normal, so long as the environment is active during the build process.

Importing two shared libraries which uses C++ streams into python results in corrupted output

I've been looking into a particularly nasty bug - wanted to find out from the community if this is just me being stupid (entirely possible) or is there something odd happening.
So, to replicate the problem, you need GCC 5.3 and boost 1.60.
First pyt.cpp -> which compiles to libpyt.so
/*
* This inclusion should be put at the beginning. It will include <Python.h>.
*/
#include <boost/python.hpp>
#include <string>
#include <sstream>
/*
* This is the C++ function we write and want to expose to Python.
*/
const std::string hello1(const std::string& name) {
std::ostringstream str;
str << "Hello: " << name << ", here is a number: " << 10 << std::endl;
return str.str();
}
/*
* This is a macro Boost.Python provides to signify a Python extension module.
*/
BOOST_PYTHON_MODULE(libpyt) {
// An established convention for using boost.python.
using namespace boost::python;
// Expose the function hello2().
def("hello1", hello1);
}
Second pyto.cpp -> which compiles to libpyto.so
/*
* This inclusion should be put at the beginning. It will include <Python.h>.
*/
#include <boost/python.hpp>
#include <string>
#include <sstream>
/*
* This is the C++ function we write and want to expose to Python.
*/
const std::string hello2(const std::string& name) {
std::ostringstream str;
str << "Hello: " << name << ", here is a number: " << 10 << std::endl;
return str.str();
}
/*
* This is a macro Boost.Python provides to signify a Python extension module.
*/
BOOST_PYTHON_MODULE(libpyto) {
// An established convention for using boost.python.
using namespace boost::python;
// Expose the function hello2().
def("hello2", hello2);
}
I compiled using the following:
/usr/local/gcc5_3_0/bin/g++ -std=c++14 pyt.cpp -fPIC -shared -o libpyt.so -I /usr/local/boost1_60_0_gcc5_3_0/include/ -I /usr/include/python2.7/ -L /usr/local/boost1_60_0_gcc5_3_0/lib64/ -Wl,-Bstatic -l boost_python.pic -Wl,-Bdynamic -lpthread -lpython2.7 -ldl -lrt -static-libstdc++ -static-libgcc
/usr/local/gcc5_3_0/bin/g++ -std=c++14 pyto.cpp -fPIC -shared -o libpyto.so -I /usr/local/boost1_60_0_gcc5_3_0/include/ -I /usr/include/python2.7/ -L /usr/local/boost1_60_0_gcc5_3_0/lib64/ -Wl,-Bstatic -l boost_python.pic -Wl,-Bdynamic -lpthread -lpython2.7 -ldl -lrt -static-libstdc++ -static-libgcc
(ignore the .pic extension for the boost library, it's just a static library with objects that were compiled with -fPIC - using this same compiler.)
Now, I simply import them into python, and call the hello1/2 functions:
bash-4.2$ python
Python 2.7.5 (default, Sep 15 2016, 22:37:39)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libpyt
>>> import libpyto
>>> libpyto.hello2("hello");
'Hello: hello, here is a number: 10\n'
>>> libpyt.hello1("hello");
'Hello: hello, here is a number: ' <<<!!! What???
>>>
bash-4.2$ python
Python 2.7.5 (default, Sep 15 2016, 22:37:39)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libpyto
>>> import libpyt
>>> libpyt.hello1("Hello")
'Hello: Hello, here is a number: 10\n'
>>> libpyto.hello2("Hello")
'Hello: Hello, here is a number: ' <<<!!! What???
As you can see, irrespective of the order of import, the second hello function fails to generate the output correctly. So my question is, why is the streaming out of the integer value failing for the second call?
EDIT: One more data point, enabling exceptions on the streams, results in a std::bad_cast being thrown on the second call.

You cannot mix multiple copies of Boost.Python in the same process. If you do, you get two type registries, and only one will be found and used.
Solution: use Boost.Python as a shared library, shared between your two Python module shared objects.

Okay - so the problem is simple to resolve in the end. The issue stems from -static-libstdc++ -static-libgcc. It appears that you cannot import multiple modules into python which have libstdc++ and libgcc statically linked.

Compile c++ Programm with CMake that uses Boost/python.hpp

I'm currently trying to compile a c++ file using CMake.
But since I'm using Boost::python it won't compile.
I set up a little test file to figure out what I need to do but I just can't get it to work.
Any Help would be greatly appreciated!!
The test file:
#include <Python.h>
#include <boost/python.hpp>
#include <iostream>
using std::cout;
using std::endl;
int main()
{
namespace py = boost::python;
Py_Initialize();
// Retrieve the main module's namespace
py::object global(py::import("__main__").attr("__dict__"));
py::exec("print 'Hello from Python!' \n", global, global);
return 0;
}
It will compile if I just use,
clang++ -I/usr/include/python2.7 -lpython2.7 -lboost_python -std=c++11 boosttest.cpp -o boosttest
I tried this CMakeLists.txt to get it to work.
cmake_minimum_required(VERSION 3.2)
FIND_PACKAGE(PythonLibs)
FIND_PACKAGE(Boost)
INCLUDE_DIRECTORIES(${Boost_INCLUDE_DIRS} ${PYTHON_INCLUDE_DIRS})
LINK_LIBRARIES(${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
add_executable(Test1 boosttest.cpp)
target_link_libraries(Test1 ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
and what I get is
undefined reference to `boost::python::import(boost::python::str)'
and a couple more of the same category.

Thanks for your help Mark, thanks to the new errors after I included
find_package(Boost REQUIRED python)
I was able to figure out that the problem was that CMake selected the libs for python 3.4 but Boost was build against 2.7.
So the Solution was to include the version as so:
FIND_PACKAGE(PythonLibs 2.7 REQUIRED)

Did you try
find_package(Boost REQUIRED python)
also run with verbosity to see what is going on
cmake . --debug-output
make VERBOSE=1