We Keep Coding

linking C++ and Python using Boost.Python, on Remote Hosting

First, pythonanywhere is an amazing remote hosting site, and it would be great if it weren't just for python.
I've read loads of solutions, trying to get this simple example to work, in an attempt to use c++ code, using python code hosted in pythonanywhere.
c++ code
char const* greet()
{
return "hello, world";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello)
{
using namespace boost::python;
def("greet", greet);
}
In my account in pythonanywhere I did the following:
Create hello.cpp file with C++ code and upload it.
Run the following two cmds:
gcc -o middle -c -fPIC -I /usr/include/python3.8/ -L /usr/include/python3.8/ hello.cpp
gcc -o hello.so -shared middle -lboost_python -lpython3.8
The second command terminates unexpectedly showing the error:
/usr/bin/ld: cannot find -lboost_python
If I run the second command without the -lboost_python argument, it succeeds, but when I use python an error is thrown.
$ python3.8
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/Ottoman/hello.so: undefined symbol: _ZNK5boost6python7objects21py_function_impl_base9max_arityEv
I need two kinds of answers. First, what am I doing wrong and how can I solve my problem? Second, What I'm trying to do is good practice?, since my ultimate goal is to execute much more complex C++ code. If not, what alternatives do I have? Thank you!!!

Instead of -lboost_python you should use -lboost_python38, since you are using python3.8.

building extension for Python 3 with Swig and distutils

I've built an application ("Checkyrs") in C++, and now am doing something external which will use large parts of Checkyrs but is built in Python 3. (This is all something in my free time, so it doesn't need to be Python 3, but that's my preference.)
To get an interface between the python and C++, I'm using SWIG and the python distutils package. I've built a dynamic library containing what I need from Checkyrs, and I've successfully built a Python extension ("checkyrsai") using the tools I mentioned. I've tested it in Python 2.7 and it works fully, all of the C++ classes and functions I need are available and function correctly.
But my preference is to work with Python 3, and while I can build the extension with Python 3 I'm unable to load it successfully:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import checkyrsai
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/chris/Documents/Programming/eggyolk/checkyrsai.py", line 28, in <module>
_checkyrsai = swig_import_helper()
File "/Users/chris/Documents/Programming/eggyolk/checkyrsai.py", line 24, in swig_import_helper
_mod = imp.load_module('_checkyrsai', fp, pathname, description)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: dlopen(/Users/chris/Documents/Programming/eggyolk/_checkyrsai.so, 2): Symbol not found: __ZN4Game11ExecuteMoveERKSt6vectorI8PositionSaIS1_EE
Referenced from: /Users/chris/Documents/Programming/eggyolk/_checkyrsai.so
Expected in: flat namespace
in /Users/chris/Documents/Programming/eggyolk/_checkyrsai.so
My process for building the extension (for Python 2) is:
swig -c++ -python checkyrsai.i
python setup.py build_ext --inplace
where my setup.py file looks like this:
from distutils.core import setup, Extension
import os
os.environ["CC"] = "g++"
checkyrsai = Extension('_checkyrsai',
sources = ['checkyrsai_wrap.cxx','../checkyrs/checkyrs/ai.cpp'],
include_dirs = ['/usr/local/include','../checkyrs/checkyrs'],
libraries = ['Checkyrs'],
library_dirs = ['../checkyrs/Build/Products/Release/','/usr/local/lib'],
extra_compile_args = ['-std=c++11']
)
setup (name = 'checkyrs',
version = '1.0',
description = 'checkyrs',
ext_modules = [checkyrsai])
As I said above, this works perfectly. From this point I can open my python (2.7) interpreter,
import checkyrsai
and off I go to play with my new toy.
When trying to build for Python 3 I use almost exactly the same process, just with the addition of Python 3 flags for SWIG and running distutils through Python 3:
swig -c++ -python -py3 checkyrsai.i
python3 setup.py build_ext --inplace
this runs through the compilation successfully and produces the extension, but when I try to
import checkyrsai
I get the ImportError […] Symbol not found issue quoted above.
I don't change my code or the setup.py script in any way between the Python 2 and Python 3 versions. The symbol refers to a method which should be found in my libCheckyrs.dylib. It evidently is available there as it is used successfully when I use the Python 2.7 extension - but it seems not to be found when I make the extension for Python 3. Does anybody have any suggestion where I'm going wrong?

I eventually solved this issue by changing XCode project settings for the C++ library I was linking against.
Specifically, changing the "C++ Language Dialect" setting to "C++ [-std=c++11]", i.e. the same version I had specified in the extra_compile_args setting for distutils. Previously it had been GNU++11, hence symbols not matching as there was a mismatch in the namespace (std::vector vs. std::__1::vector).
With that change I'm now happily and successfully able to call my C++ code from Python 3.
I don't really understand why it did work in python 2.7, since it was using all the same distutils settings, with the same C++ version specified and the linking against the same C++ lib. If anybody has an explanation for that I'd love to hear it.

Using third-party modules in pypy

The initial problem was solved, but there are more ...
__
I found this neat LTS tool online. Since it takes quite some time for training I'd like to speed things up. While searching for an approach I came across PyPy.
Pypy is now setup and seems to be working, but not with the Sequitur g2p code. When importing a module which comes from a .so file I get a No module named _sequitur_ .
I also tried installing/compiling the code with PyPy instead of Python which crashes without any useful(?) error.
I run PyPy v2.6.1 on a Ubuntu Linux 14.04 with Python 2.7.10.
Calling g2p with the default interpreter works fine.
Does the problematic .so file need to be compiled via a call from PyPy ?
Edit #1:
When trying to install g2p with PyPy instead of CPython it crashes without after this command:
error: Command "cc -O2 -fPIC -Wimplicit -DMULTIGRAM_SIZE=2 -I/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include -I/opt/pypy-2.6.1-linux64/include -c sequitur_wrap.cpp -o build/temp.linux-x86_64-2.7/sequitur_wrap.o -fpermissive" failed with exit status 1
When calling this particular command from the console, without being a part of the setup.py, there are two errors:
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:192:55: error: ‘PyArray_DescrFromType’ was not declared in this scope
PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \
&
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:194:69: error: ‘PyArray_FromAny’ was not declared in this scope
NPY_ARRAY_ENSUREARRAY, NULL)
This either didn't appear or caused an error when installing with CPhyton.
diff also shows that ndarrayobject.h under the mentioned path doesn't differ from the one in /usr/local/lib/python2.7/site-packages/numpy/andsoon .
Since I have no clue how c++ works, I'm lost there...
Edit #2:
Well, somehow it didn't work with the first include (pointing to pypy's numpy). Probably not the clean way to do it, but it helped to get rid of that problem. So I substituted the include_dirs entry in the setup.py and pointed that one to cPython's numpy include dir.
That done, the setup.py runs without any error. When I now try to import sequitur (one of the necessary files) I noticed that one .so was missing, copied it from cPython's numpy to Pypy's numpy. So far so good but now I still get this error:
$ pypy
Python 2.7.10 (f3ad1e1e1d62, Aug 28 2015, 10:45:29)
[PyPy 2.6.1 with GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import sequitur
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sequitur.py", line 32, in <module>
import sequitur_, SequenceModel, Minimization, misc
File "sequitur_.py", line 28, in <module>
_sequitur_ = swig_import_helper()
File "sequitur_.py", line 20, in swig_import_helper
import _sequitur_
ImportError: numpy.core.multiarray failed to import
Any suggestions?
Thanks

How do I compile a Fortran library for use with Python? (f2py may not be an option)

I'm trying to compile a fortran90 library (specifically this one) in order to call it from python (3.4.0). Generally in this case I would write a wrapper for f2py and call it a day, but the library itself makes use of derived types, which seems to be making f2py fail. The full stderr is pasted here, but the relevent line is
getctype: No C-type found in "{'typename': 'optim_type', 'typespec': 'type'}", assuming void.
The other option, based on the numpy documentation is to use ctypes, which also fails
Python 3.4.0 (default, Jun 19 2015, 14:20:21)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.ctypeslib.load_library('libLBFGS', '/home/kaplane/src/TOOLBOX_OPTIMIZATION_shared/lib')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/kaplane/.local/lib/python3.4/site-packages/numpy/ctypeslib.py", line 123, in load_library
return ctypes.cdll[libpath]
File "/usr/lib/python3.4/ctypes/__init__.py", line 426, in __getitem__
return getattr(self, name)
File "/usr/lib/python3.4/ctypes/__init__.py", line 421, in __getattr__
dll = self._dlltype(name)
File "/usr/lib/python3.4/ctypes/__init__.py", line 351, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /home/kaplane/src/TOOLBOX_OPTIMIZATION_shared/lib/libLBFGS.so: invalid ELF header
What I can't figure out is exactly what is invalid about the ELF header. The output from $ readelf -h is the same (excepting number, size, and locations of program and section headers) as for a shared library that works.
How I'm compiling the library
Compiling on my local machine I use gfortran instead of ifort, and have the compiler flags set as
OPTF = -O3 -shared -fPIC
OPTC = -O3 -shared -fPIC
OPTL = -O3 -shared -fPIC
AR= ar
ARFUNCT= cruvs
in the Makefile.inc file. I also run a script
find ./ -name "Makefile" | xargs sed -i -e 's/lib\([A-Z]*\)\.a/lib\1.so/g'
so that the libraries are labeled as .so instead of .a. This doesn't seem to affect the operation of the examples programs.
What I'd like to know
I think the best option is to figure out how to compile the library such that I don't get that invalid ELF error. Failing that I'd need to figure out how to compile Fortran modules with derived types, but the searching I've done is less than promising.

f2py is writen for Fortran77 Code and therefore does not support most of the features of Fortran90+, such as derived types, allocatable arrays, etc.
My own workaround incorporated writing a Fortran wrapper routine arround the subroutines I wanted to use. In this wrapper routine I copied all allocatable arrays (because that was the only unsupported feature that was used) to fixed size arrays (f2py also seems to have an undocumented maximum arrays size :/ ). These fixed size arrays, along with the size of the original arrays was then used as output of the fortran wrapper routine.
Additionally I wrote a python wrapper routine for the generated f2py library that read those fixed size arrays (read LARGE), including the size information and only returned the actual data (removing all the unused rows/ columns, etc. from the fixed size array).
This approach was only possible, because I had full control and knowledge of the source files and expected data. I would not recommend it if your work might ot be used by somebody outside of your reach.
As an alternative you should have a look at Cython. This provides an almost native way to exchange data between Fortran and Python routines using iso_c_binding [2]. For a minimum working example look here. A great talk about that can also be found in the 1st comment of this question (for reference).
I used the above workaround because I couldn't get it to work back then. But the great talks and tutorials I just mentioned have been added since, that should make it a whole lot easier.

Loading DLL from embedded python code in c

The crux of my problem is this:
I am developing code on Windows XP in C with MS Visual Studio 10.0, and I need to embed Python to do some plotting, file management, and some other things. I had problems with sys.path finding my Pure-Python modules, but I have fixed the problem by modifying PYTHONPATH.
Now, my problem is getting python to find dynamic libraries that are pulled in by some modules. In particular, my problem is to compress a folder into a bzip2 achive of the same name.
From a normal python command prompt, this works just fine:
import tarfile
tar=tarfile.open('Code.tar.bz2','w:bz2')
tar.add('Code',arcname='Code')
tar.close()
But when I call this code from my c-code, it gives me this error:
Traceback (most recent call last):
File "<string>", line 4, in <module>
File "D:\My_Documents\Code\ScrollModel\trunk\PythonCode.py", line 20, in Colle
ctFiles
tar=tarfile.open(os.path.join(runPath,'CODE.tar.bz2'),'w:bz2')
File "c:\Python26\lib\tarfile.py", line 1671, in open
return func(name, filemode, fileobj, **kwargs)
File "c:\Python26\lib\tarfile.py", line 1737, in bz2open
raise CompressionError("bz2 module is not available")
tarfile.CompressionError: bz2 module is not available
I have a suspicion the problem is similar to what is described at section 5.6 of Embedded Python, but it is a bit hard to tell. For what its worth, if I do
Py_Initialize();
PyRun_SimpleString("import ssl\n");
Py_Finalize();
it doesn't work either and I get an ImportError.
Anyone had any problems like this? Am I missing something critical?

Try this, it works on my machine.
Create a simple Windows console application in Visual Studio 2010 (remove precompiled headers option in the wizard). Replace the generated code with this one :
#include <Python.h>
int main(int argc, char *argv[]) {
Py_Initialize();
PyRun_SimpleString("import ssl \n"
"for f in dir(ssl):\n"
" print f \n" );
Py_Finalize();
return 0;
}
With PYTHONHOME set to something like c:\Python...
add C:\Python\Include to the include path
add C:\Python\Libs to the library path
add python26.lib to the linker input (adjust with your Python version)
Build. Run from anywhere and you should see a listing of the content of the ssl module.
I also tried with Mingw. Same file, build with this command line :
gcc -Wall -o test.exe embeed.c -I%PYTHONHOME%\Include -L%PYTHONHOME%\libs -lpython26

Hey, I have asked a similar question, my operation system is Linux.
When i compile c file, option $(python-config --cflags --ldflags) should be added, as
gcc test.c $(python-config --cflags --ldflags) -o test
I think in Windows you may also check python-config option, Hope this helps!

I had a similar problem with Boost C++ DLL. Any external DLL need to be in the DLL search path.
In my experience, PYTHONPATH affects Python module (the import statement in Python will end up a LoadLibrary call), and build options have nothing to do with it.
When you load a DLL, Windows doesn't care what the process is. In other words, Python follows the same DLL loading rules as Notepad. You can confirm that you are facing a Windows path problem by copying any missing DLL in the same directory as your python extension, or to a directory in your path.
To find what DLL are required by any other executable or DLL, simply open the DLL or EXE file with DependencyWalker. There is also a "Profile" menu which will allow you to run your application and watch it search and load DLLs.