We Keep Coding

Pybind11 - ImportError: .../pybindx.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E

I wrote some binding code to bind C++ code with python in pybindx.cpp file. I want to call some functions (implemented in C++) using python. When I use python setup.py build_ext command, the .so file ./build/lib.linux-x86_64-3.8/pybindx.cpython-38-x86_64-linux-gnu.so is getting created, but when I try to import(import pybindx) in test.py to call binded functions, It gives the following error:
ImportError: <path-to-.so-file>/pybindx.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E
I have added <path-to-.so-file> to PYTHONPATH and LD_LIBRARY_PATH.
My setup.py file contains following code:
import os, sys
from distutils.core import setup, Extension
from distutils import sysconfig
cpp_args = ['-std=c++11']
ext_modules = [
Extension(
'pybindx',
['class1.cpp', 'class2.cpp', 'base_class1.cpp', 'base_class2.cpp', 'pybindx.cpp'],
include_dirs=['paths/to/include/header/files', 'path/to/protobuf/include'],
language='c++',
extra_compile_args = cpp_args,
),
]
setup(
name='pybindx',
version='0.0.1',
author='xxxxx',
author_email='xxxxx',
description='desc',
ext_modules=ext_modules,
)
Where, class1.cpp, class2.cpp, base_class1.cpp, base_class2.cpp are the files having implementation of classes and functions which I want to bind with python.
I am new to pybind11, can someone help me with this?
Thanks!
I tried writing small example code without protobuf, where I am able to call the C++ function using test.py, but here I want to use protobuf.

Anyways, fixed it. I Used Makefile rather than setup.py. I have to link static libprotobuf.a with my pybindx.so.
Added following command with other required commands in Makefile:
g++ -shared -fPIC ./build/*.o <path-to-static-protobuf>/libprotobuf.a -o pybindx.so
Where, *.o are the object files created using some command(s) in Makefile.

Compiling cython with gcc: No such file or directory from #include "ios"

Given a file docprep.pyx as simple as
from spacy.structs cimport TokenC
print("loading")
And trying to cythonize it via
cythonize -3 -i docprep.pyx
I get the following error message
docprep.c:613:10: fatal error: ios: No such file or directory
#include "ios"
^~~~~
compilation terminated
As you can tell from the paths, this system has an Anaconda installation with Python 3.7. numpy, spacy and cython are all installed through conda.

In my case, it worked using #mountrix tip, just add the language="c++" to your setup.py, an example:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy
extensions = [
Extension("processing_module", sources=["processing_module.pyx"], include_dirs=[numpy.get_include()], extra_compile_args=["-O3"], language="c++")
]
setup(
name="processing_module",
ext_modules = cythonize(extensions),
)

<ios> is a c++-header. The error message shows that you try to compile a C++-code as C-code.
Per default, Cython will produce a file with extension *.c, which will be interpreted as C-code by the compiler later on.
Cython can also produce a file with the right file-extension for c++, i.e. *.cpp. And there are multiple ways to trigger this behavior:
adding # distutils: language = c++ at the beginning of the pyx-file.
adding language="c++" to the Extension definition in the setup.py-file.
call cython with option --cplus.
in IPython, calling %%cython magic with -+, i.e. %%cython -+.
for alternatives when building with pyximport, see this SO-question.
Actually, for cythonize there is no command line option to trigger c++-generation, thus the first options looks like the best way to go:
# distutils: language = c++
from spacy.structs cimport TokenC
print("loading")
The problem is that spacy/structs.pxd uses c++-constructs, for example vectors or anything else cimported from libcpp:
...
from libcpp.vector cimport vector
...
and thus also c++-libraries/headers are needed for the build.

Run compiled python script with arguments in C++ (Cython)

What I try to achieve is compiling my Python script to lib/dll and invoke it with arguments.
This is my setup.py file for the script:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize("nu3k2nu4k.py")
)
Used this command to produce compilation outputs:
python setup.py build_ext --inplace
following files produced:
nu3k2nu4k.c
nu3k2nu4k.cp36-win_amd64.pyd
nu3k2nu4k.pyx
and a subdirectory named build which contains:
nu3k2nu4k.cp36-win_amd64.exp
nu3k2nu4k.cp36-win_amd64.lib
nu3k2nu4k.obj
How can I invoke the compiled script from c++ code with arguments?
I used Cython for the task but that's not mandatory, boost or others could be used as well (I'm doing this to make the source code not accessible).
Edit:
Using the documentation provided by Mykola Shchetinin I have managed to generate an API header file for my script. I basically added a cdef api keyword to my main function. While this does help me export the method name to c++ I still need a way to pass around some arguments to my script. Since I pass string arguments I was hoping for a similar way like using PyRun to set argv and parse the arguments from the script (since I would like to avoid conversion of strings from c to python encoding if possible) Is there any easy way to pass those string arguments?
EDIT 2:
I got it working! following the sample provided by Mykola and directly compiling the c file output instead of using the .lib I got it working.

I have tried to do that by myself. So I convert char * to bytes in cython. (according to docs)
That is what I came up with. I have the following setup (gcc and centos7.2):
setup.py
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize("func.pyx")
)
main.c
#include "Python.h"
#include "func.h"
int main() {
Py_Initialize();
initfunc();
func("0123hello jorghy!!");
Py_Finalize();
return 0;
}
func.pyx
cdef public func(char * arg):
cdef bytes py_bytes = arg
print(str(py_bytes));
I build all this stuff with the following commands:
python setup.py build_ext --inplace
gcc -I/usr/include/python2.7 -lpython2.7 -Wall -Wextra -O2 -g -o test main.c func.c
Then when running the executable file test I get the following results:
0123hello jorghy!!
UPDATE
There is another way to approach this problem:
I have created a cython file test.pyx:
import sys
if __name__ == "__main__":
print("hei")
print(sys.argv)
And compiled as an executable it using commands:
cython test.pyx --embed
gcc -I/usr/include/python2.7 -lpython2.7 -Wall -Wextra -O2 -g -o test test.c
Then you can call this code as an executable file:
$ ./test 1 2 3
hei
['./test', '1', '2', '3']
Then you can call it from C++ using std::system or other (better) methods.

Cython and fortran - how to compile together without f2py

FINAL UPDATE
This question is about how to write a setup.py that will compile a cython module which accesses FORTRAN code directly, like C would. It was a rather long and arduous journey to the solution, but the full mess is included below for context.
ORIGINAL QUESTION
I have an extension which is a Cython file, which sets up some heap memory and passes it to the fortran code, and a fortran file, which is a venerable old module that I'd like to avoid reimplementing if I can.
The .pyx file compiles fine to C, but the cython compiler chokes on the .f90 file with the following error:
$ python setup.py build_ext --inplace
running build_ext
cythoning delaunay/__init__.pyx to delaunay/__init__.c
building 'delaunay' extension
error: unknown file type '.f90' (from 'delaunay/stripack.f90')
Here's (the top half of) my setup file:
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension("delaunay",
sources=["delaunay/__init__.pyx",
"delaunay/stripack.f90"])
]
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
...
)
NOTE: I originally had the fortran file's location incorrectly specified (without the directory prefix) but this breaks in exactly the same way after I fixed that.
Things I have tried:
I found this, and tried passing in the name of the fortran compiler (i.e. gfortran) like this:
$ python setup.py config --fcompiler=gfortran build_ext --inplace
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --fcompiler not recognized
And I've also tried removing --inplace, in case that was the problem (it wasn't, same as the top error message).
So, how do I compile this fortran? Can I hack it into a .o myself and get away with linking it? Or is this a bug in Cython, which will force me to reimplement distutils or hack around with the preprocessor?
UPDATE
So, having checked out the numpy.distutils packages, I understand the problem a bit more. It seems that you have to
Use cython to convert the .pyx files to cpython .c files,
Then use an Extension/setup() combination that supports fortran, like numpy's.
Having tried this, my setup.py now looks like this:
from numpy.distutils.core import setup
from Cython.Build import cythonize
from numpy.distutils.extension import Extension
cy_modules = cythonize('delaunay/sphere.pyx')
e = cy_modules[0]
ext_modules = [
Extension("delaunay.sphere",
sources=e.sources + ['delaunay/stripack.f90'])
]
setup(
ext_modules = ext_modules,
name="delaunay",
...
)
(note that I've also restructured the module a bit, since seemingly an __init__.pyx is disallowed...)
Now is where things become buggy and platform-dependent. I have two testing systems available - one Mac OS X 10.6 (Snow Leopard), using Macports Python 2.7, and one Mac OS X 10.7 (Lion) using the system python 2.7.
On Snow Leopard, the following applies:
This means that the module compiles (hurray!) (although there's no --inplace for numpy, it seems, so I had to system-wide install the testing module :/) but I still get a crash on import as follows:
>>> import delaunay
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<snip>site-packages/delaunay/__init__.py", line 1, in <module>
from sphere import delaunay_mesh
ImportError: dlopen(<snip>site-packages/delaunay/sphere.so, 2): no suitable image found. Did find:
<snip>site-packages/delaunay/sphere.so: mach-o, but wrong architecture
and on Lion, I get a compile error, following a rather confusing looking compile line:
gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f
/usr/local/bin/gfortran -Wall -arch i686 -arch x86_64 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/fortranobject.o build/temp.macosx-10.7-intel-2.7/delaunay/stripack.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.o -lgfortran -o build/lib.macosx-10.7-intel-2.7/delaunay/sphere.so
ld: duplicate symbol _initsphere in build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o ldand :build /temp.macosx-10.7-intelduplicate- 2.7symbol/ delaunay/sphere.o _initsphere in forbuild architecture /i386
temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o and build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o for architecture x86_64
Now let's just step back a moment before we pore over the details here. Firstly, I know there are a bunch of headaches over architecture clashes in 64-bit Mac OS X; I had to work very hard to get Macports Python working on the Snow Leopard machine (just to upgrade from system python 2.6). I also know that when you see gfortran -arch i686 -arch x86_64 you are sending mixed messages to your compiler. There are all manner of platform-specific problems buried in there, that we don't need to worry about in the context of this question.
But let's just look at this line:
gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f
what is numpy doing?! I don't need any f2py features in this build! I actually wrote a cython module in order to avoid dealing with f2py's insanity (I need to have 4 or 5 output variables, as well as neither-in-nor-out arguments - neither of which is well supported in f2py.) I just want it to compile .c -> .o, and .f90 -> .o and link them. I could write this compiler line myself if I knew how to include all the relevant headers.
Please tell me I don't need to write my own makefile for this... or that there's a way to translate fortran to (output-compatible) C so I can just avoid python ever seeing the .f90 extension (which fixes the whole problem.) Note that f2c is not suitable for this as it only works on F77 and this is a more modern dialect (hence the .f90 file extension).
UPDATE 2
The following bash script will happily compile and link the code in place:
PYTHON_H_LOCATION="/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/"
cython sphere.pyx
gcc -arch x86_64 -c sphere.c -I$PYTHON_H_LOCATION
gfortran -arch x86_64 -c stripack.f90
gfortran -arch x86_64 -bundle -undefined dynamic_lookup -L/opt/local/lib *.o -o sphere.so
Any advice on how to make this kind of hack compatible with a setup.py? I don't anyone installing this module to have to go find Python.h manually...

UPDATE: I've created a project on github which wraps up this generating of compile lines by hand. it's called complicated_build.
UPDATE 2: in fact, "generating by hand" is a really bad idea as it's platform specific — the project now reads the values from the distutils.sysconfig module, which is the settings used to compile python (i.e. exactly what we want,) the only setting which is guessed is fortran compiler and file extensions (which are user-configurable). I suspect it is reimplementing a fair bit of distutils now!
The way to do this is to write your own compiler lines, and hack them into your setup.py. I show an example below which works for my (very simple) case, which has the following strucutre:
imports
cythonize() any .pyx files, so you only have fortran and C files.
define a build() function which compiles your code:
maybe some easy-to-change constants, like compiler names and architecture
list up the fortran and C files
generate the shell commands that will build the modules
add the linker line
run the shell commands.
if the command was install and the target doesn't exist yet, build it.
run setup (which will build the pure python sections)
if the command was build, run the build now.
my implementation of this is shown below. It's designed for only one extension module, and it recompiles all the files every time, so may require further extension to be of more general use. Also note that I've hard coded various unix /s, so if you're porting this to windows make sure you adapt or replace with os.path.sep.
from distutils.core import setup
from distutils.sysconfig import get_python_inc
from Cython.Build import cythonize
import sys, os, shutil
cythonize('delaunay/sphere.pyx')
target = 'build/lib/delaunay/sphere.so'
def build():
fortran_compiler = 'gfortran'
c_compiler = 'gcc'
architecture = 'x86_64'
python_h_location = get_python_inc()
build_temp = 'build/custom_temp'
global target
try:
shutil.rmtree(build_temp)
except OSError:
pass
os.makedirs(build_temp) # if you get an error here, please ensure the build/ ...
# folder is writable by this user.
c_files = ['delaunay/sphere.c']
fortran_files = ['delaunay/stripack.f90']
c_compile_commands = []
for cf in c_files:
# use the path (sans /s), without the extension, as the object file name:
components = os.path.split(cf)
name = components[0].replace('/', '') + '.'.join(components[1].split('.')[:-1])
c_compile_commands.append(
c_compiler + ' -arch ' + architecture + ' -I' + python_h_location + ' -o ' +
build_temp + '/' + name + '.o -c ' + cf
)
fortran_compile_commands = []
for ff in fortran_files:
# prefix with f in case of name collisions with c files:
components = os.path.split(ff)
name = components[0].replace('/', '') + 'f' + '.'.join(components[1].split('.')[:-1])
fortran_compile_commands.append(
fortran_compiler + ' -arch ' + architecture + ' -o ' + build_temp +
'/' + name + '.o -c ' + ff
)
commands = c_compile_commands + fortran_compile_commands + [
fortran_compiler + ' -arch ' + architecture +
' -bundle -undefined dynamic_lookup ' + build_temp + '/*.o -o ' + target
]
for c in commands:
os.system(c)
if 'install' in sys.argv and not os.path.exists(target):
try:
os.makedirs('build/lib/delaunay')
except OSError:
# we don't care if the containing folder already exists.
pass
build()
setup(
name="delaunay",
version="0.1",
...
packages=["delaunay"]
)
if 'build' in sys.argv:
build()
This could be wrapped up into a new Extension class I guess, with it's own build_ext command - an exercise for the advanced student ;)

Simply build and install your vintage Fortran library outside of Python, then link to it in distutils. Your question indicates that you do not intend to temper with this library, so a once-and-for-all install will probably do (using the library's build and installation instructions). Then link the Python extension to the installed external library:
ext_modules = [
Extension("delaunay",
sources = ["delaunay/__init__.pyx"],
libraries = ["delaunay"])
]
This approach is also safe for the case that you realize that you need wrappers for other languages as well, such as Matlab, Octave, IDL, ...
Update
At some point, if you end up with more than a few such external libraries that you want to wrap, it is advantageous to add a top-level build system that installs all these libraries, and manages the building of all the wrappers as well. I have cmake for this purpose, which is great at handling system-wide builds and installations. However, it cannot build Python stuff out of the box, but it can be taught easily to call "python setup.py install" in each subdirectory python, thus invoking distutils. So the overall build process looks like this:
mkdir build
cd build
cmake ..
make
make install
make python
(make octave)
(make matlab)
It is very important to always separate core library code from wrappers for specific front-end languages (also for your own projects!), since they tend to change rather fast. What happens otherwise can be seen at the example of numpy: Instead of writing one general-purpose C library libndarray.so and creating thin wrappers for Python, there are Python C API calls everywhere in the sources. This is what is now holding back Pypy as a serious alternative to CPython, since in order to get numpy they have to support every last bit of the CPython API, which they can't do, since they have a just-in-time compiler and a different garbage collector. This means we are missing out on a lot of potential improvements.
Bottom line:
Build general purpose Fortran/C libraries separately and install them system-wide.
Have a separate build step for the wrappers, which should be kept as lightweight as possible, so that it's easy to adapt for the next big language X that comes about. If there is one safe assumption, it is that X will support linking with C libraries.

You can build the object file outside of distutils then include it at the linking step using the extra_objects argument to Extension's constructor. In setup.py:
...
e = Extension(..., extra_objects = ['holycode.o'])
...
On the command prompt:
# gfortran -c -fPIC holycode.f
# ./setup.py build_ext ...
With only one external object, this will be the easiest way for many.

Error setting up thrift modules for python

I'm trying to set up thrift in order to incorporate with Cassandra, so when I ran the
setup.py
it out puts this message in command line
running build
running build_py
running build_ext
building 'thrift.protocol.fastbinary' extension
C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python26\include -IC:\Pytho
n26\PC -c src/protocol/fastbinary.c -o build\temp.win32-2.6\Release\src\protocol
\fastbinary.o
src/protocol/fastbinary.c:24:24: netinet/in.h: No such file or directory
src/protocol/fastbinary.c:85:4: #error "Cannot determine endianness"
src/protocol/fastbinary.c: In function `writeI16':
src/protocol/fastbinary.c:295: warning: implicit declaration of function `htons'
src/protocol/fastbinary.c: In function `writeI32':
src/protocol/fastbinary.c:300: warning: implicit declaration of function `htonl'
src/protocol/fastbinary.c: In function `readI16':
src/protocol/fastbinary.c:688: warning: implicit declaration of function `ntohs'
src/protocol/fastbinary.c: In function `readI32':
src/protocol/fastbinary.c:696: warning: implicit declaration of function `ntohl'
error: command 'gcc' failed with exit status 1
Need some helping on this issue.I have already install the MigW32
Thanks.

With a bit of source file tweaking it is possible to install it with MINGW on Windows. I'm using thrift 0.9.1 and Python 27
The steps I followed were:
If you are using Python 2.7, follow the normal setup steps and workarounds for minGW. In particular you may need to open the file C:\Python27\Lib\distutils\cygwinccompiler.py, and modify the class Mingw32CCompiler to remove all references to the -mno-cygwin option. This option is deprecated and will cause the compiler to halt with an error if it is left in.
Open fastbinary.c and add the following include statement,
#include <stdbool.h>,
this includes definitions for true / false which will otherwise cause the compile to fail. (I assume they're included by default on MSVC?)
3) Modify the setup.py file to tell the linker to link to ws2_32.lib. This is done using a pragma comment on MSVC, but gcc doesn't support this option. So your ext_modules should look like :
ext_modules = [
Extension('thrift.protocol.fastbinary',
sources = ['src/protocol/fastbinary.c'],
libraries=['ws2_32'],
include_dirs = include_dirs,
)
],
4) Build as normal using the setup.py
On my setup, I didn't get much of a speed improvement when using the C extension rather than pure python (about a 5% difference), so the effort to do this might not be justified except in extreme cases.

I've only succeeded in installing Thrift with MSVC.
Install MSVC
Get Thrift
Apply thrift-252-python-msvc-1.diff patch (google it)
The fastbinary.c will be patched, but setup.py patch will fail, update manually from hints at setup.py.rej, here's a (seemingly) correct copy:
from distutils.core import setup, Extension
import sys
libraries = []
if sys.platform == 'win32':
libraries.append('ws2_32')
fastbinarymod = Extension('thrift.protocol.fastbinary',
sources = ['src/protocol/fastbinary.c'],
libraries = libraries,
)
setup(name = 'Thrift',
version = '0.1',
description = 'Thrift Python Libraries',
author = 'Thrift Developers',
author_email = 'thrift-dev#incubator.apache.org',
url = 'http://incubator.apache.org/thrift/',
license = 'Apache License 2.0',
packages = [
'thrift',
'thrift.protocol',
'thrift.transport',
'thrift.server',
],
package_dir = {'thrift' : 'src'},
ext_modules = [fastbinarymod],
)
Endianness test will fail, modify fastbinary.c (around line 68):
#ifdef _MSC_VER
#define __BYTE_ORDER __LITTLE_ENDIAN
#endif
After that run python setup.py install, hopefully, you'll get what you need.

Install python-dev
You can run:
sudo apt-get install python-dev

I would recommend you to execute the following command:
pip3 install thriftpy2