We Keep Coding

building extension for Python 3 with Swig and distutils

I've built an application ("Checkyrs") in C++, and now am doing something external which will use large parts of Checkyrs but is built in Python 3. (This is all something in my free time, so it doesn't need to be Python 3, but that's my preference.)
To get an interface between the python and C++, I'm using SWIG and the python distutils package. I've built a dynamic library containing what I need from Checkyrs, and I've successfully built a Python extension ("checkyrsai") using the tools I mentioned. I've tested it in Python 2.7 and it works fully, all of the C++ classes and functions I need are available and function correctly.
But my preference is to work with Python 3, and while I can build the extension with Python 3 I'm unable to load it successfully:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import checkyrsai
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/chris/Documents/Programming/eggyolk/checkyrsai.py", line 28, in <module>
_checkyrsai = swig_import_helper()
File "/Users/chris/Documents/Programming/eggyolk/checkyrsai.py", line 24, in swig_import_helper
_mod = imp.load_module('_checkyrsai', fp, pathname, description)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: dlopen(/Users/chris/Documents/Programming/eggyolk/_checkyrsai.so, 2): Symbol not found: __ZN4Game11ExecuteMoveERKSt6vectorI8PositionSaIS1_EE
Referenced from: /Users/chris/Documents/Programming/eggyolk/_checkyrsai.so
Expected in: flat namespace
in /Users/chris/Documents/Programming/eggyolk/_checkyrsai.so
My process for building the extension (for Python 2) is:
swig -c++ -python checkyrsai.i
python setup.py build_ext --inplace
where my setup.py file looks like this:
from distutils.core import setup, Extension
import os
os.environ["CC"] = "g++"
checkyrsai = Extension('_checkyrsai',
sources = ['checkyrsai_wrap.cxx','../checkyrs/checkyrs/ai.cpp'],
include_dirs = ['/usr/local/include','../checkyrs/checkyrs'],
libraries = ['Checkyrs'],
library_dirs = ['../checkyrs/Build/Products/Release/','/usr/local/lib'],
extra_compile_args = ['-std=c++11']
)
setup (name = 'checkyrs',
version = '1.0',
description = 'checkyrs',
ext_modules = [checkyrsai])
As I said above, this works perfectly. From this point I can open my python (2.7) interpreter,
import checkyrsai
and off I go to play with my new toy.
When trying to build for Python 3 I use almost exactly the same process, just with the addition of Python 3 flags for SWIG and running distutils through Python 3:
swig -c++ -python -py3 checkyrsai.i
python3 setup.py build_ext --inplace
this runs through the compilation successfully and produces the extension, but when I try to
import checkyrsai
I get the ImportError […] Symbol not found issue quoted above.
I don't change my code or the setup.py script in any way between the Python 2 and Python 3 versions. The symbol refers to a method which should be found in my libCheckyrs.dylib. It evidently is available there as it is used successfully when I use the Python 2.7 extension - but it seems not to be found when I make the extension for Python 3. Does anybody have any suggestion where I'm going wrong?

I eventually solved this issue by changing XCode project settings for the C++ library I was linking against.
Specifically, changing the "C++ Language Dialect" setting to "C++ [-std=c++11]", i.e. the same version I had specified in the extra_compile_args setting for distutils. Previously it had been GNU++11, hence symbols not matching as there was a mismatch in the namespace (std::vector vs. std::__1::vector).
With that change I'm now happily and successfully able to call my C++ code from Python 3.
I don't really understand why it did work in python 2.7, since it was using all the same distutils settings, with the same C++ version specified and the linking against the same C++ lib. If anybody has an explanation for that I'd love to hear it.

Matlab Engine Python - OSx Anaconda Segfault or DYLD_LIBRARY_PATH error with iPython

After installing the matlab python package in terminal using:
cd "matlabroot\extern\engines\python"
python setup.py install
And trying to run it, I get a segfault:
:~$ python
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import matlab.engine
Segmentation fault: 11
However, I can get around this by setting DYLD_LIBRARY_PATH after which matlab.engine works:
:~$ export DYLD_LIBRARY_PATH=/System/Library/Frameworks/Python.framework/Versions/Current/lib:$DYLD_LIBRARY_PATH
:~$ python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import matlab.engine
>>> eng = matlab.engine.start_matlab()
>>> exit()
However, when I try to launch iPython afterwards I get this error:
Traceback (most recent call last):
File "//anaconda/bin/ipython", line 4, in <module>
from IPython import start_ipython
File "//anaconda/lib/python2.7/site-packages/IPython/__init__.py", line 45, in <module>
from .config.loader import Config
File "//anaconda/lib/python2.7/site-packages/IPython/config/__init__.py", line 6, in <module>
from .application import *
File "//anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 19, in <module>
from IPython.config.configurable import SingletonConfigurable
File "//anaconda/lib/python2.7/site-packages/IPython/config/configurable.py", line 12, in <module>
from .loader import Config, LazyConfigValue
File "//anaconda/lib/python2.7/site-packages/IPython/config/loader.py", line 16, in <module>
from IPython.utils.path import filefind, get_ipython_dir
File "//anaconda/lib/python2.7/site-packages/IPython/utils/path.py", line 14, in <module>
import tempfile
File "//anaconda/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "//anaconda/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(//anaconda/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
Referenced from: //anaconda/lib/python2.7/lib-dynload/_io.so
Expected in: dynamic lookup
As you can see the python versions are different. I think this is a conflict between my system Python and Anaconda but I'm not sure how to fix it, any help much appreciated.
Thanks.

You need to install libraries using the conda command line install utility, not python. conda help search and conda help install should get you going.

This is not a direct solution, but I was stuck with the same problem and used this workaround. And I should say that I did finally make matlab.engine run in my Python environment, but that it just plain sucked. It couldn't run scripts with embedded scripts in other folders, and I also experienced that it couldn't find some build-in functions. The following is implemented for a Unix machine, but it would not take a lot of modification to make it work in Windows, I believe.
Here's what I did:
Wrote a main Matlab script that could do everything I needed from Matlab.
Ran that script through a subprocess in Python.
In my case I needed do a series of matrix operations for matrix X and return matrix S. The matrix operations required the use of a particular Matlab function. My first idea was to open a Matlab-session with matlab.engine and then manage the matrix operations in Python only calling the Matlab function when needed. Instead (as the bullets state) I wrote a Matlab function ComputeSimilarityMat that takes X, does all necessary operations, and returns S. Then I basically just ran that Matlab function from Python using a subprocess.
This is what the Python script, that manages the Matlab script through a subprocess, looks like:
import subprocess
import numpy as np
def run_matlab(X):
"""Produce trait-similarity matrix S from a trait-space matrix X
Parameters
----------
X : numpy.ndarray
Returns
-------
S : numpy.ndarray
"""
# Dump input in .csv, so Matlab can load it
np.savetxt('X.csv', X, delimiter=",")
# Code executed in Matlab. Important to end with quit;
matlab_code = "X=csvread('X.csv');" \
"S=ComputeSimilarityMat(X);" \
"csvwrite('S.csv',S);" \
"quit;"
# -nosplash suppresses the splash window on startup
matlab_call = ["matlab", "-nodesktop", "-nosplash", "-r", matlab_code]
subprocess.call(matlab_call)
# Load the .csv file that Matlab dumps after finishing
S = np.genfromtxt('X.csv', delimiter=",")
return S
I have to say, I'm sure there's a nicer way to pass an object to Matlab then to have to save and load like this, but it worked for me. Still hoping for an improvement of matlab.engine though, but until then I'm going with this approach.
Note: To run Matlab from command-line/subprocess you need to add the Matlab bin folder to your PATH variable like this:
export PATH="/Applications/MATLAB_R2015a.app/bin:$PATH"
I just put this in my .profile file.

In my case adding things to LD_LIBRARY_PATH (Ubuntu's version of DYLD_LIBRARY_PATH) only made things worse. Instead I had to make sure it did not refer to the Python installation, and had to add a symbolic link from /usr/lib instead. See https://stackoverflow.com/a/45161288/2524427

Python ImportError when attempting to import sqlite3 module

I am trying to cross compile Python 2.7.3 for an arm based embedded device. I have managed to compile it successfully (based on these instructions: http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html) and all of the tests pass on the target device so I'm confident that the build process works. I've cross compiled sqlite3 (version 3.8.5) and included it in the python cross compile process which it seems to pick up fine (it is no longer listed in the modules which were not found at the end of the build process).
I'm having difficulty actually trying to import the sqlite3 library on the target device, I get the error listed below (python is running with the -v flag).
Python 2.7.3 (default, Jul 7 2014, 19:06:12)
[GCC 3.4.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
import sqlite3 # directory /mnt/card/arm-python/lib/python2.7/sqlite3
# /mnt/card/arm-python/lib/python2.7/sqlite3/__init__.pyc matches /mnt/card/arm-python/lib/python2.7/sqlite3/__init__.py
import sqlite3 # precompiled from /mnt/card/arm-python/lib/python2.7/sqlite3/__init__.pyc
# /mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.pyc matches /mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.py
import sqlite3.dbapi2 # precompiled from /mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.pyc
dlopen("/mnt/card/arm-python/lib/python2.7/lib-dynload/datetime.so", 2);
import datetime # dynamically loaded from /mnt/card/arm-python/lib/python2.7/lib-dynload/datetime.so
dlopen("/mnt/card/arm-python/lib/python2.7/lib-dynload/time.so", 2);
import time # dynamically loaded from /mnt/card/arm-python/lib/python2.7/lib-dynload/time.so
dlopen("/mnt/card/arm-python/lib/python2.7/lib-dynload/_sqlite3.so", 2);
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/card/arm-python/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: File not found
It seems to be complaining about a "file not found" but I've been through all of the paths listed in the output an all of the files seem to exist. Is there anything I can do to diagnose this problem further?

I've managed to get it working although I don't think I fully understand what's going on, I'm not that familiar with compiling C/C++ code and how the whole static/shared libraries and linking works, maybe someone can shed some light on what's actually going on here. Anyway, here's how I've resolved it.
First I ran strace (Linux debugging utility) on the python process:
strace /mnt/card/arm-python/bin/python
This spits out a load of output as the python process starts, once it's settled down I tried to import the sqlite library:
import sqlite3
This will then spit out a load more output, most of it relating to opening the files involved in the module your are importing. I then noticed it managed to open _sqlite3.so but shortly afterwards it tried to load libsqlite3.so which wasn't found on the library paths ($LD_LIBRARY_PATH).
open("/mnt/card/arm-python/lib/python2.7/lib-dynload/_sqlite3.so", O_RDONLY|O_LARGEFILE) = 5
...
open("/lib/libsqlite3.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/libsqlite3.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
I copied libsqlite3.so.0.8.6 from the /lib directory within the cross compiled sqlite library on my build machine to /mnt/card on the embedded arm device and renamed it to libsqlite3.so.0. I then added /mnt/card to the $LD_LIBRARY_PATH as the existing locations in that path reside on the read-only filesystem.
I then tried to import sqlite3 again and it all seem to be working fine. So what is the role of the stuff in the lib-dynload and also the role of libsqlite3.so.0?

f2py with include files

I am compiling a fortran program called prog.f. It contains an include file called test.inc.
This below runs successfully and shows that my include file is found. I have a prog.so file generated.
f2py -c prog.f -m prog
However when I call this module from python I get this error message:
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import prog
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./prog.so: undefined symbol: unknown_function_
I suspect something to do with my compilation arguments but I am not too familiar with Fortran. Do I need to include my include file as well? If so how?

Yes, you need the include file. It might be something as simple as:
f2py -c include_file.f prog.f -m prog
although I haven't tested that. Alternatively, you might need to use the fortran include statement to include include_file into prog ... Also, if you're more familiar with C, virtually all fortran compilers that I know use the convention that prog.F is fortran code that should be preprocessed by cpp. So you could probably get your file included that way as well.

I also ran into this problem, and was able to find other discussions that simply stated that current f2py does not support preprocessing of files, such as #include statements and conditional compilation.
https://groups.google.com/forum/#!topic/f2py-dev/aW65sEoSdG8
http://mail.scipy.org/pipermail/numpy-discussion/2009-November/046381.html
Two solutions:
Simplest: If you are using gfortran to compile your code, rename any .f90 or .f file to .F90 or .F. Using a capital letter keys gfortran to automatically use the preprocessor (even if you don't specify).
Second:
Use your Fortran compiler to preprocess the file and save the result as an intermediate Fortran code to be sent to f2py.
gfortran -E -D<Defines-for-compiler> file.f -cpp -o outdir/file.f
where -E is the command to tell gfortran to just preprocess the file (may vary from compiler to compiler), and -D option to define any values to use during the preprocessing.
Saving the file to another directory (or changing the name completely) is necessary to avoid overwriting the original .f file.

Undefined Symbol in C++ When Loading a Python Shared Library

I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.
Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.
However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.
Here is the full python error:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in
import add_newdocs
File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in
from lib import add_newdoc
File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in
from type_check import *
File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in
import numpy.core.numeric as _nx
File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in
import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in
from openravepy import *
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in
openravepy_currentversion = loadlatest()
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
return _loadversion('_openravepy_')
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
mainpackage = __import__("openravepy", globals(), locals(), [targetname])
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in
from openravepy_int import *
ImportError: numpy.core.multiarray failed to import

I experienced the same problem with my application and solved it without linking python to the executable.
The setup is as follows:
Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter
The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.
dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)
This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.
It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.

The solution was linking the python2.6 library with my executable as well.
Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.
For clarification, my program model is something like:
[My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]

Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.

there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:
http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html
search for "python" in the cmake file here:
https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt
the relevant info is:
if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
find_package(PythonLibs)
if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
if( PYTHON_EXECUTABLE )
# get the site-packages directory
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
OUTPUT_VARIABLE _python_sitepackage
RESULT_VARIABLE _python_failed)
if( ${_python_failed} EQUAL 0 )
string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
else()
message(STATUS "failed to get python site-package directory")
endif()
endif()
include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
add_library(orpythonbinding SHARED orpythonbinding.cpp)
target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
if( WIN32 )
set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
endif()
endif()
endif()