I've built an application ("Checkyrs") in C++, and now am doing something external which will use large parts of Checkyrs but is built in Python 3. (This is all something in my free time, so it doesn't need to be Python 3, but that's my preference.)
To get an interface between the python and C++, I'm using SWIG and the python distutils package. I've built a dynamic library containing what I need from Checkyrs, and I've successfully built a Python extension ("checkyrsai") using the tools I mentioned. I've tested it in Python 2.7 and it works fully, all of the C++ classes and functions I need are available and function correctly.
But my preference is to work with Python 3, and while I can build the extension with Python 3 I'm unable to load it successfully:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import checkyrsai
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/chris/Documents/Programming/eggyolk/checkyrsai.py", line 28, in <module>
_checkyrsai = swig_import_helper()
File "/Users/chris/Documents/Programming/eggyolk/checkyrsai.py", line 24, in swig_import_helper
_mod = imp.load_module('_checkyrsai', fp, pathname, description)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: dlopen(/Users/chris/Documents/Programming/eggyolk/_checkyrsai.so, 2): Symbol not found: __ZN4Game11ExecuteMoveERKSt6vectorI8PositionSaIS1_EE
Referenced from: /Users/chris/Documents/Programming/eggyolk/_checkyrsai.so
Expected in: flat namespace
in /Users/chris/Documents/Programming/eggyolk/_checkyrsai.so
My process for building the extension (for Python 2) is:
swig -c++ -python checkyrsai.i
python setup.py build_ext --inplace
where my setup.py file looks like this:
from distutils.core import setup, Extension
import os
os.environ["CC"] = "g++"
checkyrsai = Extension('_checkyrsai',
sources = ['checkyrsai_wrap.cxx','../checkyrs/checkyrs/ai.cpp'],
include_dirs = ['/usr/local/include','../checkyrs/checkyrs'],
libraries = ['Checkyrs'],
library_dirs = ['../checkyrs/Build/Products/Release/','/usr/local/lib'],
extra_compile_args = ['-std=c++11']
)
setup (name = 'checkyrs',
version = '1.0',
description = 'checkyrs',
ext_modules = [checkyrsai])
As I said above, this works perfectly. From this point I can open my python (2.7) interpreter,
import checkyrsai
and off I go to play with my new toy.
When trying to build for Python 3 I use almost exactly the same process, just with the addition of Python 3 flags for SWIG and running distutils through Python 3:
swig -c++ -python -py3 checkyrsai.i
python3 setup.py build_ext --inplace
this runs through the compilation successfully and produces the extension, but when I try to
import checkyrsai
I get the ImportError […] Symbol not found issue quoted above.
I don't change my code or the setup.py script in any way between the Python 2 and Python 3 versions. The symbol refers to a method which should be found in my libCheckyrs.dylib. It evidently is available there as it is used successfully when I use the Python 2.7 extension - but it seems not to be found when I make the extension for Python 3. Does anybody have any suggestion where I'm going wrong?
I eventually solved this issue by changing XCode project settings for the C++ library I was linking against.
Specifically, changing the "C++ Language Dialect" setting to "C++ [-std=c++11]", i.e. the same version I had specified in the extra_compile_args setting for distutils. Previously it had been GNU++11, hence symbols not matching as there was a mismatch in the namespace (std::vector vs. std::__1::vector).
With that change I'm now happily and successfully able to call my C++ code from Python 3.
I don't really understand why it did work in python 2.7, since it was using all the same distutils settings, with the same C++ version specified and the linking against the same C++ lib. If anybody has an explanation for that I'd love to hear it.
Related
I have a library that I've created with PyO3 on my system using Python 3.5.2. The .so file links to the corresponding libpython3.5m file:
$ ldd my_library.so
linux-vdso.so.1 => (0x00007ffffc823000)
libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007fcac34b0000)
...
But if I try to use my library on another system with a different minor version of Python (eg, 3.6.9 or 3.7.3), that library doesn't exist:
$ ldd my_library.so
linux-vdso.so.1 (0x00007fffefaae000)
libpython3.5m.so.1.0 => not found
...
And therefore, I can't use my library:
$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_library
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory
Is there a way that I can link my library to libpython more generically to be resilient to these minor version changes or at least allow backward-compatibility (build on newer version but allow older versions of Python to also use my library)? Or anything I can do at runtime to let the library still function?
Python extension modules shouldn't link to libpython at all. The Python symbols will stay unresolved, and the dynamic loader will look them up in whichever python interpreter loads the module. That way you can even use your module with a statically linked Python!
rust-cpython has an extension-module feature for this and PyO3 seems to have copied that.
You an enable it in your Cargo.toml:
[dependencies]
pyo3 = { version = ..., features = [ "extension-module" ] }
The initial problem was solved, but there are more ...
__
I found this neat LTS tool online. Since it takes quite some time for training I'd like to speed things up. While searching for an approach I came across PyPy.
Pypy is now setup and seems to be working, but not with the Sequitur g2p code. When importing a module which comes from a .so file I get a No module named _sequitur_ .
I also tried installing/compiling the code with PyPy instead of Python which crashes without any useful(?) error.
I run PyPy v2.6.1 on a Ubuntu Linux 14.04 with Python 2.7.10.
Calling g2p with the default interpreter works fine.
Does the problematic .so file need to be compiled via a call from PyPy ?
Edit #1:
When trying to install g2p with PyPy instead of CPython it crashes without after this command:
error: Command "cc -O2 -fPIC -Wimplicit -DMULTIGRAM_SIZE=2 -I/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include -I/opt/pypy-2.6.1-linux64/include -c sequitur_wrap.cpp -o build/temp.linux-x86_64-2.7/sequitur_wrap.o -fpermissive" failed with exit status 1
When calling this particular command from the console, without being a part of the setup.py, there are two errors:
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:192:55: error: ‘PyArray_DescrFromType’ was not declared in this scope
PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \
&
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:194:69: error: ‘PyArray_FromAny’ was not declared in this scope
NPY_ARRAY_ENSUREARRAY, NULL)
This either didn't appear or caused an error when installing with CPhyton.
diff also shows that ndarrayobject.h under the mentioned path doesn't differ from the one in /usr/local/lib/python2.7/site-packages/numpy/andsoon .
Since I have no clue how c++ works, I'm lost there...
Edit #2:
Well, somehow it didn't work with the first include (pointing to pypy's numpy). Probably not the clean way to do it, but it helped to get rid of that problem. So I substituted the include_dirs entry in the setup.py and pointed that one to cPython's numpy include dir.
That done, the setup.py runs without any error. When I now try to import sequitur (one of the necessary files) I noticed that one .so was missing, copied it from cPython's numpy to Pypy's numpy. So far so good but now I still get this error:
$ pypy
Python 2.7.10 (f3ad1e1e1d62, Aug 28 2015, 10:45:29)
[PyPy 2.6.1 with GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import sequitur
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sequitur.py", line 32, in <module>
import sequitur_, SequenceModel, Minimization, misc
File "sequitur_.py", line 28, in <module>
_sequitur_ = swig_import_helper()
File "sequitur_.py", line 20, in swig_import_helper
import _sequitur_
ImportError: numpy.core.multiarray failed to import
Any suggestions?
Thanks
I am trying to cross compile Python 2.7.3 for an arm based embedded device. I have managed to compile it successfully (based on these instructions: http://randomsplat.com/id5-cross-compiling-python-for-embedded-linux.html) and all of the tests pass on the target device so I'm confident that the build process works. I've cross compiled sqlite3 (version 3.8.5) and included it in the python cross compile process which it seems to pick up fine (it is no longer listed in the modules which were not found at the end of the build process).
I'm having difficulty actually trying to import the sqlite3 library on the target device, I get the error listed below (python is running with the -v flag).
Python 2.7.3 (default, Jul 7 2014, 19:06:12)
[GCC 3.4.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
import sqlite3 # directory /mnt/card/arm-python/lib/python2.7/sqlite3
# /mnt/card/arm-python/lib/python2.7/sqlite3/__init__.pyc matches /mnt/card/arm-python/lib/python2.7/sqlite3/__init__.py
import sqlite3 # precompiled from /mnt/card/arm-python/lib/python2.7/sqlite3/__init__.pyc
# /mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.pyc matches /mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.py
import sqlite3.dbapi2 # precompiled from /mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.pyc
dlopen("/mnt/card/arm-python/lib/python2.7/lib-dynload/datetime.so", 2);
import datetime # dynamically loaded from /mnt/card/arm-python/lib/python2.7/lib-dynload/datetime.so
dlopen("/mnt/card/arm-python/lib/python2.7/lib-dynload/time.so", 2);
import time # dynamically loaded from /mnt/card/arm-python/lib/python2.7/lib-dynload/time.so
dlopen("/mnt/card/arm-python/lib/python2.7/lib-dynload/_sqlite3.so", 2);
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/card/arm-python/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/mnt/card/arm-python/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: File not found
It seems to be complaining about a "file not found" but I've been through all of the paths listed in the output an all of the files seem to exist. Is there anything I can do to diagnose this problem further?
I've managed to get it working although I don't think I fully understand what's going on, I'm not that familiar with compiling C/C++ code and how the whole static/shared libraries and linking works, maybe someone can shed some light on what's actually going on here. Anyway, here's how I've resolved it.
First I ran strace (Linux debugging utility) on the python process:
strace /mnt/card/arm-python/bin/python
This spits out a load of output as the python process starts, once it's settled down I tried to import the sqlite library:
import sqlite3
This will then spit out a load more output, most of it relating to opening the files involved in the module your are importing. I then noticed it managed to open _sqlite3.so but shortly afterwards it tried to load libsqlite3.so which wasn't found on the library paths ($LD_LIBRARY_PATH).
open("/mnt/card/arm-python/lib/python2.7/lib-dynload/_sqlite3.so", O_RDONLY|O_LARGEFILE) = 5
...
open("/lib/libsqlite3.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/libsqlite3.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
I copied libsqlite3.so.0.8.6 from the /lib directory within the cross compiled sqlite library on my build machine to /mnt/card on the embedded arm device and renamed it to libsqlite3.so.0. I then added /mnt/card to the $LD_LIBRARY_PATH as the existing locations in that path reside on the read-only filesystem.
I then tried to import sqlite3 again and it all seem to be working fine. So what is the role of the stuff in the lib-dynload and also the role of libsqlite3.so.0?
I am compiling a fortran program called prog.f. It contains an include file called test.inc.
This below runs successfully and shows that my include file is found. I have a prog.so file generated.
f2py -c prog.f -m prog
However when I call this module from python I get this error message:
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import prog
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./prog.so: undefined symbol: unknown_function_
I suspect something to do with my compilation arguments but I am not too familiar with Fortran. Do I need to include my include file as well? If so how?
Yes, you need the include file. It might be something as simple as:
f2py -c include_file.f prog.f -m prog
although I haven't tested that. Alternatively, you might need to use the fortran include statement to include include_file into prog ... Also, if you're more familiar with C, virtually all fortran compilers that I know use the convention that prog.F is fortran code that should be preprocessed by cpp. So you could probably get your file included that way as well.
I also ran into this problem, and was able to find other discussions that simply stated that current f2py does not support preprocessing of files, such as #include statements and conditional compilation.
https://groups.google.com/forum/#!topic/f2py-dev/aW65sEoSdG8
http://mail.scipy.org/pipermail/numpy-discussion/2009-November/046381.html
Two solutions:
Simplest: If you are using gfortran to compile your code, rename any .f90 or .f file to .F90 or .F. Using a capital letter keys gfortran to automatically use the preprocessor (even if you don't specify).
Second:
Use your Fortran compiler to preprocess the file and save the result as an intermediate Fortran code to be sent to f2py.
gfortran -E -D<Defines-for-compiler> file.f -cpp -o outdir/file.f
where -E is the command to tell gfortran to just preprocess the file (may vary from compiler to compiler), and -D option to define any values to use during the preprocessing.
Saving the file to another directory (or changing the name completely) is necessary to avoid overwriting the original .f file.
I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.
Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.
However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.
Here is the full python error:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in
import add_newdocs
File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in
from lib import add_newdoc
File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in
from type_check import *
File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in
import numpy.core.numeric as _nx
File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in
import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in
from openravepy import *
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in
openravepy_currentversion = loadlatest()
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
return _loadversion('_openravepy_')
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
mainpackage = __import__("openravepy", globals(), locals(), [targetname])
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in
from openravepy_int import *
ImportError: numpy.core.multiarray failed to import
I experienced the same problem with my application and solved it without linking python to the executable.
The setup is as follows:
Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter
The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.
dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)
This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.
It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.
The solution was linking the python2.6 library with my executable as well.
Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.
For clarification, my program model is something like:
[My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]
Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.
there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:
http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html
search for "python" in the cmake file here:
https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt
the relevant info is:
if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
find_package(PythonLibs)
if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
if( PYTHON_EXECUTABLE )
# get the site-packages directory
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
OUTPUT_VARIABLE _python_sitepackage
RESULT_VARIABLE _python_failed)
if( ${_python_failed} EQUAL 0 )
string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
else()
message(STATUS "failed to get python site-package directory")
endif()
endif()
include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
add_library(orpythonbinding SHARED orpythonbinding.cpp)
target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
if( WIN32 )
set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
endif()
endif()
endif()