I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.
Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.
However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.
Here is the full python error:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in
import add_newdocs
File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in
from lib import add_newdoc
File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in
from type_check import *
File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in
import numpy.core.numeric as _nx
File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in
import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in
from openravepy import *
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in
openravepy_currentversion = loadlatest()
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
return _loadversion('_openravepy_')
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
mainpackage = __import__("openravepy", globals(), locals(), [targetname])
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in
from openravepy_int import *
ImportError: numpy.core.multiarray failed to import
I experienced the same problem with my application and solved it without linking python to the executable.
The setup is as follows:
Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter
The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.
dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)
This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.
It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.
The solution was linking the python2.6 library with my executable as well.
Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.
For clarification, my program model is something like:
[My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]
Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.
there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:
http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html
search for "python" in the cmake file here:
https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt
the relevant info is:
if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
find_package(PythonLibs)
if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
if( PYTHON_EXECUTABLE )
# get the site-packages directory
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
OUTPUT_VARIABLE _python_sitepackage
RESULT_VARIABLE _python_failed)
if( ${_python_failed} EQUAL 0 )
string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
else()
message(STATUS "failed to get python site-package directory")
endif()
endif()
include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
add_library(orpythonbinding SHARED orpythonbinding.cpp)
target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
if( WIN32 )
set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
endif()
endif()
endif()
Related
There is a custom module "ETPython" generated by SWIG (consists of ETPython.py and binary _ETPython.so) and a simple script that invokes this module
sample.py
import ETPython
...
There aren't any problems if the script is run in an IDE (pycharm, internal python's IDLE, squish by froglogic so on). But if I'm trying to launch it in python shell in interactive mode or via terminal using
python3 sample.py
there is an error message like:
File "<path_to_file>/example.py", line 12, in <module>
import ETPython
File "<path_to_file>/ETPython.py", line 15, in <module>
import _ETPython
ImportError: dlopen(<path_to_file>/_ETPython.so, 2): Symbol not found: _NSLog
Referenced from: <path_to_file>/_ETPython.so
Expected in: flat namespace
Searching topics, I found that problem is related to wrong paths. So I added some code:
import os, sys
os.chdir("<path_to_dir>")
sys.path.append('<path_to_dir>')
os.system('export PYTHONPATH=<path_to_dir>:$PYTHONPATH')
This code helped to import the module in python shell in interactive mode but launching in terminal is still failing.
So the question is how to make it to work?
I found solution. The SWIG module was compiled incorrectly.
There was option for CMake that suppressed errors for undefined symbols
set(PLATFORM_LIBS "-undefined dynamic_lookup" )
That why the module doesn't contain NSLOG symbol.
The module was recompiled with additional
"-framework Foundation"
in swig_link_libraries statement. And now the module is imported correctly
I am having issues getting python to import the _analog_swig gnuradio module in order to run gnuradio code on a Windows 8.1 64bit machine.
Some background: I am running Python 2.7.10 (installed in C:\Python27) and have installed the latest gnuradio binary (v3.7.11.1/v1.3 64-Bit Any CPU) from here: http://www.gcndevelopment.com/gnuradio/downloads.htm. I have installed gnuradio to C:\Program Files\GNURadio-3.7 .
I can run gnuradio companion and run flowgraphs from GRC successfully (which calls "C:\Program Files\GNURadio-3.7\bin\run_gr.bat" gnuradio-companion.py).
I have added & verified the following system variables are set:
Path: C:\Program Files\GNURadio-3.7\bin
PYTHONPATH: C:\Program Files\GNURadio-3.7\lib\site-packages
GRC_BLOCKS_PATH: C:\Program Files\GNURadio-3.7\share\gnuradio\grc\blocks
Now to the problem: If I run e.g. CMD and type:
python C:\test\top_block.py
I am returned the following ImportError:
File "C:\test\top_block.py", line 22, in <module>
from gnuradio import analog
File "C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog\__init__.py", line 33, in <module>
from analog_swig import *
File "C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog\analog_swig.py", line 17, in <module>
_analog_swig = swig_import_helper()
File "C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog\analog_swig.py", line 16, in swig_import_helper
return importlib.import_module('_analog_swig')
File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named _analog_swig
The folder content of C:\Program Files\GNURadio-3.7\lib\site-packages\gnuradio\analog is as follows:
Comparing this to the folder content on a linux machine, which has a working install of gnuradio that works with python as I want it:
The difference seems to be that the folder in windows contains only a _analog_swig.pyc file, whereas the folder in linux contains a _analog_swig_.so file.
Any idea why the _analog_swig module can apparently not be imported in windows?
My plan is to be able to run gnuradio code directly from my python interpreter and being able to create compiled gnuradio executables so any help on how this could be fixed is much appreciated.
I've been struggling with this for the past few days, but I finally figured it out. I was trying to run GnuRadio Companion generated code in IDLE and also in PyCharm. I kept failing miserably with this same error. I finally figured it out:
-As Flexo says, the PYD file (_analog_swig.pyd) is actually a Windows DLL. The error makes it sound like Python is not finding that file, but that is not at all what was happening. The PYD file, being a DLL, has dependencies itself. Python is able to find _analog_swig.pyd just fine, but it could not find the DEPENDENCIES of that library.
-To verify if that's what wrong in your installation, download and use DependencyWalker (Google it) to check if your system can find the dependencies to _analog_swig.pyd.
-The fix for me was to add the GnuRadio-3.7/bin folder to my PATH environment variable. Inside that folder are a number of DLLs that the _analog_swig.pyd library needs to load. If you don't have the folder in your PATH, the module will fail to load in Python and throw the error you see above.
-I see that you verified that this folder is in your PATH, so this is apparently not the same problem, although your symptoms are exactly the same as mine. i.e. the GRC code would run just fine when you start with "run_gr.bat", but not when you run from a normal CMD window.
Hopefully that helps someone else that wants to use GNURadio Python code on Windows.
Friend,
As you mentioned, the GNU Companion calls \bin\run_gr.bat gnuradio-companion.py. That batch script does quite a bit of work on windows environment variables (try opening it in a text editor if you're curious).
In a sense, the run_gr.bat script puts together a temporary, custom python workspace for gnuradio so it can import anything it needs. It receives python scripts to run in this environment as command line arguments; hence, you can use it to run any GNU radio python code you want in your windows command prompt. Generally, you would call
<gnuradio_install_path>\bin\run_gr.bat <gnu_radio_code>.py
To test your import, you can try
# test.py
from gnuradio import analog
try calling the following from the command prompt, in the test.py directory:
<gnuradio_install_path>\bin\run_gr.bat test.py
I'm trying to use py2exe to build an executable on 64-bit Windows 7 with Anaconda (Python 3.4) for a project of mine that depends on a lot of libraries. Some of the more complex ones include vispy (pyopengl), PyQt4, numba, and scipy. I've been stepping through various errors to try to get a working executable, but have hit a road block with no clear way forward. Currently, the py2exe command completes, but I get the following error when running the exe:
...
from numba import jit
File "C:\Anaconda3\envs\sift_py2exe\lib\site-packages\numba\__init__.py", line
13, in <module>
from .pycc.decorators import export, exportmany
File "C:\Anaconda3\envs\sift_py2exe\lib\site-packages\numba\pycc\__init__.py",
line 12, in <module>
from .cc import CC
File "C:\Anaconda3\envs\sift_py2exe\lib\site-packages\numba\pycc\cc.py", line
4, in <module>
from distutils.command import build_ext
File "C:\Anaconda3\envs\sift_py2exe\lib\distutils\command\build_ext.py", line
17, in <module>
from site import USER_BASE
ImportError: No module named 'site'
I was able to do a small workaround by adding the C:\Anaconda3\envs\sift_py2exe\Lib directory to sys.path in my main script, but I doubt that's going to help me much later. Not to mention I had more scipy DLL issues after that.
Here are the relevant parts of my setup.py:
try:
import py2exe
from llvmlite.binding.ffi import _lib_dir, _lib_name
kwargs["data_files"] = [('.', [os.path.join(_lib_dir, _lib_name), os.path.join(_lib_dir, "MSVCP120.dll"), os.path.join(_lib_dir, "MSVCR120.dll")])]
kwargs["console"] = [{
'script': 'cspov/__main__.py',
'dest_base': "SIFT",
}]
kwargs["options"] = {'py2exe': {"includes": ["vispy.app.backends._pyqt4", "PyQt4.QtNetwork"]}}
except ImportError:
print("'py2exe' and/or 'llvmlite' not available")
I've tried adding the "Lib" directory in setup.py and then including "site", but it doesn't find the module. Any ideas? Thanks.
Side note: I'm using the Microsoft DLLs from llvmlite as a quick workaround because I couldn't get it to work in any of the normal ways.
This isn't the answer I was hoping for, but I was able to get a working executable when I switched to pyinstaller. All of the other SO questions I saw relevant to my problem had similar "solutions".
I have installed Python 2.6.7 in $HOME/local of a machine which already has a default Python in /usr (I don't have admin access on this machine). The default Python is compiled in 32bits and my local installation is a 64bits. For some unknown reasons my local Python library (which I call as an embedded python interpreter from within a C program) search for the modules in the default (wrong) installation.
This is the result of "import random"
Traceback (most recent call last):
File "test.py", line 3, in <module>
import random
File "/tmp/work/mdorier1/local/lib/python2.6/random.py", line 45, in <module>
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
ImportError: /usr/lib/python2.6/lib-dynload/math.so: wrong ELF class: ELFCLASS32
As you can see, the import statement correctly search "random.py" in the local installation of Python, but the import statement in random.py go search for math.so in the wrong location, which ends in an error since the default location has 32 bits modules.
I guessed there is a problem with an environment variable, and I tried
import sys
sys.path
to get
['/tmp/work/mdorier1/local/lib/python26.zip',
'/tmp/work/mdorier1/local/lib/python2.6',
'/tmp/work/mdorier1/local/lib/python2.6/plat-linux2',
'/tmp/work/mdorier1/local/lib/python2.6/lib-tk',
'/tmp/work/mdorier1/local/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/tmp/work/mdorier1/local/lib/python2.6/site-packages']
I noticed that on of the paths indeed points to the default installation of Python. My questions thus are:
- Why this path shows up here, as the local installation has nothing to do with the default one?
- How do I change it (in a clean and permanent way)? This path should be the path to lib-dynload in the local installation instead.
Thanks
You probably don't have your Python's bin directory in the PATH variable before the system Python.
Or perhaps you simply compiled your Python incorrectly and did not use:
./configure --prefix=/tmp/work/mdorier1/local
so now it thinks that its files are somewhere else.
The crux of my problem is this:
I am developing code on Windows XP in C with MS Visual Studio 10.0, and I need to embed Python to do some plotting, file management, and some other things. I had problems with sys.path finding my Pure-Python modules, but I have fixed the problem by modifying PYTHONPATH.
Now, my problem is getting python to find dynamic libraries that are pulled in by some modules. In particular, my problem is to compress a folder into a bzip2 achive of the same name.
From a normal python command prompt, this works just fine:
import tarfile
tar=tarfile.open('Code.tar.bz2','w:bz2')
tar.add('Code',arcname='Code')
tar.close()
But when I call this code from my c-code, it gives me this error:
Traceback (most recent call last):
File "<string>", line 4, in <module>
File "D:\My_Documents\Code\ScrollModel\trunk\PythonCode.py", line 20, in Colle
ctFiles
tar=tarfile.open(os.path.join(runPath,'CODE.tar.bz2'),'w:bz2')
File "c:\Python26\lib\tarfile.py", line 1671, in open
return func(name, filemode, fileobj, **kwargs)
File "c:\Python26\lib\tarfile.py", line 1737, in bz2open
raise CompressionError("bz2 module is not available")
tarfile.CompressionError: bz2 module is not available
I have a suspicion the problem is similar to what is described at section 5.6 of Embedded Python, but it is a bit hard to tell. For what its worth, if I do
Py_Initialize();
PyRun_SimpleString("import ssl\n");
Py_Finalize();
it doesn't work either and I get an ImportError.
Anyone had any problems like this? Am I missing something critical?
Try this, it works on my machine.
Create a simple Windows console application in Visual Studio 2010 (remove precompiled headers option in the wizard). Replace the generated code with this one :
#include <Python.h>
int main(int argc, char *argv[]) {
Py_Initialize();
PyRun_SimpleString("import ssl \n"
"for f in dir(ssl):\n"
" print f \n" );
Py_Finalize();
return 0;
}
With PYTHONHOME set to something like c:\Python...
add C:\Python\Include to the include path
add C:\Python\Libs to the library path
add python26.lib to the linker input (adjust with your Python version)
Build. Run from anywhere and you should see a listing of the content of the ssl module.
I also tried with Mingw. Same file, build with this command line :
gcc -Wall -o test.exe embeed.c -I%PYTHONHOME%\Include -L%PYTHONHOME%\libs -lpython26
Hey, I have asked a similar question, my operation system is Linux.
When i compile c file, option $(python-config --cflags --ldflags) should be added, as
gcc test.c $(python-config --cflags --ldflags) -o test
I think in Windows you may also check python-config option, Hope this helps!
I had a similar problem with Boost C++ DLL. Any external DLL need to be in the DLL search path.
In my experience, PYTHONPATH affects Python module (the import statement in Python will end up a LoadLibrary call), and build options have nothing to do with it.
When you load a DLL, Windows doesn't care what the process is. In other words, Python follows the same DLL loading rules as Notepad. You can confirm that you are facing a Windows path problem by copying any missing DLL in the same directory as your python extension, or to a directory in your path.
To find what DLL are required by any other executable or DLL, simply open the DLL or EXE file with DependencyWalker. There is also a "Profile" menu which will allow you to run your application and watch it search and load DLLs.