The crux of my problem is this:
I am developing code on Windows XP in C with MS Visual Studio 10.0, and I need to embed Python to do some plotting, file management, and some other things. I had problems with sys.path finding my Pure-Python modules, but I have fixed the problem by modifying PYTHONPATH.
Now, my problem is getting python to find dynamic libraries that are pulled in by some modules. In particular, my problem is to compress a folder into a bzip2 achive of the same name.
From a normal python command prompt, this works just fine:
import tarfile
tar=tarfile.open('Code.tar.bz2','w:bz2')
tar.add('Code',arcname='Code')
tar.close()
But when I call this code from my c-code, it gives me this error:
Traceback (most recent call last):
File "<string>", line 4, in <module>
File "D:\My_Documents\Code\ScrollModel\trunk\PythonCode.py", line 20, in Colle
ctFiles
tar=tarfile.open(os.path.join(runPath,'CODE.tar.bz2'),'w:bz2')
File "c:\Python26\lib\tarfile.py", line 1671, in open
return func(name, filemode, fileobj, **kwargs)
File "c:\Python26\lib\tarfile.py", line 1737, in bz2open
raise CompressionError("bz2 module is not available")
tarfile.CompressionError: bz2 module is not available
I have a suspicion the problem is similar to what is described at section 5.6 of Embedded Python, but it is a bit hard to tell. For what its worth, if I do
Py_Initialize();
PyRun_SimpleString("import ssl\n");
Py_Finalize();
it doesn't work either and I get an ImportError.
Anyone had any problems like this? Am I missing something critical?
Try this, it works on my machine.
Create a simple Windows console application in Visual Studio 2010 (remove precompiled headers option in the wizard). Replace the generated code with this one :
#include <Python.h>
int main(int argc, char *argv[]) {
Py_Initialize();
PyRun_SimpleString("import ssl \n"
"for f in dir(ssl):\n"
" print f \n" );
Py_Finalize();
return 0;
}
With PYTHONHOME set to something like c:\Python...
add C:\Python\Include to the include path
add C:\Python\Libs to the library path
add python26.lib to the linker input (adjust with your Python version)
Build. Run from anywhere and you should see a listing of the content of the ssl module.
I also tried with Mingw. Same file, build with this command line :
gcc -Wall -o test.exe embeed.c -I%PYTHONHOME%\Include -L%PYTHONHOME%\libs -lpython26
Hey, I have asked a similar question, my operation system is Linux.
When i compile c file, option $(python-config --cflags --ldflags) should be added, as
gcc test.c $(python-config --cflags --ldflags) -o test
I think in Windows you may also check python-config option, Hope this helps!
I had a similar problem with Boost C++ DLL. Any external DLL need to be in the DLL search path.
In my experience, PYTHONPATH affects Python module (the import statement in Python will end up a LoadLibrary call), and build options have nothing to do with it.
When you load a DLL, Windows doesn't care what the process is. In other words, Python follows the same DLL loading rules as Notepad. You can confirm that you are facing a Windows path problem by copying any missing DLL in the same directory as your python extension, or to a directory in your path.
To find what DLL are required by any other executable or DLL, simply open the DLL or EXE file with DependencyWalker. There is also a "Profile" menu which will allow you to run your application and watch it search and load DLLs.
Related
Okay so according to the answer I found here to the question titled "Calling C/C++ from Python?" here, and also on the cppyy documentation website, I made some sample classes in .h and .cpp files and tried to include them in Python. While the .h file gets included easily, when I try to use the cppyy.load_library() function, it gives me a runtime error for some reason. Can someone please help? I've tried to look for solutions online but apparently no one has got a similar problem before. This is what I'm running in Jupyter Notebook:
import cppyy
cppyy.include("foo.h")
cppyy.load_library("libfoo")
the final line is giving me the following error:
RuntimeError Traceback (most recent call last)
<ipython-input-3-eea6173ad08e> in <module>
----> 1 cppyy.load_library("libfoo")
~\anaconda3\lib\site-packages\cppyy\__init__.py in load_library(name)
219 sc = gSystem.Load(name)
220 if sc == -1:
--> 221 raise RuntimeError('Unable to load library "%s"%s' % (name, err.err))
222 return True
223
RuntimeError: Unable to load library "libfoo"
This is my .h file:
class Foo {
public:
void bar();
};
And here is my .cpp file:
#include "foo.h"
#include <iostream>
void Foo::bar() { std::cout << "Hello" << std::endl; }
I'm using the commands g++ -c -fPIC foo.cpp -o foo.o and g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o to compile my cpp code.
Please can someone help?
The example works just fine for me.
To debug, first make sure to start python in the same directory where libfoo.so is located, or to add the directory where libfoo.so lives to LD_LIBRARY_PATH (for any process to use), or call cppyy.add_library_path() with the path as argument to add it for use by cppyy only. As concerns the name, the .so extension is automatically added as needed, as is the lib part, so either one of foo, libfoo, foo.so, or libfoo.so is fine.
If that still fails, a reasonable way on Linux (only) of getting more information for what could be going wrong, is to use ctypes:
$ python
>>> import cypes
>>> lib = ctypes.CDLL("libfoo.so")
which will show you if there are different problems such as missing symbols or missing dependent libraries (but neither is the case here).
I recently worked on a python project where I need to import some .pyx files.
However, I had a hard time import it as .pyx and compile it with the following lines:
import pyximport
pyximport.install()
from . import kl
The error is
fatal error: capsule.h: No such file or directory
And in the kl.pyx file, capsule.h is used as following:
cdef extern from "capsule.h":
void* Capsule_AsVoidPtr(object ptr)
I am not sure what I should do. Please help me a little bit.
Alternatively, is it possible that the python script could import compiled .c (in Linux) file directly, instead of importing .pyx and compile it?
When I compile it with command line, it is compiled successfully, but I don't know how my IDE (PyCharm) could import that .c file.
Thank you very much.
you missed some library, you need to install the source code of capsule.h.
I wrote a simple application which uses selenium to nagivate through pages and download their source code. Now I would like to make my application Windows-executable.
My setup.py file:
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 1,
"dll_excludes": ['w9xpopen.exe', 'MSVCP90.dll', 'mswsock.dll', 'powrprof.dll', 'MPR.dll', 'MSVCR100.dll', 'mfc90.dll'],
'compressed': True,"includes":["selenium"],
}
},
windows = [{'script': "main.py", "icon_resources": [(1, "hacker.ico")]}],
zipfile = None
)
My program (main.py) (with setup.py file) is located in C:\Documents and Settings\student\Desktop. Py2exe builds my exe in C:\Documents and Settings\student\Desktop\dist.
I copied both webdriver.xpi and webdriver_prefs.json files to C:\Documents and Settings\student\Desktop\dist\selenium\webdriver\firefox\, but I'm getting the error when trying to launch my application:
Traceback (most recent call last):
File "main.py", line 73, in <module>
File "main.py", line 58, in check_file
File "main.py", line 25, in try_to_log_in
File "selenium\webdriver\firefox\webdriver.pyo", line 47, in __init__
File "selenium\webdriver\firefox\firefox_profile.pyo", line 63, in __init__
IOError: [Errno 2] No such file or directory: 'C:\\Documents and Settings\\student\\Desktop\\dist\\main.exe\\selenium\\webdriver\\firefox\\webdriver_prefs.json'
How to solve this?
Actually, it worked with such setup.py file:
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
wd_path = 'C:\\Python27\\Lib\\site-packages\\selenium\\webdriver'
required_data_files = [('selenium/webdriver/firefox',
['{}\\firefox\\webdriver.xpi'.format(wd_path), '{}\\firefox\\webdriver_prefs.json'.format(wd_path)])]
setup(
windows = [{'script': "main.py", "icon_resources": [(1, "hacker.ico")]}],
data_files = required_data_files,
options = {
"py2exe":{
"skip_archive": True,
}
}
)
But the problem is I need to build SINGLE executable.
Have you tried to have a look at this answer for the "bundle_files = 1" problems? It helped me solving that specific problem.
TL;DR --Please check out this tool I built: https://github.com/douglasmccormickjr/PyInstaller-Assistance-Tools--PAT
Might I suggest using PyInstaller instead of py2exe or anything else for that matter since PyInstaller does a far better job in terms of bundling a single executable. I'm on Windows about 90% of the time (no complaints here) with my python coding-- PyInstaller is a way better option than py2exe (for me at least -- I've used/test a great deal of Windows compilers in the past with varied success). Maybe other people suffering from compiling issues could benefit from this method as well.
PyInstaller Prerequisites:
Install PyInstaller from: http://www.pyinstaller.org/
After PyInstaller installation-- confirm both "pyi-makespec.exe" and "pyi-build.exe" are in the "C:\Python##\Scripts" directory on your machine
Download my PyInstaller-Assitance-Tools--PAT (it's just 2 batch files and 1 executable with the executable's source python file too -- for the paranoid)...The file are listed above:
Create_Single_Executable_with_NO_CONSOLE.bat
Create_Single_Executable_with_CONSOLE.bat
pyi-fixspec.exe
pyi-fixpec.py (optional -- this is the source file for the executable -- not needed)
Place the exectuable file called "pyi-fixspec.exe" inside the previous "Scripts" folder I mentioned above...this makes compiling much easier in the long run!
let's get it working now...some slight code changes to your python application
I use a standard function that references the location of applications/scripts that my python application needs to utilize to work while being executed/operated. This function operates both when the app is a standalone python script or when it's fully compiled via pyinstaller.
Here's the piece of code I use...
def resource_path(relative_path):
""" Get absolute path to resource, works for dev and for PyInstaller """
if hasattr(sys, '_MEIPASS'):
return os.path.join(sys._MEIPASS, relative_path)
return os.path.join(os.path.abspath("."), relative_path)
and here's my app using it....
source = resource_path("data\my_archive_file.zip")
that means the app/files look something like this in terms of directory structure:
C:\Path\To\Application\myscript_001.py <--- main application/script intended to be compiled
...
C:\Path\To\Application\data\my_archive_file.zip <---|
C:\Path\To\Application\data\images\my_picture.jpg <---| supporting files in the bundled app
C:\Path\To\Application\info\other_stuff.json <---|
...
Please note that the data/files/folders I'm bundling for my app are below the main executable/script that I'll be compiling...the "_MEIPASS" part in the function lets pyinstaller know that it's working as a compiled application...VERY IMPORTANT NOTE: Please use the function "resource_path" since the "pyi-fixspec.exe" application will be looking for that phrase/function while parsing/correcting the python application pathing
Goto the directory containing the 2 batch files mentioned above and type in either:
Option #1
C:\MyComputer\Downloads\PAT> Create_Single_Executable_with_NO_CONSOLE.bat C:\Path\to\the\python\file.py
The output executable file results in a GUI app when double clicked
Option #2
C:\MyComputer\Downloads\PAT> Create_Single_Executable_with_CONSOLE.bat C:\Path\to\the\python\file.py
The output executable file results in a WINDOWS CONSOLE app when double clicked -- expects commandline activity ONLY
Your new single-file-executable is done! Check for it in this location
C:\Original\Directory\ApplicationDistribution64bit\NameOfPythonFile\dist
If you do edit/change the original python file that has just been previously compiled, please delete the folder/contents of **\NameOfPythonFile** prior to next compile kickoff (you'll want to delete the historical/artifact files)
Coffee break -- or if you wany to edit/add ICONS to the executable (and other items too), please look at the generated ".spec" file and PyInstaller documentation for configuration details. You'll just need to kick off this again in the windows console:
pyi-build.exe C:\path\to\the\pythonfile.spec
You can build a single executable, which will run natively, by using Nuitka. It converts the Python code into C++ and then compiles it.
http://nuitka.net/
It does, however, require that you have a compiler installed. The appropriate versions of either Microsoft Visual C++ or GCC. Microsoft released "Microsoft Visual C++ Compiler for Python 2.7", which can be obtained here at https://www.microsoft.com/en-us/download/details.aspx?id=44266.
A nice installer of MinGW GCC for windows can be found at https://github.com/develersrl/gccwinbinaries with detailed instructions, including which MSVCRTXX.dll version to link with for which version of Python.
Things you will gain from this method of executable generation:
Doing this generates machine code, so it will be more difficult, but not impossible, to reverse engineer. Simply using Py2exe or PyInstaller, which are great for their intended use, only packages the byte compiled Python code, which is easily decompiled (http://www.simonroses.com/2013/10/appsec-myths-about-obfuscation-and-reversing-python/), into a zip appended executable.
Your application will also gain a bit of a speed boost. I wrote a blog post about this kind of thing at (I DO NOT receive money from click throughs or ads).
https://jaredfields83.wordpress.com/2015/12/21/squeezing-more-juice-from-python/.
The problem you have is that selenium is trying to open a file in a way that is not directly compatible with py2exe.
As you can see at line 63 here, selenium must open a preferences file that is usually shipped with the package. It uses the __file__ variable to find it, which doesn't play well with py2exe. As you have found, it is possible to work around that by also packaging up the prefs file in your distribution. However, that is now more than one file.
The py2exe wiki then has a recipe to use NSIS that will build a self-extracting executable of your complete distribution.
I built a DLL in VS2010 with boost::python to export some function to a python module:
myDLL.cpp:
std::string greet() { return "hello, world"; }
int square(int number) { return number * number; }
BOOST_PYTHON_MODULE(getting_started1)
{
// Add regular functions to the module.
def("greet", greet);
def("square", square);
}
Up to here, everything compiles just fine. I then get the myDLL.dll and myDLL.lib file in c:\myDLL\Debug.
According to boost doc (http://wiki.python.org/moin/boost.python/SimpleExample), I need to add this to PYTHONPATH, so I added c:\myDLL\Debug to it:
PYTHONPATH:
C:\Python27;c:\myDLL\Debug;
then, from my .py file, I try to import it:
import getting_started1
print getting_started1.greet()
number = 11
print number, '*', number, '=', getting_started1.square(number)
I have also tried from myDLL import getting_started1, and from getting_started1 import *, and all possible combinations of sorts.
Can anyone tell me how am I supposed to call my module? Thanks
EDIT:
According to cgohlke, there should be a getting_started1.pyd somewhere in my PYTHONPATH when I compile in VS? This file is inexistant... Do I have to set somethign different in VS2010? I have a default win32 DLL project.
But the boost doc says " If we build this shared library and put it on our PYTHONPATH", isn't a shared library on windows a DLL? ergo, the DLL should be in the PYTHONPATH?
The standard, portable way to build Python extensions is via distutils. However, Visual Studio 2010 is not a supported compiler for Python 2.7. The following setup.py works for me with Visual Studio 2008 and boost_1_48_0. The build command is python setup.py build_ext --inplace.
# setup.py
from distutils.core import setup
from distutils.extension import Extension
setup(name="getting_started1",
ext_modules=[
Extension("getting_started1", ["getting_started1.cpp"],
include_dirs=['boost_1_48_0'],
libraries = ['boost_python-vc90-mt-1_48'],
extra_compile_args=['/EHsc', '/FD', '/DBOOST_ALL_DYN_LINK=1']
)
])
For your Visual Studio 2010 project, try to change the linker output file to getting_started1.pyd instead of myDLL.dll.
I managed to get it working only in Release configuration and not in Debug.
From the project properties, on the General tab, modify Target Extension to .pyd
The project should be indeed a dll, as you did
In the Python script you need to specify the location of the dll as in this example:
import sys
sys.path.append("d:\\TheProjectl\\bin\\Release")
import getting_started #Dll name without the extension
I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.
Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.
However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.
Here is the full python error:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in
import add_newdocs
File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in
from lib import add_newdoc
File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in
from type_check import *
File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in
import numpy.core.numeric as _nx
File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in
import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in
from openravepy import *
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in
openravepy_currentversion = loadlatest()
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
return _loadversion('_openravepy_')
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
mainpackage = __import__("openravepy", globals(), locals(), [targetname])
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in
from openravepy_int import *
ImportError: numpy.core.multiarray failed to import
I experienced the same problem with my application and solved it without linking python to the executable.
The setup is as follows:
Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter
The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.
dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)
This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.
It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.
The solution was linking the python2.6 library with my executable as well.
Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.
For clarification, my program model is something like:
[My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]
Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.
there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:
http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html
search for "python" in the cmake file here:
https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt
the relevant info is:
if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
find_package(PythonLibs)
if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
if( PYTHON_EXECUTABLE )
# get the site-packages directory
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
OUTPUT_VARIABLE _python_sitepackage
RESULT_VARIABLE _python_failed)
if( ${_python_failed} EQUAL 0 )
string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
else()
message(STATUS "failed to get python site-package directory")
endif()
endif()
include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
add_library(orpythonbinding SHARED orpythonbinding.cpp)
target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
if( WIN32 )
set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
endif()
endif()
endif()