I am working on making a python wrapper for some existing code using CFFI, but I'm getting stuck with packaging and distribution for Linux.
I've loosely followed the example here: https://github.com/wolever/python-cffi-example
And I've read this question as well: How to use shared dynamic libraries with python-cffi (in linux)?
I can get the python packager to install the .so file into the site-packages folder, but I can't get Python to locate it there (results in an ImportError: cannot open shared object file).
If I manually copy the .so file to /usr/lib or /usr/local/lib everything works fine, but I can't get the python package to copy files there (absolute paths don't work in a wheel) and that doesn't seem like the right answer anyway.
I also looked at modifying LD_LIBRARY_PATH in my python startup code, but that doesn't take effect for the current process only for new ones so that doesn't work either.
How can I distribute a .so file with a python package in a way that it can be located when the package is imported?
Related
I have a python program that uses OpenCV to get frames from a video file for processing. I then create a standalone executable using py2exe (also tried pyinstaller and got same error). My computer and the target computer are both Windows 7, but the target computer does not have python installed. I use OpenCV to read the frame rate and individual images from a video file.
Problem: When I run the executable on the target computer the frame rate is returned as 0.0 and I cannot read frames.
If python is installed on the target machine then the executable runs as expected, otherwise it produces this error. So it seems that something is missing in the executable, but I get no errors when creating the executable to indicate what might be missing.
Others who have reported similar issues usually have not included the numpy dependency (and get an error indicating this), but I have included numpy. I have also tried including the entire PyQt4 module since this is listed as a dependency on the python xy site for OpenCV (I already have parts of PyQt4 for other parts of the code) and this does not solve the problem either.
I guess I will go ahead and post an answer for this, but solution was provided by #otterb in the comments to the question. I am pasting the text here:
"py2exe is not perfect so will often miss some libraries or dll, pyd etc needed. Most likely you are missing opencv_highgui249.dll and opencv_ffmpeg249.dll etc. I would use py2exe with no single executable option enabled. And, start manually copying files that might be needed for your app. After identifying them, modify setup.py for py2exe to include them automatically."
I will note however that I use pyinstaller rather than py2exe, since I get fewer problems while building. I still have to manually copy the opencv dll files though.On Windows 7 they are located here: "C:\Python27\DLLs" and they need to be copied into the distribution folder so that they are on the same path as the other dll files that go with the distribution.
Try using pyinstaller, download it using pip :
pip install pyinstaller
if you don't know how to install pip , try downloading python 2.7.9 or above, which has inbuilt pip, but don forget to add python path to environment varibles, this procedure is mentioned in this post:
How to run Pip commands from CMD
After installing pyinstaller, select the main file of your project and run this command
pyinstaller yourprogram.py
it will create folder with application file naming your file name, and finally make sure that numpy and opencv are in lib->site-packages folder of your python27 in your C Folder while running that command
Hope it Helps!
Java has the concept of packaging all of the code into a file called a Jar file. Does Python have an equivalent idea? If so, what is it? How do I package the files?
Python doesn't have any exact equivalent to a .jar file.
There are many differences, and without knowing exactly what you want to do, it's hard to explain how to do it. But the Python Packaging User Guide does a pretty good job of explaining just about everything relevant.
Here are some of the major differences.
A .jar file is a compiled collection of classes that can be dropped into your application, or installed anywhere on your CLASSPATH.
In Python:
A .py (or .pyc) module can be dropped into your application, or installed anywhere on your sys.path, and it can be imported and used.
A directory full of modules can be treated the same way; it becomes a package (or, if it doesn't contain an __init__.py, it merges with other directories of the same name elsewhere on sys.path into a single package).
A .zip archive containing any number of modules and packages can be stored anywhere, and its path added to your sys.path (e.g., at runtime or via PYTHONPATH) and all of its contents become importable.
Most commonly, you want things to be installed into a system, user, or virtualenv site-packages directory. The recommended way to do that is to create a pip-compatible package distribution; people then install it (and possibly automatically download it from PyPI or a private repo) via pip.
pip does a lot more than that, however. It also allows you to manage dependencies between packages. So ideally, instead of listing a bunch of prereqs that someone has to go download and install manually, you just make them dependencies, and someone just has to pip install your-library. And it keeps track of the state of your site-packages, so you can uninstall or upgrade a package without having to track down the specific files.
Meanwhile, in Java, most .jar files are cross-platform; build once, run anywhere. A few packages have JNI native code and can't be used this way, but it's not the norm.
In Python, many packages have C extensions that have to be compiled for each platform, and even pure-Python packages often need to do some install-time configuration. And meanwhile, "compiling" pure Python doesn't do anything that can't be done just as well at runtime. So in Python, you generally distribute source packages, not compiled packages.
However, .wheel is a binary package format. You can pip wheel to build binary packages for different targets from the source package; then, if someone tries to pip install your package, if there's a wheel for his system, that will be downloaded and installed.
Easy Install from setup_tools defines the .egg format for deploying Python libraries or applications. While similar to JAR, it is nowhere spread as universally as JARs in Java world. Many people just deploy the .py files.
A newer format, intended to supersede eggs, is wheel.
Though it's not a perfect susbstitute of jar due to portability issues, I would add the "auto-extracting" archive way.
One possibility is "makeself": https://makeself.io/
But if you don't need to package external files, and if you like KISS approach, the following is a nice and clean alternative:
The following is taken from Asim Jalis's website.
How to deploy a Python application as a zip file
Create a file __main__.py containing:
print "Hello world from Python"
Zip up the Python files (in this case just this one file) into app.zip by typing:
zip app.zip *
The next step adds a shebang to the zip file and saves it as app—at this point the file app is a zip file containing all your Python sources.
echo '#!/usr/bin/env python' | cat - app.zip > app
chmod 755 app
That’s it. The file app is now have a zipped Python application that is ready to deploy as a single file.
You can run app either using a Python interpreter as:
python app
Or you can run it directly from the command line:
./app
Reference: https://gist.github.com/asimjalis/4237534
I installed a local version of Python 2.7 in my home directory (Linux RedHat) under ~/opt using the --prefix flag.
More specifically, Python was placed in ~/home/opt/bin.
Now, I want to install NumPy, but I am not really sure how I would achieve this. All I found in the INSTALL.txt and online documentation was the command to use the compiler.
I tried gfortran, and it worked without any error message:
python setup.py build --fcompiler=gnu95
However, I am not sure how to install it for my local version of Python.
Also, I have to admit that I don't really understand how this whole approach works in general. E.g., what is the setup.py build doing? Is it creating module files that I have to move to a specific folder?
I hope anyone can give me some help here, and I would also appreciate a few lines of information how this approach works, or maybe some resources where I can read it up (I didn't find anything on the NumPy pages).
Your local version of python should keep all of it's files somewhere in ~/opt (presumably). As long as this is the python installation that gets used when you issue the command
python setup.py build --fcompiler=gnu95
you should be all set because in the sys module, there are a bunch of constants which the setup script uses to determine where to put the modules once they are built.
So -- running python setup.py build issues all of the necessary commands to build the module (compiling the C/Fortran code into shared object libraries that python can load dynamically and copying the pure python code to create the proper directory structure). The module is actually built somewhere in the build subdirectory which gets created during the process if it doesn't already exist. Once the library has been built (successfully), installing it should be as simple as:
python setup.py install
(You might need to sudo if you don't have write privileges in the install directory).
I am planning to use Metis for drawing some networks and graphs using Python. I downloaded Metis from here. For installing the Python wrapper, I followed the instructions given here, but when I run the easy_install command like:
$ easy_install metis
I get this error:
RuntimeError: Could not locate METIS dll. Please set the METIS_DLL environment variable to its full path.
From the second link above, I understand that I have to specify the full path to some 'METIS shared library' file in the METIS_DLL environment variable. But I don't know what the file is called and where it is located or how I should specify the path in an environment variable.
According to my research on Google, I found out that the file might be called 'libmetis.so' but I don't have any file of that name. Could someone give me a clue on this?
Any help would be appreciated, thanks.
I assume that you are working on Linux (because of the .so suffix) and that you have downloaded, compiled and installed the source code linked in the page. User libraries are installed in /usr/lib or /usr/local/lib and their subdirectories. Note that you might have lib64 or lib32 also, so try searching there. Try also in /opt and in the site-packages Python external modules folder, although it's usually placed inside /usr/lib. Entering locate libmetis.so at the command line might help.
If you have downloaded that package and not compiled yet, then you simply don't have that file. Read BUILD.txt or BUILD-windows.txt and follow the instructions.
Please note that this question belongs on Superuser rather than on Stackoverflow.
I am trying to get simplejson installed on this python 2.4.3 - I cannot upgrade, I know it is old, there is nothing I can do about it, it is not my fault, please help. however when i do the ..\python.exe .\setup.py install i get:
File "C:\Program Files (x86)\WorldViz\Vizard30\bin\lib\zipfile.py", line 188, in __init__ raise RuntimeError,\
RuntimeError: Compression requires the (missing) zlib module
Does anyone know how can I get zlib installed on this windows 64 machine? or where I can get a compiled version of simplejson or where can I find a compatible alternative for it.
Again, I can't do anything about it being python 2.4.3 - it is a proprietary modified version of python that I cannot do a thing about.
Copying the .py files into site-packages will get some stuff to work - particularly you can read files from an uncompressed zip archive, but you will not be able to inflate compressed files. The problem you are having is that ./config is not finding zlib.so (or possibly zlib.h) where it expects to find it and skips making it. This is a problem in compiling old versions of Python (pre 2.6) on recent Ubuntu boxes (Natty Narwhal - 11.0 in particular) since Canonical restructured the /lib directory contents to better (for them) support multiple architectures. So config is looking for the file /lib/libz.so and it is in /usr/lib/i386-linux-gnu/libz.so (or some other /usr/lib/$ARCH/ directory depending upon your machine).
It is supposed to be possible to tell config additional directories to search by exporting shell variables. I couldn't get that to work. What did work was the nasty hack
ln -s /usr/lib/i386-linux-gnu/libz.so /lib/libz.so
There are other parts of the standard library that config considers optional that are affected by this, but I didn't have time to track them down. After you do "make" do a "make test" and at the end it will list which packages were expected on Linux that it did not find.
The zlib module comes in the Standard Library, and I do not see a separately installable version on the Python Package Index. It is actually integral to Python, since Python cannot run its own Standard Library without it if the library is distributed as a ZIP file, so this must be a very customized Python if it lacks it. Could you build normal Python from source in another directory and in the presence of the library files for zlib, and then copy the zlib.so file over to see if it will link against this crazy Python you are using?
Edit: And, come to think of it, why do you need a compiled version of simplejson? Is the plain Python version not enough?
Edit: Here is how I can install simplejson without trying to get its accelerator routine to compile:
$ wget http://pypi.python.org/packages/source/s/simplejson/simplejson-2.1.1.tar.gz
$ tar xvfz simplejson-2.1.1.tar.gz
$ cd simplejson-2.1.1
$ cp -r simplejson /usr/lib/python2.4/site-packages
Those first three steps can all occur on some other system, and of course you can just use your web browser rather than wget. But the point is that you can just copy the whole simplejson directory from inside of the .tar.gz into your site-packages and it should work. It's a messy way to manually install, but it sure won't depend on any other dependencies! :-)