I want to send somebody my compiled fortran extension on a Mac (compiled with f2py and gfortran).
Problem is that it doesn't work on other Macs unless they also instal xcode (2 GB, yikes!) and gfortran. So apparently there are some additional files missing when I just send the compiled extension.
Does anybody know what other files to include or (better) how to compile a fortran extension without needing to send any additional files?
Thanks,
Mark
Well, when you compile a module with f2py, it essentially creates a dynamic library (.so) which uses your system libraries. For instance, on my computer, the linking step is,
gfortran [...] -lpython2.7 -lgfortran -o ./my_f2py_module.so
Therefore, if you want to be able to execute the resulting f2py module on a different computer (assuming the same architecture), libpython2.7.so and libgfortran.so should be available there.
I don't know much about OS/X deployment, but I think you should
use a compiler present by default on Mac (i.e. clang), which should work with f2py. Or alternatively install gfortran on both systems.
make sure you link to the same version of python in both cases
Also use ldd ./my_f2py_module.so to list all the libraries that it is linked to. If some of them cannot be found on the current system, you will also see it with this command.
Related
As I discussed in a previous question, I am trying to run a python script that uses f2py wrapped Fortran code on a remote server. If I compile the f2py module locally on my own pc and then run the script on the server, I get the following error,
ImportError: liblapack.so.3: cannot open shared object file: No such file or directory
This is logical since lapack and blas are both not installed on the remote server, and sadly I do not have the root privileges to install them. I am wondering if I could fix this by using static linking. On the GNU wiki it is stated that,
One way to avoid this (more ideas can be found on the binaries page) is to use the so-called "static linking", available with option -static. Gfortran then puts the library code inside the program created, thus enabling it to run without the library present (like, on a computer where gfortran is not installed).
This sounds like exactly what I would need. If the required library files are included in the f2py module then I won't need the lapack and blas modules on the remote server and I can just run it as is. However I am not sure that this static linking actually works with f2py. Currently I run,
python -m numpy.f2py -llapack -lblas -c --fcompiler=gnu95 --compiler=unix signature_file.pyf
object_files.o
object_files.o means all interdependent Fortran files compiled into object files, i.e. I ran
gfortran -c -llapack -lblas object_file.F90
on all relevant files. Now I would like to do this with static linking, but how? I have tried adding -static when compiling the object files, but this didn't change anything. I have also tried adding -static to the f2py call, but this just gives an unknown option error.
Yes, this is possible inside a setup.py script if you use Extension from numpy.distutils.core with extra_link_args=["-static", "-static-libgfortran", "-static-libgcc"].
It is explained in this answer.
Does python require microsoft visual c++ redistributable to run the code ?
I'm using pyinstaller to compile my .py code into exe. In some systems my exe is asking for microsoft visual c++ redistributable package to run.
Does pyinstaller includes microsoft visual c++ redistributable files while making exe ? If no, how can I include those files so that I don't need to install microsoft visual c++ redistributable package into other's system to run my software ?
What are the other alternatives to build a standalone software in python ? I'm reading to use other languages along with python.
I saw electron js and python can be used together to make desktop application. But how will I distribute that application as a standalone exe ?
Python itself does not depend on the presence of MSVC. You can download a portable Python package, and it will run wherever you copy it. Those are the embeddable ones from https://www.python.org/downloads/windows/
But, Python modules with native extension code inside can depend on MSVC on multiple levels:
if the native part comes in binary format (.pyd file on Windows), actually that is a .dll, and it may depend on other .dll-s, depending on how it has been built
if the native part comes as C/C++ source code, it will be built at installation time, typically via a "setup.py", and this procedure will need a C compiler installed on the system
PyInstaller is a different story. First of all, it has a documentation which you may want to read. For example the page What PyInstaller Does and How It Does It clearly gives a direct answer to at least one of your questions:
Bundling to One File
PyInstaller can bundle your script and all its dependencies into a single executable named myscript (myscript.exe in Windows).
There is also a list of packages with known compatibility and known compatibility issues: https://github.com/pyinstaller/pyinstaller/wiki/Supported-Packages, which you may find useful depending on what packages you need.
While it is not a duplicate, this question: How to package a linked DLL and a pyd file into one self contained pyd file? (and another one it links) may be interesting to read.
Of course Python requires MSVC Redistributable, any native Windows program using standard library functions requires it. Obviously, Python uses lots of them and should provide a consistent environment across all extension modules.
However, since Python 3.5 it is bundled with an installer, so there's no need to install it manually. Python installers prior to 3.5 don't include it and I wasn't able to find any clarifications whether it's downloaded during installation or not.
By default Python also enforces extension modules to be compiled with the same (or, since 3.5, compatible) version of MSVC as an interpreter itself. So except for some very rare cases extension modules will also use the same redistributable.
"Embeddable" Python releases referred by #tevemadar are NOT a "portable Python"! Here's what the documentation says about their usage:
It is intended for acting as part of another application, rather than being directly accessed by end-users.
Note: The embedded distribution does not include the Microsoft C Runtime and it is the responsibility of the application installer to provide this. The runtime may have already been installed on a user’s system previously or automatically via Windows Update, and can be detected by finding ucrtbase.dll in the system directory.
But you still don't need them if you use PyInstaller.
To check whether or not redistributable files are included in your .exe file you could probably open it with any archiver software and see it for yourself. My guess is that they can be included at least if Python is installed in a single user mode, as in such case they're installed in the Python directory as well.
Other than that, however, you should really ask your questions separately.
What I'm trying to do is ship my code to a remote server, that may have different python version installed and/or may not have packages my app requires.
Right now to achieve such portability I have to build relocatable virtualenv with interpreter and code. That approach has some issues (for example, you have to manually copy a bunch of libraries into your virtualenv, since --always-copy doesn't work as expected) and generally slow.
There's (in theory) a way to build python itself statically.
I wonder if I could pack interpreter with my code into one binary and run my application as module. Something like that: ./mypython -m myapp run or ./mypython -m gunicorn -c ./gunicorn.conf myapp.wsgi:application.
There are two ways you could go about to solve your problem
Use a static builder, like freeze, or pyinstaller, or py2exe
Compile using cython
This answer explains how you can go about doing it using the second approach, since the first method is not cross platform and version, and has been explained in other answers. Also, using programs like pyinstaller typically results in huge file sizes, while using cython will result in a file that's much smaller
First, install cython.
sudo -H pip3 install cython
Then, you can use cython to generate a C file out of the Python .py file
(in reference to https://stackoverflow.com/a/22040484/5714445)
cython example_file.py --embed
Use GCC to compile it after getting your current python version (Note: The below assumes you are trying to compile it to Python3)
PYTHONLIBVER=python$(python3 -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')$(python3-config --abiflags)
gcc -Os $(python3-config --includes) example_file.c -o output_bin_file $(python3-config --ldflags) -l$PYTHONLIBVER
You will now have a binary file output_bin_file, which is what you are looking for
Other things to note:
Change example_file.py to whatever file you are actually trying to compile.
Cython is used to use C-Type Variable definitions for static memory allocation to speed up Python programs. In your case however, you will still be using traditional Python definitions.
If you are using additional libraries (like opencv, for example), you might have to provide the directory to them using -L and then specify the name of the library using -l in the GCC Flags. For more information on this, please refer to GCC flags
The above method might not work for anaconda python, as you will likely have to install a version of gcc that is compatible with your conda-python.
You might wish to investigate Nuitka. It takes python source code and converts it in to C++ API calls. Then it compiles into an executable binary (ELF on Linux). It has been around for a few years now and supports a wide range of Python versions.
You will probably also get a performance improvement if you use it. Recommended.
You're probably looking for something like Freeze, which is able to compile your Python application with all its libraries into a static binary:
PyPi page of Freeze
Python Wiki page of Freeze
Sourceforge page of Freeze
If you are on a Mac you can use py2app to create a .app bundle, which starts your Django app when you double-click on it.
I described how to bundle Django and CherryPy into such a bundle at https://moosystems.com/articles/14-distribute-django-app-as-native-desktop-app-01.html
In the article I use pywebview to display your Django site in a local application window.
Freeze options:
https://pypi.python.org/pypi/bbfreeze/1.1.3
http://cx-freeze.sourceforge.net/
However, your target server should have the environment you want -> you should be able to 'create' it. If it doesn't, you should build your software to match the environment.
I found this handy guide on how to install custom version of python to a virtualenv, assuming you have ssh access: https://stackoverflow.com/a/5507373/5616110
In virtualenv, you should be able to pip install anything and you shouldn't need to worry about sudo privileges. Of course, having those and access to package manager like apt makes everything a lot easier.
I have created a docker image that relies on Nuitka and a custom statically linked python3.10 to create a static binary.
Did not test it extensively, if you have the chance please let me know if it works for your use case.
You can check it at:
https://github.com/joaompinto/docker-build-python-static-bin
I currently have Xcode (along with command line tools) and gfrotran from HPC installed on my Yosemite system, and would like to replace HPC's gfortran with Homebrew's (because I'm having trouble building Python packages with the HPC gfortran).
What are the steps to accomplish this?
I want to be sure that
HPC's gfortran is gone (I just want one Fortran) and
Apple's tools still work (for Xcode, Swift, OS X and iOS development, etc.)
and of course
that I have a working version of gfortran that reliably builds Python packages.
Can this be done? I see for example that Homebrew's gfortran is now packaged as part of gcc, so it looks like I'll end up with two versions gcc (which I'd like to avoid) or one that doesn't play well with Xcode.
What are the steps to accomplish 1-3?
I recently worked through this very same problem. This is how I got/have it working on my Yosemite system.
Some things about home-brew and the command line tools compilers:
The compiler you get through home-brew will overwrite the existing one if it is in the same place. If you have a fortran compiler that you obtained from the command line tools then it will not be affected.(Home-brew will warn you about this before doing anything)
The binaries for the compilers you get through the command line tools are in /usr/bin .
The compilers (and anything else) you obtain using home-brew are stored in the cellar ( /usr/local/Cellar)
and the executables are linked into the directory /usr/local/bin.
What to do?
Modify the path:
I only needed to use the compilers I got through home-brew so I moved /usr/local/bin to the top of my $path this ensures that the /usr/local/bin directory is searched before /usr/bin.
Use aliases :
You can create an alias for each compiler so that you can use them interchangeably as you wish. To create an alias you will have to modify your shell-rc file. On my system I use the tcsh, and to create an alias for the home-brew compiler I would add something similar to this to my ~/.cshrc file.
alias brewgfortran '/usr/local/bin/gfortran'
this now executes the home-brew gfortran-4.9 executable that is stored in my cellar and linked to /usr/local/bin/gfortran/
IF you absolutely want rid of the apple compilers you can of course remove them from the /usr/bin/ directory completely. I don't think this is the best idea though. I have shown you a few ways to avoid using them and if you ever needed them for some reason you would be SOL. I cant say if those tools you need will work without them as I have never used any of those, however I know it will build python packages for you (sorry hope this still helps)
NOTE: If you are not using the same shell as me the syntax for the alias is a little different (I think) just google it or man alias
I have an account in a remote computer without root permissions and I needed to install a local version of Python (the remote computer has a version of Python that is incompatible with some codes I have), Numpy and Scipy there. I've been trying to install numpy locally since yesterday, with no success.
I successfully installed a local version of Python (2.7.3) in /home/myusername/.local/, so I access to this version of Python by doing /home/myusername/.local/bin/python. I tried two ways of installing Numpy:
I downloaded the lastest stable version of Numpy from the official webpage, unpacked it, got into the unpacked folder and did: /home/myusername/.local/bin/python setup.py install --prefix=/home/myusername/.local. However, I get the following error, which is followed by a series of other errors (deriving from this one):
gcc -pthread -shared build/temp.linux-x86_64-2.7/numpy/core/blasdot/_dotblas.o
-L/usr/local/lib -Lbuild/temp.linux-x86_64-2.7 -lptf77blas -lptcblas -latlas
-o build/lib.linux-x86_64-2.7/numpy/core/_dotblas.so
/usr/bin/ld: /usr/local/lib/libptcblas.a(cblas_dptgemm.o): relocation
R_X86_64_32 against `a local symbol' can not be used when making a shared
object; recompile with -fPIC
Not really knowing what this meant (except that the error apparently has to do with the LAPACK library), I just did the same command as above, but now putting LDFLAGS='-fPIC', as suggested by the error i.e., I did
LDFLAGS="-fPIC" /home/myusername/.local/bin/python setup.py install --prefix=/home/myusername/.local.
However, I got the same error (except that the prefix -fPIC was addeded after the gcc command above).
I tried installing it using pip, i.e., doing /home/myusername/.local/bin/pip install numpy /after successfully instaling pip in my local path). However, I get the exact same error.
I searched on the web, but none of the errors seemed to be similar to mine. My first guess is that this has to do with some piece of code that needs root permissions to be executed, or maybe with some problem with the version of the LAPACK libraries.
Help, anyone?
The error message is telling you that your ATLAS library has not been built with the -fPIC flag. That means it cannot be linked into a shared library like Python extension modules. You need to rebuild ATLAS with the -fPIC flag. The ATLAS documentation describes how to do so.
It's kind of a pain to build from source. Is it possible to avoid doing that?
If we assume that you are trying to install on an x86 computer (Intel, AMD, whatever), can you just install Python on another x86 computer where you do have root, then make a tar archive of the Python installation, copy the tar to the other computer, and unpack the tar archive?
The problem with the above is that the pre-built Python might have hard-coded paths for where to look for libraries: it might need the libraries to be in /usr/share or whatever. It would be a bit of a hack, but you might be able to make a chroot jail and get Python to run.
You might also want to take a look at Enthought Python Distribution (EPD). I believe the EPD installer simply asks you where you want EPD installed, and installs it there.
http://www.enthought.com/products/epdgetstart.php?platform=linux
There is a free version of EPD. If you want 64-bit you would have to pay for EPD, but if 32-bit will work for you, EPD Free might be all you need.
http://www.enthought.com/products/epd_free.php
P.S. The Enthought web site seems to be rejecting any URL that doesn't start with www.! This means that some Google search links don't work unless you edit them to insert the www. at the beginning. I'm sure they will fix this soon.
You may want to look into EasyBuild for building your local Python version with numpy and scipy enabled, see http://hpcugent.github.com/easybuild/.
It basically takes all the nasty stuff away from you, you just need to configure it a little bit (specify where you want the software to end up, for exaple), and then you can build Python with the packages of your choice with a single command.