I have written several small scripts in python and build the binaries using pyinstaller.
When I would build them on my Ubuntu 16.04 machine -- they would run fine on the machine where I build them. But moving the file to a Centos / Redhat 7.4 machine would give me GCLIB and other .so version dependency errors.
Building the same binary on docker with the same version of Centos would not give these errors.
If I try running the binary compiled on Centos 7.4 on Centos 6.6 I would get errors again -- But building on Centos 6.6 it would work correctly with Centos 6.6
I have solved the problem using a lower version of Centos to build my binaries for now.
My specific question is -- In Python, is it a common approach of building the binaries on different OSs based on the target OS that it is intended for (assuming linux targets only) or what I am doing is a hack / bad way of solving this problem?
I am trying to understand how this problem is approached in the standard way.
As long as the binary produced by pyinstaller only depends on glibc, then it should be a valid approach to build it on the oldest system available, and it should work on future systems.
In general, glibc is designed to be backwards compatible, so that applications built against an older version of glibc will still run with a newer glibc, but not vice versa. It does this via symbol versioning, in which each symbol you link to can have a version associated with it, and any case in which a newer glibc has changed the ABI of some function, it will also have a compatibility routine with the old ABI exposed with the older symbol version, so that applications linked against the old one will dynamically link with the compatibility routine, while if you have an application linked against the newer symbol versions, there won't be the newer versions in an older glibc to dynamically link to.
While other libraries can also do this, not many library authors bother to, so newer versions may simply be incompatible, while the glibc developers generally try to preserve compatibility.
So yes, as long as the final binary links only to glibc, or to other libraries which follow a similar symbol versioning scheme to ensure that older binaries will still link properly to newer versions of the library, it is perfectly valid to build against an older version and then run it on newer versions of various Linux distros, and even generally across distros as well.
Unfortunately, there's no good way to get the linker to pick the older symbol versions if linking against a newer glibc, so frequently the easiest way to do this is within a Docker or other type of container containing an older distro that has the oldest glibc that you want to be compatible with.
Related
I'm trying to get the HDBSCAN package to run on a Windows 7 machine with no C++ compiler. Installing a compiler is not an option, unfortunately.
I read that some packages have precompiled wheel files that require no compiler to install. The installation notes state "Binary wheels for a number of platforms are available thanks to the work of Ryan Helinski". However, there is no mention of where those can be found. My questions are then:
How do I obtain the .whl file for the HDBSCAN package?
Is it possible to simply compile+install on another machine and copy it? If so, what should the machine on which I compile have in common with the one on which the code must run? Can I do it on a Windows 10 machine, or does it have to be Win7 as well, do the same Windows updates need to have been run on both, etc?
Looks like there is no pre-built wheel distribution published by the maintainers of the project themselves on PyPI.
As already mentioned by other contributors, one could get such wheel distributions from a third party such as Christoph Gohlke's "Unofficial Windows Binaries for Python Extension Packages".
It is also of course possible to build such wheels yourself on one machine and then reuse it on another. As far as I know in the case of Windows, both machines need to have the same Python interpreter (minor) version as well as the same bitness (both 32 bits or both 64 bits). The exact version of the operating system should not matter (from Windows 10 to Windows 7 or the other way around should work).
I ended up getting this to work by compiling on another machine and copying the package from it. It was critical that the required packages were of the same version on both machines, so I simply set up a new conda environment which had the same package versions as the target machine, then intalled hdbscan with pip there, and copied.
I was worried about Windows version compatibility, but this worked even though I installed on Windows 10 and moved to a machine running Win7.
here is the site you can download the wheel file :https://www.lfd.uci.edu/~gohlke/pythonlibs/
for python 3.5>
or PyPi for older versions
run python -m pip install thefiledownloadedforyourpythonversion.whl
For the second question , yes you can but is rather complicated and you should avoid it when you can :)
I just checked. Python 3.8 is coded in C. You need a C compiler, not a C++ one. MINGW is one, and TinyCC Win32 is another one (a small one, producing quickly slow executables). Look also into this list of free C or C++ compilers.
A possibility (which could take several days of work) might be to use some WSL or some Linux emulator such as JSLinux (it runs in a web browser). Then you could (painfully) build a cross compiler (starting first with a tinycc-win32, then compiling an old cross GCC 3 compiler, then compiling with that a newer C++ GCC, etc....)
But the reality is that you should not be allowed to do this. Your real issue is non technical: why are you not allowed to install then use a C++ compiler, such as MinGW? Get permission (and resources) to install some....
Alternatively, consider installing some Linux distribution (ensure you are allowed to do so). Most of them have a recent Python and GCC...
Is it possible to simply compile+install on another machine and copy it?
This is called cross-compilation and is possible in general. The point is to be permitted to do that. You should find relevant cross-compilers for your situation.
If you are allowed to, you could even use a live Linux USB stick....
Some C compilers written in Python do exist.... You could use them to cross-compile tinycc for Win32. Then you should have a C compiler for Win32. You could then compile an old GCC, etc.... Qemu exist for windows. You could run a Linux with a cross-GCC compiler in Qemu.
I am using Xcode to build a PyObjC application. The app runs fine on the build machine (running 10.8) but crashes on startup on a machine running 10.6, because it fails to find the Python 2.7 installation. Fair enough -- the preinstalled Python on 10.6 is Python 2.5. But I don't really care which Python version my app uses, I just want it to use the latest version of Python it can find.
How can I either:
A) Tell my app to use the latest version of Python available on the host system, OR
B) Bundle the entire Python source into my app?
I have been very frustrated by this issue and any help would be greatly appreciated!
Good question. I wrestled with this for an embedding problem a few years ago and ended up dynamically linking against a specific version that I knew was available on every platform.
I had assumed originally that the Python.framework's Current would be adequate, but it seems to get resolved at link time by Xcode, therefore making the link specific to a particular version. For my purposes, I was able to link directly to
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/Python
Which works for OS X 10.7 and 10.8. If you need to go back to 10.6, you'll want to try linking to
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/Python.framework/Versions/2.5/Python
Obviously, if you're compiling against the 10.6 SDK, you'll want to use that, but the key is to make sure that you link to a version-specific version of the libraries and not the framework version itself. You should be able to manually navigate to the specific version through the + button in the Link Binary with Libraries, but if you have problems, you can open the Python.framework from the SDK directly in the Finder and drag the version-specific library into the Link Binary with Libraries group.
If you find a way around this, then please post another answer here, but this was effective for me to link for embedding into my app.
Use py2app to create the application bundle, and do that using a separate install of Python (that is don't use /System/Library/Framework/Python.framework). The python install you use should be compiled with the MACOSX_DEPLOYMENT_TARGET set to the minimum OSX release you want to support.
When you do this, it should be possible to deploy to older OSX releases. I regularly do this for building apps on a 10.8 machine that get deployed to a 10.5 machine.
You do need to take some care when including other libraries, especially when those include a configure script: sometimes the configure script detects functionality that is available on the build machine, but not on the deployment machine.
BTW. You need to link against the same version of Python as you use at runtime. CPython's ABI is not compatible between feature releases (that is, the 2.6 ABI is not necessarily compatible with the 2.7 ABI). For python 3.x there is a stable ABI that is compatible between feature releases, but AFAIK that's primarily targeting Python extensions and I don't know how useful that is for embedding Python in your application.
I'm looking at two different guides for Windows HgWeb installation on IIS. The first from mercurial developers says:
On Windows, your Python version must
match the version used to compile
Mercurial. Otherwise, you'll get
"Invalid Magic Number" errors when
trying to run the CGI.
The second guide says:
Don’t waste time monkeying around with
the installers. You need a version of
Mercurial matching your Python install
so get the source code!
This is all great, except nobody wants to tell you how to match versions of Python and Mercurial. I see no guide that gives a map correlating version numbers. That is my primary need. For example, I'm using Mercurial 1.9. I have no idea what version of Python goes with that. The second guide says my version of PyWin32 must match as well. Great.
If the version-match info is not available online, is there a command I can issue to Hg that will dump all this info on me?
In the mean time, I've found that I can't build the Mercurial source code (is this because I don't have a properly matching version of Python?).
HgWeb
For HgWeb, this Mercurial download page recommends using the version that installs Mercurial as python modules, requiring Python 2.6.
From that, you would select a build of PyWin32 and choose the download for Python 2.6, as they build it for several versions.
Both decisions also have 32bit and 64bit versions.
Building
The main Mercurial download page hints that Python 2.4 through 2.7 are supported, so you might try building with 2.6 or 2.7 if you still wish to build it yourself. The wiki page for building on Windows is dated, mentioning Mercurial 1.3, but also mentions Python 2.6.
I've been using MacPorts Python 2.6 on OS X 10.6.
I am considering also installing the Enthought Python Distribution (EPD) on the same machine because it comes preconfigured with matplotlib and other nice data analysis and visualization packages.
Can the two Python distributions co-exist peacefully on the same machine?
What potential problems will I have to look out for (e.g., environment variables)?
I know that building matplotlib through MacPorts is an option, but the process is lengthy (on the order of a full day) and there are open questions about compiling some dependencies on 64bit Intel.
I would like to know about the tradeoffs before committing to one of the two approaches.
I'm not an OSX user, so I can't say for certain, but you should be fine.
EPD bundles its own python interpreter and all required libraries. The installer basically just extracts a tarball into a stand-alone folder.
The linux installer doesn't modify any environment variables (not even $PATH), so I assume the mac installer is similar.
(Because of this, you'll probably want to manually set up aliases to the EPD python/ipython/etc binaries so you can more easily call them without changing your $PATH.)
EPD really should be completely stand-alone, as far as I know. It should be safe to install along-side anything else.
I've used EPD on Windows 7. It works great. Only stalled on me a couple of times with rather large data set.
I also use OSX, and was about to install EPD but hesitating a bit.
I think python 2.7.3 that comes with EPD (early 2013) will become the default python. Normally, upgrading to 2.7 from 2.6 should not be much of problem based on my personal experience. But I am a bit concerned to have 2 versions of 2.7 and what kind of impact it will have on using GAE SDK on my OSX. (Upgrading created a bit of hassle with GAE SKD.)
Here is what EPD readme says:
A full installation will install the following:
* Python framework -- installed into:
/Library/Frameworks/Python.framework/Versions/7.3
Python applications -- installed into
/Applications/Enthought
System fixes -- makes the EPD Python your default one by adding lines to
your .bash_profile to your path.
I think I will sleep on this.
Question
What are the differences between a Framework build and a non-Framework build (i.e., standard UNIX build) of Python on Mac OS X? Also, what are the advantages and disadvantages of each?
Preliminary Research
Here is the information that I found prior to posting this question:
[Pythonmac-SIG] Why is Framework build of Python needed
B. Grainger: "I seem to recall that a Framework build of Python is needed if you want to do anything with the native Mac GUI. Is my understanding correct?"
C. Barker: "Pretty much -- to access the Mac GUI, an app needs to be in a proper Mac application bundle. The Framework build supplies that."
Apple Developer Connection: Framework Definition
"A framework is a bundle (a structured directory) that contains a dynamic shared library along with associated resources, such as nib files, image files, and header files. When you develop an application, your project links to one or more frameworks. For example, iPhone application projects link by default to the Foundation, UIKit, and Core Graphics frameworks. Your code accesses the capabilities of a framework through the application programming interface (API), which is published by the framework through its header files. Because the library is dynamically shared, multiple applications can access the framework code and resources simultaneously. The system loads the code and resources of a framework into memory, as needed, and shares the one copy of a resource among all applications."
Framework Programming Guide: What are Frameworks?
"Frameworks offer the following advantages over static-linked libraries and other types of dynamic shared libraries:
Frameworks group related, but separate, resources together. This grouping makes it easier to install, uninstall, and locate those resources.
Frameworks can include a wider variety of resource types than libraries. For example, a framework can include any relevant header files and documentation.
Multiple versions of a framework can be included in the same bundle. This makes it possible to be backward compatible with older programs.
Only one copy of a framework’s read-only resources reside physically in-memory at any given time, regardless of how many processes are using those resources. This sharing of resources reduces the memory footprint of the system and helps improve performance."
Background
Prior to Mac OS X 10.6 Snow Leopard, I hadn't thought much about this, as I simply would download and install the Python 2.6.2 Mac Installer Disk Image, which is a framework build, and go about my business using virtualenv, pip, etc. However, with the changes in Snow Leopard to 64-bit, gcc, etc., I've noticed some issues that have made me want to build/compile Python 2.6.2+ myself from source, which leads me to my question of the differences and advantages/disadvantages of building Python as a MacOSX|Darwin framework.
You've already listed all important advantages of making a framework (congratulations for excellent research and reporting thereof!); the only flip side is that it's harder to arrange to build one properly, but if you take your clues from the examples in the installer you quote, it should be doable.
BTW, what's wrong with the system Python that comes with Snow Leopard? I haven't upgraded from Leopard yet (long story... I do have the "family license" upgrade DVD, but need Snow Leopard to fix some things before I can upgrade), so I have no first-hand experience with that yet, but I do know it's a 2.6 build and it comes in both 32-bit and 64-bit versions... so why do you need to build your own framework?
There is another difference: typically the Framework installation provided by the installer from python.org has several architectures.
$ file libpython2.7.dylib
libpython2.7.dylib: Mach-O universal binary with 2 architectures
libpython2.7.dylib (for architecture i386): Mach-O dynamically linked shared library i386
libpython2.7.dylib (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
If you install from source and you do not deliberately change this, your libpython has only one architecture.
I have had cases where the two architectures actually resulted in problems (at least I believe that this was the reason), namely when installing the HDF5 python bindings (h5py).
And there is yet another difference: some tools require the framework installation. For instance PyQt, and in particular sip. While it is possible to install sip and PyQt even for the non-framework version of python, it is much more complicated.
As for the decision what to prefer, I still do not know. At the moment, I went for the non-framework option, but I must say, that it also caused me some headache.
If you are going to ship your code (have it running on another machine), you'd better use the system version of python otherwise your program behavior will be undefined on the other machines.
I use Macports on 10.6, which makes it very simple to install multiple versions of python and switch between them and Apple's version:
sudo port install python26
sudo port install python_select
sudo python_select -l
The most recent version of python26 is 2.6.2, and compiles and runs fine on 10.6.1:
trac.macports.org/browser/trunk/dports/lang/python26/Portfile
Framework builds are owned by the 'root' account when installed. A source build will be owned by the account installing it. The advantage (and disadvantage) of having ownership of the Python installation is that you don't need to change accounts to modify it.
A small difference is that Framework builds are built against the EditLine library. Source builds are usually compiled against the Readline library. Depending upon which library Python is compiled against, the readline module in the standard library works slightly differently. See 'man python' on Mac OS X for more details on this.
There is a nice buildout for automating the compile of Python 2.4, 2.5 and 2.6 from source on Mac OS X, which is explained here. This will compile against a custom build of readline. However, the usefulness of scripting the source install is that you can make additional tweaks to your custom Python builds, e.g. installing essential distributions such as virtualenv, or harder to install distributions such as PIL.