Can one use the same python package (wheel file) for Linux , windows etc.? I am asking this as some packages include not only python files but EXEs as well, which I assume are python code turned into exe (at least with pip.exe and Django admin tool). Exe files are platform specific in the same way there are separate python interpreters for windows and Linux so that arises a question.
Some wheel packages are cross-platform; some are platform-specific.
This information is included in the wheel's name. For example:
pytz-2018.4-py2.py3-none-any.whl (510kB)
That py2.py3 means that it works in any Python implementation, both Python 2.x and 3.x, and that none-any means that it works on any platform.
This one is more specific:
numpy-1.14.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
That cp36-cp36m means that it works only in CPython 3.6, and that macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64 means that it's built for x86_64 macOS versions 10.9-10.10. (Roughly speaking, that's the minimum and recommended versions of macOS; most other platforms aren't quite as complicated.)
The most common reason for a package to be platform-specific is that it includes C API extension modules, as is the case with numpy. But there can be other reasons. For example, it may include a native executable to subprocess, or it may use ctypes to access system APIs, etc.
A Python wheel is a packaging format, NOT an execution format. It's basically a .zip file.
Furthermore:
https://packaging.python.org/discussions/wheel-vs-egg/
...when the distribution only contains Python files (i.e. no compiled
extensions), and is compatible with Python 2 and 3, it’s possible for
a wheel to be “universal”, similar to an sdist.
From the same link:
A single wheel archive can indicate its compatibility with a number of
Python language versions and implementations, ABIs, and system
architectures.
In other words, the "wheel" format is designed to be as portable as possible ... and it also allows you to include platform-specific contents as required.
Related
I'm trying to understand Linux OS library dependencies to effectively run python 3.9 and imported pip packages to work. Is there a requirement for GCC to be installed for pip modules with c extention modules to run? What system libraries does Python's interpreter (CPython) depends on?
I'm trying to understand Linux OS library dependencies to effectively run python 3.9 and imported pip packages to work.
Your questions may have pretty broad answers and depend on a bunch of input factors you haven't mentioned.
Is there a requirement for GCC to be installed for pip modules with c extention modules to run?
It depends how the package is built and shipped. If it is available only as a source distribution (sdist), then yes. Obviously a compiler is needed to take the .c files and produce a laudable binary extension (ELF or DLL). Some packages ship binary distributions, where the publisher does the compilation for you. Of course this is more of a burden on the publisher, as they must support many possible target machines.
What system libraries does Python's interpreter depends on?
It depends on a number of things, including which interpreter (there are multiple!) and how it was built and packaged. Even constraining the discussion to CPython (the canonical interpreter), this may vary widely.
The simplest thing to do is whatever your Linux distro has decided for you; just apt install python3 or whatever, and don't think too hard about it. Most distros ship dynamically-linked packages; these will depend on a small number of "common" libraries (e.g. libc, libz, etc). Some distros will statically-link the Python library into the interpreter -- IOW the python3 executable will not depend on libpython3.so. Other distros will dynamically link against libpython.
What dependencies will external modules (e.g. from PyPI) have? Well that completely depends on the package in question!
Hopefully this helps you understand the limitations of your question. If you need more specific answers, you'll need to either do your own research, or provide a more specific question.
Python depends on compilers and a lot of other tools if you're going to compile the source (from the repository). This is from the offical repository, telling you what you need to compile it from source, check it out.
1.4. Install dependencies
This section explains how to install additional extensions (e.g. zlib) on Linux and macOs/OS X. On Windows, extensions are already included and built automatically.
1.4.1. Linux
For UNIX based systems, we try to use system libraries whenever available. This means optional components will only build if the relevant system headers are available. The best way to obtain the appropriate headers will vary by distribution, but the appropriate commands for some popular distributions are below.
However, if you just want to run python programs, all you need is the python binary (and the libraries your script wants to use). The binary is usually at /usr/bin/python3 or /usr/bin/python3.9
Python GitHub Repository
For individual packages, it depends on the package.
Further reading:
What is PIP?
Official: Managing application dependencies
Got a problem here, if I'm using cython in my package, the compiled .pyd file differents from different python version, for example, .pyd file compiled under python3.7 will not be recognized by python3.8 . If I'd like to release my package to pypi , let's say for example the version number be 1.0.0, how can I upload the package, let different version's python running the same command pip install package==1.0.0 and get its own version's compiled file separately?
Thanks.
Typically there are two kinds of archive formats that you should publish. They are called "distribution packages" and we have:
"source distributions" ("sdist" for short);
and "built distributions" (or "binary distributions", "bdist" for short).
Sdists should not contain any platform specific components, in your case should not contain the compiled Cython code.
And bdists on the other hand are meant to contain compiled code (compiled Cython files for example) and thus are allowed to be platform specific.
Nowadays the most common and the only recommended kind of bdist format is "wheel".
So you should distribute (publish on PyPI) the sdist of your project, and if possible try to build as many platform specific wheels as you can.
See for example the distribution packages for pandas v2.2.1, there is exactly one sdist and many different wheels that cover a wide range of:
Python interpreter versions
operating system
CPU bitness
etc.
Notice how all the file names are different. PyPI does not allow uploading files with the exact same name. pip (and other packaging-related tools) are able to interpret those file names and make educated guesses about what their contents are, and thus pick the right distributions to download and install.
To preface: my code works as I expect when compiling and running on Linux. However, this library needs to be compiled for use on a Windows machine. I looked in to a couple different options, and decided that using Cygwin to compile for Windows seemed to be the correct choice. I'm using a setup.py file with the distutils.core library and compiling using python setup.py install. When compiling on Windows in Cygwin, it fails to find pthread.h, arpa/inet.h, netinet/in.h, and sys/socket.h. I was under the impression that Cygwin came prepackaged with these headers, which is why I chose to use it. The alternative to Cygwin is putting preprocessor commands everywhere and using Windows specific libraries such as winsock2.h, which I want to avoid if at all possible. Is it possible to compile for Windows using Cygwin? If so, what have I done wrong to cause Cygwin to not recognize these headers?
You need to install the proper headers
$ cygcheck -p usr/include/pthread.h
Found 9 matches for usr/include/pthread.h
cygwin-devel-3.0.7-1 - cygwin-devel: Core development files
..
cygwin-devel-3.1.6-1 - cygwin-devel: Core development files
...
so install the cygwin-devel package
To check all the shared libraries needed by the built dll, you can use cygcheck
$ cygcheck /usr/lib/python3.8/site-packages/Cython/Compiler/FlowControl.cpython-38-x86_64-cygwin.dll
D:\cygwin64\lib\python3.8\site-packages\Cython\Compiler\FlowControl.cpython-38-x86_64-cygwin.
dll
D:\cygwin64\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
D:\cygwin64\bin\libpython3.8.dll
D:\cygwin64\bin\cygintl-8.dll
D:\cygwin64\bin\cygiconv-2.dll
D:\cygwin64\bin\cyggcc_s-seh-1.dll
As was built with Cygwin Python, you need also to transfer the cygwin python...
Most important, I think, is to follow the instructions in the Python help or on the Python doc web site for "Extending and Embedding the Python Interpreter" for the version you are building the extension for. For windows, the build instructions identify the build environment used to create the binary package that you download from python.org, usually something like VS2013 or VS2017. (As an aside, I think the Community editions have everything you need, and I don't think you actually have to use the Visual Studio GUI when you build using nmake from the CMD.EXE terminal.)
To build in Cygwin for use in a Windows version of Python, you may need to install and then use the x86_64-w64-mingw32-gcc, etc., cygwin packages to cross-compile non-cygwin (i.e. pure windows) executables and DLLs from Cygwin.
Binary extensions must be built using the source tree for a specific Python major.minor version, and bitness. For windows, you will need to build multiple versions of the extension, one for each major.minor, bitness version of Python that will import it, e.g. 3.6, 3.7, 3.8, 3.9, 32-bit, 64-bit. The extension code may not require changes between versions, but it still needs to be compiled with the right compiler and linked against exactly the same shared libraries (in this case .DLL files) as used by the Python executable. For instance, it must use exactly the same version of Microsoft's C run time library DLL as the Python executable does. This is a bit more sensitive and restrictive than on Linux, where you can rebuild the python executable and your extension with the same toolchain from your distro more easily.
Does python require microsoft visual c++ redistributable to run the code ?
I'm using pyinstaller to compile my .py code into exe. In some systems my exe is asking for microsoft visual c++ redistributable package to run.
Does pyinstaller includes microsoft visual c++ redistributable files while making exe ? If no, how can I include those files so that I don't need to install microsoft visual c++ redistributable package into other's system to run my software ?
What are the other alternatives to build a standalone software in python ? I'm reading to use other languages along with python.
I saw electron js and python can be used together to make desktop application. But how will I distribute that application as a standalone exe ?
Python itself does not depend on the presence of MSVC. You can download a portable Python package, and it will run wherever you copy it. Those are the embeddable ones from https://www.python.org/downloads/windows/
But, Python modules with native extension code inside can depend on MSVC on multiple levels:
if the native part comes in binary format (.pyd file on Windows), actually that is a .dll, and it may depend on other .dll-s, depending on how it has been built
if the native part comes as C/C++ source code, it will be built at installation time, typically via a "setup.py", and this procedure will need a C compiler installed on the system
PyInstaller is a different story. First of all, it has a documentation which you may want to read. For example the page What PyInstaller Does and How It Does It clearly gives a direct answer to at least one of your questions:
Bundling to One File
PyInstaller can bundle your script and all its dependencies into a single executable named myscript (myscript.exe in Windows).
There is also a list of packages with known compatibility and known compatibility issues: https://github.com/pyinstaller/pyinstaller/wiki/Supported-Packages, which you may find useful depending on what packages you need.
While it is not a duplicate, this question: How to package a linked DLL and a pyd file into one self contained pyd file? (and another one it links) may be interesting to read.
Of course Python requires MSVC Redistributable, any native Windows program using standard library functions requires it. Obviously, Python uses lots of them and should provide a consistent environment across all extension modules.
However, since Python 3.5 it is bundled with an installer, so there's no need to install it manually. Python installers prior to 3.5 don't include it and I wasn't able to find any clarifications whether it's downloaded during installation or not.
By default Python also enforces extension modules to be compiled with the same (or, since 3.5, compatible) version of MSVC as an interpreter itself. So except for some very rare cases extension modules will also use the same redistributable.
"Embeddable" Python releases referred by #tevemadar are NOT a "portable Python"! Here's what the documentation says about their usage:
It is intended for acting as part of another application, rather than being directly accessed by end-users.
Note: The embedded distribution does not include the Microsoft C Runtime and it is the responsibility of the application installer to provide this. The runtime may have already been installed on a user’s system previously or automatically via Windows Update, and can be detected by finding ucrtbase.dll in the system directory.
But you still don't need them if you use PyInstaller.
To check whether or not redistributable files are included in your .exe file you could probably open it with any archiver software and see it for yourself. My guess is that they can be included at least if Python is installed in a single user mode, as in such case they're installed in the Python directory as well.
Other than that, however, you should really ask your questions separately.
I thought it is an easy question but I spent a lot of google time to find the answer with no luck. Hope you can help me.
My company has a large SW system on windows which is portable, meaning copy some folders, add some folder to windows path and you are ready to go.
No registry, no dll in system directory, no shortcuts, Nothing!
I want to start using python 3.x in our system in the same paradigm. I also want the ability to add to this distribution a pip/conda 3rd packages from time to time.
I don't want to install python msi on all the systems.
I don't want to pack it to standalone executable like py2exe and pyinstaller or use special python distribution like PyWin32.
Somehow, I couldn't find a formal official solution for that.
The closest thing was here but no pip is supported, python is minimal, and the system isolation is "almost".
3.8. Embedded Distribution New in version 3.5.
The embedded distribution is a ZIP file containing a minimal Python
environment. It is intended for acting as part of another application,
rather than being directly accessed by end-users.
When extracted, the embedded distribution is (almost) fully isolated
from the user’s system, including environment variables, system
registry settings, and installed packages. The standard library is
included as pre-compiled and optimized .pyc files in a ZIP, and
python3.dll, python36.dll, python.exe and pythonw.exe are all
provided. Tcl/tk (including all dependants, such as Idle), pip and the
Python documentation are not included.
Note The embedded distribution does not include the Microsoft C
Runtime and it is the responsibility of the application installer to
provide this. The runtime may have already been installed on a user’s
system previously or automatically via Windows Update, and can be
detected by finding ucrtbase.dll in the system directory. Third-party
packages should be installed by the application installer alongside
the embedded distribution. Using pip to manage dependencies as for a
regular Python installation is not supported with this distribution,
though with some care it may be possible to include and use pip for
automatic updates. In general, third-party packages should be treated
as part of the application (“vendoring”) so that the developer can
ensure compatibility with newer versions before providing updates to
users.
Any ideas?
Thanks.
How about... installing Python in one machine and replicate that installation on others computers?
Usually, I install Python in a Windows Virtualbox machine (Microsoft usually give it for free to try it or for testing old Internet Explorer versions).
Then I copy the Python directory to my Windows machine (the real host) and usually works. This makes possible to using various python versions.
Did you try to complete the Python Embedded Distribution? Usually they not come with Tkinter, but once I could copy files and put in this distribution in a way that works. Try it too.
You can install pip with get-pip.py