With the 3.5.0 release, Python.org has introduced a distribution billed as embeddable zip file.
Unfortunately the zipped file comes without a help file (not even a readme). The download page on Python.org just lists it among the downloads.
Apparently this is a portable Python distribution. It is anyway quite different in structure and size from the standard distribution using the installer.
I realised that it is possible to install pip with get-pip.py and, thanks to pip, it is a breeze to add many other application packages, though I am still unable to add Tkinter (adjust slashes according to your shell):
curl https://www.python.org/ftp/python/3.x.x/python-3.x.x-embed-amd64.zip > epython.zip
unzip -o epython.zip -d env1
curl -L https://bootstrap.pypa.io/get-pip.py>env1/get-pip.py
env1/python env1/get-pip.py
Add what you need, e.g django:
env1/python -m pip install django
Given the size (6.5 Mega for the 3.5.1-x64), I think that it can be convenient as a means to create isolated environments.
In fact the general Python documentation says that
the embedded distribution is (almost) fully isolated from the user’s system, including environment variables, system registry settings, and installed package
Given this, in Windows there are now two isolated Python environments, the second being the standard
Virtualenv. The same process in Virtualenv is like follows:
virtualenv env2
and for django it would be:
env2/Scripts/python -m pip install django
Comparing the contents of env1 and env2, they appear to have the same files. The only significant difference is Tkinter1, which is anyway not much significant for desktop apps.
Which is the difference between Python Virtualenv and Python embeddable?
Specifically, which is the difference between the isolated web app created with the embeddable zip (env1) and Virtualenv (env2)?
As you can see from the documentation, it is mainly meant for running Python based applications on ms-windows and for embedding Python in an application. As you can see, they left out tkinter. Maybe to keep the size down?
Comparing it to a virtualenv doesn't make much sense, I think. They have completely different use cases.
In the ms-windows world, applications are generally distributed as monolithic independant entities. In contrast, basically every UNIX flavor has a working package management system which makes it easier to have packages that depend on others. So if you install a python-based app in UNIX, the package management system will basically install Python for you if it isn't installed yet. On ms-windows this doesn't work. Several Python distributions for ms-windows have sprung up because (for technical reasons) compiling and setting up stuff on ms-windows is painful [1] compared to UNIX. So having an embeddable Python could make sense for people who want to distribute Python-based programs or who want to embed Python into their application.
In general though I recommend that ms-windows users install either Canopy or Anaconda because they come with most of the external modules that you'll be likely to need.
Edit As of 2020, the python.org distribution has come a long way; you don't need a special compiler for it anymore, and more and more modules distribute precompiled binaries for ms-windows on PyPI. So my recommendation for ms-windows users has changed: use the python.org releases of Python.
Related
I wish to place a python program on GitHub and have other people download and run it on their computers with assorted operating systems. I am relatively new to python but have used it enough to have noticed that getting the assorted versions of all the included modules to work together can be problematic. I just discovered the use of requirements.txt (generated with pipreqs and deployed with the command pip install -r /path/to/requirements.txt) but was very surprised to notice that requirements.txt does not actually state what version of python is being used so obviously it is not the complete solution on its own. So my question is: what set of specifications/files/something-else is needed to ensure that someone downloading my project will actually be able to run it with the fewest possible problems.
EDIT: My plan was to be guided by whichever answer got the most upvotes. But so far, after 4 answers and 127 views, not a single answer has even one upvote. If some of the answers are no good, it would be useful to see some comments as to why they are no good.
Have you considered setting up a setup.py file? It's a handy way of bundling all of your... well setup into a single location. So all your user has to do is A) clone your repo and B) run pip install . to run the setup.py
There's a great stack discussion about this.
As well as a handle example written by the requests guy.
This should cover most use cases. Now if you want to make it truly distributable then you'll want to look into setting it up in PyPi, the official distribution hub.
Beyond that if you're asking how to make a program "OS independent" there isn't a one size fits all. It depends on what you are doing with your code. Requires researching how your particular code interacts with those OS's etc.
There are many, many, many, many, many, many, many ways to do this. I'll skate over the principles behind each, and it's use case.
1. A python environment
There are many ways to do this. pipenv, conda, requirments.txt, etc etc.
With some of these, you can specify python versions. With others, just specify a range of python versions you know it works with - for example, if you're using python 3.7, it's unlikely not to support 3.6; there's only one or two minor changes. 3.8 should work as well.
Another similar method is setup.py. These are generally used to distribute libraries - like PyInstaller (another solution I'll mention below), or numpy, or wxPython, or PyQt5 etc - for import/command line use. The python packaging guide is quite useful, and there are loads of tutorials out there. (google python setup.py tutorial) You can also specify requirements in these files.
2. A container
Docker is the big one. If you haven't heard of it, I'll be surprised. A quick google of a summary comes up with this, which I'll quote part of:
So why does everyone love containers and Docker? James Bottomley, formerly Parallels' CTO of server virtualization and a leading Linux kernel developer, explained VM hypervisors, such as Hyper-V, KVM, and Xen, all are "based on emulating virtual hardware. That means they're fat in terms of system requirements."
Containers, however, use shared operating systems. This means they are much more efficient than hypervisors in system resource terms. Instead of virtualizing hardware, containers rest on top of a single Linux instance. This means you can "leave behind the useless 99.9 percent VM junk, leaving you with a small, neat capsule containing your application,"
That should summarise it for you. (Note you don't need a specific OS for containers.)
3. An executable file
There are 2 main tools that do this at the time of writing. PyInstaller, and cx_Freeze. Both are actively developed. Both are open source.
You take your script, and the tool compiles it to bytecode, finds the imports, copies those, and creates a portable python environment that runs your script on the target system without the end user needing python.
Personally, I prefer PyInstaller - I'm one of the developers. PyInstaller provides all of its functionality through a command line script, and supports most libraries that you can think of - and is extendable to support more. cx_Freeze requires a setup script.
Both tools support windows, Linux, macOS, and more. PyInstaller can create single file exes, or a one folder bundle, whereas cx_Freeze only supports one folder bundles. PyInstaller 3.6 supports python 2.7, and 3.5-3.7 - but 4.0 won't support python 2. cx_Freeze has dropped python 2 support as of the last major release (6.0 I think).
Anyway, enough about the tools features; you can look into those yourself. (See https://pyinstaller.org and https://cx-freeze.readthedocs.io for more info)
When using this distribution method, you usually provide source code on the GitHub repo, a couple of exes (one for each platform) ready for download, and instructions on how to build the code into an executable file.
The best tool I have used so far for this is Pipenv. Not only it unifies and simplifies the whole pip+virtualenv workflow for you, developer, but it also guarantees that the exact versions of all dependencies (including Python itself) are met when other people run your project with it.
The project website does a pretty good job at explaining how to use the tool, but, for completeness sake, I'll give a short explanation here.
Once you have Pipenv installed (for instance, by running pip install --user pipenv), you can go to the directory of your project and run pipenv --python 3.7, so Pipenv will create a new virtualenv for your project, create a Pipfile and a Pipfile.lock (more on them later). If you go ahead and run pipenv install -r requirements.txt it will install all your packages. Now you can do a pipenv shell to activate your new virtualenv, or a pipenv run your_main_file.py to simply run your project.
Now let's take a look at the contents of your Pipfile. It should be something resembling this:
[packages]
Django = "*"
djangorestframework = "*"
iso8601 = "*"
graypy = "*"
whitenoise = "*"
[requires]
python_version = "3.7"
This file has the human-readable specifications for the dependencies of your project (note that it specifies the Python version too). If your requirements.txt had pinned versions, your Pipfile could have them too, but you can safely wildcard them, because the exact versions are stored in the Pipfile.lock. Now you can run things like pipenv update to update your dependencies and don't forget to commit Pipfile and Pipfile.lock to your VCS.
Once people clone your project, all they have to do is run pipenv install and Pipenv will take care of the rest (it may even install the correct version of Python for them).
I hope this was useful. I'm not affiliated in any way with Pipenv, just wanted to share this awesome tool.
If your program is less about GUI, or has a web GUI, then you can share the code using Google Colaboratory.
https://colab.research.google.com/
Everyone can run it with the same environment. No need for installation.
In case converting all your python scripts into one executable can help you, then my answer below would help ...
I have been developing a large desktop application purely in python since 3 years. It is a GUI-based tool built on top of pyqt library (python-bindings of QT C++ framework).
I am currently using "py2exe" packaging library : is a distutils extension which allows to build standalone Windows executable programs (32-bit and 64-bit) from Python scripts; all you have to do is to:
install py2exe: 'pip install py2exe'
Create a setup.py script: It is used to specify the content of the final EXE (name, icon, author, data files, shared libraries, etc ..)
Execute: python setup.py py2exe
I am also using "Inno Setup" software to create installer: Creating shortcuts, setting environment variables, icons, etc ...
I'll give you a very brief summary of some of the existing available solutions when it comes to python packaging you may choose from (knowledge is power):
Follow the guidelines provided at Structuring Your Project, these conventions are widely accepted by python community and it's usually a good starting point when newcomers start coding in python. By following these guidelines pythonists watching your project/source at github or other similar places will know straightaway how to install it. Also, uploading your project to pypi as well as adding CI by following those rules will be painless.
Once your project is structured properly according to standard conventions, the next step might be using some of the available freezers, in case you'd like to ship to your end-users a package they can install without forcing them to have python installed on their machines. Be aware though these tools won't provide you any code protection... said otherwise, extracting the original python code from the final artifacts would be trivial in all cases
If you still want to ship your project to your users without forcing them to install any dev dependency and you do also care about code protection so you don't want to consider any of the existing freezers you might use tools such as nuitka, shedskin, cython or similar ones. Usually reversing code from the artifacts produced by these tools isn't trivial at all... Cracking protection on the other hand is a different matter and unless you don't provide a physical binary to your end-user you can't do much about it other than slowing them down :)
Also, in case you'd need to use external languages in your python project another classic link that comes to mind would be https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages, adding the build systems of such tools to CI by following rules of 1 would be pretty easy.
That said, I'd suggest stick to bulletpoint 1 as I know that will be more than good enough to get you started, also that particular point should cover many of the existing use-cases for python "standard" projects.
While this is not intended to be a full guide by following those you'll be able to publish your python project to the masses in no time.
I think you can use docker with your python https://github.com/celery/celery/tree/master/docker
kindly follow the files and I think you can figure out the way to make your docker file for your python scripts!
Because it is missing from the other answers, I would like to add one completely different aspect:
Unit testing. Or testing in general.
Usually, it is good to have one known good configuration. Depending on what the dependencies of the program are, you might have to test different combinations of packages. You can do that in an automated fashion with e.g. tox or as part of a CI/CD pipeline.
There is no general rule of what combination of packages should be tested, but usually python2/3 compatability is a major issue. If you have strong dependencies on packages with major version differences, you might want to consider testing against these different versions.
I am new to Python having come from a proprietary compiled language (Xojo) that produces self-contained executables.
I understand that Python is an interpreted language. I understand that it requires an interpreter (let’s stick with CPython) and presumably it requires a number of accessory frameworks/C libraries in order to run. What I don’t understand is why is it so hard to create a folder containing the interpreter and all required files and libraries and simply bundle these up with my script to distribute.
I have discovered that there are a bunch of tools that attempt to do this (py2app, cx_freeze, etc) but many of them seem either broken, not maintained or really buggy.
I guess my question is: is there any documentation that describes the exact things I need to bundle with a “Hello World” script to get it running? This seems to be a really straightforward problem to solve but it hasn’t been (which suggests that it is far more complex than I appreciate).
My understanding is that PyInstaller works fine for making a single exe for distribution. But barring packaging tools like that, in general, there isn't an obvious "bare minimum"; the modules don't have documented dependencies, so it's usually best to ship the whole standard library.
Typically, if you need a redistributable version, you use the embedded Python zip redistributable, shipping Python alongside your main application.
The exact list of files/libraries depends on how the python interpreter is built. In windows for example, you can obtain CPython binaries built from Visual Studio, Cygwin and Mingw-w64. They have different dependency of cause. In Linux distributions, python is normally installed by default.
Below is the list of .dll and .exe files that you can find in the official CPython release for windows.
libcrypto-1_1-x64.dll python.exe python37.dll sqlite3.dll
libssl-1_1-x64.dll pythonw.exe python3.dll vcruntime140.dll
The total size of this ZIP file release is only 6.7 MB. So it would be easy to bundle it in your main executable. You can use whatever bundler at hand, not necessary those designed for python. Quoting from the documentation here:
extracting the embedded distribution to a subdirectory of the application installation is sufficient to provide a loadable Python interpreter.
I feel the absolute best way to experience Python for beginners in thonny and an esp32.
A very good way to get started with python is to use Anaconda https://www.anaconda.com/distribution/#download-section - this distribution contains the CPython interpreter and the most commonly used packages. For quite a while you will get along without installing more packages.
To be able to make a simple distributable piece of code just include a requirements.txt along with your code which should list down the packages (and versions) you are using in your code.
More on that here : https://www.idiotinside.com/2015/05/10/python-auto-generate-requirements-txt/
pip freeze generates a superset of all packages in your running environment so you would ideally go with the second smarter option in the link : pipreqs
So, in short along with your code just an additional requirements.txt should be fine using which people can install all required packages as
pip install -r requirements.txt
and they are good to go to run your code.
For advanced scenarios you might want to look up creating virtual environments using conda.
What is a conda environment?
https://docs.conda.io/projects/conda/en/latest/user-guide/concepts.html#conda-environments
How to create/manage a conda environment
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
All the best in your Python journey!
I thought it is an easy question but I spent a lot of google time to find the answer with no luck. Hope you can help me.
My company has a large SW system on windows which is portable, meaning copy some folders, add some folder to windows path and you are ready to go.
No registry, no dll in system directory, no shortcuts, Nothing!
I want to start using python 3.x in our system in the same paradigm. I also want the ability to add to this distribution a pip/conda 3rd packages from time to time.
I don't want to install python msi on all the systems.
I don't want to pack it to standalone executable like py2exe and pyinstaller or use special python distribution like PyWin32.
Somehow, I couldn't find a formal official solution for that.
The closest thing was here but no pip is supported, python is minimal, and the system isolation is "almost".
3.8. Embedded Distribution New in version 3.5.
The embedded distribution is a ZIP file containing a minimal Python
environment. It is intended for acting as part of another application,
rather than being directly accessed by end-users.
When extracted, the embedded distribution is (almost) fully isolated
from the user’s system, including environment variables, system
registry settings, and installed packages. The standard library is
included as pre-compiled and optimized .pyc files in a ZIP, and
python3.dll, python36.dll, python.exe and pythonw.exe are all
provided. Tcl/tk (including all dependants, such as Idle), pip and the
Python documentation are not included.
Note The embedded distribution does not include the Microsoft C
Runtime and it is the responsibility of the application installer to
provide this. The runtime may have already been installed on a user’s
system previously or automatically via Windows Update, and can be
detected by finding ucrtbase.dll in the system directory. Third-party
packages should be installed by the application installer alongside
the embedded distribution. Using pip to manage dependencies as for a
regular Python installation is not supported with this distribution,
though with some care it may be possible to include and use pip for
automatic updates. In general, third-party packages should be treated
as part of the application (“vendoring”) so that the developer can
ensure compatibility with newer versions before providing updates to
users.
Any ideas?
Thanks.
How about... installing Python in one machine and replicate that installation on others computers?
Usually, I install Python in a Windows Virtualbox machine (Microsoft usually give it for free to try it or for testing old Internet Explorer versions).
Then I copy the Python directory to my Windows machine (the real host) and usually works. This makes possible to using various python versions.
Did you try to complete the Python Embedded Distribution? Usually they not come with Tkinter, but once I could copy files and put in this distribution in a way that works. Try it too.
You can install pip with get-pip.py
I am using PythonXY (2.7, 32-bit) and the official Python (2.7, 32-bit).
Normally it is recommended to install according to python version, example C:\python27. But since they are both python27, can I arbitrarily change the base name (example C:\pythonxy27)?
When using python extras like pylauncher, or when utilizing the setuptools user-site, will they automatically recognize my custom installation sites (they will easily differentiate C:\python27 and C:\python33), or will both installations compete for the python27 namespace. (specifically when installing 3rd party packages to user-site, which normally locates as such \APPDATA\Python\PythonVer)
As far as setuptools/distribute are concerned, the python installer will handle custom locations for you. As long as you don't move that directory, all should be fine.
As for Pylauncher:
Things are not quite so clean. Pylauncher has simple configuration/call-parameters (for shebang lines in particular), that can handle version/platform selection quite well (2.7 vs 3.3, and 32bit vs 64bit).
As for the scenario in question (two different deployments where both are based on 32bit Python 2.7), pylauncher will attempt to guess which installation you wanted. If it is picking the wrong installation, there is some debugging information you can review to tune pylauncher's selection.
If an environment variable PYLAUNCH_DEBUG is set (to any value), the
launcher will print diagnostic information
It does not seem like there is a portable way to configure this, and will have to be done per-system (once you have your installations configured, YOU CAN set an alias that will be recognized on the shebang line)
Virtualenv and friends
I have also found (after struggling with pylauncher focused solutions), virtualenv addresses many of the deployment isolation hurdles. At the time of posting, working with virtualenv was not nearly as intuitive (on Windows) as compared to a linux shell environment. But I have discovered support packages like virtualenvwrapper which handle a lot of the ugly batch file interfaces very nicely.
Final Notes
Originally, I was also handling python globally with admin accounts. Forcing myself to stay within my user home directory (C:/Users/username), utilizing python user-site configurations, and making optimum use of ipython: have all given me a much better interactive command-line experience.
Finally making a legitimate mac installer for my product. I've made a successful Windows installer with Inno installer. I'm not sure how to do this in Mac.
This must happen:
-Python is installed
-Wx is Installed
-Py Serial is installed
-Program is copied
-Shortcut is made.
I was doing this with Bash scripts before, but my customers having been complaining about those. Perhaps X-code package maker is the solution? I know the recommended method is "just copy files" but these libraries must be installed somehow.
Thanks in advance!
Unless I am using Fink for installing packages, I usually just download the .tar.gz file from the source and install it from terminal inside the unzipped folder containing the install.py file. Terminal command:
sudo python ./setup.py install
If you would like, I can show you how to set up and use Fink, which is another easy way to install packages / libraries.
tl;dr: py2app will make a self-contained application bundle out of your Python scripts, making it real easy to employ the 'just copy files' installation process. The libraries you need are bundled into the app bundle itself, so they don't need to be installed systemwide.
Also check out Optimizing for Mac OS X from the wxPython wiki; it gives good tips on using py2app and other useful information on building a Mac-friendly Python application.
On OS X, programs are generally installed through one of three ways: the Mac App Store, a package installer (.pkg/.mpkg), or a copyable application bundle on a disk image (.app in a .dmg). Each has its strengths and weaknesses.
The Mac App Store requires that you subscribe to Apple's restrictions and requirements, and may be a good choice for apps expecting wider distribution (though, nowadays, it can be a good way to reach that wider audience easily). Copyable application bundles are by far the simplest installation method pre-App Store, and still remain one of the more popular ways to install programs. Finally, an Installer package is a user-friendly way to install more complex programs requiring more than a simple application bundle (e.g. system services, files in particular directories, system-dependent components, advanced installation logic, etc.). I should note, though, that do exist complex applications which ship as application bundles and perform the bulk of their 'installation' at first run.
My experience with the Mac App Store is limited, so I won't talk about it. You can find more details at the official website.
Python is quite amenable to being shipped as an application bundle. You can use py2app to automatically create an application bundle for the program, and then drop that bundle into a Mac disk image (.dmg) using Disk Utility to create a complete installation package. This doesn't support making shortcuts, but on OS X it is much more usual for users to just drop the app into /Applications and make the necessary dock shortcut themselves if they want.
The next way is to make a metapackage (.mpkg) which will be installed using the OS X standard Installer utility. This is in line with what users will expect from a Mac application. IIRC, both Mac Python and wxPython ship as .pkg already, which should make it easier to integrate them into a metapackage. bdist_mpkg can help with making packages for pyserial and your own program, which can be added to the metapackage. Finally, using the third-party dockutil script, you can automatically add a dock shortcut. Note, however, that installers generally do not add shortcuts to the dock; it is more usual to have a program installed to the /Applications directory.`