Currently we keep project related python packages in a subversion directory, so when someone adds or removes one it will directly be available to others.
Still, this method works well with Python packages that are not platform dependent.
Still I do have quite a few that are platform dependent and worse, when you install them using easy_install they will need a compiler to produce the .egg file.
I had to mention that the package maintainers are not providing binaries for these modules, so I need to compile it manually. I've tried to add the .egg file to the shared directory but python doesn't pick it up by default.
While in the entire team only a few have compilers, how can we share the packages in an easy way?
To make the problem even more complex, I had to specify that even if in 99% of the code is run on the same platform (Windows) with the same version of Python (2.5), we still have few scripts that are to be executed on another platform (Linux).
Related
I am currently writting a C++ application. I want to use a library that is exclusively written in Python and that uses numpy. I know that there is an extensive documentation on how to embed Python in a C++ program. My problem comes from the fact that with this method, there needs to be a Python installation on the machine where the app will be installed. However, I want to be able to distribute the program on machine where the user does not have all the rights.
If I am not mistaken, Blender is doing something very similar to what I want to achieve. Basically, when you download a portable version of Blender, you will find a structure like this:
blender/
2.90/
datafiles/
python/
bin/ <-- Contains the Python executable and Python DLL
DLLs/ <-- Contains the base packages and DLLs of a standard Python installation
(such as libssl, asyncio, ...)
lib/
[...] <-- Different python packages
site-packages/ <-- Additional packages such as numpy
scripts/ <-- Contains the python scripts relative to Blender
Basically, Blender ships with its own Python installation with the correct packages. I tried to look at the Blender source code to understand how they did it, but I was quickly overwhelmed by the complexity of the code. Still, I managed to find some clues in the blender/source/creator CMakeLists.txt.
I am using CMake (3.10) for my build system and using MinGW for the compilation. I plan to support Linux sometime in the future, but I first want to concentrate on Windows. Here is a little overview on my what my objectives are:
I want that CMake automatically creates a structure like the one shown above, with the desired packages. For me it is ok if this kind of structure is created in the ${CMAKE_BINARY_DIR}. Bonus point if it is possible to automatically download locally Python and install the packages (to avoid messing with the default Python installation on the system of the developper).
I need to tell my C++ application to use the correct Python environment (the one that will be created in 1.)
P.S.
I am also considering using pybind11, which has a embedding part. But the problems are more or less the same as above.
We develop a system that includes a lot of different Python scripts and modules. Some are deployed to a self-hosted PyPI and some are simple packaged into a zip file (stored in a self-hosted Artifactory). Finally, there is another application (not developed by us), that uses some of our Python scripts as plugins. So, the dependency graph is rather complex for a python environment, I guess. The following snippet should explain the graph:
Script (own, zip package)
Module (own, pypi)
Module (external, pypi)
Module (own, pypi)
Module (external, pypi)
This is just an example, in reality, there are much more dependencies. But in the end, it is a mix of zip packaged and pypi packaged Pyhton scripts and modules. The dependencies of the pypi modules are managed via setuptools install_requires parameter in the setup.py. But the dependencies of the zip packaged scripts are managed via a self-implemented configuration and install script.
At the end, we have our install script that creates one virtual environment and installs all dependencies in it. Either via pip or simple download the zip files and put it in the right directory. But honestly, that feels a little bit weird and we are not sure if this is the right (pythonic) way to go.
We already searched couple of days through the internet but found no answer. It also seems, that it is very uncommon to have such a complex system implemented in Python? So, the final question is: is our approach the right one or is there not really "the right way"? Or is our approach completely wrong?
I am new to Python having come from a proprietary compiled language (Xojo) that produces self-contained executables.
I understand that Python is an interpreted language. I understand that it requires an interpreter (let’s stick with CPython) and presumably it requires a number of accessory frameworks/C libraries in order to run. What I don’t understand is why is it so hard to create a folder containing the interpreter and all required files and libraries and simply bundle these up with my script to distribute.
I have discovered that there are a bunch of tools that attempt to do this (py2app, cx_freeze, etc) but many of them seem either broken, not maintained or really buggy.
I guess my question is: is there any documentation that describes the exact things I need to bundle with a “Hello World” script to get it running? This seems to be a really straightforward problem to solve but it hasn’t been (which suggests that it is far more complex than I appreciate).
My understanding is that PyInstaller works fine for making a single exe for distribution. But barring packaging tools like that, in general, there isn't an obvious "bare minimum"; the modules don't have documented dependencies, so it's usually best to ship the whole standard library.
Typically, if you need a redistributable version, you use the embedded Python zip redistributable, shipping Python alongside your main application.
The exact list of files/libraries depends on how the python interpreter is built. In windows for example, you can obtain CPython binaries built from Visual Studio, Cygwin and Mingw-w64. They have different dependency of cause. In Linux distributions, python is normally installed by default.
Below is the list of .dll and .exe files that you can find in the official CPython release for windows.
libcrypto-1_1-x64.dll python.exe python37.dll sqlite3.dll
libssl-1_1-x64.dll pythonw.exe python3.dll vcruntime140.dll
The total size of this ZIP file release is only 6.7 MB. So it would be easy to bundle it in your main executable. You can use whatever bundler at hand, not necessary those designed for python. Quoting from the documentation here:
extracting the embedded distribution to a subdirectory of the application installation is sufficient to provide a loadable Python interpreter.
I feel the absolute best way to experience Python for beginners in thonny and an esp32.
A very good way to get started with python is to use Anaconda https://www.anaconda.com/distribution/#download-section - this distribution contains the CPython interpreter and the most commonly used packages. For quite a while you will get along without installing more packages.
To be able to make a simple distributable piece of code just include a requirements.txt along with your code which should list down the packages (and versions) you are using in your code.
More on that here : https://www.idiotinside.com/2015/05/10/python-auto-generate-requirements-txt/
pip freeze generates a superset of all packages in your running environment so you would ideally go with the second smarter option in the link : pipreqs
So, in short along with your code just an additional requirements.txt should be fine using which people can install all required packages as
pip install -r requirements.txt
and they are good to go to run your code.
For advanced scenarios you might want to look up creating virtual environments using conda.
What is a conda environment?
https://docs.conda.io/projects/conda/en/latest/user-guide/concepts.html#conda-environments
How to create/manage a conda environment
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
All the best in your Python journey!
I thought it is an easy question but I spent a lot of google time to find the answer with no luck. Hope you can help me.
My company has a large SW system on windows which is portable, meaning copy some folders, add some folder to windows path and you are ready to go.
No registry, no dll in system directory, no shortcuts, Nothing!
I want to start using python 3.x in our system in the same paradigm. I also want the ability to add to this distribution a pip/conda 3rd packages from time to time.
I don't want to install python msi on all the systems.
I don't want to pack it to standalone executable like py2exe and pyinstaller or use special python distribution like PyWin32.
Somehow, I couldn't find a formal official solution for that.
The closest thing was here but no pip is supported, python is minimal, and the system isolation is "almost".
3.8. Embedded Distribution New in version 3.5.
The embedded distribution is a ZIP file containing a minimal Python
environment. It is intended for acting as part of another application,
rather than being directly accessed by end-users.
When extracted, the embedded distribution is (almost) fully isolated
from the user’s system, including environment variables, system
registry settings, and installed packages. The standard library is
included as pre-compiled and optimized .pyc files in a ZIP, and
python3.dll, python36.dll, python.exe and pythonw.exe are all
provided. Tcl/tk (including all dependants, such as Idle), pip and the
Python documentation are not included.
Note The embedded distribution does not include the Microsoft C
Runtime and it is the responsibility of the application installer to
provide this. The runtime may have already been installed on a user’s
system previously or automatically via Windows Update, and can be
detected by finding ucrtbase.dll in the system directory. Third-party
packages should be installed by the application installer alongside
the embedded distribution. Using pip to manage dependencies as for a
regular Python installation is not supported with this distribution,
though with some care it may be possible to include and use pip for
automatic updates. In general, third-party packages should be treated
as part of the application (“vendoring”) so that the developer can
ensure compatibility with newer versions before providing updates to
users.
Any ideas?
Thanks.
How about... installing Python in one machine and replicate that installation on others computers?
Usually, I install Python in a Windows Virtualbox machine (Microsoft usually give it for free to try it or for testing old Internet Explorer versions).
Then I copy the Python directory to my Windows machine (the real host) and usually works. This makes possible to using various python versions.
Did you try to complete the Python Embedded Distribution? Usually they not come with Tkinter, but once I could copy files and put in this distribution in a way that works. Try it too.
You can install pip with get-pip.py
We have a common python installation for all of our systems in order to ensure every system has the same python installation and to ease configuration issues. This installation is located on a shared drive. We also have multiple platforms that share this installation. We get around conflicting platform-specific files by setting the --exec-prefix configure option when compiling python.
My issue is that I now want to install an egg using easy_install (or otherwise) that is platform-dependent. easy_install puts the egg in the site-packages directory of the platform-independent part of the install. The name of the egg has the platform in it so there should be no conflict. But python will only load the first one it finds. (So, on Solaris it might try to load the Linux egg). Modifying the easy-install.pth file can change which one it finds, but that's pretty useless.
I can move the .egg files into a platform-depended packages directory and then use pkg_resources.require() to load them (or manually adjust the path). But it seems as though I shouldn't have to since the platform is in the name of the egg.
Is there any more generic way I can ensure that python will load the egg for the correct platform?
Try virtualenv ... http://pypi.python.org/pypi/virtualenv ... helps you create isolated environment with it's own python interpreter + site_packages folder. Thus you never have any conflicts with packages installed in say local path.
What I ended up going with was manually moving the platform-dependent egg to the platform-specific site-packages directory (as specified at http://docs.python.org/install/index.html). Then I made another easy-install.pth in that same directory, listing the eggs to be installed.
This would be much more convenient if easy_install were to honour the exec_prefix and put platform-dependent eggs in the correct "non-pure module distribution" location. Perhaps I will request this from the easy_install folks.
Python has enough hooks to make this possible but it is probably not wise. If you really want to implement this, look at PEP 302 and start hacking on the easy_install or distribute source code. As you know, there's not really any logic in easy-install.pth.
You would probably be much better off simply keeping track of which Python packages you want installed. You could for example write your own Python package that just depends on the desired set.
Use "easy_install -m" to install all the platform-specific packages, so that there is no default version on sys.path. That way, version resolution takes place at runtime, and platform information will be taken into consideration.