I am currently writting a C++ application. I want to use a library that is exclusively written in Python and that uses numpy. I know that there is an extensive documentation on how to embed Python in a C++ program. My problem comes from the fact that with this method, there needs to be a Python installation on the machine where the app will be installed. However, I want to be able to distribute the program on machine where the user does not have all the rights.
If I am not mistaken, Blender is doing something very similar to what I want to achieve. Basically, when you download a portable version of Blender, you will find a structure like this:
blender/
2.90/
datafiles/
python/
bin/ <-- Contains the Python executable and Python DLL
DLLs/ <-- Contains the base packages and DLLs of a standard Python installation
(such as libssl, asyncio, ...)
lib/
[...] <-- Different python packages
site-packages/ <-- Additional packages such as numpy
scripts/ <-- Contains the python scripts relative to Blender
Basically, Blender ships with its own Python installation with the correct packages. I tried to look at the Blender source code to understand how they did it, but I was quickly overwhelmed by the complexity of the code. Still, I managed to find some clues in the blender/source/creator CMakeLists.txt.
I am using CMake (3.10) for my build system and using MinGW for the compilation. I plan to support Linux sometime in the future, but I first want to concentrate on Windows. Here is a little overview on my what my objectives are:
I want that CMake automatically creates a structure like the one shown above, with the desired packages. For me it is ok if this kind of structure is created in the ${CMAKE_BINARY_DIR}. Bonus point if it is possible to automatically download locally Python and install the packages (to avoid messing with the default Python installation on the system of the developper).
I need to tell my C++ application to use the correct Python environment (the one that will be created in 1.)
P.S.
I am also considering using pybind11, which has a embedding part. But the problems are more or less the same as above.
Related
I am new to Python having come from a proprietary compiled language (Xojo) that produces self-contained executables.
I understand that Python is an interpreted language. I understand that it requires an interpreter (let’s stick with CPython) and presumably it requires a number of accessory frameworks/C libraries in order to run. What I don’t understand is why is it so hard to create a folder containing the interpreter and all required files and libraries and simply bundle these up with my script to distribute.
I have discovered that there are a bunch of tools that attempt to do this (py2app, cx_freeze, etc) but many of them seem either broken, not maintained or really buggy.
I guess my question is: is there any documentation that describes the exact things I need to bundle with a “Hello World” script to get it running? This seems to be a really straightforward problem to solve but it hasn’t been (which suggests that it is far more complex than I appreciate).
My understanding is that PyInstaller works fine for making a single exe for distribution. But barring packaging tools like that, in general, there isn't an obvious "bare minimum"; the modules don't have documented dependencies, so it's usually best to ship the whole standard library.
Typically, if you need a redistributable version, you use the embedded Python zip redistributable, shipping Python alongside your main application.
The exact list of files/libraries depends on how the python interpreter is built. In windows for example, you can obtain CPython binaries built from Visual Studio, Cygwin and Mingw-w64. They have different dependency of cause. In Linux distributions, python is normally installed by default.
Below is the list of .dll and .exe files that you can find in the official CPython release for windows.
libcrypto-1_1-x64.dll python.exe python37.dll sqlite3.dll
libssl-1_1-x64.dll pythonw.exe python3.dll vcruntime140.dll
The total size of this ZIP file release is only 6.7 MB. So it would be easy to bundle it in your main executable. You can use whatever bundler at hand, not necessary those designed for python. Quoting from the documentation here:
extracting the embedded distribution to a subdirectory of the application installation is sufficient to provide a loadable Python interpreter.
I feel the absolute best way to experience Python for beginners in thonny and an esp32.
A very good way to get started with python is to use Anaconda https://www.anaconda.com/distribution/#download-section - this distribution contains the CPython interpreter and the most commonly used packages. For quite a while you will get along without installing more packages.
To be able to make a simple distributable piece of code just include a requirements.txt along with your code which should list down the packages (and versions) you are using in your code.
More on that here : https://www.idiotinside.com/2015/05/10/python-auto-generate-requirements-txt/
pip freeze generates a superset of all packages in your running environment so you would ideally go with the second smarter option in the link : pipreqs
So, in short along with your code just an additional requirements.txt should be fine using which people can install all required packages as
pip install -r requirements.txt
and they are good to go to run your code.
For advanced scenarios you might want to look up creating virtual environments using conda.
What is a conda environment?
https://docs.conda.io/projects/conda/en/latest/user-guide/concepts.html#conda-environments
How to create/manage a conda environment
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
All the best in your Python journey!
I have been developing a python script for a few days now, and wanted to send the final result to a friend of mine. Is there a way to get all software requirements to run my program?
The python part is relatively easy, as I can just look at my import list on my script. The problem is external software requirements.
For example, one of the python modules I'm using is Rasterio. I could not install this through conda or pip, until I installed Visual Studios C++ Community Version. I imagine there is some specific dll file that is being used. I can't really tell though.
I would really like to make a standalone setup.bat file, that installs all requirements and finally allows the user to simply run my program. I'm not 100% sure right now of those requirements though.
How can I get these?
You can explicitly check your application dependencies using Dependency Walker get it from here
Currently we keep project related python packages in a subversion directory, so when someone adds or removes one it will directly be available to others.
Still, this method works well with Python packages that are not platform dependent.
Still I do have quite a few that are platform dependent and worse, when you install them using easy_install they will need a compiler to produce the .egg file.
I had to mention that the package maintainers are not providing binaries for these modules, so I need to compile it manually. I've tried to add the .egg file to the shared directory but python doesn't pick it up by default.
While in the entire team only a few have compilers, how can we share the packages in an easy way?
To make the problem even more complex, I had to specify that even if in 99% of the code is run on the same platform (Windows) with the same version of Python (2.5), we still have few scripts that are to be executed on another platform (Linux).
I need to make some Python applications for a work project. The target platform is AIX 5.3.
My question is: What version of Python should I be using?
My requirements are:
The Python version must be easy to install on the target machines. Others will do that according to instructions that I write, so no compiling from source or anything like that.
The Python version must have ncurses or curses support (I'm making a form handler).
I've found two different precompiled versions of Python for AIX, but one (2.1.something) didn't include the curses module, and the other (2.3.4, RPM format) had prerequisites that I failed to fulfill).
Any help would be greatly appreciated.
Use the AS Package of Python 2.6.3.7 from Activestate. They have a binary package for AIX on their download site.
If you don't have an AIX machine to test it on, the install works the same way on Solaris or Linux, so you could write your documentation based on that. Basically, you ungzip the tarball file, use tar to unpack the archive, change directory to the unpacked folder, run a shell script to install it, tell the shell script what directory to place it in, and wait.
Normally this would be used to install into a user directory, without superuser permissions, but you could install it anywhere that you like. You might also need to edit the system profile in order to make sure that all users can find the Python binary.
I suggest the latest Python 2.6, because it has a lot of bugfixes, and there is now a critical mass of 3rd party libraries ported to it. Also, the standard library includes a lot of useful stuff that you used to have to collect separately. Curses is in the standard library of Python 2.6.
Make sure to avoid Python 3.1 since it has not yet matured enough and provides few benefits for most business applications development.
I'd compile it from source myself and tell them where to download it from in the instructions
We've used ActiveState's Python as well as Pware's compiled version. Both have worked well. For AS, we've used 2.5 and 2.6. For Pware, just 2.6. Both 2.5 and 2.6 from AS support curses on our machine.
I've compiled from source but usually wind up having trouble with with ctypes or SSL. Currently I have the Frankenstein option going of AS Python2.6 installed but I pulled out a couple of *.so files from Pware's. I'm using GCC since we've never ponied up for a compiler but depending on what you need from Python, it's definitely doable if I can do it.
I will mention that AS Python claims to be 100% compatible with standard Python and it has been for everything we've done so far (mostly web applications).
I have an open source project containing both Python and C code. I'm wondering that is there any use for distutils for me, because I'm planning to do a ubuntu/debian package. The C code is not something that I could or want to use as Python extension. C and Python programs communicate with TCP/IP through localhost.
So the bottom line here is that while I'm learning packaging, does the usage of distutils specific files only make me more confused since I can't use my C-code as Python extensions? Or should I divide my C and Python functionality to separate projects to be able to understand packaging concepts better?
distutils can be used to install end user programs, but it's most useful when using it for Python libraries, as it can create source packages and also install them in the correct place. For that I would say it's more or less required.
But for an end user Python program you can also use make or whatever you like and are used to, as you don't need to install any code in the Python site-packages directory, and you don't need to put your code onto PyPI and it doesn't need to be accessible from other Python-code.
I don't think distutils will be neither more or less complicated to use in installing an end-user program compared to other tools. All such install/packaging tools are hella-complex, as Cartman would have said.
Because it uses an unified python setup.py install command? distutils, or setuptools? Whatever, just use one of those.
For development, it's also really useful because you don't have to care where to find such and such dependency. As long as it's standard Python/basic system library stuff, setup.py should find it for you. With setup.py, you don't require anymore ./configure stuff or ugly autotools to create huge Makefiles. It just works (tm)