I am developing a proprietary Python library. The library is currently Windows-only, and I want to also make it available for other platforms (Linux & Mac).
The library is currently shipped to the (paying) customers in the form of a Zip file. This archive contains the compiled Python code for the implementation of my library, plus all dependencies. By placing the Zip file on his PYTHONPATH, the customer can use the library from his Python scripts.
Shipping a Zip file with all dependencies included has the following advantages:
No internet access or administrator privileges are required to install the library.
The customer does not have to worry about installing / managing dependencies of my library.
It also has the disadvantage that the customer is not (easily) able to use his own versions of my library's dependencies.
Even though I am not generating an EXE, I am using py2exe to obtain the distributable Zip file for my library. This "hack" is very convenient, as py2exe allows me to simply say which packages I require and does the work of performing a dependency analysis of the required libraries for me. py2exe automatically generates the Zip file with my (compiled) library code, and all dependencies.
Unfortunately, py2exe is only available for Windows. I need to also be able to build it on Linux & Mac, hence change the build process to not use py2exe.
My questions are:
Is it considered a bad idea in the Python community to ship one large Zip file with all dependencies? From what I have seen, it seems to be an unusual approach, at the least.
Is it possible to distribute the library in a form that allows for an offline installation without administrator privileges using other tools, such as setuptools?
My insight into the state of the art in Python regarding these matters is limited, so I would appreciate advice from someone with more experience in the subject.
A hard requirement is that I can only ship binary distributions of my library, as this is a proprietary product. I looked at Distutils and Setuptools, where the recommended approach seems to be to simply ship all sources.
Many thanks!
Related
I want to make a python script that I can turn into an .exe file and give to others in my company. How can I do this without requiring them to download the Anaconda Distribution, conda installing the correct libraries, etc? Does turning a file into an .exe file take care of this?
Technically it depends on how you turn it into an exe. Exe's can depend on DLLs that the others in your company don't have. However, if you are using a standard tool like https://pyinstaller.readthedocs.io/en/stable/operating-mode.html pyinstaller, there will be no outside dependencies and you can bundle your python scripts as exes and have no issues with your coworkers.
The existing tools I'm aware of (PyInstaller, py2exe, py2app and cx_Freeze) are all designed to encapsulate all dependencies into a single executable, with no external dependencies, so you should be able to distribute the (possibly quite large) executable without worrying about dependencies.
To be clear, there is nothing intrinsic to the .exe conversion that causes it to avoid dependencies. .exe files can depend on .dll library files, external data files, etc. It's just that most people who want to make such an executable are trying to remove dependencies on the installed Python entirely, not just third-party library dependencies, so the tooling tends to support this use case by avoiding any dependencies at all.
I have a python script that depends on pycrypto, boto3 and pyYaml. I have to deliver this script to a client who is strictly non-technical and maynot even have access to the internet.
Is there a way to package everything up together so that the client can just extract the zip file and have the script working for him out of the box? Also, the script should be able to run on Windows/Mac/Linux and they can each have their own .zip files as well.
I have worked with pip, requirements.txt and setuptools in the past but have never done something very rudimentary like this. What is the best way of achieving this?
If you are deploying on Windows, py2exe will help you build a stand-alone executable. Look for the -c and --bundle-files options, which together will bundle files into a compressed archive with your executable.
If deploying to Mac, try py2app instead.
In my experience, such bundlers are fussy, requiring at least some fiddling and experimentation. Some popular Python packages (e.g. PIL/Pillow, lxml, numpy, pandas, anything requiring a native C runtime) may have gotchas when mixed with bundler tools. So do not just bundle and deliver to a non-technical customer. Test and (if necessary) tweak before delivery.
This isn't a complete list of ways to install or deliver Python code. A few others from vinta's awesome python list.
I've built a few pieces of C++ software I want to release for Ubuntu. What ways are there and what can you recommend? Is building .deb files and setting up an apt repo for them the best way? What about make install, is it considered an acceptable way to install software?
By far simplest for me, and perhaps most transparent for the user, would be to just have a github repository in which one could run make install to get all programs installed at one go.
Do I always install the binaries into /usr/bin?
One of the programs contains Python 3 library code, should that be installed in /usr/lib/python3/dist-packages? (I don't want to create a pip package, that would make the installation harder -- and waste more of my time.) The program also contains Python 3 examples/tutorials intended for the user to tweak and learn from, where do I install those? Do I create a ~/my-prog-tutorial-dir/ to put them in? If so: how should I name that directory?
Edit: if I simply release the statically linked binaries in a tarball, what will break eventually? Libc? Are there any major application APIs that usually change between Ubuntu LTSs? I only use pthreads, X11 and OpenGL so I suspect statically linked binaries could be a fairly stable option?
In general, building a binary package will make your software much easier for your users to install and keep up to date. The same goes for python packages. There are generally tools to generate apt packages from pip packages, so you can just list your python code as a dependency of your binary package(s).
You may see packaging and installers as a waste of your time, but only providing a source distribution wastes your users' time. Users don't want to constantly have to check github for new versions, and they often don't want to have to install all of your build dependencies if they just want to use your software. If your software is targeted towards developers this may be less of an issue, but it's still extra work that your users have to go through.
As for examples, the general convention is to put those in /usr/share/doc/myprogram/samples or a samples directory in your python package.
I was asked to expand my comment in an answer, and so I do.
The project I was talking about is called Woodpecker hash Bruteforce, and I distribute it as plain archived executables for Mac OS, Windows and Linux.
Woodpecker hash Bruteforce has only two dependencies I have to care about (the users don't need to install anything): OpenSSL and Botan - libraries to do hashing. I've got two virtual machines on my Mac where I build the project and several scripts to automate the process. I'm using Docker (in collaboration with VirtualBox) and VMware Fusion.
Above I said the users don't need to worry about any third-party libraries because everything's linked statically with the executable: you just download the appropriate file from the official website, unarchive it (if needed), sudo chmod +x the executable and that's it!
This works on any version of Linux, including Ubuntu (this is where I perform the build) and Kali Linux.
The best way to release software for Ubuntu depends on the software itself and its target audience, as Miles Budnek already pointed out.
Your goal is to lower the barriers to software usage. If you are targeting developers of your software (i.e., you develop source files that are supposed to be edited by others) or you are developing piece of code supposed to be included in other projects (e.g., gnulib), probably it is best to just provide sources and documentation.
In any other case that I currently imagine (including when you are targeting developers), providing precompiled binaries is a better option. In this case the optimal solution would be to have the software in Ubuntu. In this case How to get my software into Ubuntu? provides a lot of useful information, as suggested by Mark K.
Getting software into Debian or Ubuntu can be difficult and may require a large amount of time (you have to respect a lot of policies that you may not be aware of and you have to find a sponsor) and you will soon learn that a key point is to use a decent and popular build system for your software (e.g., autotools, cmake, distutils, ...) as mentioned in the Debian Upstream Guide. Being compliant with the guide will also be beneficial for users of other distributions.
In general I suggest to proceed in this order:
provide sources;
use a common build system (from the point of view of system administrators, i.e., people installing the software, autotools works best in my experience for Posix systems);
create a binary package (please keep in mind that you have to maintain it or your users will likely encounter binary incompatibilities);
add the package in private repository (I suggest aptly for this task);
try to get the package in the distribution of choice (please keep in mind maintainance costs).
Another option that I do not suggest, is to provide statically linked builds. This reduces the possibility of binary incompatibilities, but augment the costs of bug fixing (e.g., if the bug is in a dependency) and security, as explained in this and following comments. Another reason to avoid static linking is if several implementations of the same ABI exists, in order to exploit hardware acceleration (e.g., OpenGL), but you can also mix static and dynamic linking.
Finally you may also provide a container, like docker, to ship your software and all its dependencies: your users will just need docker to run your application very portably. However it is probably overkill in most situations and if it is a practical solution or not depends on your application and target audience.
I'm working on porting a Python package over to Cython, but would rather provide as many pre-compiled binary packages of it as possible so that users don't need to have Cython on their system.
I've had no lucking finding one so it probably doesn't exist but is there a Cython package build service available somewhere? Basically, I want to be able to build for Windows, Linux, Mac, ARM in both x86 and x64 varieties which means I need to create at least 8 separate builds. I'd certainly be willing to either pay for or go through the hassle of setting up an automated build system that would do that for me, on demand.
Also, I don't currently own a Mac and would rather not have to buy one just for the sake of building this package.
I don't think this question is specific to Cython, but rather any Python module which has Extensions.
Several of the Continuous Integration services allow you to create build "artifacts". You can use this feature to automate your builds.
I've used Appveyor to do this for Windows (https://www.appveyor.com/) - see an example project here: https://ci.appveyor.com/project/snorfalorpagus/appveyordemo
I'm aware that Travis can do this for Linux and OS X, although I've not tried this myself.
I've found that using Anaconda greatly simplifies things (http://conda.pydata.org/miniconda.html).
I'm not aware of a single system that does everything but that may just be because I've never looked.
Currently we keep project related python packages in a subversion directory, so when someone adds or removes one it will directly be available to others.
Still, this method works well with Python packages that are not platform dependent.
Still I do have quite a few that are platform dependent and worse, when you install them using easy_install they will need a compiler to produce the .egg file.
I had to mention that the package maintainers are not providing binaries for these modules, so I need to compile it manually. I've tried to add the .egg file to the shared directory but python doesn't pick it up by default.
While in the entire team only a few have compilers, how can we share the packages in an easy way?
To make the problem even more complex, I had to specify that even if in 99% of the code is run on the same platform (Windows) with the same version of Python (2.5), we still have few scripts that are to be executed on another platform (Linux).