I have a simple script that has a dependency on dnspython for parsing zone files. I would like to distribute this script as a single .py that users can run just so long as they have 2.6/2.7 installed. I don't want to have the user install dependencies site-wide as there might be conflicts with existing packages/versions, nor do I want them to muck around with virtualenv. I was wondering if there was a way to embed a package like dnspython inside the script (gzip/base64) and have that script access that package at runtime. Perhaps unpack it into a dir in /tmp and add that to sys.path? I'm not concerned about startup overhead, I just want a single .py w/ all dependencies included that I can distribute.
Also, there would be no C dependencies to build, only pure python packages.
Edit: The script doesn't have to be a .py. Just so long as it is a single executable file.
You can package multiple Python files up into a .egg. Egg files are essentially just zip archives with well defined metadata - look at the setuptools documentation to see how to do this. Per the docs you can make egg files directly executable by specifying the entry point. This would give you a single executable file that can contain your code + any other dependencies.
EDIT: Nowadays I would recommend building a pex to do this. pex is basically an executable zip file with non stdlib dependencies. It doesn't contain a python distribution (like py2app/py2exe) but holds everything else and can be built with a single command line invocation. https://pex.readthedocs.org/en/latest/
The simplest way is just to put your python script named __main__.py with pure Python dependencies in a zip archive, example.
Otherwise PyInstaller could be used to produce a stand-alone executable.
please don't do this. If you do DO NOT make a habit of it.
pydns is BDS licensed but if you try to "embed" a gpl module in this way you could get in trouble
you can learn to use setuptools and you will be much happier in the long run
setuptools will handle the install of dependencies you identified (I'm not sure if the pydns you are using is pure python so you might create problems for your users if you try to add it yourself without knowing their environment)
you can set a url or pypi so that people could upgrade your script with easy_install -U
Related
We develop a system that includes a lot of different Python scripts and modules. Some are deployed to a self-hosted PyPI and some are simple packaged into a zip file (stored in a self-hosted Artifactory). Finally, there is another application (not developed by us), that uses some of our Python scripts as plugins. So, the dependency graph is rather complex for a python environment, I guess. The following snippet should explain the graph:
Script (own, zip package)
Module (own, pypi)
Module (external, pypi)
Module (own, pypi)
Module (external, pypi)
This is just an example, in reality, there are much more dependencies. But in the end, it is a mix of zip packaged and pypi packaged Pyhton scripts and modules. The dependencies of the pypi modules are managed via setuptools install_requires parameter in the setup.py. But the dependencies of the zip packaged scripts are managed via a self-implemented configuration and install script.
At the end, we have our install script that creates one virtual environment and installs all dependencies in it. Either via pip or simple download the zip files and put it in the right directory. But honestly, that feels a little bit weird and we are not sure if this is the right (pythonic) way to go.
We already searched couple of days through the internet but found no answer. It also seems, that it is very uncommon to have such a complex system implemented in Python? So, the final question is: is our approach the right one or is there not really "the right way"? Or is our approach completely wrong?
Java has the concept of packaging all of the code into a file called a Jar file. Does Python have an equivalent idea? If so, what is it? How do I package the files?
Python doesn't have any exact equivalent to a .jar file.
There are many differences, and without knowing exactly what you want to do, it's hard to explain how to do it. But the Python Packaging User Guide does a pretty good job of explaining just about everything relevant.
Here are some of the major differences.
A .jar file is a compiled collection of classes that can be dropped into your application, or installed anywhere on your CLASSPATH.
In Python:
A .py (or .pyc) module can be dropped into your application, or installed anywhere on your sys.path, and it can be imported and used.
A directory full of modules can be treated the same way; it becomes a package (or, if it doesn't contain an __init__.py, it merges with other directories of the same name elsewhere on sys.path into a single package).
A .zip archive containing any number of modules and packages can be stored anywhere, and its path added to your sys.path (e.g., at runtime or via PYTHONPATH) and all of its contents become importable.
Most commonly, you want things to be installed into a system, user, or virtualenv site-packages directory. The recommended way to do that is to create a pip-compatible package distribution; people then install it (and possibly automatically download it from PyPI or a private repo) via pip.
pip does a lot more than that, however. It also allows you to manage dependencies between packages. So ideally, instead of listing a bunch of prereqs that someone has to go download and install manually, you just make them dependencies, and someone just has to pip install your-library. And it keeps track of the state of your site-packages, so you can uninstall or upgrade a package without having to track down the specific files.
Meanwhile, in Java, most .jar files are cross-platform; build once, run anywhere. A few packages have JNI native code and can't be used this way, but it's not the norm.
In Python, many packages have C extensions that have to be compiled for each platform, and even pure-Python packages often need to do some install-time configuration. And meanwhile, "compiling" pure Python doesn't do anything that can't be done just as well at runtime. So in Python, you generally distribute source packages, not compiled packages.
However, .wheel is a binary package format. You can pip wheel to build binary packages for different targets from the source package; then, if someone tries to pip install your package, if there's a wheel for his system, that will be downloaded and installed.
Easy Install from setup_tools defines the .egg format for deploying Python libraries or applications. While similar to JAR, it is nowhere spread as universally as JARs in Java world. Many people just deploy the .py files.
A newer format, intended to supersede eggs, is wheel.
Though it's not a perfect susbstitute of jar due to portability issues, I would add the "auto-extracting" archive way.
One possibility is "makeself": https://makeself.io/
But if you don't need to package external files, and if you like KISS approach, the following is a nice and clean alternative:
The following is taken from Asim Jalis's website.
How to deploy a Python application as a zip file
Create a file __main__.py containing:
print "Hello world from Python"
Zip up the Python files (in this case just this one file) into app.zip by typing:
zip app.zip *
The next step adds a shebang to the zip file and saves it as app—at this point the file app is a zip file containing all your Python sources.
echo '#!/usr/bin/env python' | cat - app.zip > app
chmod 755 app
That’s it. The file app is now have a zipped Python application that is ready to deploy as a single file.
You can run app either using a Python interpreter as:
python app
Or you can run it directly from the command line:
./app
Reference: https://gist.github.com/asimjalis/4237534
I have a simple python shell script (no gui) who uses a couple of dependencies (requests and BeautifulfSoup4).
I would like to share this simple script over multiple computers. Each computer has already python installed and they are all Linux powered.
At this moment, on my development environments, the application runs inside a virtualenv with all its dependencies.
Is there any way to share this application with all the dependencies without the needing of installing them with pip?
I would like to just run python myapp.py to run it.
You will need to either create a single-file executable, using something like bbfreeze or pyinstaller or bundle your dependencies (assuming they're pure-python) into a .zip file and then source it as your PYTHONPATH (ex: PYTHONPATH=deps.zip python myapp.py).
The much better solution would be to create a setup.py file and use pip. Your setup.py file can create dependency links to files or repos if you don't want those machines to have access to the outside world. See this related issue.
As long as you make the virtualenv relocatable (use the --relocatable option on it in its original place), you can literally just copy the whole virtualenv over. If you create it with --copy-only (you'll need to patch the bug in virtualenv), then you shouldn't even need to have python installed elsewhere on the target machines.
Alternatively, look at http://guide.python-distribute.org/ and learn how to create an egg or wheel. An egg can then be run directly by python.
I haven't tested your particular case, but you can find source code (either mirrored or original) on a site like github.
For example, for BeautifulSoup, you can find the code here.
You can put the code into the same folder (probably a rename is a good idea, so as to not call an existing package). Just note that you won't get any updates.
I am thinking of a good way to ship my application which is a python package. Installing my package is easy making use of pythons distutils package.
The trouble comes with the dependencies my package relies on. If the dependencies are python packages I can deal with them easily again using distutils, but non python packages? Some of them even need a lot of care while building and installing them since very special compiler flags need to be set and so forth...
If I want to automate the installation procedure for the user what is the best way to go about it?
Writing a make file that downloads and installs the dependencies
Write a script that installs the dependencies
No automation is best. simply write a manual that tells the user how to install the
dependencies
???
Thx in advance for any answer or suggestion
We have a project named Kivy ( http://kivy.org/ ), that have exactly the same issue. At the early stage, we've done a all-in-one package that include every setup of every dependencies. However the user was having a lot of "Next >" button to click... for every deps (Windows). So now, we have managed to take care ourself of the dependencies.
Except linux related (since all our deps are already packaged on "linux"), we have taken the approach of managing what we named "portable-deps" zipfile for each platform. And then, we have a script that:
Download the portable-deps zip
Include the latest version of our project
Add launcher script in the root directory
Zip the root directory, and rename the zip to project--.zip
With a special case for MacOSX, where the zip is a dmg with a little UI.
The good part is that the user don't have to care about deps, and developers know exactly what binaries are delivered with the project :)
For information, we have build_portable commands for distutils :
https://github.com/kivy/kivy/blob/master/kivy/tools/packaging/osx/build.py
https://github.com/kivy/kivy/blob/master/kivy/tools/packaging/win32/build.py
The most important thing to help you decide is to consider your audience.
Are they technically-inclined and likely to be comfortable following instructions specifying how to build the dependencies themselves? If so, go with (3). If not, writing a python or shell script, or a makefile to automate the task may be the way to go. Pick whichever you feel most comfortable writing.
I'm new to python and I'm writing my first program. I would like after I finish to be able to run the program from the source code on a windows or mac machine. My program has dependencies on 3rd party modules.
I read about virtualenv but I don't think it helps me because it says it's not relocatable and it's not cross-platform (see Making Environments Relocatable http://pypi.python.org/pypi/virtualenv).
The best scenario is to install the 3rd party modules locally in my project, aka xcopy installation.
I will be really surprised if python doesn't support this easily especially since it promotes simplicity and frictionless programming.
You can do what you want, you just have to make sure that the directory containing your third-party modules is on the python path.
There's no requirement to install modules system-wide.
Note, while packaging your whole app with py2exe may not be an option, you can use it to make a simple launcher environment. You make a script with imports your module/package/whatever and launches the main() entry-point. Package this with py2exe but keep your application code outside this, as python code or an egg. I do something similar where I read a .pth text file to learn what paths to add to the sys.path in order to import my application code.
Simply, that's generally not how python works. Modules are installed site-wide and used that way. Are you familiar with pip and/or easy_install? Those + pypi let you automatically install dependencies no matter what you need.
If you want to create a standalone executable typically you'd use py2exe, py2app or something like that. Then you would have no dependencies on python at all.
I also found about zc.buildout that can be used to include dependencies in an automatic way.