In this case, the application consists of one or more Python files, plus a settings.ini file. Now the Python files when being installed need to be installed in ~/.hg (as default) or prompted where the user want them installed. The installation also requires text to be appended to files like hgrc.
Is there already a specific Python package that does all of this, or if anyone has any experience in this area please share.
As far as I have looked, Python packaging refers to setuptools and easy_install.
The basis for packaging is a setup.py file. A problem with this is that such a setup file is used for a couple of dissimilar tasks:
Generating documentation.
Creating a release (source/binary).
Actually installing the software.
Combining these tasks in one file is a bit of a hazard and leads to problems now and then.
or distutils, but I am not sure if these packages support the notion of user prompting and deployment like appending text to existing files, and creating new ones.
I would include a custom script (bin/ command) which will poke the users' .hgrc and others. Doing it without the user consent would be rude.
User story
Install package: easy_install pkgname and this deploys myproject-init-hg (UNIX executable, can be also written in Python)
The installation finished and tells the user to run commmand myproject-init-hg
setup.py include mechanism to distribute and deploy bin/ style scripts.
Related
We've had the case where we released a Python source distribution (sdist) file on PyPI, and there was an issue that was only noticed after making the release. Namely for some files, the permissions were set to not allow read by all, and then in some cases those files couldn't be read on user machines after they installed the package with sudo.
We're considering starting our own script to check an sdist for this issue and to run it before uploading to PyPI.
Is there a tool (or something in setuptools or twine or somewhere) to run this or other basic checks on an sdist before uploading it available already?
This a question about setuptools and creating a python distribution for a web application.
Context
It is a django project and we distribute it via a private pypi, to ourselves, not external users (yet). Currently we run python setup.py sdist to create the distribution and we upload that to our pypi. I suspect it is just a copy of the source code as a tar.gz. We checkout the source to do development. We don't install for dev via pip (perhaps we should?).
Recently the project has started using nodejs. Which now means that now we need to do a "build" to create new files which are not part of the source code, but do need to be deployed. The build process requires a bunch of node packages which are not a necessary, or desired, part of the final deployment to a webserver.
Reading through the packaging documentation it describes sdist, bdist_wheel, and also develop targets for setup.py, but I'm not clear on how to apply these to our situation.
Question 1
When we pull from our pypi, what should we be pulling?
the pre-built deployable version of code for the webserver, without source code?, or
code on which we run something else which builds and arranges things as they need to be? If so, what would the appropriate setup.py target be (sdist, bdist_wheel, a new custom target)?
Question 1 (put another way)
When, and how, should we run the build?
To 'build' the app, we run nodejs-javascript scripts eg: npm install, and less x y x, or webpack -p, or python, scripts which convert the source files into deployable files. We don't want these things on the web server.
Discussion
So, generally speaking, as you can tell, first we're a bit confused about when to convert the source files into distribution files. And second, how to actually implement that: subprocess calls to shell scripts from a subclassed setuptools.command?!?
We'd prefer not to push the finished product to pypi because then all the bits and bobs we do to create the build are not captured and automated in a neat setuptools centric way. So when, where, and how do we incorporate the build process into our setuptools distribution?
* Note: adding more confusion about what/where we're also using tox
Java has the concept of packaging all of the code into a file called a Jar file. Does Python have an equivalent idea? If so, what is it? How do I package the files?
Python doesn't have any exact equivalent to a .jar file.
There are many differences, and without knowing exactly what you want to do, it's hard to explain how to do it. But the Python Packaging User Guide does a pretty good job of explaining just about everything relevant.
Here are some of the major differences.
A .jar file is a compiled collection of classes that can be dropped into your application, or installed anywhere on your CLASSPATH.
In Python:
A .py (or .pyc) module can be dropped into your application, or installed anywhere on your sys.path, and it can be imported and used.
A directory full of modules can be treated the same way; it becomes a package (or, if it doesn't contain an __init__.py, it merges with other directories of the same name elsewhere on sys.path into a single package).
A .zip archive containing any number of modules and packages can be stored anywhere, and its path added to your sys.path (e.g., at runtime or via PYTHONPATH) and all of its contents become importable.
Most commonly, you want things to be installed into a system, user, or virtualenv site-packages directory. The recommended way to do that is to create a pip-compatible package distribution; people then install it (and possibly automatically download it from PyPI or a private repo) via pip.
pip does a lot more than that, however. It also allows you to manage dependencies between packages. So ideally, instead of listing a bunch of prereqs that someone has to go download and install manually, you just make them dependencies, and someone just has to pip install your-library. And it keeps track of the state of your site-packages, so you can uninstall or upgrade a package without having to track down the specific files.
Meanwhile, in Java, most .jar files are cross-platform; build once, run anywhere. A few packages have JNI native code and can't be used this way, but it's not the norm.
In Python, many packages have C extensions that have to be compiled for each platform, and even pure-Python packages often need to do some install-time configuration. And meanwhile, "compiling" pure Python doesn't do anything that can't be done just as well at runtime. So in Python, you generally distribute source packages, not compiled packages.
However, .wheel is a binary package format. You can pip wheel to build binary packages for different targets from the source package; then, if someone tries to pip install your package, if there's a wheel for his system, that will be downloaded and installed.
Easy Install from setup_tools defines the .egg format for deploying Python libraries or applications. While similar to JAR, it is nowhere spread as universally as JARs in Java world. Many people just deploy the .py files.
A newer format, intended to supersede eggs, is wheel.
Though it's not a perfect susbstitute of jar due to portability issues, I would add the "auto-extracting" archive way.
One possibility is "makeself": https://makeself.io/
But if you don't need to package external files, and if you like KISS approach, the following is a nice and clean alternative:
The following is taken from Asim Jalis's website.
How to deploy a Python application as a zip file
Create a file __main__.py containing:
print "Hello world from Python"
Zip up the Python files (in this case just this one file) into app.zip by typing:
zip app.zip *
The next step adds a shebang to the zip file and saves it as app—at this point the file app is a zip file containing all your Python sources.
echo '#!/usr/bin/env python' | cat - app.zip > app
chmod 755 app
That’s it. The file app is now have a zipped Python application that is ready to deploy as a single file.
You can run app either using a Python interpreter as:
python app
Or you can run it directly from the command line:
./app
Reference: https://gist.github.com/asimjalis/4237534
I have a simple script that has a dependency on dnspython for parsing zone files. I would like to distribute this script as a single .py that users can run just so long as they have 2.6/2.7 installed. I don't want to have the user install dependencies site-wide as there might be conflicts with existing packages/versions, nor do I want them to muck around with virtualenv. I was wondering if there was a way to embed a package like dnspython inside the script (gzip/base64) and have that script access that package at runtime. Perhaps unpack it into a dir in /tmp and add that to sys.path? I'm not concerned about startup overhead, I just want a single .py w/ all dependencies included that I can distribute.
Also, there would be no C dependencies to build, only pure python packages.
Edit: The script doesn't have to be a .py. Just so long as it is a single executable file.
You can package multiple Python files up into a .egg. Egg files are essentially just zip archives with well defined metadata - look at the setuptools documentation to see how to do this. Per the docs you can make egg files directly executable by specifying the entry point. This would give you a single executable file that can contain your code + any other dependencies.
EDIT: Nowadays I would recommend building a pex to do this. pex is basically an executable zip file with non stdlib dependencies. It doesn't contain a python distribution (like py2app/py2exe) but holds everything else and can be built with a single command line invocation. https://pex.readthedocs.org/en/latest/
The simplest way is just to put your python script named __main__.py with pure Python dependencies in a zip archive, example.
Otherwise PyInstaller could be used to produce a stand-alone executable.
please don't do this. If you do DO NOT make a habit of it.
pydns is BDS licensed but if you try to "embed" a gpl module in this way you could get in trouble
you can learn to use setuptools and you will be much happier in the long run
setuptools will handle the install of dependencies you identified (I'm not sure if the pydns you are using is pure python so you might create problems for your users if you try to add it yourself without knowing their environment)
you can set a url or pypi so that people could upgrade your script with easy_install -U
In Trac on Admin -> Plugins there is an option to install Plug-ins. Now this option expect you to upload an Python egg.
This would be all well but for the fact that all the Trac plug-ins I found are either plain .py files or zip files and are incompatible with the upload function (I tried it).
This leaves my with a bunch of questions:
Are there any Trac plug-ins which come as an Python egg?
What is an (Trac compatible) Python egg?
Is it difficult to repackage an .py file into a Trac compatible Python egg?
If not: how is it done?
Haven't used trac for a year, but what I remember is that most plugins are available trough subversion and already packed as an egg (which is kind of an installer in the python world, but I am not very familiar with the concept).
Most plugins are available at http://trac-hacks.org/ and the easiest way to install a plugin is
easy_install http://svn.domain.tdl/path/to/plugin/
the folder should contain a setup.py and a setup.cfg file.
easy_install checks the files out from svn and installs the plugin. You can find details here: http://trac.edgewall.org/wiki/TracPlugins
If the plugin makes database changes you have to call
trac-admin upgrade
from console.
http://trac.edgewall.org/wiki/TracAdmin
If I remember right, the install through the webinterface installs the plugin locally (for the instance) while easy_install installs it globally (for all running trac sites) and is the more common way to install a plugin.
Hint: After every plugin install you have to restart trac
Hint2: Most plugins don't tell you how to install and only give a link to the root of their svn. You only have to browse the svn folder and locate the folder containing the setup.py.
The rest is done with easy_install.
Example:
Plugin: http://trac-hacks.org/wiki/GoogleChartPlugin
Wiki pages tells you:
You can check out GoogleChartPlugin from here using Subversion, or browse the source with Trac.
where here links to http://trac-hacks.org/svn/googlechartplugin/
The svn contains two versions. Browse to http://trac-hacks.org/svn/googlechartplugin/0.11/trunk/ and copy the path.
Then do
easy_install http://trac-hacks.org/svn/googlechartplugin/0.11/trunk/
Answers to your questions in order.
Python eggs are binary packages which contain the code for the application and some metadata. They're not very different from debs or rpms in this sense. The egg itself is basically just a zip file which contains all the above mentioned files with specific names and layouts. For more information on eggs (the format and how to create them), please refer to http://www.ibm.com/developerworks/library/l-cppeak3.html. It's probably a little dated since the future (and present) of python packaging is a little hazy.
A trac plugin is a python program that uses the Trac plugin API to extend the functionality of trac. It can be packaged as an egg.
If your package is properly laid out and contains a setuptools/distribute setup.py file, then issuing the command python setup.py bdist_egg will create a .egg file for you. For details on this please refer to this(a little dated but complete) and this (more upto date but still in progress). The Trac Growl plugin mentions this on it's documentation page.
Please see above point.