I am newbie.
I am buidling rpm package for my own app and decided to use distutils to do achieve it. I managed to create some substitue of %post by using advice from this website, which i really am thankfull for, but i am having problems with %postun.
Let me describe what i have done. In setup.py i run command that creates symbolic link which is needed to run application. It works good but problem is when i want to remove rpm, link stays there. So i figured that i should use %postun in spec file. My question is: is there a way to do this in setup.py or do i have to manually edit spec file?
Please advise or point me some manuals or anything.
Thank you
Yes, you can specify a post install script, all you need is to declare in the bdist_rpm in the options arg the file you want to use:
setup(
...
options = {'bdist_rpm':{'post_install' : 'post_install',
'post_uninstall' : 'post_uninstall'}},
...)
In the post_uninstall file, put he code you need to remove the link, somethink like:
rm -f /var/lib/mylink
Neither distutils nor setuptools have uninstall functionality.
At some point, the python community agreed that uninstall should be handled by the packaging system. In this case you want to use rpm, so there is probably a way inside of rpm system to remove packages, but you will not find that in distutils or setuptools.
# pycon2009, there was a presentation on distutils and setuptools. You can find all of the videos here
Eggs and Buildout Deployment in Python - Part 1
Eggs and Buildout Deployment in Python - Part 2
Eggs and Buildout Deployment in Python - Part 3
There is a video called How to Build Applications Linux Distributions will Package. I have not seen it, but it seems to be appropriate.
Related
I am wondering what my options are if I write a Python script that makes use of installed libraries on my computer (like lxml for example) and I want to deploy this script on another computer.
Of course having Python installed on the other machine is a given but do I also have to install all the libraries I use in my script or can I just use the *.pyc file?
Are there any options for making an installer for this kind of problem that copies all the dependencies along with the script in question?
I am talking about Windows machines by the way.
Edit -------------------
After looking through your answers i thought i should add this:
The thing is....the library required for this to work won't come along quietly with either pip or easy_install on account on it requiring either a windows installer (witch i found after some searching) or being rebuilt on the target computer from sources (witch i'm trying to avoid) .
I thought there was some way to port a pyc file or something to another pc and the interpreter on that station will not require the dependencies on account on it already being translated to bytecode.
If this is not possible, can anyone show me a guide of some sort for making windows package installers ?
You should create setup.py for use with setuptools. Dependencies should be included in the field install_requires. Afterwards if you install that package using easy_install or pip, dependencies will be automatically downloaded and installed.
You can also use distutils. A basic setup.py looks as follows:
from distutils.core import setup
setup(
name='My App',
version='1.0',
# ... snip ...
install_requires=[
"somedependency >= 1.2.3"
],
)
For differences between distutils and setuptools see this question.
I am thinking of a good way to ship my application which is a python package. Installing my package is easy making use of pythons distutils package.
The trouble comes with the dependencies my package relies on. If the dependencies are python packages I can deal with them easily again using distutils, but non python packages? Some of them even need a lot of care while building and installing them since very special compiler flags need to be set and so forth...
If I want to automate the installation procedure for the user what is the best way to go about it?
Writing a make file that downloads and installs the dependencies
Write a script that installs the dependencies
No automation is best. simply write a manual that tells the user how to install the
dependencies
???
Thx in advance for any answer or suggestion
We have a project named Kivy ( http://kivy.org/ ), that have exactly the same issue. At the early stage, we've done a all-in-one package that include every setup of every dependencies. However the user was having a lot of "Next >" button to click... for every deps (Windows). So now, we have managed to take care ourself of the dependencies.
Except linux related (since all our deps are already packaged on "linux"), we have taken the approach of managing what we named "portable-deps" zipfile for each platform. And then, we have a script that:
Download the portable-deps zip
Include the latest version of our project
Add launcher script in the root directory
Zip the root directory, and rename the zip to project--.zip
With a special case for MacOSX, where the zip is a dmg with a little UI.
The good part is that the user don't have to care about deps, and developers know exactly what binaries are delivered with the project :)
For information, we have build_portable commands for distutils :
https://github.com/kivy/kivy/blob/master/kivy/tools/packaging/osx/build.py
https://github.com/kivy/kivy/blob/master/kivy/tools/packaging/win32/build.py
The most important thing to help you decide is to consider your audience.
Are they technically-inclined and likely to be comfortable following instructions specifying how to build the dependencies themselves? If so, go with (3). If not, writing a python or shell script, or a makefile to automate the task may be the way to go. Pick whichever you feel most comfortable writing.
I had my fair chance of getting through the python management of modules, and every time is a challenge: packaging is not what people do every day, and it becomes a burden to learn, and a burden to remember, even when you actually do it, since this happens normally once.
I would like to collect here the definitive overview of how import, package management and distribution works in python, so that this question becomes the definitive explanation for all the magic that happens under the hood. Although I understand the broad level of the question, these things are so intertwined that any focused answer will not solve the main problem: understand how all works, what is outdated, what is current, what are just alternatives for the same task, what are the quirks.
The list of keywords to refer to is the following, but this is just a sample out of the bunch. There's a lot more and you are welcome to add additional details.
PyPI
setuptools / Distribute
distutils
eggs
egg-link
pip
zipimport
site.py
site-packages
.pth files
virtualenv
handling of compiled modules in eggs (with and without installation via easy_install)
use of get_data()
pypm
bento
PEP 376
the cheese shop
eggsecutable
Linking to other answers is probably a good idea. As I said, this question is for the high-level overview.
For the most part, this is an attempt to look at the packaging/distribution side, not the mechanics of import. Unfortunately, packaging is the place where Python provides way more than one way to do it. I'm just trying to get the ball rolling, hopefully others will help fill what I miss or point out mistakes.
First of all there's some messy terminology here. A directory containing an __init__.py file is a package. However, most of what we're talking about here are specific versions of packages published on PyPI, one of it's mirrors, or in a vendor specific package management system like Debian's Apt, Redhat's Yum, Fink, Macports, Homebrew, or ActiveState's pypm.
These published packages are what folks are trying to call "Distributions" going forward in an attempt to use "Package" only as the Python language construct. You can see some of that usage in PEP-376 PEP-376.
Now, your list of keywords relate to several different aspects of the Python Ecosystem:
Finding and publishing python distributions:
PyPI (aka the cheese shop)
PyPI Mirrors
Various package management tools / systems: apt, yum, fink, macports, homebrew
pypm (ActiveState's alternative to PyPI)
The above are all services that provide a place to publish Python distributions in various formats. Some, like PyPI mirrors and apt / yum repositories can be run on your local machine or within your companies network but folks typically use the official ones. Most, if not all provide a tool (or multiple tools in the case of PyPI) to help find and download distributions.
Libraries used to create and install distributions:
setuptools / Distribute
distutils
Distutils is the standard infrastructure on which Python packages are compiled and built into distributions. There's a ton of functionality in distutils but most folks just know:
from distutils.core import setup
setup(name='Distutils',
version='1.0',
description='Python Distribution Utilities',
author='Greg Ward',
author_email='gward#python.net',
url='http://www.python.org/sigs/distutils-sig/',
packages=['distutils', 'distutils.command'],
)
And to some extent that's a most of what you need. With the prior 9 lines of code you have enough information to install a pure Python package and also the minimal metadata required to publish that package a distribution on PyPI.
Setuptools provides the hooks necessary to support the Egg format and all of it's features and foibles. Distribute is an alternative to Setuptools that adds some features while trying to be mostly backwards compatible. I believe Distribute is going to be included in Python 3 as the successor to Distutil's from distutils.core import setup.
Both Setuptools and Distribute provide a custom version of the distutils setup command
that does useful things like support the Egg format.
Python Distribution Formats:
source
eggs
Distributions are typically provided either as source archives (tarball or zipfile). The standard way to install a source distribution is by downloading and uncompressing the archive and then running the setup.py file inside.
For example, the following will download, build, and install the Pygments syntax highlighting library:
curl -O -G http://pypi.python.org/packages/source/P/Pygments/Pygments-1.4.tar.gz
tar -zxvf Pygments-1.4.tar.gz
cd Pygments-1.4
python setup.py build
sudo python setup.py install
Alternatively you can download the Egg file and install it. Typically this is accomplished by using easy_install or pip:
sudo easy_install pygments
or
sudo pip install pygments
Eggs were inspired by Java's Jarfiles and they have quite a few features you should read about here
Python Package Formats:
uncompressed directories
zipimport (zip compressed directories)
A normal python package is just a directory containing an __init__.py file and an arbitrary number of additional modules or sub-packages. Python also has support for finding and loading source code within *.zip files as long as they are included on the PYTHONPATH (sys.path).
Installing Python Packages:
easy_install: the original egg installation tool, depends on setuptools
pip: currently the most popular way to install python packages. Similar to easy_install but more flexible and has some nice features like requirements files to help document dependencies and reproduce deployments.
pypm, apt, yum, fink, etc
Environment Management / Automated Deployment:
bento
buildout
virtualenv (and virtualenvwrapper)
The above tools are used to help automate and manage dependencies for a Python project. Basically they give you tools to describe what distributions your application requires and automate the installation of those specific versions of your dependencies.
Locations of Packages / Distributions:
site-packages
PYTHONPATH
the current working directory (depends on your OS and environment settings)
By default, installing a python distribution is going to drop it into the site-packages directory. That directory is usually something like /usr/lib/pythonX.Y/site-packages.
A simple programmatic way to find your site-packages directory:
from distuils import sysconfig
print sysconfig.get_python_lib()
Ways to modify your PYTHONPATH:
Python's import statement will only find packages that are located in one of the directories included in your PYTHONPATH.
You can inspect and change your path from within Python by accessing:
import sys
print sys.path
sys.path.append("/home/myname/lib")
Besides that, you can set the PYTHONPATH environment variable like you would any other environment variable on your OS or you could use:
.pth files: *.pth files located in directories that are already on your PYTHONPATH are read and each line of the *.pth file is added to your PYTHONPATH. Basically any time you would copy a package into a directory on your PYTHONPATH you could instead create a mypackages.pth. Read more about *.pth files: site module
egg-link files: Internal structure of python eggs they are a cross platform alternative to symbolic links. Creating an egg link file is similar to creating a pth file.
site.py modifications
To add the above /home/myname/lib to site-packages with a *.pth file you'd create a *.pth file. The name of the file doesn't matter but you should still probably choose something sensible.
Let's create myname.pth:
# myname.pth
/home/myname/lib
That's it. Drop that into sysconfig.get_python_lib() on your system or any other directory in your PYTHONPATH and /home/myname/lib will be added to the path.
For packaging question, this should help http://guide.python-distribute.org/
For import, the old article from Fredrik Lundh http://effbot.org/zone/import-confusion.htm still a very good starting point.
I recommend Tarek Ziadek's Book on Python. There's a chapter dedicated to packaging and distribution.
I don't think import needs to be explored (Python's namespacing and importing functionality is intuitive IMHO).
I use pip exclusively now. I haven't run into any issues with it.
However, the topic of packaging and distribution is something worth exploring. Instead of giving a lengthy answer, I will say this:
I learned how to package and distribute my own "packages" by simply copying how Pylons or many other open-source packages do it. I then combined that sort-of template with reading up of the docs to flesh it out even further and have come up with a solid distribution method.
When you grok package management and distribution for python (distutils and pypi) it's actually quite powerful. I like it a lot.
[edit]
I also wanted to add in a bit about virtualenv. USE IT. I create a virtualenv for every project and I always use --no-site-packages; I install all the packages I need for that particular project (even if it's something common amongst them all, like lxml) inside the virtualev. It keeps everything isolated and it's much easier for me to maintain the grouping in my head (rather than trying to keep track of what's where and for which version of python!)
[/edit]
I downloaded an open source project http://gmapcatcher.googlecode.com/files/GMapCatcher-0.7.2.0.tar.gz and I am trying to modify a few things in the code but don't know how to test the code!
I tried to make an installer for the project but nothing worked till now maybe I didn't follow the right steps or I am missing somthing.
my question is how can I modify the code and test it ? and how can I make an installer for this project (I know there is an installer already in google but I want to make it myself).
Looks like the package has a setup.py for the use with distutils. The setup.py works kind of like Makefile for python. The way you use it is (in the directory where setup.py is located:
$ python setup.py command
Where "command" is... well... a command. Type
$ python setup.py --help
for more information. The two basic commands are build and install. install installs, as the name suggests, the package to your system. It is not probably going to do anything like create shortcuts on your desktop or anything like that. It simply installs the Python package into your Python installation. Judging by the contents of setup.py, it seems they're somehow using py2exe (google it; being a newbie I can only include one hyperlink in my answer) to prepare the Windows installer.
To simply run the software, however, it seems all you need to do is to unpack it and do
$ python maps.py
in the package's root directory - provided you have all the necessary dependencies already installed, of course.
I wrote a quick program in python to add a gtk GUI to a cli program. I was wondering how I can create an installer using distutils. Since it's just a GUI frontend for a command line app it only works in *nix anyway so I'm not worried about it being cross platform.
my main goal is to create a .deb package for debian/ubuntu users, but I don't understand make/configure files. I've primarily been a web developer up until now.
edit: Does anyone know of a project that uses distutils so I could see it in action and, you know, actually try building it?
Here are a few useful links
Ubuntu Python Packaging Guide
This Guide is very helpful. I don't know how I missed it during my initial wave of gooling. It even walks you through packaging up an existing python application
The Ubuntu MOTU Project
This is the official package maintaining project at ubuntu. Anyone can join, and there are lots of tutorials and info about creating packages, of all types, which include the above 'python packaging guide'.
"Python distutils to deb?" - Ars Technica Forum discussion
According to this conversation, you can't just use distutils. It doesn't follow the debian packaging format (or something like that). I guess that's why you need dh_make as seen in the Ubuntu Packaging guide
"A bdist_deb command for distutils
This one has some interesting discussion (it's also how I found the ubuntu guide) about concatenating a zip-file and a shell script to create some kind of universal executable (anything with python and bash that is). weird. Let me know if anyone finds more info on this practice because I've never heard of it.
Description of the deb format and how distutils fit in - python mailing list
See the distutils simple example. That's basically what it is like, except real install scripts usually contain a bit more information. I have not seen any that are fundamentally more complicated, though. In essence, you just give it a list of what needs to be installed. Sometimes you need to give it some mapping dicts since the source and installed trees might not be the same.
Here is a real-life (anonymized) example:
#!/usr/bin/python
from distutils.core import setup
setup (name = 'Initech Package 3',
description = "Services and libraries ABC, DEF",
author = "That Guy, Initech Ltd",
author_email = "that.guy#initech.com",
version = '1.0.5',
package_dir = {'Package3' : 'site-packages/Package3'},
packages = ['Package3', 'Package3.Queries'],
data_files = [
('/etc/Package3', ['etc/Package3/ExternalResources.conf'])
])
apt-get install python-stdeb
Python to Debian source package conversion utility
This package provides some tools to produce Debian packages from Python packages via a new distutils command, sdist_dsc. Automatic defaults are provided for the Debian package, but many aspects of the resulting package can be customized via a configuration file.
pypi-install will query the Python Package Index (PyPI) for a
package, download it, create a .deb from it, and then install
the .deb.
py2dsc will convert a distutils-built source tarball into a Debian
source package.
Most Python programs will use distutils. Django is a one - see http://code.djangoproject.com/svn/django/trunk/setup.py
You should also read the documentation, as it's very comprehensive and has some good examples.
I found the following tutorial to be very helpful. It's shorter than the distutils documentation and explains how to setup a typical project step by step.
distutils really isn't all that difficult once you get the hang of it. It's really just a matter of putting in some meta-information (program name, author, version, etc) and then selecting what files you want to include. For example, here's a sample distutils setup.py module from a decently complex python library:
Kamaelia setup.py
Note that this doesn't deal with any data files or or whatnot, so YMMV.
On another note, I agree that the distutils documentation is probably some of python's worst documentation. It is extremely inclusive in some areas, but neglects some really important information in others.