We Keep Coding

Python: lightweight package install, without pip?

I'm packaging up a minimal Ubuntu distro to fit in a 4GB disk image, for use on a VPS. This image is a (C++) webapp which (among other things) writes and runs simple Python scripts to handle conversions between csv and xls files, with csvkit and XlsxWriter doing the heavy lifting. My entire Python knowledge is unfortunately limited to writing and running these scripts.
Problem: I install pip in the image to handle the download and install of csvkit and XlsxWriter. This creates a huge amount of cruft, including what seems to be a C++ development environment, just to install what I imagine (presumably incorrectly) is simply Python source code. I can't really afford this in a 4GB distribution.
Is there a lightweight alternative to using pip to do this? Can I just copy over a handful of files from the dev machine, for example? I suppose one alternative is simply to uninstall pip after use, but I'd rather keep the disk image clean if possible (if nothing else, it will compress better).

XlsxWriter doesn't have any dependencies so it can be installed from source using python setup.py install. See the XlsxWriter installation docs.
However csvkit has a number of dependencies and may prove difficult to install from source. Which is why it is best to stick with pip since it handles that for you, which I'm sure you know.
Maybe try to install pip via get-pip which should give the smallest possible bootstrapping installation:
$ curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ python get-pip.py

If you are using python3.4 or newer you might harness ensurepip from standard library. It allows installing pip if it was not installed alongside with python, after doing
python -m ensurepip
you should be able to use pip as if it was installed together with python.

Trouble installing scipy despite having python2.7 and numpy installed already

I'm having trouble installing scipy via the binaries provided at http://sourceforge.net/projects/scipy/files/scipy/
Double clicking on the mpkg file after mounting the dmg installer gives the following error:
"scipy 0.13.0 can't be installed on this disk. scipy requires System Python 2.7 to install"
However, I already have python 2.7 and numpy installed. The python 2.7 came default with OSX Lion, so I assume it is System Python. With other python modules, one normally can download the binary then run
python setup.py install
Is there a way to cd through the mpkg file and locate a setup.py? Any advice install via this dmg installer?
I know there are other ways to manage python modules, like port and brew. However, I already installed a bunch of packages through setup.py, and I couldn't figure out how to get port to recognize those packages (for example, it will try to reinstall python and numpy via port)
Thanks!

If you have Mavericks and XCode 5, then you'll have to install Command Line Tools manually from the Apple Developer Site. I found this helpful post

You've got a few misconceptions here.
With other python modules, one normally can download the binary then run python setup.py install
No, that's what you do with source packages.
Is there a way to cd through the mpkg file and locate a setup.py?
No. What's inside an mpkg are pkg files. Which are filled with xar archives filled with cpio archives. Inside there is the built version of SciPy—that is, the files that setup.py would have copied to your site-packages if you'd run it—not the source package.
But you can download the source package yourself.
Or, better, let pip (or easy_install, but pip is better) download and run the setup.py for you.
Any advice install via this dmg installer?
If it won't work, my advice would be to not use it, and instead install with pip.
This blog post explains it, but I'll give you the details relevant to you below.
I know there are other ways to manage python modules, like port and brew. However, I already installed a bunch of packages through setup.py, and I couldn't figure out how to get port to recognize those packages.
You can't. MacPorts will not touch your system Python; it builds its own separate Python 2.7, with a completely independent site-packages directory and everything else. You would have to reinstall everything for this second Python 2.7. And deal with the confusion of having two Python 2.7 installations on the same machine.
Don't do that unless you absolutely have to.
In fact, if you want to use Homebrew for anything (and you do, see below), uninstall MacPorts, unless you really need it for something.
So, here are the steps:
Uninstall MacPorts.
I assume you already have Xcode and its Command Line Tools.
I assume you already have Homebrew.
Install a Fortran compiler with brew install gfortran.
Lion's Python 2.7 comes with easy_install, but not pip. So sudo easy_install pip to fix that. While you're at it, I'd suggest sudo easy_install readline, because you'll want that for ipython, and it won't work right with pip.
Apple's pre-installed NumPy has to be upgraded, and rebuilt with Fortran support, to make SciPy work. Fix that with sudo pip install --upgrade --force-reinstall numpy.
If you want ipython, pandas, etc. sudo pip install each of them as well.
In case you're considering upgrading soon, the exact same steps worked for me with OS X 10.9.0, except for some extra work to get the Xcode 5 command line tools set up.

associate python packages to a different version of python on ubuntu

I am using ubuntu 11.04, which comes with the system-wide python 2.6. Now, I installed the python2.7 in addition to the v2.6.
Now, the question is, if I want to install the latest version of numpy, scipy, matplotlib, etc to make them associated with the python2.7, what should I do to make sure they are not associated with the python 2.6?
Thanks.
J.

You have a few options. Which is best depends on what you want to use those libraries for. If you're doing development, virtualenv is a good idea:
$ virtualenv -p /usr/bin/python2.7 py27env && . py27env/bin/activate
py27env$ pip install numpy scipy matplotlib

Pull down the latest tarballs for numpy, scipy, and matplotlib. You can get numpy and scipy from here:
http://scipy.org/Download
Matplotlib can be found here:
http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.0/
Then open up a terminal and use python 2.7 to install them using the setup.py scripts that come with the tarballs. For example, do the following for numpy (assuming you've pulled down the latest tarball already from sourceforge and it's sitting on your desktop:
$ mv Desktop/numpy-1.6.2.tar.gz /tmp/
$ cd /tmp/
$ tar -xvzf numpy-1.6.2.tar.gz
$ cd numpy-1.6.2
$ python2.7 setup.py install
That should do it. Tarballs for python code generally come with a setup.py script that will install things in the right place for the version of python you run it with.

Seems like this post answers your question:
Newbie hint on installing Python and it’s modules and packages
You install every Python separately, you install every module and
package separately in those Python install, and you use everything
explicitly.

How do Homebrew, PIP, easy_install etc. work so that I can clean up

I have a problem that comes from me following tutorials without really understanding what I'm doing. The root of the problem I think is the fact that I don't understand how the OS X filesystem works.
The problem is bigger than Python but it was when I started learning about Python that I realized how little I really understand. So in the beginning I started following tutorials which led me to use the easy_install command a lot and when a lot of tutorials recommended PIP I never got it running. So I have run a lot of commands and installed a lot of different packages.
As I have understood Lion comes with a python install. I have been using this a lot and from this I have installed various packages with easy_install. Is there any way to go back to default installation and begin from the very beginning? Is this something I want to do? If so why?
Is there any advantage of using a Python version I have installed with Homebrew? How can I see from where Python is run when I run the Python command?
When I do install something with either easy_install, homebrew, macports etc where do things actually end up?

Homebrew installs its software inside the /usr/local subdirectory on your Mac. OS X doesn't install anything there on its own; in fact, /usr/local is reserved for user-installed stuff. Since Homebrew never installs files outside /usr/local (and doesn't even have the ability to, unless you run brew using sudo - which is not recommended_) and OS X never installs files inside there, never the two shall mix.
easy_install and pip install files into system directories by default. That's why you have to run those commands with sudo to install packages with them.
I can't recommend virtualenv enough, regardless of which OS you're using. It installs a copy of Python, along with any packages or modules you want, inside a directory of your choosing. For example:
$ cd /tmp
$ virtualenv foo
New python executable in foo/bin/python
Installing setuptools............done.
Installing pip...............done.
$ cd foo
$ bin/pip install sqlalchemy
Downloading/unpacking sqlalchemy
Downloading SQLAlchemy-0.7.7.tar.gz (2.6Mb): 2.6Mb downloaded
Running setup.py egg_info for package sqlalchemy
[...]
Successfully installed sqlalchemy
Cleaning up...
[work, work, work]
[decide this was a bad idea]
$ cd /tmp; rm -rf foo
...and all traces of the project are now completely gone.
Use easy_install to install virtualenv into OS X itself - like you've done for those other packages - but then do all new development inside isolated directories that you can wipe clean at a moment's notice. This is pretty much the standard way of developing and deploying Python applications these days.

The advantage of using a Python installed via a package manager like Homebrew or MacPorts would be that this provides a simple way of removing the Python installation and reinstalling it. Also, you can install a more recent version than the one Mac OS X provides.

How to install pycairo on osx?

I am trying to install the pycairo (Python bindings for the cairo graphics library) under OSX.
I started with
easy_install pycairo
and got:
Requested 'cairo >= 1.8.8' but version of cairo is 1.0.4
error: Setup script exited with Error: cairo >= 1.8.8 not found
So I went to cairo's site and downloaded the latest package (1.8.8) of cairo, and also the latest package of something called pixman (both source packages -- couldn't find osx binaries)
unzipped both, each in own directory.
for pixman, the regular ./configure ; make ; sudo make install worked just find
for cairo, ./configure seemed to work, but make failed with:
In file included from cairo-analysis-surface.c:37:
cairoint.h:71:20: error: pixman.h: No such file or directory
What am I doing wrong?
And why do I have to struggle so much to get a software library to work on an os that "just works"? Why isn't darwin more like linux?

If you already have homebrew, these two commands should be helpful:
$ brew install cairo --use-clang
$ brew install py2cairo
For a non-Homebrew installed Python, set the PYTHONPATH to find pycairo. You can set your PYTHONPATH in your .bashrc/.profile/.whatever to the following:
PYTHONPATH=/usr/local/lib/python2.7/site-packages:$PYTHONPATH.
I personally didn't need to use this last part but it might help you.

It appears you are mixing various install options here. The MacPorts package system port install command should automatically pull in all the dependencies needed for a particular package so the trick is to start with the right top-level project. For python packages, MacPorts has a general convention currently: packages that start with py- are for python 2.4, those with py25- are for 2.5, and py26- for 2.6. There are currently py-cairo, py25-cairo, and py26-cairo packages available in MacPorts.
By choosing py-cairo you picked the python2.4 version and you'll probably find that MacPorts built and installed a python2.4 for you (linked at /opt/local/bin/python2.4) and, if you launch it, you'll probably find that you can import cairo there. Now that may be OK for your needs but Python 2.4 is quite old and no longer supported so, if you're just starting, it might be better to start with Python 2.6, one of the two current versions of Python. To do so, all you should need to do is:
sudo port install py26-cairo
That should bring in any missing dependencies, mainly the MacPorts python2.6, which you can run from /opt/local/bin/python2.6. You may want to change your $PATH in your shell startup script, probably .bash_profile, to put /opt/local/bin early on the search path.
Because installing Cairo and its python bindings seems to be fairly complex, it should be easier and better to stick to using a complete MacPorts solution for this. That does mean you've needlessly (and harmlessly) installed a couple of Python instances that you won't need. But if you do want to clean things up a bit, you can easily remove the MacPorts python24 with:
sudo port uninstall py-cairo python24
Completely removing the python.org installed python is more complicated. I've explained the process here. But there's no pressing need to remove either as long as you keep your paths straight.

Ok. I solved it. Putting solution here for future reference, it might help someone.
Basically, the whole ports/fink system is a bit messed up, and osx doesn't really play nice with the linux-y world.
So, the steps I needed to install pycairo on OSX were:
download the latest source versions of pixman, cairo, pycairo
extract everything. Then:
cd PIXMAN_DIR ; ./configure ; make ; sudo make install
cd CAIRO_DIR ; cp PIXMAN_DIR/pixman/*.h . ; ./configure ; make ; sudo make install
cd PYCAYRO_DIR; locate cairo.pc
hopefully, several locations are returned. choose the most likely one (one with newest cairo). For me it was "/opt/local/lib/pkgconfig/cairo.pc" and do:
export PKG_CONFIG_PATH=/opt/local/lib/pkgconfig/
after this, still in PYCAIRO_DIR, do:
python setup.py install
This should do it...

The port command installs the library for the darwinports python installation, which is different to the framework build (so steps 2 and 3 shouldn't work). Try sudo easy_install pycairo instead (although your step 4 should be equivalent to this).
Look at which python too, to check that you are in fact running the python you think you are.

On Mac OS you can have multiple Python versions installed. You can have even more if you decide to install Python via Fink or MacPorts. When you compile libraries from the source, you should make sure they point to the correct installation.
I currently have Python 2.5.1 and Python 2.6.4 installed on my machine, which I can call via python2.5 and python respectively. They live in two different folders:
/System/Library/Frameworks/Python.framework/Versions/2.5and /Library/Frameworks/Python.framework/Versions/2.6
I was running into a similar problem when compiling pycairo 1.8.8 from the tarball. The INSTALL file in this case is your friend, as it contains the correct instructions to avoid potential version conflicts. You basically need to specify the correct prefix so that the package will be installed in the correct folder.
$ python -c "import sys; print sys.prefix"
# make a note of the python prefix
$ ./configure --prefix=[python_prefix]
$ make
$ make install # may require superuser access
Running these instructions with python2.5 and python you will be able to correctly install pycairo for both versions (or for any version installed via MacPorts / Fink).

Step 1: Run this from terminal ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null
Step 2: brew install cairo
Step 3: pip install pycairo