We Keep Coding

Can I have my pip user-installed package be preferred over system?

I would like to figure out a "fool-proof" installation instruction to put in the README of a Python project, call it footools, such that other people in our group can install the newest SVN version of it on their laptops and their server accounts.
The problem is getting the user-installed libs to be used by Python when they call the scripts installed by pip. E.g., we're using a server that has an old version of footools in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/.
If I do python2.7 setup.py install --user and run the main entry script, it uses the files in /Users/unhammer/Library/Python/2.7/lib/python/site-packages/. This is what I want, but setup.py alone doesn't install dependencies.
If I (revert the installation and) instead do pip-2.7 install --user . and run the main entry script, it uses the old files in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ – that's not what I want.
If I (revert the installation and) instead do pip-2.7 install --user -e . and run the main entry script, it uses the files in . – that's not what I want, the user should be able to remove the source dir (and be able to svn up without that affecting their install).
I could use (and recommend other people to use) python2.7 setup.py install --user – but then they have to first do
pip-2.7 install -U --user -r requirements.txt -e .
pip-2.7 uninstall -y footools
in order to get the dependencies installed (since pip has no install --only-deps option). That's rather verbose though.
What is setup.py doing that pip is not doing here?
(Edited to make it clear I'm looking for simpler+safer installation instructions.)

Install virtualenvwrapper. I allows setting up separate python environments to alleviate any conflicts you might be having. Here is a tutorial for installing and using virtualenv.
Related:
https://virtualenvwrapper.readthedocs.org/en/latest/

Console scripts generated by pip in the process of installation should use user installed versions of libraries as according to PEP 370:
The user site directory is added before the system site directories
but after Python's search paths and PYTHONPATH. This setup allows the
user to install a different version of a package than the system
administrator but it prevents the user from accidently overwriting a
stdlib module. Stdlib modules can still be overwritten with
PYTHONPATH.
Sidenote
Setuptools use hack by inserting code in easy_install.pth file which is placed in site-packages directory. This code makes packages installed with setuptools come before other packages in sys.path so they shadow other packages with the same name. This is referred to as sys.path modification in the table comparing setuptools and pip. This is the reason console scripts use user installed libraries when you install with setup.py install instead of using pip.
Taking all of the above into account the reason for what you observe might be caused by:
PYTHONPATH pointing to directories with system-wide installed libraries
Having system-wide libraries installed using sudo python.py install (...)
Having OS influence sys.path construction in some way
In the first case either clearing PYTHONPATH or adding path to user installed library to the beginning of PYTHONPATH should help.
In the second case uninstalling system-wide libraries and installing them with distro package manager instead might help (please note that you never should use sudo with pip or setup.py to install Python packages).
In the third case it's necessary to find out how does OS influence sys.path construction and if there's some way of placing user installed libraries before system ones.
You might be interested in reading issue pip list reports wrong version of package installed both in system site and user site where I asked basically the same question as you:
Does it mean that having system wide Python packages installed with easy_install thus having them use sys.path manipulation breaks scripts from user bin directory? If so is there any workaround?
Last resort solution would be to manually place directory/directories with user installed libraries in the beginning of sys.path from your scripts before importing these libraries.
Having said that if your users do not need direct access to source code I would propose packaging your app together with all dependencies using tool like pex or Platter into self-contained bundle.

Setuptools, pip, virtualenv and windows

I am working on a Windows system with limited download access. My workaround is to download an app into a local folder. Then in the virtualenv I use pip install -e download folder to install. For instance, pip install -e c:\django\test_virtualenv\South-0.8.4 and South is installed in the virtual environment.
In the virtualenv I am trying to install django-user-accounts, using the same method. Unfortunately, this app has two dependencies listed in setup.py, namely, "django-appconf>=0.6", "pytz>=2013.9". You can see these in the install_requires section in the setup.py file.
I was unable to redirect to a local folder such as c:\download\django-appconf
This lead to the question whether arguments were available when using install_required (part of setuptools). install_requires in setup.py depending on installed Python version points out install_requires takes just a list.
In http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies the text talks about "limited network access" and "manually download all the eggs to a single directory". This doc also suggests using easy_install with the -f option.
I have a neat system operational at the moment using pip and I am asking if anyone has found a way to redirect the downloading of dependencies from a local folder, no access to network resources for download.
Tommy.

setup.py not honoring PIP_INDEX_URL

I am running a local pypi server. I can install packages from this server by either specifying it with the -i option of the pip command or by setting the PIP_INDEX_URL environment variable. When I install a package that has prerequisites, setup.py has historically honored the PIP_INDEX_URL environment variable, pulling the additional packages from my local server.
However, on a couple of systems that have been recently installed, it is behaving differently. Running, for instance, python setup.py develop fails because it tries to install prerequisites packages from pypi.python.org.
I have updated all of the related python packages (python, distribute, virtualenv, pip, etc...) on all the systems I'm testing on and continue to see this discrepancy. On my "original" system, setup.py downloads prerequisites from the pypi server specified in my PIP_INDEX_URL environment variable. On the newer systems, I can't seem to make it honor this variable.
What am I missing?

Create setup.cfg in the same folder as your setup.py with following content:
[easy_install]
allow_hosts = *.myintranet.example.com
From: http://pythonhosted.org/setuptools/easy_install.html#restricting-downloads-with-allow-hosts
You can use the --allow-hosts (-H) option to restrict what domains EasyInstall will look for links and downloads on.
--allow-hosts=None prevents downloading altogether.

I ran into the same issue. Fundamentally, setup.py is using setuptools which leverages easy_install, not pip. Thus, it ignores any pip-related environment variables you set.
Rather than use python setup.py develop you can run pip (from the top of the package) pip install -e . to produce the same effect.

How to install python packages without root privileges?

I am using numpy / scipy / pynest to do some research computing on Mac OS X. For performance, we rent a 400-node cluster (with Linux) from our university so that the tasks could be done parallel. The problem is that we are NOT allowed to install any extra packages on the cluster (no sudo or any installation tool), they only provide the raw python itself.
How can I run my scripts on the cluster then? Is there any way to integrate the modules (numpy and scipy also have some compiled binaries I think) so that it could be interpreted and executed without installing packages?

You don't need root privileges to install packages in your home directory. You can do that with a command such as
pip install --user numpy
or from source
python setup.py install --user
See https://stackoverflow.com/a/7143496/284795
The first alternative is much more convenient, so if the server doesn't have pip or easy_install, you should politely ask the admins to add it, explaining the benefit to them (they won't be bothered anymore by requests for individual packages).

You could create a virtual environment through the virtualenv package.
This creates a folder (say venv) with a new copy of the Python executable and a new site-packages directory, into which you can "install" any number of packages without needing any kind of administrative access at all. Thus, activating the environment through source venv/bin/activate will give Python an environment that's equivalent to having those packages installed.
I know this works for SGE clusters, although how the virtual environment is activated might depend on your cluster's configuration.
You can try installing virtualenv on your cluster within your own site-packages directory using the following steps:
Download virtualenv from here, put it on your cluster
Install it using setup.py to a specific, local directory to serve as your own site-packages:
python setup.py build
python setup.py install --install-base /path/to/local-site-packages
Add that directory to your PYTHONPATH:
export PYTHONPATH="/path/to/local-site-packages:${PYTHONPATH}"
Create a virtualenv:
virtualenv venv

You can import a module from an arbitrary path by calling:
sys.path.append()

The Python Distribution Anaconda solves many of the issues discussed in this questions. Anaconda does not require Admin or root access and is able to install to your home directory. Anaconda comes with many of the packages in question (scipy, numpy, sklearn, etc...) as well as the conda installer to install additional packages should additional ones be necessary.
It can be downloaded from https://www.continuum.io/downloads

Override default installation directory for Python bdist Windows installer

Is it possible to specify during the installer generation (or during the actual installation) a custom path for Python modules? By way of example, let's say I have 5 modules for which I generate an installer using:
c:\>python setup.py bdist
Everything gets packaged up correctly, but when I install, I am forced to install into site-packages. I need to be able to specify a custom directory of my (or the installer's choosing). At a minimum, I need to be able to override the default so my custom path appears as the default.
Is this possible using a built distribution?

You should write setup.cfg where you can specify installation options(see python setup.py install --help output) and then run python setup.py bdist. When creating binary distro python will do the dumb installation under the "build" subdir with this options and create the installer from this dumb installation. For example, if you want to create bdist which installs libraries to /some/lib/path and scripts to /some/bin/path create the following setup.cfg:
[install]
prefix=/
install_lib=/some/lib/path
install_scripts=/some/bin/path
And then run python setup.py bdist

From running python setup.py --help install:
Options for 'install' command:
--prefix installation prefix
--exec-prefix (Unix only) prefix for platform-
specific files
--home (Unix only) home directory to install
under
--user install in user site-package
'/home/jterrace/.local/lib/python2.7/si
te-packages'
--install-base base installation directory (instead of
--prefix or --home)
--install-platbase base installation directory for
platform-specific files (instead of --
exec-prefix or --home)
--root install everything relative to this
alternate root directory

I do beleive that MaxSin's answer was somewhat correct. But to use his answer for the command: "python setup.py bdist_wininst" you would have to do it like this:
[bdist_wininst]
prefix=/
install_lib=/some/lib/path
install_scripts=/some/bin/path
Seeing as the syntax here is:
[command]
option=value
...
edit:
It looks like this doesnt work :( not sure of a possible other solution.