We Keep Coding

How to package python into virtualenv?

How to package Python itself into virtualenv? Is this even possible?
I'm trying to run python on a machine which it is not installed on, and I thought virtualenv made this possible. It activates, but can't run any Python.

When setting up the virtualenv (this can also be done if it already set up) simply do:
python -m virtualenv -p python env
And Python will be added to the virtualenv, and will become the default python of it.
The version of Python can also be passed, as python uses the first version found in the PATH.

virtualenv makes it convenient to use multiple python versions in different projects on the same machine, and isolate the pip install libraries installed by each project. It doesn’t install or manage the overall python environment. Python must be installed on the machine before you can install or configure the virtualenv tool itself or switch into a virtual environment.
Side note, consider using virtualenvwrapper — great helper for virtualenv.

You haven't specified the Operating System you are using.
In case you're using Windows, you don't use virtualenv for this. Instead you:
Download the Python embeddable package
Unpack it
Uncomment import site in the python37._pth file (only if you want to add additional packages)
Manually copy your additional packages (the ones you usually install with pip) to Lib\site-packages (you need to create that directory first, of course)
Such a python installation is configured in such a way that it can be moved and run from any location.
You only have to ensure the Microsoft C Runtime is installed on the system (but it almost always already is). See the documentation note:
Note The embedded distribution does not include the Microsoft C Runtime and it is the responsibility of the application installer to provide this. The runtime may have already been installed on a user’s system previously or automatically via Windows Update, and can be detected by finding ucrtbase.dll in the system directory.

You might need to install python in some location you have the permissions to do so.

Will Anaconda Python config scripts clash with Homebrew's?

Will Anaconda Python config scripts clash with Homebrew's? Note that I do not use these config scripts in any of my workflows, I'm just wondering if any of these config scripts may get called "behind the scenes". Sample output below (with username replaced by '..'):
$ brew doctor
...
Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
/Users/../anaconda/bin/curl-config
/Users/../anaconda/bin/freetype-config
/Users/../anaconda/bin/libdynd-config
/Users/../anaconda/bin/libpng-config
/Users/../anaconda/bin/libpng15-config
/Users/../anaconda/bin/llvm-config
/Users/../anaconda/bin/python-config
/Users/../anaconda/bin/python2-config
/Users/../anaconda/bin/python2.7-config
/Users/../anaconda/bin/xml2-config
/Users/../anaconda/bin/xslt-config
Clearly some of these clash with some Homebrew-installed packages.
$ ls /usr/local/bin/*-config
/usr/local/bin/Magick++-config /usr/local/bin/libpng-config
/usr/local/bin/Magick-config /usr/local/bin/libpng16-config
/usr/local/bin/MagickCore-config /usr/local/bin/pcre-config
/usr/local/bin/MagickWand-config /usr/local/bin/pkg-config
/usr/local/bin/Wand-config /usr/local/bin/python-config
/usr/local/bin/freetype-config /usr/local/bin/python2-config
/usr/local/bin/gdlib-config /usr/local/bin/python2.7-config

Such clashes are entirely possible. When you install softwares that depends on Python using Homebrew, you want it to see Python packages and libraries installed via Homebrew but not those installed by Anaconda.
My solution to this is not putting
export PATH=$HOME/anaconda/bin:$PATH
into .bashrc. Normally, you'll just use Python and pip installed via Homebrew and packages installed by that pip. Sometimes, when you are developing Python projects that is convenient to use Anaconda's environment management mechanism (conda create -n my-env), you can temporarily do export PATH=$HOME/anaconda/bin:$PATH to turn it on. From what I gathered, one important benefit of using Anaconda compared to using regular Python is that conda create -n my-env anaconda will not duplicate package installations unnecessarily as virtualenv my-env will when you have a large number of virtual environments. If you do not mind having some degree of duplication, you could just avoid installing Anaconda all together and just use virtualenv.

It's entirely possible you won't notice any problems. On the other hand, you may have some pretty frustrating ones. It all depends on what you use and how your $PATH is ordered. Homebrew will take whatever file has precedence in your $PATH; if another Homebrew package needs to use Homebrew-installed config files and it sees the Anaconda versions first, it doesn't known any better than to use the wrong ones. In a sense, that's what you told it to do.
My recommendation is to keep things simple and clean. Unless you have a particular reason to keep Anaconda on your $PATH, you should probably pop it out and alias anything you need. Alternatively you could just install the things you require (e.g., numpy) via Homebrew and eliminate Anaconda altogether. (Actually, that's really what I would do. Anaconda comes with way more stuff than I have any reason to be dumping onto my machine.)
I don't know what your $PATH looks like, but in my experience, keeping it short and systematic has a lot of advantages.

Python packages in wrong location after installing Homebrew Python?

After installing Homebrew's Python onto a system with an established Apple Python, the last entries listed by sys.path using Homebrew's Python are
/Library/Python/2.7/site-packages
/usr/local/lib/python2.7/site-package
This is the reverse of the order I expect. Shouldn't Homebrew's packages be searched first? (In fact, shouldn't it be the only place searched?) That's what's implied in the documentation. Where is it set and how can I (or should I) change it?
Or is this the way Brewed Python is supposed to work? Is it designed to prevent duplicate packages and assume that any package in the system site-packages is meant to stay there unless explicitly uninstalled and then subsequently installed (into Brew's); with the exception of pip and setuptools which are duplicated (and put first in Brewed Python's path).

That is the intended behaviour. The rationale behind it is that you can keep using your old installed modules despite the fact that you are now using a new homebrewed Python.
Now this has some drawbacks, for example some libraries like numpy, won't work across different Python versions, so if you had installed numpy it will be imported from the old system's site-packages and won't work.
There are at least two ways to change sys.path:
Use a .pth file:
Python will pick that from some of the builtin locations (ex: ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth). This appends to sys.path which is not ideal but has the advantage that it won't be picked by Python 3. It is currently the recommended method. You can achieve this with:
echo "$(brew --prefix)/lib/python2.7/site-packages" > ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth
Set PYTHONPATH:
This gets prepended to sys.path, it has the drawback that is global to all python versions so it is not recommended if you are going to use different python versions. You can do it by adding to your .bash_profile:
export PYTHONPATH=`brew --prefix`/lib/python2.7/site-packages:$PYTHONPATH
I personally used option 2 with homebrew-python (I now use and recommend Anaconda). My reasons were that I didn't care about system's Python or Python 3 at the time.

How to make macports python use its own packages over another installation's?

I have spent days struggling to get a scientific Python environment running on Mac OS Lion. I tried the SciPack Superpack route, and also various manual installations via pip and easy_install, but still got errors trying to import or use various modules. Based on the advice in this Stackoverflow thread I set up a fresh installation using MacPorts.
However, when I run the macports Python, it is ignoring packages in the macports installation and instead trying to load incompatible packages from the old installation. I am absolutely sure that I am running the newly installed macports Python. I have checked the symlinks and have checked python_select and have launched Python by directly typing the path to the new installation. But when I try to import, say, statsmodels, it pulls in the old version from another directory.
Here are the contents of sys.path (edited for brevity):
['',
'/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg',
'/Library/Python/2.7/site-packages/statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg',
...
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/Library/Python/2.7/site-packages/twilio-3.6.6-py2.7.egg',
'/Library/Python/2.7/site-packages/six-1.6.1-py2.7.egg',
'/Library/Python/2.7/site-packages/httplib2-0.9-py2.7.egg',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
'/Library/Python/2.7/site-packages']
The macports installation is in /opt/local, while the older installation is under /Library/Python. As you can see, the older packages are higher up in the list, which means that they have higher priority.
Environmental variable PYTHONPATH is empty. If I do put anything into PYTHONPATH, that appears in sys.path after the /Library entries but before the /opt entries. So it does not solve the problem.
If I invoke the new python with the -S option, sys.path becomes:
['',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']
That succeeds in getting rid of the alien site packages entries, but it also nukes macports' site-packages entries so then I can't load anything.
I believe that the culprit is a file called /Library/Python/2.7/site-packages/easy-install.pth , with the following contents (again edited for brevity):
import sys; sys.__plen = len(sys.path)
./mrjob-0.4.3_dev-py2.7.egg
...
./statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg
...
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
./twilio-3.6.6-py2.7.egg
./six-1.6.1-py2.7.egg
./httplib2-0.9-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
If I rename that file before launching macports python, python no longer is adding these alien packages to its sys.path. Now sys.path looks a lot like it does when I use "python -S" option except it has these additional entries at the end:
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
'/Library/Python/2.7/site-packages'
Now I am able to load macports' own packages, say for example statsmodels. By checking statsmodels.__file__ I have confirmed that it is importing the local package and not the alien package.
However, I consider this a workaround/kludge rather than a real solution. A solution should make it behave as expected, which is: Macports' Python, launched from /opt/local/... installation, should prioritize packages installed via macports into the /opt/local/... directory tree and go looking elsewhere only if the module does not exist locally. So for example I would expect the /opt/local entries to come first in sys.path, with the /Library entries further down the list.
It seems like this should be the default behavior and I've seen a lot of comments here on Stackoverflow that just assert that this is how Macports Python behaves. So...how do I make it so?

There are at least two separate root causes for the problem you are seeing. One is that non-system OS X framework builds of Python 2.7, including MacPorts Python 2.7, deliberately include the system Python site-packages location, /Library/Python/2.7/site-packages, in sys.path, normally at the end of sys.path after that instance's own site-packages directory. This has been a debated feature in upstream Python and there is an open MacPorts issue to remove it (https://trac.macports.org/ticket/34763), the argument being that it is better to keep the system Python and the MacPorts Python totally separate.
The second root cause is the behavior of the original setuptools package and its easy_install command as supplied with OS X. As you have discovered, it does some fancy manipulation of sys.path by some magic tricks with .pth files, including easy-install.pth, to ensure that the packages that have been installed using easy_install show up first in sys.path and override other installed versions of those packages. Also as you have found, one way - and the easiest way - to remove that behavior is to delete the easy-install.pth file in /Library/Python/2.7/site-packages. That is assuming you don't want to use any of the packages installed there with the Apple-supplied system Python.
The long-term strategy to avoid this problem in the future is to make sure you do not use the Apple-supplied easy_install or easy_install-2.7 commands found in /usr/bin. They will install packages to /Library/Python for use by the system Python and will create or update easy-install.pth. In general, you should avoid use of easy_install all together. Its modern replacement is pip which provides better control and avoids tricks with .pth files. If MacPorts doesn't already provide a port for the Python package you want to install, usually of the form py27-xxxx, install and use the MacPorts py27-pip port instead of easy_install:
/opt/local/bin/pip-2.7 install xxxx
You can also use the usual MacPorts features like port select pip pip27.

Replicate virtualenv without downloading all the packages again on the same machine

I have a couple projects that require similar dependencies, and I don't want to have pip going out and DLing the dependencies from the web every time. For instance I am using the norel-django package which would conflict with my standard django (rdbms version) if I installed it system wide.
Is there a way for me to "reuse" the downloaded dependancies using pip? Do I need to DL the source tar.bz2 files and make a folder structure similar to that of a pip archive or something? Any assistance would be appreciated.
Thanks

Add the following to $HOME/.pip/pip.conf:
[global]
download_cache = ~/.pip/cache
This tells pip to cache downloads in ~/.pip/cache so it won't need to go out and download them again next time.

it looks like virtualenv has a virtualenv-clone command, or perhaps virtualenvwrapper does?
Regardless, it looks to be a little more involved then just copyin and pasting virtual environment directories:
https://github.com/edwardgeorge/virtualenv-clone
additionally it appears virtualenv has a flag that will facilitate in moving your virtualenv.
http://www.virtualenv.org/en/latest/#making-environments-relocatable
$ virtualenv --relocatable ENV from virtualenv doc:
This will make some of the files created by setuptools or distribute
use relative paths, and will change all the scripts to use
activate_this.py instead of using the location of the Python
interpreter to select the environment.
Note: you must run this after you’ve installed any packages into the
environment. If you make an environment relocatable, then install a
new package, you must run virtualenv --relocatable again.
Also, this does not make your packages cross-platform. You can move
the directory around, but it can only be used on other similar
computers. Some known environmental differences that can cause
incompatibilities: a different version of Python, when one platform
uses UCS2 for its internal unicode representation and another uses
UCS4 (a compile-time option), obvious platform changes like Windows
vs. Linux, or Intel vs. ARM, and if you have libraries that bind to C
libraries on the system, if those C libraries are located somewhere
different (either different versions, or a different filesystem
layout).
If you use this flag to create an environment, currently, the
--system-site-packages option will be implied.