We Keep Coding

Shouldn't virtualenv be used on Ansible target nodes?

In most cases I think that Ansible engineers install pip packages 1) without using a virtualenv and 2) under root.
If we do this manually we would see a warning
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Typically when our Ansible automation becomes more advanced we would need additional pip packages to make Ansible modules work. More often than not this also requires additional OS packages to be installed. For example for python-ldap pip package on Ubuntu 18.04 requires
build-essential
python3-dev
python3-wheel
libsasl2-dev
libldap2-dev
libssl-dev
The way that Ansible is made to work on target nodes by installing additional pip packages as root while this clearly not the recommended way to use Python and Pip makes me wonder if there is not a better way to do this.
Should we not use virtualenv and another account than root for installing pip for Ansible?

There are probably multiple aspects to this. The one that came to my mind first, is this:
Using the "global" python outside of any venv, will probably work for the vast majority of users very well, while using a venv can lead to all kinds of unexpected behavior, if you are not familiar with the concept.
For example, if a venv was used by default, people will install python packages and then wonder why python claims they are not available when they try to import them in python on the host.
On the other hand, it is probably relatively easy to use a venv, if you want to do that. In any playbook, you can create the venv and then just update the ansible_python_interpreter variable:
- pip:
name: pip
virtualenv: /path/to/venv
- set_fact:
ansible_python_interpreter: /path/to/venv/bin/python
- setup:
Disclaimer: I just tried that very quickly and saw that it worked, so there might be glitches in some situations.
Obviously, it is not very neat to add something like this to every play, but it can be made a lot nicer, e.g. by creating the venv on the first setup and then using the interpreter_python key in ansible.cfg.
tl;dr:
Using global python is probably the way it works the best for most users, while "power-users" still have ways to achieve what they want with some additional actions.

setup.py + virtualenv = chicken and egg issue?

I'm a Java/Scala dev transitioning to Python for a work project. To dust off the cobwebs on the Python side of my brain, I wrote a webapp that acts as a front-end for Docker when doing local Docker work. I'm now working on packaging it up and, as such, am learning about setup.py and virtualenv. Coming from the JVM world, where dependencies aren't "installed" so much as downloaded to a repository and referenced when needed, the way pip handles things is a bit foreign. It seems like best practice for production Python work is to first create a virtual environment for your project, do your coding work, then package it up with setup.py.
My question is, what happens on the other end when someone needs to install what I've written? They too will have to create a virtual environment for the package but won't know how to set it up without inspecting the setup.py file to figure out what version of Python to use, etc. Is there a way for me to create a setup.py file that also creates the appropriate virtual environment as part of the install process? If not — or if that's considered a "no" as this respondent stated to this SO post — what is considered "best practice" in this situation?

You can think of virtualenv as an isolation for every package you install using pip. It is a simple way to handle different versions of python and packages. For instance you have two projects which use same packages but different versions of them. So, by using virtualenv you can isolate those two projects and install different version of packages separately, not on your working system.
Now, let's say, you want work on a project with your friend. In order to have the same packages installed you have to share somehow what versions and which packages your project depends on. If you are delivering a reusable package (a library) then you need to distribute it and here where setup.py helps. You can learn more in Quick Start
However, if you work on a web site, all you need is to put libraries versions into a separate file. Best practice is to create separate requirements for tests, development and production. In order to see the format of the file - write pip freeze. You will be presented with a list of packages installed on the system (or in the virtualenv) right now. Put it into the file and you can install it later on another pc, with completely clear virtualenv using pip install -r development.txt
And one more thing, please do not put strict versions of packages like pip freeze shows, most of time you want >= at least X.X version. And good news here is that pip handles dependencies by its own. It means you do not have to put dependent packages there, pip will sort it out.
Talking about deploy, you may want to check tox, a tool for managing virtualenvs. It helps a lot with deploy.

Python default package path always point to system environment, that need Administrator access to install. Virtualenv able to localised the installation to an isolated environment.
For deployment/distribution of package, you can choose to
Distribute by source code. User need to run python setup.py --install, or
Pack your python package and upload to Pypi or custom Devpi. So the user can simply use pip install <yourpackage>
However, as you notice the issue on top : without virtualenv, they user need administrator access to install any python package.
In addition, the Pypi package worlds contains a certain amount of badly tested package that doesn't work out of the box.
Note : virtualenv itself is actually a hack to achieve isolation.

For real, too many installations of Python on OSX Mountain Lion

I have three different Python 2.7s at:
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
I use a number of packages that come from different sources. I am currently installing packages from port (MacPorts), easy_install, pip (installed by easy_install), and Mercurial. There are also some that I have to install from image or build from source. I have more control over those.
The problem is that easy_install and pip seem to be installing to one location (/Library/Frameworks/...) and MacPorts installs to another (/opt/local/Library/Frameworks/...).
What's my best action now? Delete /Library/Frameworks/.../python2.7 and move easy_install and pip to the MacPorts one at /opt/local/...? Link the two directories? Move the MacPorts installation to /Library/Frameworks/...?
How can I consolidate these Pythons? I have tried putting both site-packages locations in my path, but only certain packages are available only for one Python and not the other and others vice versa, and I need them all available at once.

It seems that you have control over the stuff you're building yourself. This is how I consolidate macports with pip:
I like using Macports for all my stuff, so I just make sure that pip and easy_install build into macports' installation of python (the one in /opt/local/...).
You can tell where pip and easy_install will install things by using:
readlink `which pip`
(those are backticks)
If you want pip to install to the macports direcectories, use macports to install pip:
sudo port install py-pip
Then, be sure that which pip points to something like:
askewchan#rock:~$ which pip
/opt/local/bin/pip
askewchan#rock:~$ readlink `which pip`
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip-2.7
From the comments below (thanks #Jonathan and #Ned) you can do the same with easy_install but its port is called py-distribute:
sudo port install py-distribute
But as far as I know, you never need to use easy_install because anything that can be easy_installed can be piped better.
Note the port descriptions:
askewchan#rock:Tracking {master *}$ port search *easy*install*
py-pip #1.2.1 (python, www)
An easy_install replacement
askewchan#rock:Tracking {master *}$ port search py*distribute
py-distribute #0.6.35 (python, devel)
Replacement for setuptools

I suggest deciding on one and only one Python for your development work ( personally, I use distribution from Python.org )
You can't get rid of /Library/Frameworks - that's the default OSX one, and you could break things
of the two remaining Pythons, I'm assuming one is Macports and the other is Python.org -- you need to choose which one you want to be your development env and to stick with that.
I would strongly recommend against using pip or easy_install from one Python to install modules for another. The reason is that there can be differences in the compile options. It can be hard enough as-is to get certain packages to compile on OSX properly -- if you start compiling against different binaries ( which might support different architectures ) you're just going to increase your headaches.
I personally chose the following path:
I use the Python.org package for all development.
On a terminal login, I run shell scripts to prioritize my Python choice
All of my projects have their own virtualenv , and I disable system packages
When starting to work on any project, I tend to have an environment setup script. I just type in go_myproject.source ; that cds me to the right directory and runs the source /path/to/virtualenv/bin/activate to get me set up for that project.
There's a tiny bit of overhead on getting things setup, but I have been in complete heaven ever since. Managing projects and not needing to worry about dependencies/upgrades for one thing killing something else is... blissful.

While not a general solution, I install Mercurial and other Python-based applications using virtualenv. In particular, pip and easy_install will install to the respective virtual environment only and not clutter any system folder. The downside is, of course, that I will have duplicates of some packages; the advantage is that I have a clean, self-contained environment with a known version of Python (which for things such as Mercurial and other mission-critical applications is more important for me).
Another downside is that I need to link individual applications to my personal bin directory or add the bin directories of the virtual environments to my path. (Personally, I manage this with some simple scripts that do the symlinking for me.)

I sugest to move all python installations to the one place and create symlinks.
After that configure python environment to avoid problems with imports and "visibility" of the modules.
Try to use commands:
# easy_install
env PYTHONPATH=/custom/path easy_install –install-dir /custom/path
#pip
pip install --install-option="--prefix=$PREFIX_PATH" package_name

What are the Python equivalents to Ruby's bundler / Perl's carton?

I know about virtualenv and pip. But these are a bit different from bundler/carton.
For instance:
pip writes the absolute path to shebang or activate script
pip doesn't have the exec sub command (bundle exec bar)
virtualenv copies the Python interpreter to a local directory
Does every Python developer use virtualenv/pip? Are there other package management tools for Python?

From what i've read about bundler — pip without virtualenv should work just fine for you. You can think of it as something between regular gem command and bundler. Common things that you can do with pip:
Installing packages (gem install)
pip install mypackage
Dependencies and bulk-install (gemfile)
Probably the easiest way is to use pip's requirements.txt files. Basically it's just a plain list of required packages with possible version constraints. It might look something like:
nose==1.1.2
django<1.3
PIL
Later when you'd want to install those dependencies you would do:
$ pip install -r requirements.txt
A simple way to see all your current packages in requirements-file syntax is to do:
$ pip freeze
You can read more about it here.
Execution (bundler exec)
All python packages that come with executable files are usually directly available after install (unless you have custom setup or it's a special package). For example:
$ pip install gunicorn
$ gunicorn -h
Package gems for install from cache (bundler package)
There is pip bundle and pip zip/unzip. But i'm not sure if many people use it.
p.s. If you do care about environment isolation you can also use virtualenv together with pip (they are close friends and work perfectly together). By default pip installs packages system-wide which might require admin rights.

You can use pipenv, which has similar interface with bundler.
$ pip install pipenv
Pipenv creates virtualenv automatically and installs dependencies from Pipfile or Pipfile.lock.
$ pipenv --three # Create virtualenv with Python3
$ pipenv install # Install dependencies from Pipfile
$ pipenv install requests # Install `requests` and update Pipfile
$ pipenv lock # Generate `Pipfile.lock`
$ pipenv shell # Run shell with virtualenv activated
You can run command with virtualenv scope like bundle exec.
$ pipenv run python3 -c "print('hello!')"

There is a clone pbundler.
The version that is currently in pip simply reads the requirements.txt file you already have, but is much out of date. It's also not totally equivalent: it insists on making a virtualenv. Bundler, I notice, only installs what packages are missing, and gives you the option of giving your sudo password to install into your system dirs or of restarting, which doesn't seem to be a feature of pbundler.
However, the version on git is an almost complete rewrite to be much closer to Bundler's behaviour... including having a "Cheesefile" and now not supporting requirements.txt. This is unfortunate, since requirements.txt is the de facto standard in pythonland, and there's even Offical BDFL-stamped work to standardize it. When that comes into force, you can be sure that something like pbundler will become the de facto standard. Alas, nothing quite stable yet that I know of (but I would love to be proven wrong).

I wrote one — https://github.com/Deepwalker/pundler .
On PIP its pundle, name was already taken.
It uses requirements(_\w+)?.txt files as your desired dependencies and creates frozen(_\w+)?.txt files with frozen versions.
About (_\w+)? thing — this is envs. You can create requirements_test.txt and then use PUNDLEENV=test to use this deps in your run with requirements.txt ones alongside.
And about virtualenv – you need not one, its what pundle takes from bundler in first head.

Python Poetry is the closest to Ruby bundler as of 2020 (and already since 2018). It's already more than two years old, still very active, has great documentation. One might complain about curl-pipe-python-style being the recommended way of installing, but there are alternatives, e.g. homebrew on macOS.
Primary website: https://python-poetry.org/
Github: https://github.com/python-poetry/poetry
Documentation: https://python-poetry.org/docs/
It uses virtualenvs behind the scenes (in contrast to bundler), but it provides and uses a lock-file, takes care of sub dependencies, adheres to specified version constraints and allows automatically updating outdated packages. There's even autocompletion for your favorite shell.
With its use of a pyproject.toml file, it's also going a bit further than bundler (closer to a gemspec. It's also comparable to JavaScript's and TypeScript's npm and yarn).
Poetrify (a complementing project) helps converting projects from requirements.txt to pyproject.toml for Poetry.
The lock file can be exported to requirements.txt by poetry export -f requirements.txt > requirements.txt, if you need that for other tooling (or the unlikely case want to go back).

I'd say Shovel is worth a look. It was developed specifically to the Pythonish version of Rake. There's not a ton of commit activity on the project, but seems stable and useful.

You can use pipx to install and run Python Applications in Isolated Environments automatically.
You can use pipenv to create and manage a virtualenv for your projects automatically.
Both wraps pip with virtual environment tools and aiming for different use cases.
All of these are one of the most stared project listed in the github PyPA repository.
FYI: Debian bullseye/testing currently lacks pipx. But package from sid should work fine. (2021-06-19)

No, no all the developers use virtualenv and/or pip, but many developers use/prefer these tools
And now, for package development tools and diferent environments that is your real question. Exist any other tools like Buildout (http://www.buildout.org/en/latest/) for the same purpose, isolate your environment Python build system for every project that you manage. For some time I use this, but not now.
Independent environments per project, in Python are a little different that the same situation in Ruby. In my case i use pyenv (https://github.com/yyuu/pyenv) that is something like rbenv but, for Python. diferent versions of python and virtualenvs per project, and, in this isolated environments, i can use pip or easy-install (if is needed).

Does Python have a package/module management system?

Does Python have a package/module management system, similar to how Ruby has rubygems where you can do gem install packagename?
On Installing Python Modules, I only see references to python setup.py install, but that requires you to find the package first.

Recent progress
March 2014: Good news! Python 3.4 ships with Pip. Pip has long been Python's de-facto standard package manager. You can install a package like this:
pip install httpie
Wahey! This is the best feature of any Python release. It makes the community's wealth of libraries accessible to everyone. Newbies are no longer excluded from using community libraries by the prohibitive difficulty of setup.
However, there remains a number of outstanding frustrations with the Python packaging experience. Cumulatively, they make Python very unwelcoming for newbies. Also, the long history of neglect (ie. not shipping with a package manager for 14 years from Python 2.0 to Python 3.3) did damage to the community. I describe both below.
Outstanding frustrations
It's important to understand that while experienced users are able to work around these frustrations, they are significant barriers to people new to Python. In fact, the difficulty and general user-unfriendliness is likely to deter many of them.
PyPI website is counter-helpful
Every language with a package manager has an official (or quasi-official) repository for the community to download and publish packages. Python has the Python Package Index, PyPI. https://pypi.python.org/pypi
Let's compare its pages with those of RubyGems and Npm (the Node package manager).
https://rubygems.org/gems/rails RubyGems page for the package rails
https://www.npmjs.org/package/express Npm page for the package express
https://pypi.python.org/pypi/simplejson/ PyPI page for the package simplejson
You'll see the RubyGems and Npm pages both begin with a one-line description of the package, then large friendly instructions how to install it.
Meanwhile, woe to any hapless Python user who naively browses to PyPI. On https://pypi.python.org/pypi/simplejson/ , they'll find no such helpful instructions. There is however, a large green 'Download' link. It's not unreasonable to follow it. Aha, they click! Their browser downloads a .tar.gz file. Many Windows users can't even open it, but if they persevere they may eventually extract it, then run setup.py and eventually with the help of Google setup.py install. Some will give up and reinvent the wheel..
Of course, all of this is wrong. The easiest way to install a package is with a Pip command. But PyPI didn't even mention Pip. Instead, it led them down an archaic and tedious path.
Error: Unable to find vcvarsall.bat
Numpy is one of Python's most popular libraries. Try to install it with Pip, you get this cryptic error message:
Error: Unable to find vcvarsall.bat
Trying to fix that is one of the most popular questions on Stack Overflow: "error: Unable to find vcvarsall.bat"
Few people succeed.
For comparison, in the same situation, Ruby prints this message, which explains what's going on and how to fix it:
Please update your PATH to include build tools or download the DevKit from http://rubyinstaller.org/downloads and follow the instructions at http://github.com/oneclick/rubyinstaller/wiki/Development-Kit
Publishing packages is hard
Ruby and Nodejs ship with full-featured package managers, Gem (since 2007) and Npm (since 2011), and have nurtured sharing communities centred around GitHub. Npm makes publishing packages as easy as installing them, it already has 64k packages. RubyGems lists 72k packages. The venerable Python package index lists only 41k.
History
Flying in the face of its "batteries included" motto, Python shipped without a package manager until 2014.
Until Pip, the de facto standard was a command easy_install. It was woefully inadequate. The was no command to uninstall packages.
Pip was a massive improvement. It had most the features of Ruby's Gem. Unfortunately, Pip was--until recently--ironically difficult to install. In fact, the problem remains a top Python question on Stack Overflow: "How do I install pip on Windows?"

And just to provide a contrast, there's also pip.

The Python Package Index (PyPI) seems to be standard:
To install a package:
pip install MyProject
To update a package
pip install --upgrade MyProject
To fix a version of a package pip install MyProject==1.0
You can install the package manager as follows:
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
References:
http://guide.python-distribute.org/
http://pypi.python.org/pypi/distribute

As a Ruby and Perl developer and learning-Python guy, I haven't found easy_install or pip to be the equivalent to RubyGems or CPAN.
I tend to keep my development systems running the latest versions of modules as the developers update them, and freeze my production systems at set versions. Both RubyGems and CPAN make it easy to find modules by listing what's available, then install and later update them individually or in bulk if desired.
easy_install and pip make it easy to install a module ONCE I located it via a browser search or learned about it by some other means, but they won't tell me what is available. I can explicitly name the module to be updated, but the apps won't tell me what has been updated nor will they update everything in bulk if I want.
So, the basic functionality is there in pip and easy_install but there are features missing that I'd like to see that would make them friendlier and easier to use and on par with CPAN and RubyGems.

There are at least two, easy_install and its successor pip.

As of at least late 2014, Continuum Analytics' Anaconda Python distribution with the conda package manager should be considered. It solves most of the serious issues people run into with Python in general (managing different Python versions, updating Python versions, package management, virtual environments, Windows/Mac compatibility) in one cohesive download.
It enables you to do pretty much everything you could want to with Python without having to change the system at all. My next preferred solution is pip + virtualenv, but you either have to install virtualenv into your system Python (and your system Python may not be the version you want), or build from source. Anaconda makes this whole process the click of a button, as well as adding a bunch of other features.

That'd be easy_install.

It's called setuptools. You run it with the "easy_install" command.
You can find the directory at http://pypi.python.org/

I don't see either MacPorts or Homebrew mentioned in other answers here, but since I do see them mentioned elsewhere on Stack Overflow for related questions, I'll add my own US$0.02 that many folks seem to consider MacPorts as not only a package manager for packages in general (as of today they list 16311 packages/ports, 2931 matching "python", albeit only for Macs), but also as a decent (maybe better) package manager for Python packages/modules:
Question
"...what is the method that Mac python developers use to manage their modules?"
Answers
"MacPorts is perfect for Python on the Mac."
"The best way is to use MacPorts."
"I prefer MacPorts..."
"With my MacPorts setup..."
"I use MacPorts to install ... third-party modules tracked by MacPorts"
SciPy
"Macs (unlike Linux) don’t come with a package manager, but there are a couple of popular package managers you can install.
Macports..."
I'm still debating on whether or not to use MacPorts myself, but at the moment I'm leaning in that direction.

On Windows install http://chocolatey.org/ then
choco install python
Open a new cmd-window with the updated PATH. Next, do
choco install pip
After that you can
pip install pyside
pip install ipython
...

Since no one has mentioned pipenv here, I would like to describe my views why everyone should use it for managing python packages.
As #ColonelPanic mentioned there are several issues with the Python Package Index and with pip and virtualenv also.
Pipenv solves most of the issues with pip and provides additional features also.
Pipenv features
Pipenv is intended to replace pip and virtualenv, which means pipenv will automatically create a separate virtual environment for every project thus avoiding conflicts between different python versions/package versions for different projects.
Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.
Automatically install required Pythons, if pyenv is available.
Automatically finds your project home, recursively, by looking for a Pipfile.
Automatically generates a Pipfile, if one doesn’t exist.
Automatically creates a virtualenv in a standard location.
Automatically adds/removes packages to a Pipfile when they are un/installed.
Automatically loads .env files, if they exist.
If you have worked on python projects before, you would realize these features make managing packages way easier.
Other Commands
check checks for security vulnerabilities and asserts that PEP 508 requirements are being met by the current environment. (which I think is a great feature especially after this - Malicious packages on PyPi)
graph will show you a dependency graph, of your installed dependencies.
You can read more about it here - Pipenv.
Installation
You can find the installation documentation here
P.S.: If you liked working with the Python Package requests , you would be pleased to know that pipenv is by the same developer Kenneth Reitz

In 2019 poetry is the package and dependency manager you are looking for.
https://github.com/sdispater/poetry#why
It's modern, simple and reliable.

Poetry is what you're looking for. It takes care of dependency management, virtual environments, running.