In most cases I think that Ansible engineers install pip packages 1) without using a virtualenv and 2) under root.
If we do this manually we would see a warning
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Typically when our Ansible automation becomes more advanced we would need additional pip packages to make Ansible modules work. More often than not this also requires additional OS packages to be installed. For example for python-ldap pip package on Ubuntu 18.04 requires
build-essential
python3-dev
python3-wheel
libsasl2-dev
libldap2-dev
libssl-dev
The way that Ansible is made to work on target nodes by installing additional pip packages as root while this clearly not the recommended way to use Python and Pip makes me wonder if there is not a better way to do this.
Should we not use virtualenv and another account than root for installing pip for Ansible?
There are probably multiple aspects to this. The one that came to my mind first, is this:
Using the "global" python outside of any venv, will probably work for the vast majority of users very well, while using a venv can lead to all kinds of unexpected behavior, if you are not familiar with the concept.
For example, if a venv was used by default, people will install python packages and then wonder why python claims they are not available when they try to import them in python on the host.
On the other hand, it is probably relatively easy to use a venv, if you want to do that. In any playbook, you can create the venv and then just update the ansible_python_interpreter variable:
- pip:
name: pip
virtualenv: /path/to/venv
- set_fact:
ansible_python_interpreter: /path/to/venv/bin/python
- setup:
Disclaimer: I just tried that very quickly and saw that it worked, so there might be glitches in some situations.
Obviously, it is not very neat to add something like this to every play, but it can be made a lot nicer, e.g. by creating the venv on the first setup and then using the interpreter_python key in ansible.cfg.
tl;dr:
Using global python is probably the way it works the best for most users, while "power-users" still have ways to achieve what they want with some additional actions.
Related
I'm getting familiar with nix by using it instead of brew on my Mac. I like it pretty much so far, but there is an issue with python environment. I can't figure out how to configure it properly.
I'm using powerline status, which is a python package itself, but also available as derivation (nixpkgs.python37Packages.powerline). So, I installed it together with python and some other packages:
# file: python.nix
with import <nixpkgs> {};
python3.withPackages (ps: with ps; [ pip pipenv powerline ipython ])
This is working as expected.
However, I'm also using some additional segments for powerline (ex. powerline-gitstatus), which are not available as derivations. So, I just installed them via pip install --user .... And then set PYTHONPATH=$HOME/.local/lib/python3.7/site-packages.
This is also working.
Apart from that I need to have awscli, which is a python package and available as standalone derivation (not nixpkgs.python37Packages.awscli), so I can't install it as a part of my custom python derivation, as far as I understand.
I tried to get it with nix-env -i awscli, but it didn't work together with set PYTHONPATH.
I tried to get it with pip install --user awscli, but it didn't work also.
In both cases it crashed because of lack of dependencies or version incompatibilities.
So, my question is: how to setup global python environment properly with nix?
I guess following well-documented very convenient approach could help in your case also.
How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
10.09.2020 Update:
Developing Python with Poetry & Poetry2nix: Reproducible flexible Python environments describes an other working with python on NixOS approach.
For our prod apps, we manage all packages in RPM. In a perfect yum-only world, this is fine.
However, each language or framework recently are deploying their own package managers. For python for example, I am almost always using pip and then having to build the packages for rpm again when going to production. Same goes for npm or gems.
Is this an issue that a lot of people here are running into? What is best way to avoid all this? Are people dealing with this by just sucking it up and packaging everything manually, or is there some automated way to integrate them?
I would love to hear thoughts on this.
I have used pyp2rpm to build rpm package for rstr module and I dislike random installations of some files via pip as well:
pyp2rpm -n rstr > ~/rpmbuild/SPECS/rstr.spec
rpmbuild -ba ~/rpmbuild/SPECS/rstr.spec
but I do not know if this is good enough for your production use.
You can install fpm: https://github.com/jordansissel/fpm
Effing package management! Build packages for multiple platforms (deb, rpm, etc) with great ease and sanity. http://fpm.readthedocs.io/en/latest/
Next you can run:
$ fpm -s python -t rpm pyramid
no value for epoch is set, defaulting to nil {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"python-pyramid-1.9a2-1.noarch.rpm"}
-s source type
-t type of package to create
and as last parameter the name of the package
https://fpm.readthedocs.io/en/latest/source/python.html
I prefer installing to a different sitelib than the default system one. Hence I do:
$ pip install --user django
Far from ideal, doesn't play with dnf/rpm/yum but at the same time doesn't mess with your system modules. And you can wipe them whenever you want.
I have three different Python 2.7s at:
/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
I use a number of packages that come from different sources. I am currently installing packages from port (MacPorts), easy_install, pip (installed by easy_install), and Mercurial. There are also some that I have to install from image or build from source. I have more control over those.
The problem is that easy_install and pip seem to be installing to one location (/Library/Frameworks/...) and MacPorts installs to another (/opt/local/Library/Frameworks/...).
What's my best action now? Delete /Library/Frameworks/.../python2.7 and move easy_install and pip to the MacPorts one at /opt/local/...? Link the two directories? Move the MacPorts installation to /Library/Frameworks/...?
How can I consolidate these Pythons? I have tried putting both site-packages locations in my path, but only certain packages are available only for one Python and not the other and others vice versa, and I need them all available at once.
It seems that you have control over the stuff you're building yourself. This is how I consolidate macports with pip:
I like using Macports for all my stuff, so I just make sure that pip and easy_install build into macports' installation of python (the one in /opt/local/...).
You can tell where pip and easy_install will install things by using:
readlink `which pip`
(those are backticks)
If you want pip to install to the macports direcectories, use macports to install pip:
sudo port install py-pip
Then, be sure that which pip points to something like:
askewchan#rock:~$ which pip
/opt/local/bin/pip
askewchan#rock:~$ readlink `which pip`
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip-2.7
From the comments below (thanks #Jonathan and #Ned) you can do the same with easy_install but its port is called py-distribute:
sudo port install py-distribute
But as far as I know, you never need to use easy_install because anything that can be easy_installed can be piped better.
Note the port descriptions:
askewchan#rock:Tracking {master *}$ port search *easy*install*
py-pip #1.2.1 (python, www)
An easy_install replacement
askewchan#rock:Tracking {master *}$ port search py*distribute
py-distribute #0.6.35 (python, devel)
Replacement for setuptools
I suggest deciding on one and only one Python for your development work ( personally, I use distribution from Python.org )
You can't get rid of /Library/Frameworks - that's the default OSX one, and you could break things
of the two remaining Pythons, I'm assuming one is Macports and the other is Python.org -- you need to choose which one you want to be your development env and to stick with that.
I would strongly recommend against using pip or easy_install from one Python to install modules for another. The reason is that there can be differences in the compile options. It can be hard enough as-is to get certain packages to compile on OSX properly -- if you start compiling against different binaries ( which might support different architectures ) you're just going to increase your headaches.
I personally chose the following path:
I use the Python.org package for all development.
On a terminal login, I run shell scripts to prioritize my Python choice
All of my projects have their own virtualenv , and I disable system packages
When starting to work on any project, I tend to have an environment setup script. I just type in go_myproject.source ; that cds me to the right directory and runs the source /path/to/virtualenv/bin/activate to get me set up for that project.
There's a tiny bit of overhead on getting things setup, but I have been in complete heaven ever since. Managing projects and not needing to worry about dependencies/upgrades for one thing killing something else is... blissful.
While not a general solution, I install Mercurial and other Python-based applications using virtualenv. In particular, pip and easy_install will install to the respective virtual environment only and not clutter any system folder. The downside is, of course, that I will have duplicates of some packages; the advantage is that I have a clean, self-contained environment with a known version of Python (which for things such as Mercurial and other mission-critical applications is more important for me).
Another downside is that I need to link individual applications to my personal bin directory or add the bin directories of the virtual environments to my path. (Personally, I manage this with some simple scripts that do the symlinking for me.)
I sugest to move all python installations to the one place and create symlinks.
After that configure python environment to avoid problems with imports and "visibility" of the modules.
Try to use commands:
# easy_install
env PYTHONPATH=/custom/path easy_install –install-dir /custom/path
#pip
pip install --install-option="--prefix=$PREFIX_PATH" package_name
I know about virtualenv and pip. But these are a bit different from bundler/carton.
For instance:
pip writes the absolute path to shebang or activate script
pip doesn't have the exec sub command (bundle exec bar)
virtualenv copies the Python interpreter to a local directory
Does every Python developer use virtualenv/pip? Are there other package management tools for Python?
From what i've read about bundler — pip without virtualenv should work just fine for you. You can think of it as something between regular gem command and bundler. Common things that you can do with pip:
Installing packages (gem install)
pip install mypackage
Dependencies and bulk-install (gemfile)
Probably the easiest way is to use pip's requirements.txt files. Basically it's just a plain list of required packages with possible version constraints. It might look something like:
nose==1.1.2
django<1.3
PIL
Later when you'd want to install those dependencies you would do:
$ pip install -r requirements.txt
A simple way to see all your current packages in requirements-file syntax is to do:
$ pip freeze
You can read more about it here.
Execution (bundler exec)
All python packages that come with executable files are usually directly available after install (unless you have custom setup or it's a special package). For example:
$ pip install gunicorn
$ gunicorn -h
Package gems for install from cache (bundler package)
There is pip bundle and pip zip/unzip. But i'm not sure if many people use it.
p.s. If you do care about environment isolation you can also use virtualenv together with pip (they are close friends and work perfectly together). By default pip installs packages system-wide which might require admin rights.
You can use pipenv, which has similar interface with bundler.
$ pip install pipenv
Pipenv creates virtualenv automatically and installs dependencies from Pipfile or Pipfile.lock.
$ pipenv --three # Create virtualenv with Python3
$ pipenv install # Install dependencies from Pipfile
$ pipenv install requests # Install `requests` and update Pipfile
$ pipenv lock # Generate `Pipfile.lock`
$ pipenv shell # Run shell with virtualenv activated
You can run command with virtualenv scope like bundle exec.
$ pipenv run python3 -c "print('hello!')"
There is a clone pbundler.
The version that is currently in pip simply reads the requirements.txt file you already have, but is much out of date. It's also not totally equivalent: it insists on making a virtualenv. Bundler, I notice, only installs what packages are missing, and gives you the option of giving your sudo password to install into your system dirs or of restarting, which doesn't seem to be a feature of pbundler.
However, the version on git is an almost complete rewrite to be much closer to Bundler's behaviour... including having a "Cheesefile" and now not supporting requirements.txt. This is unfortunate, since requirements.txt is the de facto standard in pythonland, and there's even Offical BDFL-stamped work to standardize it. When that comes into force, you can be sure that something like pbundler will become the de facto standard. Alas, nothing quite stable yet that I know of (but I would love to be proven wrong).
I wrote one — https://github.com/Deepwalker/pundler .
On PIP its pundle, name was already taken.
It uses requirements(_\w+)?.txt files as your desired dependencies and creates frozen(_\w+)?.txt files with frozen versions.
About (_\w+)? thing — this is envs. You can create requirements_test.txt and then use PUNDLEENV=test to use this deps in your run with requirements.txt ones alongside.
And about virtualenv – you need not one, its what pundle takes from bundler in first head.
Python Poetry is the closest to Ruby bundler as of 2020 (and already since 2018). It's already more than two years old, still very active, has great documentation. One might complain about curl-pipe-python-style being the recommended way of installing, but there are alternatives, e.g. homebrew on macOS.
Primary website: https://python-poetry.org/
Github: https://github.com/python-poetry/poetry
Documentation: https://python-poetry.org/docs/
It uses virtualenvs behind the scenes (in contrast to bundler), but it provides and uses a lock-file, takes care of sub dependencies, adheres to specified version constraints and allows automatically updating outdated packages. There's even autocompletion for your favorite shell.
With its use of a pyproject.toml file, it's also going a bit further than bundler (closer to a gemspec. It's also comparable to JavaScript's and TypeScript's npm and yarn).
Poetrify (a complementing project) helps converting projects from requirements.txt to pyproject.toml for Poetry.
The lock file can be exported to requirements.txt by poetry export -f requirements.txt > requirements.txt, if you need that for other tooling (or the unlikely case want to go back).
I'd say Shovel is worth a look. It was developed specifically to the Pythonish version of Rake. There's not a ton of commit activity on the project, but seems stable and useful.
You can use pipx to install and run Python Applications in Isolated Environments automatically.
You can use pipenv to create and manage a virtualenv for your projects automatically.
Both wraps pip with virtual environment tools and aiming for different use cases.
All of these are one of the most stared project listed in the github PyPA repository.
FYI: Debian bullseye/testing currently lacks pipx. But package from sid should work fine. (2021-06-19)
No, no all the developers use virtualenv and/or pip, but many developers use/prefer these tools
And now, for package development tools and diferent environments that is your real question. Exist any other tools like Buildout (http://www.buildout.org/en/latest/) for the same purpose, isolate your environment Python build system for every project that you manage. For some time I use this, but not now.
Independent environments per project, in Python are a little different that the same situation in Ruby. In my case i use pyenv (https://github.com/yyuu/pyenv) that is something like rbenv but, for Python. diferent versions of python and virtualenvs per project, and, in this isolated environments, i can use pip or easy-install (if is needed).
Can someone please explain to me what is going on with python in ubuntu 9.04?
I'm trying to spin up virtualenv, and the --no-site-packages flag seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with easy_install (which I've upgraded to setuptools 0.6c9) and everything seems to be installed to /usr/local/lib/python2.6/dist-packages
I assume that when installing a package using apt-get, it's placed in /usr/lib/python2.6/dist-packages/ ?
The issue is, there is a /usr/local/lib/python2.6/site-packages as well that just sits there being empty. It would seem (by looking at the path in a virtualenv) that this is the folder virtualenv uses as backup. Thus even thought I omit --no-site-packages, I cant access my local systems packages from any of my virtualenv's.
So my questions are:
How do I get virtualenv to point to one of the dist-packages?
Which dist-packages should I point it to? /usr/lib/python2.6/dist-packages or /usr/local/lib/python2.6/dist-packages/
What is the point of /usr/lib/python2.6/site-packages? There's nothing in there!
Is it first come first serve on the path? If I have a newer version of package XYZ installed in /usr/local/lib/python2.6/dist-packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/python2.6/dist-packages, which one gets imported when I import xyz? I'm assuming this is based on the path list, yes?
Why the hell is this so confusing? Is there something I'm missing here?
Where is it defined that easy_install should install to /usr/local/lib/python2.6/dist-packages?
Will this affect pip as well?
Thanks to anyone who can clear this up!
I believe Mike Orr's answer from the virtual-env mailing list seems to be the best. Note the OP published this question in both places.
Original content of mail:
Years ago Debian created /usr/local/lib/pythonVERSION/site-packages,
and compiled the Python binary to include it in the default search
path. Ubuntu followed Debian's lead as it normally does. The Python
developers did not like this because you'd get interference with a
locally-installed /usr/local/bin/python using the same site-packages
directory. Ubuntu finally decided to abandon site-packages and use
dist-packages instead, a name which they invented so it wouldn't
interfere with anything. The loing story is out there somewhere if
you google it, somewhere in the Python bug tracker or distutils SIG or
such.
The system works, at least if you use the Ubuntu virtualenv package.
Some people have had problems using a locally-installed virtualenv on
Ubuntu because the magic sys.path entries weren't being added or
something. I'm not sure about --no-site-packages because I never use
that option: I run PIL and mysqldb from the Ubuntu packages because it
can be hard to compile their C dependencies sometimes. (Need the
right header files, Python is ignoring the header files, etc.)
So Ubuntu Python packages go into
/usr/lib/pythonVERSION/dist-packages. Or that python-support
directory for some reason. Locally-installed Python packages go into
/usr/local/lib/pythonVERSION/dist-packages by default. Whenever I
install an Ubuntu 9.04 system I run:
$ sudo apt-get install python-setuptools (6.0c9)
$ sudo apt-get install python-virtualenv (1.3.3)
$ sudo easy_install pip
$ sudo pip install virtualenvwrapper
The virtualenvs work fine this way, although I haven't tried --no-site-packages.
I'm trying to spin up virtualenv, and the --no-site-packages flag
seems to do nothing with ubuntu. I installed virtualenv 1.3.3 with
easy_install (which I've upgraded to setuptools 0.6c9)
These versions are both in Ubuntu 9.04, so you're making it harder on
yourself by installing them locally.
and everything
seems to be installed to /usr/local/lib/python2.6/dist-packages
Yes
I assume that when installing a package using apt-get, it's placed in /
usr/lib/python2.6/dist-packages/ ?
Yes
Is it first come first serve on the path? If I have a newer
version of package XYZ installed in /usr/local/lib/python2.6/dist-
packages/ and and older one (from ubuntu repos/apt-get) in /usr/lib/
python2.6/dist-packages, which one gets imported when I import xyz?
I'm assuming this is based on the path list, yes?
sys.path is scanned in order. The only funny thing is that .pth eggs
get put earlier or later in the path than some people expect. But if
you're using pip for everything it can do (i.e. except to install pip
itself, precompiled eggs, and a snapshot of a local directory that's a
copy rather than an egg link), you won't have many .pth eggs anyway.
Why the hell is this so confusing? Is there something I'm
missing here?
It's not well documented. I figured it out by scanning the web.
Will this affect pip as well?
Yes, pip will automatically install to
/usr/local/lib/pythonVERSION/site-packages. Use "pip install -E
$VIRTUAL_ENV packagename" to install into a virtualenv.
I'd be tempted to hack it by making site-packages a link to dist-packages, but I guess this might affect other cases where you want to install some extension other than from the ubuntu dist. I can't think of another answer to 1 except tweaking virtualenv's sources (with both ubuntu and virtualenv being so popular I wouldn't be surprised to find tweaked versions already exist).
Re 2, if you're using /usr/local/bin/python you should use the /usr/local version of the lib (including site-packages) and conversely if you're using /usr/bin/python.
Re 3, there will be something there if you ever install an extension for /usr/bin/python from sources (not via easy_install or from ubuntu's distro).
Re 4, yes, earlier entries on the path take precedence.
Re 5, easy_install is easy only in its name -- it does so much dark magic that it's been carefully kept out of the standard python library despite its convenience because the consensus among us python committers is that deep dark magic for convenience is "easy" only on the surface.
Re 6, I think that's an ubuntu modification to easy_install -- if that's right then it's defined wherever Canonical or other ubuntu maintainers make their collective decisions.
Re 7, sorry, no idea -- I have no reasonably recent ubuntu at hand to check.
You really should not touch Ubuntu's Python installation unless you are building system admin tools, or building something that could be considered to be a new system service.
If you are using Ubuntu to develop or deploy Python applications, always build your own Python from source, tar it up, and use that for deployment. That way you will have all the directories in the right place and virtualenv will work normally. If you will deploy several Python apps on the server, then make your Python live in some place like /home/python or /opt/python or somewhere outside of your home directory. Make sure that you have write permissions for the developers group (users?) so that people can easily add packages.
This also allows you to have two tiers of packages. The ones that are your in-house standard tools can be installed in your Python distro and be part of the tarball that you deploy, and only the app-specific packages would be in a virtualenv.
Do not upgrade or modify the Ubuntu system installed Python.
Well I have a Ubuntu 9.04 and quickly tried setting up a couple sandboxes with site-packages and one without. And things are working fine.
The only difference in the approach I took is I used Ubuntu's python-virtualenv package (1.3.3). And presume that it is tweaked by Ubuntu team to suit Ubuntu setups.
To sum up disable easy_installed virtualenv for a while, use packaged python-virtualenv and see if that meets your expectations.
In fact we use similar setup for production without any problem. Rest is already answered by Alex.
Another way to fix it:
https://stackoverflow.com/a/17265840/202168
Have to remember to do that in each virtualenv where you need it, but doesn't rely on hacks or a special version of virtualenv