I’m trying to figure out how to install a second python environment alongside anaconda.
On windows I can just install python in a different folder stand reference the desired python environment using env variables. I’d like to do the same on Mac.
A virtual env won’t do the trick as it does not copy the standard library and other things. It needs to be a complete stand alone environment. I guess I could compile it, but is there an easier way?
Thank you very much for any input.
You can do that using pyenv.
It allows you to have several python versions, and even different distributions.
It works, mostly on user space. So, no additional requirements are needed (apart from compilation tools)
Related
I'm new to coding and I was wondering if someone could explain what exactly python interpreters/environments do and the how that relates to the python versions and packages that can be used in certain projects.
I was trying to code a twitter bot with python and I use VS code as my IDE. While trying to import tweepy into my python code, I noticed I kept getting an error that it couldn't import. After some googling I realized it was because I had to set the python interpreter of the python file to the one that had tweepy installed to. So I changed it and the error with tweepy was solved. But I noticed that I have a bunch of python interpreters in seemingly different locations (screenshot of the interpreter options I'm given). The interpreter that fixed the issue was the one in the pyenv path.
I had installed pyenv some months back because I wanted my terminal (I'm on macOS) to automatically launch python3 when I typed python into the terminal. However, I don't really know what it does beyond that.
So my questions are:
Why do I have multiple python interpreters? Is there a way to get rid of the ones and just keep the one from pyenv (like clean up the ones in /usr/local/bin/python3, /opt/homebrew/bin/python or /usr/bin/python3 since I'm not using them)? or should I not do that?
What exactly does pyenv do? Is it okay that my python libraries are getting installed to pyenv by default? Or should I change it so that it's getting installed to homebrew or one of the usr/bin paths?
Sorry, this is my first time asking a question here so I might not sound that cohesive.
The interpreter is the program which executes a python source file. This is a program like any other, and you can have various versions of it at once (indeed python is quite good at being self contained, and multiple versions will live alongside one another quite happily).
Macs ship with an outdated python 2, used for something in the system. If you remove or update it, things may break. So you install your own python 3 somewhere. If you want to load the new python when you type python in a shell, that shell's $PATH needs to point to the right python. You generally set this in something like ~/.profile.
Pyenv is a tool for managing multiple python interpreters. It's usually used on a per-project basis, to test code against multiple pythons. Further to complicate things, it's often used alongside a virtual environment tool like pipenv. Using it to avoid manually fiddling with $PATH is fine, but a slightly orthogonal use case. All pyenv does is to put little scripts with the same name as python executables somewhere in your $PATH where they supercede anything else. These scripts then figure out which python should get called in this case. So pyenv isn't installing anything, it's just working out which python is going to do it.
The solution to all your problems turns out to be quite simple:
python -m pip install abc
Get into the habit of installing stuff with the python you want to use already set up in the shell, and calling pip like that. That way whatever you use it will always be installed in the right env.
Many modern software has dependency on python language and they -as a consequence- install their own versions of python with the necessary libraries for each particular software to work properly.
In my case, I have my own python that I downloaded intentionally using anaconda distribution, but I also have the ones came with ArcGIS, QGIS, and others.
I have difficulties distinguishing which python -say- I am updating or adding libraries to when reaching them from the command line, and these are not environments but rather the full python packages.
What is the best way to tackle that?
Is there a way to force new software to create new environments within one central python distribution instead of loosing track of all the copies existing in my computer?!
Note that I am aware that QGIS can be downloaded now through conda, which reduces the size of my problem, but doesn't completely solve it. Moreover, that version of QGIS comes with its own issues.
Thank you very much.
as Nukala suggested, that's exactly what virtual environments are for. It contains a particular version of a python interpreter and a set of libraries to be used by a single (or sometimes multiple) project. If you use IDE:s such as Pycharm, it handles the venvs for you automatically.
You can use pyenv to manage python versions in your system. Using pyenv you can easily switch between multiple versions.
And as suggested - each project can create a virtual environment. You have multiple options here - venv, virtualenv, virtualenvwrapper, pipenv, poetry ... etc.
I'm running MacOS Sierra 10.12.6
By default the system came with Python 2.7.10
I installed Python 3.6.3 (with IDLE) so I can learn Python (3). I understand that this is normal as MacOS may rely on Python 2.x for some programs. Either way, Python3 runs just fine if I run python3 from the command line/terminal, or if I use IDLE (which defaults to Python 3).
Now I want to install some libraries like Beautiful Soup.
And I believe I can install it as follows:
pip3 install beautifulsoup4
which should automatically install it. However, I read that it's recommended to use virtualenv on Mac BEFORE I run the above command. As a newbie, I don't want to mess anything up on my PC, so can anyone point me out how I can do this correctly?
For example, I can follow this link: http://sourabhbajaj.com/mac-setup/Python/virtualenv.html
But I just want to write here to make sure I'm following the right article/commands before I do it. Just being super careful!
Also, can I make a folder with my "virtual environment" and then add sub-folders inside that for each project? Meaning, I don't need to do this everytime, I have one virtual environment and any project that I do just is a subfolder within that space so I can use any libraries that I installed. Just trying to grasp the concept.
Thanks!
Sorry to add confusion.. this can be a tough subject for someone starting out.
The official docs recommend venv, which is similar to, but slightly different than virtualenv.
I would strongly recommend pycharm. It will create your venv for you as part of your project, which you might find helpful.
[Edit: Some other virtual environment features of pycharm that will help you].
If you type in an import statement for a package that isn't installed, it will offer to install it for you.
typing alt-F12 will bring you up a console with your virtual environment active
It syncs up your requirements.txt document for you
It manages your virtual environment path for you (as long as you are running inside pycharm), helping avoid import problems that many newcomers have with virtual environments.
I am not affiliated with pycharm, btw -- I just think it is a great tool for python developers, especially for newcomers, and its treatment of virtual environments is especially helpful.
You create one virtualenv for each project as a way of keeping track of the specific dependencies to keep them minimal which then makes it easier when you want to share projects with other people.
But this is not something you need. No harm comes from installing packages in your real environment as well. So you can safely run
pip3 install beautifulsoup4
I'm setting up a new system for a group of Python rookies to do a specific kind of scientific work using Python. It's got 2 different pythons on it (32 and 64 bit), and I want to install a set of common modules that users on the system will use.
(a) Some modules work out of the box for both pythons,
(b) some compile code and install differently depending on the python, and
(c) some don't work at all on certain pythons.
I've been told that virtualenv (+ wrapper) is good for this type of situation, but it's not clear to me how.
Can I use virtualenv to set up sandboxed modules across multiple user accounts without having to install each module for each user?
Can I use virtualenv to save me some time for case (a), i.e. install a module, but have all pythons see it?
I like the idea of isolating environments, and then having them just type "workon science32", "workon science64", depending on the issues with case (c).
Any advice is appreciated.
With virtualenv, you can allow each environment to use globally installed system packages simply by omitting the --no-site-packages option. This is the default behavior.
If you want to make each environment install all of their own packages, then use --no-site-packages and you will get a bare python installation to install your own modules. This is useful when you do not want packages to conflict with system packages. I normally do this just to keep system upgrades from interfering with working code.
I would be careful about thinking about these as sandboxes, because they are only partially isolated. The paths to python binaries and libraries are modified to use the environment, but really that is all that is going on. Virtualenv does nothing to prevent code running from doing destructive things to the system. Best way to sandbox is set Linux/Unix permissions properly, and give them their own user accounts.
EDIT For Version 1.7+
The default for 1.7 is to not include system packages, so if you want the behavior of using system packages, use the --system-site-packages option. Check the docs for more info.
short version: how can I get rid of the multiple-versions-of-python nightmare ?
long version: over the years, I've used several versions of python, and what is worse, several extensions to python (e.g. pygame, pylab, wxPython...). Each time it was on a different setup, with different OSes, sometimes different architectures (like my old PowerPC mac).
Nowadays I'm using a mac (OSX 10.6 on x86-64) and it's a dependency nightmare each time I want to revive script older than a few months. Python itself already comes in three different flavours in /usr/bin (2.5, 2.6, 3.1), but I had to install 2.4 from macports for pygame, something else (cannot remember what) forced me to install all three others from macports as well, so at the end of the day I'm the happy owner of seven (!) instances of python on my system.
But that's not the problem, the problem is, none of them has the right (i.e. same set of) libraries installed, some of them are 32bits, some 64bits, and now I'm pretty much lost.
For example right now I'm trying to run a three-year-old script (not written by me) which used to use matplotlib/numpy to draw a real-time plot within a rectangle of a wxwidgets window. But I'm failing miserably: py26-wxpython from macports won't install, stock python has wxwidgets included but also has some conflict between 32 bits and 64 bits, and it doesn't have numpy... what a mess !
Obviously, I'm doing things the wrong way. How do you usally cope with all that chaos ?
I solve this using virtualenv. I sympathise with wanting to avoid further layers of nightmare abstraction, but virtualenv is actually amazingly clean and simple to use. You literally do this (command line, Linux):
virtualenv my_env
This creates a new python binary and library location, and symlinks to your existing system libraries by default. Then, to switch paths to use the new environment, you do this:
source my_env/bin/activate
That's it. Now if you install modules (e.g. with easy_install), they get installed to the lib directory of the my_env directory. They don't interfere with existing libraries, you don't get weird conflicts, stuff doesn't stop working in your old environment. They're completely isolated.
To exit the environment, just do
deactivate
If you decide you made a mistake with an installation, or you don't want that environment anymore, just delete the directory:
rm -rf my_env
And you're done. It's really that simple.
virtualenv is great. ;)
Some tips:
on Mac OS X, use only the python installation in /Library/Frameworks/Python.framework.
whenever you use numpy/scipy/matplotlib, install the enthought python distribution
use virtualenv and virtualenvwrapper to keep those "system" installations pristine; ideally use one virtual environment per project, so each project's dependencies are fulfilled. And, yes, that means potentially a lot of code will be replicated in the various virtual envs.
That seems like a bigger mess indeed, but at least things work that way. Basically, if one of the projects works in a virtualenv, it will keep working no matter what upgrades you perform, since you never change the "system" installs.
Take a look at virtualenv.
What I usually do is trying to (progressively) keep up with the Python versions as they come along (and once all of the external dependencies have correct versions available).
Most of the time the Python code itself can be transferred as-is with only minor needed modifications.
My biggest Python project # work (15.000+ LOC) is now on Python 2.6 a few months (upgrading everything from Python 2.5 did take most of a day due to installing / checking 10+ dependencies...)
In general I think this is the best strategy with most of the interdependent components in the free software stack (think the dependencies in the linux software repositories): keep your versions (semi)-current (or at least: progressing at the same pace).
install the python versions you need, better if from sources
when you write a script, include the full python version into it (such as #!/usr/local/bin/python2.6)
I can't see what could go wrong.
If something does, it's probably macports fault anyway, not yours (one of the reasons I don't use macports anymore).
I know I'm probably missing something and this will get downvoted, but please leave at least a little comment in that case, thanks :)
I use the MacPorts version for everything, but as you note a lot of the default versions are bizarrely old. For example vim omnicomplete in Snow Leopard has python25 as a dependency. A lot of python related ports have old dependencies but you can usually flag the newer version at build time, for example port install vim +python26 instead of port install vim +python. Do a dry run before installing anything to see if you are pulling, for example, the whole of python24 when it isn't necessary. Check portfiles often because the naming convention as Darwin ports was getting off the ground left something to be desired. In practice I just leave everything in the default /opt... folders of MacPorts, including a copy of the entire framework with duplicates of PyObjC, etc., and just stick with one version at a time, retaining the option to return to the system default if things break unexpectedly. Which is all perhaps a bit too much work to avoid using virtualenv, which I've been meaning to get around to using.
I've had good luck using Buildout. You set up a list of which eggs and which versions you want. Buildout then downloads and installs private versions of each for you. It makes a private "python" binary with all the eggs already installed. A local "nosetests" makes things easy to debug. You can extend the build with your own functions.
On the down side, Buildout can be quite mysterious. Do "buildout -vvvv" for a while to see exactly what it's doing and why.
http://www.buildout.org/docs/tutorial.html
At least under Linux, multiple pythons can co-exist fairly happily. I use Python 2.6 on a CentOS system that needs Python 2.4 to be the default for various system things. I simply compiled and installed python 2.6 into a separate directory tree (and added the appropriate bin directory to my path) which was fairly painless. It's then invoked by typing "python2.6".
Once you have separate pythons up and running, installing libraries for a specific version is straightforward. If you invoke the setup.py script with the python you want, it will be installed in directories appropriate to that python, and scripts will be installed in the same directory as the python executable itself and will automagically use the correct python when invoked.
I also try to avoid using too many libraries. When I only need one or two functions from a library (eg scipy), I'll often see if I can just copy them to my own project.