We Keep Coding

Multiple versions of Python in PATH

I've installed Python 3.7, and since installed python 3.8.
I've added both their folders and script folders to PATH, and made sure 3.8 is first as I'd like that to be default.
I see that the Python scripts folder has pip, pip3 and pip3.8 and the python 3.7 folder has the same (but with pip3.7 of course), so in cmd typing pip or pip3 will default to version 3.8 as I have that first in PATH.
This is great, as I can explicitly decide which pip version to run. However I don't know how to do to the same for Python. ie. run Python3.7 from cmd.
And things like Jupyter Notebooks only see a "Python 3" kernel and don't have an option for both.
How can I configure the PATH variables so I can specify which version of python3 to run?

What OS are you running? If you are running linux and used the system package panager to install python 3.8 you should be able to invoke python 3.8 by typing python3.8. Having multiple binaries named python3 in your PATH is problematic, and having python3 in your PATH point to python 3.8 instead of the system version (which is likely a lower version for your OS) will break your system's package manager. It is advisable to keep python3 in your PATH pointing to whatever the system defaults to, and use python3.8 to invoke python 3.8.
The python version that Jupyter sees will be the version from which you installed it. If you want to be able to use Jupyter with multiple python versions, create a virtual environment with your desired python version and install Jupyter in that environment. Once you activate that specific virtual env you will be sure that the jupyter command that you invoke will activate the currect python runtime.

I recommend you use pyenv a great tool for manage multiple python versions on the same system. Once installed you need to create a virtualenv, then activate the virtualenviroment and there you can install any libraries you want in a safe way.
By the way also come with an automatic installer pyenv-installer
Regards

Poetry doesn't use the correct version of Python

I've recently installed both Pyenv and Poetry and want to create a new Python 3.8 project. I've set both the global and local versions of python to 3.8.1 using the appropriate Pyenv commands (pyenv global 3.8.1 for example). When I run pyenv version in my terminal the output is 3.8.1. as expected.
Now, the problem is that when I create a new python project with Poetry (poetry new my-project), the generated pyproject.toml file creates a project with python 2.7:
[tool.poetry]
name = "my-project"
version = "0.1.0"
description = ""
authors = ["user <user#email.com>"]
[tool.poetry.dependencies]
python = "^2.7"
[tool.poetry.dev-dependencies]
pytest = "^4.6"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
It seems that Poetry defaults back to the system version of Python. How do I change this so that it uses the version installed with Pyenv?
Edit
I'm using MacOS, which comes bundled with Python 2.7. I think that might be causing some of the issues here. I've reinstalled Python 3.8 again with Pyenv, but when I hit Poetry install I get the following error:
The currently activated Python version 2.7.16 is not supported by the project (^3.8).
Trying to find and use a compatible version.
[NoCompatiblePythonVersionFound]
Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.
Should I create an environment explicitly for the project using Pyenv or should the project be able to access the correct Python version after running pyenv local 3.8.1.? When I do the latter, nothing changes and I still get the same errors.

pyproject.toml is used to define all the dependencies for your project, including the supported python version.
The line your complaining about is just saying that the versions of python supported by the project is python2.7 or greater, this is independent of what versions of python you've installed with pyenv.
python = "^2.7"
If you want to update the versions of python supported by the project you can edit the file directly and run poetry update.
If you want to use multiple versions of python you need to make sure poetry is using the correct dependencies for the version of python you are using. To change the specific version poetry is using you should use poetry env,
poetry env list show the versions of python poetry can use
poetry env use <python> switches poetry to use that version.
For instance on my machine poetry has 3 virtual environments installed and is using the one associated with python3.6:
↪ poetry env list
sipy-a9sqc5pb-py3.6 (Activated)
sipy-a9sqc5pb-py3.7
sipy-a9sqc5pb-py3.8
I'm not sure how these virtual environments with interact with the shivs used by pyenv but their docs have a section relating to it
Managing Virtual Environments
There is a pyenv plugin named pyenv-virtualenv which comes with various features to help pyenv users to manage virtual environments created by virtualenv or Anaconda. Because the activate script of those virtual environments are relying on mutating $PATH variable of user's interactive shell, it will intercept pyenv's shim style command execution hooks. We'd recommend to install pyenv-virtualenv as well if you have some plan to play with those virtual environments.

Alright, I figured the problem. A little embarrassingly, I had not run pyenv shell 3.8.1 before running any of the other commands. Everything works now. Thank you all for your efforts.

you can specify an explicit python executable for poetry using
poetry env use <path to python executable>
This worked for me.

On my machine I was able to fix the "currently activated Python version is not supported by the project" error by reinstalling Poetry:
curl -sSL https://install.python-poetry.org | python3 - --uninstall
curl -sSL https://install.python-poetry.org | python3 -
After that,poetry was able to find the correct version installed by pyenv.

In my case, I had to delete and recreate the virtualenv used by poetry. This is because I added the python version restrictions (e.g. python = ">=3.6.2 <3.7") after creating the virtualenv.
Steps
Delete the original one: run poetry env remove myApp-XkghI9p6-py3.6
Run any poetry step, to create it, or run poetry shell, and confirm poetry run python --version is the correct version.

Even though this issue has been resolved, I am writing this for somebody who comes across this problem again.
After all attempts my python -V always resulted in 2.7 and no discussions mentioned running pyenv shell (surprising to me!)
Adding pyenv to path
$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
In my case I had to add it to .bashrc and not bash_profile.
https://ggkbase-help.berkeley.edu/how-to/install-pyenv/
Worked!

In my case, the environment was messed up in some way that poetry failed to activate the virtualenv properly.
Try using a different shell: perhaps, sh, or zsh. If everything works in that shell, this proves that your environment is as messed up as mine was :)
Use this command in both shells:
$ env
and try to spot the difference

You can remove the python version from pyproject.toml file and then run Poetry install

What worked for me was to run python3.8 -m poetry install.

Use python 3.8 instead of 3.6 on elementary OS (or Ubuntu)

I recently started using elementary OS. It's based on Ubuntu.
During installation it installs python 3.6.
I installed python 3.8 manually by installing the following packages: python3.8, python3.8-dev, python3.8-minimal, python3.8-venv.
I also updated the link to the python binary with:
sudo ln -sf /usr/bin/python3.8 /usr/bin/python3
After this a couple of things stopped working.
For example when I tried to execute a non-existing command it didn't print the error message that it cannot found the command, but it displayed a python stack trace. That one I solved with:
cd /usr/lib/python3/dist-packages
sudo ln -s apt_pkg.cpython-36m-x86_64-linux-gnu.so apt_pkg.so
sudo ln -s apt_inst.cpython-36m-x86_64-linux-gnu.so apt_inst.so
As you can see the python error was because it couldn't find the compiled binaries for the apt module.
So this issue was solved, but there are a couple of similar ones, and none of them can be solved this way, as the module binary is not compatible with python 3.8.
Is it possible to remove python 3.6 completely and override it with 3.8 so that the module binaries also get updated? Or can python 3.8 and 3.6 coexist? I would be fine with the /usr/bin/python3 link pointing to python 3.6 and I would manually execute /usr/bin/python3.8 or create a different alias or link for it. However when I print out the sys.path with /usr/bin/python3.8 I get this:
/usr/lib/python38.zip
/usr/lib/python3.8
/usr/lib/python3.8/lib-dynload
/home/{username}/.local/lib/python3.8/site-packages
/usr/local/lib/python3.8/dist-packages
/usr/lib/python3/dist-packages
The trouble is with the last one. That's where the modules are with the 3.6 compatible so files. Can I somehow force python 3.8 to completely ignore the last module search path (without always stating sys.path.remove in my scripts) ?

I recommend you undo your symlink overwriting your systemwide version of Python 3. As you have found, replacing the python3 executable might create some problems, as it is used under the hood.
In general, you should leave your system python[3] installation alone for this reason and it is common to use virtual environments. This can be done as follows (assuming you are in some project directory):
python3.8 -m venv venv
source venv/bin/activate
The first command will create a virtual environment ('venv') in the directory venv. The second command will 'activate': now python (and in this case, python3 and python3.8) all refer to your original python3.8 in the context of this shell. You will have to repeat this if you launch a new shell.
This will also allow you to install packages using pip without cluttering your system installation. The use of virtual environments and pip is an incredibly common workflow in the Python development world.
In terms of shell and 'global' Python management, you can also use pyenv to manage your Python versions and what is available in the shell. pyenv is quite nice if you want to run a particular version of python, say 3.8.0 but not 3.8.1.

How to install different versions of Python in Mac (OS X) [duplicate]

I want to have multiple installs of Python: 2.1, 2.4, 2.7, 3.x
My IDE is Eclipse (Helios)/Pydev on MacOSX, which works great. I have a couple of Python codebases that are/will be running on different versions of Python. Also, I like Eclipse PyDev's crosslinking from source-code to documentation.
The standard recommendation seems to be: use virtualenv, and keep the installs totally separate from the builtin MacPython (2.6.1). Eclipse should never be pointing to the MacPython install. (Should PYTHONPATH even be set in such an environment?)
Before I get on with virtualenv, is there anything else I should know about this?
virtualenv doesn't impose any overhead, and I shouldn't be worried with occasional comments about breakage to nose, coverage etc?
I'm interested in hearing from Eclipse or Pydev users on MacOS.
Also if anyone has other tips on migrating a Python codebase from 2.1 -> 2.7.

A good way is to use macport to install the different version of pythons. It will have different versions of all packages for all versions of pythons that you want. They will be installed in /opt/local/.
So in Eclipse with PyDev for a particular project you can right click into the name of the project -> properties: There, in the left you choose PyDev - Interpreter/Grammar. And you click the link Click here to configure an interpreter not lister. You click on new on the top right of the new window. You give the name of the version of python you want to create the interpreter for, if you have already installed it with all required package through macport. And in the field Intepreter Executable you give the path: /opt/local/bin/pythonX.X .
After, in the previous option window: PyDev - Intepreter/Grammar, you can choose in the Interpreter menu the python version that you just installed.
For the shell, and the default path pointing to /usr/bin/python you must use: python_select (installed through macport) instead of playing with the env variables. Afterward you can use macport to update/install new packages, make sure everything is alway clean.

Having Home Brew already installed in your system, I recommend you pyenv. Most of the following information comes from this quick installation/use guide:
Installing pyenv + pyenv-virtualenv
Go to your terminal and install pyenv and the external libraries needed by Python:
brew update
brew install pyenv openssl readline sqlite3 xz zlib
Add pyenv init to your shell to enable shims and autocompletion running the following command:
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
Install the pyenv-virtualenv plugin so you can create virtual environments for each version of Python:
git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
Reset your terminal to apply the changes exec "$SHELL" or just close your terminal and open a new one.
Gettings different Python versions
We can list all available versions with pyenv install --list:
pyenv install --list
Available versions:
2.7.15
3.0.1
..
3.7.2
For installing the Python version 3.7.2:
pyenv install 3.7.2
With pyenv versions you can see all the Python versions installed in your computer:
root#Host ~$ pyenv versions
* 3.7.2
Creating a virtual environment
In the previous step we have downloaded the Python 3.7.2 interpreter, we can now use it to create an isolated virtual environment. This is very useful for software development, keeping each project completely isolated from the others:
pyenv virtualenv 3.7.2 MY_VIRTUALENV_NAME
To list all your virtual environments:
pyenv virtualenvs
To activate a virtual environment:
pyenv activate MY_VIRTUALENV_NAME
To deactivate the currently active virtual environment:
pyenv deactivate

From the README text file of python
Installing multiple versions
On Unix and Mac systems if you intend
to install multiple versions of Python
using the same installation prefix
(--prefix argument to the configure
script) you must take care that your
primary python executable is not
overwritten by the installation of a
different version. All files and
directories installed using "make
altinstall" contain the major and
minor version and can thus live
side-by-side. "make install" also
creates ${prefix}/bin/python which
refers to ${prefix}/bin/pythonX.Y. If
you intend to install multiple
versions using the same prefix you
must decide which version (if any) is
your "primary" version. Install that
version using "make install". Install
all other versions using "make
altinstall".
For example, if you want to install
Python 2.5, 2.6 and 3.0 with 2.6 being
the primary version, you would execute
"make install" in your 2.6 build
directory and "make altinstall" in the
others.
Virtualenv is an option but you could use the above mentioned option instead of venv which seems much simpler.

Personally, I use conda to create multiple environments (mostly, you create a new env, activate it and install the packages you want there -- using conda itself if possible or pip if it's not available in conda). See: https://conda.io/docs/installation.html.
After you have an environment created, you just need to add a new interpreter inside PyDev which points to the interpreter in the created environment. Ssee: http://www.pydev.org/manual_101_interpreter.html.

pytest running with another version of python

I've installed pyenv and have different versions of python installed with it:
$ pyenv versions
system
2.7.1
3.2.5
3.5.0
3.5.1
* 3.5.2
I use the following command to switch to python 3.5.2:
pyenv shell 3.5.2
And when I check the python version this is what I get:
$ python --version
Python 3.5.2
But when I run pytest, it still runs under python 2.7.6:
pytest -v
==================================================================== test session starts ====================================================================
platform linux2 -- Python 2.7.6, pytest-3.0.3, py-1.4.31, pluggy-0.4.0 -- /usr/bin/python
Why is pytest running under the older version?

Bottom line: run
python -m pytest, or
py.test-<version> if your alternative Python and pytest are installed with system package manager, or
if your alternative Python has been installed with pyenv, switch with pyenv to that version and make sure you have pytest installed for it. Then you can just run pytest.
since the pip executable is also among the ones being switched, you need to switch to the alternative Python before installing pytest for it, too.
As I can see, /usr/bin/pytest (that belongs to the system package manager's python-pytest package) has a shebang !#/usr/bin/python since it corresponds to the system python's installation.
pyenv, as its README.md says, does not replace /usr/bin/python - because it indeed should not be replaced to avoid breaking system packages.
Instead, it adds its directory to PATH and inserts a launcher there (called "shim") which is what gets invoked when you type "python". As you probably guessed, this hack is ignored by a shebang like the above - as it should.
Running python -m pytest will make whichever python that launches itself use the package from its installation.
Alternatively, pytest for your other Python version may include versioned executables on the PATH named py.test-<version> (e.g. py.test-3 or py.test-3.6) depending on the way you installed it.
If it's from a system package manager's package for nonstandard python - like python36-pytest - this is virtually guaranteed.
I checked that if you install a version with pip, it only creates an unversioned executable (though you can create a versioned one yourself). Moreover, if you install the same package for a different Python version but with the same --prefix, it will overwrite the existing one's executable!
pyenv's suggested way seems to be to install all python versions of interest and packages for them under ~/.pyenv/versions.
This is not applicable for the system's Python but the default /usr/local can be used for it.
Once you switch to an alternative Python version, it claims to create shims for all scripts (including pip!) that are currently installed for that version, so invoking those scripts without a path would run those shims.
So, if a package (and thus its script) is not installed for the alternative version but installed for system version, trying to run its executable would "fall through" to /usr/local with just the result you're seeing now.

I found this related question. For them it worked with this:
python -m pytest tests/my_test.py
I hope it works

I just wanted to run "black" for Python 2.7, but I have installed it in pyenv's Python 3.7.9. It worked correctly with the script /usr/local/bin/black27:
PYENV_VERSION=3.7.9 black -t py27 "$#"
It didn't work with:
pyenv shell 3.7.9
black -t py27 "$#"
nor
pyenv shell 3.7.9
pyenv exec black -t py27 "$#"