We Keep Coding

In need of a little assistance in configuring my python installation

Being a "normal" user (i.e. not overly familiar with code), I find myself in a bit of a python conundrum and would very much like to try to muster some kind of assistance with the following:
I use svtplay-dl and youtube-dl via homebrew in Mac OSX 10.14.6, which also requires python (among many other extensions such as ffmpeg et al). However, I now seem to have three different python versions installed, and since python3.5 is no longer supported, I thought I would delete this and make sure that homebrew/svtplay-dl/youtube-dl uses a path to python3.9.
When I type "which" followed by the following in Terminal:
python
python3.5
python3.9
...I get the following respective results:
/usr/local/bin/python
/Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5
/usr/local/bin/python3.9
Based upon this short background info, can someone advise how I best uninstall/remove python3.5 and get homebrew et al to use python3.9 every time python is required?
Any help with this would be very much appreciated!
/Mark

The normal answer is to use something like virtualenv (https://virtualenv.pypa.io/en/latest/) to isolate your python environment and dependencies locally.
You need to install virtualenv on your system, then you tell it what python version to install. Once activated, any dependencies are installed and isolated in that environment. Also see: https://gist.github.com/pandafulmanda/730a9355e088a9970b18275cb9eadef3
If you want to go a step further you can use docker (https://docs.docker.com/language/python/build-images/) to isolate everything, including the operating system, in that case you don't need virtualenv, but to start with I'd play around with virtualenv.

macOS Big Sur; Command Line Tools and Python 3.8.2

I am experiencing difficulty with installing Python correctly on my Mac (Fresh Install, I wiped my PC clean). I installed firstly
Home Brew
Xcode through App Store
Command line tools using command: "xcode-select --install"
On typing the following command:
which python3
Output is as follows:
gaurangsmacbookpro#Gaurangs-New-MacBook-Pro ~ % which python
/usr/bin/python
gaurangsmacbookpro#Gaurangs-New-MacBook-Pro ~ % which python3
/usr/bin/python3
upon running the following command which I was following off a blog from this site:
https://www.pyimagesearch.com/2016/12/05/macos-install-opencv-3-and-python-3-5/
brew install eigen tbb
Home brew installed python 3.9 dependencies so no when I type the following commands:
brew list python3
I get the following output:
gaurangsmacbookpro#Gaurangs-New-MacBook-Pro ~ % brew list python3
/usr/local/Cellar/python#3.9/3.9.1_8/bin/2to3
/usr/local/Cellar/python#3.9/3.9.1_8/bin/2to3-3.9
/usr/local/Cellar/python#3.9/3.9.1_8/bin/easy_install-3.9
/usr/local/Cellar/python#3.9/3.9.1_8/bin/idle3
/usr/local/Cellar/python#3.9/3.9.1_8/bin/idle3.9
/usr/local/Cellar/python#3.9/3.9.1_8/bin/pip3
/usr/local/Cellar/python#3.9/3.9.1_8/bin/pip3.9
/usr/local/Cellar/python#3.9/3.9.1_8/bin/pydoc3
/usr/local/Cellar/python#3.9/3.9.1_8/bin/pydoc3.9
/usr/local/Cellar/python#3.9/3.9.1_8/bin/python3
/usr/local/Cellar/python#3.9/3.9.1_8/bin/python3-config
/usr/local/Cellar/python#3.9/3.9.1_8/bin/python3.9
/usr/local/Cellar/python#3.9/3.9.1_8/bin/python3.9-config
/usr/local/Cellar/python#3.9/3.9.1_8/bin/wheel3
/usr/local/Cellar/python#3.9/3.9.1_8/Frameworks/Python.framework/ (3019 files)
/usr/local/Cellar/python#3.9/3.9.1_8/IDLE 3.app/Contents/ (8 files)
/usr/local/Cellar/python#3.9/3.9.1_8/lib/pkgconfig/ (4 files)
/usr/local/Cellar/python#3.9/3.9.1_8/libexec/bin/ (7 files)
/usr/local/Cellar/python#3.9/3.9.1_8/libexec/pip/ (480 files)
/usr/local/Cellar/python#3.9/3.9.1_8/libexec/setuptools/ (334 files)
/usr/local/Cellar/python#3.9/3.9.1_8/libexec/wheel/ (44 files)
/usr/local/Cellar/python#3.9/3.9.1_8/Python Launcher 3.app/Contents/ (16 files)
/usr/local/Cellar/python#3.9/3.9.1_8/share/man/ (2 files)
The contents of my ~./bash_profile file are as follows:
# Add Homebrew's executable directory to the front of the PATH
export PATH=/usr/local/bin:$PATH
export PATH="/usr/local/sbin:$PATH"
export PATH="/usr/local/opt/python#3.8/bin:$PATH"
What am I doing wrong? I just want a fresh clean install of Python 3.8 not 3.9 as there is no support for TF Lite on 3.9 yet. Can someone help me fix this mess? I will be very grateful!

I'm a bit late at the game, but I hope this will be useful to somebody. The best solution here is to use a virtual environment, but for the sake of argument and because it is a little bit more involved, I won't discuss that as an option here and will assume you just want a specific version installed on your machine where you can just add all the dependencies you need.
From looking at the page OP was using, the easiest solution here is probably to use a distribution like Anaconda if you are particularly interested in OpenCV or that type of libraries. Then it's just a case of setting up your path like export PATH=$HOME/anaconda/bin:$PATH or whatever path you used to install Anaconda with your shell of choice.
But let's imagine that Anaconda doesn't help you and you still need a specific version of Python, and you figure 3.8.2 is good enough for your needs and a newer version wouldn't work. Using the Python version that comes with the Xcode command line tools is probably a bad idea, just like using the system installed Python is. That version is really meant to support the Apple toolchain and while I've never run into any issues with it, I would not recommend it.
Next option is is to use homebrew. Great option since most people will already be using it, and you might already have python installed as part of a dependency but maybe like OP it's not the version you were hoping for. You can still make it work by installing an older version like so brew install python#3.8 however, it comes with a few caveats. Whenever one of the python dependencies is updated, you might need to unlink and relink it so that you can continue to use the correct version. Believe me, it's no way to live.
What I've found works (in combination with virtual environments is to use pyenv. Install it through homebrew, set your shell as described in the install doc and use it to install the python version you want (very easy) and then set the global interpreter to be whatever version you want and your system will stick to it until you set it to something else. It also works great in combination with tools like pipenv which allow you to manage virtual environments.

How to package python into virtualenv?

How to package Python itself into virtualenv? Is this even possible?
I'm trying to run python on a machine which it is not installed on, and I thought virtualenv made this possible. It activates, but can't run any Python.

When setting up the virtualenv (this can also be done if it already set up) simply do:
python -m virtualenv -p python env
And Python will be added to the virtualenv, and will become the default python of it.
The version of Python can also be passed, as python uses the first version found in the PATH.

virtualenv makes it convenient to use multiple python versions in different projects on the same machine, and isolate the pip install libraries installed by each project. It doesn’t install or manage the overall python environment. Python must be installed on the machine before you can install or configure the virtualenv tool itself or switch into a virtual environment.
Side note, consider using virtualenvwrapper — great helper for virtualenv.

You haven't specified the Operating System you are using.
In case you're using Windows, you don't use virtualenv for this. Instead you:
Download the Python embeddable package
Unpack it
Uncomment import site in the python37._pth file (only if you want to add additional packages)
Manually copy your additional packages (the ones you usually install with pip) to Lib\site-packages (you need to create that directory first, of course)
Such a python installation is configured in such a way that it can be moved and run from any location.
You only have to ensure the Microsoft C Runtime is installed on the system (but it almost always already is). See the documentation note:
Note The embedded distribution does not include the Microsoft C Runtime and it is the responsibility of the application installer to provide this. The runtime may have already been installed on a user’s system previously or automatically via Windows Update, and can be detected by finding ucrtbase.dll in the system directory.

You might need to install python in some location you have the permissions to do so.

How to make macports python use its own packages over another installation's?

I have spent days struggling to get a scientific Python environment running on Mac OS Lion. I tried the SciPack Superpack route, and also various manual installations via pip and easy_install, but still got errors trying to import or use various modules. Based on the advice in this Stackoverflow thread I set up a fresh installation using MacPorts.
However, when I run the macports Python, it is ignoring packages in the macports installation and instead trying to load incompatible packages from the old installation. I am absolutely sure that I am running the newly installed macports Python. I have checked the symlinks and have checked python_select and have launched Python by directly typing the path to the new installation. But when I try to import, say, statsmodels, it pulls in the old version from another directory.
Here are the contents of sys.path (edited for brevity):
['',
'/Library/Python/2.7/site-packages/mrjob-0.4.3_dev-py2.7.egg',
'/Library/Python/2.7/site-packages/statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg',
...
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/Library/Python/2.7/site-packages/twilio-3.6.6-py2.7.egg',
'/Library/Python/2.7/site-packages/six-1.6.1-py2.7.egg',
'/Library/Python/2.7/site-packages/httplib2-0.9-py2.7.egg',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
'/Library/Python/2.7/site-packages']
The macports installation is in /opt/local, while the older installation is under /Library/Python. As you can see, the older packages are higher up in the list, which means that they have higher priority.
Environmental variable PYTHONPATH is empty. If I do put anything into PYTHONPATH, that appears in sys.path after the /Library entries but before the /opt entries. So it does not solve the problem.
If I invoke the new python with the -S option, sys.path becomes:
['',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']
That succeeds in getting rid of the alien site packages entries, but it also nukes macports' site-packages entries so then I can't load anything.
I believe that the culprit is a file called /Library/Python/2.7/site-packages/easy-install.pth , with the following contents (again edited for brevity):
import sys; sys.__plen = len(sys.path)
./mrjob-0.4.3_dev-py2.7.egg
...
./statsmodels-0.6.0-py2.7-macosx-10.9-intel.egg
...
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
./twilio-3.6.6-py2.7.egg
./six-1.6.1-py2.7.egg
./httplib2-0.9-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
If I rename that file before launching macports python, python no longer is adding these alien packages to its sys.path. Now sys.path looks a lot like it does when I use "python -S" option except it has these additional entries at the end:
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyObjC',
'/Library/Python/2.7/site-packages'
Now I am able to load macports' own packages, say for example statsmodels. By checking statsmodels.__file__ I have confirmed that it is importing the local package and not the alien package.
However, I consider this a workaround/kludge rather than a real solution. A solution should make it behave as expected, which is: Macports' Python, launched from /opt/local/... installation, should prioritize packages installed via macports into the /opt/local/... directory tree and go looking elsewhere only if the module does not exist locally. So for example I would expect the /opt/local entries to come first in sys.path, with the /Library entries further down the list.
It seems like this should be the default behavior and I've seen a lot of comments here on Stackoverflow that just assert that this is how Macports Python behaves. So...how do I make it so?

There are at least two separate root causes for the problem you are seeing. One is that non-system OS X framework builds of Python 2.7, including MacPorts Python 2.7, deliberately include the system Python site-packages location, /Library/Python/2.7/site-packages, in sys.path, normally at the end of sys.path after that instance's own site-packages directory. This has been a debated feature in upstream Python and there is an open MacPorts issue to remove it (https://trac.macports.org/ticket/34763), the argument being that it is better to keep the system Python and the MacPorts Python totally separate.
The second root cause is the behavior of the original setuptools package and its easy_install command as supplied with OS X. As you have discovered, it does some fancy manipulation of sys.path by some magic tricks with .pth files, including easy-install.pth, to ensure that the packages that have been installed using easy_install show up first in sys.path and override other installed versions of those packages. Also as you have found, one way - and the easiest way - to remove that behavior is to delete the easy-install.pth file in /Library/Python/2.7/site-packages. That is assuming you don't want to use any of the packages installed there with the Apple-supplied system Python.
The long-term strategy to avoid this problem in the future is to make sure you do not use the Apple-supplied easy_install or easy_install-2.7 commands found in /usr/bin. They will install packages to /Library/Python for use by the system Python and will create or update easy-install.pth. In general, you should avoid use of easy_install all together. Its modern replacement is pip which provides better control and avoids tricks with .pth files. If MacPorts doesn't already provide a port for the Python package you want to install, usually of the form py27-xxxx, install and use the MacPorts py27-pip port instead of easy_install:
/opt/local/bin/pip-2.7 install xxxx
You can also use the usual MacPorts features like port select pip pip27.

Migrating to pip+virtualenv from setuptools

So pip and virtualenv sound wonderful compared to setuptools. Being able to uninstall would be great. But my project is already using setuptools, so how do I migrate? The web sites I've been able to find so far are very vague and general. So here's an anthology of questions after reading the main web sites and trying stuff out:
First of all, are virtualenv and pip supposed to be in a usable state by now? If not, please disregard the rest as the ravings of a madman.
How should virtualenv be installed? I'm not quite ready to believe it's as convoluted as explained elsewhere.
Is there a set of tested instructions for how to install matplotlib in a virtual environment? For some reason it always wants to compile it here instead of just installing a package, and it always ends in failure (even after build-dep which took up 250 MB of disk space). After a whole bunch of warnings it prints src/mplutils.cpp:17: error: ‘vsprintf’ was not declared in this scope.
How does either tool interact with setup.py? pip is supposed to replace easy_install, but it's not clear whether it's a drop-in or more complicated relationship.
Is virtualenv only for development mode, or should the users also install it?
Will the resulting package be installed with the minimum requirements (like the current egg), or will it be installed with sources & binaries for all dependencies plus all the build tools, creating a gigabyte monster in the virtual environment?
Will the users have to modify their $PATH and $PYTHONPATH to run the resulting package if it's installed in a virtual environment?
Do I need to create a script from a text string for virtualenv like in the bad old days?
What is with the #egg=Package URL syntax? That's not part of the standard URL, so why isn't it a separate parameter?
Where is #rev included in the URL? At the end I suppose, but the documentation is not clear about this ("You can also include #rev in the URL").
What is supposed to be understood by using an existing requirements file as "as a sort of template for the new file"? This could mean any number of things.

Wow, that's quite a set of questions. Many of them would really deserve their own SO question with more details. I'll do my best:
First of all, are virtualenv and pip
supposed to be in a usable state by
now?
Yes, although they don't serve everyone's needs. Pip and virtualenv (along with everything else in Python package management) are far from perfect, but they are widely used and depended upon nonetheless.
How should virtualenv be installed?
I'm not quite ready to believe it's as
convoluted as explained elsewhere.
The answer you link is complex because it is trying to avoid making any changes at all to your global Python installation and install everything in ~/.local instead. This has some advantages, but is more complex to setup. It's also installing virtualenvwrapper, which is a set of convenience bash scripts for working with virtualenv, but is not necessary for using virtualenv.
If you are on Ubuntu, aptitude install python-setuptools followed by easy_install virtualenv should get you a working virtualenv installation without doing any damage to your global python environment (unless you also had the Ubuntu virtualenv package installed, which I don't recommend as it will likely be an old version).
Is there a set of tested instructions
for how to install matplotlib in a
virtual environment? For some reason
it always wants to compile it here
instead of just installing a package,
and it always ends in failure (even
after build-dep which took up 250 MB
of disk space). After a whole bunch of
warnings it prints
src/mplutils.cpp:17: error: ‘vsprintf’
was not declared in this scope.
It "always wants to compile" because pip, by design, installs only from source, it doesn't install pre-compiled binaries. This is a controversial choice, and is probably the primary reason why pip has seen widest adoption among Python web developers, who use more pure-Python packages and commonly develop and deploy in POSIX environments where a working compilation chain is standard.
The reason for the design choice is that providing precompiled binaries has a combinatorial explosion problem with different platforms and build architectures (including python version, UCS-2 vs UCS-4 python builds, 32 vs 64-bit...). The way easy_install finds the right binary package on PyPI sort of works, most of the time, but doesn't account for all these factors and can break. So pip just avoids that issue altogether (replacing it with a requirement that you have a working compilation environment).
In many cases, packages that require C compilation also have a slower-moving release schedule and it's acceptable to simply install OS packages for them instead. This doesn't allow working with different versions of them in different virtualenvs, though.
I don't know what's causing your compilation error, it works for me (on Ubuntu 10.10) with this series of commands:
virtualenv --no-site-packages tmp
. tmp/bin/activate
pip install numpy
pip install -f http://downloads.sourceforge.net/project/matplotlib/matplotlib/matplotlib-1.0.1/matplotlib-1.0.1.tar.gz matplotlib
The "-f" link is necessary to get the most recent version, due to matplotlib's unusual download URLs on PyPI.
How does either tool interact with
setup.py? pip is supposed to replace
easy_install, but it's not clear
whether it's a drop-in or more
complicated relationship.
The setup.py file is a convention of distutils, the Python standard library's package management "solution." distutils alone is missing some key features, and setuptools is a widely-used third-party package that "embraces and extends" distutils to provide some additional features. setuptools also uses setup.py. easy_install is the installer bundled with setuptools. Setuptools development stalled for several years, and distribute was a fork of setuptools to fix some longstanding bugs. Eventually the fork was resolved with a merge of distribute back into setuptools, and setuptools development is now active again (with a new maintainer).
distutils2 was a mostly-rewritten new version of distutils that attempted to incorporate the best ideas from setuptools/distribute, and was supposed to become part of the Python standard library. Unfortunately this effort failed, so for the time being setuptools remains the de facto standard for Python packaging.
Pip replaces easy_install, but it does not replace setuptools; it requires setuptools and builds on top of it. Thus it also uses setup.py.
Is virtualenv only for development
mode, or should the users also install
it?
There's no single right answer to that; it can be used either way. In the end it's really your user's choice, and your software ideally should be able to be installed inside or out of a virtualenv; though you might choose to document and emphasize one approach or the other. It depends very much on who your users are and what environments they are likely to need to install your software into.
Will the resulting package be
installed with the minimum
requirements (like the current egg),
or will it be installed with sources &
binaries for all dependencies plus all
the build tools, creating a gigabyte
monster in the virtual environment?
If a package that requires compilation is installed via pip, it will need to be compiled from source. That also applies to any dependencies that require compilation.
This is unrelated to the question of whether you use a virtualenv. easy_install is available by default in a virtualenv and works just fine there. It can install pre-compiled binary eggs, just like it does outside of a virtualenv.
Will the users have to modify their
$PATH and $PYTHONPATH to run the
resulting package if it's installed in
a virtual environment?
In order to use anything installed in a virtualenv, you need to use the python binary in the virtualenv's bin/ directory (or another script installed into the virtualenv that references this binary). The most common way to do this is to use the virtualenv's activate or activate.bat script to temporarily modify the shell PATH so the virtualenv's bin/ directory is first. Modifying PYTHONPATH is not generally useful or necessary with virtualenv.
Do I need to create a script from a
text string for virtualenv like in the
bad old days?
No.
What is with the #egg=Package URL
syntax? That's not part of the
standard URL, so why isn't it a
separate parameter?
The "#egg=projectname-version" URL fragment hack was first introduced by setuptools and easy_install. Since easy_install scrapes links from the web to find candidate distributions to install for a given package name and version, this hack allowed package authors to add links on PyPI that easy_install could understand, even if they didn't use easy_install's standard naming conventions for their files.
Where is #rev included in the URL? At
the end I suppose, but the
documentation is not clear about this
("You can also include #rev in the
URL").
A couple sentences after that quoted fragment there is a link to "read the requirements file format to learn about other features." The #rev feature is fully documented and demonstrated there.
What is supposed to be understood by
using an existing requirements file as
"as a sort of template for the new
file"? This could mean any number of
things.
The very next sentence says "it will keep the packages listed in devel-req.txt in order and preserve comments." I'm not sure what would be a better concise description.

I can't answer all your questions, but hopefully the following helps.
Both virtualenv and pip are very usable. Many Python devs use these everyday.
Since you have a working easy_install, the easiest way to install both is the following:
easy_install pip
easy_install virtualenv
Once you have virtualenv, just type virtualenv yourEnvName and you'll get your new python virtual environment in a directory named yourEnvName.
From there, it's as easy as source yourEnvName/bin/activate and the virtual python interpreter will be your active. I know nothing about matplotlib, but following the installation interactions should work out ok unless there are weird hard-coded path issues.
If you can install something via easy_install you can usually install it via pip. I haven't found anything that easy_install could do that pip couldn't.
I wouldn't count on users being able to install virtualenv (it depends on who your users are). Technically, a virtual python interpreter can be treated as a real one for most cases. It's main use is not cluttering up the real interpreter's site-packages and if you have two libraries/apps that require different and incompatible versions of the same library.
If you or a user install something in a virtualenv, it won't be available in other virtualenvs or the system Python interpreter. You'll need to use source /path/to/yourvirtualenv/bin/activate command to switch to a virtual environment you installed the library on.
What they mean by "as a sort of template for the new file" is that the pip freeze -r devel-req.txt > stable-req.txt command will create a new file stable-req.txt based on the existing file devel-req.txt. The only difference will be anything installed not already specified in the existing file will be in the new file.