I recently installed pipenv using the following command: pip3 install --user pipenv. (It's also worth mentioning that I'm following Python's official guide here.) Most online resources seem to indicate that the default installation directory for user-scoped packages is at ~/.local/bin. However, it would seem that my installation of pipenv resides in ~/Library/Python/3.6/bin. I'm concerned that keeping the installation in a version specific directory (i.e. Python 3.6) could lead to problems down the road. What happened? Should I be worried?
This is entirely correct behaviour, and not something that you need to worry about.
Python packages with native compiled extensions are tied to the specific Python version into which it is installed and should not be shared. Because you can't detect a-priori what package will contain native extensions, all Python packages are installed in a version-specific location.
The --user switch installs in the User Scheme location:
With Python 2.6 came the "user scheme" for installation, which means that all Python distributions support an alternative install location that is specific to a user. The default location for each OS is explained in the python documentation for the site.USER_BASE variable. This mode of installation can be turned on by specifying the --user option to pip install.
You can always list your USER_BASE location by running:
python3 -m site
(using the same Python binary as tied to your pip command).
The Python module search path automatically includes the user location, and because that location is Python version (major.minor) specific, won't interfere with other Python versions.
~/Library/Python/3.6/ is the Mac OS X specific path used when you have a framework build. You can override the path by setting the PYTHONUSERBASE environment variable.
Related
I have a Windows 7 machine running Python 3.8.5 with a very large number of physics/electronics/data analysis/simulation packages. As it turned out, I must have - for some inexplicable reason - installed the 32-bit version of Python instead of the 64-bit one despite having a 64-bit system. And I didn't notice until very recently when I was trying to install some packages that require 64-bit Python. Hence I've now downloaded and installed the latest Python version that is supported by Windows 7, which seems to be 3.8.10.
Question: What is the easiest and also fail-safe way to reinstall all the user packages - that I currently have under 3.8.5 - to 3.8.10?
For some reason, I couldn't find any "canonical" solution for this online. As it seems, Python does not come with any built-in support for updating or system migration and I'm honestly wondering why...
Anyway, my first idea was to get a list of all user (= "local"?) packages currently installed under 3.8.5, but I don't know how. Reason: Doing help('modules') inside the interpreter will list all packages and I don't see a way to "selectively apply" pip to a specific Python version, e.g. something like python-3.8.5 -m pip list --local is not supported.
After getting a list of the user packages, I was thinking to pack it into a batch command pip install package_1 package_2 <...> package_N, thus reinstalling everything to Python 3.8.10. And afterwards uninstalling Python 3.8.5 and removing all environment variables from system PATH.
Is this the proper way to do this?
Anyway, my first idea was to get a list of all user (= "local"?) packages currently installed under 3.8.5, but I don't know how.
Create a list of installed packages with pip freeze > pkglist.txt or pip list --format=freeze. If you already have one, that's great.
Then uninstall 32-bit Python 3.8.5 and clean your path for all Python related variables. Now, install 64-bit Python 3.8.10.
After reinstalling, you can install back all the packages with pip install -r pkglist.txt and it will restore the exact versions of the packages.
If you insist on having both 32-bit and 64-bit versions installed and also have the Python Launcher installed, you could invoke 32 and 64 bit versions separately with py -3.8-64 -m pip and py -3.8-32 -m pip.
I don't see a way to "selectively apply" pip to a specific Python version.
This is possible with the Python Launcher on Windows. But only between major/minor versions and not the patch versions according to its help message.
I would also recommend creating a virtual environment this time before installing the packages and leaving the root environment alone. You can create one named venv with just python -m venv venv, activate it with ./venv/Scripts/activate and proceed with the installation of packages.
Nope, doesn't work. After installing the packages with the newer Python version in PATH, e.g. Jupyter won't start.
If the Jupyter error persists, you could try pinning packages to their most recent patch/minor versions to update them and yet not break your code.
As a last resort, you could try installing Python 3.10 alongside your current Python installation (without uninstall or editing the PATH) and then installing the absolute latest versions of the packages in a 3.10 virtual environment to see if it works for you. You would invoke the two versions with Py Launcher, e.g. py -3.10 and py -3.8.
If I understood correctly, you have multiple packages like NumPy, pandas etc. installed on your machine, and you want to reinstall them "automatically" on a fresh installation of python.
The method (I use) to perform such an operation is by creating a file named setup.py which includes a list of all the packages.
Bellow, I am attaching an example of such a file I use in one of my projects:
from setuptools import setup, find_packages
setup(
name='surface_quality_tools',
version='0.1',
install_requires=["matplotlib", "psutil", "numpy", "scipy", "pandas", "trimesh", "pyglet", "networkx", "protobuf",
"numpy-stl", "sklearn", "opencv-python", "seaborn", "scikit-image", "flask", "tqdm", "pytest"],
package_data={'': ['*.json']},
packages=find_packages(include=[])
)
to run the installation you should open a command prompt from inside the project directory and run:
pip install -e .
You can find a nice example in this blog page
One common way of handling packages in Python is via virtual environments. You can use Anaconda (conda), venv or any of several other solutions. For example, see this post:
https://towardsdatascience.com/virtual-environments-104c62d48c54#:~:text=A%20virtual%20environment%20is%20a,a%20system%2Dwide%20Python).
The way this works in by keeping the Python interpreter separate from the virtual environment that contains all the necessary packages.
Probably the main reason Python doesn't feature migration tools (at least as part of standard library) is because pip - the main package tool - doesn't handle conflict resolution all too well. When you update a version of Python it might so happen (especially with niche packages) that some of them won't work any more and pip often won't be able to solve the dependencies. This is why it's a good idea to keep a separate venv for different Python versions and different projects.
The other tool you could use for easy migration is Docker which is a semi-virtual machine working on top of your host OS and containing usually some linux distribution, Python along with the necessary packages necessary for running and development.
It takes a bit of time to set up a container image initially but afterwards setting everythin on a new machine or in the cloud becomes a breeze.
Listing currently installed packages is done via pip freeze command, the output of which you can then pipe into a file to keep a record of project requirements, for example pip freeze > requirements.txt.
I have python2.7 and python3.6 installed side by side in my computer. Now when I install a package using "pip install", how can I know in which python's site-packages is my package going to be installed?
Thank you.
When you have both version 2 and 3 installations pip and pip3 differentiate the target installtion.
For installing anything on Python 3(versions 3.5 and above) use pip3
for Python 2.7 use pip
Make sure python path is set in environment variables too.
also you can use where pip or which pip as #mshsayem mentioned.
Additional Reference
if you use virtualenv, the modules are located in:
{path_to_your_virtualenv}/lib/python{your_python_version}/site-packages/
and if you don't use virtualenv, normally are installed in:
/usr/local/lib/python{your_python_version}
You have to use pip3 for install python3 modules.
Check where a specific package is installed by:
pip3 show <package_name>
List all installed packages with install locations by:
pip3 list -v
Check the install location used by default when installed without sudo:
pip3 --version
and the location for packages installed with sudo, meaning system-wide installation:
sudo pip3 --version
You can find the location of pip by which pip. Then you view the pip executable header using head `which pip` or using your preferred editor. You can find the python interpreter location on the first line. You may have a pip2 and a pip3 executable.
By the way, you can run pip as a python module by python -m pip <command>. In this way, you can specify your python interpreter.
The answer to you question is divided to two parts:
1. Which python version the native terminal selects for me?
2. How do I specify which python version to use?
Which python version the native terminal selects for me?
In windows, the default pip that will be used is the one associated with the default python version you use. You can edit it in the PATH environmental variable (Start->find-type "Environmental" and click "Edit system variables"). Look the PATH variable and see which version of python is listed. If both versions are listed, windows will select the first.
See more information on system environmental variables here.
In Ubuntu/Linux, usually pip is associated with the native legacy version (2.7), pip3 is associated with Python3.5.x and pip3.6 is associated with Python3.6.x.
However, if you are using Unix OS (such as Ubuntu) or Mac, it is highly recommended to use virtualenv and activate it. See Official documentation to see how to use it. It's true for both Python2.7 and
Python3.6. In short, you will create a lightweight copy of you python installation without any packages, and, your installed packages will be installed within this virtual environment. Once you activate a virtual environment, the pip is associated with this environment.
How do I specify which python version to use?
You have multiple choices to specify in which environment you want to install the package. It depends if you are on Windows/Linux/MAC.
Shortly, you have the following options:
Use an IDE and let it help you manage your packages (e.g. Pycharm). Using PyCharm, you will find it very easy to use its package manager. You can also open the IDE's terminal and when you use pip, it will use the package manager of the selected interpreter. See official documentation.
Use OS native terminal and specify the version. In windows, the easiest way is to go to a command line or powershell, and type "c:\path\to\python.exe -m pip install ". On Ubuntu, use pip/pip3/pip3.6. Again, on Ubuntu it is highly recommended to use venv (virtual environment) since installing wrong package on the wrong version can interrupt the native python (Ubuntu uses python for multiple reasons such as the GNOME GUI).
Use virtual environments. You can look it up, there are plenty of threads explaining on that, as well as the Official documentation.
I went ahead and downloaded the latest version of python3 onto my mac from python.org/downloads/, however, I was having trouble using pip in my terminal, so I used Homebrew to install python via the command line. It looks like Homebrew installed a 2.x version. I also know that my mac already has some python2.x version built in. I would like to remove the 2.x versions and only use the python3.6.1 that I have. How can I accomplish this?
There are several aspects to your question: invoking different major Python versions, removing unneeded Homebrew-installed packages, and identifying which installation is in use.
Major Python versions, 2 and 3, use different command names to invoke them. The first, python, uses the system default according to your configured PATH environment variable and should default to Python 2, but does not on all systems. Reference: https://www.python.org/dev/peps/pep-0394/
Instead, explicitly invoke the version you want, e.g. by running python3. Note that virtual environments (created by running python3 -m venv <name> or virtualenv) will create an isolated environment using the Python version of your choice where python will always use that version, and pip will install packages relative to, instead of system-wide, which may avoid the requirement of superuser (sudo) privileges.
Note that Homebrew might not override the default versions provided by the system, requiring you to make changes to your PATH to force Homebrew's versions to be preferred. After installing a package Homebrew will inform you of this type of information, and you can get the messages later by running: brew info <package>
Uninstalling a brew-installed package requires running: brew uninstall <package>
For more information, run: brew help
As mentioned above, identifying just which Python you're using starts with the PATH. To quickly identify the executable you invoke when typing python, python2, or python3 into your shell is by running: which python — this can help identify if you're using the Homebrew installed version or the system version, or if you are using a version from an active virtual environment. You can also use which to identify the pip or pip3 command location.
This gets you the first part of the equation. The second then boils down to: where does Python think things are? Invoking it then running the following will tell you exactly where it thinks things (imports) are:
import sys, print
pprint(sys.path)
This can help track down issues related to why you can't import that package you just installed with pip.
I can highly recommend using a virtual environment to isolate your projects from system level packages and differences. Within one, python and pip will basically always behave as expected, as you determined when creating the environment.
I would like to figure out a "fool-proof" installation instruction to put in the README of a Python project, call it footools, such that other people in our group can install the newest SVN version of it on their laptops and their server accounts.
The problem is getting the user-installed libs to be used by Python when they call the scripts installed by pip. E.g., we're using a server that has an old version of footools in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/.
If I do python2.7 setup.py install --user and run the main entry script, it uses the files in /Users/unhammer/Library/Python/2.7/lib/python/site-packages/. This is what I want, but setup.py alone doesn't install dependencies.
If I (revert the installation and) instead do pip-2.7 install --user . and run the main entry script, it uses the old files in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ – that's not what I want.
If I (revert the installation and) instead do pip-2.7 install --user -e . and run the main entry script, it uses the files in . – that's not what I want, the user should be able to remove the source dir (and be able to svn up without that affecting their install).
I could use (and recommend other people to use) python2.7 setup.py install --user – but then they have to first do
pip-2.7 install -U --user -r requirements.txt -e .
pip-2.7 uninstall -y footools
in order to get the dependencies installed (since pip has no install --only-deps option). That's rather verbose though.
What is setup.py doing that pip is not doing here?
(Edited to make it clear I'm looking for simpler+safer installation instructions.)
Install virtualenvwrapper. I allows setting up separate python environments to alleviate any conflicts you might be having. Here is a tutorial for installing and using virtualenv.
Related:
https://virtualenvwrapper.readthedocs.org/en/latest/
Console scripts generated by pip in the process of installation should use user installed versions of libraries as according to PEP 370:
The user site directory is added before the system site directories
but after Python's search paths and PYTHONPATH. This setup allows the
user to install a different version of a package than the system
administrator but it prevents the user from accidently overwriting a
stdlib module. Stdlib modules can still be overwritten with
PYTHONPATH.
Sidenote
Setuptools use hack by inserting code in easy_install.pth file which is placed in site-packages directory. This code makes packages installed with setuptools come before other packages in sys.path so they shadow other packages with the same name. This is referred to as sys.path modification in the table comparing setuptools and pip. This is the reason console scripts use user installed libraries when you install with setup.py install instead of using pip.
Taking all of the above into account the reason for what you observe might be caused by:
PYTHONPATH pointing to directories with system-wide installed libraries
Having system-wide libraries installed using sudo python.py install (...)
Having OS influence sys.path construction in some way
In the first case either clearing PYTHONPATH or adding path to user installed library to the beginning of PYTHONPATH should help.
In the second case uninstalling system-wide libraries and installing them with distro package manager instead might help (please note that you never should use sudo with pip or setup.py to install Python packages).
In the third case it's necessary to find out how does OS influence sys.path construction and if there's some way of placing user installed libraries before system ones.
You might be interested in reading issue pip list reports wrong version of package installed both in system site and user site where I asked basically the same question as you:
Does it mean that having system wide Python packages installed with easy_install thus having them use sys.path manipulation breaks scripts from user bin directory? If so is there any workaround?
Last resort solution would be to manually place directory/directories with user installed libraries in the beginning of sys.path from your scripts before importing these libraries.
Having said that if your users do not need direct access to source code I would propose packaging your app together with all dependencies using tool like pex or Platter into self-contained bundle.
I like to use Python with numpy, scipy and some other packages. I am an absolute Python beginner and have some issues with the installation under Mac OS X.
I am following these two tutorials to install python: 1 and 2.
Here, HomeBrew is used to install Python (with pip) and virtualenv. I do not have an opinion about what is better, MacPorts, HomeBrew, Fink... I just found this tutorial inspiring confidence.
If I understand things correctly, OS X system Python, which I should never touch, is under /System/Library/Frameworks/Python.Framework. And I cannot use this one in Xcode because it does not have my wanted packages. The HomeBrew Python will be installed somewhere in /usr/local/. I found a framework there but as the system framework it does not have the additional packages. The tutorial explains that it might be better to install additional packages in virtual environments only which is done via pip. But I cannot find a framework there.
So my question is: How can I get a Python installation in a virtual environment that includes a framework that I can include into Xcode?
The Apple Python is functional and the normal site-packages folder is /Library/Python/2.7/site-packages (and not /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages). You can use it without problem.
I never had any problem to install all modules (numpy, scipy, matplotlib, pandas, shapely, and others...) that I wanted as frameworks, or with pip or easy_install, including virtualenv (simply install them in the conventional way in Python) or creating virtual environments.
when you install a framework module, It is placed in in the normal site-packages folder.
the only problem is possibly the "old" Python version (not a problem for me, using 2.6.x, 2.7.x and 3.3.x versions)
But if you want, you can install others versions of Python (in 64-bits, not 32 !):
a) the way prescribed by Apple: as a framework
the official versions of Python.org are installed in /Library/Frameworks/Python.framework with site-packages folder in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
same for the Enthought scientific version of Python (scientific distribution with many modules preinstalled, numpy, scipy, matplotlib,...),
(you can also install the Homebrew Python version as a framework, see below)
You must change the PATH of the Python executable in /usr/bin (usually, this is done automatically by the distribution by symlinks or in the /Users/me/.bash_profile file ).
The modules installed in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages do not interfere with those installed in /Library/Python/2.7/site-packages if you use the appropriate Python executable and viceversa.
b) the package management system way
MacPorts install its own version of Python in the folder /opt/;
sudo port -v install python27
Fink install its own version of Python in the folder /sw/;
fink install Python27
Homebrew installs Python in /usr/local/Cellar with symlinks in /usr/local/bin.
brew install python
or
brew install python --framework
To use them, you must add /sw/bin, /sw/lib/ or /opt/bin, /opt/lib/ to the PATH and change the PATH of the Python executable
For me, the main problem with Fink and MacPorts is that they do not take into account what is already installed and install all in their respective folders which can create real problems in the management of library paths.
The Homebrew solution is "cleaner" (in /usr/local) and is based on existing libraries if they are up to date, otherwise it installs its own versions of the libraries
c) the "autonomous" way
the perfect solution is Anaconda (another scientific distribution with many modules preinstalled, ),
Installs cleanly into a single directory (where you want as /Users/me/anaconda)
doesn’t require root privileges
doesn’t affect other Python installs on your system, or interfere with OS X Frameworks
switch to/from Anaconda just by setting $PATH or creating an alias in /Users/me/.bash_profile
alias anaconda='/Users/me/anaconda/bin/python'
alias anaconda3='/Users/me/anaconda/envs/py33/bin/python3'
you can install Python versions from 2.6.x to 3.3.x
Innovative package and environment manager for Python, named conda, but you can use pip or easy_install without problem
for me now, it is the best solution to install virtual environments (as /Users/me/anaconda/envs/py33 )
d) the "hard" way
you can compile your own version of Python in a classical form (results in /usr/) or as a framework. It takes time but it is not difficult.
So your question:
How can I get a Python installation in a virtual environment that includes a framework that I can include into Xcode?
Unless you are a Unix specialist (PATHs management) , you must use the Apple's recommended solution, a frameworks distribution (including the Apple Python)