Previously, I have been using Python 2.7. If I installed a module using pip, it would place the module in /usr/local/lib/python2.7/site-packages. Then, if I were to import that module in a python script, the site-packages directory would be searched to find that module.
However, I want to start using the Anaconda distribution. After downloading this, I notice that there are a number of packages located at /home/karnivaurus/Anaconda/pkgs, for example scikit-learn. Then, in the directory of each packages, there is the directory structure lib/python2.7/site-packages, which contains the modules for that package.
So, it seems that I have now gone from the situation where I only had one site-packages directory, to where I have a number of site-packages directories, one for each package.
My question is: When I create a python script, and want to import a module, how does python know where to look for these modules? Will it look in /usr/local/lib/python2.7/site-packages as well as the site-packages directories that come with Anaconda? What if I want to install another package that does not come with Anaconda -- where should this be installed to?
Thanks!
The paths in which Python will import packages can be seen with the following command:
python -c "import sys; print sys.path"
Please see the documentation, also (as stated in the documentation) if you need to modify the search path, look into PYTHONPATH.
Related
Context
On a provided Linux-server without root access I am bound to preinstalled Python-packages (e.g. cannot upgrade them).
However, I am able to install a package directly from a wheel (.whl) by using pip3 install /path/package_name --user which will install the package to a site-package-folder preserved to my user.
In my special case I want to upgrade the scikit-learn-package. This package is already preinstalled and I cannot upgrade it (root access missing), however, I can install the newer version in the --user-folder.
In the filesystem I can see that both installations are now present in their respective folders.
Both paths are known to python (checked by using sys.path).
Question/Problem
When I import scikit-learn via import sklearn and print the version (sklearn.__version__), I'll always end up with the preinstalled version and not the new one I installed in my --user-directory.
Given two installations of the same module with a differing version: How can I define in Python during the import which module/version to load?
The premise is that I cannot disable/uninstall the old version (root access again..).
Thanks to #0 0 I was able to figure out the solution as well as a workaround:
Solution
When I compared sys.path with the defined PYTHONPATH in the shell (the PYTHONPATH initializes sys.path like answered here on SO) I saw that the PYTHONPATH did not know about the --user-folder (even though sys.path knew - this puzzles me a bit).
Thus, I added to my .bashrc:
cd # Go to the user folder where .bashrc is stored
nano .bashrc
export PYTHONPATH=$(python3 -m site --user-site)$PYTHONPATH
python3 -m site --user-site identifies the --user-folder pip uses.
In order to load the module first, put it before the current PYTHONPATH.
When now loading sklearn in Python I get the correct version.
Workaround
For the sake of completeness:
One can also just rename the doubled package and import that.
cd $(python3 -m site --user-site) # Go to the folder pip installed the package while using --user
mv package_name package_name_new # Rename it
Then in Python load it this way:
import package_name_new
NOTE: This workaround will cause trouble if you have dependencies relying on it! Thus, better use the correct solution.
In short, my question is, how do I install the latest version of scikit-image into my usr/lib/python3/dist-packages so I can actually use it? I think there is a problem with my understanding of how third-party modules are installed. As a newb, I don’t know how to rectify that, hence this post.
I need help to understand how to install packages in python3 up until now I have used pip/pip3/apt-get/synaptic etc and it has worked fine for many packages. However, I have hit several barriers (Skimage, opencv, plantcv in python3). I must emphasise, the problem I am having is using these packages in python3, not 2.7.
For example, I want to use the latest version of scikit-image (0.14) with python3. (http://scikit-image.org/) I have tried using the installation instructions and have not yet successfully managed to install it. I have navigated to my usr/lib/python3/dist-packages and copied scikit-image into this directory (I have all the dependencies installed in here already).
Image of my folder for dist-packages as proof
As you can see, the folder containing skimage is in the directory I want to be installed in, how do I actually install it? Do I have to extract skimage out of the folder into the directory and then run the install command? If I navigate to usr/lib/python3/dist-packages/scikit-image and then run pip install -e . I get an error, stating that I need numpy. If I write a python script using python3 I can clearly see I have it installed (and I have been using it for a long time). So, there must be a problem in how I have this package in my file system. I think a janky workaround would be to copy all the modules into my working directory and Import them that way as if they were modules I have made myself, but this obviously negates the whole point of installing packages.
This has also happened with another package called plantcv. Where I went into the directory usr/lib/python3/dist-packages then cloned the source from git hub and installed as per instructions. When I import plantcv in my python3 script. It Imports fine. But, there is nothing in it, as python cannot see the modules which are inside this folder at usr/lib/python3/dist-packages/plantcv/plantcv.
There is clearly some comprehension here that I am missing, as I have a similar problem for two packages now. Please, Internet. Help me understand what I am missing!
You simply need to copy the folder in /usr/lib/python3/dist-packages/package-name
However, there are certain things that are specific to python packages. The folder named package name should be a valid package. A good indicator of that is it will contain a file "__init__.py". It is very likely that every sub-directory inside this package directory will contain a "__init__.py" file. It depends on whether there are modules inside these sub-directories.
In your code simply import the package like the following.
import package-name
where package-name can be skimage
I already figured out how one can retrieve the include and site-package paths of a python environment. For instance the following is one of multiple possibilities:
from distutils.sysconfig import get_python_lib, get_python_inc
print(get_python_lib()) # Prints the location of site-packages
print(get_python_inc()) # Prints the location of the include dir
However, I was not able to find a robust method to retrieve the bin folder of a python environment, that is, the folder where python itself and tools like pip, pyinstaller, easy_install, etc., typically reside. Does anyone know how I can get this path from within python?
Some may want to suggest binpath = os.path.dirname(sys.executable). On Mac however, this does not work if python was installed as a Framework (binpath would point at: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS)
What could I use instead that works cross-platform?
The stdlib way to get this location is via sysconfig module. Works on 2.7 and 3.3+. Works whether or not you are using a virtual environment.
>>> from sysconfig import get_path
>>> get_path('scripts')
'/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/bin'
Almost any package with scripts to install will put them there. However, note that with an imperative installer the setup.py code is actually executed, which means it could literally lay down scripts anywhere on the filesystem, given permission.
A Python module is just a .py source file. A Python package is simply a collection of modules.
So why do we need programs such as pip to 'install' Python modules? Why not just download the files, put them in our project's folder and import them?
What exactly does it mean to 'install' a module or a package? And what exactly does pip do?
Are things different on Windows and on Linux?
So why do we need programs such as pip to 'install' Python modules? Why not just download the files, put them in our project's folder and import them?
It's just meant to facilitate the installation of softwares without having to bundle all the dependencies nor ask the user to download the files.
You can type pip install mysoftware and that will also install the required dependencies. You can also upgrade a software easily.
What exactly does it mean to 'install' a module or a package? And what exactly does pip do?
It will copy the files in a directory that is in your Python path. This way you will be able to import the package without having to copy the directory in your project.
With your proposal, for each and every project you have to download the required modules as dependencies. You have to download them again and again and add them with your project which is not very suitable though some platform like node.us do it.
What pip do is to keep the modules you installed in /use/lib/python*/site-packages/ so clearly it is included in your Python's path. So, when you try to import a module or package it checks in site-package if it exists. If exists,then this code will be used with your project. If not, you will get an error.
I am new to Python. I am running Mac OS X 10.8.2, Python 2.7.3, Xcode 4.5.1.
I am not able to import pyobjc to python.I used easy_install pyobjc or manually downloading it from http://pypi.python.org/pypi/pyobjc/2.3 and running python setup.py install. Here is a screenshot of my site-packages folder
How do I solve this?
Here is a screenshot of sys.path.PyOBJc is present in sys.path
I don't see a .pth file for pyobjc in your site-packages directory there.
.pth files, placed inside a directory already on Python's search path, contain directories to add to that search path. They're simple text files; you can review the ones already there to get a feel for how they work.
As to why you didn't get .pth files for pyobjc, I'm not sure. But you could create some to fix the problem up.
Further reading: Modifying Python’s Search Path
The module name for PyObjC is not "pyobjc".
To use the low-level bridge use "import objc"
To access the Cocoa framework use "import Cocoa".
If "import objc" gives an import error you probably have a number of Python interpreters, check if the value of "sys.prefix" is what you expect it to be. Also check if "/Library/Frameworks/Python.framework/Version/2.7/lib/python2.7/site-packages" is on "sys.path".