Previously, I have been using Python 2.7. If I installed a module using pip, it would place the module in /usr/local/lib/python2.7/site-packages. Then, if I were to import that module in a python script, the site-packages directory would be searched to find that module.
However, I want to start using the Anaconda distribution. After downloading this, I notice that there are a number of packages located at /home/karnivaurus/Anaconda/pkgs, for example scikit-learn. Then, in the directory of each packages, there is the directory structure lib/python2.7/site-packages, which contains the modules for that package.
So, it seems that I have now gone from the situation where I only had one site-packages directory, to where I have a number of site-packages directories, one for each package.
My question is: When I create a python script, and want to import a module, how does python know where to look for these modules? Will it look in /usr/local/lib/python2.7/site-packages as well as the site-packages directories that come with Anaconda? What if I want to install another package that does not come with Anaconda -- where should this be installed to?
Thanks!
The paths in which Python will import packages can be seen with the following command:
python -c "import sys; print sys.path"
Please see the documentation, also (as stated in the documentation) if you need to modify the search path, look into PYTHONPATH.
Problem description:
Using pip, I upgraded to the latest version of requests (version 2.7.0, with pip show requests giving the location /usr/local/lib/python2.7/dist-packages). When I import requests and print requests.__version__ in the interactive command line, though, I am seeing version 2.2.1. It turns out that Python is using the pre-installed Ubuntu version of requests (requests.__file__ is /usr/lib/python2.7/dist-packages/requests/__init__.pyc -- not /user/local/lib/...).
From my investigation, this fact is caused by Ubuntu's changes to the Python search path (I run Ubuntu 14.04) by prepending the path to Ubuntu's Python package (for my machine, this happens in usr/local/lib/python2.7/dist-packages/easy-install.pth). In my case, this causes the apt-get version of requests, which is pre-packaged with Ubuntu, to be used, rather than the pip version I want to use.
What I'm looking for:
I want to globally prepend pip's installation directory path to Python's search path (sys.path), before the path to Ubuntu's Python installation directory. Since requests (and many other packages) are used in many Python scripts of mine, I don't want to manually change the search path for every single file on my machine.
Unsatisfactory Solution 1: Using virtualenv
Using virtualenv would cause an unnecessary amount of change to my machine, since I would have to reinstall every package that exists globally. I only want to upgrade from Ubuntu's packages to pip's packages.
Unsatisfactory Solution 2: Changing easy-install.pth
Since easy-install.pth is overwritten every time easy-install is used, my changes to easy-install.pth would be removed if a new package is installed. This problem makes it difficult to maintain the packages on my machine.
Unsatisfactory (but best one I have so far) Solution 3: Adding a separate .pth file
In the same directory as easy-install.pth I added a zzz.pth with contents:
import sys; sys.__plen = len(sys.path)
/usr/lib/python2.7/dist-packages/test_dir
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
This file is read by site.py when Python is starting. Since its file name comes after easy-install.pth alphanumerically, it is consumed by site.py afterwards. Taken together, the first and last lines of the file prepend the path to sys.path (these lines were taken from easy-install.pth).
I don't like how this solution depends on the alphanumeric ordering of the file name to correctly place the new path.
PYTHONPATHs come after Ubuntu's paths
Another answer on Stack Overflow didn't work for me. My PYTHONPATH paths come after the paths in easy-install.pth, which uses the same code I mention in "Unsatisfactory solution 3" to prepend its paths.
Thank you in advance!
This is not recommended, as it hard-codes a path and makes it difficult to run the script elsewhere, but you can do
>>> import sys
>>> sys.path.insert(0,'/home/anand/')
>>> print(sys.path)
['/home/anand/', '', '/usr/local/lib/python2.7/dist-packages/_pdbpp_path_hack', '/usr/local/lib/python2.7/dist-packages/goose-0.0.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/jieba-0.33-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/cssselect-0.9.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/nanoservice-0.1.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/nanomsg-1.0a2-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/msgpack_python-0.4.2-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/DecisionTree-2.2.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/nudepy-0.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/wsgilog-0.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/distribute-0.7.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/PIL-1.1.7-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/munkres-1.0.7-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/parsedatetime-1.4-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/argparse-1.3.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/tusker-0.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/SQLAlchemy-1.0.3-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/numpy-1.9.2-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/turkic-0.2.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/scikits.bootstrap-0.3.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/pyvision-0.1-py2.7-linux-x86_64.egg', '/home/anand/playspace/languages/python_pkgs/ets', '/usr/local/lib/python2.7/dist-packages/Scrapy-1.1.0dev1-py2.7.egg', '/usr/lib/python2.7/dist-packages', '/home/anand/playspace', '/home/anand/workspace/pyvision/src', '/home/anand/playspace/yapf', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/Orange/orng', '/usr/local/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/wx-3.0-gtk2']
>>>
After this your imports will look into the prepended path before looking anywhere else.
You shouldn't need to mess with pip's path, python actually handles it's pathing automatically in my experience. It appears you have two pythons installed. If you type:
which pip
which python
what paths do you see? If they're not in the same /bin folder, then that's your problem. I'm guessing that the python you're running (probably the original system one), doesn't have it's own pip installed. You probably just need make sure the path for the python you want to run should come before /usr/bin in your .bashrc or .zshrc
If this is correct, then you should see that:
which easy_install
shares the same path as the python installation you're using, maybe under /usr/local/bin. Then just run:
easy_install pip
And start installing the right packages for the python that you're using.
Answer to direct question
You could create a directory called sitecustomize in your site-packages directory. We'll turn this into a sitecustomize module as described here (Python 2 here). Specifically:
an attempt is made to import a module named sitecustomize, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in your site-packages directory.
In the sitecustomize directory create a file called __init__.py and add the manipulations you want to perform there. A very simple example is:
import sys
sys.path = ['/your/path/to/pip/install'] + sys.path
In your case, I think your/path... would be /usr/local/lib/python2.7/dist-packages. You might want to do something more sophisticated, but this crudely prepends to sys.path and is run whenever python is started (e.g. starting the interpreter in the command line, or running a python script from a file).
Caveat
I'm not a huge advocate of doing this - it's a bit of a blunt way to do what you want. But you specifically say that using a virtualenv is undesirable for you and you want to make the change "globally" and I think this will do what you want.
Thoughts on underlying issue
I think #fivetentaylor's answer is on the right track here - it appears you are using pip from one install with python executable for another. Masking this by messing with the path could get very confusing very quickly. I'd definitely ensure you have a separate pip for each install of python and you use that. That should keep the directory structures for the separate installs separate. Otherwise, you are forcing one install to use packages from a different installation's directories. No problem technically, but confusing logistically.
Using virtualenv would cause an unnecessary amount of change to my machine, since I would have to reinstall every package that exists globally. I only want to upgrade from Ubuntu's packages to pip's packages.
Nope, you can use --system-site-packages.
Edit
# make your new virtualenv
user#darkstar:~$ mkvirtualenv --system-site-packages max
(max)user#darkstar:~$ python
>>> pprint(sys.path)
['',
'/home/user/.virtualenvs/max/lib64/python27.zip',
'/home/user/.virtualenvs/max/lib64/python2.7',
'/home/user/.virtualenvs/max/lib64/python2.7/plat-linux2',
'/home/user/.virtualenvs/max/lib64/python2.7/lib-tk',
'/home/user/.virtualenvs/max/lib64/python2.7/lib-old',
'/home/user/.virtualenvs/max/lib64/python2.7/lib-dynload',
'/usr/lib64/python2.7',
'/usr/lib/python2.7',
'/usr/lib64/python2.7/lib-tk',
'/home/user/.virtualenvs/max/lib/python2.7/site-packages',
'/usr/lib64/python2.7/site-packages/google_api_python_client-1.2-py2.7.egg',
'/usr/lib64/python2.7/site-packages',
'/usr/lib64/python2.7/site-packages/PIL',
'/usr/lib64/python2.7/site-packages/gtk-2.0',
'/usr/lib64/python2.7/site-packages/IPython/extensions']
As you see, the path of this virtualenv include the system path.
To check if it was working, i installed a package system-wide after making the virtualenv.
root#darkstar:~: pip install igraph
Collecting igraph
Downloading igraph-0.1.8-py2.py3-none-any.whl (119kB)
100% |████████████████████████████████| 122kB 1.7MB/s
Collecting ipython (from igraph)
Downloading ipython-3.2.1-py2-none-any.whl (3.4MB)
100% |████████████████████████████████| 3.4MB 203kB/s
Installing collected packages: ipython, igraph
Successfully installed igraph-0.1.8 ipython-3.2.1
root#darkstar:~: python -c 'print __import__("igraph")'
<module 'igraph' from '/usr/lib64/python2.7/site-packages/igraph/__init__.pyc'>
(max)user#darkstar:max$ python -c 'print __import__("igraph")'
<module 'igraph' from '/usr/lib64/python2.7/site-packages/igraph/__init__.pyc'>
Obviously, what is installed inside the virtualenv takes precedence over the system-wide libraries.
I belive that answer your needs.
Well, the alternatives presented by the others are very acceptable and might even be better. However, if you are intent on using the sys.path() way, then just treat it like a list and use the insert method.
import sys
sys.path.insert(0, "path_to_pip")
from subprocess import call
call("sudo pip install requests")
I would do this with sitecustomize as described in the site.py docs. This file is imported after the initial sys.path is configured and you can use it to alter sys.path in arbitrary ways as needed.
I've used it as a sysadmin to include custom release locations and it does the job quite nicely.
https://docs.python.org/2/library/site.html
While bufh's answer will solve your problem now, you'll likely find that there'll be some other package where you don't want to use the Ubuntu-provided version. So here's why you want to use virtualenvs to manage the versions of packages (and not try to override the system versions).
As you've noticed, the order of sys.path sets the order that python packages are found. This means that changing sys.path affects how python scripts find their imports, both scripts you've written, and those provided by Ubuntu. Given that python scripts are used in Ubuntu programs, it's possible to "break" Ubuntu in interesting ways by changing which version of python packages Ubuntu programs use (which is the reason dist-packages exists).
To avoid this, virtualenv was created, which effectively allows there to be different sets of packages to be used. There's now a bunch of utilities which make using and managing virtualenvs easier. The one that's probably of most interest for you is pipsi, which creates a virtualenv per script, and avoids the need to activate it.
From the sounds of things, Ubuntu is using a package path configuration file as documented here to set up any packages it installs.
Looking at site.py I see that there is a specific order of path resolution that invokes the configuration files as it resolves the site packages directories.
I think that gives you three options that I can see:
Use virtualenv --system-site-packages as per #bufh's answer.
Use pip user installs to set up the packages you need in the path before the standard site packages.
Use sitecustomize to re-write your sys.path (e.g. to put your local directories first).
I use a script under virtualenv, that requires bzrlib package which is not available in my virtualenv but is included in my system python packages: /usr/lib/python2.7/dist-packages/bzrlib/
If I want to use it, one option is to extend sys.path, but I would have to include the parent folder /usr/lib/python2.7/dist-packages/ which contains many other packages, that I don't want to make available. Is there any easy way to include just bzrlib package?
What about creating a link in different directory and importing using that? Or even in your own project.
ln -s /package/dir/path /project/dir/path
If you have to load it remotely, here is the link provided by #unutbu showing how to do it:
How to import a module given the full path?
import imp
foo = imp.load_source('module.name', '/path/to/file.py')
foo.MyClass()
I want to be able to write a line of code like this and have it run smoothly:
python /path/to/python_file.py -arg1 -arg2 -etc
I figured out an easy way to discover all modules and add them to the current Python path, but it still doesn't seem to recognize the .py file, even though it's supposedly in the sys.path. I know the sys.path addition is working because I can perform this in the interpreter just fine:
>>>import ModuleManager # My Python script to discover modules in lower directories
>>>import testModule # Module I want to run from lower directory
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named testModule
>>>ModuleManager.discoverModules() # Now, I find the modules and add them to path
Discovering Modules...Complete!
Discovered 6 Modules.
>>>import testModule # No error now.
>>>
What I would like to do at this point is be able to go into my terminal and say:
python testModule -arg1 -arg2 -etc
and have it perform how I would expect.
To be clear, I want to write a line of code in ModuleManager.py (from my application root folder) that allows me to access a file named testModule.py which is found in /root/path/to/testModule.py in a way such that I can use arguments as in python testModule.py -arg1 -arg2 -etc. Any suggestions?
sys.path.append(os.getcwd()) work for me.
Now you can run commands like ./manage/my_cli_command.py
How to write python scripts efficiently
There are multiple ways to write Python script (a script, which is supposed to be called from command line).
For flexible use, forget about manipulating PYTHONPATH - such solutions are difficult to maintain and re rather shaky.
Script with Python stdlib imports only
Betting purely on packages and modules, which are part of Python stdlib makes the script easy to run from anywhere.
Extension to this is importing modules and packages, which are globally installed. Globally installing packages is mostly considered bad practice, as it spoils global Python environment. I have globally installed just small set of packages, which I use often: pyyaml, docopt, lxml, jinja2.
Script importing any installed package
Packages and modules, even your own, can be installed by means of setup.py. It took me a while to get used to this, I was initially ignoring setup.py as too complex solution, but later on I used to use it for any home-made package which I need to import in my programs.
Importing packages installed into virtualenv
This is by far my most popular solution now.
You will need your packages having setup.py.
After virtualenv is created, you use pip to install all the packages you need.
With given virtualenv being active, you can than simply use the script, which will see the packages available for import regardless of where you are your script starting from.
Script installed by means of setup.py
Again, this solution seems first too complex "just for simple script", but finally it can become the simplest method.
You have to create simple Python project with your script and setup.py only. The setup.py shall declare the script for installing. There are two methods, either by scripts argument, or by entry_points, later one being more flexible - use that one.
The script is best installed from within virtualenv - after the installation is complete, find location of the script and move it whenever you want to use it. For using the script, there is no need to activate the virtualenv, it is enough to have it in PATH. The script inside includes refernces to Python from given virtualenv and this ensures, it uses it including all installed packages.
Note, that the best solution for having required packages installed for your script is mentioning it in install_requires argument in setup.py, it ensures, they get installed with your script automatically.
Using zc.buildout
zc.buildout used to be my favourite solution until pip and virtualenv became mature solutions.
You have to create buildout.cfg file and there you define all what is needed for your script. this moslty includes having the setup.py in place, but this is not always necessary.
Personally, I found zc.buildout powerfull, but rather difficult to learn and I would recommend using virtualenv solution.
Using other solutions like PyInstaller
There are other solution, which allow turning Python script into single executable file.
PyInstaller looks like good way to go. There might be other similar solutions.
Conclusions
Forget about making your modules and packages importable by manipulating PYTHONPATH, such solutions are quite shaky.
My recommendation is to go for project having setup.py using entry_points to install your script.
Installation into virtualenv seems to be the cleanest method.
Setting up this environment (the project) the first time takes some effort, but after that, you can simply copy and adopt the solution and it will become natural way to go.
How to call from Python another Python script
You want to call a Python script, which works well only if run from particular directory as it has some imports which work well only from given location.
Assuming there is a subdirectory named "script_dir" and there is the "local_script_name.py" file:
import subprocess
subprocess.call(["python", "local_script_name.py"], cwd="script_dir")
If your script accepts command line arguments "arg1", "arg2", "-etc"
import subprocess
subprocess.call(["python", "local_script_name.py", "arg1", "arg2", "-etc"], cwd="script_dir")
Append the parent directory temporarily to the path:
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir)))
I am new to Python. I am running Mac OS X 10.8.2, Python 2.7.3, Xcode 4.5.1.
I am not able to import pyobjc to python.I used easy_install pyobjc or manually downloading it from http://pypi.python.org/pypi/pyobjc/2.3 and running python setup.py install. Here is a screenshot of my site-packages folder
How do I solve this?
Here is a screenshot of sys.path.PyOBJc is present in sys.path
I don't see a .pth file for pyobjc in your site-packages directory there.
.pth files, placed inside a directory already on Python's search path, contain directories to add to that search path. They're simple text files; you can review the ones already there to get a feel for how they work.
As to why you didn't get .pth files for pyobjc, I'm not sure. But you could create some to fix the problem up.
Further reading: Modifying Python’s Search Path
The module name for PyObjC is not "pyobjc".
To use the low-level bridge use "import objc"
To access the Cocoa framework use "import Cocoa".
If "import objc" gives an import error you probably have a number of Python interpreters, check if the value of "sys.prefix" is what you expect it to be. Also check if "/Library/Frameworks/Python.framework/Version/2.7/lib/python2.7/site-packages" is on "sys.path".