Related
Experimenting with using python in a virtualenv on my shared hosting account. Based on this dreamhost tutorial have installed pip and another module or two (echonest, remix), but trying to install numpy the long list of errors starts with non-existing path in 'numpy/distutils': 'site.cfg'
/bin/sh: svnversion: command not found.
The virtualenv instructions I read say, "make sure that your path gives preference to ~/local/bin to /usr/bin so that your "local" copy of Python runs, and that your scripts refer to that location."
Does that suggest to make a link somewhere that points calls to /usr/bin/ to ~/local/bin?
Is the solution to find the install package and edit the paths in it's setup.py file?
this is referring to the linux environment variable $PATH which lists the directories in which to look for executables for commands when you don't specify an absolute path. this will contain a list of comma separated paths eg:
/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
You just need to make sure that the /usr/local... stuff comes first (left) like this:
export PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
Are you sure you're running ~/local/bin/python when running setup.py?
One virtualenv-specific thing you can do is source ~/local/bin/activate, which automatically sets your virtualenv to take preference over everything else in your path. It only works until you log out of your terminal instance or run deactivate
I'm seeing confusion in the comments--- between "local" and "your virtualenv". And there is a parallel problem with $PATH. Is there just one path issue? No... there are two.
True, my own analogous efforts on a different shared-hosting account are ultimately crashing too, but I'm further along and I appear to have some things figured out. The confusion here partly comes about because there are two Dreamhost tutorials that are linked to in the question, but the authors of those two wiki chapters didn't make any effort to integrate the two.
And in the second tutorial on Python the mistake is made of introducing the discussion of the local directory in the virtualenv section though it's only really explained in the next section on building your own Python. And the discussion on building your own isn't complete regarding what to do about $PATH. You don't need to build your own Python or have a local directory just to use virtualenv.
It may even be detrimental. The language at this official virtualenv page should fill you with despair, as it shows no effort at clarity. I think that it means that mod_python and mod_wsgi, either of which you must be using for your Flask experiment, don't use any Python that you might come up with--- they use their own Python interpreter, the system's version. But virtualenv, the program, also puts a Python interpreter inside your virtual environment (the interpreter that you will use to do installs and also ultimately to run flask if you let it). The point of the warning from the virtualenv folks is that you can't run with two different interpreters in the same server process. They show you the workaround.
But I have digressed. The second, Python tutorial instructs about building your own local Python, in the directory /home/yourusername/local. A more complete, yet succinct discussion is to be found at Sugath Mudali's Blog.
Your virtualenv should occupy a second directory, which I'll call yourvirtenv. Alongside ~/local should be OK.
So, per Sugath Mudali's helpful instructions, you need to execute export PATH=/home/yourusername/local/bin:$PATH, which puts /home/yourusername/local/bin at the front of your $PATH, which you can confirm with "echo $PATH". Once you've done that, whenever you use "python" in your shell you'll be using the Python that you just installed, whose path you just put first.
Having done that, if you then cd into to your virtualenv source folder (somewhere in /home/youruserhame) and run "python virtualenv.py /home/youruserhame/yourvirtenv" it will create a directory ~/yourvirtenv, into which it will put a copy of the interpreter of your local Python. Generally it will not only be a distinct copy but it will differ even as to version from that of your system, which is at /usr/bin/ as /usr/bin/python ("usr" here is not you but is the oddly-named directory of the entire shared Linux machine to which no unprivileged user such as you has access, into which program binaries can be installed, by the privileged, for system-wide use).
Note that the Dreamhost Python wiki says "DreamHost has begun upgrading servers to Python 2.6.6 as of February 2012." Version 2.6 is adequate for virtualenv and practically everything else. So I would question the need to build your own Python in your case. (Note that Dreamhost even suggests that you should try to forget about it entirely.)
Anyway, on to the second PATH deal, as limasxgoesto0 noted, once you run virtualenv.py and have thus created myvirtenv, if you're still in the /home/myusername user-root directory you run source myvirtenv/bin/source. That puts /home/myusername/myvirtenv/bin at the front of the path--- until you later simply run "deactivate" (no source in front of it). That is all that activate does.
Now, when you return days later and go back to work consider this: export PATH=/home/yourusername/local/bin:$PATH has evaporated. It's not permanent; goes away when the shell is closed. To make it permanent you have to add that statement to .bash_profile. Here's mine:
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
PATH=$PATH:$HOME/bin
export PATH
export PATH=$HOME/local/bin:$PATH
export LD_LIBRARY_PATH=/home/myusername/local/lib
unset USERNAME
I found it necessary to reassert the LD_LIBRARY_PATH in order to make sqlite3 function, even though I had compiled sqlite3 and done that export prior to building Python.
Note that this .bash_profile invokes .bashrc. Depending on how Dreamhost has arranged things, you may have to put these exports in .bashrc.
Likewise, when you return to the shell myvirtenv will not be found to still be activated even if you never typed-in "deactivate". You only activate it temporarily in the shell to install stuff inside it and to start your server or run some other program that you've installed. Thereafter, the installed stuff knows to use the Python interpreter that has been copied to myvirtenv. To make that not happen, to make the stuff run off of the system Python interpreter you have to do what the aforementioned official virtualenv docs on mod_python, mod_wsgi say.
It looks as though this may be necessary even if you have not built your own Python, because otherwise you would have two Python interpreters running in the same server process. I am however sketchy on this. Flask is, like Django, a Python framework. An A2 Hosting tutorial--- they use Phusion Passerger too--- may be of interest. On their site search for "Django shared" and the tutorial will come up on top. Note that Chris C.'s instructions there tell you to use the --python=/path/to/python switch. In your case that could be /home/myusername/local/bin/python if you have locally installed Python, but in their case it's /usr/bin/python2.6. I think that their point is they don't want to get tech-support tickets from customers on problems caused by upgrades to their system python2.6 that could break your site. So that --python switch (-p directory is the short form; --python=directory is the long form) is to freeze the Python that your app will be using, to avoid such problems. It does so by copying all of the Python into your virtual environment, not just the interpreter.
I don't understand how they are avoiding the problem of mod_wsgi and the customer's apps using two different interpreters, but maybe that's what Phusion Passenger helps out with.
Whenever I use sys.path.append, the new directory will be added. However, once I close python, the list will revert to the previous (default?) values. How do I permanently add a directory to PYTHONPATH?
If you're using bash (on a Mac or GNU/Linux distro), add this to your ~/.bashrc
export PYTHONPATH="${PYTHONPATH}:/my/other/path"
You need to add your new directory to the environment variable PYTHONPATH, separated by a colon from previous contents thereof. In any form of Unix, you can do that in a startup script appropriate to whatever shell you're using (.profile or whatever, depending on your favorite shell) with a command which, again, depends on the shell in question; in Windows, you can do it through the system GUI for the purpose.
superuser.com may be a better place to ask further, i.e. for more details if you need specifics about how to enrich an environment variable in your chosen platform and shell, since it's not really a programming question per se.
Instead of manipulating PYTHONPATH you can also create a path configuration file. First find out in which directory Python searches for this information:
python -m site --user-site
For some reason this doesn't seem to work in Python 2.7. There you can use:
python -c 'import site; site._script()' --user-site
Then create a .pth file in that directory containing the path you want to add (create the directory if it doesn't exist).
For example:
# find directory
SITEDIR=$(python -m site --user-site)
# create if it doesn't exist
mkdir -p "$SITEDIR"
# create new .pth file with our path
echo "$HOME/foo/bar" > "$SITEDIR/somelib.pth"
This works on Windows
On Windows, with Python 2.7 go to the Python setup folder.
Open Lib/site-packages.
Add an example.pth empty file to this folder.
Add the required path to the file, one per each line.
Then you'll be able to see all modules within those paths from your scripts.
In case anyone is still confused - if you are on a Mac, do the following:
Open up Terminal
Type open .bash_profile
In the text file that pops up, add this line at the end:
export PYTHONPATH=$PYTHONPATH:foo/bar
Save the file, restart the Terminal, and you're done
You could add the path via your pythonrc file, which defaults to ~/.pythonrc on linux. ie.
import sys
sys.path.append('/path/to/dir')
You could also set the PYTHONPATH environment variable, in a global rc file, such ~/.profile on mac or linux, or via Control Panel -> System -> Advanced tab -> Environment Variables on windows.
To give a bit more explanation, Python will automatically construct its search paths (as mentioned above and here) using the site.py script (typically located in sys.prefix + lib/python<version>/site-packages as well as lib/site-python). One can obtain the value of sys.prefix:
python -c 'import sys; print(sys.prefix)'
The site.py script then adds a number of directories, dependent upon the platform, such as /usr/{lib,share}/python<version>/dist-packages, /usr/local/lib/python<version>/dist-packages to the search path and also searches these paths for <package>.pth config files which contain specific additional search paths. For example easy-install maintains its collection of installed packages which are added to a system specific file e.g on Ubuntu it's /usr/local/lib/python2.7/dist-packages/easy-install.pth. On a typical system there are a bunch of these .pth files around which can explain some unexpected paths in sys.path:
python -c 'import sys; print(sys.path)'
So one can create a .pth file and put in any of these directories (including the sitedir as mentioned above). This seems to be the way most packages get added to the sys.path as opposed to using the PYTHONPATH.
Note: On OSX there's a special additional search path added by site.py for 'framework builds' (but seems to work for normal command line use of python): /Library/Python/<version>/site-packages (e.g. for Python2.7: /Library/Python/2.7/site-packages/) which is where 3rd party packages are supposed to be installed (see the README in that dir). So one can add a path configuration file in there containing additional search paths e.g. create a file called /Library/Python/2.7/site-packages/pip-usr-local.pth which contains /usr/local/lib/python2.7/site-packages/ and then the system python will add that search path.
On MacOS, Instead of giving path to a specific library. Giving full path to the root project folder in
~/.bash_profile
made my day, for example:
export PYTHONPATH="${PYTHONPATH}:/Users/<myuser>/project_root_folder_path"
after this do:
source ~/.bash_profile
On linux you can create a symbolic link from your package to a directory of the PYTHONPATH without having to deal with the environment variables. Something like:
ln -s /your/path /usr/lib/pymodules/python2.7/
For me it worked when I changed the .bash_profile file. Just changing .bashrc file worked only till I restarted the shell.
For python 2.7 it should look like:
export PYTHONPATH="$PYTHONPATH:/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python"
at the end of the .bash_profile file.
Adding export PYTHONPATH="${PYTHONPATH}:/my/other/path" to the ~/.bashrc might not work if PYTHONPATH does not currently exist (because of the :).
export PYTHONPATH="/my/other/path1"
export PYTHONPATH="${PYTHONPATH}:/my/other/path2"
Adding the above to my ~/.bashrc did the trick for me on Ubuntu 16.04
This is an update to this thread which has some old answers.
For those using MAC-OS Catalina or some newer (>= 10.15), it was introduced a new Terminal named zsh (a substitute to the old bash).
I had some problems with the answers above due to this change, and I somewhat did a workaround by creating the file ~/.zshrc and pasting the file directory to the $PATH and $PYTHONPATH
So, first I did:
nano ~/.zshrc
When the editor opened I pasted the following content:
export PATH="${PATH}:/Users/caio.hc.oliveira/Library/Python/3.7/bin"
export PYTHONPATH="${PYTHONPATH}:/Users/caio.hc.oliveira/Library/Python/3.7/bin"
saved it, and restarted the terminal.
IMPORTANT: The path above is set to my computer's path, you would have to adapt it to your python.
The script below works on all platforms as it's pure Python. It makes use of the pathlib Path, documented here https://docs.python.org/3/library/pathlib.html, to make it work cross-platform. You run it once, restart the kernel and that's it. Inspired by https://medium.com/#arnaud.bertrand/modifying-python-s-search-path-with-pth-files-2a41a4143574. In order to run it it requires administrator privileges since you modify some system files.
from pathlib import Path
to_add=Path(path_of_directory_to_add)
from sys import path
if str(to_add) not in path:
minLen=999999
for index,directory in enumerate(path):
if 'site-packages' in directory and len(directory)<=minLen:
minLen=len(directory)
stpi=index
pathSitePckgs=Path(path[stpi])
with open(str(pathSitePckgs/'current_machine_paths.pth'),'w') as pth_file:
pth_file.write(str(to_add))
Just to add on awesomo's answer, you can also add that line into your ~/.bash_profile or ~/.profile
The add a new path to PYTHONPATH is doing in manually by:
adding the path to your ~/.bashrc profile, in terminal by:
vim ~/.bashrc
paste the following to your profile
export PYTHONPATH="${PYTHONPATH}:/User/johndoe/pythonModule"
then, make sure to source your bashrc profile when ever you run your code in terminal:
source ~/.bashrc
Hope this helps.
I added permanently in Windows Vista, Python 3.5
System > Control Panel > Advanced system settings > Advanced (tap) Environment Variables > System variables > (if you don't see PYTHONPATH in Variable column) (click) New > Variable name: PYTHONPATH > Variable value:
Please, write the directory in the Variable value. It is details of Blue Peppers' answer.
Fix Python Path issues when you switch from bash to zsh
I ran into Python Path problems when I switched to zsh from bash.
The solution was simple, but I failed to notice.
Pip was showing me, that the scripts blah blah or package blah blah is installed in ~/.local/bin which is not in path.
After reading some solutions to this question, I opened my .zshrc to find that the solution already existed.
I had to simply uncomment a line:
Take a look
I found a solution to do this in a anaconda environment here: https://datacomy.com/python/anaconda/add_folder_to_path/
Just:
conda develop /your_path
In Python 3.6.4 you can persist sys.path across python sessions like this:
import sys
import os
print(str(sys.path))
dir_path = os.path.dirname(os.path.realpath(__file__))
print(f"current working dir: {dir_path}")
root_dir = dir_path.replace("/util", '', 1)
print(f"root dir: {root_dir}")
sys.path.insert(0, root_dir)
print(str(sys.path))
I strongly suggest you use virtualenv and virtualenvwrapper otherwise you will clutter your path
Inspired by andrei-deusteanu answer, here is my version. This allows you to create a number of additional paths in your site-packages directory.
import os
# Add paths here. Then Run this block of code once and restart kernel. Paths should now be set.
paths_of_directories_to_add = [r'C:\GIT\project1', r'C:\GIT\project2', r'C:\GIT\project3']
# Find your site-packages directory
pathSitePckgs = os.path.join(os.path.dirname(os.__file__), 'site-packages')
# Write a .pth file in your site-packages directory
pthFile = os.path.join(pathSitePckgs,'current_machine_paths.pth')
with open(pthFile,'w') as pth_file:
pth_file.write('\n'.join(paths_of_directories_to_add))
print(pthFile)
After multiple bashing into wall. Finally resolved, in my CentOS 8 the pip3 was old, which was showing error to install the recent packages.
Now, I had downloaded the Python source package, which is Python-3.10.4 and installed the usual way, however the post-installation check generated errors in bash.
And I could not remove the existing Python, because that would break the CentOS desktop features.
Solution:
For building
./configure //don't not add --prefix=/usr, which you need to set proper care
make -j8
sudo make install
Now, as you have multiple Python installed, you can set alias python=python3
And for setting PYTHONPATH
export PYTHONPATH="/usr/local/bin/python3.10:/usr/local/lib/python3.10/lib-dynload:/usr/local/lib/python3.10/site-packages"
Don't add PYTHONHOME
For those who (like me) don't want to get too deeply involved in Python file management (which seems hopelessly overcomplicated), creating a .pth file works perfectly on my Windows 11 laptop (I'm using Visual Studio Code in Windows). So just go to the folder for your virtual environment site packages - there's mine:
Create a text file with a .pth extension - I called mine wheal.pth:
Add paths to it:
The best thing about this in VS Code is that import statements recognise this path (I had to exit VS Code and go back in), so now more typing # type: ignore to suppress linting warning messages!
on Mac :
user#terminal$ env PYTHONPATH=module_path python3
>>> import sys
>>> sys.path
['module_path', 'plus_other_python3_paths',...]
Shortest path between A <-> B is a straight line;
import sys
if not 'NEW_PATH' in sys.path:
sys.path += ['NEW_PATH']
I have a local git repository on my machine, let's say under /develop/myPackage.
I'm currently developing it as a python package (a Django app) and I would like to access it from my local virtualenv. I've tried to include its path in my PYTHONPATH (I'm on a Mac)
export PATH="$PATH:/develop/myPackage"
The directory already contains a __init__.py within its root and within each subdirectory.
No matter what I do but I can't get it work, python won't see my package.
The alternatives are:
Push my local change to github and install the package within my virtualenv from there with pip
Activate my virtualenv and install the package manually with python setup.py install
Since I often need to make changes to my code the last two solution would require too much work all the time even for a small change.
Am I doing something wrong? Would you suggest a better solution?
Install it in editable mode from your local path:
pip install -e /develop/MyPackage
This actually symlinks the package within your virtualenv so you can keep on devving and testing.
The example you show above uses PATH, and not PYTHONPATH. Generally, the search path used by python is partially predicated on the PYTHONPATH environment variable (PATH has little use for this case.)
Try this:
export PYTHONPATH=$PYTHONPATH:/develop/myPackage
Though in reality, you likely want it to be pointing to the directory that contains your package (so you can do 'import myPackage', rather than importing things within the package. That being said, you likely want:
export PYTHONPATH=$PYTHONPATH:/develop/
Reference the python docs here for more information about Python's module/package search path: http://docs.python.org/2/tutorial/modules.html#the-module-search-path
By default, Python uses the packages that it was installed with as it's default path, and as a result PYTHONPATH is unset in the environment.
My current setup is Python 2.5/ Django 1.1.1 on Windows. I want to start using Django 1.2 on some projects, but can't use it for everything. Which is just the sort of thing I've got virtualenv for. However, I'm running into a problem I've never encountered and it's hard to Google for: installing Django 1.2 into a virtualenv has no effect for me. If I then activate the environment and do
python
import django
django.VERSION
I get "1.1.1 Final". Django is installed in the site-packages directory of my environment and the init file in the root shows that it is 1.2. But the environment falls back to 1.1.1, even if I create the environment with the --no-site-packages flag. What am I screwing up?
Based on the bug you filed at bitbucket, it looks like you're using the PYTHONPATH environment variable to point to a directory with some packages, including Django 1.1.1. By design, PYTHONPATH always comes first in your sys.path, even when you have a virtualenv activated (because PYTHONPATH is under your direct and immediate control, and people use it for local overrides).
In this case, if you don't want that PYTHONPATH when this virtualenv is activated, you'll need to take care of that yourself; perhaps by creating a custom batch file that both calls the virtualenv's activate.bat and also modifies PYTHONPATH.
Some tools you can use to diagnose these problems:
To see where django is coming from, print django.__file__. It will show the file indicating django's location on the file system.
To see all the places Python will look for packages, print sys.path. It's a list of directories.
To see imports as they happen, start python as python -v, and you'll see lots of debugging information about where packages are being imported.
I'm trying to change my PYTHONPATH. I've tried to change it in "My Computer" etc, but it doesn't exist there. I searched in the registry in some places, and even ran a whole search for the word 'PYTHONPATH', but to no avail.
However, it Python I can easily see it exists. So where is it?
At runtime, you can change it with:
import sys
sys.path.append('...')
In My Computer, right-click Properties (or press Win-Break), System tab, Environment Variables, System. You can add it if it's not already there.
Finally, in the CMD prompt:
set PYTHONPATH C:\Python25\Lib;C:\MyPythonLib
Or in bash:
PYTHONPATH=/usr/share/python/lib:/home/me/python
export PYTHONPATH
Or, more succinctly:
export PYTHONPATH=/home/me/python
Python does some stuff up front when it is started, probably also setting that path in windows. Just set it and see, if it is changed in sys.path.
Setting environment variables in the Python docs say:
My Computer ‣ Properties ‣ Advanced ‣ Environment Variables
You can add it under "My Computer" if it doesn't exist. PYTHONPATH just adds to the default sys.path.
On unix/linux/osx you can:
$ export PYTHONPATH=/to/my/python/libs
You can also use .pth files to point to libraries:
http://docs.python.org/library/site.html#module-site
And of course:
import sys
sys.path.append('/path/to/libs/')
Also, check out virtualenv for managing libraries for multiple projects.
Here's how I solved it.
First, get the current path. There's a lot more there than I expected.
import sys
print ';'.join(sys.path)
Copy that result to the clipboard. Go to My Computer and create the new environment variable PYTHONPATH, and for the value paste from the clipboard. Modify as necessary.
MacOS 10.5.8, Python 2.6, Eclipse+Pydev 1.5.7
Python installation's site-package is, for example:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
create symlinks YOUR LIBRARY inside into site-package, for example:
cd /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
ln -s /path/to/YOUR/LIBRARY/ YOUR_LIBRARY_NAME
Now You can use in commandline: import YOUR_LIBRARY_NAME
run Eclipse with Pydev, go to Preferences->Pydev->Interpreter Python
remove Your Python interpreter record, if exists;
click New and add Python 2.6 interpreter path, for example:
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
notice, that Eclipse Pydev display Python System Library, accept that
in Library section click New Folder and write path to YOUR LIBRARY, for example:
/path/to/YOUR/LIBRARY/
click Apply - it is essential, because Eclipse Pydev built now his own "library map", when this operation finish - click [OK]
close Eclipse
run Eclipse again - now You should use in Pydev: import YOUR_LIBRARY_NAME
And, as with all good things in life, you can find it in the documentation:
http://docs.python.org/install/index.html#modifying-python-s-search-path
What's it set to? Have you tried creating a PYTHONPATH environment variable?
You need modify your environment variables. How to do this depends on which version of Windows you have.
If the PYTHONPATH variable doesn't exist, you have to create it. It might not exist if you haven't already created it.
I had same problem and oliver-zehentleitner's answer in github solved my problem.
He said: Maybe You install package with pip for python2 and run with python3, just try to install with pip3 or python3 -m pip install python-binance and then run your script again.
I hope this can solve yours too.