I'm developing several different Python packages with my team. Say that we have ~/src/pkg1, ~/src/pkg2, and ~/src/pkg3. How do we add these to PYTHONPATH without each of us having to manage dot-files?
We could add, say, ~/src/site/sitecustomize.py, which is added once to PYTHONPATH, but is it "guaranteed" that there won't be a global sitecustomize.py.
virtualenv seems like the wrong solution, because we don't want to have to build/install the packages after each change.
You have a lot of options...
1) Why not dotfiles?
You could centralize the management of dotfiles with a centralized repository and optionally with version control. I use a Dropbox folder named dotfiles but many people use github or other services like that to manage dotfiles.
If you do that, you will guarantee every people on your development team to share some dotfiles. So you could define a dotfile say .python_proys which export the appropriate PATH and PYTHONPATH which by convention every developer should source in their environment.
Suppose pkg1 is only an script, pkg2 is an script and also a module and pk3 is only a module. Then, python_proys Example:
export PATH=$PATH:~/src/pkg1:~/src/pkg2
export PYTHONPATH=$PYTHONPATH:~/src/pkg2:~/src/pkg3
And then, every developer have to source this dotfile somewhere by convetion. Each one will do the way he like. One could source the dotfile manually before using the packages. Another one could source it in his .bashrc or .zshenv or whatever dotfile apply to him.
The idea is to have one centralized point of coordination and only one dotfile to maintain: the .python_proys dotfile.
2) Use symlinks
You could define a directory in your home, like ~/dist (for modules) and ~/bin (for scripts) and set symbolic links there to the specific pakages in ~/src/, and make every developer have this PATH and PYTHONPATH setting:
export PATH=$PATH:~/bin
export PYTHONPATH=$PYTHONPATH:~/dist
So, using the same example at Why not dotfiles?, where pkg1 is only an script, pkg2 is an script and also a module and pkg3 is only a module, then you could symlink like:
cd ~/bin
ln -s ../src/pkg1
ln -s ../src/pkg2
cd ~/dist
ln -s ../src/pkg2
ln -s ../src/pkg3
Those commands could be do automatically with an script. You could write a bootstrap script, or simply copy and paste the commands and save it in a shell script. In any way, maintain it and centralize it the same way i explain it before.
This way the .dotfiles will not change, only the script defining the symlinks.
I suggest looking into creating a name.pth path configuration file as outlined in thesitemodule's documentation. These files can hold multiple paths that will be added tosys.pathand be easily edited since they're simply text files.
First, you don't add a python module to PYTHONPATH, you just add the path component.
If you want all your team to be working on some python package, you can install the package as editable with the -e option in a virtual environment.
This way you can continue development and you don't have to mess with the PYTHONPATH. Keep in mind that the working directory is always included in the PYTHONPATH, so unless you have an external requirement; you don't need a virtual environment, just the source in your working directory.
Your workflow would be the following:
Create virtual environment
Create a .pth file, to modify your PYTHONPATH.
Work as usual.
This would be my preferred option. If you have a standard layout across your projects, you can distribute a customized bootstrap script which will create the environment, and then adjust the PYTHONPATH automatically. Share this bootstrap script across the team, or add it as part of the source repository.
I assume that your other modules are at predictable path (relative to $0)
We can compute absolute path of $0
os.path.realpath(sys.argv[0])
then arrive at your module path and append it
sys.path.append(something)
Related
I'm confused as to where I should put my virtualenvs.
With my first django project, I created the project with the command
django-admin.py startproject djangoproject
I then cd'd into the djangoproject directory and ran the command
virtualenv env
which created the virtual environment directory at the same level as the inner djangoproject directory.
Is this the wrong place in which to create the virtualenv for this particular project?
I'm getting the impression that most people keep all their virtualenvs together in an entirely different directory, e.g. ~/virtualenvs, and then use virtualenvwrapper to switch back and forth between them.
Is there a correct way to do this?
Many people use the virtualenvwrapper tool, which keeps all virtualenvs in the same place (the ~/.virtualenvs directory) and allows shortcuts for creating and keeping them there. For example, you might do:
mkvirtualenv djangoproject
and then later:
workon djangoproject
It's probably a bad idea to keep the virtualenv directory in the project itself, since you don't want to distribute it (it might be specific to your computer or operating system). Instead, keep a requirements.txt file using pip:
pip freeze > requirements.txt
and distribute that. This will allow others using your project to reinstall all the same requirements into their virtualenv with:
pip install -r requirements.txt
Changing the location of the virtualenv directory breaks it
This is one advantage of putting the directory outside of the repository tree, e.g. under ~/.virtualenvs with virutalenvwrapper.
Otherwise, if you keep it in the project tree, moving the project location will break the virtualenv.
See: Renaming a virtualenv folder without breaking it
There is --relocatable but it is known to not be perfect.
Another minor advantage: you don't have to .gitignore it.
The advantages of putting it gitignored in the project tree itself are:
keeps related stuff close together.
you will likely never reuse a given virtualenv across projects, so putting it somewhere else does not give much advantage
This is an annoying design flaw in my opinion. They should implement virutalenv in a way that does not matter where the directory is, as storing in-tree is just simpler and more isolated. Node.js' NPM package manager does it without any problem. And while we are at it: pip should just use local directories by default just like NPM. Having this separate virtualenv layer is wonky. Node.js just have NPM that does it all without extra typing. I can't believe I'm prasing the JavaScript ecosystem on a Python post, but it's true.
The generally accepted place to put them is the same place that the default installation of virtualenvwrapper puts them: ~/.virtualenvs
Related: virtualenvwrapper is an excellent tool that provides shorthands for the common virtualenv commands. http://www.doughellmann.com/projects/virtualenvwrapper/
If you use pyenv install Python, then pyenv-virtualenv will be a best practice. If set .python-version file, it can auto activate or deactivate virtual env when you change work folder. Pyenv-virtualenv also put all virtual env into $HOME/.pyenv/versions folder.
From my personal experience, I would recommend to organize all virtual environments in one single directory. Unless someone has extremely sharp memory and can remember files/folders scattered across file system.
Not a big fan of using other tools just to mange virtual environments. In VSCode if I configure(python.venvPath) directory containing all virtual environments, it can automatically recognize all of them.
For Anaconda installations of Python, the "conda create" command puts it in a directory within the anaconda3 folder by default. Specifically (for Windows):
C:\Users\username\anaconda3\envs
This allows other conda commands to work without specifying the path. One advantage, not noted above, is that putting environments in the project folder allows you to use the same name for all of them (but that is not much of an advantage for me). For more info, see:
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
now I have a folder named my_funcs which have __init__.py and some .py files containing some functions and classes I wrote that I want to use for several projects.
So I want to know the best practice for these projects to direct import from this folder.
one solution is to sys.path.append('.../my_funcs'), in this case I will have to put this in front of the import statement for every .py file.
Any suggestions? BTW, I'm on Windows
Best is to use PYTHONPATH. Set it to the path where your common modules are found, before running Python. Then you can just do import my_funcs for example.
Checkout PEP370, "Per user site-packages directory".
You set PYTHONUSERBASE to somewhere under your control, and you can see
$ PYTHONUSERBASE=/home/me/.local
$ python -m site --user-site
/home/me/.local/lib/python2.7/site-packages
Your personal directory now appears in sys.path. easy_install respects this (using the --user option) so you can install "real" packages there, also, but make them available only to you.
Depending on your configuration, sounds like you could move your my_funcs directory under the site-packages directory and you're done!
I have a number of django projects organized with the following directory structure using win7 (I'm using GIT_BASH/mingw for my command line) :
envs--r1--project1
--project2
pPython275--
My files are on portable flash drive which is in an adjacent directory to 'envs'
I want to end up with the different projects having a common environment that I activate from each projects root directory using :
$ source ../Scripts/activate
Is this file structure OK or do I need to make changes to create a common virtualenv using the python interpreter at:
f:/pPython275/python.exe
Personal Opinion
I personally prefer keeping my projects outside the virtualenv, which helps me if i need to clone (copy) the virtualenv.
the structure I use is
envs--r1--
--Library
--Scripts--python.exe
projects--
--project1
--project2
pPython275--
You could activate your venv from anywhere. and relative path like you specified would work too
$ source ../envs/r1/Scripts/activate
I have a Python project which contains three components: main executable scripts, modules which those scripts rely on, and data (sqlite3 databases, flat files, etc.) which those scripts manipulate. The top level has an __init__.py file so that other programs can also borrow from the modules if needed.
The question is, is it more "Pythonic" or "correct" to move my project into the default site-packages directory, or to modify PYTHONPATH to include one directory above my project (so that the project can be imported from)? On the one hand, what I've described is not strictly a "package", but a "project" with data that can be treated as a package. So I'm leaning in the direction of modifying PYTHONPATH (after all, PYTHONPATH must exist for a reason, right?)
Definitely do not add your project to site-packages this is going to spoil your system Python installation and will fire back at the moment some other app would come there or you would need to install something.
There are at last two popular options for installing python apps in isolated manner
Using virtualenv
See virtualenv project. It allows
creation of new isolating python environment - python for this environment is different from system one and has it's own PYTHONPATH setup this allows to keep all installed packages private for it.
activation and deactivation of given virtualenv from command line for command line usage. After activate, you can run pip install etc. and it will affect only given virtualenv install.
calling any Python script by starting by virtualenv Python copy - this will use related virtualenv (note, that there is no need to call any activate)
using zc.buildout
This package provides command buildout. With this, you can use special configuration file and this allows creation of local python environment with all packages and scripts.
Conclusions
virtualenv seems more popular today and I find it much easier to learn and use
zc.buildout might be also working for you, but be prepared for a bit longer learning time
installing into system Python directories shall be always reserved for very special cases (pip, easy_install), better avoid it.
installing into private directories and manipulatig PYTHONPATH is also an option, but you would repeat, what virtualenv already provides
I have a local git repository on my machine, let's say under /develop/myPackage.
I'm currently developing it as a python package (a Django app) and I would like to access it from my local virtualenv. I've tried to include its path in my PYTHONPATH (I'm on a Mac)
export PATH="$PATH:/develop/myPackage"
The directory already contains a __init__.py within its root and within each subdirectory.
No matter what I do but I can't get it work, python won't see my package.
The alternatives are:
Push my local change to github and install the package within my virtualenv from there with pip
Activate my virtualenv and install the package manually with python setup.py install
Since I often need to make changes to my code the last two solution would require too much work all the time even for a small change.
Am I doing something wrong? Would you suggest a better solution?
Install it in editable mode from your local path:
pip install -e /develop/MyPackage
This actually symlinks the package within your virtualenv so you can keep on devving and testing.
The example you show above uses PATH, and not PYTHONPATH. Generally, the search path used by python is partially predicated on the PYTHONPATH environment variable (PATH has little use for this case.)
Try this:
export PYTHONPATH=$PYTHONPATH:/develop/myPackage
Though in reality, you likely want it to be pointing to the directory that contains your package (so you can do 'import myPackage', rather than importing things within the package. That being said, you likely want:
export PYTHONPATH=$PYTHONPATH:/develop/
Reference the python docs here for more information about Python's module/package search path: http://docs.python.org/2/tutorial/modules.html#the-module-search-path
By default, Python uses the packages that it was installed with as it's default path, and as a result PYTHONPATH is unset in the environment.