We Keep Coding

Importing files from different folder, *project-wide*

There is a popular Python question called Importing files from different folder.
But the top answer there mentions adding stuff to "some_file.py", and that will obviously only apply to imports inside that file.
What if I want to specify an additional dir to import from, project-wide?
I don't want to modify PYTHONPATH, as I believe a per-project solution is cleaner.
I don't want to use python packages for this, because I feel they'll probably just complicate stuff. E.g. maybe I'll need to manually recompile .py files into .pyc files every time I make a change to the code in the other folder.

What if I want to specify an additional dir to import from, project-wide?
Solution 1: Package installed in develop mode
Create a regular package with setup.py and install it with -e option:
python -m pip install -e /path/to/dir_with_setup_py/
-e, --editable Install a project in editable mode (i.e. setuptools "develop mode") from a local project
path or a VCS url.
Now, as soon as you update your code, the new version will be used at import
without reinstalling anything.
Solution 2: Dynamically modify sys.path
You can add as many directories dynamically to the search path for Python packages as you want. Make this the very first lines of code you execute:
import sys
sys.path.append('my/path/to/my/file1')
sys.path.append('my/path/to/my/file2')
or to make the first to be found:
sys.path.insert(0, 'my/path/to/my/file1')
sys.path.insert(0, 'my/path/to/my/file2')
Now the files:
my/path/to/my/file1/myscript1.py
my/path/to/my/file2/myscript2.py
can be imported anywhere in your project:
import myscript1
import myscript2
No need to modify sys.path again as long as this Python process is running.

how to set different PYTHONPATH variables for python3 and python2 respectively

I want to add a specific library path only to python2. After adding export PYTHONPATH="/path/to/lib/" to my .bashrc, however, executing python3 gets the error: Your PYTHONPATH points to a site-packages dir for Python 2.x but you are running Python 3.x!
I think it is due to that python2 and python3 share the common PYTHONPATH variable.
So, can I set different PYTHONPATH variables respectively for python2 and python3. If not, how can I add a library path exclusively to a particular version of python?

PYTHONPATH is somewhat of a hack as far as package management is concerned. A "pretty" solution would be to package your library and install it.
This could sound more tricky than it is, so let me show you how it works.
Let us assume your "package" has a single file named wow.py and you keep it in /home/user/mylib/wow.py.
Create the file /home/user/mylib/setup.py with the following content:
from setuptools import setup
setup(name="WowPackage",
packages=["."],
)
That's it, now you can "properly install" your package into the Python distribution of your choice without the need to bother about PYTHONPATH. As far as "proper installation" is concerned, you have at least three options:
"Really proper". Will copy your code to your python site-packages directory:
$ python setup.py install
"Development". Will only add a link from the python site-packages to /home/user/mylib. This means that changes to code in your directory will have effect.
$ python setup.py develop
"User". If you do not want to write to the system directories, you can install the package (either "properly" or "in development mode") to /home/user/.local directory, where Python will also find them on its own. For that, just add --user to the command.
$ python setup.py install --user
$ python setup.py develop --user
To remove a package installed in development mode, do
$ python setup.py develop -u
or
$ python setup.py develop -u --user
To remove a package installed "properly", do
$ pip uninstall WowPackage
If your package is more interesting than a single file (e.g. you have subdirectories and such), just list those in the packages parameter of the setup function (you will need to list everything recursively, hence you'll use a helper function for larger libraries). Once you get a hang of it, make sure to read a more detailed manual as well.
In the end, go and contribute your package to PyPI -- it is as simple as calling python setup.py sdist register upload (you'll need a PyPI username, though).

You can create a configuration file mymodule.pth under lib/site-packages (on Windows) or lib/pythonX.Y/site-packages (on Unix and Macintosh), then add one line containing the directory to add to python path.
From docs.python2 and docs.python3:
A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path. Non-existing items are never added to sys.path, and no check is made that the item refers to a directory rather than a file. No item is added to sys.path more than once. Blank lines and lines beginning with # are skipped. Lines starting with import (followed by space or tab) are executed.

I found that there is no way to modify PYTHONPATH that is only for python2 or only for python3. I had to use a .pth file.
What I had to do was:
make sure directory is created in my home: $HOME/.local/lib/python${MAJOR_VERSION}.${MINOR_VERSION}/site-packages
create a .pth file in that directory
test that your .pth file is work
done
For more info on `.pth. file syntax and how they work please see: python2 docs and python3 docs.
(.pth files in a nutshell: when your python interpreter starts it will look in certain directories and see the .pth file, open those files, parse the files, and add those directories to your sys.path (i.e. the same behavior as PYTHONPATH) and make any python modules located on those directories available for normal importing.)

If you don't want to bother with moving/adding documents in lib/site-packages, try adding two lines of code in the python2.7 script you would like to run (below.)
import sys
sys.path = [p for p in sys.path if p.startswith(r'C:\Python27')]
This way, PYTHONPATH will be updated (ignore all python3.x packages) every time you run your code.

Managing PYTHONPATH for development environment

I'm developing several different Python packages with my team. Say that we have ~/src/pkg1, ~/src/pkg2, and ~/src/pkg3. How do we add these to PYTHONPATH without each of us having to manage dot-files?
We could add, say, ~/src/site/sitecustomize.py, which is added once to PYTHONPATH, but is it "guaranteed" that there won't be a global sitecustomize.py.
virtualenv seems like the wrong solution, because we don't want to have to build/install the packages after each change.

You have a lot of options...
1) Why not dotfiles?
You could centralize the management of dotfiles with a centralized repository and optionally with version control. I use a Dropbox folder named dotfiles but many people use github or other services like that to manage dotfiles.
If you do that, you will guarantee every people on your development team to share some dotfiles. So you could define a dotfile say .python_proys which export the appropriate PATH and PYTHONPATH which by convention every developer should source in their environment.
Suppose pkg1 is only an script, pkg2 is an script and also a module and pk3 is only a module. Then, python_proys Example:
export PATH=$PATH:~/src/pkg1:~/src/pkg2
export PYTHONPATH=$PYTHONPATH:~/src/pkg2:~/src/pkg3
And then, every developer have to source this dotfile somewhere by convetion. Each one will do the way he like. One could source the dotfile manually before using the packages. Another one could source it in his .bashrc or .zshenv or whatever dotfile apply to him.
The idea is to have one centralized point of coordination and only one dotfile to maintain: the .python_proys dotfile.
2) Use symlinks
You could define a directory in your home, like ~/dist (for modules) and ~/bin (for scripts) and set symbolic links there to the specific pakages in ~/src/, and make every developer have this PATH and PYTHONPATH setting:
export PATH=$PATH:~/bin
export PYTHONPATH=$PYTHONPATH:~/dist
So, using the same example at Why not dotfiles?, where pkg1 is only an script, pkg2 is an script and also a module and pkg3 is only a module, then you could symlink like:
cd ~/bin
ln -s ../src/pkg1
ln -s ../src/pkg2
cd ~/dist
ln -s ../src/pkg2
ln -s ../src/pkg3
Those commands could be do automatically with an script. You could write a bootstrap script, or simply copy and paste the commands and save it in a shell script. In any way, maintain it and centralize it the same way i explain it before.
This way the .dotfiles will not change, only the script defining the symlinks.

I suggest looking into creating a name.pth path configuration file as outlined in thesitemodule's documentation. These files can hold multiple paths that will be added tosys.pathand be easily edited since they're simply text files.

First, you don't add a python module to PYTHONPATH, you just add the path component.
If you want all your team to be working on some python package, you can install the package as editable with the -e option in a virtual environment.
This way you can continue development and you don't have to mess with the PYTHONPATH. Keep in mind that the working directory is always included in the PYTHONPATH, so unless you have an external requirement; you don't need a virtual environment, just the source in your working directory.
Your workflow would be the following:
Create virtual environment
Create a .pth file, to modify your PYTHONPATH.
Work as usual.
This would be my preferred option. If you have a standard layout across your projects, you can distribute a customized bootstrap script which will create the environment, and then adjust the PYTHONPATH automatically. Share this bootstrap script across the team, or add it as part of the source repository.

I assume that your other modules are at predictable path (relative to $0)
We can compute absolute path of $0
os.path.realpath(sys.argv[0])
then arrive at your module path and append it
sys.path.append(something)

PYTHONPATH conflict

I am trying to import ZipCodeDatabase in helloworld.py.
helloworld.py exists at /google-app-engine/helloworld
ZipCodeDatabase module exists /usr/local/lib/python/python2.7/dist-packages
PYTHONPATH = /usr/local/lib/python/python2.7/dist-packages;/usr/local/lib/python/
When compiling helloworld I am still getting "ZipCodeDatabase module not found". Why isn't it being picked from the PYTHONPATH?

I highly doubt you've got a module called ZipCodeDatabase. That naming convention is typically reserved for a class that resides within a module. Modules are usually lowercase or lower_snake_case, to represent the file containing the module. I'm assuming you've installed pyzipcode here, but it may be a different module.
# assuming pyzipcode.py in the dist-packages directory
$ python -c 'from pyzipcode import ZipCodeDatabase'
If I'm wrong above, then are you sure you're running the version of python that has the ZipCodeDatabase module installed?
Some troubleshooting steps:
$ which python
$ python --version
$ python -c 'import ZipCodeDatabase'
$ ls -l /usr/local/lib/python2.7/dist-packages/ | grep -i zip
Also, is it really necessary for you to specify the PYTHONPATH line? Typically, the site-packages folder (and by extension I assume the dist-packages folder on Ubuntu) is included in the default PYTHONPATH, along with the current directory of the python module you're using.
How did you install the ZipCodeDatabase? Did you just drop the file in there? Try putting it alongside your helloworld.py file and try importing it then. Also, a full stack trace is useful information here, especially when others are trying to diagnose the problem you're having.
Edit:
Ok, now that I know you're using google app engine (should have been obvious from your use of paths - I'm sorry), it looks like it doesn't use the site-packages or dist-packages to load modules. You should create a sub-directory in your project with the relevant third party libraries, and add that sub-directory to your path. Disclaimer: I've never used GAE so I might be missing the mark with this.
Check out this answer for how to structure your project and add the extra directory to your path from within the application.

Python: using downloaded modules

I am new to Python and mostly used my own code. But so now I downloaded a package that I need for some problem I have.
Example structure:
root\
externals\
__init__.py
cowfactory\
__init__.py
cow.py
milk.py
kittens.py
Now the cowfactory's __init__.py does from cowfactory import cow. This gives an import error.
I could fix it and change the import statement to from externals.cowfactory import cow but something tells me that there is an easier way since it's not very practical.
An other fix could be to put the cowfactory package in the root of my project but that's not very tidy either.
I think I have to do something with the __init__.py file in the externals directory but I am not sure what.

Inside the cowfactory package, relative imports should be used such as from . import cow. The __init__.py file in externals is not necessary. Assuming that your project lies in root\ and cowfactory is the external package you downloaded, you can do it in two different ways:
Install the external module
External Python packages usually come with a file "setup.py" that allows you to install it. On Windows, it would be the command "setup.py bdist_wininst" and you get a EXE installer in the "dist" directory (if it builds correctly). Use that installer and the package will be installed in the Python installation directory. Afterwards, you can simply do an import cowfactory just like you would do import os.
If you have pip or easy_install installed: Many external packages can be installed with them (pip even allows easy uninstallation).
Use PYTHONPATH for development
If you want to keep all dependencies together in your project directory, then keep all external packages in the externals\ folder and add the folder to the PYTHONPATH. If you're using the command line, you can create a batch file containing something like
set PYTHONPATH=%PYTHONPATH%:externals
yourprogram.py
I'm actually doing something similar, but using PyDev+Eclipse. There, you can change the "Run configurations" to include the environment variable PYTHONPATH with the value "externals". After the environment variable is set, you can simply import cowfactory in your own modules. Note how that is better than from external import cowfactory because in the latter case, it wouldn't work anymore once you install your project (or you'd have to install all external dependencies as a package called "external" which is a bad idea).
Same solutions of course apply to Linux, as well, but with different commands.

generally, you would use easy_install our pip to install it for you in the appropriate directory. There is a site-packages directory on windows where you can put the package if you can't use easy_install for some reason. On ubuntu, it's /usr/lib/pythonX.Y/dist-packages. Google for your particular system. Or you can put it anywhere on your PYTHONPATH environment variable.
As a general rule, it's good to not put third party libs in your programs directory structure (although there are differing opinions on this vis a vis source control). This keeps your directory structure as minimalist as possible.

The easiest way is to use the enviroment variable $PYTHONPATH. You set it before running your scripts as follows:
export $PYTHONPATH=$PYTHONPATH:/root/externals/
You can add as many folders as you want (provided their separate by :) and python will look in all those folders when importing.