We Keep Coding

Is there a way to always execute a script when python starts? (similar site.profile in R)

In the R programming language, there is a site.profile file that defines some code that R processes execute on start up. Is there similar functionality in Python?
Edit: to clarify, this script should be executed if the user calls python from the command line, but also if python is spawned from another process (e.g. if the user's script uses subprocess to spawn another python).

If you only want this for interactive sessions (as opposed to also happening every time you run a script with python myscript.py or ./myscript or a module with python -m mymodule), what you want is the environment variable PYTHONSTARTUP:
If this is the name of a readable file, the Python commands in that file are executed before the first prompt is displayed in interactive mode. The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session…
If you want this to always happen forever, of course, you need to set this environment variable in some appropriate global place—e.g., your shell profile on most *nix platforms, or both your shell profile and your launchd profile on macOS, or the appropriate part of the Control Panel on Windows (the appropriate part changes with almost every new version of Windows, but it usually has "System" in the name).
If you want this to happen for all users, not just the current user… the details for how to set a system-wide environment variable are more platform-specific, but otherwise the idea is the same.
If you want this to happen for every Python session, even when some other program is running a Python script and you didn't even know it was doing that… what you want is either usercustomize or sitecustomize, as documented in the site documentation:
This module is automatically imported during initialization. The automatic import can be suppressed using the interpreter’s -S option.
…
After these path manipulations, an attempt is made to import a module named sitecustomize, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in the site-packages directory.
After this, an attempt is made to import a module named usercustomize, which can perform arbitrary user-specific customizations, if ENABLE_USER_SITE is true. This file is intended to be created in the user site-packages directory (see below), which is part of sys.path unless disabled by -s…
So, you want to find an appropriate place to override this. First try this:
python3 -m site
Then, if this didn't give you sys.path (probably only on pretty old Python, but just in case…), also do this:
python3 -c 'import sys; print('\n'.join(sys.path))'
If you want this customization to happen only for the current user, you want to create a usercustomize.py file in the USER_SITE directory listed by python3 -m site. If the directory doesn't exist, create it.
If you want it to happen for all users, you want a sitecustomize.py file in one of the sys.path directories. The problem is that there may already be one. For example, most linux distros' builtin Python packages have their own sitecustomize modules. If there is, python3 -c 'import sitecustomize; print(sitecustomize.__file__) will tell you where it is. Then, you can edit, or you can copy it, edit that copy, and place that copy somewhere that comes earlier in sys.path than the original. As a general rule, /usr/local is probably better than /usr, and site-packages is probably better than dist-packages is probably better than anything else.

The Python mechanism is called... site. It is a module that is automatically imported and sets up your environment. If it finds modules sitecustomize.py and usercustomize.py, it will import them as well. So these two are where you would put site-wide and personal customizations that you want to be a permanent part of your Python environment. See the site documentation for more details.

File pointed by your environmental variable PYTHONSTARTUP would be run on starting an interactive python session
And USER_SITE/usercustomize.py will be run for non-interactive python session.
Run python -c "import site; print(site.USER_SITE)" to find the USER_SITE dir location

Permanently define environment variable (for Python gdal undefined symbol)

I have installed gdal version 1.11.2, however I have problems with undefined symbols. A proposed solution for this is given by gerrit:
export LD_PRELOAD=/usr/local/lib/libgdal.so.1
However, this only works if I call the python script from the command line. If I call the script from a subprocess.Popen, this will obviously not work.
Is there a way to define the LD_PRELOAD permanently?

you can save the export in your /etc/environment if you want to have it globally for every user
or you can use the env argument like so:
subprocess.Popen(command, env={'LD_PRELOAD': '/usr/local/lib/libgdal.so.1'})
see also here:
Python subprocess/Popen with a modified environment

The best solution IMHO is to fix the problem at OS level. If you want gdalto be accessible to only some users of your platform, each user should add the line
export LD_PRELOAD=/usr/local/lib/libgdal.so.1
in his own .profile file (assuming they all use bash or a compatible shell)
Alternatively, if all users should be able to use gdal, that line should be added to the system /etc/profile file.

Suppress warning without modifying third party code

We use a third party open source tool and it generates a warning:
DeprecationWarning: The compiler package is deprecated and removed in Python 3.x.
How can I suppress this warning without modifying the third party code?
I know how to use warnings.filter() but I can't use it: I call py.test from the command line, thus no single source code line of my code gets executed before the warning gets created.
Creating a wrapper around the console script is not an option.

You can add a usercustomize or sitecustomize module that calls warnings.filter(). It'll be loaded as the interpreter starts.
See The Customization Modules:
Python provides two hooks to let you customize it: sitecustomize and usercustomize. To see how it works, you need first to find the location of your user site-packages directory. Start Python and run this code:
>>> import site
>>> site.getusersitepackages()
'/home/user/.local/lib/python2.7/site-packages'
Now you can create a file named usercustomize.py in that directory and
put anything you want in it. It will affect every invocation of
Python, unless it is started with the -s option to disable the
automatic import.
sitecustomize works in the same way, but is typically created by an administrator of the computer in the global site-packages directory, and is imported before usercustomize. See the documentation of the site module for more details.
You can influence where Python looks with the PYTHONUSERBASE environment variable, so you can point Python to a per-project usercustomize.py file here, provided you take into account the path lib/python/site-packages is added to the base:
$ python -m site --user-site
/Users/someuser/Library/Python/2.7/lib/python/site-packages
$ PYTHONUSERBASE=/foo/bar python -m site --user-site
/foo/bar/lib/python/site-packages
In the above example, with PYTHONUSERBASE set to /foo/bar, Python will load /foo/bar/lib/python/site-packages/usercustomize.py if it exists.
In a Python virtualenv a customised site.py file is used that is based on a site.py from before Python 2.6; this version omits the getusersitepackages() function. If the file lib/pythonX.X/no-global-site-packages.txt exists, the usercustomize module will not be imported. You'll have to use a sitecustomize.py file in the virtual env lib/python/site-packages directory instead.

What is the best way to organize my local python scripts?

I have a bunch of useful scripts that I want to import from time to time. How to best organize them? I would want them to be on my /home/ folder -- is that possible? Is that the best way?
On a related note, when my other scripts import my local scripts, is there a best practice to make them portable? Should I include notes in my script to alert readers / myself that I'm importing from self-written script?
Thank you!

In your .bashrc you can specify the $PYTHONSTARTUP and $PYTHONPATH parameters. I have the following in my own .bashrc:
export PYTHONSTARTUP=$HOME/.config/python/pythonrc.py
export PYTHONPATH=$PYTHONPATH:$HOME/.config/python/path
Note that The .bashrc file is for bash specifically. Other shells may have other files loaded at startup.
The $PYTHONSTARTUP script is run every time you start a python console. This is useful if you want to add tab completion for example. For example in the case I specified, whenever you run python from the terminal, the script .config/python/pythonrc.py is executed before the console starts.
You can put python packages which should be importable anywhere in the $PYTHONPATH you specified. So basically $PYTHONPATH for python has some similarities to $PATH for bash. Note this is not $PATH. I do not recommend messing with the $PYTHONPATH though. I think it is better to append the paths to sys.path in the $PYTHONSTARTUP script.
And then there is the usercustomize module. If there is a module named usercustomize anywhere in the path, it will be imported by all python processes. For usercustomize to work you do need to make sure that this is in your $PYTHONPATH. For usercustomize you do need to set it in $PYTHONPATH, but you could append more paths in usercustomize.py just like in $PYTHONSTARTUP, so you only need to add 1 more directory to the $PYTHONPATH.

How to add a folder to your path for Mac
You may want to organize your custom scripts into a folder that is not mixed in with the Anaconda modules
1) Find your bash profile. In Terminal on the command line type (without the $ sign, which is the prompt)
$ cd
$ ls -a
This means "change directory to my home folder, list all files including hidden files". You should see a file called .bash_profile listed there
2) Edit your bash profile.Into the command line type
$ open .bash_profile
This should open in TextEdit. Add two lines to the text file:
# Added by My Name on My Date
export PYTHONPATH="/Users/myusername/path/to/folder:$PYTHONPATH"
3) Check that this worked. Open a python session:
>>> import sys
>>> sys.path
The new folder should now be listed in your path

How can I start IPython running a script?

My use case is I want to initialize some functions in a file and then start up IPython with those functions defined. Is there a way to do something like this?
ipython --run_script=myscript.py

In recent versions of IPython, you do need to add the -i option to get into the interactive environment afterwards. Without the -i it just runs the code in myfile.py and returns to the prompt.
ipython -i myfile.py

Per the docs, it's trivial:
You start IPython with the command:
$ ipython [options] files
If invoked with no options, it
executes all the files listed in
sequence and drops you into the
interpreter while still acknowledging
any options you may have set in your
ipythonrc file. This behavior is
different from standard Python, which
when called as python -i will only
execute one file and ignore your
configuration setup.
So, just use ipython myfile.py... and there you are!-)

You can use IPython profiles to define startup scripts that will run every time you start IPython. A full description of profiles, is given here. You can create multiple profiles with different startup files.
Assuming you only need one profile, and always want the same startup files every time you start IPython, you can simply modify the default profile. To do this, first find out where your IPython configuration directory is in an ipython session.:
In [1]: import IPython
In [2]: IPython.paths.get_ipython_dir() # As of IPython v4.0
In [3]: print(ipython_config_dir)
/home/johndoe/.config/ipython
For this example, I am using Ubuntu Linux, and the configuration directory is in /home/johndoe/.config/ipython, where johndoe is the username.
The default_profile is in the profile_default subdirectory. Put any starting scripts in profile_default/startup. In the example here, the full path would be /home/johndoe/.config/ipython/profile_default/startup.

Nowadays, you can use the startup folder of IPython, which is located in your home directory (C:\users\[username]\.ipython on Windows). Go into the default profile and you'll see a startup folder with a README file. Just put any Python scripts in there, or if you want IPython commands, put them in a file with an .ipy extension.

You seem to be looking for IPython's %run magic command.
By typing in IPython:
%run hello_world.py
you'll run the hello.py program saved in your home directory. The functions and variables defined in that script will be accessible to you too.

The following is for the case when you want your startup scripts to automatically be run whenever you use IPython (instead of having a script that you must specify each time you run IPython).
For recent versions (i.e., 5.1.0) of IPython, place one or more python scripts you wish to have executed in the IPYTHON_CONFIG_DIR/profile_PROFILENAME/startup folder.
On Linux, for example, you could put your Python startup code into a file named ~/.ipython/profile_default/startup/10-mystartupstuff.py if you want it to run when no IPython profile is specified.
Information on creating and using IPython profiles is available here.

Update to Caleb's answer for Python 3.5 in Ubuntu 14.04 (Trusty Tahr): I made this answer self-contained by copying relevant parts of Caleb's answer.
You can use IPython profiles to define startup scripts that will run every time you start IPython. A full description of profiles, is given here. You can create multiple profiles with different startup files.
Assuming you only need one profile, and always want the same startup files every time you start IPython, you can simply modify the default profile. To do this, first find out where your IPython configuration directory is in an IPython session:
Input:
import IPython
ipython_config_dir = IPython.paths.get_ipython_dir()
print(ipython_cofig_dir)
Output:
/home/johndoe/.ipython
For this example, johndoe is the username.
Inside the /.ipython folder, the default_profile is in the profile_default subdirectory. Put any starting scripts in profile_default/startup. In the example here, the full path would be
/home/johndoe/.ipython/profile_default/startup