Python uses the PYTHONPATH environment-variable to determine in which folders it should look for modules.
You can play around with it by modifying sys.path, which works nicely for pure Python-Modules.
But when a module uses shared object files or static libraries, it looks for those in LD_LIBRARY_PATH (on linux), but this can't be changed as easily and is platform dependent as far as I know.
The quick-fix for this problem is of course to set the environment-variable or invoke the script like LD_LIBRARY_PATH=. ./script.py, but then you'll have to set it again for every new shell you open.
Also, the .so files in my case will always be in the same directory as the .py file, but may very well be moved to another absolute path, so I'd like to set them automatically every time I invoke the script.
How can I edit the path in which the Python interpreter looks for libraries platform-independently on runtime?
EDIT:
I already tried os.environ['LD_LIBRARY_PATH'] = os.getcwd(), but to no avail.
UPDATE: see the EDIT below.
I would use:
import os
os.environ['LD_LIBRARY_PATH'] = os.getcwd() # or whatever path you want
This sets the LD_LIBRARY_PATH environment variable for the duration/lifetime of the execution of the current process only.
EDIT: it looks like this needs to be set before starting Python: Changing LD_LIBRARY_PATH at runtime for ctypes
So I'd suggest going with a wrapper .sh (or .py if you insist) script. Also, as #chepner pointed out, you might want to consider installing your .so files in a standard location (within the virtualenv).
See also Setting LD_LIBRARY_PATH from inside Python
My solution to this problem is to put this as the first line of a Python script (instead of the usual shebang):
exec env LD_LIBRARY_PATH=/some/path/to/lib /path/to/specific/python -x "$0" "$#"
And here is how this works:
with no shebang the current shell treats the file as a shell script,
"exec" ensures that this first line is also the last command from this file executed by the shell,
"env" is used here to set any environment variables, e.g. LD_LIBRARY_PATH,
an exact path to Python's interpreter can specified or "env" can find one in PATH,
"-x" is a Python's option which causes the first line to be ignored by the Python interpreter,
"$0" is the script name, "$#" is substituted by positional parameters.
Python, when gets the values of environment variables as in os.environ['LD_LIBRARY_PATH'] or os.environ['PATH'], it copies the values, into a dictionary, from it's parent process's environment, generally bash (bash process's environment gets carried to the child process, the python running instance).
you can see this environment variable section with env command output from bash.
you can also see/read this env data from /proc/<pid>/environ, by introducing an infinite loop(while 1: pass) after modifying any environment variable.
If you see/read this variable value/data from /proc/<pid>/environ after modifying it inside the python script, you would get to see that the real variable's data doesn't get modified, though the python script shows a modified dictionary key value, updated.
What actually happens when you modify an env variable inside python script, as in os.environ['LD_LIBRARY_PATH']='/<new_location>', is that it just updates the value in local dictionary, which is not mapped to process's env variable section. Hence it won't propagate all the way back to reflect in current process's environment, because ONLY a local dictionary was modified/updated/populated.
Hence if we want to have the new environment variable to be reflected, we should overwrite the memory image of the process with new environment variable data, using execv.
Example:
new_lib = '/<new_location>'
if not new_lib in os.environ['LD_LIBRARY_PATH']:
os.environ['LD_LIBRARY_PATH'] += ':'+new_lib
try:
os.execv(sys.argv[0], sys.argv)
except Exception as e:
sys.exit('EXCEPTION: Failed to Execute under modified environment, '+e)
import xyz
#do something else
Limitation: Ideally, python should not allow such modification of os.environ variables.
But because there is no constant dictionary data type, it allows modification of the data variable. There is absolutely no use of modifying the values, as it does nothing useful to reflect in running process's real environment, unless execv is used.
The solution works fine if the env is reinit
import os
os.environ['LD_LIBRARY_PATH'] = os.getcwd() # or whatever path you want
The code need to be taken in place....
os.execv(sys.argv[0], sys.argv)
Since coreutils 8.30 it is possible to use env -S to split shebang line to separate arguments:
#!/usr/bin/env -S LD_LIBRARY_PATH=/path/to/lib python options
For compatibility with older systems, you can use the fact that the shell allows all commands to be enclosed in quotes, whereas in python it will be just strings:
#!/bin/sh
"export" "LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH"
"exec" "python3" "$0" "$#"
# Further python program
import somemodule
Related
I am trying to access constants set in a python file from a bash script. The constants will be accessed by both python files and bash scripts. At first I thought to just set the variables in python like:
import os
...
constant_variable=1
constant_path='/path/to/file'
etc...
and then
source constants_file.py
in the bash script.
but I have been told to set these constants as environmental variables in python instead as the python file will likely include stuff that I don't want to load into bash using "source" in the future.
I have tried doing something like this in python...
os.environ['CONSTANT_VARIABLE'] = '1'
...
constant_variable = int(os.environ.get('CONSTANT_VARIABLE'))
and then accessing it in bash like so...
constant_variable=${CONSTANT_VARIABLE}
which does not work because I am no longer using source to read in the constants from the python file...
I don't know if this is even possible, but is there some way to just import env variables created by the python file without sourcing the whole file? Or am I way off base here?
Thanks!
If you set the environment variable in python, it will only be set for the process's child process. Unless you're running the bash script FROM the python process, you'll probably need to either generate a file to source with bash, or just define the variables in bash and then read them in python.
I want to know how (if there is a way) to write a function and have it automatically defined when you start a python shell, or to keep certain defined functions after the shell closes so you don't have to keep re defining it if you use it a lot.
Yes, you can, you need to set your environment variable PYTHONSTARTUP to a python script you would like to be executed at every python startup.
By convention, such files are named by adding the rc suffix to the program name you're trying to tweak, and are usually located at the root of your home directory. In that case I would create the python file under $HOME/.pythonrc and then run export PYTHONSTARTUP=~/.pythonrc (for UNIX-like systems, it could be slightly different on Windows if you don't have a MinGW or equivalent).
Here's an example of .pythonrc file that you can play with: https://gist.github.com/twneale/5245670
In the R programming language, there is a site.profile file that defines some code that R processes execute on start up. Is there similar functionality in Python?
Edit: to clarify, this script should be executed if the user calls python from the command line, but also if python is spawned from another process (e.g. if the user's script uses subprocess to spawn another python).
If you only want this for interactive sessions (as opposed to also happening every time you run a script with python myscript.py or ./myscript or a module with python -m mymodule), what you want is the environment variable PYTHONSTARTUP:
If this is the name of a readable file, the Python commands in that file are executed before the first prompt is displayed in interactive mode. The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session…
If you want this to always happen forever, of course, you need to set this environment variable in some appropriate global place—e.g., your shell profile on most *nix platforms, or both your shell profile and your launchd profile on macOS, or the appropriate part of the Control Panel on Windows (the appropriate part changes with almost every new version of Windows, but it usually has "System" in the name).
If you want this to happen for all users, not just the current user… the details for how to set a system-wide environment variable are more platform-specific, but otherwise the idea is the same.
If you want this to happen for every Python session, even when some other program is running a Python script and you didn't even know it was doing that… what you want is either usercustomize or sitecustomize, as documented in the site documentation:
This module is automatically imported during initialization. The automatic import can be suppressed using the interpreter’s -S option.
…
After these path manipulations, an attempt is made to import a module named sitecustomize, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in the site-packages directory.
After this, an attempt is made to import a module named usercustomize, which can perform arbitrary user-specific customizations, if ENABLE_USER_SITE is true. This file is intended to be created in the user site-packages directory (see below), which is part of sys.path unless disabled by -s…
So, you want to find an appropriate place to override this. First try this:
python3 -m site
Then, if this didn't give you sys.path (probably only on pretty old Python, but just in case…), also do this:
python3 -c 'import sys; print('\n'.join(sys.path))'
If you want this customization to happen only for the current user, you want to create a usercustomize.py file in the USER_SITE directory listed by python3 -m site. If the directory doesn't exist, create it.
If you want it to happen for all users, you want a sitecustomize.py file in one of the sys.path directories. The problem is that there may already be one. For example, most linux distros' builtin Python packages have their own sitecustomize modules. If there is, python3 -c 'import sitecustomize; print(sitecustomize.__file__) will tell you where it is. Then, you can edit, or you can copy it, edit that copy, and place that copy somewhere that comes earlier in sys.path than the original. As a general rule, /usr/local is probably better than /usr, and site-packages is probably better than dist-packages is probably better than anything else.
The Python mechanism is called... site. It is a module that is automatically imported and sets up your environment. If it finds modules sitecustomize.py and usercustomize.py, it will import them as well. So these two are where you would put site-wide and personal customizations that you want to be a permanent part of your Python environment. See the site documentation for more details.
File pointed by your environmental variable PYTHONSTARTUP would be run on starting an interactive python session
And USER_SITE/usercustomize.py will be run for non-interactive python session.
Run python -c "import site; print(site.USER_SITE)" to find the USER_SITE dir location
I have installed gdal version 1.11.2, however I have problems with undefined symbols. A proposed solution for this is given by gerrit:
export LD_PRELOAD=/usr/local/lib/libgdal.so.1
However, this only works if I call the python script from the command line. If I call the script from a subprocess.Popen, this will obviously not work.
Is there a way to define the LD_PRELOAD permanently?
you can save the export in your /etc/environment if you want to have it globally for every user
or you can use the env argument like so:
subprocess.Popen(command, env={'LD_PRELOAD': '/usr/local/lib/libgdal.so.1'})
see also here:
Python subprocess/Popen with a modified environment
The best solution IMHO is to fix the problem at OS level. If you want gdalto be accessible to only some users of your platform, each user should add the line
export LD_PRELOAD=/usr/local/lib/libgdal.so.1
in his own .profile file (assuming they all use bash or a compatible shell)
Alternatively, if all users should be able to use gdal, that line should be added to the system /etc/profile file.
I'm finding that python is modifying my path in problematic ways -- in particular, it's pre-pending the path to my github development folder, which results in the wrong libraries being loaded.
In my terminal session, if I run echo $PATH I get:
~$echo $PATH
/Users/Nick/anaconda/bin:/usr/local/bin:/usr/bin:
/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/usr/texbin
Which is perfect. But when I launch python from that same terminal session (either as python or ipython) and check my PATH from within python, I'm getting:
>>> print(sys.path)
['', '/Users/Nick/anaconda/lib/python3.4/site-packages/Cython-0.22.1-py3.4-
macosx-10.5-x86_64.egg', '/Users/Nick/github/pandas',
'/Users/Nick/anaconda/lib/python34.zip', '/Users/Nick/anaconda/lib/python3.4',
'/Users/Nick/anaconda/lib/python3.4/plat-darwin',
'/Users/Nick/anaconda/lib/python3.4/lib-dynload',
'/Users/Nick/anaconda/lib/python3.4/site-packages',
'/Users/Nick/anaconda/lib/python3.4/site-packages/Sphinx-1.3.1-py3.4.egg',
'/Users/Nick/anaconda/lib/python3.4/site-packages/aeosa',
'/Users/Nick/anaconda/lib/python3.4/site-packages/setuptools-18.0.1-py3.4.egg']
Where are these coming from and how do I stop them?
Thanks!
PATH has nothing to do with the Python module search path; that environment variable is used by your shell to find executables, instead.
You need to look at the PYTHONPATH variable here.
If that variable doesn't contain your extra elements, start Python with the -S command line switch to disable loading site.py; it may be that the extra elements are set by a .pth file. Also see the site module documentation:
A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path.
While $PATH seems like it may be used by Python, what you actually want to look at is your $PYTHONPATH -- which is used by the import machinery and logic.
You should look into using virtualenvironments to control the pathing of Python module lookups.