I have installed gdal version 1.11.2, however I have problems with undefined symbols. A proposed solution for this is given by gerrit:
export LD_PRELOAD=/usr/local/lib/libgdal.so.1
However, this only works if I call the python script from the command line. If I call the script from a subprocess.Popen, this will obviously not work.
Is there a way to define the LD_PRELOAD permanently?
you can save the export in your /etc/environment if you want to have it globally for every user
or you can use the env argument like so:
subprocess.Popen(command, env={'LD_PRELOAD': '/usr/local/lib/libgdal.so.1'})
see also here:
Python subprocess/Popen with a modified environment
The best solution IMHO is to fix the problem at OS level. If you want gdalto be accessible to only some users of your platform, each user should add the line
export LD_PRELOAD=/usr/local/lib/libgdal.so.1
in his own .profile file (assuming they all use bash or a compatible shell)
Alternatively, if all users should be able to use gdal, that line should be added to the system /etc/profile file.
Related
I am having difficulty accessing my custom scripts from the command line (python3 unzipit.py "C:\Users\Me\downloads\zipfilehome"). I want to have my scripts act on any file, no matter where they are. I don't want to have the script file in the same directory in order for it to work. I followed this question's top answer to no avail. Note: I am using Windows 10 and all my python versions are in the path with no issues accessing them.
What I did (I always refresh my CLI with every change)
In the system environmental variables:
Path: (unchanged since installation) C:\Path2Python27;C:\Path2Python27\scripts;C:\Path2Python37;C:\Path2Python37\scripts;
PYTHONPATH: C:\Path2Python37;C:\Path2Python37\scripts;C:\Users\Me\myscripts\py
Things I also tried
system environmental variables
Path: C:\Path2Python27;C:\Path2Python27\scripts;C:\Path2Python37;C:\Path2Python37\scripts;C:\Users\Me\myscripts\py
moving the same stuff to user env's Path
moving the above-shown system PYTHONPATH to the user env
What else am I missing? I don't understand.
All in all, what I needed to get this working:
system environmental variables
Path: C:\Path2Python27;C:\Path2Python27\scripts;C:\Path2Python37;C:\Path2Python37\scripts;
PYTHONPATH: C:\Users\Me\myscripts\py
and
making sure to use Andriy's comment. It won't work using python3 unzipit.py "C:\link\to\folder".
In order to achieve what you want you have to specify -m flag and the module name, so python will retrieve module by looking up python module path. See more here in interface-options. The command shall be:
python3 -m unzipit "C:\Users\Me\downloads\zipfilehome"
I have installed some command line clients using pip that should run straight from the command line without the python keyword and using the path to the file.
For instance shub from scrapinghub or turbolift
All I get is:
shub: command not found
and
turbolift: command not found
What environment variables should I add to .bash_profile to enable the desired command line behaviour?
The directory which contains the script you want to run must be added to your PATH.
For example, to add $HOME/bin/pip/, you would use
PATH=$HOME/bin/pip:$PATH
to add it at the front. (If it's not frequently used, and doesn't need to override system commands, maybe add to the end instead.)
Many guidelines add export PATH but this is normally unnecessary, as the system startup files will already have declared this particular variable to be exported.
This is an extremely basic and very frequently asked question.
Python uses the PYTHONPATH environment-variable to determine in which folders it should look for modules.
You can play around with it by modifying sys.path, which works nicely for pure Python-Modules.
But when a module uses shared object files or static libraries, it looks for those in LD_LIBRARY_PATH (on linux), but this can't be changed as easily and is platform dependent as far as I know.
The quick-fix for this problem is of course to set the environment-variable or invoke the script like LD_LIBRARY_PATH=. ./script.py, but then you'll have to set it again for every new shell you open.
Also, the .so files in my case will always be in the same directory as the .py file, but may very well be moved to another absolute path, so I'd like to set them automatically every time I invoke the script.
How can I edit the path in which the Python interpreter looks for libraries platform-independently on runtime?
EDIT:
I already tried os.environ['LD_LIBRARY_PATH'] = os.getcwd(), but to no avail.
UPDATE: see the EDIT below.
I would use:
import os
os.environ['LD_LIBRARY_PATH'] = os.getcwd() # or whatever path you want
This sets the LD_LIBRARY_PATH environment variable for the duration/lifetime of the execution of the current process only.
EDIT: it looks like this needs to be set before starting Python: Changing LD_LIBRARY_PATH at runtime for ctypes
So I'd suggest going with a wrapper .sh (or .py if you insist) script. Also, as #chepner pointed out, you might want to consider installing your .so files in a standard location (within the virtualenv).
See also Setting LD_LIBRARY_PATH from inside Python
My solution to this problem is to put this as the first line of a Python script (instead of the usual shebang):
exec env LD_LIBRARY_PATH=/some/path/to/lib /path/to/specific/python -x "$0" "$#"
And here is how this works:
with no shebang the current shell treats the file as a shell script,
"exec" ensures that this first line is also the last command from this file executed by the shell,
"env" is used here to set any environment variables, e.g. LD_LIBRARY_PATH,
an exact path to Python's interpreter can specified or "env" can find one in PATH,
"-x" is a Python's option which causes the first line to be ignored by the Python interpreter,
"$0" is the script name, "$#" is substituted by positional parameters.
Python, when gets the values of environment variables as in os.environ['LD_LIBRARY_PATH'] or os.environ['PATH'], it copies the values, into a dictionary, from it's parent process's environment, generally bash (bash process's environment gets carried to the child process, the python running instance).
you can see this environment variable section with env command output from bash.
you can also see/read this env data from /proc/<pid>/environ, by introducing an infinite loop(while 1: pass) after modifying any environment variable.
If you see/read this variable value/data from /proc/<pid>/environ after modifying it inside the python script, you would get to see that the real variable's data doesn't get modified, though the python script shows a modified dictionary key value, updated.
What actually happens when you modify an env variable inside python script, as in os.environ['LD_LIBRARY_PATH']='/<new_location>', is that it just updates the value in local dictionary, which is not mapped to process's env variable section. Hence it won't propagate all the way back to reflect in current process's environment, because ONLY a local dictionary was modified/updated/populated.
Hence if we want to have the new environment variable to be reflected, we should overwrite the memory image of the process with new environment variable data, using execv.
Example:
new_lib = '/<new_location>'
if not new_lib in os.environ['LD_LIBRARY_PATH']:
os.environ['LD_LIBRARY_PATH'] += ':'+new_lib
try:
os.execv(sys.argv[0], sys.argv)
except Exception as e:
sys.exit('EXCEPTION: Failed to Execute under modified environment, '+e)
import xyz
#do something else
Limitation: Ideally, python should not allow such modification of os.environ variables.
But because there is no constant dictionary data type, it allows modification of the data variable. There is absolutely no use of modifying the values, as it does nothing useful to reflect in running process's real environment, unless execv is used.
The solution works fine if the env is reinit
import os
os.environ['LD_LIBRARY_PATH'] = os.getcwd() # or whatever path you want
The code need to be taken in place....
os.execv(sys.argv[0], sys.argv)
Since coreutils 8.30 it is possible to use env -S to split shebang line to separate arguments:
#!/usr/bin/env -S LD_LIBRARY_PATH=/path/to/lib python options
For compatibility with older systems, you can use the fact that the shell allows all commands to be enclosed in quotes, whereas in python it will be just strings:
#!/bin/sh
"export" "LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH"
"exec" "python3" "$0" "$#"
# Further python program
import somemodule
How do I install the pythonstartup file so that it runs on command like python myfile.py?
I tried to install it into my /home/myuser directory for Ubuntu, but it said that I had insufficient permissions. Furthermore, different places say alternately that it should be in all caps or in all lower case with a period before it. The Python command line guide does not seem to explain where to put the file, or how to change which environment variable to 'point' to it.
In your ~/.bashrc:
export PYTHONSTARTUP=$HOME/.pythonstartup
and put your python code in $HOME/.pythonstartup, like:
import rlcompleter
import readline
readline.parse_and_bind("tab: complete")
Then run the interactive shell:
python
See the imports from PYTHONSTARTUP are processed. This only works in python interactive mode.
For more information about PYTHONSTARTUP variable, read the python man page:
$ man python
How to execute the python file defined in $PYTHONSTARTUP when you execute a file like python foobar.py
Run this command to find out where your OS has defined USER_SITE:
$ python -c "import site; site._script()"
Mine says:
USER_SITE: '/home/el/.local/lib64/python2.7/site-packages'
Create a new file there called /home/el/.local/lib64/python2.7/site-packages/usercustomize.py, put this code in there:
try:
import your_things
import readline
print("ROCKETMAN!")
except ImportError:
print("Can't load your things")
print("Either exit here, or perform remedial actions")
exit()
Close the terminal and reopen it to clear out any shenanigans.
Make a new file python foobar.py anywhere on the filesystem, put this code in there:
#Note there is no your_things module imported here.
#Print your_things version:
print(your_things.__version__)
Run it:
python foobar.py
ROCKETMAN!
'1.12.0'
What just happened.
You used the python sitewide specific python configuration hook and imported libraries in the usercustomize.py file which ran before foobar.py.
Documentation: https://docs.python.org/2/library/site.html
Where I found this trick: https://nedbatchelder.com/blog/201001/running_code_at_python_startup.html
On Windows you can put your startup script just about anywhere as long as you put its path into your PYTHONSTARTUP environment variable. On Windows the lettercase of environment variable names doesn't matter.
You can define the values of user and system environment variables as described in a somewhat related answer of mine here.
I'm assuming you mean the PYTHONSTARTUP environment variable? Try putting a file my-python-startup.py with some interesting contents in your home dir, then issue the following on the command line:
export PYTHONSTARTUP=$HOME/my-python-startup.py
python
and observe what happens. Then put the first of the above lines in the (hidden!) file .bashrc in your homedir to have it persist across terminal sessions.
I have a MacBook Pro with Snow Leopard, and the Python 2.6 distribution that comes standard. Numpy does not work properly on it. Loadtxt gives errors of the filename being too long, and getfromtxt does not work at all (no object in module error). So then I tried downloading the py26-numpy port on MacPorts. Of course when I use python, it defaults the mac distribution. How can I switch it to use the latest and greatest from MacPorts. This seems so much simpler than building all the tools I need from source...
Thanks!
First of all, add the MacPorts path (/opt/local/bin) to your $PATH. In .bashrc (or whatever shell config file you use):
export PATH="/opt/local/bin:${PATH}"
If you have multiple versions of Python installed via MacPorts, and/or want to easily switch between the MacPorts and Apple distributions, you can install the python_select port as well.
Also note that the MacPorts version of Python 2.6 is installed into /opt/local/bin/python2.6, so to use that interpreter, you'll have to do one of three things:
Start the interpreter using python2.6 (not just python).
Set up a shell alias so that python calls python2.6 (alias python=python2.6).
Manually set up a symlink from /opt/local/bin/python -> /opt/local/bin/python2.6.
Use python_select to set the Python used by calling python.
Options #3 or #4 are probably the best bet.
You need to update your PATH so that the stuff from MacPorts is in front of the standard system directories, e.g., export PATH=/opt/local/bin:/opt/local/sbin:/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin/:$PATH.
UPDATE: Pay special attention to the fact that /opt/local/Library/Frameworks/Python.framework/Versions/Current/bin is in front of your old PATH value.
The existing answers are quite useful, but I noticed that neither of them spell out how to make the change stick. If you're not familiar with the unix command line this might be important.
First, and explanation: In unix based operating systems, important configuration information in the shell is stored in things called environment variables. The environment variable called PATH directs your shell to a list of places to look for programs. When you type a command, it starts at the leftmost end of the PATH variable, and looks in that folder for the program you tried to run. If it finds it, it runs it; else it looks in the next folder. When you have multiple versions of the same program installed, you can use the PATH variable to give one precedence.
To make use of this, put the folder with the shiny new version in front of the path, like this:
PATH=/opt/local/bin:/usr/bin:/usr/local/bin
To make this change in a single version of your shell, you can type
export PATH=/opt/local/bin:/usr/bin:/usr/local/bin
To make the change in every shell you open, you need to instruct your shell to set this variable every time it starts. There is a file called .bashrc, and another one called .bash_profile that bash will read when it starts up. The .bashrc file is generally used to contain instructions for all shells, and .bash_profile is used to contain instructions only for interactive shells. So, to make this change stick, you can edit /Users/yourname/.bashrc to include a line like this:
export PATH="/opt/local/bin:$PATH"
What that does is add /opt/local/bin to the front of the path variable while leaving the rest of the path alone. If that change doesn't seem to work, you will need to either ensure .bashrc is getting called by adding source $HOME/.bashrc to your .bash_profile script, or just move the necessary line into .bash_profile.