I am sorry if my question not very clear.
I am trying to create a variable pass it to the environment in linux. Then, I want to be able to get this variable some where else. What i have tried so far on the linux command line:
local_pc:~/home$ export variable=10
local_pc:~/home$ python -c 'import os; print os.getenv("variable")'
10
which all sound fine. But when I set export in python I will not be able to get it
subprocess.call(["export","variable=20"],shell = True)
print(os.getenv("variable"))
None
So my question here is how to do xport variable=10 in python
You can change environment variable only for current process or its children. To change environment in its parent process would require hacks e.g., using gdb.
In your example export variable=10 is run in the same process and python -c .. command is a child process (of the shell). Therefore it works.
In your Python example, you are trying (incorrectly) to export variable in a child process and get it in a parent process.
To summarize:
working example: parent sets environment variable for a child
non-working example: child tries to set environment variable for a parent
To reproduce your bash example:
import os
import sys
from subprocess import check_call
#NOTE: it works but you shouldn't do it, there are most probably better ways
os.environ['variable'] = '10' # set it for current processes and its children
check_call([sys.executable or 'python', '-c',
'import os; print(os.getenv("variable"))'])
Subprocess either inherits parent's environment or you could set it explicitly using env argument.
For example, to change a local timezone that is used by time module, you could change TZ environment variable for the current python process on posix systems:
import os
import time
os.environ['TZ'] = ':America/New_York'
time.tzset()
is_dst = time.daylight and time.localtime().tm_isdst > 0
# local time = utc time + utc offset
utc_offset = -time.timezone if not is_dst else -time.altzone
print("%s has utc offset: %.1f hours" % (
os.environ.get('TZ').lstrip(':'), utc_offset/3600.))
update:
There's no feasible way to establish commuication over processes through environment variables.
The child may inherit the environment variables from the parent, but a change to the parent environment variables after the childs invocation won't be passed through to the child and the child's environment changes are completely intransparent to the parent. So no way!
I tested it by trying to establish a round-robin, token-based message passing approach, but i don't see a way to get the changes passed between the process environment variables.
Related
Consider I have two python scripts caller.py and being_called.py
Caller.py:
p = Popen(path_to_being_called, shell=True, stdout=fh, stderr=fh1)
path_to_being_called contains executable path of script 'being_called.py'
Now being_called.py does following:
os.environ['hello'] = 'Yolo'
I want to access this value of environment variable hello inside the script caller.py
Is there a way to do this with popen calls?
Note that I know I can update environment variable for 'being_called' from caller itself as a paramter env and mapping of os.environ and kwargs, but thats not what I want to do. Env variable will be updated from being_called.py only and I need to access its value in caller.py
I tried to see what dir(p) offers but not getting any help from any of that.
Is it possible using Python 3.5 to create and update environment variables in Windows and Linux so that they get persisted?
At the moment I use this:
import os
os.environ["MY_VARIABLE"] = "TRUE"
However it seems as if this does not "store" the environment variable persistently.
I'm speaking for Linux here, not sure about Windows.
Environment variables don't work that way. They are a part of the process (which is what you modify by changing os.environ), and they will propagate to child processes of your process (and their children obviously). They are in-memory only, and there is no way to "set and persist" them directly.
There are however several configuration files which allow you to set the environment on a more granular basis. These are read by various processes, and can be system-wide, specific to a user, specific to a shell, to a particular type of process etc.
Some of them are:
/etc/environment for system-wide variables
/etc/profile for shells (and their children)
Several other shell-specific files in /etc
Various dot-files in a user's home directory such as .profile, .bashrc, .bash_profile, .tcshrc and so on. Read your shell's documentation.
I believe that there are also various ways to configure environment variables launched from GUIs (e.g. from the gnome panel or something like that).
Most of the time you'll want to set environment variables for the current user only. If you only care about shells, append them to ~/.profile in this format:
NAME="VALUE"
The standard way to 'persist' an environment variable is with a configuration file. Write your application to open the configuration file and set every NAME=VARIABLE pair that it finds. Optionally this step could be done in a wrapper startup script.
If you wish to 'save' the state of a variable, you need to open the configuration file and modify its contents. Then when it's read in again, your application will set the environment accordingly.
You could of course store the configuration in some other way. For example in a configuration_settings class that you pickle/shelve. Then on program startup you read in the pickled class and set the environment. The important thing to understand is that when a process exits its environment is not saved. Environments are inherited from the parent process as an intentional byproduct of forking.
Config file could look like:
NAME=VALUE
NAME2=VALUE2
...
Or your config class could look like:
class Configuration():
env_vars = {}
import os
def set(name, val):
env_vars[name] = val
os.environ[name] = val
def unset(name):
del env_vars[name]
del os.environ[name]
def init():
for name in env_vars:
os.environ[name] = env_vars[name]
Somewhere else in our application
import shelve
filename = "config.db"
d = shelve.open(filename)
# get our configuration out of shelve
config = d['configuration']
# initialize environment
config.init()
# setting an environment variable
config.set("MY_VARIABLE", "TRUE")
#unsetting
config.unset("MY_VARIABLE")
# save our configuration
d['configuration'] = config
Code is not tested but I think you get the jist.
I'm writing a build script to resolve dependent shared libraries (and their shared libraries, etc.). These shared libraries do not exist in the normal PATH environment variable.
For the build process to work (for the compiler to find these libraries), the PATH has been changed to include the directories of these libraries.
The build process is thus:
Loader script (changes PATH) -> Python-based build script -> Configure -> Build -> Resolve Dependencies -> Install.
The Python instance inherits a changed PATH variable from its parent shell.
From within Python, I'm trying to get the default PATH (not the one inherited from its parent shell).
The idea:
The idea to resolve the 'default' PATH variable is to somehow 'signal' the OS to start a new process (running a script that prints PATH) but this process is NOT a child of the current Python process (and presumably won't inherit its modified environment variables).
The attempted implementation:
import os
import sys
print os.environ["PATH"]
print "---"
os.spawnl(os.P_WAIT, sys.executable, "python", "-c \"import os;print(os.environ['PATH']);\"")
os.spawn appears to use the same environment variables as the Python process which calls it. I've also tried this approach with subprocess.POpen, with no success.
Can this approach be implemented ? If not, what is an alternative approach (given that the loader script and the overall process can't change)?
I'm currently using Windows but the build script is to be cross-platform.
EDIT:
The cross-platform constraint appears to be too restrictive. Different implementations of the same concept can now be considered.
As an example, using code from this answer, the Windows registry can be used to get the 'default' system PATH variable.
try:
import _winreg as winreg
except ImportError:
try:
import winreg
except ImportError:
winreg = None
def env_keys(user=True):
if user:
root = winreg.HKEY_CURRENT_USER
subkey = "Environment"
else:
root = winreg.HKEY_LOCAL_MACHINE
subkey = r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
return root, subkey
def get_env(name, user=True):
root, subkey = env_keys(user)
key = winreg.OpenKey(root, subkey, 0, winreg.KEY_READ)
try:
value, _ = winreg.QueryValueEx(key, name)
except WindowsError:
return ""
value = winreg.ExpandEnvironmentStrings(value)
return value
print get_env("PATH", False)
A consistent approach for *nix is needed.
Using subprocess.Popen, you can provide an environment for the child process to use:
default_path = os.environ['PATH'] # save the default path before changing it
os.environ['PATH'] = # whatever you want
child_env = os.environ.copy()
child_env['PATH'] = default_path
# change env
subprocess.Popen(..., env=child_env)
The documentation states that the provided environment will be used instead of inheriting it from the parent:
If env is not None, it must be a mapping that defines the environment
variables for the new process; these are used instead of inheriting
the current process’ environment, which is the default behavior.
What do you really mean by 'default' value of PATH? The value it had when you logged in? Some system-wide default? The value the loader script started with before it made changes?
The easiest thing would be to wrap the loader script (if you really cannot change it) with one of your own that saves the current value of PATH in some other environment variable like OLD_PATH. Then you can use something like:
os.spawnle( ... , {'PATH' : os.environ['OLD_PATH']})
Or you could spawn a shell as a login or at least interactive shell and let it source the user's .bashrc (or other startup) before invoking python.
** update ** for windows, and assuming you just want to get the PATH:
Spawn CMD.EXE, have it execute command 'echo %PATH%'
What would be a pythonic way of resetting os.environ to the default values one would find in a command shell? I could handle this by first pushing os.environ into a default dictionary, but that method would fail in the event os.environ is changed by another module before mine is imported.
In Windows i'm currently able to reset values like so:
import os, subprocess, tempfile
def is_locked(filepath):
''' Needed to determine when the set command below completes
'''
locked = None
file_object = None
if os.path.exists(filepath):
try:
buffer_size = 8
file_object = open(filepath, 'a', buffer_size)
if file_object:
locked = False
except IOError, message:
locked = True
finally:
if file_object:
file_object.close()
else:
locked = True
return locked
# Define a Windows command which would dump default env variables into a tempfile
#
# - start /i will give default cmd.exe environment variable to what follows.
# It's the only way i found to get cmd.exe's default env variables and because
# start terminates immediately i need to use is_locked defined above to wait for
# Window's set command to finish dumping it's variables into the temp file
# - /b is will prevent a command shell to pop up
temp = tempfile.mktemp()
cmd = 'start /i /b cmd /c set>%s'%temp
p = subprocess.Popen(cmd, shell=True)
p.wait()
# Wait for the command to complete and parse the tempfile
data = []
while(is_locked(temp)):
pass
with open(temp,'r') as file:
data = file.readlines()
os.remove(temp)
# defaults will contain default env variables
defaults = dict()
for env in data:
env = env.strip().split('=')
defaults[env[0]]=env[1]
print '%s %s'%(env[0],env[1])
os.environ = dict(defaults)
My way works currently in python 2.7.3 64 bit, but i just noticed that when i run this in 32 bit, both the PROGRAMFILES and PROGRAMFILES(x86) env variables point to "Program Files (x86)" which is an issue discussed here.
Thanks in advance for all you help!
I wonder that your hack seems to work, because it is by design that the child inherits the current environment of the parent. So I think there's no alternative to saving the environment at app startup. This is not necessarily the env of a shell started from the start menu, though, as your parent could have manipulated it, too.
Do the following experiment: start an explorer, run a shell from it and investigate the env. Then start a shell from the start menu direcly and check again. Now go to control panel -> system and add a variable. Restart the two shells (but not the explorer) and note the difference.
To get the "root" environment (the one that is created at login) you could read the keys at HKCU/Environment. But that's probably hacky, too.
You may need something like the modified_environ context manager describe in this question to set/restore the environment variables.
I suggest you to wrap your main function with this context manager:
with modified_environ():
main()
That way, everything will be restored at function completion.
You can also wrap the imports:
with modified_environ():
import pkg.which.can.change.env
main()
Is it possible to change environment variables of current process?
More specifically in a python script I want to change LD_LIBRARY_PATH so that on import of a module 'x' which depends on some xyz.so, xyz.so is taken from my given path in LD_LIBRARY_PATH
is there any other way to dynamically change path from where library is loaded?
Edit: I think I need to mention that I have already tried thing like
os.environ["LD_LIBRARY_PATH"] = mypath
os.putenv('LD_LIBRARY_PATH', mypath)
but these modify the env. for spawned sub-process, not the current process, and module loading doesn't consider the new LD_LIBRARY_PATH
Edit2, so question is can we change environment or something so the library loader sees it and loads from there?
The reason
os.environ["LD_LIBRARY_PATH"] = ...
doesn't work is simple: this environment variable controls behavior of the dynamic loader (ld-linux.so.2 on Linux, ld.so.1 on Solaris), but the loader only looks at LD_LIBRARY_PATH once at process startup. Changing the value of LD_LIBRARY_PATH in the current process after that point has no effect (just as the answer to this question says).
You do have some options:
A. If you know that you are going to need xyz.so from /some/path, and control the execution of python script from the start, then simply set LD_LIBRARY_PATH to your liking (after checking that it is not already so set), and re-execute yourself. This is what Java does.
B. You can import /some/path/xyz.so via its absolute path before importing x.so. When you then import x.so, the loader will discover that it has already loaded xyz.so, and will use the already loaded module instead of searching for it again.
C. If you build x.so yourself, you can add -Wl,-rpath=/some/path to its link line, and then importing x.so will cause the loader to look for dependent modules in /some/path.
Based on the answer from Employed Russian, this is what works for me
oracle_libs = os.environ['ORACLE_HOME']+"/lib/"
rerun = True
if not 'LD_LIBRARY_PATH' in os.environ:
os.environ['LD_LIBRARY_PATH'] = ":"+oracle_libs
elif not oracle_libs in os.environ.get('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] += ":"+oracle_libs
else:
rerun = False
if rerun:
os.execve(os.path.realpath(__file__), sys.argv, os.environ)
In my experience trying to change the way the loader works for a running Python is very tricky; probably OS/version dependent; may not work. One work-around that might help in some circumstances is to launch a sub-process that changes the environment parameter using a shell script and then launch a new Python using the shell.
The below code is to set the LD_LIBRARY_PATH or any other environment variable paths that is required by the import modules.
if os.getenv('LD_LIBRARY_PATH')==None:
os.environ['LD_LIBRARY_PATH']='<PATH>'
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
elif <path> not in os.getenv('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] = ':'.join([os.getenv('LD_LIBRARY_PATH'),'<PATH>'])
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
# import X
The function os.execl will replace the current process. In UNIX a new executable will be loaded into the current process.
By having this code before the import of the 'X' module, now it will be looking for the files in the new path that was set.
More on execl
well, the environment variables are stored in the dictionary os.environ, so if you want to change , you can do
os.environ["PATH"] = "/usr/bin"