Is it possible using Python 3.5 to create and update environment variables in Windows and Linux so that they get persisted?
At the moment I use this:
import os
os.environ["MY_VARIABLE"] = "TRUE"
However it seems as if this does not "store" the environment variable persistently.
I'm speaking for Linux here, not sure about Windows.
Environment variables don't work that way. They are a part of the process (which is what you modify by changing os.environ), and they will propagate to child processes of your process (and their children obviously). They are in-memory only, and there is no way to "set and persist" them directly.
There are however several configuration files which allow you to set the environment on a more granular basis. These are read by various processes, and can be system-wide, specific to a user, specific to a shell, to a particular type of process etc.
Some of them are:
/etc/environment for system-wide variables
/etc/profile for shells (and their children)
Several other shell-specific files in /etc
Various dot-files in a user's home directory such as .profile, .bashrc, .bash_profile, .tcshrc and so on. Read your shell's documentation.
I believe that there are also various ways to configure environment variables launched from GUIs (e.g. from the gnome panel or something like that).
Most of the time you'll want to set environment variables for the current user only. If you only care about shells, append them to ~/.profile in this format:
NAME="VALUE"
The standard way to 'persist' an environment variable is with a configuration file. Write your application to open the configuration file and set every NAME=VARIABLE pair that it finds. Optionally this step could be done in a wrapper startup script.
If you wish to 'save' the state of a variable, you need to open the configuration file and modify its contents. Then when it's read in again, your application will set the environment accordingly.
You could of course store the configuration in some other way. For example in a configuration_settings class that you pickle/shelve. Then on program startup you read in the pickled class and set the environment. The important thing to understand is that when a process exits its environment is not saved. Environments are inherited from the parent process as an intentional byproduct of forking.
Config file could look like:
NAME=VALUE
NAME2=VALUE2
...
Or your config class could look like:
class Configuration():
env_vars = {}
import os
def set(name, val):
env_vars[name] = val
os.environ[name] = val
def unset(name):
del env_vars[name]
del os.environ[name]
def init():
for name in env_vars:
os.environ[name] = env_vars[name]
Somewhere else in our application
import shelve
filename = "config.db"
d = shelve.open(filename)
# get our configuration out of shelve
config = d['configuration']
# initialize environment
config.init()
# setting an environment variable
config.set("MY_VARIABLE", "TRUE")
#unsetting
config.unset("MY_VARIABLE")
# save our configuration
d['configuration'] = config
Code is not tested but I think you get the jist.
Related
I'm trying to find out how to configure the "python interactive shell" on linux (i.e. the program executed when typing python in terminal). I'm looking forward to configure stuff such as coloring ps[12], tab completion (?), etc.
I'm aware of ipython/jupyter, bpython, etc. but I don't want fancy (unnecessary?) stuff but only some colored terminal :)
Is it possible to have a kind of config file?
There is an environment variable you can set, PYTHONSTARTUP:
If this is the name of a readable file, the Python commands in that file are executed before the first prompt is displayed in interactive mode. The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session. You can also change the prompts sys.ps1 and sys.ps2 and the hook sys.__interactivehook__ in this file.
To set the environment variable, go to your terminal and type:
$ export PYTHONSTARTUP=/path/to/my/python/file.py
($ is the prompt, not something you should type.)
Since that won't last beyond the current session, you probably want to put it in your .bashrc file.
In your /path/to/my/file.py you can do whatever Python stuff you want. For example, you might want to have the primary and secondary prompts to be green:
import sys
color = "\x1b[32m{}\x1b[m"
sys.ps1 = color.format(sys.ps1)
sys.ps2 = color.format(sys.ps2)
# We don't want variables to be around in our interactive sessions.
del color
del sys
You can do whatever you want in there. In mine, I set a history file that is written to atexit. That way, Up can go beyond the current session. I also added tab completion.
I'm writing a build script to resolve dependent shared libraries (and their shared libraries, etc.). These shared libraries do not exist in the normal PATH environment variable.
For the build process to work (for the compiler to find these libraries), the PATH has been changed to include the directories of these libraries.
The build process is thus:
Loader script (changes PATH) -> Python-based build script -> Configure -> Build -> Resolve Dependencies -> Install.
The Python instance inherits a changed PATH variable from its parent shell.
From within Python, I'm trying to get the default PATH (not the one inherited from its parent shell).
The idea:
The idea to resolve the 'default' PATH variable is to somehow 'signal' the OS to start a new process (running a script that prints PATH) but this process is NOT a child of the current Python process (and presumably won't inherit its modified environment variables).
The attempted implementation:
import os
import sys
print os.environ["PATH"]
print "---"
os.spawnl(os.P_WAIT, sys.executable, "python", "-c \"import os;print(os.environ['PATH']);\"")
os.spawn appears to use the same environment variables as the Python process which calls it. I've also tried this approach with subprocess.POpen, with no success.
Can this approach be implemented ? If not, what is an alternative approach (given that the loader script and the overall process can't change)?
I'm currently using Windows but the build script is to be cross-platform.
EDIT:
The cross-platform constraint appears to be too restrictive. Different implementations of the same concept can now be considered.
As an example, using code from this answer, the Windows registry can be used to get the 'default' system PATH variable.
try:
import _winreg as winreg
except ImportError:
try:
import winreg
except ImportError:
winreg = None
def env_keys(user=True):
if user:
root = winreg.HKEY_CURRENT_USER
subkey = "Environment"
else:
root = winreg.HKEY_LOCAL_MACHINE
subkey = r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
return root, subkey
def get_env(name, user=True):
root, subkey = env_keys(user)
key = winreg.OpenKey(root, subkey, 0, winreg.KEY_READ)
try:
value, _ = winreg.QueryValueEx(key, name)
except WindowsError:
return ""
value = winreg.ExpandEnvironmentStrings(value)
return value
print get_env("PATH", False)
A consistent approach for *nix is needed.
Using subprocess.Popen, you can provide an environment for the child process to use:
default_path = os.environ['PATH'] # save the default path before changing it
os.environ['PATH'] = # whatever you want
child_env = os.environ.copy()
child_env['PATH'] = default_path
# change env
subprocess.Popen(..., env=child_env)
The documentation states that the provided environment will be used instead of inheriting it from the parent:
If env is not None, it must be a mapping that defines the environment
variables for the new process; these are used instead of inheriting
the current process’ environment, which is the default behavior.
What do you really mean by 'default' value of PATH? The value it had when you logged in? Some system-wide default? The value the loader script started with before it made changes?
The easiest thing would be to wrap the loader script (if you really cannot change it) with one of your own that saves the current value of PATH in some other environment variable like OLD_PATH. Then you can use something like:
os.spawnle( ... , {'PATH' : os.environ['OLD_PATH']})
Or you could spawn a shell as a login or at least interactive shell and let it source the user's .bashrc (or other startup) before invoking python.
** update ** for windows, and assuming you just want to get the PATH:
Spawn CMD.EXE, have it execute command 'echo %PATH%'
i am using SCONS Construction tool.
i am unable to use the environment variable which is initialized in python script.
In My project USER can change some variables to work with the compiler.
For that we have 2 files.
Config.py
Sconstruct
Config.py is having all the variables which are like Include directories, CFLAGS , CPPDEFINES etc. So, Here we can set some variables. Those variables i need to use in Sconstruct file. In config.py i set a variable like below
SCONS_INC = "Include files"
os.environ["SCONS_INC"] = SCONS_INC
I need to use those variables in Sconstruct File. The code is
env["CPPPATH"] = os.environ["SCONS_INC"]
But I am getting an error like Undefined variable SCONS_INC.
How to do this?
SCons by default does not use the invoked environment, this is to make sure that you can reproduce the build no matter which configurations your environment have.
The environment variables are stored within the scons environment under the key ENV so you access the general environment variables like this:
env = Environment()
variable = env['ENV']['SomeVariable']
env['ENV']['SomeVariable'] = SomeValue
I understand your question like you need to use variables set in the python script within SCons. To do this you need to transfer them using the two method you describe in combination.
env = Enviroment()
python_variable = os.environ['SomeVariable']
env['ENV']['SomeVariable'] = python_variable
I would however perhaps recommend other ways of controlling the build, so you do not have to go with the hassle of transferring environment variable. IMHO using arguments are simpler. The arguments are simply a dict that are generated by the invocation of scons, so when you say:
scons -D some_argument=blob
You can get that argument by simply:
some_variable = ARGUMENTS["some_argument"]
Of course I do not know why you need the environment variables, so this might be completely irrelevant for you.
I once had a similar need, where the compiler was looking for a certain Env variable that hadnt been set. I was able to solve this problem as follows:
env = Environment()
env['ENV']['THE_VARIABLE'] = 'SomeValue'
I am using the pydev plugin to debug a remote application.
This (remote) application has a structure of files that differs from the structure where my Eclipse is running. This leads to problems when I set the breakpoints from the Eclipse IDE because the pydev debugger server cannot match the absolute path of the file with the file on the remote application and hence the breakpoint isn´t hit.
I dont want to hardcode the pydevd_file_utils.PATHS_FROM_ECLIPSE_TO_PYTHON constant to enable filepath translations.
Do you know some way to modify this value without changing the file?
Thanks!
There are 2 ways of setting the path translation:
Use an environment variable such as PATHS_FROM_ECLIPSE_TO_PYTHON that maps the paths from the client to the server side.
The value is a json string with a list(list(str, str)) such that:
PATHS_FROM_ECLIPSE_TO_PYTHON=[["c:/local/path", "/path/in/server"]]
Note that you may set the environment variable in any place you'd like (such as the Environment tab in the Python interpreter preferences page, in the OS itself, in the launch config, etc).
Use the pydevd API to set the tracing at runtime from the python process:
from pydevd_file_utils import setup_client_server_paths
MY_PATHS_FROM_ECLIPSE_TO_PYTHON = [
('/home/user/local-project', '/remote/path/to/project'),
]
setup_client_server_paths(MY_PATHS_FROM_ECLIPSE_TO_PYTHON)
# At this point we could connect to the remote debugger client with:
import pydevd
pydevd.settrace("10.0.0.12")
See: https://www.pydev.org/manual_adv_remote_debugger.html for more info on the Remote Debugging.
Note: the mapping set in Window > Preferences select PyDev > Debug > Source Locator doesn't really map to that environment variable nor the actual debugger mapping (that's a separate translation that only translates paths which are found on Eclipse locally and it's not really passed on to the debugger to hit breakpoints remotely).
You can do that by setting a new environment variable like this:
PATHS_FROM_ECLIPSE_TO_PYTHON='[["client_src_fullpath", "remote_src_fullpath"]]'
In linux simply run that before starting the program from the command line, or set is as a global variable.
In windows you will need to set it as a global system variable.
Variable name: PATHS_FROM_ECLIPSE_TO_PYTHON
Variable value: [["client_src_path", "remote_src_path"]]
As an alternative, you can also do this in code, BUT you need to do it BEFORE you import pydevd:
import os
os.environ.setdefault("PATHS_FROM_ECLIPSE_TO_PYTHON",'[["client_src_path","remote_src_path"]]')
import pydevd
pydevd.settrace("10.0.2.2", port=5678,stdoutToServer=True, stderrToServer=True, suspend=False,patch_multiprocessing=True)
(I'm aware this is a very old question, but none of the answers were updated to the current code)
Unfortunately there is no good way to do that.
As a workaround I explicitly replaced function NormFileToServer by adding the following code at the beginning of my source file.
def SrcPathMapping(file):
eclipse_src_path = 'C:\\tmp\\workspace\\test\\Scripts\\'
server_src_path = '/home/tester/test/Scripts/'
return file.replace(eclipse_src_path, server_src_path)
import pysrc.pydevd as pydevd
pydevd.NormFileToServer = SrcPathMapping
This simplistic mapping is sufficient when all source files are located in one directory. For proper implementation of the mapping function check NormFileToServer in the pydevd_file_utils.
Is it possible to change environment variables of current process?
More specifically in a python script I want to change LD_LIBRARY_PATH so that on import of a module 'x' which depends on some xyz.so, xyz.so is taken from my given path in LD_LIBRARY_PATH
is there any other way to dynamically change path from where library is loaded?
Edit: I think I need to mention that I have already tried thing like
os.environ["LD_LIBRARY_PATH"] = mypath
os.putenv('LD_LIBRARY_PATH', mypath)
but these modify the env. for spawned sub-process, not the current process, and module loading doesn't consider the new LD_LIBRARY_PATH
Edit2, so question is can we change environment or something so the library loader sees it and loads from there?
The reason
os.environ["LD_LIBRARY_PATH"] = ...
doesn't work is simple: this environment variable controls behavior of the dynamic loader (ld-linux.so.2 on Linux, ld.so.1 on Solaris), but the loader only looks at LD_LIBRARY_PATH once at process startup. Changing the value of LD_LIBRARY_PATH in the current process after that point has no effect (just as the answer to this question says).
You do have some options:
A. If you know that you are going to need xyz.so from /some/path, and control the execution of python script from the start, then simply set LD_LIBRARY_PATH to your liking (after checking that it is not already so set), and re-execute yourself. This is what Java does.
B. You can import /some/path/xyz.so via its absolute path before importing x.so. When you then import x.so, the loader will discover that it has already loaded xyz.so, and will use the already loaded module instead of searching for it again.
C. If you build x.so yourself, you can add -Wl,-rpath=/some/path to its link line, and then importing x.so will cause the loader to look for dependent modules in /some/path.
Based on the answer from Employed Russian, this is what works for me
oracle_libs = os.environ['ORACLE_HOME']+"/lib/"
rerun = True
if not 'LD_LIBRARY_PATH' in os.environ:
os.environ['LD_LIBRARY_PATH'] = ":"+oracle_libs
elif not oracle_libs in os.environ.get('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] += ":"+oracle_libs
else:
rerun = False
if rerun:
os.execve(os.path.realpath(__file__), sys.argv, os.environ)
In my experience trying to change the way the loader works for a running Python is very tricky; probably OS/version dependent; may not work. One work-around that might help in some circumstances is to launch a sub-process that changes the environment parameter using a shell script and then launch a new Python using the shell.
The below code is to set the LD_LIBRARY_PATH or any other environment variable paths that is required by the import modules.
if os.getenv('LD_LIBRARY_PATH')==None:
os.environ['LD_LIBRARY_PATH']='<PATH>'
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
elif <path> not in os.getenv('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] = ':'.join([os.getenv('LD_LIBRARY_PATH'),'<PATH>'])
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
# import X
The function os.execl will replace the current process. In UNIX a new executable will be loaded into the current process.
By having this code before the import of the 'X' module, now it will be looking for the files in the new path that was set.
More on execl
well, the environment variables are stored in the dictionary os.environ, so if you want to change , you can do
os.environ["PATH"] = "/usr/bin"