I'm writing a build script to resolve dependent shared libraries (and their shared libraries, etc.). These shared libraries do not exist in the normal PATH environment variable.
For the build process to work (for the compiler to find these libraries), the PATH has been changed to include the directories of these libraries.
The build process is thus:
Loader script (changes PATH) -> Python-based build script -> Configure -> Build -> Resolve Dependencies -> Install.
The Python instance inherits a changed PATH variable from its parent shell.
From within Python, I'm trying to get the default PATH (not the one inherited from its parent shell).
The idea:
The idea to resolve the 'default' PATH variable is to somehow 'signal' the OS to start a new process (running a script that prints PATH) but this process is NOT a child of the current Python process (and presumably won't inherit its modified environment variables).
The attempted implementation:
import os
import sys
print os.environ["PATH"]
print "---"
os.spawnl(os.P_WAIT, sys.executable, "python", "-c \"import os;print(os.environ['PATH']);\"")
os.spawn appears to use the same environment variables as the Python process which calls it. I've also tried this approach with subprocess.POpen, with no success.
Can this approach be implemented ? If not, what is an alternative approach (given that the loader script and the overall process can't change)?
I'm currently using Windows but the build script is to be cross-platform.
EDIT:
The cross-platform constraint appears to be too restrictive. Different implementations of the same concept can now be considered.
As an example, using code from this answer, the Windows registry can be used to get the 'default' system PATH variable.
try:
import _winreg as winreg
except ImportError:
try:
import winreg
except ImportError:
winreg = None
def env_keys(user=True):
if user:
root = winreg.HKEY_CURRENT_USER
subkey = "Environment"
else:
root = winreg.HKEY_LOCAL_MACHINE
subkey = r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
return root, subkey
def get_env(name, user=True):
root, subkey = env_keys(user)
key = winreg.OpenKey(root, subkey, 0, winreg.KEY_READ)
try:
value, _ = winreg.QueryValueEx(key, name)
except WindowsError:
return ""
value = winreg.ExpandEnvironmentStrings(value)
return value
print get_env("PATH", False)
A consistent approach for *nix is needed.
Using subprocess.Popen, you can provide an environment for the child process to use:
default_path = os.environ['PATH'] # save the default path before changing it
os.environ['PATH'] = # whatever you want
child_env = os.environ.copy()
child_env['PATH'] = default_path
# change env
subprocess.Popen(..., env=child_env)
The documentation states that the provided environment will be used instead of inheriting it from the parent:
If env is not None, it must be a mapping that defines the environment
variables for the new process; these are used instead of inheriting
the current process’ environment, which is the default behavior.
What do you really mean by 'default' value of PATH? The value it had when you logged in? Some system-wide default? The value the loader script started with before it made changes?
The easiest thing would be to wrap the loader script (if you really cannot change it) with one of your own that saves the current value of PATH in some other environment variable like OLD_PATH. Then you can use something like:
os.spawnle( ... , {'PATH' : os.environ['OLD_PATH']})
Or you could spawn a shell as a login or at least interactive shell and let it source the user's .bashrc (or other startup) before invoking python.
** update ** for windows, and assuming you just want to get the PATH:
Spawn CMD.EXE, have it execute command 'echo %PATH%'
Related
I've created a command line program that I would like to distribute to some folks at work. Getting them all to install the python interpreter is simply unrealistic. Hence, I've created a single .exe file using PyInstaller. I am coming to realize however, that most people don't even know how to navigate to the directory where the .exe is, in order to invoke it. (And as of now, I haven't yet figured out how to get the program to run when clicked.) Is there a way to make the program add it's self to the users sys PATH when it is run the first time or would this require an installer? Thanks!
The common trap would be to read the PATH env. variable by using os.environ('PATH'). That would be a big mistake because this variable contains user & system paths mixed together. That's a special case for the PATH variable.
What you need to do is to fetch PATH env variable from the registry (user part), update it if needed, and write it back.
You can achieve that using winreg module, modifying the user PATH environment variable (or create if doesn't exist for this particular user)
read user PATH variable
if exists, tokenize the paths (else, path list defaults to empty)
compute the path of the current module (using os.path.dirname(__file__))
check if already in the path, if so, exit (I print the path list in that case so you can test)
create/update PATH user env. variable with the updated path list if necessary
Code:
import winreg,os
script_directory = os.path.dirname(__file__)
paths = []
key_type = winreg.REG_EXPAND_SZ # default if PATH doesn't exist
try:
keyQ = winreg.OpenKey(winreg.HKEY_CURRENT_USER, 'Environment', 0, winreg.KEY_QUERY_VALUE)
path_old, key_type = winreg.QueryValueEx(keyQ, "PATH")
winreg.CloseKey(keyQ)
paths = path_old.split(os.pathsep)
except WindowsError:
pass
if script_directory in paths:
# already set, do nothing
print(paths)
else:
# add the new path
paths.append(script_directory)
# change registry
keyQ = winreg.OpenKey(winreg.HKEY_CURRENT_USER, 'Environment', 0, winreg.KEY_WRITE)
winreg.SetValueEx(keyQ, 'PATH', 0, key_type, os.pathsep.join(paths))
winreg.CloseKey(keyQ)
Note that the user will have to logoff/logon for changes to take effect. Another solution would be to call setx on the PATH variable. System call, ugly, but effective immediately.
# change registry with immediate effect
import subprocess
subprocess.call(["setx","PATH",os.pathsep.join(paths)])
Or, courtesy to eryksun, some python code to propagate the registry changes to new processes. No need to logoff, no need for ugly setx, just call broadcast_change('Environment') using the code below:
import ctypes
user32 = ctypes.WinDLL('user32', use_last_error=True)
HWND_BROADCAST = 0xFFFF
WM_SETTINGCHANGE = 0x001A
SMTO_ABORTIFHUNG = 0x0002
ERROR_TIMEOUT = 0x05B4
def broadcast_change(lparam):
result = user32.SendMessageTimeoutW(HWND_BROADCAST, WM_SETTINGCHANGE,
0, ctypes.c_wchar_p(lparam), SMTO_ABORTIFHUNG, 1000, None)
if not result:
err = ctypes.get_last_error()
if err != ERROR_TIMEOUT:
raise ctypes.WinError(err)
(seems that I have to refactor some code of my own with that last bit :))
env. variable read code took from here: How to return only user Path in environment variables without access to Registry?
Is it possible using Python 3.5 to create and update environment variables in Windows and Linux so that they get persisted?
At the moment I use this:
import os
os.environ["MY_VARIABLE"] = "TRUE"
However it seems as if this does not "store" the environment variable persistently.
I'm speaking for Linux here, not sure about Windows.
Environment variables don't work that way. They are a part of the process (which is what you modify by changing os.environ), and they will propagate to child processes of your process (and their children obviously). They are in-memory only, and there is no way to "set and persist" them directly.
There are however several configuration files which allow you to set the environment on a more granular basis. These are read by various processes, and can be system-wide, specific to a user, specific to a shell, to a particular type of process etc.
Some of them are:
/etc/environment for system-wide variables
/etc/profile for shells (and their children)
Several other shell-specific files in /etc
Various dot-files in a user's home directory such as .profile, .bashrc, .bash_profile, .tcshrc and so on. Read your shell's documentation.
I believe that there are also various ways to configure environment variables launched from GUIs (e.g. from the gnome panel or something like that).
Most of the time you'll want to set environment variables for the current user only. If you only care about shells, append them to ~/.profile in this format:
NAME="VALUE"
The standard way to 'persist' an environment variable is with a configuration file. Write your application to open the configuration file and set every NAME=VARIABLE pair that it finds. Optionally this step could be done in a wrapper startup script.
If you wish to 'save' the state of a variable, you need to open the configuration file and modify its contents. Then when it's read in again, your application will set the environment accordingly.
You could of course store the configuration in some other way. For example in a configuration_settings class that you pickle/shelve. Then on program startup you read in the pickled class and set the environment. The important thing to understand is that when a process exits its environment is not saved. Environments are inherited from the parent process as an intentional byproduct of forking.
Config file could look like:
NAME=VALUE
NAME2=VALUE2
...
Or your config class could look like:
class Configuration():
env_vars = {}
import os
def set(name, val):
env_vars[name] = val
os.environ[name] = val
def unset(name):
del env_vars[name]
del os.environ[name]
def init():
for name in env_vars:
os.environ[name] = env_vars[name]
Somewhere else in our application
import shelve
filename = "config.db"
d = shelve.open(filename)
# get our configuration out of shelve
config = d['configuration']
# initialize environment
config.init()
# setting an environment variable
config.set("MY_VARIABLE", "TRUE")
#unsetting
config.unset("MY_VARIABLE")
# save our configuration
d['configuration'] = config
Code is not tested but I think you get the jist.
What would be a pythonic way of resetting os.environ to the default values one would find in a command shell? I could handle this by first pushing os.environ into a default dictionary, but that method would fail in the event os.environ is changed by another module before mine is imported.
In Windows i'm currently able to reset values like so:
import os, subprocess, tempfile
def is_locked(filepath):
''' Needed to determine when the set command below completes
'''
locked = None
file_object = None
if os.path.exists(filepath):
try:
buffer_size = 8
file_object = open(filepath, 'a', buffer_size)
if file_object:
locked = False
except IOError, message:
locked = True
finally:
if file_object:
file_object.close()
else:
locked = True
return locked
# Define a Windows command which would dump default env variables into a tempfile
#
# - start /i will give default cmd.exe environment variable to what follows.
# It's the only way i found to get cmd.exe's default env variables and because
# start terminates immediately i need to use is_locked defined above to wait for
# Window's set command to finish dumping it's variables into the temp file
# - /b is will prevent a command shell to pop up
temp = tempfile.mktemp()
cmd = 'start /i /b cmd /c set>%s'%temp
p = subprocess.Popen(cmd, shell=True)
p.wait()
# Wait for the command to complete and parse the tempfile
data = []
while(is_locked(temp)):
pass
with open(temp,'r') as file:
data = file.readlines()
os.remove(temp)
# defaults will contain default env variables
defaults = dict()
for env in data:
env = env.strip().split('=')
defaults[env[0]]=env[1]
print '%s %s'%(env[0],env[1])
os.environ = dict(defaults)
My way works currently in python 2.7.3 64 bit, but i just noticed that when i run this in 32 bit, both the PROGRAMFILES and PROGRAMFILES(x86) env variables point to "Program Files (x86)" which is an issue discussed here.
Thanks in advance for all you help!
I wonder that your hack seems to work, because it is by design that the child inherits the current environment of the parent. So I think there's no alternative to saving the environment at app startup. This is not necessarily the env of a shell started from the start menu, though, as your parent could have manipulated it, too.
Do the following experiment: start an explorer, run a shell from it and investigate the env. Then start a shell from the start menu direcly and check again. Now go to control panel -> system and add a variable. Restart the two shells (but not the explorer) and note the difference.
To get the "root" environment (the one that is created at login) you could read the keys at HKCU/Environment. But that's probably hacky, too.
You may need something like the modified_environ context manager describe in this question to set/restore the environment variables.
I suggest you to wrap your main function with this context manager:
with modified_environ():
main()
That way, everything will be restored at function completion.
You can also wrap the imports:
with modified_environ():
import pkg.which.can.change.env
main()
So I have this uber script which constantly checks the system path for a program (openvpn). When you install openvpn it adds itself to the system path. I run my script in the console and, while it runs and checks, I install openvpn. In that console my script will never find openvpn in sys path. If I open a new console and run the same script it finds it.
Any idea how I can make my script a little less dumb?
import os
import time
import subprocess
def cmd( command ):
return subprocess.check_output( command, shell = True )
def program_in_path( program ):
path = cmd( "path" ).split(";")
for p in path:
if "openvpn" in p.lower():
return True
return False
if __name__ == '__main__':
while True:
print program_in_path("openvpn")
time.sleep( 2 )
I presume it's from the shell = True thing but how else would I find it if not with path or WHERE openvpn /Q ? Running with no sehll I get WindowsError: [Error 2] The system cannot find the file specified
Here's slightly the same program done in ruby which works 100%:
loop do
puts system( "WHERE openvpn /Q" )
sleep( 5 )
end
Unfortunately my project is too deep into python to switch languages now. Too bad.
It's actually because when your program starts, it has an environment configured. Part of that environment is the system path. When you start a subshell, it inherits the environment of the parent process.
I'm not a Windows programmer, and I don't have a Windows machine available to test on right now. But according to that bug report, if you import nt in your script and reload(nt) in your while True loop that it will pull down a fresh copy of the environment from the system. I don't know whether that's true or not. It might be worth a try.
For what it's worth, you can see the same behavior from the cmd window by, for instance, opening a command window, adding a program folder to the System Path, and then trying to run an exe from that program folder in your existing cmd window. It won't work -- but open a new cmd window, and it will.
The bug report you cite is about a different problem. That problem outlined there is that from within Python, if you load in one of the system DLLs and use a particular function Windows provides for manipulating your environment, Python does not reflect the change. However, if you make a change to os.environ, Python recognizes that change. The conclusion from the community was that the particular function that the reporter was using, was not the correct function to use to get the results he expected.
Perhaps this approach works for you, getting the PATH variable straight from the registry (since you're on Windows).
For instance you could do something like this:
import winreg
def PathFromReg():
loc = r'SYSTEM\CurrentControlSet\Control\Session Manager\Environment'
reg = winreg.ConnectRegistry(None, winreg.HKEY_LOCAL_MACHINE)
key = winreg.OpenKey(reg, loc)
n_val = winreg.QueryInfoKey(key)[1]
for i in range(n_val):
val = winreg.EnumValue(key, i)
if val[0] == 'Path':
return val[1]
path = PathFromReg()
print('openvpn' in path.lower())
I think you only need to assign the key once and then query the values inside the loop.
Note: In Python 2 the module is called _winreg.
Is it possible to change environment variables of current process?
More specifically in a python script I want to change LD_LIBRARY_PATH so that on import of a module 'x' which depends on some xyz.so, xyz.so is taken from my given path in LD_LIBRARY_PATH
is there any other way to dynamically change path from where library is loaded?
Edit: I think I need to mention that I have already tried thing like
os.environ["LD_LIBRARY_PATH"] = mypath
os.putenv('LD_LIBRARY_PATH', mypath)
but these modify the env. for spawned sub-process, not the current process, and module loading doesn't consider the new LD_LIBRARY_PATH
Edit2, so question is can we change environment or something so the library loader sees it and loads from there?
The reason
os.environ["LD_LIBRARY_PATH"] = ...
doesn't work is simple: this environment variable controls behavior of the dynamic loader (ld-linux.so.2 on Linux, ld.so.1 on Solaris), but the loader only looks at LD_LIBRARY_PATH once at process startup. Changing the value of LD_LIBRARY_PATH in the current process after that point has no effect (just as the answer to this question says).
You do have some options:
A. If you know that you are going to need xyz.so from /some/path, and control the execution of python script from the start, then simply set LD_LIBRARY_PATH to your liking (after checking that it is not already so set), and re-execute yourself. This is what Java does.
B. You can import /some/path/xyz.so via its absolute path before importing x.so. When you then import x.so, the loader will discover that it has already loaded xyz.so, and will use the already loaded module instead of searching for it again.
C. If you build x.so yourself, you can add -Wl,-rpath=/some/path to its link line, and then importing x.so will cause the loader to look for dependent modules in /some/path.
Based on the answer from Employed Russian, this is what works for me
oracle_libs = os.environ['ORACLE_HOME']+"/lib/"
rerun = True
if not 'LD_LIBRARY_PATH' in os.environ:
os.environ['LD_LIBRARY_PATH'] = ":"+oracle_libs
elif not oracle_libs in os.environ.get('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] += ":"+oracle_libs
else:
rerun = False
if rerun:
os.execve(os.path.realpath(__file__), sys.argv, os.environ)
In my experience trying to change the way the loader works for a running Python is very tricky; probably OS/version dependent; may not work. One work-around that might help in some circumstances is to launch a sub-process that changes the environment parameter using a shell script and then launch a new Python using the shell.
The below code is to set the LD_LIBRARY_PATH or any other environment variable paths that is required by the import modules.
if os.getenv('LD_LIBRARY_PATH')==None:
os.environ['LD_LIBRARY_PATH']='<PATH>'
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
elif <path> not in os.getenv('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] = ':'.join([os.getenv('LD_LIBRARY_PATH'),'<PATH>'])
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
# import X
The function os.execl will replace the current process. In UNIX a new executable will be loaded into the current process.
By having this code before the import of the 'X' module, now it will be looking for the files in the new path that was set.
More on execl
well, the environment variables are stored in the dictionary os.environ, so if you want to change , you can do
os.environ["PATH"] = "/usr/bin"