Is it possible to change environment variables of current process?
More specifically in a python script I want to change LD_LIBRARY_PATH so that on import of a module 'x' which depends on some xyz.so, xyz.so is taken from my given path in LD_LIBRARY_PATH
is there any other way to dynamically change path from where library is loaded?
Edit: I think I need to mention that I have already tried thing like
os.environ["LD_LIBRARY_PATH"] = mypath
os.putenv('LD_LIBRARY_PATH', mypath)
but these modify the env. for spawned sub-process, not the current process, and module loading doesn't consider the new LD_LIBRARY_PATH
Edit2, so question is can we change environment or something so the library loader sees it and loads from there?
The reason
os.environ["LD_LIBRARY_PATH"] = ...
doesn't work is simple: this environment variable controls behavior of the dynamic loader (ld-linux.so.2 on Linux, ld.so.1 on Solaris), but the loader only looks at LD_LIBRARY_PATH once at process startup. Changing the value of LD_LIBRARY_PATH in the current process after that point has no effect (just as the answer to this question says).
You do have some options:
A. If you know that you are going to need xyz.so from /some/path, and control the execution of python script from the start, then simply set LD_LIBRARY_PATH to your liking (after checking that it is not already so set), and re-execute yourself. This is what Java does.
B. You can import /some/path/xyz.so via its absolute path before importing x.so. When you then import x.so, the loader will discover that it has already loaded xyz.so, and will use the already loaded module instead of searching for it again.
C. If you build x.so yourself, you can add -Wl,-rpath=/some/path to its link line, and then importing x.so will cause the loader to look for dependent modules in /some/path.
Based on the answer from Employed Russian, this is what works for me
oracle_libs = os.environ['ORACLE_HOME']+"/lib/"
rerun = True
if not 'LD_LIBRARY_PATH' in os.environ:
os.environ['LD_LIBRARY_PATH'] = ":"+oracle_libs
elif not oracle_libs in os.environ.get('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] += ":"+oracle_libs
else:
rerun = False
if rerun:
os.execve(os.path.realpath(__file__), sys.argv, os.environ)
In my experience trying to change the way the loader works for a running Python is very tricky; probably OS/version dependent; may not work. One work-around that might help in some circumstances is to launch a sub-process that changes the environment parameter using a shell script and then launch a new Python using the shell.
The below code is to set the LD_LIBRARY_PATH or any other environment variable paths that is required by the import modules.
if os.getenv('LD_LIBRARY_PATH')==None:
os.environ['LD_LIBRARY_PATH']='<PATH>'
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
elif <path> not in os.getenv('LD_LIBRARY_PATH'):
os.environ['LD_LIBRARY_PATH'] = ':'.join([os.getenv('LD_LIBRARY_PATH'),'<PATH>'])
try:
sys.stdout.flush()
os.execl(sys.executable,sys.executable, *sys.argv)
except OSError as e:
print(e)
# import X
The function os.execl will replace the current process. In UNIX a new executable will be loaded into the current process.
By having this code before the import of the 'X' module, now it will be looking for the files in the new path that was set.
More on execl
well, the environment variables are stored in the dictionary os.environ, so if you want to change , you can do
os.environ["PATH"] = "/usr/bin"
Related
Can someone help me with this please?
I am trying to compile a program in this case programmed in python that I can run in win9Xdos, that I can call/start from a 9xDos batchfile, that will find the Current working Dir & by that I mean identify the cwd (current working directory) from where the python program and batchfile are executed. The python program will be called getcwd.py which I am hoping someone will outline what I need to do to convert to EXE/COM file. there is a program called Py2EXE but not sure if this will compile for Ms-dos file. Anyways heres my simple code thus far. Can someone tell me if I am on the right track please? Oh by the way what I am trying to do is find the CWD & inject the resultant path into a variable that can be read from 9Xdos. The current Var would be %cwd%
# importing os module
import os
# some websites say use: del cwd (to remove variable if it exists)
cwd = none
cwd = os.getcwd()
print(cwd)
The print line may need interchanging with code below, not sure help needed:
print(type(path))
# <class 'str'>
would the above code work, say in the root e.g. C:\ with & work in obtaining the CWD variable & if correct how would I proceed to compile it to an exe/com file? do I need to take into account LFN's & Spaces between possible paths e.g C:\Program Files & possible backslashes etc?
Your code isn't perfect but it is on the right track. All you need is this:
import os
if __name__ == '__main__':
print(os.getcwd())
There is no need for an auxiliary variable, and I don't know what websites are recommending that you delete the variable before creating it. Trying to delete a nonexistent Python variable is a runtime error. So I would stay away from those websites.
But your question is about setting an environment variable. Calling print() won't do that. All it will do is echo the current working directory to the console. There is no way to change the environment of a running process that will affect the parent process. This is not a Python restriction nor a Windows restriction: it is quite general. The OS sets up the environment of the process when it creates the process. You can make changes to the environment (using os.environ[env-var]) but those changes will only be visible inside that Python process and will not be visible to the environment of the batch file that runs the Python program. To do that, you need to pass the value back to the calling process.
One way to do that is like this:
In Python:
import os
if __name__ == '__main__':
print(f"set CWDIR={os.getcwd()}", file=open("mycd.bat","w"))
I haven't had a Python 1.5.2 environment for 15 years, so I can't test this, but I think the equivalent would have been
if __name__ == '__main__':
print >> open("mycd.bat","w"), "set CWDIR=%s" % (os.getcwd(),)
In a cmd.exe console:
call mycd.bat
Though if your Win9XDos doesn't provide %cd% (which, as far as I recall, was available in MS-DOS 5, or maybe further back still) there is no way of telling if it supports call either. (Are you maybe running command.com instead of cmd.exe? That would explain why things that should be there are missing).
I used pyinstaller to create a 64-bit .exe and that resulted in a file of about 6MB. Now, 32-bit executables are smaller, but it might be that the resulting executable is still too big to load.
So I think the Python route may turn out to be more trouble than it is worth.
I've created a command line program that I would like to distribute to some folks at work. Getting them all to install the python interpreter is simply unrealistic. Hence, I've created a single .exe file using PyInstaller. I am coming to realize however, that most people don't even know how to navigate to the directory where the .exe is, in order to invoke it. (And as of now, I haven't yet figured out how to get the program to run when clicked.) Is there a way to make the program add it's self to the users sys PATH when it is run the first time or would this require an installer? Thanks!
The common trap would be to read the PATH env. variable by using os.environ('PATH'). That would be a big mistake because this variable contains user & system paths mixed together. That's a special case for the PATH variable.
What you need to do is to fetch PATH env variable from the registry (user part), update it if needed, and write it back.
You can achieve that using winreg module, modifying the user PATH environment variable (or create if doesn't exist for this particular user)
read user PATH variable
if exists, tokenize the paths (else, path list defaults to empty)
compute the path of the current module (using os.path.dirname(__file__))
check if already in the path, if so, exit (I print the path list in that case so you can test)
create/update PATH user env. variable with the updated path list if necessary
Code:
import winreg,os
script_directory = os.path.dirname(__file__)
paths = []
key_type = winreg.REG_EXPAND_SZ # default if PATH doesn't exist
try:
keyQ = winreg.OpenKey(winreg.HKEY_CURRENT_USER, 'Environment', 0, winreg.KEY_QUERY_VALUE)
path_old, key_type = winreg.QueryValueEx(keyQ, "PATH")
winreg.CloseKey(keyQ)
paths = path_old.split(os.pathsep)
except WindowsError:
pass
if script_directory in paths:
# already set, do nothing
print(paths)
else:
# add the new path
paths.append(script_directory)
# change registry
keyQ = winreg.OpenKey(winreg.HKEY_CURRENT_USER, 'Environment', 0, winreg.KEY_WRITE)
winreg.SetValueEx(keyQ, 'PATH', 0, key_type, os.pathsep.join(paths))
winreg.CloseKey(keyQ)
Note that the user will have to logoff/logon for changes to take effect. Another solution would be to call setx on the PATH variable. System call, ugly, but effective immediately.
# change registry with immediate effect
import subprocess
subprocess.call(["setx","PATH",os.pathsep.join(paths)])
Or, courtesy to eryksun, some python code to propagate the registry changes to new processes. No need to logoff, no need for ugly setx, just call broadcast_change('Environment') using the code below:
import ctypes
user32 = ctypes.WinDLL('user32', use_last_error=True)
HWND_BROADCAST = 0xFFFF
WM_SETTINGCHANGE = 0x001A
SMTO_ABORTIFHUNG = 0x0002
ERROR_TIMEOUT = 0x05B4
def broadcast_change(lparam):
result = user32.SendMessageTimeoutW(HWND_BROADCAST, WM_SETTINGCHANGE,
0, ctypes.c_wchar_p(lparam), SMTO_ABORTIFHUNG, 1000, None)
if not result:
err = ctypes.get_last_error()
if err != ERROR_TIMEOUT:
raise ctypes.WinError(err)
(seems that I have to refactor some code of my own with that last bit :))
env. variable read code took from here: How to return only user Path in environment variables without access to Registry?
I'm writing a build script to resolve dependent shared libraries (and their shared libraries, etc.). These shared libraries do not exist in the normal PATH environment variable.
For the build process to work (for the compiler to find these libraries), the PATH has been changed to include the directories of these libraries.
The build process is thus:
Loader script (changes PATH) -> Python-based build script -> Configure -> Build -> Resolve Dependencies -> Install.
The Python instance inherits a changed PATH variable from its parent shell.
From within Python, I'm trying to get the default PATH (not the one inherited from its parent shell).
The idea:
The idea to resolve the 'default' PATH variable is to somehow 'signal' the OS to start a new process (running a script that prints PATH) but this process is NOT a child of the current Python process (and presumably won't inherit its modified environment variables).
The attempted implementation:
import os
import sys
print os.environ["PATH"]
print "---"
os.spawnl(os.P_WAIT, sys.executable, "python", "-c \"import os;print(os.environ['PATH']);\"")
os.spawn appears to use the same environment variables as the Python process which calls it. I've also tried this approach with subprocess.POpen, with no success.
Can this approach be implemented ? If not, what is an alternative approach (given that the loader script and the overall process can't change)?
I'm currently using Windows but the build script is to be cross-platform.
EDIT:
The cross-platform constraint appears to be too restrictive. Different implementations of the same concept can now be considered.
As an example, using code from this answer, the Windows registry can be used to get the 'default' system PATH variable.
try:
import _winreg as winreg
except ImportError:
try:
import winreg
except ImportError:
winreg = None
def env_keys(user=True):
if user:
root = winreg.HKEY_CURRENT_USER
subkey = "Environment"
else:
root = winreg.HKEY_LOCAL_MACHINE
subkey = r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
return root, subkey
def get_env(name, user=True):
root, subkey = env_keys(user)
key = winreg.OpenKey(root, subkey, 0, winreg.KEY_READ)
try:
value, _ = winreg.QueryValueEx(key, name)
except WindowsError:
return ""
value = winreg.ExpandEnvironmentStrings(value)
return value
print get_env("PATH", False)
A consistent approach for *nix is needed.
Using subprocess.Popen, you can provide an environment for the child process to use:
default_path = os.environ['PATH'] # save the default path before changing it
os.environ['PATH'] = # whatever you want
child_env = os.environ.copy()
child_env['PATH'] = default_path
# change env
subprocess.Popen(..., env=child_env)
The documentation states that the provided environment will be used instead of inheriting it from the parent:
If env is not None, it must be a mapping that defines the environment
variables for the new process; these are used instead of inheriting
the current process’ environment, which is the default behavior.
What do you really mean by 'default' value of PATH? The value it had when you logged in? Some system-wide default? The value the loader script started with before it made changes?
The easiest thing would be to wrap the loader script (if you really cannot change it) with one of your own that saves the current value of PATH in some other environment variable like OLD_PATH. Then you can use something like:
os.spawnle( ... , {'PATH' : os.environ['OLD_PATH']})
Or you could spawn a shell as a login or at least interactive shell and let it source the user's .bashrc (or other startup) before invoking python.
** update ** for windows, and assuming you just want to get the PATH:
Spawn CMD.EXE, have it execute command 'echo %PATH%'
So I have this uber script which constantly checks the system path for a program (openvpn). When you install openvpn it adds itself to the system path. I run my script in the console and, while it runs and checks, I install openvpn. In that console my script will never find openvpn in sys path. If I open a new console and run the same script it finds it.
Any idea how I can make my script a little less dumb?
import os
import time
import subprocess
def cmd( command ):
return subprocess.check_output( command, shell = True )
def program_in_path( program ):
path = cmd( "path" ).split(";")
for p in path:
if "openvpn" in p.lower():
return True
return False
if __name__ == '__main__':
while True:
print program_in_path("openvpn")
time.sleep( 2 )
I presume it's from the shell = True thing but how else would I find it if not with path or WHERE openvpn /Q ? Running with no sehll I get WindowsError: [Error 2] The system cannot find the file specified
Here's slightly the same program done in ruby which works 100%:
loop do
puts system( "WHERE openvpn /Q" )
sleep( 5 )
end
Unfortunately my project is too deep into python to switch languages now. Too bad.
It's actually because when your program starts, it has an environment configured. Part of that environment is the system path. When you start a subshell, it inherits the environment of the parent process.
I'm not a Windows programmer, and I don't have a Windows machine available to test on right now. But according to that bug report, if you import nt in your script and reload(nt) in your while True loop that it will pull down a fresh copy of the environment from the system. I don't know whether that's true or not. It might be worth a try.
For what it's worth, you can see the same behavior from the cmd window by, for instance, opening a command window, adding a program folder to the System Path, and then trying to run an exe from that program folder in your existing cmd window. It won't work -- but open a new cmd window, and it will.
The bug report you cite is about a different problem. That problem outlined there is that from within Python, if you load in one of the system DLLs and use a particular function Windows provides for manipulating your environment, Python does not reflect the change. However, if you make a change to os.environ, Python recognizes that change. The conclusion from the community was that the particular function that the reporter was using, was not the correct function to use to get the results he expected.
Perhaps this approach works for you, getting the PATH variable straight from the registry (since you're on Windows).
For instance you could do something like this:
import winreg
def PathFromReg():
loc = r'SYSTEM\CurrentControlSet\Control\Session Manager\Environment'
reg = winreg.ConnectRegistry(None, winreg.HKEY_LOCAL_MACHINE)
key = winreg.OpenKey(reg, loc)
n_val = winreg.QueryInfoKey(key)[1]
for i in range(n_val):
val = winreg.EnumValue(key, i)
if val[0] == 'Path':
return val[1]
path = PathFromReg()
print('openvpn' in path.lower())
I think you only need to assign the key once and then query the values inside the loop.
Note: In Python 2 the module is called _winreg.
I am using the pydev plugin to debug a remote application.
This (remote) application has a structure of files that differs from the structure where my Eclipse is running. This leads to problems when I set the breakpoints from the Eclipse IDE because the pydev debugger server cannot match the absolute path of the file with the file on the remote application and hence the breakpoint isn´t hit.
I dont want to hardcode the pydevd_file_utils.PATHS_FROM_ECLIPSE_TO_PYTHON constant to enable filepath translations.
Do you know some way to modify this value without changing the file?
Thanks!
There are 2 ways of setting the path translation:
Use an environment variable such as PATHS_FROM_ECLIPSE_TO_PYTHON that maps the paths from the client to the server side.
The value is a json string with a list(list(str, str)) such that:
PATHS_FROM_ECLIPSE_TO_PYTHON=[["c:/local/path", "/path/in/server"]]
Note that you may set the environment variable in any place you'd like (such as the Environment tab in the Python interpreter preferences page, in the OS itself, in the launch config, etc).
Use the pydevd API to set the tracing at runtime from the python process:
from pydevd_file_utils import setup_client_server_paths
MY_PATHS_FROM_ECLIPSE_TO_PYTHON = [
('/home/user/local-project', '/remote/path/to/project'),
]
setup_client_server_paths(MY_PATHS_FROM_ECLIPSE_TO_PYTHON)
# At this point we could connect to the remote debugger client with:
import pydevd
pydevd.settrace("10.0.0.12")
See: https://www.pydev.org/manual_adv_remote_debugger.html for more info on the Remote Debugging.
Note: the mapping set in Window > Preferences select PyDev > Debug > Source Locator doesn't really map to that environment variable nor the actual debugger mapping (that's a separate translation that only translates paths which are found on Eclipse locally and it's not really passed on to the debugger to hit breakpoints remotely).
You can do that by setting a new environment variable like this:
PATHS_FROM_ECLIPSE_TO_PYTHON='[["client_src_fullpath", "remote_src_fullpath"]]'
In linux simply run that before starting the program from the command line, or set is as a global variable.
In windows you will need to set it as a global system variable.
Variable name: PATHS_FROM_ECLIPSE_TO_PYTHON
Variable value: [["client_src_path", "remote_src_path"]]
As an alternative, you can also do this in code, BUT you need to do it BEFORE you import pydevd:
import os
os.environ.setdefault("PATHS_FROM_ECLIPSE_TO_PYTHON",'[["client_src_path","remote_src_path"]]')
import pydevd
pydevd.settrace("10.0.2.2", port=5678,stdoutToServer=True, stderrToServer=True, suspend=False,patch_multiprocessing=True)
(I'm aware this is a very old question, but none of the answers were updated to the current code)
Unfortunately there is no good way to do that.
As a workaround I explicitly replaced function NormFileToServer by adding the following code at the beginning of my source file.
def SrcPathMapping(file):
eclipse_src_path = 'C:\\tmp\\workspace\\test\\Scripts\\'
server_src_path = '/home/tester/test/Scripts/'
return file.replace(eclipse_src_path, server_src_path)
import pysrc.pydevd as pydevd
pydevd.NormFileToServer = SrcPathMapping
This simplistic mapping is sufficient when all source files are located in one directory. For proper implementation of the mapping function check NormFileToServer in the pydevd_file_utils.