I work in tc shell and have environment set script with commands like setenv foo bar. I use to run this script by command source set_env.sh and then, when environment is set, run compilation command make my_ target...
I would like to automate the whole process by executing it from Python by calling subprocess.Popen or even os.system. The troubles are:
Shell opened form Python do not recognize command source. My assumption that /bin/bash shell is opened. How do I force Python to open /bin/tcsh?
How do I preserve changed environment setting for second command (make my_ target...)?
If you use the subprocess package in Python then it won't run the shell by default, it will just launch a process (without using the shell). You can add shell=True, but this won't do you much good since environment variables are set for the current process only, and are not sent back to the parent process.
Python starts a tcsh child process, and child processes can't affect parent processes.
The solution would be to make your script print out the environment variables, and then parse the output of that in Python.
So if your script prints something like:
XX=one
YY=two
Then in Python do something along the likes of:
for line in output.split('\n'):
if line == '':
continue
k, v = line.split('=')
os.environ[k] = v
Related
In Linux When I invoke python from the shell it replicates its environment, and starts the python process. Therefore if I do something like the following:
import os
os.environ["FOO"] = "A_Value"
When the python process returns, FOO, assuming it was undefined originally, will still be undefined. Is there a way for the python process (or any child process) to modify the environment of its parent process?
I know you typically solve this problem using something like
source script_name.sh
But this conflicts with other requirements I have.
No process can change its parent process (or any other existing process' environment).
You can, however, create a new environment by creating a new interactive shell with the modified environment.
You have to spawn a new copy of the shell that uses the upgraded environment and has access to the existing stdin, stdout and stderr, and does its reinitialization dance.
You need to do something like use subprocess.Popen to run /bin/bash -i.
So the original shell runs Python, which runs a new shell. Yes, you have a lot of processes running. No it's not too bad because the original shell and Python aren't really doing anything except waiting for the subshell to finish so they can exit cleanly, also.
It's not possible, for any child process, to change the environment of the parent process. The best you can do is to output shell statements to stdout that you then source, or write it to a file that you source in the parent.
I would use the bash eval statement, and have the python script output the shell code
child.py:
#!/usr/bin/env python
print 'FOO="A_Value"'
parent.sh
#!/bin/bash
eval `./child.py`
I needed something similar, I ended up creating a script envtest.py with:
import sys, os
sys.stdout = open(os.devnull, 'w')
# Python code with any number of prints (to stdout).
print("This is some other logic, which shouldn't pollute stdout.")
sys.stdout = sys.__stdout__
print("SomeValue")
Then in bash:
export MYVAR=$(python3 envtest.py)
echo "MYVAR is $MYVAR"
Which echos the expected: MYVAR is SomeValue
I am using python to run a bash script which contains a bash source command. For some reason the source command is not working.
Bash script code:
#!/bin/bash
SETTINGSFILE=/apps/settings
if test -f "$SETTINGSFILE"; then
source $SETTINGSFILE #This is not working
echo "settings file exists" #reaches here
else
echo "settings file not found"
fi
...
Python code:
import subprocess
rc = subprocess.call(["./<name-of-bash-script>.sh"])
Basically, I want to run the command source /apps/settings using a script. The control reaches to the echo statement (marked with comment), but still source command is not executed by the bash script. How to get it working?
I don't know why running "source file.sh" in a terminal would not load your environment variables (I am not on a linux system right now and can't really test this). However, in general, if you want to manipulate your environment, you should do that directly in python and not call a child process. For example,
import os
os.environ['THING'] = 'WORD'
I believe if you wish to change your environment variables from a child process then you need to imbed that child process in your current parent process (so that you become the child with the new environment) and you need to keep that child process open (you don't want it to terminate). You probably need something like "/bin/bash -i" at the end of your source file (to keep the subprocess running). You then need to use something like the pexpect module to imbed subprocess in current process.
import pexpect
child = pexpect.spawn('file.sh')
child.interact()
#check your os.environ to see if it is updated from your source file.
Sorry I can't be of more help.
I'm trying to generate an encryption key for a file and then save it for use next time the script runs. I know that's not very secure, but it's just an interim solution for keeping a password out of a git repo.
subprocess.call('export KEY="password"', shell=True) returns 0 and does nothing.
Running export KEY="password" manually in my bash prompt works fine on Ubuntu.
subprocess.call('export KEY="password"', shell=True)
creates a shell, sets your KEY and exits: accomplishes nothing.
Environment variables do not propagate to parent process, only to child processes. When you set the variable in your bash prompt, it is effective for all the subprocesses (but not outside the bash prompt, for a quick parallel)
The only way to make it using python would be to set the password using a master python script (using os.putenv("KEY","password") or os.environ["KEY"]="password") which calls sub-modules or processes.
Using Python:
#SET:
os.environ["EnvVar"] = "1"
#GET:
print os.environ["EnvVar"]
This isn't a thing you can do. Your subprocess call creates a subshell and sets the env var there, but doesn't affect the current process, let alone the calling shell.
recently, I want to use python script to set environment in linux.This is one line of my code:
p = subprocess.call(['/bin/csh', '-c', "source setup.csh"])
My setup.csh file is below:
add questa10.2b
add ds5-2013.06
setenv MODELSIM modelsim.ini
But when I run my python, it shows that the files have sourced on screen, but it turns out I have to type myself on command line.
How could I solve these problem? Can any one please help me with this?
You're creating a new csh shell as a subprocess and then running your commands inside that shell, which then terminates. The commands do not run in, or affect, the parent shell within which Python is running. When you just run the commands yourself, they affect the current shell.
If you need these settings to persist in your current shell after Python terminates, your best bet in general is to source setup.csh rather than putting it in a Python script. If other child processes of the Python script need your environment variables, you can alter os.environ.
I am writing a python script (Linux) that is adding some shell aliases (writes them to HOME/.bash_aliases).
In order to make an alias available immediately after it was written I should issue the following bash built-in:
source HOME/.bashrc
source is a bash built-in so I cannot just:
os.system(source HOME/.bashrc)
If i try something like:
os.system('/bin/bash -c source HOME/.bashrc')
...will freeze the script (just like is waiting for something).
Any suggestions ?
What you want is not possible. A program (your script) cannot modify the environment of the caller (the shell you run it from).
Another approach which would allow you to do something close is to write it in terms of a bash function, which is run in the same process and can modify the caller. Note that sourcing during runtime can have possible negative side-effects depending on what the user has in their bashrc.
what you are trying to do is impossible. or better: how you are trying to do it is impossible.
your bash command is wrong. bash -s command does not execute command. it just stores the string "command" in the variable $1 and then drops you to the prompt. that is why the python script seems to freeze. what you meant to do is bash -c command.
why do you source .bashrc? would it not be enough to just source .bash_aliases?
even if you got your bash command right, the changes will only take effect in the bash session started from python. once that bash session is closed, and your python script is done, you are back at your original bash session. all changes in the bash session started from python is lost.
everytime you want to change something in the current bash session, you have to do it from inside the current bash session. most of the commands you run from bash (system commands, python scripts, even bash scripts) will spawn another process, and everything you do in that other process will not affect your first bash session.
source is a bash builtin which allows you to execute commands inside the currently running bash session, instead of spawning another process and running the commands there. defining a bash function is another way to execute commands inside the currently running bash session.
see this answer for more information about sourcing and executing.
what you can do to achieve what you want
modify your python script to just do the changes necessary to .bash_aliases.
prepare a bash script to run your python script and then source .bash_aliases.
#i am a bash script, but you have to source me, do not execute me.
modify_bash_aliases.py "$#"
source ~/.bash_aliases
add an alias to your .bashrc to source that script
alias add_alias='source modify_bash_aliases.sh'
now when you type add_alias some_alias in your bash prompt it will be replaced with source modify_bash_aliases.sh and then executed. since source is a bash builtin, the commands inside the script will be executed inside the currently running bash session. the python script will still run in another process, but the subsequent source command will run inside your currently running bash session.
another way
modify your python script to just do the changes necessary to .bash_aliases.
prepare a bash function to run your python script and then source .bash_aliases.
add_alias() {
modify_bash_aliases.py "$#"
source ~/.bash_aliases
}
now you can call the function like this: add_alias some_alias
I had an interesting issue where I needed to source an RC file to get the correct output in my python script.
I eventually used this inside my function to bring over the same variables from the bash file I needed to source. Be sure to have os imported.
with open('overcloudrc') as data:
lines = data.readlines()
for line in lines:
var = line.split(' ')[1].split('=')[0].strip()
val = line.split(' ')[1].split('=')[1].strip()
os.environ[var] = val
Working solution from Can I use an alias to execute a program from a python script :
import subprocess
sp = subprocess.Popen(["/bin/bash", "-i", "-c", "nuke -x scriptpath"])
sp.communicate()