I have a bash backup script run as root (cron) that delegates certain tasks to other specific bash scripts owned by different users. (simplified example, principle is, some things have to be done as root, different tasks are delegated to users with the appropriate environment (oracle, amazon, ...)
mkdir -p /tmp/backup$NAME
su - oracle -c "~/.backups/export-test.sh"
tar cf /tmp/backup/$NOW.tar /tmp/backup$NAME
su - amazon upload_to_amazon.sh /tmp/backup/$NOW.tar
This script itself does then some tasks as user oracle:
mkdir -p $TMP_LOCATION
cd ~/.backups
exp $TMP_LOCATION/$NAME-$NOW
When I try to mimic this behaviour in python I came up with the following (started from cron as root)
name = "oracle"
# part run as root
os.makedirs(tmp_backup + name)
os.setegid(pwd.getpwnam(name)[3])
os.seteuid(pwd.getpwnam(name)[2])
# part run as oracle
os.makedirs(tmp_location)
os.chdir(os.path.expanduser("~{user}/.backups".format(user=name)))
subprocess.check_call(["exp",
"os.path.join(tmp_location, name+'-'+now)"
])
In bash when using su -, a real new shell is invoked and all environment variables of that user are set.
How can I improve this for my python script? Is there a standard recipe I can follow? I'm thinking of environment variables, umask, ...
the environment is Solaris if that might matter.
all environment variables of that user are set
Usually because a shell runs a .profile file when it starts up.
You have several choices.
Create a proper subprocess with subprocess.Popen to execute the shell .profile -- same as su -.
Carefully locate the environment variable settings and mimic them in Python. The issue is that a .profile can do all kinds of crazy things, making it a potential problem to determine the exact effects of the .profile.
Or you can extract the relevant environment variables to make the accessible to both the shell environment and your Python programs.
First. Read the .profile for each user to be clear on what environment variables it sets (different from things like aliases or other craziness that doesn't apply to your Python script). Some of these environment variables are relevant to the scripts you're running. Some aren't relevant.
Second. Split the "relevant" environment variables into a tidy env_backups.sh script or env_uploads.sh script.
Once you have those environment variable scripts, update your .profile files to replace the environment variables settings with source env_backup.sh or source env_uploads.sh.
Third. Source the relevant env_this and env_that scripts before running the Python program. Now your Python environment shares the variables with your shell environment and you only maintain them in one place.
my_script.sh
source ~oracle/env_backup.sh
source ~amazon/env_uploads.sh
python my_script.py
That seems best to me. (Since that's how we do it.)
I can run amazon as root, without needing environment variables after all.
I used boto for that.
As for the oracle environment variables I used this piece of code:
if "ORACLE_HOME" not in os.environ or os.environ["ORACLE_HOME"] != ORACLE_HOME:
logger.debug("setting ORACLE_HOME='{oh}'".format(oh=ORACLE_HOME))
os.environ['ORACLE_HOME'] = ORACLE_HOME
if ORACLE_HOME + "/bin" not in os.environ["PATH"].split(":"):
logger.debug("setting PATH='{p}'".format(p=os.path.expandvars(ORACLE_PATH)))
os.environ['PATH'] = os.path.expandvars(ORACLE_PATH)
if "NLS_LANG" not in os.environ or os.environ["NLS_LANG"] != NLS_LANG:
logger.debug("setting NLS_LANG='{n}'".format(n=NLS_LANG))
os.environ['NLS_LANG'] = NLS_LANG
Related
I'm using MacOS and working on a python project.
In my project there is a line:
num_workers = int(os.environ.get('NUM_SAS_WORKERS', 1))
I want to add NUM_SAS_WORKERS in my environment variables and set it to 10 so that my python project can load the value 10 into num_workers
I've tried executing this in my terminal:
export NUM_SAS_WORKERS=10
and I was able to verify that NUM_SAS_WORKERS = 10 exists by running printenv in terminal
But it doesn't work. In my python script num_workers is still loaded with 1
How to do it correctly?
If you want to access the NUM_SAS_WORKERS environment variable in the Python shell, run this command from your terminal:
NUM_SAS_WORKERS=10 python
And once in the Python shell:
>>> import os
>>> int(os.environ.get('NUM_SAS_WORKERS', 1))
10
If you want to access it in a file, very similar:
NUM_SAS_WORKERS=10 python yourfile.py
If you set up environment variables from the terminal command it gets erased as soon as you close that terminal. To set up and keep them you have to update them in .bash_profile file which resides in the home directory as a hidden file. Follow the below commands to update it.
Open terminal
cd ~
(To take you to the home directory of the mac)
open -a "Visual Studio Code" .bash_profile
( To open .bash_profile file, in the place of "Visual Studio Code" you can use any text editor name)
Now, .bash_profile will be opened and there you can set your environment variables and save it.
Example:
NUM_SAS_WORKERS=10
Now run echo $[NUM_SAS_WORKERS] in terminal to read that
After performing the above steps some times still environment variables not available to read in your python project/file. In such cases, you have to restart or log off and log in to your machine that can fix the issue.
To check whether the environment variable is available to your python code run the below script in your python console. This should print all your environment variables.
import os
print(os.eviron)
I would like to set environment variables for the scope of a terminal session.
Eg, the user would open a terminal, type something like setproject <project_name> and it would export an environment variable PROJECT. When they close the terminal, the variable setting is lost. What was impressive with this system was that it would know what project name was valid and what wasn’t. I’ve worked at places that have done this quite well and I’m wondering how it’s done.
I’d prefer the user not have to type source setproject <project_name> and I also wanted to check if the project they specified was valid or not.
I’ve written something in python, which checks the validity of a project name (against a yaml config file), and passes it on to a bash script, but it doesn’t really work (the environment variable isn’t properly set. It’s set, but only for the scope of the python script)
So, how’d I do this? Are there any good examples of what I’m looking for online?
You can't do this with a normal command -- normal commands (shell scripts, python scripts, binaries, whatever) run as a subprocess under the main shell process, and cannot affect the main shell process's environment. But the shell supports a couple of command-like things that "happen" in the shell itself: functions and aliases. These both need to be defined in the shell before they can be used, so you'd need to put the definition in .bash_profile (or whatever the appropriate shell init file is in your setup -- see here).
The way I would do this would be to define setproject as a shell function, something like this:
# ...in shell init file...
setproject() {
if [ some_test_for_project_validity ]; then
export PROJECT="$1"
else
echo "setproject: $1 is not a valid project name" >&2
return 1
fi
}
The logic you need here is too complicated to put in an alias (they're much simpler and dumber than functions), but you could put the actual script in a separate file and use an alias to avoid having to explicitly source it:
# ...in shell init file...
alias setproject='source /path/to/setproject'
# .../path/to/setproject...
# note no shebang, since it won't be run as a normal command
if [ some_test_for_project_validity ]; then
export PROJECT="$1"
else
echo "setproject: $1 is not a valid project name" >&2
return 1 # In a file to be source'd, use return instead of exit
fi
Inside of my scons script I execute another python script:
fs = env.Command('fs', None, 'python updatefs.py')
AlwaysBuild(fs)
Depends(fs, main)
In python script I am trying to access an environment variable:
import os
mode = os.environ['PROC_MODE']
The variable was previously set up in the shell:
export PROC_MODE='some_mode'
Python complain:
KeyError: 'PROC_MODE'
What is the proper way to propagate environment to an external script?
This is covered in lightly in the FAQ:
FAQ
Basically SCons constructs a clean reproducible set of environment variables so that differences in any user's environment won't break a build.
So if you want to propagate a particular variable from your shell you can explicitly do it as such:
env['ENV']['MY_VARIABLE']=os.environ['MY_VARIABLE']
If you wanted to progagate all environment variables you'd do this:
env['ENV'] = os.environ
Where env is your Environment()
I have a rails 4 app on AWS EC2 Ubuntu. It will call a python program using backticks:
pyresult = `python /path/to/abc.py`
The python programe works perfect on my local dev. But it failed on the production server, with no error. (pyresult is empty)
I've debugged for long time, and found a possible cause: $PYTHONPATH
On local, i have the follow in my .bashrc, as the python program asked to add:
export PYTHONPATH="$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH"
On server, i also added them in .bashrc on server. But run echo $PYTHONPATH in my app gives me empty string.
Then I added explicitly in my rails app,
`export PYTHONPATH="$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH"`
pyresult = `python /path/to/abc.py`
but $PYTHONPATH is still empty. and calling python program still failed with no error.
Is it correct to guess the missing $PYTHONPATH caused the issue? if yes, how to solve the problem? Thanks for helping.(sorry i'm new to linux and not familiar with python)
Linux doesn't have system variables, only environment variables, which are only inherited by child processes. This line
`export PYTHONPATH="$HOME/.local/lib/python2.7/site-packages/:$PYTHONPATH"`
only sets PYTHONPATH for the subshell started by the backquotes (and any processes started from that subshell). Once the shell exists, the setting for PYTHONPATH is gone.
You need to use whatever facility Ruby has for modifying the environment of the current process, which I gather (as someone who doesn't use Ruby) is
ENV['PYTHONPATH'] = '...'
Then the shell started by
pyresult = `python /path/to/abc.py`
will inherit its PYTHONPATH value from the current Ruby process.
After using Open4 gem to run the command, I found the rootcause is paython programe can't find the lib, which is installed via setup.py. So it's another issue.
I am trying to call a shell script that sets a bunch of environment variables on our server from a mercurial hook. The shell script gets called fine when a new changegroup comes in, but the environment variables aren't carrying over past the call to the shell script.
My hgrc file on the respository looks like this:
[hooks]
changegroup = shell_script
changegroup.env = env
I can see the output of the shell script, and then the output of the env command, but the env command doesn't include the new environment variables set by the shell script.
I have verified that the shell script works fine when run by itself but when run in the context of the mercurial hook it does not properly set the environment.
Shell scripts can't modify their enviroment.
http://tldp.org/LDP/abs/html/gotchas.html
A script may not export variables back to its parent process, the shell, or to the environment. Just as we learned in biology, a child process can inherit from a parent, but not vice versa
$ cat > eg.sh
export FOO="bar";
^D
$ bash eg.sh
$ echo $FOO;
$
also, the problem is greater, as you have multiple calls of bash
bash 1 -> hg -> bash 2 ( shell script )
-> bash 3 ( env call )
it would be like thinking I could set a variable in one php script and then magically get it with another simply by running one after the other.