i am using SCONS Construction tool.
i am unable to use the environment variable which is initialized in python script.
In My project USER can change some variables to work with the compiler.
For that we have 2 files.
Config.py
Sconstruct
Config.py is having all the variables which are like Include directories, CFLAGS , CPPDEFINES etc. So, Here we can set some variables. Those variables i need to use in Sconstruct file. In config.py i set a variable like below
SCONS_INC = "Include files"
os.environ["SCONS_INC"] = SCONS_INC
I need to use those variables in Sconstruct File. The code is
env["CPPPATH"] = os.environ["SCONS_INC"]
But I am getting an error like Undefined variable SCONS_INC.
How to do this?
SCons by default does not use the invoked environment, this is to make sure that you can reproduce the build no matter which configurations your environment have.
The environment variables are stored within the scons environment under the key ENV so you access the general environment variables like this:
env = Environment()
variable = env['ENV']['SomeVariable']
env['ENV']['SomeVariable'] = SomeValue
I understand your question like you need to use variables set in the python script within SCons. To do this you need to transfer them using the two method you describe in combination.
env = Enviroment()
python_variable = os.environ['SomeVariable']
env['ENV']['SomeVariable'] = python_variable
I would however perhaps recommend other ways of controlling the build, so you do not have to go with the hassle of transferring environment variable. IMHO using arguments are simpler. The arguments are simply a dict that are generated by the invocation of scons, so when you say:
scons -D some_argument=blob
You can get that argument by simply:
some_variable = ARGUMENTS["some_argument"]
Of course I do not know why you need the environment variables, so this might be completely irrelevant for you.
I once had a similar need, where the compiler was looking for a certain Env variable that hadnt been set. I was able to solve this problem as follows:
env = Environment()
env['ENV']['THE_VARIABLE'] = 'SomeValue'
Related
I believe that running an external command with a slightly modified environment is a very common case. That's how I tend to do it:
import subprocess, os
my_env = os.environ
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)
I've got a gut feeling that there's a better way; does it look alright?
I think os.environ.copy() is better if you don't intend to modify the os.environ for the current process:
import subprocess, os
my_env = os.environ.copy()
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)
That depends on what the issue is. If it's to clone and modify the environment one solution could be:
subprocess.Popen(my_command, env=dict(os.environ, PATH="path"))
But that somewhat depends on that the replaced variables are valid python identifiers, which they most often are (how often do you run into environment variable names that are not alphanumeric+underscore or variables that starts with a number?).
Otherwise you'll could write something like:
subprocess.Popen(my_command, env=dict(os.environ,
**{"Not valid python name":"value"}))
In the very odd case (how often do you use control codes or non-ascii characters in environment variable names?) that the keys of the environment are bytes you can't (on python3) even use that construct.
As you can see the techniques (especially the first) used here benefits on the keys of the environment normally is valid python identifiers, and also known in advance (at coding time), the second approach has issues. In cases where that isn't the case you should probably look for another approach.
With Python 3.5 you could do it this way:
import os
import subprocess
my_env = {**os.environ, 'PATH': '/usr/sbin:/sbin:' + os.environ['PATH']}
subprocess.Popen(my_command, env=my_env)
Here we end up with a copy of os.environ and overridden PATH value.
It was made possible by PEP 448 (Additional Unpacking Generalizations).
Another example. If you have a default environment (i.e. os.environ), and a dict you want to override defaults with, you can express it like this:
my_env = {**os.environ, **dict_with_env_variables}
you might use my_env.get("PATH", '') instead of my_env["PATH"] in case PATH somehow not defined in the original environment, but other than that it looks fine.
To temporarily set an environment variable without having to copy the os.envrion object etc, I do this:
process = subprocess.Popen(['env', 'RSYNC_PASSWORD=foobar', 'rsync', \
'rsync://username#foobar.com::'], stdout=subprocess.PIPE)
The env parameter accepts a dictionary. You can simply take os.environ, add a key (your desired variable) (to a copy of the dict if you must) to that and use it as a parameter to Popen.
I know this has been answered for some time, but there are some points that some may want to know about using PYTHONPATH instead of PATH in their environment variable. I have outlined an explanation of running python scripts with cronjobs that deals with the modified environment in a different way (found here). Thought it would be of some good for those who, like me, needed just a bit more than this answer provided.
In certain circumstances you may want to only pass down the environment variables your subprocess needs, but I think you've got the right idea in general (that's how I do it too).
This could be a solution :
new_env = dict([(k,(':'.join([env[k], v]) if k in env else v)) for k,v in os.environ.items()])
I believe that running an external command with a slightly modified environment is a very common case. That's how I tend to do it:
import subprocess, os
my_env = os.environ
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)
I've got a gut feeling that there's a better way; does it look alright?
I think os.environ.copy() is better if you don't intend to modify the os.environ for the current process:
import subprocess, os
my_env = os.environ.copy()
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)
That depends on what the issue is. If it's to clone and modify the environment one solution could be:
subprocess.Popen(my_command, env=dict(os.environ, PATH="path"))
But that somewhat depends on that the replaced variables are valid python identifiers, which they most often are (how often do you run into environment variable names that are not alphanumeric+underscore or variables that starts with a number?).
Otherwise you'll could write something like:
subprocess.Popen(my_command, env=dict(os.environ,
**{"Not valid python name":"value"}))
In the very odd case (how often do you use control codes or non-ascii characters in environment variable names?) that the keys of the environment are bytes you can't (on python3) even use that construct.
As you can see the techniques (especially the first) used here benefits on the keys of the environment normally is valid python identifiers, and also known in advance (at coding time), the second approach has issues. In cases where that isn't the case you should probably look for another approach.
With Python 3.5 you could do it this way:
import os
import subprocess
my_env = {**os.environ, 'PATH': '/usr/sbin:/sbin:' + os.environ['PATH']}
subprocess.Popen(my_command, env=my_env)
Here we end up with a copy of os.environ and overridden PATH value.
It was made possible by PEP 448 (Additional Unpacking Generalizations).
Another example. If you have a default environment (i.e. os.environ), and a dict you want to override defaults with, you can express it like this:
my_env = {**os.environ, **dict_with_env_variables}
you might use my_env.get("PATH", '') instead of my_env["PATH"] in case PATH somehow not defined in the original environment, but other than that it looks fine.
To temporarily set an environment variable without having to copy the os.envrion object etc, I do this:
process = subprocess.Popen(['env', 'RSYNC_PASSWORD=foobar', 'rsync', \
'rsync://username#foobar.com::'], stdout=subprocess.PIPE)
The env parameter accepts a dictionary. You can simply take os.environ, add a key (your desired variable) (to a copy of the dict if you must) to that and use it as a parameter to Popen.
I know this has been answered for some time, but there are some points that some may want to know about using PYTHONPATH instead of PATH in their environment variable. I have outlined an explanation of running python scripts with cronjobs that deals with the modified environment in a different way (found here). Thought it would be of some good for those who, like me, needed just a bit more than this answer provided.
In certain circumstances you may want to only pass down the environment variables your subprocess needs, but I think you've got the right idea in general (that's how I do it too).
This could be a solution :
new_env = dict([(k,(':'.join([env[k], v]) if k in env else v)) for k,v in os.environ.items()])
I am trying to create a pipeline and have defined some variable in my gitlab job which will be used by python script to populate results.
Is there any way I can call/define those variable in my python script.
For example:
.test-report:
extends: .test_report_setup
tags: *def_runners_tags
variables:
value-path: ${CI_PROJECT_DIR}/value
Now in my python script I want to call the variable 'value-path' to fetch or read files located in that directory. (Please note that variable is a custom variable on gitlab)
file_path = '<value-path>' <---- To get the gitlab job variable here
file_in = open(file_path + "id.txt")
Please help me how I can get it in my python script as I am a bit stuck on it.
Any suggestion/help on this will be appreciated. Thanks in advance.
You can access environment variables with os.environ or os.getenv. os.environ is a dict with the environment variables and will fail if you attempt to retrieve a key that doesn't exist. os.getenv is basically os.environ.get, which allows you to set a default value if there is no environment variable with that name.
Running python tests in VSCode. I have a .env file that is loading (I can add new vars, and see the change reflected in a print(... of os.environ.
However, variable expansion within that file is doing something odd. It seems to expand ${workspaceFolder} in some cases (specifically, PYTHONPATH=) but not others. ??
Here's my .env file. Again, it is loading, as I can see changes if i add vars, or change the contents of the (fully working) PYTHONPATH var:
FOOBAR1=${workspaceFolder}/foobar1.txt
PYTHONPATH=${workspaceFolder}/backend/src:/tmp/foo-2:${env:PYTHONPATH}
however print('ing all the os.environ's shows:
...
?
v
FOOBAR1=/foobar1.txt
PYTHONPATH=/home/-omitted-/backend/src:/tmp/foo-2:
...
${workspaceFolder} is expanded for one var, but not the other?? I've tried re-ordering. Don't see anything here or on google.
Help !? Thank you!
There is no variable expansion in vscode .env files at all. Any ${workspaceFolder} or similar variables are replaced by empty strings.
Note: Variable substitution is only supported in VS Code settings files, it will not work in .env environment files.
https://code.visualstudio.com/docs/python/environments#_environment-variable-definitions-file
Something else is probably prepending the /home/-omitted- to PYTHONPATH.
Is it possible using Python 3.5 to create and update environment variables in Windows and Linux so that they get persisted?
At the moment I use this:
import os
os.environ["MY_VARIABLE"] = "TRUE"
However it seems as if this does not "store" the environment variable persistently.
I'm speaking for Linux here, not sure about Windows.
Environment variables don't work that way. They are a part of the process (which is what you modify by changing os.environ), and they will propagate to child processes of your process (and their children obviously). They are in-memory only, and there is no way to "set and persist" them directly.
There are however several configuration files which allow you to set the environment on a more granular basis. These are read by various processes, and can be system-wide, specific to a user, specific to a shell, to a particular type of process etc.
Some of them are:
/etc/environment for system-wide variables
/etc/profile for shells (and their children)
Several other shell-specific files in /etc
Various dot-files in a user's home directory such as .profile, .bashrc, .bash_profile, .tcshrc and so on. Read your shell's documentation.
I believe that there are also various ways to configure environment variables launched from GUIs (e.g. from the gnome panel or something like that).
Most of the time you'll want to set environment variables for the current user only. If you only care about shells, append them to ~/.profile in this format:
NAME="VALUE"
The standard way to 'persist' an environment variable is with a configuration file. Write your application to open the configuration file and set every NAME=VARIABLE pair that it finds. Optionally this step could be done in a wrapper startup script.
If you wish to 'save' the state of a variable, you need to open the configuration file and modify its contents. Then when it's read in again, your application will set the environment accordingly.
You could of course store the configuration in some other way. For example in a configuration_settings class that you pickle/shelve. Then on program startup you read in the pickled class and set the environment. The important thing to understand is that when a process exits its environment is not saved. Environments are inherited from the parent process as an intentional byproduct of forking.
Config file could look like:
NAME=VALUE
NAME2=VALUE2
...
Or your config class could look like:
class Configuration():
env_vars = {}
import os
def set(name, val):
env_vars[name] = val
os.environ[name] = val
def unset(name):
del env_vars[name]
del os.environ[name]
def init():
for name in env_vars:
os.environ[name] = env_vars[name]
Somewhere else in our application
import shelve
filename = "config.db"
d = shelve.open(filename)
# get our configuration out of shelve
config = d['configuration']
# initialize environment
config.init()
# setting an environment variable
config.set("MY_VARIABLE", "TRUE")
#unsetting
config.unset("MY_VARIABLE")
# save our configuration
d['configuration'] = config
Code is not tested but I think you get the jist.