This question already has answers here:
How to set environment variables in Python?
(19 answers)
Closed 3 years ago.
My python script which calls many python functions and shell scripts. I want to set a environment variable in Python (main calling function) and all the daughter processes including the shell scripts to see the environmental variable set.
I need to set some environmental variables like this:
DEBUSSY 1
FSDB 1
1 is a number, not a string. Additionally, how can I read the value stored in an environment variable? (Like DEBUSSY/FSDB in another python child script.)
Try using the os module.
import os
os.environ['DEBUSSY'] = '1'
os.environ['FSDB'] = '1'
# Open child processes via os.system(), popen() or fork() and execv()
someVariable = int(os.environ['DEBUSSY'])
See the Python docs on os.environ. Also, for spawning child processes, see Python's subprocess docs.
First things first :) reading books is an excellent approach to problem solving; it's the difference between band-aid fixes and long-term investments in solving problems. Never miss an opportunity to learn. :D
You might choose to interpret the 1 as a number, but environment variables don't care. They just pass around strings:
The argument envp is an array of character pointers to null-
terminated strings. These strings shall constitute the
environment for the new process image. The envp array is
terminated by a null pointer.
(From environ(3posix).)
You access environment variables in python using the os.environ dictionary-like object:
>>> import os
>>> os.environ["HOME"]
'/home/sarnold'
>>> os.environ["PATH"]
'/home/sarnold/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
>>> os.environ["PATH"] = os.environ["PATH"] + ":/silly/"
>>> os.environ["PATH"]
'/home/sarnold/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/silly/'
If you want to pass global variables into new scripts, you can create a python file that is only meant for holding global variables (e.g. globals.py). When you import this file at the top of the child script, it should have access to all of those variables.
If you are writing to these variables, then that is a different story. That involves concurrency and locking the variables, which I'm not going to get into unless you want.
Use os.environ[str(DEBUSSY)] for both reading and writing (http://docs.python.org/library/os.html#os.environ).
As for reading, you have to parse the number from the string yourself of course.
Related
I have two source files that I am running in Python 3.9. (The files are big...)
File one (fileOne.py)
# ...
sessionID = uuid.uuid4().hex
# ...
File two (fileTwo.py)
# ...
from fileOne import sessionID
# ...
File two is executed using module multiprocessing.
When I run on my local machine and print the UUID in file two, it is always unique.
When I run the script on Centos OS, it somehow remained the same
If I restart the service, the UUID will change once.
My question: Why does this work locally (Windows OS) as expected, but not on a CentOS VM?
UPDATE 1.0:
To make it clear.
For each separate process, I need that UUID will be the same across FileOne and FileTwo. WHich mean
processOne = UUID in file one and in file two will be 1q2w3e
processTwo = UUID in file one and in file two will be r4t5y6 (a different one)
Your riddle is likely is caused by the way multi-processing works in different operating systems. You don't mention, but your "run locally" is certainly Windows or MacOS, not a Linux or other Unix Flavor.
The thing is that multiprocessing on Linux (and up to a time ago on MacOS, but changed that on Python 3.8), used a system fork call when using multiprocessing: the current process is duplicatesd "as is" with all its defined variables and classes - since your sessionID is defined at import time, it stays the same in all subprocesess.
Windows lacks the fork call, and multiprocessing resorts to start a new Python interpreter which re-imports all modules from the current process (this leads to another, more common cause of confusion, where any code not guarded by an if __name__ == "__main__": on the entry Python file is re-executed). In your case the value for sessionID is regenerated.
Check the docs at: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
So, if you want the variable to behave reliably and have teh same value across all processes when running multiprocessing, you should either pass it as a parameter to the target functions in the other processes, or use a proper structure meant to share values across processes as documented here:
https://docs.python.org/3/library/multiprocessing.html#sharing-state-between-processes
(you can also check this recent question about the same topic: why is a string printing 3 times instead of 1 when using time.sleep with multiprocessing imported?)
If you need a unique ID across files for each different process:
(As is more clear from the edit and comments)
Have a global (plain) dictionary which will work as a per-process registry for the IDs, and use a function to retrieve the ID - the function can use os.getpid() as a key to the registry.
file 1:
import os
import uuid
...
_id_registry = {}
def get_session_id():
return _id_registry.setdefault(os.getpid(), uuid.uuid4())
file2:
from file1 import get_session_id
sessionID = get_session_id()
(the setdefault dict method takes care of providing a new ID value if none was set)
NB.: the registry set up in this way will keep at most the master process ID (if multiprocessing is using fork mode) and itself - no data on the siblings, as each process will hold its own copy of the registry. If you need a working inter-process dictionary (which could hold a live registry for all processes, for example) you will probably be better using redis for it (https://redis.io - certainly one of the Python bindings have a transparent Python-mapping-over-redis, so you don´t have to worry with the semantics of it)
When you run your script it generate the new value of the uuid, but when you run it inside some service you code the same as:
sessionID = 123 # simple constant
so to fix the issue you can try wrap the code to the function, for example:
def get_uuid():
return uuid.uuid4().hex
in you second file:
from frileOne import get_uuid
get_uuid()
I'm a bit experienced without other languages but, novice with Python. I have come across made codes in jupyter notebooks where sys is imported.
I can't see the further use of the sys module in the code. Can someone help me to understand what is the purpose of importing sys?
I do know about the module and it's uses though but can't find a concise reason of why is it used in many code blocks without any further use.
If nothing declared within sys is actually used, then there's no benefit to importing it. There's not a significant amount of cost either.
Sys module is a rather useful module as it allows you to work with your System and those things. Eg:
You can access any command line arguments using sys.argv[1:]
You can see the Path to files.
Version of your Python Interpreter using sys.version
Exit the running code with sys.exit
Mostly you will use it for accessing the Command Line arguments.
I'm a new pythonista bro, I learned to import it whenever I want to exit the program with a nice exit text in red
import sys
name = input("What's your name? ")
if name == "Vedant":
print(f"Hello There {name}.")
else:
sys.exit(f"You're not {name}!")
The sys includes "functions + variable " to help you control and change the python environment #runtime.
Some examples of this control includes:
1- using other sources data as input via using:
sys.stdin
2- using data in the other resources via using:
sys.stdout
3- writing errors when an exception happens, automatically in :
sys.stderr
4- exit from the program by printing a message like:
sys.exit("Finish with the calculations.")
5- The built-in variable to list the directories which the interpreter will looking for functions in them:
sys.pasth
6- Use a function to realize the number of bytes in anonymous datatype via:
sys.getsizeof(1)
sys.getsizeof(3.8)
This question already has answers here:
How can I create multiple variables from a list of strings? [duplicate]
(2 answers)
How do I create variable variables?
(17 answers)
Closed 5 years ago.
I'm working with python 3.4.1 interpreter in pycharm.
I'm trying to do several things on a changing number of variables.
I'v found it convenient to use exec() in order of changing the variable name and execute the same command on a different variable name.
After doing it successfully for some time, for some reason, errors started to appear.
Those errors happened because I had variables initialization within an exec() command so they weren't recognized by the interpreter.
While changing those lines and removing unnecessary variables, I wrote one command that seems to be running just fine when executes in python console, but won't run in run time or debugging mode.
Usually I use the console as a expression evaluator, so it helps me understand and correct wrong commands, but this time it seems to be the right command and still nothing happens when i'm running the script.
This is the code, doc2text_path was initialized erlier:
$newdir = ""
exec("newdir = " + "doc2text_path%s" % scrape_number)
The command executes and the value of newdir remains "".
newdir is a new variable that i want to create, doc2txt_path is a string variable containing the path of a file I want to write in to later on.
I have about 4 files like that one all referred to with variables in following name format: doc2text_path and a serial number: doc2text_path1 doc2text_path2 etc'
That way i can use a for loop in order of referring to each one of them.
thanks.
This question already has answers here:
Actual meaning of 'shell=True' in subprocess
(7 answers)
Closed 7 years ago.
It seems whenever I try to use Python's subprocess module, I find I still don't understand some things. Currently, I was trying to join 3 mp4 files from within a Python module.
When I tried
z ='MP4Box -cat test_0.mp4 -cat test_1.mp4 -cat test_2.mp4 -new test_012d.mp4'
subprocess.Popen(z,shell=True)
Everything worked.
When I tried
z = ['MP4Box', '-cat test_0.mp4', '-cat test_1.mp4', '-cat test_2.mp4', '-new test_012d.mp4']
subprocess.Popen(z,shell=False)
I got the following error:
Option -cat test_0.mp4 unknown. Please check usage
I thought that for shell=False I just needed to supply a list where the first element was the executable I wanted to run and each succeeding element was an argument to that executable. Am I mistaken in this belief, or is there a correct way to create the command I wanted to use?
Also, are there any rules for using Shell=True in subprocess.Popen? So far, all I really know(?) is "don't do it - you can expose your code to Shell injection attacks". Why does Shell=False avoid this problem? Is there ever an actual advantage to using 'Shell=True`?
If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
When shell=True is dangerous?
If we execute shell commands that might include unsanitized input from an untrusted source, it will make a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input
Eg. (Taken from docs)
>>> from subprocess import call
>>> filename = input("What file would you like to display?\n")
What file would you like to display?
non_existent; rm -rf / #
>>> call("cat " + filename, shell=True) # Uh-oh. This will end badly..
You have to give every single argument as one element of a list:
z = ['MP4Box', '-cat', 'test_0.mp4', '-cat', 'test_1.mp4', '-cat', 'test_2.mp4', '-new', 'test_012d.mp4']
subprocess.Popen(z,shell=False)
This is normally what you want to do, because you don't need to escape especial characters of the shell in filenames.
Is there a way to modify the PATH environment variable in a platform independent way using python?
Something similar to os.path.join()?
You should be able to modify os.environ.
Since os.pathsep is the character to separate different paths, you should use this to append each new path:
os.environ["PATH"] += os.pathsep + path
or, if there are several paths to add in a list:
os.environ["PATH"] += os.pathsep + os.pathsep.join(pathlist)
As you mentioned, os.path.join can also be used for each individual path you have to append in the case you have to construct them from separate parts.
Please note that os.environ is not actually a dictionary. It's a special dictionary-like object which actually sets environment variables in the current process using setenv.
>>> os.environ.__class__
<class os._Environ at 0x100472050>
>>> import os
>>> os.environ["HELLO"] = "WORLD"
>>> os.getenv("HELLO")
'WORLD'
This means that PATH (and other environment variables) will be visible to C code run in the same process.
(Since comments can't contain formatting, I have to put this in an answer, but I feel like it's an important point to make. This is really a comment on the comment about there being no equivalent to 'export'.)
The caveat to be aware of with modifying environment variables in Python, is that there is no equivalent of the "export" shell command. There is no way of injecting changes into the current process, only child processes.
You could refresh it like this
os.environ["PATH"] = os.environ["PATH"]