When trying to write script with python, I have a fundamental hole of knowledge.
Update: Thanks to the answers I corrected the word shell to process/subprocess
Nomenclature
Starting with a Bash prompt, lets call this BASH_PROCESS
Then within BASH_PROCESS I run python3 foo.py, the python script runs in say PYTHON_SUBPROCESS
Within foo.py is a call to subprocess.run(...), this subprocess command runs in say `SUBPROCESS_SUBPROCESS
Within foo.py is subprocess.run(..., shell=True), this subprocess command runs in say SUBPROCESS_SUBPROCESS=True
Test for if a process/subprocess is equal
Say SUBPROCESS_A starts SUBPROCESS_B. In the below questions, when I say is SUBPROCESS_A == SUBPROCESS_B, what I means is if SUBPROCESS_B sets an env variable, when it runs to completion, will they env variable be set in SUBPROCESS_A? If one runs eval "$(ssh-agent -s)" in SUBPROCESS_B, will SUBPROCESS_A now have an ssh agent too?
Question
Using the above nomenclature and equality tests
Is BASH_PROCESS == PYTHON_SUBPROCESS?
Is PYTHON_SUBPROCESS == SUBPROCESS_SUBPROCESS?
Is PYTHON_SUBPROCESS == SUBPROCESS_SUBPROCESS=True?
If SUBPROCESS_SUBPROCESS=True is not equal to BASH_PROCESS, then how does one alter the executing environment (e.g. eval "$(ssh-agent -s)") so that a python script can set up the env for the calller?
You seem to be confusing several concepts here.
TLDR No, there is no way for a subprocess to change its parent's environment. See also Global environment variables in a shell script
You really don't seem to be asking about "shells".
Instead, these are subprocesses; if you run python foo.py in a shell, the Python process is a subprocess of the shell process. (Many shells let you exec python foo.py which replaces the shell process with a Python process; this process is now a subprocess of whichever process started the shell. On Unix-like systems, ultimately all processes are descendants of process 1, the init process.)
subprocess runs a subprocess, simply. If shell=True then the immediate subprocess of Python is the shell, and the command(s) you run are subprocesses of that shell. The shell will be the default shell (cmd on Windows, /bin/sh on Unix-like systems) though you can explicitly override this with e.g. executable="/bin/bash"
Examples:
subprocess.Popen(['printf', '%s\n', 'foo', 'bar'])
Python is the parent process, printf is a subprocess whose parent is the Python process.
subprocess.Popen(r"printf '%s\n' foo bar", shell=True)
Python is the parent process of /bin/sh, which in turn is the parent process of printf. When printf terminates, so does sh, as it has reached the end of its script.
Perhaps notice that the shell takes care of parsing the command line and splitting it up into the four tokens we ended up explicitly passing directly to Popen in the previous example.
The commands you run have access to shell features like wildcard expansion, pipes, redirection, quoting, variable expansion, background processing, etc.
In this isolated example, none of those are used, so you are basically adding an unnecessary process. (Maybe use shlex.split() if you want to avoid the minor burden of splitting up the command into tokens.) See also Actual meaning of 'shell=True' in subprocess
subprocess.Popen(r"printf '%s\n' foo bar", shell=True, executable="/bin/bash")
Python is the parent process of Bash, which in turn is the parent process of printf. Except for the name of the shell, this is identical to the previous example.
There are situations where you really need the slower and more memory-hungry Bash shell, when the commands you want to execute require features which are available in Bash, but not in the Bourne shell. In general, a better solution is nearly always to run as little code as possible in a subprocess, and instead replace those Bash commands with native Python constructs; but if you know what you are doing (or really don't know what you are doing, but need to get the job done rather than solve the problem properly), the facility can be useful.
(Separately, you should probably avoid bare Popen when you can, as explained in the subprocess documentation.)
Subprocesses inherit the environment of their parent when they are started. On Unix-like systems, there is no way for a process to change its parent's environment (though the parent may participate in making that possible, as in your eval example).
To perhaps accomplish what you may ultimately be asking about, you can set up an environment within Python and then start your other command as a subprocess, perhaps then with an explicit env= keyword argument to point to the environment you want it to use:
import os
...
env = os.environ.copy()
env["PATH"] = "/opt/foo:" + env["PATH"]
del env["PAGER"]
env["secret_cookie"] = "xyzzy"
subprocess.Popen(["otherprogram"], env=env)
or have Python print out values in a form which can safely be passed to eval in the Bourne shell. (Caution: this requires you to understand the perils of eval in general and the target shell's quoting conventions in particular; also, you will perhaps need to support the syntax of more than one shell, unless you are only targeting a very limited audience.)
... Though in many situations, the simplest solution by far is to set up the environment in the shell, then run Python as a subprocess of that shell instance (or exec python if you want to get rid of the shell instance after it has performed its part; see also What are the uses of the exec command in shell scripts?)
Python without an argument starts the Python REPL, which could be regarded as a "shell", though we would commonly not use that term (perhaps instead call it "interactive interpreter" - see also below); but python foo.py simply runs the script foo.py and exits, so there is no shell there.
The definition of "shell" is slightly context-dependent, but you don't really seem to be asking about shells here. (Some GUIs have a concept of "graphical shell" etc but we are already out of the scope of what you were trying to ask about.) Some programs are command interpreters (the Python executable interprets and executes commands in the Python language; the Bourne shell interprets and executes shell scripts) but generally only those whose primary purposes include running other programs are called "shells".
None of those equalities are true, and half of those "shells" aren't actually shells.
Your bash shell is a shell. When you launch your Python script from that shell, the Python process that runs the script is a child process of the bash shell process. When you launch a subprocess from the Python script, that subprocess is a child process of the Python process. If you launch the subprocess with shell=True, Python invokes a shell to parse and run the command, but otherwise, no shell is involved in running the subprocess.
Child processes inherit environment variables from their parent on startup (unless you take specific steps to avoid that), but they cannot set environment variables for their parent. You cannot run a Python script to set environment variables in your shell, or run a subprocess from Python to set your Python script's environment variables.
I'm trying to run my virtualenv called env (created before) using call() from subprocess and it doesn't work. Command is fine, it's working when I type it directly in terminal.
python code:
import subprocess
subprocess.call("source env/bin/activate", shell=True)
I was trying also:
import os
os.system("source env/bin/activate")
any ideas why command is not performed or what should I use instead os.system() and subprocess.call()?
In both examples, your code launches a subprocess. That subprocess, which has its own environment, attempts to run the "source" command, and then exits. Since the subprocess has its own environment, this has no effect on the current process.
Assuming your end goal is to run some other command in the subprocess, you should just run it directly. You don't specifically need to activate the virtual environment.
subprocess.call(["./env/bin/pip", "list"])
Avoid using the shell=True option if at all possible; it can be quite dangerous if you're not extremely careful with it.
If you really need to set the environment variables that the activate script sets in this script, you need to manually set them in os.environ. You can read the activate script to see what they are. They usually won't matter, though.
I want to write a python program that runs in the background.
I mean, like we install Python package. And later, we can run any script using python in front of the script name. This means that some python process is running in background which can take inputs and perform actions.
And in case of linux, you can call grep from anywhere. That means grep is also running in the background somehow.
I want to write something like that in python. When I call certain function with name and arguments at any time, it should perform the intended action without caring for the original code. But I am not able to find how to achieve that.
Can anyone please help me here?
Thanks in advance.
Clarification: the fact that you can run python or grep in a console just by typing their name, does not mean that they run in background. It means that there exist an executable file in some location, and this location is listed in the environment variable PATH.
For example, on my system I can run Python by typing python. The python executable is installed at /usr/local/bin/python, and has the execute permission bit on.
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
and yes, /usr/local/bin is contained in PATH.
You can do the same with python scripts:
ensure that the very first line of your script contains #!/usr/bin/python or #!/usr/bin/env python
give your script execute permissions: chmod a+x yourScript
either move your script to one of the directories contained in $PATH, or add the directory where your script is located to PATH: export PATH=$PATH:/home/you/scripts
Have a look at
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
you can roll out your own daemon by inheriting the Daemon class and overriding run method
from daemon import Daemon
class run_daemon(Daemon):
def run(self):
import sys
run_daemon.execute_shell_command(sys.argv[1])
#staticmethod
def execute_shell_command(ShellCommand):
import subprocess
process = subprocess.Popen(ShellCommand, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.communicate()
I have a program with two different versions on this computer. I can't get rid of the older one because I don't have root access, but I put the newer one first in 'bin' in my home directory (which is the first thing in my $PATH)
I tried calling it with Python's Popen
Popen(['clingo'...]...)
and it worked fine
But then I needed to set an environment variable before calling it so I renamed 'clingo' to 'run_clingo' in the bin directory and replaced it with a script:
# File: ~/bin/clingo
export LD_LIBRARY_PATH=/usr/local/gcc4.8.1/lib64:$LD_LIBRARY_PATH
run_clingo $#
When I run 'clingo' from terminal, it works fine, but Python Popen calls the older version (which is later on the PATH).
How can I get Popen to call the right one? Why does changing an executable to a script cause Popen to search elsewhere?
Add shebang #!/bin/sh at the very top of ~/bin/clingo and run:
$ chmod +x ~/bin/clingo
to make the file executable.
Popen() calls os.execvp() to execute a program. Its behaviour differs from your shell that emulates execvp(): if there is no the shebang (!#.. at the top) in the script then the shell reruns it as a shell script but Python's os.execvp() runs the next file in PATH instead.
Wild guess is that Popen does not use the same PATH as your terminal. I would check the shebang line (#!) sometimes people put env -i python there, which explicitly asks fro empty, i.e. default environment.
Note that subprocess.call is the recommended way, not using the Popen directly.
I need to execute the simple command below in windows 7 command prompt using Python26.
cd C:\Python26\main project files\Process
C:\Aster\runtime\waster Analysis.comm
It runs a FEM simulation and I tried it manually and it worked well. Now, I want to automate the write procedure using Python26.
I studied the other questions and found that the os.system works but it didn't. Also I saw subprocess module but it didn't work.
The current directory is a process property: Every single process has its own current directory. A line like
os.system("cd xyz")
starts a command interpreter (cmd.exe on Windows 7) and execute the cd command in this subprocess, not affecting the calling process in any way. To change the directory of the calling process, you can use os.chdir() or the cwd keyword parameter to subprocess.Popen().
Example code:
p = subproces.Popen(["C:/Aster/runtime/waster", "Analysis.comm"],
cwd="C:/Python26/main project files/Process")
p.wait()
(Side notes: Use forward slashes in path names in Python files. You should avoid os.system() and passing shell=True to the function in the subprocess module unless really necessary.)