We Keep Coding

Unexpected output of bash 'ps -p $$' command returned by 'subprocess.run()'

I am running Linux Mint 18.1 and Python 3.9. To find out which shell is executing shell commands I have started to use ps -p $$ which is expected to return the info about the shell as value of CMD.
When using subprocess.run() in Python not specifying the shell or specifying the shell as executable='sh' the CMD value is sh for both passed commands (see code below), but when I specify executable='bash' I get different results (ps and bash).
The GNOME-Terminal which is running bash prints bash as CMD value when running ps -p $$
What is the reason for the different values of CMD being ps printed by the code below in case of ps -p $$ and bash in case of ps -p $$;echo $0?
from subprocess import run
print(run('ps -p $$ ', capture_output=True, shell=True,
encoding='utf-8', executable='bash').stdout)
print(run('ps -p $$;echo $0', capture_output=True, shell=True,
encoding='utf-8', executable='bash').stdout)
which prints:
PID TTY TIME CMD
22928 ? 00:00:00 ps
PID TTY TIME CMD
22929 ? 00:00:00 bash
bash
UPDATE to respond to the given answer and comments:
#Charles Duffy : YES, without Python involved when running the commands in GNOME Terminal using bash -c I get the same behavior as if run with subprocess.run() in Python, but ... I don't get it when running without the preliminary bash -c.
#Barmar : Trying to check out the explanation in your answer I have introduced a third command echo $0;ps -p $$ to see if the last command in the sequence will give a CMD value of ps. Below the result of a terminal session:
$ bash -c 'ps -p $$'
PID TTY TIME CMD
23386 pts/1 00:00:00 ps
$ bash -c 'ps -p $$; echo $0'
PID TTY TIME CMD
23388 pts/1 00:00:00 bash
bash
$ bash -c 'echo $0;ps -p $$'
bash
PID TTY TIME CMD
23395 pts/1 00:00:00 bash
What have I misunderstood in your answer expecting from the third command to give ps as CMD value?

This is a bash optimization. If the command line is just a single command, it's is implemented by simply calling execv() rather than forking a child to execute the command. This replaces the shell process with the ps program, keeping the same PID. It's as if you executed.
print(run('exec ps -p $$', ...))
You don't see it in the second attempt because ps is not the last command in the sequence. It has to fork a child process, while the shell keeps running to wait for it to exit and execute the following commands.

Running shell command from python script with \n

I am trying to run the shell command
echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest -
from jupyter notebook using subprocesses
I have tried the code
p = subprocess.Popen('''echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest - ''', shell=True)
p.communicate()
and some iterations with run() or call(), but everytime the output is
-e 'FROM busybox
It seems that the new line character \n causes the problem. Any ideas to solve the problem?

The \n gets parsed by Python into a literal newline. You can avoid that by using a raw string instead,
p = subprocess.run(
r'''echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest - ''',
shell=True, check=True)
but I would recommend running a single process and passing in the output from Python; this also avoids a shell, which is generally desirable.
p = subprocess.run(['docker', 'build', '-t', 'myimage:latest', '-'],
input='FROM busybox\nRUN echo "hello world"',
text=True, check=True)
Notice also how we prefer subprocess.run() over the more primitive subprocess.Popen(); as suggested in the documentation, you want to avoid this low-level function whenever you can. With check=True we also take care to propagate any subprocess errors up to the Python parent process.
As an aside, printf is both more versatile and more portable than echo -e; I would generally recommend you to avoid echo -e altogether.
This ideone demo with nl instead of docker build demonstrates the variations, and coincidentally proves why you want to avoid echo -e even if your login shell is e.g. Bash (in which case you'd think it should be supported; but subprocess doesn't use your login shell).

How to pipe a Python's cmd class' input/output to another Python process?

Currently, I'm running an experiment with Mininet-Wifi. It's CLI is Python's Cmd module and this is how I am able to get accurate information about the simulated network environment. The simulator is running as its own process as sudo python on Ubuntu 14.04 or greater.
The remote controller for this network is POX. This time, only a script is running; everything is automated with preset commands - no more human interaction. What I would like to do is: the POX process needs to inject commands into the Mininet's process and retrieve the results of the execution of that command. This is because POX's logic must constantly query the state of the network via Mininet to be able to make decisions. When a decision has been made, POX must again inject a command into the Mininet process to alter the state of the network.
ADDENDUM: Currently, I am only able to access the hosts that are spawned by Mininet when I run the sudo python a_mininet_script, thanks to the utility function called m. After spawning the hosts, Mininet enters its CLI function, which is what I want to communicate with but can't. This is Mininet's m function.
#!/bin/bash
# Attach to a Mininet host and run a command
if [ -z $1 ]; then
echo "usage: $0 host cmd [args...]"
exit 1
else
host=$1
fi
pid=`ps ax | grep "mininet:$host$" | grep bash | grep -v mnexec | awk '{print $1};'`
if echo $pid | grep -q ' '; then
echo "Error: found multiple mininet:$host processes"
exit 2
fi
if [ "$pid" == "" ]; then
echo "Could not find Mininet host $host"
exit 3
fi
if [ -z $2 ]; then
cmd='bash'
else
shift
cmd=$*
fi
cgroup=/sys/fs/cgroup/cpu/$host
if [ -d "$cgroup" ]; then
cg="-g $host"
fi
# Check whether host should be running in a chroot dir
rootdir="/var/run/mn/$host/root"
if [ -d $rootdir -a -x $rootdir/bin/bash ]; then
cmd="'cd `pwd`; exec $cmd'"
cmd="chroot $rootdir /bin/bash -c $cmd"
fi
cmd="exec sudo mnexec $cg -a $pid $cmd"
eval $cmd
For example, to access the terminal of h1 from any terminal, not from the POX script, I'd call it thus:
sh m h1 ifconfig
But to call it from subprocess, it would be:
p = subprocess.Popen('echo my passwd | sudo -kS sh m h1 ifconfig', shell = True)
To repeat my question, I want to communicate with the CLI of the Mininet process from the POX controller, and not just the spawned hosts.

I guess you want to realize the function something like netstat -oan | findstr 80 (this is on windows to find the port 80), this is the pipeline command to pass the output of netstat -oan to the command findstr.
Then the python code is something like:
import subprocess
p1 = subprocess.Popen('netstat -oan', stdout=subprocess.PIPE, shell=True)
p2 = subprocess.Popen('findstr 80', stdin=p1.stdout, stdout=subprocess.PIPE, shell=True)
pipeline_output = p2.communicate()[0]
print pipeline_output
Then, the p1 process output will be passed to p2 process, FYI.

`ps -ef` shows running process twice if started with `subprocess.Popen`

I use the following snippet in a larger Python program to spawn a process in background:
import subprocess
command = "/media/sf_SharedDir/FOOBAR"
subprocess.Popen(command, shell=True)
After that I wanted to check whether the process was running when my Python program returned.
Output of ps -ef | grep -v grep | grep FOOBAR:
ap 3396 937 0 16:08 pts/16 00:00:00 /bin/sh -c /media/sf_SharedDir/FOOBAR
ap 3397 3396 0 16:08 pts/16 00:00:00 /bin/sh /media/sf_SharedDir/FOOBAR
I was surprised to see two lines of and they have differend PIDs so are those two processes running? Is there something wrong with my Popen call?
FOOBAR Script:
#!/bin/bash
while :
do
echo "still alive"
sleep 1
done
EDIT: Starting the script in a terminal ps displayes only one process.
Started via ./FOOBAR
ap#VBU:/media/sf_SharedDir$ ps -ef | grep -v grep | grep FOOBAR
ap 4115 3463 0 16:34 pts/5 00:00:00 /bin/bash ./FOOBAR
EDIT: shell=True is causing this issue (if it is one). But how would I fix that if I required shell to be True to run bash commands?

There is nothing wrong, what you see is perfectly normal. There is no "fix".
Each of your processes has a distinct function. The top-level process is running the python interpreter.
The second process, /bin/sh -c /media/sf_SharedDir/FOOBAR' is the shell that interprets the cmd line (because you want | or * or $HOME to be interpreted, you specified shell=True).
The third process, /bin/sh /media/sf_SharedDir/FOOBAR is the FOOBAR cmd. The /bin/sh comes from the #! line inside your FOOBAR program. If it were a C program, you'd just see /media/sf_SharedDir/FOOBAR here. If it were a python program, you'd see /usr/bin/python/media/sf_SharedDir/FOOBAR.
If you are really bothered by the second process, you could modify your python program like so:
command = "exec /media/sf_SharedDir/FOOBAR"
subprocess.Popen(command, shell=True)

Changing Process Name using Shell for nagios monitoring with check_procs

I have a python script to start a process which I want to monitor using Nagios. When I run that script and perform ps -ef on my ubuntu EC2 instance, it shows process as python <filename>.py --arguments. For Nagios to monitor that process using check_procs, we need to supply process name. Here process name becomes 'python'.
/usr/lib/nagios/plugins/check_procs -C python
It returns the output that one python process is running. This is fine when I'm running one python process. But If I'm running multiple python scripts and monitor only few, then I have to give that particular process name. If in the above command, I give python script name, it throws an error. So I want to mask whole python <filename>.py --arguments to some other name so that while performing check_procs, I can give that new name.
If anyone have any idea, please let me know. I have checked other stackoverflow questions which suggest changing python process name using setproctitle but I want to perform it using shell.
Regards,
Sanket

You can use the check_procs command to look at arguments, which includes the module name. The following command will let you know if the python module 'module.py' is running.
/usr/lib/nagios/plugins/check_procs -c 1:1 -a module.py -C python
The -c argument lets you set the critical range. 1:1 will trigger a critical status if there is more or less than 1 process that matches running.
The -a argument will filter based on processes that contain the args 'module.py' (change it to the name of the module you want to monitor)
The -C argument will make sure that the process is a python process
If you need help figuring out how to create the service definition, I had to figure that out too. Just let me know.
REFERENCE:
check_procs plugin manpage
http://nagiosplugins.org/man/check_procs

You can't change the process name from pure Python, although you can use a wrapper (for example, written in C) to do so.
However, what you should do instead is making your program a daemon, and using a pidfile. Have a look at the python Daemon API and its implementation python-daemon.

check_procs already handles this situation.
check_procs can tell the difference between scripts launched as an argument to the interpreter vs jobs run directly a hashbang interpreter. Even though both of these look the same in the ps output!! The latter case will not be listed in check_procs -C python!
If you run your scripts explicitly via python: python <filename.py>, then you can monitor them with the check_procs -C python -a filename.py.
If you put #!/usr/bin/python in your scripts and run them as ./filename.py, then you can monitor with check_procs -C filename.py.
Example command line session showing this behavior:
#make test.py directly executable. See code below
$ chmod a+x test.py
#launch via python explicitly:
$ /usr/bin/python ./test.py &
[1] 27094
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 1 process with command name 'python'
PROCS OK: 0 processes with command name 'test.py'
PROCS OK: 1 process with args 'test.py'
#launch via python implicitly
$ ./test.py &
[2] 27134
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 1 process with command name 'python'
PROCS OK: 1 process with command name 'test.py'
PROCS OK: 2 processes with args 'test.py'
#PS 'COMMAND' output looks the same
$ ps 27094 27134
PID TTY STAT TIME COMMAND
27094 pts/6 S 0:00 /usr/bin/python ./test.py
27134 pts/6 S 0:00 /usr/bin/python ./test.py
#kill the explicit test
$ kill 27094
[1] - terminated /usr/bin/python ./test.py
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 0 processes with command name 'python'
PROCS OK: 1 process with command name 'test.py'
PROCS OK: 1 process with args 'test.py'
#kill the implicit test
$ kill 27134
[2] + terminated ./test.py
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 0 processes with command name 'python'
PROCS OK: 0 processes with command name 'test.py'
PROCS OK: 0 processes with args 'test.py'
test.py is a python script that sleeps for 2 minutes. It is chmod +x and has a hashbang #! line invoking /usr/bin/python.
#!/usr/bin/python
import time
time.sleep(120)

Create a pid file and use that file for the process lookup with nagios.

I'm not saying this is the best solution (it wouldn't scale well at all), but you can create a symbolic link to the python command and execute your script using this link. e.g.
ln -s `which python` ~/mypython
~/mypython myscript.py
Scripts launched using the link should show up as mypython in ps.

You can use subprocess.Popen to change the executable name, but you'd have to use a wrapper script (or some weird fork magic). The following code causes ps to list the executable as kwyjibo /tmp/test.py instead of /usr/bin/python /tmp/test.py:
import subprocess
p = subprocess.Popen(['kwyjibo', '/tmp/test.py'], executable='/usr/bin/python')