We Keep Coding

Unexpected output of bash 'ps -p $$' command returned by 'subprocess.run()'

I am running Linux Mint 18.1 and Python 3.9. To find out which shell is executing shell commands I have started to use ps -p $$ which is expected to return the info about the shell as value of CMD.
When using subprocess.run() in Python not specifying the shell or specifying the shell as executable='sh' the CMD value is sh for both passed commands (see code below), but when I specify executable='bash' I get different results (ps and bash).
The GNOME-Terminal which is running bash prints bash as CMD value when running ps -p $$
What is the reason for the different values of CMD being ps printed by the code below in case of ps -p $$ and bash in case of ps -p $$;echo $0?
from subprocess import run
print(run('ps -p $$ ', capture_output=True, shell=True,
encoding='utf-8', executable='bash').stdout)
print(run('ps -p $$;echo $0', capture_output=True, shell=True,
encoding='utf-8', executable='bash').stdout)
which prints:
PID TTY TIME CMD
22928 ? 00:00:00 ps
PID TTY TIME CMD
22929 ? 00:00:00 bash
bash
UPDATE to respond to the given answer and comments:
#Charles Duffy : YES, without Python involved when running the commands in GNOME Terminal using bash -c I get the same behavior as if run with subprocess.run() in Python, but ... I don't get it when running without the preliminary bash -c.
#Barmar : Trying to check out the explanation in your answer I have introduced a third command echo $0;ps -p $$ to see if the last command in the sequence will give a CMD value of ps. Below the result of a terminal session:
$ bash -c 'ps -p $$'
PID TTY TIME CMD
23386 pts/1 00:00:00 ps
$ bash -c 'ps -p $$; echo $0'
PID TTY TIME CMD
23388 pts/1 00:00:00 bash
bash
$ bash -c 'echo $0;ps -p $$'
bash
PID TTY TIME CMD
23395 pts/1 00:00:00 bash
What have I misunderstood in your answer expecting from the third command to give ps as CMD value?

This is a bash optimization. If the command line is just a single command, it's is implemented by simply calling execv() rather than forking a child to execute the command. This replaces the shell process with the ps program, keeping the same PID. It's as if you executed.
print(run('exec ps -p $$', ...))
You don't see it in the second attempt because ps is not the last command in the sequence. It has to fork a child process, while the shell keeps running to wait for it to exit and execute the following commands.

Script works differently when ran from the terminal and ran from Python

I have a short bash script foo.sh
#!/bin/bash
cat /dev/urandom | tr -dc 'a-z1-9' | fold -w 4 | head -n 1
When I run it directly from the shell, it runs fine, exiting when it is done
$ ./foo.sh
m1un
$
but when I run it from Python
$ python -c "import subprocess; subprocess.call(['./foo.sh'])"
ygs9
it outputs the line but then just hangs forever. What is causing this discrepancy?

Adding the trap -p command to the bash script, stopping the hung python process and running ps shows what's going on:
$ cat foo.sh
#!/bin/bash
trap -p
cat /dev/urandom | tr -dc 'a-z1-9' | fold -w 4 | head -n 1
$ python -c "import subprocess; subprocess.call(['./foo.sh'])"
trap -- '' SIGPIPE
trap -- '' SIGXFSZ
ko5o
^Z
[1]+ Stopped python -c "import subprocess; subprocess.call(['./foo.sh'])"
$ ps -H -o comm
COMMAND
bash
python
foo.sh
cat
tr
fold
ps
Thus, subprocess.call() executes the command with the SIGPIPE signal masked. When head does its job and exits, the remaining processes do not receive the broken pipe signal and do not terminate.
Having the explanation of the problem at hand, it was easy to find the bug in the python bugtracker, which turned out to be issue#1652.

The problem with Python 2 handling SIGPIPE in a non-standard way (i.e., being ignored) is already coined in Leon's answer, and the fix is given in the link: set SIGPIPE to default (SIG_DFL) with, e.g.,
import signal
signal.signal(signal.SIGPIPE,signal.SIG_DFL)
You can try to unset SIGPIPE from within your script with, e.g.,
#!/bin/bash
trap SIGPIPE # reset SIGPIPE
cat /dev/urandom | tr -dc 'a-z1-9' | fold -w 4 | head -n 1
but, unfortunately, it doesn't work, as per the Bash reference manual
Signals ignored upon entry to the shell cannot be trapped or reset.
A final comment: you have a useless use of cat here; it's better to write your script as:
#!/bin/bash
tr -dc 'a-z1-9' < /dev/urandom | fold -w 4 | head -n 1
Yet, since you're using Bash, you might as well use the read builtin as follows (this will advantageously replace fold and head):
#!/bin/bash
read -n4 a < <(tr -dc 'a-z1-9' < /dev/urandom)
printf '%s\n' "$a"
It turns out that with this version, you'll have a clear idea of what's going on (and the script will not hang):
$ python -c "import subprocess; subprocess.call(['./foo'])"
hcwh
tr: write error: Broken pipe
tr: write error
$
$ # script didn't hang
(Of course, it works well with no errors with Python3). And telling Python to use the default signal for SIGPIPE works well too:
$ python -c "import signal; import subprocess; signal.signal(signal.SIGPIPE,signal.SIG_DFL); subprocess.call(['./foo'])"
jc1p
$
(and also works with Python3).

how to integrate "at" command in python

I need to integrate " echo /bin/meteo | at 23:00 today " in to a python script.
In the python script the command "at 23:00 today" should call the bash script /bin/meteo
I did install plumbum and intergrated this in my python scrip.
from plumbum.cmd import echo, grep
Unfortunately I have no clue how to proceed from here.
I tryed:
#!/usr/bin/python2.7
if pfd.input_pins[0].value ==0:
cmd = "echo /bin/meteo | at 06:36 today"
subprocess.Popen(cmd, shell=True)
but the lights in /bin/meteo are randomly swiching on and off (not blinking as they should)
They do it from 06:36 until 06:37 and not only 5 times.
/bin/meteo:
#!/bin/bash -x
for i in {1..5}; do #blink 5x
echo -n -e "\x37\x00\x55" | nc -u -q 1 192.168.0.6 8899 #Zone 3 on
sleep 0.1
echo -n -e "\x3A\x00\x55" | nc -u -q 1 192.168.0.6 8899 #Zone 3 off
done
sleep 0.1
exit

subprocess.Popen will run the command:
import subprocess
cmd = "echo /bin/meteo | at 23:00 today "
subprocess.Popen(cmd, shell=True)
Execute a child program in a new process. On Unix, the class uses os.execvp()-like behavior to execute the child program. On Windows, the class uses the Windows CreateProcess() function. The arguments to Popen are as follows.
args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence.
It is not totally clear what you want but you can run any commands like:
In [9]: cmd = "date"
In [10]: subprocess.call(cmd, shell=True)
Sun Jul 6 22:30:47 IST 2014
Or using sudo:
import subprocess
cmd = "sudo which python"
my_pass="xxxx"
subprocess.call('echo {} | sudo -S {}'.format(my_pass,cmd), shell=True)
In [29]: subprocess.call('echo {} | sudo -S {}'.format(my_pass,cmd), shell=True)
/usr/local/bin/python
Out[29]: 0

With Python 3.4, it's easy to call a command and exchange input/output in a bulk:
subprocess.check_output(["at", "23:00", "today"], input="/bin/meteo")
Therefore in this very case, shell=True shouldn't be needed as we just call the at command with arguments and give it the script on input.
With older versions of python, this needs to be rewritten as:
process = subprocess.Popen(["at", "23:00", "today"])
process.communicate(input="/bin/meteo")
With the plumbum module, you could instead use:
from plumbum.cmd import at, echo
(echo["/bin/meteo"] | at["23:30", "today"])()
But I don't believe that it's very useful.

Kill python interpeter in linux from the terminal

I want to kill python interpeter - The intention is that all the python files that are running in this moment will stop (without any informantion about this files).
obviously the processes should be closed.
Any idea as delete files in python or destroy the interpeter is ok :D (I am working with virtual machine).
I need it from the terminal because i write c code and i use linux commands...
Hope for help

pkill -9 python
should kill any running python process.

There's a rather crude way of doing this, but be careful because first, this relies on python interpreter process identifying themselves as python, and second, it has the concomitant effect of also killing any other processes identified by that name.
In short, you can kill all python interpreters by typing this into your shell (make sure you read the caveats above!):
ps aux | grep python | grep -v "grep python" | awk '{print $2}' | xargs kill -9
To break this down, this is how it works. The first bit, ps aux | grep python | grep -v "grep python", gets the list of all processes calling themselves python, with the grep -v making sure that the grep command you just ran isn't also included in the output. Next, we use awk to get the second column of the output, which has the process ID's. Finally, these processes are all (rather unceremoniously) killed by supplying each of them with kill -9.

pkill with script path
pkill -9 -f path/to/my_script.py
is a short and selective method that is more likely to only kill the interpreter running a given script.
See also: https://unix.stackexchange.com/questions/31107/linux-kill-process-based-on-arguments

You can try the killall command:
killall python

pgrep -f <your process name> | xargs kill -9
This will kill the your process service.
In my case it is
pgrep -f python | xargs kill -9

pgrep -f youAppFile.py | xargs kill -9
pgrep returns the PID of the specific file will only kill the specific application.

If you want to show the name of processes and kill them by the command of the kill, I recommended using this script to kill all python3 running process and set your ram memory free :
ps auxww | grep 'python3' | awk '{print $2}' | xargs kill -9

to kill python script while using ubuntu 20.04.2 intead of Ctrl + C just push together
Ctrl + D

I have seen the pkill command as the top answer. While that is all great, I still try to tread carefully (since, I might be risking my machine whilst killing processes) and follow the below approach:
First list all the python processes using:
$ ps -ef | grep python
Just to have a look at what root user processes were running beforehand and to cross-check later, if they are still running (after I'm done! :D)
then using pgrep as :
$ pgrep -u <username> python -d ' ' #this gets me all the python processes running for user username
# eg output:
11265 11457 11722 11723 11724 11725
And finally, I kill these processes by using the kill command after cross-checking with the output of ps -ef| ...
kill -9 PID1 PID2 PID3 ...
# example
kill -9 11265 11457 11722 11723 11724 11725
Also, we can cross check the root PIDs by using :
pgrep -u root python -d ' '
and verifying with the output from ps -ef| ...

Changing Process Name using Shell for nagios monitoring with check_procs

I have a python script to start a process which I want to monitor using Nagios. When I run that script and perform ps -ef on my ubuntu EC2 instance, it shows process as python <filename>.py --arguments. For Nagios to monitor that process using check_procs, we need to supply process name. Here process name becomes 'python'.
/usr/lib/nagios/plugins/check_procs -C python
It returns the output that one python process is running. This is fine when I'm running one python process. But If I'm running multiple python scripts and monitor only few, then I have to give that particular process name. If in the above command, I give python script name, it throws an error. So I want to mask whole python <filename>.py --arguments to some other name so that while performing check_procs, I can give that new name.
If anyone have any idea, please let me know. I have checked other stackoverflow questions which suggest changing python process name using setproctitle but I want to perform it using shell.
Regards,
Sanket

You can use the check_procs command to look at arguments, which includes the module name. The following command will let you know if the python module 'module.py' is running.
/usr/lib/nagios/plugins/check_procs -c 1:1 -a module.py -C python
The -c argument lets you set the critical range. 1:1 will trigger a critical status if there is more or less than 1 process that matches running.
The -a argument will filter based on processes that contain the args 'module.py' (change it to the name of the module you want to monitor)
The -C argument will make sure that the process is a python process
If you need help figuring out how to create the service definition, I had to figure that out too. Just let me know.
REFERENCE:
check_procs plugin manpage
http://nagiosplugins.org/man/check_procs

You can't change the process name from pure Python, although you can use a wrapper (for example, written in C) to do so.
However, what you should do instead is making your program a daemon, and using a pidfile. Have a look at the python Daemon API and its implementation python-daemon.

check_procs already handles this situation.
check_procs can tell the difference between scripts launched as an argument to the interpreter vs jobs run directly a hashbang interpreter. Even though both of these look the same in the ps output!! The latter case will not be listed in check_procs -C python!
If you run your scripts explicitly via python: python <filename.py>, then you can monitor them with the check_procs -C python -a filename.py.
If you put #!/usr/bin/python in your scripts and run them as ./filename.py, then you can monitor with check_procs -C filename.py.
Example command line session showing this behavior:
#make test.py directly executable. See code below
$ chmod a+x test.py
#launch via python explicitly:
$ /usr/bin/python ./test.py &
[1] 27094
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 1 process with command name 'python'
PROCS OK: 0 processes with command name 'test.py'
PROCS OK: 1 process with args 'test.py'
#launch via python implicitly
$ ./test.py &
[2] 27134
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 1 process with command name 'python'
PROCS OK: 1 process with command name 'test.py'
PROCS OK: 2 processes with args 'test.py'
#PS 'COMMAND' output looks the same
$ ps 27094 27134
PID TTY STAT TIME COMMAND
27094 pts/6 S 0:00 /usr/bin/python ./test.py
27134 pts/6 S 0:00 /usr/bin/python ./test.py
#kill the explicit test
$ kill 27094
[1] - terminated /usr/bin/python ./test.py
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 0 processes with command name 'python'
PROCS OK: 1 process with command name 'test.py'
PROCS OK: 1 process with args 'test.py'
#kill the implicit test
$ kill 27134
[2] + terminated ./test.py
$ check_procs -C python && check_procs -C test.py && check_procs -a test.py
PROCS OK: 0 processes with command name 'python'
PROCS OK: 0 processes with command name 'test.py'
PROCS OK: 0 processes with args 'test.py'
test.py is a python script that sleeps for 2 minutes. It is chmod +x and has a hashbang #! line invoking /usr/bin/python.
#!/usr/bin/python
import time
time.sleep(120)

Create a pid file and use that file for the process lookup with nagios.

I'm not saying this is the best solution (it wouldn't scale well at all), but you can create a symbolic link to the python command and execute your script using this link. e.g.
ln -s `which python` ~/mypython
~/mypython myscript.py
Scripts launched using the link should show up as mypython in ps.

You can use subprocess.Popen to change the executable name, but you'd have to use a wrapper script (or some weird fork magic). The following code causes ps to list the executable as kwyjibo /tmp/test.py instead of /usr/bin/python /tmp/test.py:
import subprocess
p = subprocess.Popen(['kwyjibo', '/tmp/test.py'], executable='/usr/bin/python')