Why is it that the subprocess pid (Popen.pid) has different value from that the ps command returns?
I've noticed this when ps called both from inside python (with subprocess.call()) and from another terminal.
Here's a simple python file to test:
#!/usr/bin/python3
'''
Test subprocess termination
'''
import subprocess
command = 'cat'
#keep pipes so that cat doesn't complain
proc = subprocess.Popen(command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
shell=True)
print('pid = %d' % proc.pid)
subprocess.call("ps -A | grep -w %s" % command,
shell=True)
proc.terminate()
proc.wait() # make sure its dead before exiting pytyhon
Usually the pid reported by ps is 1 or 2 more than that reported by Popen.pid.
Because the command is run with shell=True, the pid returned by subprocess is that of the shell process used to run the command.
Related
We're having an issue kicking off a subprocess.Popen within an airflow operator. We're using the following code to kick off sqlcl:
import subprocess
cmd = '/usr/local/bin/sqlcl -V'
p = subprocess.Popen(
cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
executable='/bin/bash')
for line in iter(p.stdout.readline, ''):
self.log.info('%s', line)
p.wait()
# we have also tried p.communicate() and p.poll() here
The snippet above works when run from ipython but hangs with no output when run from within airflow. Any suggestions?
I try to measure CPU usage via top and python:
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
proc = subprocess.Popen("top -c -b -n 1 | grep /usr/local/bin/wineserver | grep -v grep | awk '{print $9}'", shell=True, stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
print len(stdout)
print len(stderr)
Output:
0
0
If I run the cmd via shell I get:
54
It seems the piping is the issue but I am not sure.
Solution:
os.system("top -c -b -n 1 | grep /usr/local/bin/wineserver | grep -v grep | awk '{print $9}' > top")
stdout = open("top").read().strip("\n")
When run interactively, top limits its display to your screen width. When run through Popen with stdout=PIPE, top is not running in a terminal and reverts to its default column width. This can be changed with an environment variable.
You could ditch the shell completely and process with python:
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
import os
myenv = os.environ.copy()
myenv["COLUMNS"] = "512"
proc = subprocess.Popen(["top", "-c", "-b", "-n", "1"], stdout=PIPE, stderr=PIPE, env=myenv)
for line in proc.stdout:
columns = print line.strip().split()
if columns[-1] == '/usr/local/bin/wineserver':
print columns
proc.wait()
Alternately, you can get CPU information through ps and use its filtering and output format specifiers to just grab the information you want. Below I use filters to display CPU and command line for "wineserver" only.
#!/usr/bin/python
import subprocess
from subprocess import PIPE, Popen
proc = subprocess.Popen(["ps", "-ww", "--no-headers", "-C", "wineserver", "-o", "pcpu args"],
stdout=PIPE, stderr=PIPE, env=myenv)
for line in proc.stdout:
pcpu, command = print line.strip().split(" ", 1)
print pcpu, command
proc.wait()
I'm trying to write the return of terminal in a file, called debug.log.
I would like also to get the pid to kill the process, for this moment the kill is working.
But the debug.log is empty
cmd1 = "cvlc rtp://232.0.2.183:8200 --sout file/mkv:/media/file.mkv"
with open("/home/user/.cache/debug.log", 'w') as out:
proc = subprocess.Popen(cmd1, stdout=out, shell=True, preexec_fn=os.setsid)
pid = proc.pid
with open("/home/user/.cache/pid.log", 'w') as f:
f.write(str(pid))
f.close()
Edit: I'm using this method to kill the process
and this method (from here) to write the log:
###########kill the process############
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(pro.pid, signal.SIGTERM) # Send the signal to all the process groups
######### write the log #############
import subprocess
cmd = ['ls', '-l'] # example of command
with open('output.txt', 'w') as out:
return_code = subprocess.call(cmd, stdout=out)
In fact, I would like to mix both of examples.
Thanks
You need to redirect the 'stderr' (not 'stdout') to 'out'.
proc = subprocess.Popen(cmd1, stderr=out, shell=True, preexec_fn=os.setsid)
I'm trying to run a program from python and print the output based on its exit status. The code below is outputting directly to my vim screen (messing it) instead of opening a shell with the output.
python << EOF
import subprocess
import vim
cmd = "BAD_COMMAND"
p = subprocess.Popen(cmd, shell=True)
retcode = p.poll()
if retcode > 0:
output_of_error = p.communicate()[0]
vim.command("!echo show errors here")
EOF
You want to redirect your subprocess to a pipe:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
communicate() will then return a 2-tuple containing the contents of stdout and stderr of the process.
I want to open a process and run two commands in the same process. I have :
cmd1 = 'source /usr/local/../..'
cmd2 = 'ls -l'
final = Popen(cmd2, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
stdout, nothing = final.communicate()
log = open('log', 'w')
log.write(stdout)
log.close()
If I use popen two times, these two commands will be executed in different processes. But I want them to run in the same shell.
The commands will always be two (unix) processes, but you can start them from one call to Popen and the same shell by using:
from subprocess import Popen, PIPE, STDOUT
cmd1 = 'echo "hello world"'
cmd2 = 'ls -l'
final = Popen("{}; {}".format(cmd1, cmd2), shell=True, stdin=PIPE,
stdout=PIPE, stderr=STDOUT, close_fds=True)
stdout, nothing = final.communicate()
log = open('log', 'w')
log.write(stdout)
log.close()
After running the program the file 'log' contains:
hello world
total 4
-rw-rw-r-- 1 anthon users 303 2012-05-15 09:44 test.py